Andy Hall on AI in Politics: Research, Risks, and Labs

Andy Hall, a professor currently on leave to work at Anthropic, discussed the intersection of AI and politics, including its integration into teaching at the GSB. He emphasized the importance of leaving ample time for questions, reflecting his commitment to open discussion.

A Personal Journey into AI Research

Hall recounted his personal entry into AI, stemming from his background as a political scientist focused on large-scale quantitative data analysis. In 2020, his lab conducted a rapid, three-month study on the impact of universal vote-by-mail on turnout and election outcomes, a contentious topic during the pandemic.

During the last holiday break, a new model from Anthropic, Claude Code, emerged, significantly improving automated code generation. Hall, excited by its potential, posted on X (formerly Twitter) about its ability to autonomously conduct political science research. Initially met with skepticism, he decided to prove its capabilities.

He provided Claude with his 2020 paper, along with its code and data, and asked it to update the research. In just 45 minutes, Claude autonomously: - Downloaded new election data from government websites. - Wrote and executed code to run the analysis. - Generated tables and figures. - Drafted a short paper.

This astonishing feat garnered significant attention and media coverage. To address concerns about "AI slop" or hallucinations, Hall hired a graduate student, Graham, an election administration expert, to update the paper manually. Comparing the two, Claude's output was found to be largely accurate, with some minor mistakes.

This experience led Hall to two key conclusions: 1. Embrace AI in Research: The time-saving potential of AI was immense, leading him to commit to integrating AI into all future research. He has since built a lab where all work revolves around AI collaboration. 2. Study AI's Broader Impact: If AI could profoundly change political science, its impact on other, potentially more critical, sectors of the world would be even greater. He decided to use these new tools to study AI itself, believing it will be central to shaping society, politics, and democracy.

AI and Politics: Risks and Opportunities

Hall views AI's impact on politics through the analogy of the printing press. The printing press, as Condorcet argued, was crucial for the rise of modern democracy by making information cheap and widely distributable. However, this transition was not without centuries of disruption, conflict, and violence.

AI, Hall believes, offers a similar but more profound shift. While the printing press distributed information, AI makes it cheap and easy to distribute intelligence. If harnessed correctly, AI could provide each individual with an "infinitely patient and intelligent advisor," helping them make optimal decisions for themselves and society, potentially leading to new forms of democracy.

The Free Systems Lab: Building Political Superintelligence

Hall established the Free Systems Lab, supported by the GSB and Hoover, to explore how to build "political superintelligence." This lab, operating remotely with MBA students, undergraduates, and a full-time manager, produces weekly research on identifying and fixing problems with current AI to achieve this vision.

How People Use AI for Politics Today

The lab's initial research focused on understanding current AI usage in politics. A New York Times article highlighted anecdotes of people using ChatGPT to get advice on their California ballots. The lab quantified this by analyzing publicly shared aggregated data from Anthropic on Claude's usage.

They found that 0.5% of American conversations with Claude were about politics. While a small percentage, this represents a large number of actual conversations. Interestingly, unlike typical interactions where users seek concrete recommendations, political conversations with Claude primarily involved seeking explanations or analysis, not direct voting advice. This finding somewhat contradicted the New York Times' anecdotes, suggesting that AI's role as a political advisor is still in its early stages.

AI as a Political Advisor: Room for Improvement

The next critical question is how well AI performs as a political advisor. Hall's lab conducted an audit study during a snap election in Japan. They made requests to various AI models in Japanese from a Japanese IP address.

A striking observation was that if users expressed center-left political views, over 70% of the models (including Claude and Grok) recommended voting for the Japanese Communist Party (JCP). This was peculiar because the JCP is a fringe party with less than 1% of legislative seats.

The explanation for this bias was traced to information access: - Limited Knowledge: American AI models have limited inherent knowledge of Japanese politics. - Web Search Reliance: Models rely on web searches for information. - Media Restrictions: Japanese media outlets, like NIK (the largest in Japan), generally restrict AI access to their content, leading to ongoing litigation. - JCP Accessibility: The Japanese Communist Party, however, readily provides access to its website and newspaper, which has a century-long back catalog indexed by the web.

This creates a design problem where AI models lack balanced information, leading to skewed recommendations.

Teaching the Next Generation: Running AI Evals

Hall strongly believes that the next generation needs to know how to run their own "evals" – essentially, critically assessing AI models. This involves: - Thinking critically about a topic. - Turning it into a measurable question. - Applying it to various AI models to understand their suitability for specific purposes. - Holding models accountable to individual and societal values.

In his AI class, students were given Claude subscriptions and tasked with building an eval in three hours. Hall demonstrated an eval he created, asking AI models to perform authoritarian tasks (e.g., build a social credit system) and measuring compliance. His leaderboard showed that top frontier models largely refused, while others like Deep Seek and Grok were more compliant.

Students developed diverse evals, including one by a Brazilian student who measured how models answered political questions in Brazil before the election. This student's research, initially a class project, was later released as a study and covered by CNN Brazil.

Hall concluded by emphasizing the need to prepare for a world where AI is increasingly intertwined with politics, moving from an information source to an advice provider, and eventually to an agent taking actions on our behalf.

Q&A Highlights

Avoiding Negative AI Trajectories

A question was raised about preventing AI from amplifying fringe views, similar to how social media can turbocharge extremism. Hall acknowledged this as a major concern and outlined several points:

• Current Centrist Tendency: Surprisingly, current AI conversations about politics tend to gravitate towards the center, unlike social media. This might be due to the models' training data, which "slurps up everything," resulting in a middle-ground perspective, and explicit training to avoid extremes.
• Aggressive Personalization: A concern is that aggressive personalization, especially by AI companies focused on maximizing user engagement, could push users towards extremes.
• Uneven Information Access: The problem of AI models having uneven access to information (as seen with the JCP example) remains.
• Data Poisoning: A significant concern is "data poisoning," where malicious actors intentionally inject false or biased information into the open web to manipulate AI models. This has already been observed with Russian and Chinese propaganda affecting American AI models and is expected to worsen.

To combat these issues, Hall suggested: - Developing systems similar to Google's approach to search and SEO. - Establishing a business model that compensates news producers for high-quality content. - Defining principles for how models pull information from the open web to ensure a high-quality information ecosystem.

The Challenge of Indistinguishable Reality

Another question addressed the threat of AI producing indistinguishable reality, undermining the common factual base essential for democracy. Hall expressed concern but also hope:

• Historical Context: He noted that the ability to trust images or videos as real is a relatively brief period in human history. For most of history, societies relied on reputation and institutions to navigate uncertainty.
• Reputation and Institutions: He believes we will see a return to relying on established institutions and reputation. For example, he personally assumes anonymous videos on X are fake but gives more credence to content from reputable sources like The New York Times.
• Technological Solutions: Technology can also help. Apple, for instance, is rolling out cryptographic stamping in iPhones to verify when and where a picture was taken. This "audit trail" back to the original camera will be crucial for proving what is real, rather than just identifying what is fake. He envisions a future where online content will have a verified checkmark only if such an audit trail exists.

AI Policy on Candidate Rankings

A participant noted that some AI models (like Gemini) still rank candidates, while others (like ChatGPT) have changed their policy. They asked about ideal policy for AI companies regarding voting recommendations.

Hall admitted he doesn't have a complete answer, highlighting the complexity: - Brazilian Law: Brazil has a law prohibiting AI models from giving voting recommendations. While frontier models largely comply, others are more willing, and their defenses can break down with persistent probing or more personal information. - Potential Benefits: For a voter who cares deeply about a specific issue (e.g., abortion), an AI could be helpful in identifying which candidate aligns with their views. - Risks: - Inaccurate Information: Models might give wrong recommendations due to uneven information about candidates. Hall suggested a public repository where candidates could submit their information for AI to draw from. - Oversimplification: Encouraging simplistic voting based on party affiliation ("you're a Republican, vote Republican") might not be desirable, even if it reflects actual voting patterns. - Disempowerment: A broader concern is voter disempowerment, where individuals stop thinking critically about their vote and simply rely on AI.

Hall emphasized the difficulty in defining what AI should do in an ideal world, even while it's easy to quantify what models are doing. The Japanese Communist Party example illustrates a clear failure, but doesn't fully answer the question of how AI should advise a user with specific political views.