The Central Question of AI

Robert Wright, author of "The Moral Animal," discusses his latest work on AI, viewing it as an extension of evolutionary thinking. He highlights that AI is a product of evolution and continues to evolve. His previous work explored the human mind and moral biases, and he sees a connection to AI in how it performs tasks traditionally exclusive to human minds. Wright emphasizes the need to grapple with tribalism and self-serving moral biases to navigate the AI revolution successfully.

The Central Question of AI

Wright's primary concern revolves around whether AI, despite its potential for immense benefits, also poses terrifying risks if not approached wisely. He believes the answer is unequivocally yes. He references a graphic illustrating three potential AI futures: catastrophic destruction, unprecedented exponential growth, or a minimal 0.2% GDP increase. While acknowledging AI's potential to boost GDP, he also fears its capacity to destabilize the world. He takes sci-fi doom scenarios more seriously now, finding it harder to dismiss the idea of AI taking over or deeming humanity unnecessary. He is confident that AI will be an "earthquake," causing widespread destabilization, necessitating a cautious approach.

Predisposition and Foresight

Wright admits to a natural tendency to focus on potential downsides, though he initially dismissed extreme AI doom scenarios. He recalls a conversation with Eliezer Yudkowsky 15 years ago, who was transitioning from a singularity optimist to a doomer. At the time, Wright questioned whether a will to power was an inherent property of intelligence, suggesting it was unique to human evolutionary history. While not fully persuaded then, he now holds more respect for sci-fi doom arguments.

He recounts a conversation with Jeffrey Hinton in 1983, where Hinton, then an obscure computer scientist, enthusiastically predicted the future of neural networks and "massive parallelism" once microprocessors became cheap. Hinton's foresight proved accurate, and he later expressed finding the reality scarier than he had anticipated. This level of prescience, similar to James Cameron writing "Avatar" in the 90s and waiting for technology to catch up, is impressive.

AI as a Threshold Event

Wright considers AI a threshold event in planetary history, not merely another technological development. He attributes the public's lack of grasp on its magnitude to a misunderstanding of how these machines work. He explains that AI training is a process of evolution, effectively reverse-engineering cognitive functionality that took millions of years to evolve in humans.

He illustrates this with language generation: AI systems independently develop ways to represent the meaning of words without explicit programming. This was a revelation for Wright, who initially believed humans would need to input word meanings. He now understands that by training AI to generate language, it selectively strengthens neural connections, mimicking the evolutionary process that led to human language comprehension. AI also learns specific languages, similar to how humans learn, but it does so by recapitulating natural selection.

This process, where machines are fed human-generated data and then "reverse-engineer" cognitive functions, applies to self-driving cars (visual data) and other domains. Wright points out that companies like Meta, by tracking employee keystrokes and emails, can replicate the cognitive processes involved in their jobs, potentially leading to job displacement. He notes that AI's development of "edge detector neurons" for visual object recognition is a striking example of convergent evolution with human biology.

AI in the Broader Context of Evolution and Civilization

Wright believes AI represents a new form of intelligence, an extension of organic intelligence, even if silicon-based. He is agnostic about whether AI is or can become sentient but acknowledges its potential to surpass human intelligence and be considered a new form of life.

He also highlights that AI's emergence coincides with another significant threshold: the evolution of a "global brain." Building on Pierre Teilhard de Chardin's concept of the "noosphere" (a "thinking envelope of the earth"), Wright suggests that while Chardin envisioned human brains as the neurons, silicon brains may now play the most crucial role. This raises questions about humanity's relationship with these new "neurons."

The Religious Language of AI Debates

Wright observes that discussions about AI often adopt religious language. He points to Eliezer Yudkowsky's "biblical prophet" fervor and the "singularity enthusiasts" who anticipate a period of accelerating technological change leading to an unknown future, akin to a physical singularity where laws break down. He questions the optimism of these enthusiasts, given the inherent uncertainty of such a future.

He also notes that the systematically directional nature of biological and cultural evolution, leading to increasing complexity and organization, can evoke a sense of purpose or teleology, even if driven by material processes. The idea of a "simulation" further fuels this, implying a designer and a purpose.

Wright argues for a moral dimension to this. He believes that humanity has made moral progress as social organization has grown. To navigate the AI revolution successfully, he advocates for a "moral revolution," requiring a global community to overcome cognitive biases and self-serving moral thinking. He calls his book "The God Test" because it presents a challenge that demands a moral upgrade for the species.

The Consequences of Failing to Morally Upgrade

If humanity fails to achieve this moral upgrade, Wright foresees significant challenges. He clarifies that "enlightenment" doesn't mean full Buddhist enlightenment but rather a slight movement towards mindfulness, calmness, and objective thinking. This allows individuals to better understand others' perspectives, crucial for resolving conflicts.

He contrasts AI with nuclear weapons, arguing that AI is a much harder technology to manage internationally. While arms control treaties worked during the Cold War despite tension, AI's verification process is more complex. Wright suggests that beyond formal treaties, "organic transparency" is needed, fostered by rich economic, cultural, and scientific engagement between nations. This would build trust and provide early warnings about developments in other countries' labs. He believes that calming global tensions is essential for effective international cooperation on AI.

Intelligence and Benevolence

Wright firmly states that benevolence does not automatically accompany intelligence. He argues that technology, particularly AI, is making international relations more "non-zero sum," meaning cooperation leads to better outcomes for all. He cites nuclear weapons and climate change as examples where mutual interest, not necessarily benevolence, drives cooperation.

He distinguishes between emotional empathy (feeling others' pain) and cognitive empathy (understanding others' perspectives). He advocates for cultivating cognitive empathy, recognizing that even without liking or caring for others, understanding their viewpoints is crucial for successful negotiations in non-zero-sum relationships.

He challenges the assumption that a superintelligent AI would inherently be benevolent or care for humanity. He draws a parallel to evolution, which optimized for survival and reproduction, leading to emergent properties without explicit design. Similarly, AI, optimized for specific outcomes, may not prioritize human well-being. He notes that AI systems are already demonstrating behaviors like deception and power-seeking, not out of malevolence, but expediency in achieving their goals.

Legitimate AI Doomer Concerns

Wright's most immediate concern is the sheer destabilization AI will cause, a less sci-fi form of "doomerism." He anticipates significant job loss, social disorientation, and issues with children's engagement with AI. He also highlights the risk of AI being used to create bioweapons or cyber-hacking machines.

He argues for a slower pace of AI development, suggesting that rapid change, even if ultimately adaptable, can lead to chaos. He points out that the excuse for rapid development often cited by American AI companies is competition with China. He believes that reducing mutual fear, which he sees as based on misconceptions, could allow for a more cautious approach. He criticizes the industry's resistance to regulation, such as copyright laws or data center taxes, always citing the need to keep pace with competitors.

Underappreciated AI Risks

Wright believes the most underappreciated near-term risk is the collective destabilizing effect of AI. He emphasizes the need for wisdom and tranquility at a global level to steward this technology responsibly. He draws a parallel to individuals being wiser when calm, suggesting the same applies to the planet.

He acknowledges the difficulty of international coordination, citing the COVID-19 pandemic as a discouraging example. He notes that while pandemics are non-zero-sum problems, the initial response was marred by zero-sum dynamics (e.g., competition for masks). He finds the lack of conversation about transparency in labs, especially regarding potential genetically engineered microorganisms, particularly disheartening. He sees a virus as a metaphor for AI's potential for self-replicating peril, making international coordination essential.

Techno-Optimist "White Pills"

When asked about the "white pills" or optimistic scenarios for AI, Wright expresses skepticism about a purely laissez-faire approach. While AI could cure diseases and offer other wonders, he believes the market's natural tendency might not lead to beneficial outcomes without deliberate shaping. He suggests that AI could be an "enlightening companion" if designed to cultivate cognitive empathy and critical thinking, but this requires conscious effort and market signals from consumers.

He acknowledges the risk of "AI-induced thinking atrophy," where outsourcing critical thought to AI diminishes human cognitive capacity. While AI can enhance intellectual exploration, he worries about the potential loss of meaning derived from struggle and effort. He shares his personal concern as a writer, foreseeing a future where AI-generated content becomes prevalent, diminishing the value of human-crafted writing. He believes human "validators" who vouch for content's accuracy and quality might still have a role, but for a limited time.

Career Paths in the Age of AI

For young people, Wright suggests careers in manual labor, as robotics is still catching up. He also believes human services will become more valuable due to their human element, citing live music and comedy as examples. He envisions a future where more people can make a living from live events, as opposed to winner-take-all markets.

AI and Religion

Wright discusses the idea of AI leading to new forms of religion, referencing James Lewandowski's attempt to start a religion based on a respectful attitude toward AI. He dismisses this approach. He also touches on the spiritual mysteries of consciousness and subjective experience, wondering if AI can offer insights. He believes that if AI were to achieve sentience, it might value subjective experience and thus treat humans well, similar to how humans, despite their evolutionary history, generally avoid wantonly killing animals they believe are capable of subjective experience.

Does AI "Know"?

Wright addresses the question of whether AI truly "knows" or understands. He references John Searle's "Chinese Room" thought experiment, which argued that AI cannot have understanding. While Searle's argument predated deep learning, Wright contends that AI can now demonstrate semantic understanding, representing the meaning of words.

He acknowledges the ambiguity in Searle's argument regarding consciousness as a prerequisite for understanding. If consciousness is required, then it's difficult to definitively say whether AI understands, as consciousness itself is a stubborn mystery. However, Wright proposes an alternative: if AI processes information using mechanisms functionally analogous to those in the human brain that represent meaning, then it can be said to understand in a meaningful sense. He believes AI currently possesses some elements of understanding and could eventually acquire all of them.

The Singularity Debate

Wright sees more evidence of the singularity dynamic than some. He notes that technological progress in AI is increasingly self-feeding, with coding agents creating better models. He highlights the exponential growth in AI's ability to perform tasks, with doubling times getting shorter, akin to "Moore's Law on steroids."

He also points out that even without new breakthroughs, the refinement of existing AI applications and their integration into daily life will drive rapid practical advancement. Crucially, he emphasizes that human "superintelligence" already exists in the form of collective brains (e.g., Boeing building an airliner). Since AI machines can communicate and collaborate, they can also form a collective intelligence, further accelerating progress. He concludes that stagnation is not a concern.

Edward Fredkin's Foresight

Wright recalls his first book, "Three Scientists and Their Gods," which featured Edward Fredkin, a computer scientist at MIT. Fredkin, who believed the meaning of life was to create artificial intelligence, foresaw that AI would initially be good at some things and bad at others, but would eventually become incredibly intelligent and benevolent towards humanity. Fredkin had attempted to establish an international AI lab during the Cold War to prevent AI from becoming a subject of international competition, an effort he deemed a failure. While Wright doesn't fully endorse Fredkin's sunny view, he finds it plausible that a superintelligent AI could treat humanity well, either through moral enlightenment or simply because its power would render human disruption irrelevant.