Waymo’s Seven Lessons for Building Safe Physical AI at Scale

Waymo has been at the forefront of developing AI that operates in the physical world, specifically through its autonomous driving technology. This technology, known as the Waymo Driver, represents a mature application of physical AI, with the company having accumulated significant experience and learned crucial lessons over the years.

The Unnoticed Safety of Physical AI

A key characteristic of effective physical AI is its seamless and safe operation, often going unnoticed by users. For instance, during a Waymo ride, the autonomous system can react smoothly and safely to unexpected events, such as other drivers cutting in, without passengers even realizing an incident occurred. This "nothing happened" experience is a testament to the AI's ability to perform its task safely and efficiently.

The Waymo Driver currently facilitates approximately 500,000 trips and drives over 4 million fully autonomous miles weekly across 15 U.S. cities. This translates to over 300 years of an average American driver's annual mileage every week, all while maintaining a superhuman safety record.

Bridging the Gap Between Digital and Physical AI

Developing AI for the physical world presents unique challenges compared to digital AI. The common Silicon Valley mantra of "move fast and break things" is unacceptable when dealing with physical systems where errors can have severe consequences. Instead, the focus must be on "move fast and ship safely," requiring robust systems and safety-first AI models from day one.

Four main gaps differentiate physical AI from digital AI:

1. Cost of Error Gap: In digital AI, a mistake might cost a retry. In the physical world, the cost can be human lives, with no "undo" button.
2. Latency Gap: Digital assistants can take seconds or minutes to respond. A car at freeway speeds covers 100 feet in one second, meaning decisions must be made in milliseconds using on-board compute.
3. Data Gap: Digital AI benefits from the vast, pre-labeled human knowledge of the internet. There's no equivalent digitized resource for the physical world.
4. Validation Gap: Digital AI can often be shipped as "good enough," with users identifying edge cases. Physical AI, due to the high cost of errors, requires a very high level of safety and confidence before initial deployment.

Despite these challenges, real-world experience is invaluable for physical AI. Systems cannot be perfected in a lab and then deployed overnight. Therefore, clearly defining operating conditions and deployment parameters, along with a rigorous framework for responsible scaling, is crucial for earning trust from customers, communities, and regulators. Waymo views these gaps in the context of autonomous vehicles, but they are relevant to any non-trivial physical agent.

Seven Lessons Learned from Waymo's Journey

Waymo has distilled its experience into seven technical lessons for building and deploying physical AI:

Lesson 1: The Chasm Between Demo and Product

A working demo represents, at best, 1% of the effort required for a real product. Achieving the "many nines" of performance and reliability is where the true work lies. Waymo achieved its first 90% milestone in autonomous driving around 2010, handling various conditions. However, transitioning from a demo to a scalable service with no human behind the wheel took another 10 years, and then five more to scale to half a million trips per week.

The "exponential ladder of nines" dictates that each additional "nine" of reliability or performance requires approximately ten times more effort. A demo might need one "nine," an assist product a few, but a fully autonomous agent interacting with the public demands many. At scale, rare events become daily occurrences, making the "long tail" the primary problem space. Achieving higher "nines" requires fundamentally different approaches, not just more of the same.

Technological breakthroughs, while making demos easier, do not significantly reduce the difficulty of the "tail" problems. This leads to hype cycles where spectacular demos are abundant, but real products are scarce. The advice is to be honest about the product's demands and the number of "nines" required, rather than cutting corners.

Lesson 2: Architecture Dictated by "Nines"

The required level of performance and reliability ("nines") fundamentally dictates the architectural and technical approach. Technologies have performance-versus-effort curves that eventually flatten. A common pitfall is choosing a technology for its fast early ramp, only to hit a plateau before reaching the necessary performance.

For autonomous vehicles, this is evident in the debate over sensing modalities. While humans drive with eyes, aiming for superhuman autonomy requires robust sensing. Waymo employs multiple modalities:

• Cameras: High resolution and color, but passive and degrade in darkness/glare.
• Lidar: Direct 3D structure measurement.
• Radar: Excellent at penetrating adverse weather (fog, rain, snow) and measuring velocity via Doppler effect.

Lidar and radar are active sensors, performing well in darkness or blinding sunlight. These modalities are not backups but complementary, with information fused into a single, precise world view. Examples show lidar detecting pedestrians in dust storms or darkness where cameras see little. Redundancy is also crucial; a single obstruction (like a leaf) shouldn't disable the system.

Hardware costs will decrease over time. Waymo is on its sixth generation of hardware, consistently improving capability while reducing cost. Companies should design for future commoditization and be ready to upgrade.

Lesson 3: Riding Tech Waves Repeatedly

Technology evolves rapidly, necessitating continuous adaptation. Waymo rebuilds its driver around major AI breakthroughs, often pushing the state-of-the-art. They leveraged CNNs for computer vision (2013), then transformers for perception, behavior prediction, and decision-making (2017). Driving, with its social aspects and sequential nature, shares similarities with language modeling. Today, Waymo uses VLMs and frontier world models.

The challenge isn't just adopting new tech for performance gains, but integrating bleeding-edge research into production in a safety-critical environment without regressions or hindering scaling. The goal is to add capability while simultaneously reducing fragmentation and complexity. This ability to repeatedly integrate innovation is a crucial "muscle" for a company.

Advice: * When pursuing new tech, clearly define the path for its integration into the entire product if successful. Avoid dead ends. * Ask not only what new tech offers in capability, but also if it simplifies the stack and promotes unification over fragmentation. Aim for breakthrough performance and radical simplification.

Lesson 4: The Waymo Foundation Model

This philosophy led to the Waymo Foundation Model, a multimodal world-action language model:

• Multimodal: Processes camera, lidar, and radar inputs.
• World Model: Understands physics, dynamics, social, and semantic aspects of the world.
• Action Model: Understands the effects of the agent's actions and distinguishes good from bad.
• Language Aligned: Leverages general world knowledge from visual language models for rare semantic situations.

The architecture is an encoder-decoder model. The encoder compresses multimodal sensor inputs into an efficient representation for the generative decoder. It's an end-to-end model, allowing gradient backpropagation from the final task to early layers. It uses a "system one, system two" (think fast, think slow) architecture:

• Think Fast Path: Fuses raw data for split-second, safety-critical decisions (e.g., instant braking for a pedestrian). This is the "lizard brain" handling geometric tasks in milliseconds.
• Think Slow Path: Handles complex semantic and scene-level understanding (e.g., recognizing a car on fire and choosing a different route, even if the path is geometrically clear). This path can tolerate more latency for higher reasoning.
• Generate Component (Decoder): Understands and produces behavior, predicts other actors' behavior, and plans driving decisions.

This foundation model powers different generations of Waymo hardware and vehicle platforms (JLR I-PACE, Hyundai Ioniq) and will extend to trucking and personally owned vehicles. By centralizing complexity in this high-capacity foundation model, the on-car specialization layer remains lightweight, accelerating development.

Lesson 5: Structure-Augmented End-to-End Models

Richard Sutton's "bitter lesson" states that general methods leveraging massive compute and data always outperform methods relying on handcrafted knowledge. Waymo's foundation model approach aligns with this, as high-capacity models with ample data and compute yield better scaling laws.

The use of structure in models is critical. Structure that fights scale will lose, while structure that channels scale wins. While end-to-end models are easy to build and train for initial progress, achieving superhuman performance in safety-critical environments requires more. The key is whether structure boosts scale or limits it.

Using a toy example of a Go-playing robot, a simple intermediate representation (the 19x19 board state) captures the game's complete state, allowing for efficient scaling. In the physical world, while such a simple representation doesn't exist, inherent structure (laws of physics, rules of the road, predictable object behavior) can be leveraged.

Waymo's "structure-augmented end-to-end" approach combines learned embeddings with materialized structured representations, offering advantages:

• Validation at Inference Time: The model isn't a black box; powerful real-time correctness and safety validation layers can be built.
• Efficiency in Training and Evaluation: Intermediate structured representations allow for mixed training and evaluation, some at scale in the compact structured space, and some in the full end-to-end setup.
• Strong Verifiable Feedback Signals: Materialized structure provides powerful tools for evaluation, metrics, loss function crafting, and reinforcement learning.

The lesson is to bet on a maximally learned, minimally constrained system, intentionally leveraging structure to boost performance and scaling laws in both training and evaluation.

Lesson 6: The Critical Role of Simulation

Building and deploying a physical AI agent requires a large-scale, realistic, high-fidelity simulator. Training and evaluation can be open-loop (passively observing input-output pairs) or closed-loop (taking an action, observing its effect, updating the world view, and taking another action). The ability to evaluate counterfactuals in a closed-loop environment is vital for safety-critical agents.

A simulator is not just a tool; it's a complex AI model itself, understanding physics, semantics, traffic, and weather. Its quality must be high enough to confidently train and evaluate agents for safety-critical deployment. This means building a highly accurate generative world model.

Waymo has built "behavioral world models" for years, and now, with end-to-end models, "sensing world models" are also crucial for realistic sensor simulations. By leveraging structure-augmented representations, Waymo's simulation uses a behavioral world model in the structured space and a tightly coupled sensor world model for realistic sensor outputs.

Waymo's world model, leveraging Google DeepMind's Gen3, produces controllable and highly realistic scenarios for both behavior and sensing. This allows training and evaluation not only in previously encountered situations but also in purely synthetic, rare scenarios never seen in the real world (e.g., a car stopped on a freeway, a plane landing, an elephant loose, snow on the Golden Gate Bridge, or even a dinosaur). Closed-loop simulation is essential for evaluation and extremely valuable for training physical AI agents.

Lesson 7: Eval and Metrics as Strategic Moats

When dealing with such complexity, an entire ecosystem is needed, powered by a flywheel. This ecosystem comprises three AI agents:

1. The Agent: The Waymo Driver that drives the car.
2. The Simulator: The virtual playground for the agent to learn.
3. The Critic: Rigorously evaluates and judges the agent's performance, guiding improvements.

The fundamental reasoning and generative capabilities of all three are shared, based on the same foundation world model. This creates a powerful flywheel: real-world deployment generates data, which grounds and improves the simulator. The simulator generates harder edge cases for the critic, which in turn helps the agent learn, leading to smarter agents, more deployments, and more data, accelerating progress.

To guide this flywheel, metrics are paramount. "Eval and metrics" are the most important strategic moat. Build evaluation and metrics before building the technology or product. Without quantitatively defining "good enough," a product isn't being built, only a demo iterated upon. While model architectures are well-known, and data is important, without good metrics, progress is blind. Eval and metrics form the foundation that steers the entire tech stack.

For physical AI, model-level evaluation is insufficient. Validation must be much deeper and broader, covering every system component from the physical to the behavioral layer, on-board and off-board components, and all operational processes. Waymo's "safety and readiness framework" guides its development and deployment, considered one of its most important assets.

Trust is everything in the physical world, and eval and metrics are how that trust is earned. Trust is built gradually by relentlessly proving safety and functionality in the field, not by clever tech or flashy demos. Waymo openly publishes its safety data and research to build this trust. This earned trust becomes a business advantage, as models and algorithms can be replicated, but hundreds of millions of miles of fully autonomous operations, backed by evidence-grade evaluation and public audits, are much harder to replicate.

The Compounding Effect

These lessons are interconnected. The "nines" set the bar and guide technology choices. Intentional use of structure and the ability to ride tech waves help achieve the required "nines." The AI ecosystem (agent, simulator, critic), guided by eval and metrics, builds a powerful flywheel. This integrated playbook has enabled Waymo to achieve superhuman safety performance.

Waymo's latest safety data, based on over 220 million fully autonomous miles, shows the Waymo Driver is about 17 times better than human drivers in preventing crashes causing serious injury in its operating areas. This translates to preventing a serious injury every eight days, highlighting the profound impact of physical AI on human lives.

The opportunity for physical AI is massive, akin to where digital AI was years ago. With generative world models, advanced architectures, affordable compute and sensing, proven scaling laws, and real products operating at scale, the next decade of AI is poised to unfold in the physical world. For those building in this space, the advice is to focus on the mission and customers, as technology without purpose is merely a science project. The joy of making a difference in people's lives is the ultimate reward.