Physical Intelligence Scales Robot Autonomy with Efficient RL

Physical Intelligence is a company focused on enabling any robot to perform any task in the real world. Over the past year, they have advanced robot capabilities from folding laundry to washing greasy pans, peeling carrots, making grilled cheese sandwiches, and slicing zucchini. The core challenge is developing general-purpose robots that are truly useful in real-world environments. This involves creating general-purpose models and effectively deploying them.

Bringing AI to the Real World

Historically, AI adoption in the real world began with applications like product recommendations and ad ranking. The introduction of deep learning significantly advanced these applications due to its ability to handle complex inputs and outputs. A major turning point was the launch of ChatGPT in 2022, which demonstrated the widespread utility of a general-purpose model, reaching a million users in five days. More recently, tools like GitHub Copilot have further showcased the utility of AI in practical settings.

Key takeaways from this progression include: - Increasing use of generalist models: AI is moving towards general-purpose models capable of performing diverse tasks. - Tolerance for mistakes: Many successful AI applications (e.g., recommendations) involve a human in the loop who can override or account for AI errors. This reduces the pressure for perfect performance.

Physical AI, however, operates differently. Robots directly interact with and affect the physical world, requiring them to make decisions autonomously. This demands a much higher degree of accuracy and fewer mistakes than current machine learning systems. Waymo's achievement of over a quarter-million weekly autonomous rides demonstrates the feasibility of reliable, autonomous physical AI.

Achieving Long-Term Autonomy in Robotics

For robots to be truly useful, they need to operate autonomously for extended periods. Consider a robot making espresso: - Precision and force: Operating a portafilter requires precise and forceful control. - Delicate handling: Smoothly handling cups with liquid without spilling is crucial. - Accurate timing: This is often overlooked but critical in physical tasks. - High reliability: The goal is over 90% reliability for practical use.

The traditional machine learning approach involves data collection, model training, and evaluation. Initial attempts rarely yield high reliability. Iterative refinement—collecting more data, improving label quality, focusing on edge cases—helps but is limited by human effort. The ideal solution is for the AI system itself to iterate, automatically identifying areas needing more data or supervision. This self-improvement process, akin to reinforcement learning, is key to achieving 99%+ reliability.

Scalable Reinforcement Learning for Robotics

Reinforcement learning (RL) algorithms like PPO and TRPO have scaled language models to achieve complex reasoning, often requiring millions of attempts. However, directly applying this to robotics is challenging: - Cost of physical attempts: One million one-minute robot trajectories would take 700 robot days, which is impractical due to hardware usage and real-world operation. - Inefficiencies in current RL: - Dead-end trajectories: Robots often spend time on unproductive paths. For example, trying to fold two stuck cardboard boxes instead of separating them. - Many attempts per prompt: Algorithms like PPO make multiple attempts (10-50) for a single prompt to estimate value, which is costly in physical environments.

To make RL more efficient for robotics: 1. Human intervention for recovery: Instead of letting robots pursue dead-end trajectories, a human can intervene to show the robot how to recover, providing useful data and preventing wasted time. 2. Amortized value estimation: Instead of estimating value for each prompt individually, a general-purpose value function can be trained across diverse robot experiences. This function learns what constitutes "good" or "bad" progress (e.g., unfolding a shirt is bad, making forward progress is good) and can generalize across different tasks (e.g., folding laundry and retrieving items from a fridge). This significantly reduces the number of attempts needed for learning.

This improved RL system involves: - Training a foundation model on diverse data. - Collecting experience with human intervention to prevent dead ends. - Training a general-purpose value function to estimate good/bad outcomes. - Using this value function to improve the model.

Real-World Applications and Reliability

This approach has been applied to various tasks: - Latte making: A robot collaborated with a human (robot makes espresso, human steams milk). The robot successfully handled precise tasks like portafilter insertion, espresso dispensing, and delicately transferring a full latte cup. This policy ran for 13 hours straight, demonstrating long-term reliability. - Chocolate factory workflow: A robot was trained to construct, label, and stack cardboard boxes, mirroring a real-world workflow at Dandelion Chocolate Factory. - Laundry folding: The algorithm enabled robots to fold unseen clothing items in new environments autonomously for extended periods.

Quantitative measurements show significant improvements: - Throughput: A 2x increase in throughput (success rate x speed) was observed from the RL post-training stage. - Success rate: Over 90% success rate was achieved for the espresso task.

These results demonstrate a scalable recipe for high reliability in complex robotic manipulation tasks, achieving long-term autonomy in real-world workflows. Further iterations and improvements are expected to enhance reliability and speed, as robots are still slower than humans and make mistakes.

The Role of Memory in Long-Term Autonomy

Most state-of-the-art robot foundation models lack memory or context, operating solely on current sensor observations. While sufficient for short, repetitive tasks, memory is critical for long, multi-step tasks to track progress.

The challenge with memory is computational cost: - Naively feeding 10 seconds of video (50 Hz, 4 cameras, 256 tokens/image) results in half a million tokens, which is computationally prohibitive in real-time. Even subsampling to 1 frame/second yields 10,000 tokens, still too expensive for current models.

A solution involves multi-timescale memory: - Short-term video memory: Approximately 10 seconds of video memory, computed efficiently. - Long-term textual memory: For minutes or hours of memory, video is not always needed. Instead, events are summarized in compressed text, which is then incorporated into the model.

This multi-timescale memory enables robots to perform complex, non-repetitive tasks lasting 10-15 minutes autonomously, such as cleaning a kitchen (wiping counters, drying, throwing away towels, putting away items, washing dishes).

Developing a General-Purpose Robot Model

To create a single general-purpose model, it's useful to contextualize robotics within the broader timeline of generalist AI: - 2012: Deep learning surpasses specialists: A deep learning system trained from scratch outperformed specialized methods on benchmarks like ImageNet. - 2014: Pre-training and fine-tuning: Models pre-trained on large datasets (e.g., ImageNet, BERT) became standard for fine-tuning on downstream tasks. - 2018: Out-of-the-box generalist models: Models like GPT-2 began to work effectively without extensive fine-tuning for many consumer applications. - 2021: Compositional generalization: Models like DALL-E showed the ability to combine disparate concepts (e.g., "avocado chair"), indicating conceptual understanding and data efficiency.

Robotics has historically lagged, with bespoke datasets and training from scratch being common even three years ago. The goal is to move robotics towards out-of-the-box generalist models with compositional generalization.

Goals for a General-Purpose Model:

1. Out-of-the-box performance: Moving from fine-tuned specialists (like BERT) to models that work immediately (like GPT). Current best robot performance still requires fine-tuning for specific tasks.
2. Compositional generalization: Inspired by DALL-E, this means the model can combine concepts it has learned independently (e.g., interacting with an object it hasn't seen before, or performing a task on a new robot platform). This implies conceptual understanding and data efficiency (not needing to see every combination in training data).

Recipe for a General-Purpose Model:

1. Large and diverse dataset:
   • Diverse robot demonstration data (including low-quality data).
   • Policy rollout data (robot attempts).
   • Videos of humans.
   • Web data.
2. Model with sufficient capacity:
   • A large model.
   • Detailed prompting: Crucial for handling heterogeneous data. The model receives:
     • Memory.
     • Overall instruction.
     • Subtask instruction (immediate next step).
     • Metadata (data quality, episode length).
     • Optional: Sub-goal image (what the scene should look like in a few seconds).

To deploy this model, a high-level policy predicts subtask instructions, and a world model generates sub-goal images.

PIO7 Model Capabilities

The PIO7 model is a single general-purpose model trained with these principles. It can: - Fold a collared shirt. - Perform precise assembly (inserting and drilling a screw into a robot arm). - Replace a trash bag.

Performance: The PIO7 model matches or outperforms fine-tuned specialist models (PIO6) across various tasks (e.g., coffee making, box building), demonstrating high performance out-of-the-box.

Compositional Generalization: 1. Object interaction: The robot successfully interacted with an air fryer (opening, putting in a sweet potato, closing), an appliance barely represented in its training data. 2. Task-robot platform generalization: A large industrial "barm" robot, which had no prior folding data, successfully folded clothes. The model generalized folding skills learned on a different robot platform to this new, physically distinct robot.

Importance of Ingredients: - Diverse data: Removing the most diverse data dramatically decreases performance on held-out tasks. - Detailed prompting (metadata): Prompting with metadata significantly improves performance. Crucially, it allows the model to leverage even low-quality data, whereas without metadata, adding low-quality data can degrade performance.

These results indicate that robotics is entering a "GPT and DALL-E like era," with single models achieving high performance and compositional generalization out-of-the-box.

Real-World Deployment and Future Outlook

Physical intelligence models are already being deployed by companies like Ultra and Weave for tasks such as laundry folding and warehouse packaging. These models are adaptable to diverse robot embodiments, including bimanual platforms, drones, surgical robots, and tractors, demonstrating their potential for real-world impact beyond research demos.

Q&A Highlights

• ChatGPT moment for robotics: Unlikely to be as sudden due to physical deployment constraints, but capabilities are rapidly approaching usefulness.
• Switching to generalist policies: Small teams should start with generalist policies (e.g., open-source PIO5) and fine-tune them immediately, unless severely constrained by hardware or connectivity.
• PhD in robotics: Offers unique opportunities to learn problem-solving under uncertainty and conduct frontier research, valuable for startups. Industry also offers diverse roles in engineering and research.
• Robotics equivalent of internet-scale data: Real-world robot operational data, including teleoperation and autonomous attempts. Human videos and web data are useful but not a substitute for robot-specific experience.
• Democratization of robotics models: While hardware and data costs are higher, open-sourcing of models and datasets is likely to continue, fostering an ecosystem, though perhaps not mirroring language models exactly.
• Model output: Current models output target joint positions, which are then achieved by a PD controller. This works well and is not a current bottleneck.
• Imagination in robots: The PIO7 model has some imagination (predicting future images), which helps but isn't strictly critical for impressive performance.
• Improving speed: Reinforcement learning and specific techniques (like RL token) have shown speed improvements, even surpassing human teleoperation. Future work focuses on faster data or faster-than-data learning.
• Most surprising task: A robot learning to use its left gripper to insert a pin into paper, despite only being trained with its right gripper, demonstrating emergent equivariance.
• Breaking into robotics from software engineering: Join a robotics company as a software engineer, or get hands-on experience with cheap robots and open-source models, then share your work.