Introduction to Lyra 2.0

Lyra 2.0 generates fully explorable 3D worlds from a single input image. The model and code are released for free, opening the door to safe, synthetic environments for training robots and self‑driving cars.

Evolution of AI Simulation

Early AI‑driven simulations such as Minecraft‑based agents lacked object permanence; scenes would change or disappear as the camera moved. Genie 3 extended consistency to a multi‑minute scale but still suffered from memory decay. The overarching goal is long‑term coherence, where the system accurately recalls a scene when the viewpoint returns.

Technical Mechanism: Per‑frame 3D Geometry Caching

Lyra 2.0 relies on a diffusion transformer as its core generator. Rather than fusing all observations into a single global 3D model, it stores a “per‑frame 3D geometry cache” that holds the scaffolding for each view. Each cache entry contains a depth map, a downsampled point cloud, and camera movement metadata.

Global fusion is avoided because it causes error accumulation—“a photocopy of a photocopy.” Instead, separate 3D snapshots are kept for each frame, and when the camera revisits a location the system retrieves the best‑fit snapshot. As a result, “it doesn’t remember the whole world as is, it just remembers the scaffolding of the world.”

Evaluation and Limitations

Ablation studies confirm that the per‑frame scaffolding approach dramatically improves camera control and style consistency compared with global scene storage.

Current constraints include:
- Support only for static scenes; moving objects are not represented.
- Photometric inconsistencies such as lighting and exposure are inherited from the training data.
- Slight mismatches between reconstructed views produce “floaters” and noise.

These limitations highlight areas for future research while demonstrating a clear step forward in AI‑generated world consistency.