Recursive Language Models: Overcoming Context Rot and Enabling Complex Reasoning in AI Agents

The Core Challenge: Context Length vs. Task Complexity

• Context length isn’t enough – large documents (legal contracts, codebases) contain many internal references that create high task complexity.
• Context rot: performance drops not only when the token limit is reached but also as the reasoning task becomes more intricate.
• Lost‑in‑the‑middle problem: retrieving isolated "needles" from a haystack is solved, but multihop reasoning over inter‑linked clauses remains unsolved.

Why Traditional Approaches Fail

1. Naïve stuffing – dumping the entire document into an LLM leads to noise, high cost, and rapid degradation.
2. Summarization (e.g., Claude’s autocompact) – lossy; essential details for the task are often omitted, causing drift.
3. Retrieval‑Augmented Generation (RAG) – works for simple Q&A but cannot capture logical relationships needed for multihop reasoning; also depends heavily on fragile chunking strategies.

A Better Mental Model: Dependency Graphs

• Treat contracts or codebases as nodes (clauses, functions) linked by edges (references, calls) rather than linear text.
• This graph view mirrors how humans navigate cross‑referencing sections and enables systematic reasoning.

Introducing Recursive Language Models (RLM)

• Ripple = Read‑Evaluate‑Print‑Loop (REPL) executed inside a Python script.
• Read: fetch the current state of the data object (e.g., a contract variable).
• Evaluate: run any programmatic operation – slicing, keyword search, custom logic.
• Print: return results to the interpreter.
• Loop: repeat until the query is resolved.
• Recursion: the primary model can hand off sub‑tasks to a smaller model, creating a controlled, one‑layer deep recursion that mimics a hand‑off rather than an infinite loop.
• This structure reduces required context, enables flexible searching, and builds the dependency graph on‑the‑fly.

Experimental Results

• Tested on GPT‑5 and a 340‑billion‑parameter Quen model.
• RLMs achieved higher accuracy at lower cost compared to plain context stuffing, summarization, or RAG.
• They could reason over contexts orders of magnitude larger than the model’s native window without severe performance loss.

Limitations & Guardrails

1. Model size matters – small models showed noticeable degradation; high‑capacity models are still preferred.
2. Recursion safety – infinite loops can become expensive; the paper enforces a single‑layer recursion and synchronous execution.
3. When not to use RLM – for low‑complexity, short‑context tasks a single‑shot LLM often outperforms the ripple approach.
4. Operational complexity – monitoring, observability, and prompt engineering become more demanding.

Practical Implications

• Beyond software engineering: legal analysis, policy review, internal document synthesis, and any domain with large, self‑referencing data assets.
• Data provenance remains essential to mitigate hallucinations.
• The approach opens a path to AI agents that can reliably handle high‑complexity, large‑context workloads without prohibitive cost.

Key Takeaways

• Model complex documents as dependency graphs, not linear text.
• Use code execution + recursion (RLM/Ripple) to intelligently search and synthesize information, dramatically reducing context requirements.
• Apply this method selectively: ideal for large‑context, high‑complexity retrieval and synthesis tasks, but keep guardrails and model size considerations in mind.

Treating intricate documents as dependency graphs and leveraging a simple read‑evaluate‑print‑loop with controlled recursion lets AI agents overcome context rot, enabling accurate, cost‑effective reasoning over massive, self‑referencing data sets.