AI Harnesses: Self‑Improving Agents, Prime Agent, Open Jarvis, YC QM

The YC Harness Club hosted a "harness night" event, highlighting the growing importance and evolution of AI harnesses. The event featured presentations on Prime Agent, Open Jarvis, and QM, showcasing advancements in self-improving harnesses, on-device AI, and agentic systems for work.

The Evolution of AI Harnesses

The speaker began by addressing the historical belittling of "harnesses" or "prompt engineering" as subpar research, despite their significant impact on AI performance. He cited examples where harnesses led to an 18% performance bump and were crucial for systems like Arc AGI to function. The progress in AI agents, particularly in their ability to run for extended periods, is largely attributed to the development of harnesses.

Initially, the "static harness era" involved harnesses without self-improvement. However, the last six months have seen a shift towards "self-improving harnesses." The speaker referenced a plot from the CEO of Trajectory, illustrating that while models are becoming more intelligent (higher IQ), they are not effectively leveraging test-time experience. Harnesses aim to bridge this gap, allowing models to adapt quickly to new problems and distributions.

Arc AGI, for instance, demonstrated how quickly it adapts to new distributions. While Claude Opus achieved 30% on Arc AGI's private holdout set, simple harnesses, often dismissed as mere wrappers or scaffolding, boosted performance to 95%, with Nvidia's AVO reaching 100%.

The speaker shared his personal experience building a harness by accident while trying to create a user interface for Carpathy's auto-researcher. This harness allowed him to specify a purpose (e.g., "diffusion LM don't beat ARLM"), seed ideas, vary ensemble sizes, and define evaluation metrics. It incorporated various agents: - Scoping agent: Researches similar papers and GitHub repositories. - PI agent (Chris Ray): Oversees the research process. - Research agent (John South Khan): Conducts the research. - Council (speaker and Yaso): Provides feedback. - Author agent: Writes ablations and papers once the idea is frozen.

This system, accessible via a cockpit and email updates, has been publishing papers, initially of varying quality, but now consistently good. The speaker noted that he can give eight ideas to eight H100 nodes, and they return papers, demonstrating the power of scaffolding.

A Brief History of Harnesses

The speaker provided a non-chronological overview of harness development:

• Initial Harness (GPT-2, February 2019): A simple end-of-sequence loop with top-P sampling and an environment. No tool calling or skills.
• Context Innovation (July 2020): Introducing in-context examples to improve learning, as seen in few-shot learners.
• Chain of Thought: Smearing computational logic over more tokens to arrive at an answer, rather than directly predicting it.
• Tool Former and WebGPT: Introducing tools (JSON objects) that models can call, such as Python for calculations.
• MeGPT (Memory): Enabling models to read, write, update, and delete their own context, separating a "memory" chunk.
• Voyager (Skills): Chaining tools to achieve tasks and distilling these learned procedures into the system prompt as "skills."
• Intercode: Allowing models to output code, creating on-the-fly tools or skills.
• React, Self-Refine, Reflection: Introducing multi-agent systems where agents have different roles and can self-improve by reflecting on their outputs and getting feedback from internal evaluators or the environment.
• Multi-Agent Spawning: Agents can spawn sub-agents that persist in ripples, allowing for complex, coordinated tasks.
• RLM (Recursive Language Model): Recursively calling RLM queries to solve larger problems, with a main orchestrator agent.

This entire progression represents "harness v1" or "static harnesses," where the system prompt and harness itself are not improved.

Harness v1 Summary

A static harness typically involves: - An agent specification (system prompt). - Defined turns and tool call limits. - A list of tools, skills, and sub-agents. - A loop that compiles context, makes an LLM call, receives an action, and potentially invokes tools, appending results back to the context.

The Self-Improving Harness Era

The most exciting advancements are in harnesses that learn and improve themselves, either by learning the optimal system prompt or by modifying their own code.

• DSPY (Demonstrate, Search, Predict): Uses genetic programming to iterate and find optimal system prompts based on a small training set. It provides CRUD (Create, Read, Update, Delete) operations over the system prompt.
• Darwin Machines: Go a step further by allowing changes to the harness code itself. A meta-harness produces harnesses, creating an archive of agents (harness + system prompt). Agents are sampled, evaluated, and can modify their own harness code, leading to continuous improvement.
• Continual Harness: Adds extra memory classes (history) and incorporates Dagger-style online learning, allowing the weight file itself to be updated based on small amounts of newly learned examples.

Prime Agent: A Self-Improving RLM Harness

Seth, a researcher at Prime Intellect and author of Prime Agent, presented his work. He emphasized a first-principles approach to harness building.

LLM as a Sequential Processor

At its core, an LLM is a sequential processor with fixed weights, taking tokens in and outputting tokens. The harness acts as the layer between the LLM and the world, adding persistent state, tools, and compute.

Prime Agent Architecture

From a human perspective, Prime Agent offers: - An agents view providing an overview of all active agents and their summaries. - A root session that orchestrates sub-agents, leveraging them when useful. - Programmatically called sub-agents based on the Recursive Language Model (RLM) principle, operating within an IPython shell. - Messaging paradigms for coordination between agents. - Direct interaction with the environment (files, programs, H200 clusters). - A persistent daemon ensuring agents continue running in the background even if the laptop is closed. - Live CRUD operations on memory, skills, sub-agents, and the system prompt, inspired by continual harness.

Context Management as a Cache Hierarchy

Seth conceptualized context management as a cache hierarchy: - L1 (Model Weights): Fastest, most readily available information. - Active Input Context: Uses many tokens, includes in-context examples. - Compaction: A generalized tool for agents to summarize their context history, working around context length limitations. - L2 (Live REPL): An IPython shell where variables are saved in RAM, allowing agents to programmatically manipulate them and run programs, saving tokens. Sub-agents also save context by performing specific operations and reporting back. - Agentic Garbage Collection: Cleaning up variables and sub-agents in RAM to prevent crashes. - L3 (Dispatch State/File System): Main memory for reading and writing files. - Refinement: Updating and deleting skills, memories, and prompts stored on the system to manage hard drive space.

This framework allows for expressing and revising information over time.

Harness as a Von Neumann Computer

Seth drew an analogy: - A raw LLM is like a Turing machine (ticker tape, instructions, operations). - A harness is like a Von Neumann computer (read/write operations on external memory), making it more powerful.

A good harness should be highly expressible, allowing models to call compaction, run Python REPLs, programmatically create sub-agents, and access state with various feedback mechanisms. These are model-controlled expressibility features.

Persistent Subsessions and Continual Harness

Prime Agent manages sub-agents as persistent subsessions. A parent session can spin up RLM sub-agents, which run tasks, report back, and then become idle. The parent can reactivate them by sending messages, preserving context. Inactive sub-agents can be offloaded to save RAM.

Continual harness allows CRUD operations on the entire harness state, including trajectories, actions, outcomes, system prompts, skills (instructions/programs), memories (long-term storage), and sub-agent specifications. This enables reflection and refinement, allowing models to bootstrap themselves to higher performance.

Messaging and Long-Horizon Performance

Prime Agent allows messaging between any two agents (parent, children, siblings) for better coordination and management of complex tasks.

For long-horizon performance, Seth emphasized the importance of evaluating how long a model can sustain work without human intervention. He introduced the concept of a "practical plateau" – the point where only incremental gains are made with more test-time tokens.

Prime Agent Results

• ARC AGI: Prime Agent achieved 78% with GPT-Soul and 95.5% with Opus, significantly outperforming other harnesses and even the original Claude Opus results (30%). This was achieved by leveraging the REPL for coding, image analysis, and programmatic checks.
• Cost-Effectiveness: Prime Agent demonstrated a better cost-to-performance ratio compared to other harnesses like Air Agent, which incurred high costs for limited performance.
• Long-Horizon Evals: On tasks like Oolong and Emulator Bench (a Program Bench alternative), Prime Agent showed parity or slight improvements over other harnesses.
• Emulator Bench: Its REPL access allowed it to perform "out-of-experiment loop" designs, trying things in a more expressive way before submitting final solutions.
• GPU Kernels: Achieved comparable results across different models.
• Auto-Research (NanoGPT Speedrun): In a week-long experiment with 8 H200s, models like Deep 6v4, GLM 5.3, and Kim K3 performed "out-of-loop experiments," optimizing CPU-based parameter searches and hyperparameter tuning before running expensive H200 experiments.
• 7-Day Factorial Run: Used 633 agents and 23 million output tokens to make steady technological advancements, leveraging sub-agents for task division (research, build, gather resources) and refinement to avoid getting stuck.

Seth concluded by recommending agentic context management, swarms (RLMs), and standardized evaluations for anyone building their own harness.

Open Jarvis: Personal AI on Device

John Sadvalone, a PhD student at Stanford, presented Open Jarvis, a project focused on building a personal AI stack entirely on device.

The Problem with Cloud-Bound Personal AI

Most personal AI projects (e.g., OpenClaw, Hermes agent) rely on cloud LLMs, leading to: - High Cost: Thousands of dollars in API costs annually. - Lack of Privacy: Personal data sent to cloud LLMs with uncertain data handling. - Renting Intelligence: Dependence on external services rather than owning the intelligence. - High Energy Consumption: Orders of magnitude more energy than local LLMs.

The Rise of Local LLMs

Local LLMs are now "good enough," often only 6-12 months behind state-of-the-art frontier models. Examples like Quen 3.8 27B achieve performance comparable to older cloud models like Claude 4.6 Opus. Hardware accelerators (Apple Silicon, Nvidia) are also improving rapidly for personal use cases.

Open Jarvis Goal

The project aims to build the core of a personal AI stack (model inference, agent execution, memory, learning) entirely on device, while remaining competitive with cloud-only stacks, and preserving security, privacy, and quality.

Open Jarvis Primitives

Open Jarvis defines a simple set of primitives for any harness or personal AI stack: 1. User Interfaces: Desktop, continuous agents (cron jobs). 2. Agentic Logic: Composable reasoning, using different intelligences and tools. 3. Intelligence (LLM): Quen, GBDO, OSS, Gemma 3N. 4. Inference Engine: O Lama, Llama CBP, VLM, SG Lang, running on various hardware (Apple Silicon, Nvidia). 5. Tools and Memory: Run through a standard MCP protocol. 6. Learning: Prompt-based (Japa, DSPI) or weight-based (gpo, SFT, Laura) techniques for continuous improvement and personalization.

Open Jarvis aims to be plug-and-play with existing workflows, providing a first on-device LLM experience similar to early ChatGPT or Claude.

Cloud LLM Optimization for Local Stacks

A key innovation is using cloud LLMs to automatically optimize the entire local Open Jarvis stack. This leverages the diagnostic and proposal capabilities of cloud LLMs (e.g., Claude, ChatGPT) to create improved solutions for local LLMs, without incurring cloud costs during inference.

Open Jarvis Results

• On-Device Competitiveness: Even with today's on-device LLMs, Open Jarvis can rival cloud LLMs for personal AI, coding, and agentic tasks.
• Efficiency Gains: Achieved 800x lower cost and significant latency reduction.
• Cloud LLM Effectiveness: Any cloud LLM (Opus, GBD 5.6 Soul, Gemini, Kimmy, GLM) was useful for optimizing local configurations, allowing users to capture efficiency and performance gains for local inference.
• Optimization Cost: The Open Jarvis harness was cheaper to optimize than alternatives requiring more data or LLM calls, due to its simplified spec and primitives.

John believes that a majority of daily inference calls will shift to local devices and on-prem workstations in the near future due to improving accelerators and LLMs.

QM: YC's Open-Source Agent Harness for Work

Josh and Rean presented QM, YC's open-source agent harness designed for work.

QM Overview

QM provides an OpenClaw-like assistant to every YC employee, fully customizable and available in Slack or via a web UI. Each user has a personal context with sandboxed files and crons. It also supports multiplayer settings in Slack channels.

QM is used for: - Email triage, legal, and finance workflows. - Editing documents and pulling data from internal databases. - Spinning up live internal web apps. - Planning events.

Evolution of YC's Internal Agent Projects

1. January 2025 - "General Agent": A straightforward system prompt with tools in a loop. One-size-fits-all, but surprisingly good at data questions. Its capabilities grew with better underlying models. Integrated with Slack, crons, and more tools.
2. June 2025 - Code Agents: Engineers used cloud code and codecs in VMs, hooked up to Slack. The bot could describe and solve bugs, even for non-coders. A small loop observed failures and updated the agents.mmd to improve the system.
3. January 2026 - OpenClaw for Partners: YC partners, who are extremely busy, found OpenClaw valuable as it provided an agent with its own "computer," offering unprecedented customizability and acting as a personal assistant.
4. April 2026 - Hermes Fleet: YC provisioned 50+ Hermes agents in VMs to provide similar capabilities to all employees. However, this fleet was difficult to manage and configure, requiring constant maintenance.

QM's Design Philosophy: Unhobbling Agents

QM was built to address the downsides of managing a large fleet of agents while retaining personalization. The core idea is "unhobbling" agents, giving them more capabilities.

• Centralized Brain: Instead of agents being trapped in individual sandboxes, QM offloads all agent conversations and context into PostgreSQL, making it centralized and accessible to the agent.
• Sandboxes as Resources: Sandboxes are treated as resources that agents can dip into as needed, rather than fixed homes. Agents can choose sandboxes with more resources for heavier workloads or simpler ones for lighter tasks.
• Automated Improvement Loop (Mixed Results): QM accumulates a large evaluation set from user conversations. While the goal is automated improvement, using LLMs as judges can lead to "main character syndrome," where agents make narrow fixes. Human-in-the-loop remains important.
• Access to Company Resources: Agents are wired to YC's internal CLI and can use arbitrary API keys. Device code and OAuth are ingested into a keychain for seamless integration.
• Human-Reviewed Writes: While mostly read-only, agents can propose database edits via human-reviewed bulk upserts, which are increasingly rubber-stamped as trust in the agent grows.
• Agent-Controlled Runtime: Agents can choose their LLM provider (e.g., switching models to avoid refusals in sensitive tasks like AI research or cybersecurity) and sandbox providers.
• Thin Harness: The core harness is kept extremely thin, focusing on three tools:
  1. Execution in a remote sandbox.
  2. Reading and writing from object storage.
  3. Publishing internal apps. Other tools (memory, crons) are seen as temporary solutions for rough edges.

Challenges and Solutions

• Agents Giving Up Too Early: Agents often give up prematurely despite a capable environment. QM introduced a "grind tool" that sets budgets (time or token spend) on goals, forcing agents to persist, leading to better research outputs and reports.
• Confusion in Social Contexts: Agents can get confused about their situation, especially in multiplayer (Slack) environments, even with clear system prompts. Local affordances are needed.
• Lack of Social Understanding: Agents don't intuitively understand social contexts or information sharing boundaries. Privileged information can easily leak. QM relies on YC's existing fine-grained permissioning system to bound the information agents can access, a challenge for organizations without such infrastructure.

QM is open-source, and the team is hiring.