Opus 5 AI Leap: Runtime, Prompt Injection Immunity, and Agentic Coding

Boris, the creator of Cloud Code, recently discussed the latest advancements in AI models, particularly with the release of Opus 5, and shared insights into building agentic products.

Opus 5: A Leap in AI Performance

Opus 5 has demonstrated significant improvements, achieving 30% on Arc AGI 3, a substantial increase from previous models that scored in the low single digits or teens. This new model introduces several groundbreaking capabilities:

• Extended Runtime: Opus 5, especially when combined with auto mode, can run for days, weeks, or even months without stopping. It doesn't require scaffolding or external tools like SLGO, as it inherently understands and pursues its tasks.
• Prompt Injection Resistance: A major breakthrough is Opus 5's apparent immunity to prompt injection. This has been a critical challenge in AI, where models could be tricked into executing harmful instructions. Opus 5, building on improvements from Opus 4.7 and 4.8, achieves this through a combination of:
  • Well-aligned model: Three years of research into alignment contribute to its robustness.
  • Prompt injection classifier: This system, based on Chrisola's mechanistic interpretability work, literally identifies neurons in the model's "brain" that activate during prompt injection attempts, allowing for detection even if the model doesn't explicitly indicate it.
  • Auto mode classifier: This third layer further enhances its resistance. With these three layers, prompt injection has become virtually impossible to demonstrate.

The Evolving Nature of System Prompts and Harnesses

A surprising aspect of building with new models is the constant need to adapt and even discard previous configurations.

• Dynamic System Prompts: With each new model release, a significant portion of the system prompt for Claude Code is deleted or changed. For Opus 5, over 80% of the system prompt was removed. This is because newer models are more intelligent and can perform tasks that previously required explicit instructions.
• Ablation for Optimization: The process involves "ablation," where the entire system prompt is deleted, and then lines are gradually added back to understand their individual impact. This empirical approach helps determine what instructions are truly necessary.
• Simple Mode: An undocumented feature, claude code simple=1, deletes all system prompts, including those for tools. Interestingly, the model often performs more intelligently without these prompts. However, for product usability, some prompts are retained to guide the model's behavior in a user-friendly way.
• Constant Iteration: This iterative process extends to tools and the harness code itself. Tools are frequently un-shipped, and code is deleted as models evolve. The current Claude Code harness primarily focuses on safety, permissions, static analysis, and UI, with much of the other code having been removed.

Building Agentic Products: The "Unhobbling" Approach

The philosophy behind Claude Code's development is to "unhobble" the model, allowing it to fully express its capabilities.

• Product Overhang: This concept describes the gap between what a model is capable of doing and what existing products allow it to do. Often, products inadvertently "hobble" the model by imposing unnecessary constraints or failing to elicit its full potential.
• Birth of Claude Code: The original Claude Code was born from the realization that models like Sonnet 3.5 (a leading coding model at the time) could write entire functions and files, but existing coding products only offered single or multi-line autocompletion or read-only access to codebases. By removing scaffolding and providing a simple harness, Claude Code enabled the model to write entire files and build features.
• Empirical Development: Building with models requires an empirical, scientific mindset. It's less about upfront system design and more about treating the model like an organic, living creature with a unique "personality" in each generation. Developers must observe its behavior, identify its strengths and weaknesses, and adjust the harness accordingly.
• Evals and Saturation: While evals (evaluation metrics) are crucial, they also have a limited lifespan. As models improve rapidly, evals can become "saturated," meaning the model consistently performs perfectly on them. In such cases, new evals must be developed based on where the model still struggles.

Practical Applications and Future Opportunities

Opus 5 opens up new possibilities for AI agents, encouraging developers to push boundaries.

• Challenging Tasks: Instead of overly specific instructions, users should provide high-level tasks, define guardrails, and set exit criteria, then let the model "cook." This approach, which wouldn't have worked six months ago, is now effective.
• Codebase Rewriting: Opus 5 can rewrite entire codebases from one language to another. A notable example is the Bun JavaScript runtime, originally written in Zig, which was rewritten to Rust by Claude. This process, which would have taken human engineers over a year, was completed in 11 days with steering, demonstrating the model's ability to handle complex, large-scale transformations.
• Creative Exploration: Beyond practical applications, Opus 5 can be used for creative tasks. For instance, it can be given OpenCV and asked to draw images like portraits, animals, or landscapes, a capability not explicitly trained but discovered through experimentation. This highlights the "solicitation gap" – the model's ability to perform tasks if asked correctly.
• Model Elicitation and Verification: The key to effective interaction with models is not just prompt engineering, but "model elicitation" – figuring out how to give Claude a challenging task and, crucially, how to enable it to verify its own work.
  • Example: Native Desktop App Rewrite: A Claude Tag session was initiated to rewrite the Electron-based Claude desktop app into Swift. The prompt involved running the Electron app in a Mac virtual machine, screenshotting it, and then pixel-by-pixel comparing it to the Swift version, continuing until completion. This task has been running for over two weeks, with Claude even live-blogging its progress in an internal Slack channel. This demonstrates the model's ability to sustain long-term, complex tasks with proper verification mechanisms.

Leveraging Agents for Automation

The ability to spawn thousands of agents is a game-changer for automation.

• Dynamic Workflows: This feature in Claude Code allows the model to orchestrate numerous agents within a Bun sandbox. It can fan out tasks, run agents in sequence or parallel, and efficiently manage tokens to perform complex work, such as rewriting codebases or in-depth data analysis. This represents a new form of "test-time compute," significantly increasing the computational power applied to a task.
• Loops and Routines:
  • Loops: Local cron jobs for Claude.
  • Routines: Cloud-based cron jobs that run even when the laptop is closed. These are used for repetitive tasks that may share memory but not necessarily context.
• Self-Maintaining Codebases: Claude is now used to maintain its own codebase. Routines run daily to:
  • Clean up dead code using static and dynamic analysis.
  • Ship experiments that have reached 100% rollout.
  • Write tests for areas needing coverage.
  • Delete unnecessary tests.
  • Unify nearly duplicated abstractions (the "abstraction police"). These routines involve hundreds, sometimes thousands, of agents daily, effectively automating the maintenance work of dozens or hundreds of engineers, freeing them to focus on new product development and user interaction.

The Future of Coding and Learning

With AI agents increasingly handling coding tasks, the definition of an "exceptional builder" is shifting.

• Coding is Solved (for many): While not universally true for all complex systems or highly precise UI tasks, coding is becoming increasingly automated for many common scenarios.
• Empirical Mindset: The most successful users of Claude are those who adopt an empirical mindset, letting go of old assumptions about computer science theory or past model behaviors. They observe the model's performance, identify struggles, and adjust their approach accordingly.
• Advice for CS Students: The advice for aspiring computer scientists is to learn not just the theoretical aspects but also how to apply them practically. This involves building products, developing design and business sense, learning data science, and interacting with users. The speaker's personal journey, starting with programming a TI-83 calculator to cheat on math tests, exemplifies learning through practical problem-solving.

Special Announcement

Attendees of the discussion received a special offer: 20x max usage for Claude Code, encouraging them to experiment and build innovative solutions.