Kimmy K3.1 Model Beats Benchmarks Amid Distillation Controversy
Last week, an anonymous model named Kibitz appeared on Chapot Arena, demonstrating frontier-level performance. The name "Kibitz," a Yiddish word for someone giving unsolicited advice during a chess game, is seen as a fitting analogy for a coding agent. It was quickly deduced that Moonshot AI was behind Kibitz, a suspicion confirmed when Moonshot officially released Kimmy K3.1, a new three-trillion-parameter open-weight model, on Friday. This model is expected to be fully downloadable on November 4th.
Kimmy K3.1's performance has surprised many, especially given recent claims by Anthropic about blocking distillation attacks.
The Distillation Debate
Earlier this year, Anthropic accused Moonshot AI, along with other Chinese companies like DeepSeek and MiniMax, of conducting large-scale distillation attacks against Claude. These attacks allegedly involved using swarms of fake users to steal Claude's output and reasoning tokens, which could then be used to train open models.
The speaker views distillation as a fundamental human right, arguing that large AI companies have themselves "distilled a collective human brain" by scraping internet content, GitHub repositories, scientific papers, and even destructively scanning books (referencing Project Panama). However, OpenAI and Anthropic consider such activities "grand theft autocomplete" and have invested millions to prevent them.
Just weeks ago, Anthropic claimed to have largely solved the distillation problem, blocking 90% to 99% of attacks from China. OpenAI has also demonstrated technology capable of stopping smaller-scale distillation efforts. If these claims are true, the performance of an open Chinese model like Kimmy should theoretically be impossible, as the prevailing narrative suggests Chinese models achieve rapid improvement by stealing outputs from larger AI models.
Kimmy K3.1's Benchmark Performance
Despite the claims of blocked distillation, Kimmy K3.1 has achieved impressive benchmark results:
- Terminal Bench 4.0: Kimmy K3.1 is within one point of Opus 5.5, a significant achievement, with no other open model coming close. Artificial Analysis independently verified these numbers, scoring Kimmy at 67%, the highest among tested models. However, it's noted that the "harness" used for testing plays a crucial role, as different harnesses can yield vastly different results.
- S. Marathon: Kimmy also excels in S. Marathon, a test for multi-hour software requirements, indicating its ability to handle complex tasks that would take humans weeks to complete without AI assistance.
Cybersecurity Caveats
A significant caveat regarding Kimmy's coding capabilities, particularly in cybersecurity, is Moonshot's deliberate removal of cybersecurity data from its training dataset. This was done to make the model less effective at hacking. Internally, an unfiltered checkpoint of the model solved 90.5% of illegal exploit tests, while the final release model only solved 81%.
This decision is seen as a strategic business move, especially after the US recently banned China's DeepSeek model from government use. Moonshot likely aims to avoid similar bans on all Chinese models, which would benefit OpenAI and Anthropic.
Despite these limitations, Kimmy still performed at the top of all publicly available models on the new Cyber Index at Artificial Analysis. This is largely attributed to Claude and GPT-6 declining most prompts on safety grounds, effectively taking a lower score.
The 3D Horse "Thoroughbench" Test
To objectively assess AI capabilities, a "Thoroughbench" was developed, focusing on generating a realistic, interactive 3D horse using 3JS. The process involves typing "create a realistic interactive 3D horse with 3JS" into the chat window and ranking the results from 0 to 10 based on a proprietary loss function.
Previous scores on Thoroughbench: * Google Argon: 0.69 * GPT-6 Astra: 5.21 * Opus: 6.39
Testing Kimmy K3.1 required a paid subscription (250 yuan per month). Kimmy produced a visually stunning horse with accurate coloring and realistic appearance, including a reasonably sized copulatory organ. However, a significant flaw was discovered in its animation. The animation was achieved by simply warping the original mesh, a rudimentary and unsophisticated method.
In contrast, Claude, while producing a less aesthetically pleasing horse, built a mesh with 95,000 vertices, each tied to one of 35 independently animated bones. Even Gemini Argon used bones, albeit with an extra one. Skeletal animation, though more complex, prevents the horse from appearing distorted during movement.
Kimmy K3.1's final score on Thoroughbench was 4.20, which is impressive but still lags behind other frontier models due to its animation shortcomings.
Mobin: A Solution for UI Design
For those looking to build advanced 3D horses or user interfaces, Mobin is recommended. Mobin's MCP server allows coding agents to study a library of 600,000 real UI screens before designing anything. Instead of generic designs, agents like Claude Code, Cursor, or Codeex can use Mobin to gather references, build visual reports of common patterns, and then recreate and adapt those flows to specific applications. Every result links back to the original screen for source verification, ensuring designs have "actual taste." Mobin offers a free trial and a 20% discount.
Takeaways
- Moonshot AI released the three‑trillion‑parameter open‑weight model Kimmy K3.1, which will be fully downloadable on November 4th and has already outperformed other open models on several benchmarks.
- Despite Anthropic’s claim that it has blocked 90‑99 % of distillation attacks, Kimmy K3.1 achieved a Terminal Bench 4.0 score within one point of the proprietary Opus 5.5, suggesting that large‑scale stealing may still be influencing Chinese model progress.
- The model’s cybersecurity capabilities were intentionally reduced by removing hacking‑related data, dropping its exploit‑test success from 90.5 % in an unfiltered checkpoint to 81 % in the released version, a move likely aimed at avoiding U.S. bans.
- In the Thoroughbench 3D‑horse test, Kimmy scored 4.20, producing a visually appealing horse but using a simplistic mesh‑warping animation, whereas competitors like Claude employed sophisticated skeletal rigs with thousands of vertices.
- The Mobin MCP server, offering a library of 600 k UI screens, is recommended for agents that need high‑quality UI design references, providing free trials and discounts to improve design fidelity beyond generic code generation.
Frequently Asked Questions
Why does Anthropic claim it has blocked 90‑99 % of distillation attacks, and how does Kimmy K3.1’s performance challenge that claim?
Anthropic says its new defenses stop 90‑99 % of large‑scale distillation that copies Claude’s outputs. Kimmy K3.1’s benchmark scores, which are within a point of the proprietary Opus 5.5, suggest the model still benefited from stolen data or other sources, indicating Anthropic’s block may not be as comprehensive as claimed.
What is the impact of removing cybersecurity data from Kimmy K3.1’s training set?
Moonshot stripped hacking‑related content to reduce the model’s ability to generate illegal exploits, dropping its success rate on exploit tests from 90.5 % in an unfiltered checkpoint to 81 % in the released version. This trade‑off improves regulatory acceptance but also limits the model’s usefulness for security‑research tasks.
Does this page include the full transcript of the video?
Yes, the full transcript for this video is available on this page. Click 'Show transcript' in the sidebar to read it.
Helpful resources related to this video
If you want to practice or explore the concepts discussed in the video, these commonly used tools may help.
Links may be affiliate links. We only include resources that are genuinely relevant to the topic.