90/30 Club (ML reading) #46: "AutoHarness: improving LLM agents by automatically synthesizing a code harness"
Week 46: "AutoHarness: improving LLM agents by automatically synthesizing a code harness" by Lou et al
https://arxiv.org/pdf/2603.03329
LLMs are brilliant strategists but terrible rule-followers, often losing complex games simply by attempting illegal moves. Instead of wasting human hours hard-coding safety "harnesses" to keep the AI in bounds, researchers introduced AutoHarness. This framework forces the LLM to write its own Python guardrails through a rapid, feedback-driven tree search. By testing actions against the game environment and refining the code based on the resulting errors, the model autonomously builds a flawless safety net that filters out 100% of illegal actions across 145 different TextArena games.The performance gains are massive. Equipped with AutoHarness, the lightweight and cost-effective Gemini-2.5-Flash consistently outmaneuvers its heavier sibling, Gemini-2.5-Pro. But the real breakthrough happens when the technique is pushed to the extreme: synthesizing the entire game policy into pure Python code. By offloading the decision-making entirely to the generated code, the agent plays the game natively without ever calling the LLM at runtime. This "harness-as-policy" approach drops inference costs to practically zero while actively beating the average reward scores of both Gemini-2.5-Pro and GPT-5.2-High.
Join us at Mox to explore:
Creating harnesses for AI with AI (meta research)
Safety Concerns (p hacking)
What is a harness?
Discussion at 20:00, (optional) quiet reading from 19:00.