Learning Layer Paper Reading Club - Week 33 - SlopCodeBench
This week's paper: SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks
Link: https://arxiv.org/abs/2603.24755 Site: https://www.scbench.ai/ Code: SprocketLab/slop-code-bench
Abstract:
Almost every coding benchmark asks an agent to solve a problem once, from scratch. Real software is not built that way — it is extended, again and again, by someone working on top of code that already exists. A single-shot setup hides the thing practitioners actually worry about: not whether an agent can pass a test today, but what its code looks like after twenty rounds of "now also handle this."
SlopCodeBench measures that. It is a benchmark of 36 problems and 196 checkpoints in which an agent repeatedly extends its own prior solution as new requirements arrive. The specifications deliberately describe only external behavior — a CLI or API contract — and say nothing about architecture, function signatures, or module boundaries, so early design decisions are the agent's own and compound across the run. The test suite stays hidden so it can't leak structural hints.
Alongside correctness, the authors track two forms of degradation: structural erosion (complexity concentrating in already-complex functions) and verbosity (redundant, duplicated code growth). The results are bleak. Across 15 open and closed coding agents, none fully solves a single problem end to end, and the best passes 14.8% of checkpoints. Quality falls as trajectories lengthen: structural erosion rises in 77% of runs and verbosity in 75.5%.
The comparison that gives the paper its name: measured against 473 open-source Python repositories, agent code is 2.3x more verbose and 2.0x more eroded, and the human repos degrade less often and by smaller margins across their git histories. Explicitly telling the agent to write clean code cuts initial verbosity and erosion by up to a third — but does nothing to the rate of decay.
Relevant to anyone who ships code with an agent in the loop, and to anyone thinking about how to evaluate long-horizon agentic behavior. No preparation required — come read and discuss with us.
Jason Carver will be presenting the paper this week
What is the Learning Layer Labs Paper Reading Club?
An initiative from https://www.learninglayer.ai, a lab with the goal of reducing AI anxiety in the world.
What is the format?
Discussion based. Expect a low pressure environment to share insights and opinions with the group.
What are the group goals?
Stay on top of AI research and improve understanding of AI fundamentals + math.
Who is welcome?
Everyone! Try to put in at least some time on the paper and come prepared with questions or things you'd like to discuss, but it's ok to just show up!
Learning Layer Labs team:
Thomas Redfern ( The backbone who runs every paper reading )
Mat Allen (The new fella bring order and organization)
Devinder Sodhi (The guy you talk to for sponsoring)