Get early access
Back to episodes
#7922 days ago

SlopCodeBench: The New Metrics

Back in August we dug into SlopCodeBench, the benchmark that makes coding agents build on their own prior code across checkpoints and measures whether they bury themselves in slop. Since then the benchmark has grown (more problems, more checkpoints, more models) and shipped a new set of metrics for measuring how code degrades. This week we're going back to see what's new: verbosity and structural erosion scores alongside strict and isolated pass rates, how effort levels and cost change the picture, and what the numbers say about whether higher-effort runs actually leave you with cleaner code. If you missed the original episode, catch up on the August 4th session first, but you don't need it to follow along. Meet the Speakers🧑‍💻​ ​​Meet Vaibhav Gupta, one of the creators of BAML and YC alum. He spent 10 years in AI performance optimization at places like Google, Microsoft, and D. E. Shaw. He loves diving deep and chatting about anything related to Gen AI and Computer Vision! ​Meet Dex Horthy, founder at HumanLayer and coiner of the term Context Engineering. He spent 10+ years building devops tools at Replicated, Sprout Social and JPL. DevOps junkie turned AI Engineer.

Project details

Open in GitHub

Dex walks through the latest SlopCodeBench results across model families and effort levels, then Vaibhav shows how baking maintainability rules and metrics into BAML gives coding agents back pressure on code quality. A short one, but dense.

Video

Links:

Episode Highlights

"If you just do the lights off factory thing and just feed a model features and high-level design, you will over time accumulate so much slop."

"Some of these bigger, juicier models, if you ask them to think more, they will think themselves out of the right solution."

"Many people now would sacrifice a little bit of cost in favor of maintainability."

Key Takeaways

  • Strict pass rate is the number to watch. A checkpoint only counts if its tests and every earlier checkpoint's tests pass. If checkpoint one shipped a bug, no later checkpoint can earn a strict pass, which mirrors how real bugs behave. In the latest run, Astra edged out Opus 5.5.
  • More effort doesn't reliably buy better code. Opus 5.5 on medium beat Opus 5.5 on x-high by about three percentage points for less money, while the OpenAI models did better on x-high. Fable 5.1 on x-high was the expensive outlier with little to show for it. Run your own tasks at two effort levels before changing your team's default.
  • Counting slop means counting the trash the model leaves behind. Test scripts and verifier files stay in the metrics, because if you merge without looking and the model never cleans up, that mess is part of the slop. Vaibhav's team bans stray .md files in the repo for the same reason: they go stale, then the next agent reads them as truth.
  • The detectors are aggressive and aging. Around 212 slop detectors mix custom AST linters with standard metrics like cyclomatic complexity, and many date from March and May. Opus 5.5 x-high triggered the fewest, while Fable wrote nearly the most functions with the lowest average complexity, which says nothing yet about whether anyone would want to read them.
  • Put maintainability rules in the compiler, not just the prompt. Vaibhav wrote guidelines from good and bad examples (no magic numbers, match over if/else), ran each task from scratch ten times, and saw nearly every complexity metric improve, with the worst function's cyclomatic complexity the holdout. Having the compiler push back when code got worse is cheap back pressure.
  • Optimize for maintainability months from now, not for today's cost. Tokens and agent time matter far less than complexity that compounds. Turn your own style rules into a skill or check, run the same task with and without them, and see in a chart whether they actually simplify the code.

Resources

Whiteboards

All episodes