Agent scaffolding goes stale with each new model

Claim

Skills that harnesses used to patch in (edit application, codebase exploration, self-verification, running tests, persistence across long tasks, deciding when to call a tool) are now native to frontier models. Evidence from independent papers: harness choice moves pass rates by 0 to 8 points with confidence intervals including zero, but tokens per solved task vary up to 40x. The harness's job on standard coding tasks is now cost and reliability, not capability.

Evidence

• Anthropic (2026-03-24): context resets added for Sonnet 4.5's "context anxiety" became "dead weight" on Opus 4.5; the sprint construct was dropped for Opus 4.6. "Every harness component reflects an assumption about what the model cannot do alone." The planner and evaluator were kept; the evaluator moved to a single end-of-run pass. Agents "reliably skew positive when grading their own work", so a separate skeptical evaluator is "far more tractable".
• Anthropic (2026-04-02): letting Opus 4.6 filter its own tool outputs via code execution raised BrowseComp from 45.3% to 61.6%; replace preloaded instructions with progressively loaded skills; promote high-stakes actions to typed declarative tools so the harness can gate, log and replay them; periodically ask "what can I stop doing?".
• Cursor retired: post-task verify, commit and push logic (now agent-controlled tools), hardcoded multi-repo logic (now repo layout plus branch and PR tools), CI-log fetching (now gh plus searchable files). Still scaffolded: computer use, as a dedicated subagent type. "The harness isn't going away so much as what it contains is changing." Cursor once ran a separate "apply" model because frontier models could not apply edits.
• The Scaffold Effect (2026-06-08): three harnesses times two models on Terminal-Bench Pro; up to 40x difference in tokens per solved task while pass-rate gaps were 0 to 8 points; each harness had a model-independent failure signature.
• Don't Blame the LLM (2026-07-04): 35 sequential Qwen Code releases with the model fixed showed quality swings attributable to specific harness PRs; harness release velocity over 2 per day.
• Stripe (2026-05-14): passive steering (AGENTS.md hints, SDK comments, warnings) failed; hard steering (errors, blocking responses, CLI help text) worked. Agents are "focused, literal, and impatient"; "errors block progress but warnings don't". Modular progressive-disclosure skills beat one monolithic file by about 10%.

Open questions

• The UC Berkeley and Apple/EPFL studies Katz cites were not located as papers; the two arXiv papers above are the independent corroboration.

Sources

• https://www.anthropic.com/engineering/harness-design-long-running-apps — blog, Mon Mar 23
• https://cursor.com/blog/cloud-agent-lessons — blog, Mon Jun 01
• https://arxiv.org/abs/2607.22585 — paper, Sun Jun 07
• https://arxiv.org/abs/2607.03691 — paper, Fri Jul 03
• https://claude.com/resources/articles/harnessing-claudes-intelligence — blog, Wed Apr 01
• https://stripe.dev/blog/ai-steering-experiments — blog, Wed May 13

AI-assisted research: gathered and summarised with AI from the sources above, each claim checked against them.