METR: AI models still lag at judgment
An update to AI's time horizon graph
Research
Harvard working paper by Fiona Chen and James Stratton · 718 firms
Written with AI assistance, reviewed by the ACN desk.
“The paper locates the shortfall in code review.”
A Harvard working paper has found that AI coding agents shifted the bottleneck in software work to code review. ComplexDiscovery reports that Ph.D. candidates Fiona Chen and James Stratton wrote the paper, which covers 718 firms.
At firms that adopted coding agents, lines of code added rose about 30%. Commits rose 20% and pull requests rose 23%. Resolved issues showed no statistically detectable rise. Median time from pull request to merge rose 3.45 days on a 7.03-day baseline, a 49% increase. Review comments per pull request rose 35%.
The data came from Jellyfish, an engineering analytics company, joined to LinkedIn employment records from Revelio Labs. According to ComplexDiscovery, the authors offer two possible causes. Faster drafting may overload a fixed pool of reviewers. Alternatively, more change requests may signal lower draft quality.
ComplexDiscovery reported the paper on October 13. It notes the paper is a job-market paper with no indication of completed peer review. Jellyfish has a commercial interest, and the authors state the conclusions are their own.
“The paper locates the shortfall in code review.”
Sponsor
Spot an error? Email corrections@aicodingnews.si. Fixes are logged on the corrections page.
Sponsor
Sponsor
An update to AI's time horizon graph
OpenAI's second report on agents in its research
Runta FrontierHarness Eval v1.1 · open models only
The Riemann proof passes its formal check; understanding it comes next
Sponsor
Sponsor