Study: AI agents shift bottleneck to code review
Harvard working paper by Fiona Chen and James Stratton · 718 firms
Research
An update to AI's time horizon graph
Written with AI assistance, reviewed by the ACN desk.
“By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify”
METR, a nonprofit that evaluates AI models, has updated its time horizon graph to include tasks that need judgment. MIT Technology Review reports that some AI capabilities are still improving at a rapid, exponential pace.
The original graph tracked the length of tasks frontier models could complete. That length doubled every four months. The new report adds longer and more open-ended tasks. Many completion times in it are estimates rather than measurements.
Models did worse on the judgment-heavy tasks. In one, they designed AI evaluations for METR. According to MIT Technology Review, none was good enough for METR staff to use in their research. In another, models scored philosophical arguments and again fell short of people.
“By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,” Matt Putz, a member of METR’s technical staff, told the magazine. Sayash Kapoor, an incoming professor at UC Berkeley, said the move toward open-ended tasks is promising. Some researchers argue that models trained on easy-to-verify tasks will eventually handle open-ended ones too.
“By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify”
“We're at the point where AI evaluations need to get more creative”
Sponsor
Spot an error? Email corrections@aicodingnews.si. Fixes are logged on the corrections page.
Sponsor
Sponsor
Harvard working paper by Fiona Chen and James Stratton · 718 firms
OpenAI's second report on agents in its research
Runta FrontierHarness Eval v1.1 · open models only
The Riemann proof passes its formal check; understanding it comes next
Sponsor
Sponsor