ACN · Presented by Levr
ACN WIRE

Research

METR: AI models still lag at judgment

An update to AI's time horizon graph

Written with AI assistance, reviewed by the ACN desk.

“By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify”
Matt Putz, Member of technical staff, METR
0:00 / 0:00

Image: MIT Technology Review

The quick hit

  • METR has updated its time horizon graph with judgment-heavy tasks.
  • Some AI capabilities are still improving at an exponential rate, the report finds.
  • No AI-designed evaluation was good enough for METR staff to use.

METR, a nonprofit that evaluates AI models, has updated its time horizon graph to include tasks that need judgment. MIT Technology Review reports that some AI capabilities are still improving at a rapid, exponential pace.

The original graph tracked the length of tasks frontier models could complete. That length doubled every four months. The new report adds longer and more open-ended tasks. Many completion times in it are estimates rather than measurements.

Models did worse on the judgment-heavy tasks. In one, they designed AI evaluations for METR. According to MIT Technology Review, none was good enough for METR staff to use in their research. In another, models scored philosophical arguments and again fell short of people.

“By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify,” Matt Putz, a member of METR’s technical staff, told the magazine. Sayash Kapoor, an incoming professor at UC Berkeley, said the move toward open-ended tasks is promising. Some researchers argue that models trained on easy-to-verify tasks will eventually handle open-ended ones too.

“By their nature, hard-to-verify or judgment-laden tasks are much harder to study, and especially to quantify”
Matt Putz, Member of technical staff, METR
“We're at the point where AI evaluations need to get more creative”
Sayash Kapoor, Computer scientist, incoming professor at UC Berkeley, UC Berkeley

Sponsor

Sources

  1. MIT Technology Review · Nov 20, 2026

Spot an error? Email corrections@aicodingnews.si. Fixes are logged on the corrections page.

From our sponsors

Sponsor

Sponsor

Also in

Sponsor

Sponsor