METR updates AI time horizon plot with open-ended tasks; models fall short on judgment work
The nonprofit's new report keeps the exponential trend on verifiable tasks, but no AI-designed evaluation was useful to METR staff, though Opus 5.5 reached "borderline above the bar."
What happened
METR, a nonprofit that tests AI, updated a well-known chart of how long a task AI can finish. Before, the chart used tasks with clear right answers, like coding. The new version adds fuzzier tasks that need judgment, such as designing a test for AI.
Why it matters to you
It shows AI is far better at checkable work like coding than at open-ended work. That affects how soon AI could take over research jobs.
The takeaway
AI keeps improving fast at coding-style tasks, but it still lags humans on tasks that need judgment.
What it means for you
Models trained on verifiable tasks like code and math may not transfer to open-ended work, so test them on your own fuzzy tasks.
Expect AI to handle well-specified coding work better than tasks that call for judgment or taste.
METR's report and an OpenAI blog post both indicate lower performance on real-world tasks than on easily verifiable ones.