AIAIAI.FYI
LIVE

METR updates AI time horizon plot with open-ended tasks; models fall short on judgment work

The nonprofit's new report keeps the exponential trend on verifiable tasks, but no AI-designed evaluation was useful to METR staff, though Opus 5.5 reached "borderline above the bar."

What happened

METR, a nonprofit that tests AI, updated a well-known chart of how long a task AI can finish. Before, the chart used tasks with clear right answers, like coding. The new version adds fuzzier tasks that need judgment, such as designing a test for AI.

Why it matters to you

It shows AI is far better at checkable work like coding than at open-ended work. That affects how soon AI could take over research jobs.

The takeaway

AI keeps improving fast at coding-style tasks, but it still lags humans on tasks that need judgment.

What it means for you

If you build software

Models trained on verifiable tasks like code and math may not transfer to open-ended work, so test them on your own fuzzy tasks.

If you run a business

Expect AI to handle well-specified coding work better than tasks that call for judgment or taste.

If you invest

METR's report and an OpenAI blog post both indicate lower performance on real-world tasks than on easily verifiable ones.

Get stories like this every morning