← All writing

More Time Horizons Research

METR's long-awaited time horizon update shows exponential progress continuing, and how hard that progress is becoming to measure.

AIAI ProgressBenchmarks

METR has just published its long-awaited update on the time horizon statistic, the graph that has become the best way to describe the continuity of AI’s progress but that hadn’t been updated for many months. Alongside the new numbers, the update comes as a long article explaining the research.

Here is what I take from it.

First, the exponential increase in the time horizon seems to continue. METR finds a doubling time of 3.6 months (OpenAI’s recent internal estimate of 3.8 months seems to confirm this). On the subset of new tasks alone, the doubling is much faster, at around 1.7 months.

Across TH1.1 and TH2, capabilities are still increasing quickly, doubling roughly every 3.6 months.

Second, and this is the main substance of the article, measuring models at this level is getting increasingly complex. Benchmarks with tasks this long are expensive to create. It’s difficult to recruit people who work at this level, and it’s unclear whether the evaluators METR was able to recruit really match the skills of the people whose work the models would need to replicate. On top of that, the tasks that can be measured relatively easily and cheaply in a benchmark are increasingly similar to those used in reinforcement learning, i.e. during the training of the large language model.

Models have lower 50% time horizons on TH1.1 and TH2 tasks that are estimated to be costlier to make, even controlling for length.

In summary, for me this shows that exponential progress in time horizons, and so in model capability, is continuing. However, there is also some evidence that the benchmarks might be systematically overestimating the level of model capability: whatever can be easily benchmarked can also be easily trained, which makes it likely that easier tasks are oversampled. At the same time, there is some evidence that the slope might be an underestimation. This means the actual time horizon today is likely lower than the graph shows, but the rate of increase might be even faster than estimated. It is also important to remember that this benchmark focuses on software engineering and research tasks, not knowledge work in general. While there are similar trend lines in other domains of artificial intelligence, they are likely to be lower, since software engineering and AI research are also a focus of model development, with the goal of using these models to accelerate AI progress itself.

Valentin Zacharias Lead Architect, Generative AI · Mercedes-Benz
Discuss on LinkedIn ↗