Anthropic says it uses AI to build more powerful AI, and according to a new report from AI safety organization METR, the approach appears to work well. The company's models reportedly improve about twice as fast as a year ago, with no sign of a slowdown. The findings are based on an 11-week in-house investigation that was part of an evaluation of the company's AI research and development.
What the report shows
According to METR, benchmark scores suggest that Anthropic has made roughly two years of historical progress over the last year. The researchers say they are confident this acceleration was not simply due to scaling up training compute. As of October, they say the majority of the acceleration is due to AI, with its contribution growing over time. This is supported by informal Slack polls and interviews with Anthropic employees. Employees using the company's most capable internal model reported on average being around 12 times as productive as without AI assistance, roughly double what staff had reported a few months earlier while using Mythos 5. For now, the acceleration appears to come mainly from AI assisting human researchers rather than working independently, though METR warns that fully autonomous contributions may play a larger role soon. The researchers found no clear signs of diminishing returns or observable bottlenecks.
METR says three newer, unreleased models completed training within six months, codenamed Model 2, Model 3 and Model 4. It also identified an unreleased model dating back to 2022.
Redactions limit verifiability
Anthropic gave METR access to a range of non-public information, beyond the originally scoped project. The researchers did not receive unrestricted access, however. METR writes that its report runs about 100 pages, but in the public version Anthropic redacted a majority of the evidence the researchers found most informative. Much of the redacted evidence could have multiple plausible interpretations, so the public has to rely more on METR's judgment than would be ideal. Anthropic cofounder Sam McCandlish argued that sharing exactly what drives the speedup is commercially sensitive and could provide a roadmap for others. The company said it plans to publish its own report on the pace of AI progress in the coming weeks.
Criticism and assessment
The publication has drawn sharp criticism from AI safety researchers, who point to an apparent contradiction: Anthropic is a prominent advocate of safety and transparency, yet the report suggests it is accelerating the technology. Daniel Kokotajlo of the AI Futures Project accused the company of "recklessly gambling with our lives" and criticized citing rivals going faster as a reason for redactions when it does not appear to be slowing down. Anthropic declined to answer The Verge's questions about the report.
Assessment: If the pattern holds, it could become relevant for companies and regulators how verifiable the claims of major labs about their own progress are. Whether the acceleration continues and how independent review can work in future remains open.
Sources
Editorially owned by Ideal Syka. Sources and method: Newsroom & method. Tips and corrections: ai@i6eal.de.
