Anthropic AI development metrics show 26% of R&D led by Claude

Anthropic has published a proposed framework for monitoring the pace of frontier AI development, releasing internal readings on how much of its research work Claude performs, how it oversees AI agents and how it allocates compute across development work. The Anthropic AI development metrics are intended to give governments, third parties and the public more visibility into development processes, though the company says the approach is not yet a shared standard across labs.

The release follows chief executive Dario Amodei’s September call to “pace the frontier”, his proposal to slow the rate of improvement in AI capabilities enough to allow alignment measures, safeguards and outside confirmation to catch up. Amodei said pacing would not mean stopping model training or technical progress.

What are Anthropic’s AI development metrics?

Anthropic’s framework addresses three parts of the production process: AI-led AI research and development, oversight of AI agents, and the compute resources allocated across development work. The company distinguishes these process measures, which describe how models are made, from capability evaluations, which assess what models can do.

• AI-led R&D: Anthropic’s prototype R&D Automation Index inventories its AI research tasks, assigns each an automation rating and combines the results. Its scale ranges from AL0, with no AI involvement, to AL5, where AI completes work fully autonomously without a human in the loop. At AL3, an AI system carries out substantial work under close human direction; at AL4, it completes most of a task from a high-level instruction while a person supervises.
• Agent oversight: The company proposes measuring coverage, review latency and escalation rate. Coverage is the proportion of an agent’s actions that pass through a monitor before or after execution. Review latency measures the interval before automated and human review. Escalation rate records the share of activities blocked, redirected or flagged for further examination.
• Compute allocation: This records how resources that power the development of more capable models are distributed, including the share directed to safety work.

Anthropic said that, as of August, Claude was not fully autonomous in any measured subset of the company’s AI R&D. It classified Claude as leading 26% of that work, while more than 90% was at or above the level where AI collaborates with humans. Those are company-reported findings from its own internal methodology.

Its agent-oversight snapshot found about 30,000 agents conducting research and engineering work at any one time on Anthropic’s most-used internal platform in August. Anthropic said the reported measurements cover that platform only.

What did Anthropic’s compute snapshot show?

For the week from July 13 to July 20, Anthropic said roughly 6% of the compute used for AI R&D was assigned to safety. It put the safety share at roughly 12% when looking only at compute allocated to AI-driven R&D.

The figures offer a baseline rather than an independently audited comparison of companies. Anthropic said cross-lab reporting faces two immediate obstacles: no common methodology exists, and its current system relies on Anthropic models to assess Anthropic systems, raising the possibility of similar errors in the model being assessed and the model acting as judge.

The company said third-party verification, or assessments using other developers’ models, could help address those limits if safeguards protect competitively sensitive information. Anthropic also said it plans to place independent evaluators from multiple organisations inside the company with access comparable to that of internal risk-assessment teams, allowing them to verify safety practices, report incidents and monitor key measurements.

Anthropic presents the framework as a starting point for outside assessment of development pace, rather than as proof of model safety or a standard already adopted across the industry.

This story draws on original reporting from CNBC.