Anthropic Lifts the Hood: Claude "Leads" 26% of Its AI R&D

The bet: measuring inputs, not just capabilities

Capability evaluations tell you what a model can do. Anthropic proposes the opposite here: measuring how models are made. The idea is to be able to correlate verifiable inputs — compute, human time, degree of automation — with outputs, rather than discovering capability jumps only once a model is released.

The political framing is explicit right from the introduction: "as the world considers slowing the pace of frontier AI development, the public needs more information." This is the direct continuation of Dario Amodei's call for coordination on pace, and of the regulatory framework Anthropic has proposed, the Advanced AI Framework.

In other words: these metrics aren't just transparency for transparency's sake. They're designed to serve as triggers. Anthropic suggests, for instance, that a certain level of automation could require a mandatory testing window before a new model is used to do AI R&D.

26%: what the number means (and doesn't)

The index relies on Epoch AI's scale, from AL0 (no AI) to AL5 (full autonomy). AL3, "collaborates": AI handles large chunks of work under tight human direction. AL4, "leads": it completes most of the task from a high-level instruction, with a human supervising. As of August 2026, 26% of Anthropic's AI R&D work is at AL4 or above, more than 90% at AL3 or above, and nothing reaches AL5.

The methodology is unusually detailed. For each week of July 2026, 20% of staff in departments involved in the R&D loop were sampled; a Claude agent read their Slack traces and internal documentation to list their tasks. Result: ~15,000 granular tasks, reorganized into a tree of 542 nodes including 378 leaves, of the type "diagnosing eval platform defects" or "serving incident postmortems." The tree is frozen so that subsequent measurements cover the same basket of work.

The limitations are acknowledged. The judge is a Claude model, so it may share the blind spots of what it's evaluating; exact model-human agreement is 59%, versus 35% between humans, and 97% of ratings fall within one level of disagreement. Time-weighted person-hours are admitted to be "a rough approximation." And a frozen basket, by construction, doesn't capture new categories of work that humans might invent as they shift focus.

30,000 agents and a control room

The second component is the most operationally concrete. On its most heavily used internal platform, Anthropic runs roughly 30,000 research and engineering agents simultaneously. Two types of safeguards: online monitors, which inspect every action before execution and block anything irreversible — typically an agent attempting to exfiltrate model weights — and offline monitors, which review after the fact to catch slow drifts, such as signs of misalignment.

The three proposed metrics are coverage, review latency, and escalation rate. Coverage: 100% of actions pass through the online monitor, usually within seconds, and 100% are ingested by the offline monitor. Blocking rate: out of more than a billion decisions in August 2026, 0.002% were blocked. On the offline side, one to two transcripts per thousand are flagged, i.e. ~100,000 per week, filtered through classifiers before around fifty reach humans.

Two architectural choices are worth noting. Each agent has a persistent identity, not tied to the underlying model, which allows its history to be audited across updates and prevents it from mistaking what another agent says for its own thinking. And agents communicate on a shared, open messaging system, not privately: they can correct each other, and monitoring can follow the conversation between them.

The key point is statistical: Anthropic notes that individual bad behaviors are rare, but that "when millions or billions of agents operate in the economy, even rare events happen regularly." METR has already independently red-teamed the offline monitoring platform.

6% of compute for safety: the most debatable number

During the week of July 13–20, 2026, roughly 6% of the compute allocated to AI R&D went to safety work, and roughly 12% of AI-driven R&D compute did. Anthropic presents these estimates as deliberately conservative: any work that serves both capabilities and safety was counted on the R&D side, and safeguard classifiers — a comparable amount of compute — are not included.

The company itself defuses the naive reading of the number. Safety research is inherently light on compute: it's mostly about designing experiments, which takes human time and little GPU. Worse, the metric is perversely sensitive to efficiency: a more efficient safety classifier lowers the safety share without any less safety work being done. Its value is therefore comparative — over time, and across labs — rather than absolute.

The classification was done by a Claude classifier reading run metadata and code, on a sample of ~14% of the nearly 10,000 runs that week, weighted toward the most compute-intensive ones. Anthropic acknowledges that some edge cases remained undecidable even after several hours of human review, and that "our definition is one reasonable choice among others."

Hence the sharpest recommendation in the piece, which applies as much to Anthropic as to its competitors: every developer will be tempted to draw the line generously, so the burden of proof must fall on them, and the definition should be negotiated in advance with regulators and the community.

The real deliverable: a methodology others could copy

For each of the three measures, Anthropic specifies what "any frontier developer could publish today." This is the heart of the maneuver: turning an in-house report into a de facto standard, before a regulator invents a worse one.

Two obstacles are identified for cross-lab comparison: the lack of a shared methodology, and the fact that a lab uses its own models to self-evaluate. The proposed remedies are third-party verification, or cross-evaluation using other developers' models, with safeguards against exchanging commercially sensitive data.

Finally, Anthropic announces it is integrating third-party evaluators from several organizations, with access to internal processes, systems, and data comparable to that of internal risk-evaluation teams. Their mandate: verify safety practices, flag incidents, and track these metrics. This is where the credibility of the whole exercise will be decided.