Claude's Protein Design Hit Rate Was 26.8%. One Target Returned 0 for 90.
Anthropic's autonomous protein design campaign achieved a pooled hit rate of 26.8% (354 binders from 1,320 designs), but this masks enormous per-target variance ranging from 80% on TREM2 to 0% on maltose-binding protein. A beta-binomial model fitted to published per-target counts reveals that for a new unseen target, the probability of falling below the 10% industry floor is approximately 38%, with a median hit rate of only 18.6%. The pipeline's in-silico confidence scores failed to distinguish
Analysis
TL;DR
- Anthropic's autonomous protein design campaign achieved a pooled hit rate of 26.8% (354 binders from 1,320 designs), but this masks enormous per-target variance ranging from 80% on TREM2 to 0% on maltose-binding protein.
- A beta-binomial model fitted to published per-target counts reveals that for a new unseen target, the probability of falling below the 10% industry floor is approximately 38%, with a median hit rate of only 18.6%.
- The pipeline's in-silico confidence scores failed to distinguish between targets that would succeed and those that would completely fail, scoring MBP and BBF-14 designs similarly to designs against working targets.
- Under the fitted model, achieving 95% confidence of at least one binder requires approximately 638 designs per target — a 64× increase over what the headline 26.8% rate would suggest.
- The campaign's single-target runs at 35.1% hit rate conflated focused prompting with a 2.8× larger compute budget per target, making the "narrow scope is free" narrative incomplete.
Why It Matters
This analysis exposes a critical flaw in how agentic AI benchmark results are reported and interpreted: pooled pass rates across heterogeneous tasks obscure the variance that any single deployment will actually experience. For AI practitioners building autonomous systems for scientific discovery, the finding that confidence scores cannot discriminate between solvable and closed targets is a direct warning about over-reliance on in-silico filtering without wet-lab validation. The protein design case serves as a template for evaluating every agentic benchmark — SWE-bench, ARC-AGI, τ-bench — where a single aggregate score similarly hides the distribution of per-item difficulty.
Technical Details
- Campaign architecture: Claude (Mythos Preview and Opus 4.8) orchestrated 16 open-source protein design tools across 24 distinct workflows without human intervention on design decisions. Backbones came from PXDesign (358 designs), RFdiffusion3 (267), Genie 3 (185), FreeBindCraft (135), BoltzGen (134), RFdiffusion (118), and Proteina-Complexa (100), with sequences from SolubleMPNN filtered through ESMFold2 and Protenix v2.
- Validation: Independent wet-lab validation at Adaptyv Bio using surface plasmon resonance at five target concentrations in duplicate. Designs arrived anonymized. 95% of designs expressed; 354 of 1,320 bound.
- Statistical methodology: A beta-binomial model was fitted by maximum likelihood using the Nelder-Mead optimizer from nine starting points in Python 3.11 with SciPy 1.15.3. The fitted parameters were α = 0.459, β = 1.169, implying a mean of 28.2%. A likelihood-ratio test comparing a single shared rate against per-target rates rejected the shared-rate model at p ≈ 1.8 × 10⁻⁶³.
- Per-target results: TREM2 — 72/90 (80.0%); VEGF-A — 54/90 (60.0%); IL-7Rα — 49/90 (54.4%); RBX1 — 28/90 (31.1%); TNFα — 12/150 (8.0%); BBF-14 — 3/90 (3.3%); 15-PGDH — 1/30 (3.3%); MBP — 0/90 (0.0%).
- Confidence score calibration: The co-folding model's confidence scores were discriminative within a target (top-ranked designs bound 49% of the time vs. 26.8% overall) but blind across targets, providing no warning for the MBP and BBF-14 failures.
Industry Insight
- Demand per-item distributions, not just means: Any agentic AI benchmark reporting a single aggregate pass rate should be treated as incomplete. The variance across items is the missing half of the result. Practitioners should request or infer per-item difficulty distributions before making deployment decisions based on published scores.
- Budget and scope are confounded in agent performance claims: The 35.1% single-target hit rate cannot be attributed to narrow scoping alone — it came with 2.8× the compute budget per target. Organizations should be skeptical of claims that focusing an agent's scope is a free improvement; resource allocation changes are likely contributing factors.
- Uncalibrated self-assessment is a systemic risk for agentic discovery: When an agent's internal confidence scores cannot distinguish solvable from closed targets, the agent will confidently ship failures at scale. This pattern — high average performance, enormous variance, and uncalibrated self-assessment — is likely to appear across agentic AI benchmarks in 2026 and beyond, making independent validation a non-negotiable component of any deployment pipeline.
Disclaimer: The above content is generated by AI and is for reference only.