What happened
Collinear AI published CWE-bench v2 on November 3, 2026 (Hugging Face blog). The public set has 100 Harbor tasks covering 91 CWEs: an agent gets a real open-source repo checkout and a short instruction, and must find and fix a weakness without breaking existing functionality. Scoring is programmatic. Collinear also keeps a held-out set (contact Collinear’s cyber contact address on the post). On the published comparison of 11 models, Gemini 4 Argon leads at 67.8% pass@1 and 81.0% pass@4; Claude Opus 5.5 is 67.3% / 78.0%. Ninety-five tasks are solved by at least one model; the leader still fails 19% of the set on every attempt. On cost, DeepSeek V4.1 Flash (1017) reaches 57.0% pass@1 at $0.04 per rollout, Opus adds about ten points at $1.83, and Gemini edges ahead at $3.92 — Collinear’s Pareto frontier. Dataset, grader, and harness are open on the Hub (gated access).
Why it matters
Cyber capability claims often cite a private suite. CWE-bench v2 is a named public set with deterministic verifiers — the useful habit is pass@1 vs pass@4, open vs held-out, and dollars per rollout, not a single “solved cyber” slide.
Every task is a Harbor environment; Harbor can run the suite from the Hugging Face dataset repo without a local clone once access is granted.
Collinear says verifiers were shaped with LLM judge panels and hand review, and that memorized upstream patches should fail the checks.
Carbon’s October 28 DeepMind post cited 71.0% on Collinear’s CWE-bench under a different setup — treat that as a separate measurement from this v2 public leaderboard.