Collinear AI releases CWE-bench v2, 100 runnable tasks testing whether coding agents can patch vulnerabilities

Collinear AI, a start-up based in Sunnyvale, California, has published a second version of CWE-bench, its benchmark for checking whether AI coding agents can find and fix security flaws. The new tasks are on Hugging Face. A write-up of the release dates the announcement to November 3. The benchmark is named after MITRE's Common Weakness Enumeration (CWE), the standard catalogue of software weakness types.

What the benchmark contains

According to the dataset card, version 2 has 100 tasks covering 91 different CWE types. The tasks span eight languages: Go (27), Java (15), JavaScript (15), C++ (14), C (10), Python (10), Rust (5) and TypeScript (4). In each task, an agent gets a codebase containing one weakness and has to fix it without breaking how the software normally behaves. Every task comes as a self-contained directory with an instruction brief, a Dockerfile, a reference fix and a verifier, and it runs on the Harbor evaluation framework.

Scoring is all or nothing. A task counts as solved only if the verifier confirms that every exploit input is closed and every legitimate case still passes. If the environment fails to build, the run is thrown out rather than scored as a failure. Agents get an hour per task, two CPUs and 2 GB of memory. During the agent phase they have no network access beyond the model's API.

Early results and access

The dataset card lists results for open-weight models as pass@1, averaged over four attempts. DeepSeek V4.1 Flash scores 57%, Tencent's Hy4-preview 52%, Mistral Large 4 46.3% and GLM-5.3 44%. The explainx.ai summary of Collinear's announcement adds two closed models, Gemini 4 Argon at 67.8% and Claude Opus 5.5 at 67.3%. Those figures come from Collinear's own harness and have not been reproduced independently.

The dataset is released under the Apache 2.0 license, but it is gated. Users have to sign in and accept its conditions, and the card asks that the reference fixes never be shown to the agents being tested. A held-out set stays private and is available on request.

Context

The benchmark has grown quickly. In a press release in September, which was labelled a paid placement, Collinear presented an earlier version with 100 tasks covering 54 weakness types. No agent solved a majority of those tasks. The bigger, runnable set makes it easier for others to measure defensive skill. Like any public benchmark, though, it could end up in training data over time. The private held-out set gives Collinear a separate check that is not exposed in the same way.