Vals publishes CyberOffense: gated GPT-6 Sol completes 80.8% of a simulated network attack, open-weight DeepSeek V4.1 Flash 72.8%

Vals AI has published its CyberOffense benchmark in full. Seven AI models ran as autonomous agents against two simulated corporate networks, with no human help. GPT-6 Sol, which OpenAI offers for this kind of work only through its restricted Daybreak program, completed the most: 80.8% of objectives on Vals' scoring. DeepSeek V4.1 Flash 1017, an open-weight model anyone can download, scored 72.8%. Vals released the benchmark on November 6; its page is dated October 23.

THE NUMBERS

Objectives completed, from Vals' table. Nova has 21 objectives and Lighthouse 31, and the overall score averages the two ranges' shares equally.
GPT-6 Sol Daybreak: Nova 15.7 (average of three runs, 12 to 19), Lighthouse 27.
DeepSeek V4.1 Flash 1017: Nova 16.3 (average of three runs, 15 to 18), Lighthouse 21.
Kimi K3.1 Pro: Nova 13, Lighthouse 20.
Meta Wonder: Nova 7, Lighthouse 28.
Grok 4.7: Nova 12, Lighthouse 16.
Kimi K3: Nova 9, Lighthouse 10.
GLM 5.3 Flash: Nova 5, Lighthouse 5.
By the same equal-weight rule, our arithmetic puts Kimi K3.1 Pro, whose weights went public on November 4, at 63.2% and Meta Wonder at 61.8%. Meta Wonder has the highest Lighthouse count on the table, 28 of 31, and one of the lowest on Nova.

WHAT CHANGED SINCE OCTOBER 20

Our October 20 story covered Vals' first result: Nova alone, six models, and a promise to publish the full benchmark. The published version adds the second range, Lighthouse, where an agent has to take over one company's network and use what it finds there to reach a second company. It also adds Kimi K3.1 Pro and Meta Wonder. Claude Opus 5.5, which the October chart showed at 1 objective after refusing partway through, is not on the published table.

THE COST

Vals says DeepSeek scored level with Daybreak on Nova for under $3 a run, against about $22 for Daybreak. It completed nothing in its first 25 million tokens, made most of its progress after 100 million and used two to four times as many tokens as GPT-6 Sol, which reached 25 of Lighthouse's 31 objectives inside 100 million tokens. Sol's best Lighthouse run took 27 objectives in under three hours.

HOW IT IS SCORED

Each model runs in the open-source Pi agent on an isolated Kali Linux machine that can reach only the model provider and the range. Every objective leaves a hidden marker, and Vals credits it only when that marker appears in the recorded command output, never on the model's own claim. A run counts the objectives completed within 500 million cumulative tokens, with a 48-hour limit. Vals says every run ended on its own before either limit, and that it reviewed every run and found no shortcuts.

WHERE THEY STALLED

No model completed Lighthouse's final objectives, and only one run reached the final stage of Nova, per Vals. GLM 5.3 Flash spent roughly 36 hours and a thousand turns on one range, much of it polling for a background job that never returned, and finished at five objectives.

WHAT IS SOURCED

Every figure above comes from Vals' CyberOffense page and its November 6 thread on X, except the two percentages marked as our arithmetic. Vals' thread says WIRED's Will Knight wrote about the work; this story does not rely on that article.

WHAT WE DO NOT KNOW

How these agents would fare against a real network with defenders: the ranges are simulated, with planted weaknesses and no one watching. No other lab has replicated the runs, and the Nova figures for the top two are three-run averages whose ranges overlap. Vals says it will add ranges and models, without naming which.