Vals: DeepSeek V4.1 Flash beat GPT-6 Sol on its offensive cybersecurity task, using about five times the tokens

Vals AI says DeepSeek V4.1 Flash 1017, an open-weight model, completed more objectives than OpenAI's GPT-6 Sol on a multi-step offensive cybersecurity task built on simulated corporate networks. Vals posted the result on X on October 20 as the first finding from a cybersecurity benchmark it plans to publish in full in the coming weeks, calling it an indicator of the cyber risk a model poses in the hands of a bad actor.

THE NUMBERS

Objectives completed out of 21, from Vals' chart: DeepSeek V4.1 Flash 1017 16.3 and GPT-6 Sol 15.7, each the average of three runs. The same chart shows Grok 4.7 at 12, Kimi K3 at 9, GLM 5.3 Flash at 5, and Claude Opus 5.5 at 1, which Vals marks as having refused partway through the run. Vals says it tested Sol through OpenAI's Daybreak trusted access program.

THE CATCH

Tokens. On Vals' log-scale chart, DeepSeek reached its score after close to 300 million cumulative tokens, against roughly 60 million for Sol, by our reading of the chart, so about five times as many. @scaling01 made the same point on X: a narrow win on score, at several times the cost in tokens.

WHY IT MATTERS

V4.1 Flash's weights are public under the MIT license, so the model that scored highest here can be downloaded and run by anyone. The result lands a day after the US Department of Homeland Security ordered DeepSeek products out of federal civilian agencies on supply chain security grounds.

WHAT WE DO NOT KNOW

The tasks, the scoring rules and the full model list, until Vals publishes the benchmark. This is one early result from three runs per model, shared before the full evaluation.