Cybersecurity firms and the makers of foundational models are improving the performance and efficiency of their AI systems. Quantifying that performance improvement is a challenge, however.
While claims of more efficient models with smaller token costs arrive every week, gauging whether those models perform as well as the more general-purpose members of an AI model’s family remains a challenge. The lesson for organizations: Test models against your code and workflows to make the best choice for your business requirements.
National sovereign AI and cyber-defense firm Dream Group, for example, used 220,000 vulnerability records, threat intelligence reports, detection rules, and other cybersecurity documents to train an 8-billion-parameter model — among the smallest modern models — to perform efficiently on cybersecurity tasks. The model — dubbed the Mid-trained Security Transformer (MiST) — improved average performance across 13 cybersecurity tasks, consisting mainly of multiple-choice and true-false questions about vulnerability analysis and cybersecurity domains.
The results show that stronger security capabilities do not necessarily need a more complex model; more specialized training and adding better software around the model both improve results, says Elad Ben Zaken, an AI researcher at Dream Group.
“I expect efficiency to keep improving in coding and security tasks through specialization and better workflows,” he says. “The model supplies capability; the harness determines how effectively it uses context and tools. Adding more of either can increase cost without improving results.”
Dream is not alone in finding ways to make models more efficient. When Anthropic released Opus 5.5 in September, for example, the company said that the model improved performance while costing 40% less than Opus 5 on “typical workloads.”
A Plethora of Benchmarks
However, for cybersecurity tasks, very little is typical. One model’s results may mean little to companies looking for an expert system to generate secure code or to effectively find weaknesses in the enterprise’s attack surface. On the offensive side, researchers are creating their own benchmarks to measure the performance of cybersecurity-focused models.
For offensive-security firm XBOW, the metrics that matter concern the ability to find and exploit vulnerabilities. Because different customers may want to use specific models, the company creates its own harness to improve outcomes regardless of the model used, says Albert Ziegler, head of AI at XBOW. While other organizations may want to use AI to analyze defensive telemetry and signals to boost detection of potential attacks, XBOW’s metrics focus on finding vulnerabilities and exploit chains.
“In its most atomic form, that’s the ability of the model to find an exploit — that is our bread and butter in terms of seeing what’s the value that the model produces,” Ziegler says. “For other companies, this bread and butter could obviously be something else.”
Anthropic’s Opus 5 Thinking leads the board on one metric, correctness, among large language models. Source: SonarQube’s LLM Leaderboard
Another benchmark, the SEC-Bench Pro, attempts to measure an AI agent’s ability to independently rediscover and trigger a known memory-safety bug in a browser engine or the Linux kernel, but only after the general location and type of bug are specified. Created by a research team at the University of Illinois Urbana-Champaign, the metric ranks Codex and GPT-5.5 as the current favorites, with an overall solve rate of 58%. It also finds that open-weight models tend to struggle, with GLM-5 only succeeding in 13 of the 344 test cases.
Other metrics measure the hacking capabilities of AI models over an entire attack chain. Government contractor Booz Allen, for example, has published its Cyber Weapons Index (CWI), which combines a model’s ability to find and exploit vulnerabilities with its effectiveness at executing attacks on a target. The latest version of the list puts OpenAI’s GPT-6 Astra on top with a CWI of 80.5, just edging out Claude Mythos with a score of 80.
“The CWI offers one empirical approach to offensive cyber capability — test models against a common enterprise network and score demonstrated actions using network evidence, rather than relying on what models say they can do,” says Eric Syphard, senior vice president and head of AI at Booz Allen. “There is no single measure that captures every security risk. Evaluations need to be designed around the capability being assessed and should make their methods, environment and success criteria clear.”
Similarly, the UK’s AI Security Institute (AISI) continues to benchmark the capabilities of AI models to complete a 32-step attack chain. So far, there are two models that can: Anthropic’s Claude Mythos Preview and OpenAI GPT 5.5.
In their own research, the Dream researchers found that expert curation of a primary knowledge database — a “seed corpus” — can be used to create domain-specific synthetic training data, thus avoiding relying on general knowledge databases for training specialized models. The latest MiST models, MiST-8B and MiST-30B-A3B, were evaluated using 13 different test datasets, including 1,072 multiple-choice questions on MITRE ATT&CK (SECURE-MAET) and 870 true-false questions on whether a particular piece of code is vulnerable (the PrimeVul P-C).
Security Benchmarks are Good, Internal Testing is Better
Overall, the various features, capabilities, and issues are hard to benchmark in any comprehensive way. While offensive security metrics and measurements of defensive capabilities are both evolving quickly, other areas — such as safety — are more difficult to benchmark. The AISI found that the most sophisticated models tend to execute unwanted actions, such as OpenAI’s GPT Astra conducting unsanctioned supply chain attacks, but overall safety measurements are more anecdotal than comprehensive.
Creating a metric for a model’s likelihood of taking unwanted actions, for example, is very difficult, XBOW’s Ziegler says.
“Benchmarking the possibility of the model to do harm is incredibly hard,” he says. “You run [the test case for the model] a thousand times, and it doesn’t do any harm, and then you run it the thousandth-and-first time, it drops your database. That thousand-and-first time negates all the first thousand runs that went well.”
In addition, just because a model is newer does not mean it’s more secure or has more advanced security capabilities, says Joe Tyler, general manager and AI researcher at SonarQube, a provider of code-quality and security tools. The company regularly runs the latest large language models (LLMs) through a testing pipeline to generate scores across a series of benchmarks — from cognitive complexity to vulnerability occurrence in its code — which it then posts to its LLM Leaderboard for Code Quality & Security.
The ranking can give companies a general sense of model performance, but the rankings should only be the starting point for companies evaluating LLM security performance, Tyler says. Instead, organizations should tailor their test pipelines to their own situations.
“Start with the task and the failures you can’t accept, rather than assuming the newest model is the right choice,” he says. “Public benchmarks help shortlist candidates, but the decision needs evaluations that reflect your own code and workflows.”
In addition, for most organizations, the choice of model may be constrained by the technology platforms they’ve adopted. For that reason, security teams should not necessarily focus on selecting specific models, but instead seek out AI-powered services that lead to more efficiency, better coverage, and faster response.
Attackers’ adoption of AI means companies will need to deal with a greater influx of vulnerabilities and faster time to exploitation, says Nico Waisman, XBOW’s CISO. As a result, organizations should invest in tools and processes that help security teams move faster, automate repetitive work, and reduce the time to remediation — focus on the impact of AI-based solutions, not their raw benchmarks, he says.
“Ultimately, I wouldn’t evaluate AI security tools based on which model they use or how impressive their demos look,” he says. “I would evaluate them based on measurable outcomes: Are they helping you identify real risks? Are they reducing your exposure? Are they making your team significantly faster? That’s where the security budget should go.”
