The cybersecurity capabilities of large language models (LLMs) are often summarized by a single benchmark score.
However, new research presented at the BSides 2026 conference suggests those scores reveal little about how AI agents actually behave during offensive security tasks.
Rather than focusing solely on benchmark solve rates, researcher Tarun Koyalwar analyzed the decision-making processes of AI agents during offensive security tasks.
His research examined what the models know, how they reason through attacks, where they fail, and what those behaviors reveal about future cybersecurity assessments.
Key takeaways of the LLM benchmark research
- Traditional benchmark scores reveal little about how AI agents actually perform offensive security tasks.
- Many LLM failures stem from execution gaps rather than a lack of cybersecurity knowledge.
- Leading open-weight models performed nearly as well as frontier closed models under identical testing.
- AI agents relied more on pattern recognition than traditional penetration testing methodologies.
- Behavioral telemetry provides deeper insight into AI security risks than benchmark scores alone.
Benchmark scores miss the bigger picture
Koyalwar’s research evaluated both open- and closed-weight LLMs using a modified version of the Argus validation benchmark consisting of 54 usable black-box web application targets.
Each model received only a minimal prompt instructing it to attack a target application without source code or implementation details.
Rather than measuring only whether a model successfully completed a challenge, the research examined every step the models took throughout the attack process.
The findings suggest that conventional cybersecurity benchmarks oversimplify AI capability.
A successful exploit does not reveal whether the model found the vulnerability through systematic testing, pattern recognition, or an unintended attack path.
Likewise, failed attempts often reveal models that understood the correct attack technique but could not successfully execute it.
AI knows more than it can execute
One of the study’s most notable findings is that many LLM failures are not caused by a lack of cybersecurity knowledge.
Instead, models frequently identified the correct vulnerability, referenced the appropriate exploitation technique, or even named the relevant CVE identifier before failing to complete the attack.
Examples included models that repeatedly recognized the correct attack chain yet made small execution mistakes, such as submitting malformed requests or repeatedly targeting the wrong host despite identifying the correct one earlier in the session.
According to Koyalwar, this highlights a significant gap between knowing what to do and successfully performing the required actions.
The research also notes that current benchmark scores cannot distinguish between genuine capability limitations and what researchers refer to as an “elicitation gap,” where a model possesses the capability but fails to demonstrate it during a particular run.
Open models narrow the performance gap
Another observation is how closely leading open-weight models performed compared to frontier closed models.
Under identical testing conditions, the collection of open models successfully completed 52 of the 54 benchmark targets, while the closed-model group completed 48.
The strongest individual performer among the open models was Kimi K3, although multiple open models collectively achieved the highest overall coverage.
Cost also proved to be an unreliable indicator of performance.
According to the research, execution costs varied by nearly two orders of magnitude, and neither open nor closed licensing consistently predicted operational expense.
AI agents do not think like human pentesters
Rather than following traditional penetration testing methodologies that emphasize reconnaissance, enumeration, and exploitation, many models relied heavily on recognizing vulnerability patterns.
The research found that models often guessed likely vulnerabilities immediately after observing application behavior and frequently guessed correctly.
This pattern recognition approach enabled models to reach successful outcomes more efficiently than a structured human-style methodology in many cases.
Koyalwar suggested this behavior appears to emerge naturally from broad exposure to publicly available cybersecurity knowledge rather than explicit offensive security training.
Behavioral analysis improves AI risk assessment
The study also identified instances where models interacted with components of the testing environment rather than the intended target application.
According to the research, these behaviors typically occurred when models treated supporting infrastructure as part of the challenge instead of intentionally attempting to escape containment.
More importantly, models often failed to recognize or report when they achieved success through unintended side channels, such as exposed credentials or benchmark artifacts.
Traditional scoring methods recorded these as successful exploits even though they did not demonstrate the intended offensive capability.
For security teams evaluating AI-powered offensive security tools, the findings suggest that behavioral telemetry may provide more actionable insight than benchmark percentages alone.
As organizations adopt autonomous AI for offensive security, understanding how agents reason and where they fail will become increasingly important.
Increasing AI autonomy is forcing organizations to rethink the trust models that govern agent behavior.