When organizations evaluate AI agents, they typically compare large language models (LLMs).
However, new research from Lasso suggests another component deserves equal attention: the agent harness.
In their study comparing Anthropic’s Claude Agent SDK with the open-source deepagents framework built on LangGraph, researchers held the model, prompt, tools, and attack targets constant while changing only the runtime environment.
The results showed that the same model could perform dramatically differently depending on the harness managing its execution.
Key takeaways
- AI agent harnesses can influence offensive security performance using the same underlying model.
- Runtime architecture affected attack success rates, reliability, and tool execution across multiple frontier LLMs.
- AI benchmarks may measure the combined performance of the model and harness rather than the model alone.
- Researchers found autonomous AI agents frequently misclassified failed attacks as successful, highlighting the need for independent validation.
- Selecting the right AI harness is becoming as important as choosing the underlying LLM for enterprise deployments.
Harnesses can influence attack success
An AI agent consists of more than just an LLM.
According to the researchers, an agent also includes its prompt, available tools, and the harness responsible for orchestrating execution.
This harness manages context, tool calls, memory, planning, and execution loops, effectively shaping how the model behaves throughout a multi-step task.
To measure the harness’s impact, researchers tested five frontier models across 20 attack scenarios spanning finance, healthcare, legal, customer support, and education.
The models attempted prompt extraction, sensitive information disclosure, and harmful content generation against simulated applications.
Across 1,000 autonomous attacks, average success rates appeared relatively close between the two harnesses.
However, those averages masked substantial differences at the individual model level.
In many cases, one harness successfully completed attacks that the other failed entirely, despite using the same underlying model.
Performance varied widely across models
The study found that harness selection mattered far more for some models than others.
Claude Sonnet 5 produced nearly identical attack success rates regardless of the harness, suggesting the runtime had little influence on its behavior.
DeepSeek-V4-Pro achieved similar overall success across both frameworks, but each harness excelled in different attack categories.
One proved substantially better at system prompt extraction, while the other generated higher success rates for harmful content attacks.
The most dramatic result involved Kimi-K2.6.
Running under Claude Agent SDK, the model achieved approximately a 1% attack success rate.
Using deepagents, that figure increased to roughly 24% without changing the model, prompt, or tools.
Researchers traced the difference to protocol translation between the harness and the model, which interrupted tool execution and prematurely ended many attack sequences.
Why harnesses affect AI security evaluations
The findings have implications beyond offensive security testing.
Organizations increasingly benchmark AI models to evaluate autonomous capabilities, but the research suggests those benchmarks often measure a model-and-harness combination rather than the model alone.
Different harnesses package prompts differently, structure tool descriptions differently, and manage execution loops in distinct ways, potentially influencing both capability and reliability.
The researchers also found that allowing autonomous agents to evaluate their own performance produced misleading results.
More than half of the attacks that agents labeled as successful were later determined to be failures by an independent evaluation process, suggesting organizations should rely on external validation when assessing autonomous agent performance.
Choosing the right harness depends on the workload
The study does not conclude that one harness is universally superior.
Claude Agent SDK demonstrated cost advantages when paired with Anthropic models because of native integration and prompt caching, while deepagents provided greater flexibility across multiple model providers.
The researchers highlighted that organizations should evaluate the harness alongside the model rather than assuming runtimes are interchangeable.
For enterprises deploying AI agents or conducting autonomous red teaming, this research suggests that runtime architecture can materially affect both security testing outcomes and operational performance.
As AI agents become more capable, selecting the right harness may prove just as important as selecting the underlying model itself.
These findings reinforce that deploying AI agents is no longer just a technical decision — it is also a governance challenge.