A new study from 1Password’s Off-by-1 Labs suggests that organizations should be cautious about relying on large language models (LLMs) to autonomously remediate software vulnerabilities.
After evaluating more than 6,000 AI-generated security patches across six recently disclosed, high-impact vulnerabilities, researchers found that fully successful patches were the exception rather than the rule.
Key takeaways of the AI patch research
- 1Password found AI-generated security patches failed to fully fix vulnerabilities in more than half of tested cases.
- Only 26% of AI-generated patches completely remediated vulnerabilities without introducing unintended side effects.
- Incorrect remediation guidance significantly reduced patch success, while richer prompts improved results but did not eliminate new security risks.
- Researchers found AI models often produced fragile or incomplete fixes by addressing symptoms instead of root causes.
- 1Password recommends keeping experienced security engineers involved in reviewing AI-generated patches before deployment.
Testing AI patching performance on complex vulnerabilities
The research evaluated ChatGPT 5.5 and Claude Opus 4.8 using a newly developed testing framework called FLAWED (Fix-Like Artifacts With Embedded Defects).
Researchers tested both models against six complex vulnerabilities affecting widely used open-source software, including Linux, Google Chrome, Apache ActiveMQ, Spring AI, Exim, and Gemini CLI.
The goal was not to compare the two models directly, but to assess how reliably frontier AI systems can generate secure software patches.
“One of the clearest findings from our research is that generating a patch isn’t the same as fixing a vulnerability,” said Keith Hoodlet, Head of Off-By-1-Labs at 1Password, in an email to eSecurityPlanet.
He explained, “After filtering out what we classified as ‘fragile’ patches, only about 17% of the Scenario 1 patches actually addressed the underlying root cause rather than masking the immediate issue.”
Keith added, “AI is becoming an incredibly powerful tool for software security, but our research shows that expert human review remains essential to verify that a patch is both effective and secure.”
AI-generated security patches frequently fell short
Across 6,080 valid patch attempts, only 26.0% completely fixed the targeted vulnerability without introducing unintended side effects.
Approximately 20.1% resolved the vulnerability but altered application behavior, while 49.3% failed to fix at least one exploit path.
Researchers also found that 2.2% of generated patches introduced a new exploit path.
According to the researchers, these results indicate that fully autonomous AI-generated security patches remain unreliable for production environments.
They concluded that human review remains essential because AI-generated fixes frequently addressed only part of a vulnerability or created new security issues while attempting remediation.
Prompt quality significantly influenced patch success
One of the study’s interesting findings involved the quality of guidance provided to AI models.
When models received accurate remediation guidance, they achieved a 65.0% fix-success rate.
However, incorrect guidance reduced successful remediation to just 15.2%, substantially worse than providing no guidance at all, which resulted in a 50.4% success rate.
Researchers observed that models often followed flawed instructions even when evidence generated during execution contradicted those instructions.
The study also found that providing richer, more detailed prompts improved performance. The highest-quality prompts increased fix-success rates from 51.8% to 76.3%.
However, richer prompts did little to reduce the likelihood of introducing new vulnerabilities, suggesting that additional context alone cannot eliminate AI’s tendency to produce incomplete or fragile fixes.
Common AI patching failures highlight the need for human oversight
Researchers identified several recurring failure patterns.
Models frequently exhibited “tunnel vision,” focusing narrowly on the exploit demonstrated in a proof-of-concept while overlooking similar vulnerable code paths elsewhere in the application.
In many cases, AI systems also fixed the immediate symptom without addressing the underlying root cause, resulting in patches that passed testing but left the software vulnerable through alternate execution paths.
Another notable finding was the prevalence of fragile fixes.
Among patches that successfully resolved the original vulnerability, approximately 37.5% were classified as fragile because they relied on defensive checks instead of fully eliminating the underlying vulnerability.
Bottom line
Based on these findings, 1Password recommends that organizations validate AI-assisted patching within their own codebases before adopting automated remediation workflows.
The researchers also emphasize that experienced security engineers should continue reviewing all AI-generated patches, particularly for complex vulnerabilities.
Additionally, they advise developers to avoid supplying uncertain remediation guidance, as inaccurate instructions can reduce patch quality.
These concerns extend beyond individual patches, underscoring the broader need to secure the software supply chain against compromised code, dependencies, and automated development workflows.