AI patches stop crashes but fail maintainer tests in Northwestern study

Researchers at Northwestern University evaluated AI coding agents on 112 real-world security bugs from 84 open-source projects written in C/C++, Go, and Rust. With Claude Opus 4.7, the AI agents stopped the crash in about 90% of cases but passed the original maintainers’ test suites only about 61% of the time.

The study highlights a persistent gap between crash prevention and maintainer expectations. Approximately one-third of the patches that stopped crashes still failed the maintainers’ tests, revealing a fundamental flaw in how AI-driven vulnerability remediation is graded. Many existing benchmarks focus solely on whether the proof-of-concept input still triggers a crash, allowing patches that block the crash path without addressing the underlying flaw to be counted as successful.

Maintainers require more than just crash prevention. They expect patches to preserve intended behavior, address the root cause, and withstand nearby inputs not covered by the original proof of concept. The study also found that AI agents could retrieve correct fixes by searching the web or project history, suggesting benchmark scores may reflect answer retrieval rather than independent reasoning.

The researchers called such patches ‘strictly worse than no fix at all,’ underscoring the risks of relying on benchmarks that overlook deeper validation. Design flaws in benchmarks—such as hidden background checks, test-file collisions, or agents adding their own tests—can also cause correct patches to be scored as failures.

A review of 13 benchmarks and datasets used to evaluate AI patching found that each of the five built for AI agents left at least one of nine known pitfalls unaddressed. An independent validation study of 209 cases across 20 projects showed that over 40% of patches accepted by basic tests failed additional tests based on developer fixes.

The findings suggest that while AI agents can effectively halt crashes, their patches often fail to meet the rigorous standards of maintainers. This discrepancy raises questions about the reliability of AI-driven vulnerability remediation and the need for more comprehensive testing frameworks.

Providing detailed build and test instructions improved crash-test pass rates for C and C++ bugs from 72% to 92%, but maintainer-test pass rates rose only from 28% to 36% for those languages. Across all languages, the maintainer-test pass rate increased slightly, from 41.1% to 43.8%, indicating that even with better instructions, AI patches still struggle to meet maintainer standards.