A new research framework shows that autonomous penetration-testing systems can dramatically improve accuracy and efficiency by replacing large language models (LLMs) with lightweight decision models at key judgment moments. Instead of asking the same generative AI that wrote an exploit narrative to also judge whether it's real, researchers propose inserting fast, specialized classifiers that return calibrated, auditable verdicts. This architectural shift addresses a fundamental problem in AI-driven security testing: the same model that generates plausible-sounding attack descriptions often inflates severity scores and wastes computing resources evaluating irrelevant attack surfaces.

What's the Problem With LLMs Making Security Decisions?

When large language models handle both exploit generation and verdict-making in autonomous penetration-testing harnesses, three critical failures emerge. First, calibration collapse occurs when a model's confidence in its own output doesn't match reality; a fluent narrative about a critical SQL injection vulnerability might rest on a parameter that was actually escaped and harmless. Second, class-driven severity assigns risk scores based on vulnerability type rather than demonstrated impact, inflating threat levels and eroding client trust. Third, compute waste happens when agent selection relies on LLM reasoning to evaluate attack surfaces that a lightweight classifier could dismiss in milliseconds.

The stakes are high. Current autonomous pentest systems already achieve impressive capability metrics: multi-agent harnesses reach 76.9% success on benchmark challenges, while plain coding agents with frontier models hit 92.3% under model scaling alone. However, capability and assurance are orthogonal concerns. A system can find vulnerabilities effectively while producing reports clients cannot trust because findings lack grounding in evidence or severity scores lack credibility.

How Do System One Models Fix This?

Researchers propose integrating "System One" decision models, a term borrowed from cognitive science that describes fast, pattern-matching judgment as opposed to slow, deliberative reasoning. These lightweight, non-generative classifiers return typed, probabilistically calibrated verdicts instead of free-form narratives. The framework formalizes four critical decision points where a System One model replaces LLM judgment with structured, auditable verdicts:

• Finding Adjudication: Determining whether a reported vulnerability is real or a false positive based on evidence quality rather than narrative plausibility.
• Severity Recalibration: Assigning risk scores based on demonstrated impact rather than vulnerability class, preventing inflated threat assessments.
• Agent Pruning: Selecting which attack agents to deploy by dismissing irrelevant surfaces quickly, reducing wasted compute tokens.
• Confirmation Loops: Validating findings through structured verification rather than asking the generative model to second-guess its own output.

The research team tested this approach on NeuroSploit, an open-source Rust-based penetration-testing harness, comparing single runs with and without a System One model called TypeSafe (Jev) against a 13-vulnerability web target. The observations showed meaningful differences in severity distribution, wall-clock time, and data-type-aware grading, though the team notes this exploratory case study does not constitute a controlled experiment with statistical power.

What Models Are Available Today?

The emerging landscape of System One models includes both proprietary and open-source options. The proprietary Jev family, developed by TypeSafe AI, includes browser-optimized variants like Jev-Ultrafast designed for rapid decision-making. The open-source Laya project offers an alternative for researchers and organizations seeking transparency and customization. Researchers surveyed published specifications for both families but noted that cross-benchmark comparisons do not yet extrapolate reliably to the penetration-testing domain.

The training paradigms underlying these calibrated decision models vary significantly. Approaches include Reinforcement Learning from Human Feedback (RLHF), Reinforcement Learning from AI Feedback (RLAIF), Reinforcement Learning from Calibrated Distributions (RLCD), and a proposed Reinforcement Learning from Human Verdicts (RLHV) method. Each training approach carries distinct implications for trust in security-critical pipelines, where miscalibration can lead to missed vulnerabilities or false alarms that undermine client confidence.

What's Next for This Research?

The research team sketches a future direction called Rave: a domain-adapted System One variant to be fine-tuned specifically on offensive-security decision distributions. This specialized model would address the unique patterns and edge cases that arise in penetration testing, where generic classifiers may lack context. The proposed training data requirements, evaluation protocol, and projected impact on harness assurance properties remain under development, but the framework suggests that domain adaptation could significantly improve both accuracy and efficiency in real-world security engagements.

The broader implication is that test-time compute, the computing resources spent during inference rather than training, may be better allocated to specialized decision-making rather than generative reasoning. By replacing expensive LLM inference at critical junctures with lightweight classifiers, autonomous security systems could reduce operational costs while improving the trustworthiness of their outputs. This architectural insight extends beyond penetration testing to any domain where AI systems must make high-stakes judgments based on their own outputs.