Automation introduces new risks
In another post, we made the case for continuous, automated penetration testing as the answer to AI-accelerated attacks. This blog is the other half of that argument: Automation only delivers if the right people are wrapped around it.
By now, most of us have experimented with generative AI, spinning up everything from email messages to presentation decks. Large language models generate highly polished content at a speed none of us could achieve. But you’ve probably also seen that AI-generated deliverables can look impressive while being flat-out wrong.
That’s why having a human in the loop is critical to AI-enabled workflows. Penetration Testing as a Service is no exception.
Speed without judgment moves problems downstream faster
Humans can’t match the sheer speed of automated penetration testing tools, which can combine several low- and medium-severity vulnerabilities into a working exploit chain in minutes. That same work might take a skilled person a day or longer. And while there’s no question that kind of speed is genuinely useful, relying solely on automation introduces another set of risks.
When any automated system reports a working exploit, each claim needs confirmation that the exploit functioned as described. Today’s tools do their best to weed out false positives, but human oversight really keeps this process humming. Without a person reviewing that screenshot or log, false positives can end up in a report and get treated the same as confirmed findings.
Once false positives enter the prioritization process, they skew decisions about what to fix first. Teams can waste time chasing a long list of red herrings. Meanwhile, a tool that misidentifies a vulnerability as low priority or recommends the wrong fix can do real damage if nobody catches it.
Keep security experts in the loop
Automated penetration testing must be surrounded by experienced, certified penetration testers.
These experts need an offensive security mindset, with training to think like an attacker. Only they can review automated findings to confirm that the testing methodology was sound and that the evidence supports the conclusion.
Dedicated security experts also bring something a system cannot: knowledge of what the finding means for a specific business. For example, an automated tool doesn’t know that one server holds an organization’s most sensitive intellectual property while another is a low-value laptop. Both may have the same technical vulnerability, but only an expert who has operational context can say which matters more. That context builds over time, as operators supporting an organization month after month get progressively more knowledgeable about that business.
Clear communication matters, too. Unless they’re translated into an organization’s language and operating context, even technically accurate findings can be misread or ignored.
AI security best practices: Where to start
Automated tools are a significant advance in how quickly organizations can find exploitable vulnerabilities. But evidence needs validation. Findings need business context. And recommendations need to come from an expert who understands the technical and human sides of the environment they’re protecting.
For organizations exploring Penetration Testing as a Service, a Cybersecurity Maturity Assessment focused on your Continuous Threat Exposure Management (CTEM) posture is a reasonable starting point. It gives you an objective review of your current cadence for vulnerability management, patch management, and penetration testing.
Talk with a CBTS security specialist about how human-validated Penetration Testing as a Service works and what your current findings might be missing.