Cyber

Can open-source agents replace pentesters: Testing Strix and PentAGI

We test open-source AI agents Strix and PentAGI on real infrastructure. Analyzing performance, false positives, and token consumption.

·2 min read
Can open-source agents replace pentesters: Testing Strix and PentAGI

With autonomous AI agents becoming a major discussion point in cybersecurity, many wonder if free open-source tools can actually replace human pentesters in real-world scenarios.

We decided to put two prominent open-source LLM agents, Strix and PentAGI, to the test against real infrastructure to evaluate their autonomy, accuracy, and overall practicality.

  • Strix focuses heavily on web application analysis and code review with strict verification of results via PoC.
  • PentAGI acts as a more comprehensive system capable of full-scale penetration testing with extended context support.
  • Both tools operate inside isolated Docker sandboxes and require predefined targets to function efficiently.

Architecture and operational differences of Strix and PentAGI

Both agents are capable of planning checks, executing CLI tools, writing scripts, and generating reports. However, high-level reconnaissance and asset discovery should still be performed manually, as the agents are optimized for targeted execution rather than broad asset enumeration.

Read alsoWorkOS breached by escaping AI hacking agent during security test

Strix employs multiple sub-agents to verify findings, resulting in a low false-positive rate. PentAGI offers broader execution capabilities and accepts free-form prompts, but its higher autonomy introduces risks of wandering outside defined rules of engagement.

Test results, accuracy, and false positives

During the evaluation, Strix produced almost no false positives but tended to overestimate the business criticality of findings. Out of 15 discoveries, most were minor informational issues wrongly flagged as significant risks.

PentAGI generated a massive report containing 296 vulnerabilities, including dozens marked as critical and high. However, subsequent manual checks revealed that none of these major findings were actually exploitable.

Resource consumption and efficiency

The pursuit of autonomy comes at a significant computational cost. Strix consumed over 145 million tokens due to extensive context sharing among dozens of sub-agents.

PentAGI utilized roughly 31 million tokens, but its aggressive context compression techniques carry the risk of losing crucial details during extensive infrastructure scans.

Conclusion and industry implications

The research demonstrates that current open-source agents cannot fully replace a human pentester. While they serve as powerful auxiliary tools for automation, scoping control, false-positive triage, and risk assessment remain strictly human responsibilities.

For businesses and development teams, integrating such AI agents requires rigorous monitoring, strict adherence to rules of engagement, and thorough manual validation before deploying any remediation strategies.

Source: Все публикации подряд на Хабре

Related articles