A demo shows an offensive agent finding a bug.
A real engagement asks more: can it log in, stay in scope, and give a tester findings worth reading?
At DEF CON’s Adversary Village, Dominika Pietrzak (RAID leader) and Ibai Castells (Senior Red Team Hive Member) shared what a small team at CovertSwarm learned building offensive agents that attack web applications alongside experienced testers.
Four areas shaped the result: reusability, reliability, evaluation, and safety.
Video credit: DEF CON, licensed under Creative Commons Attribution.
What makes offensive agents reusable?
Work done once should pay off again. The framework can use different models, because some handle client-side flaws better and others handle server-side ones.
Once one agent works well, such as a cross-site scripting tester, its prompt becomes a template for the next. Tools like the browser are shared across agents instead of rebuilt for each.
What keeps agents reliable in a live engagement?
Authentication was one of the biggest challenges. Modern logins involve redirects, MFA and passkeys, and they gate the interesting functionality. Early attempts with curl managed only basic flows. The fix was a dedicated subsystem with a vault and a knowledge base of login flows, tracking session expiry and cookies.
Validation is the other half.
During the engagement, deterministic gates check that a finding isn’t fabricated. Before anything reaches a client report, a human validates it.
The goal is a low noise-to-signal ratio, so testers don’t become triagers. Structured output makes agent reports consistent and easy to parse, and retries keep long runs from crashing when a model API or tool fails.
How do you test an agent before a client sees it?
Start with realistic internal labs. The team built labs with specific functionality and made them as lifelike as possible, because models are getting better at spotting test conditions.
Can the agent get through redirects, finish a login flow or handle MFA without mailbox access? Better to find out in a lab than mid-engagement.
Bug bounty targets come next, for wider variety. Some benchmarks stay private so they don’t end up in training data.
The speakers also found little industry guidance on evaluating agents.
They suggest three questions: what to measure, how to measure it, and what makes a result reliable. Counting the tasks a planning agent produces, for example, rewards volume over quality.
What keeps an attacking agent safe?
Adversaries don’t worry about safety. Offensive teams do.
Several controls help: human checkpoints, plus an ask-human tool any agent can call mid-run; a deterministic proxy that keeps agents and their tools inside the scope the consultant sets; and a kill switch the consultant can use at any moment.
Where do agents fit in an offensive team?
Beside the people. In the speakers’ experience, agents are a force multiplier, not a replacement. Consultants hack alongside them like a sidecar.
RAID agents support the Swarm’s employed ethical hackers in offensive engagements. People still lead the attack.
Watch the full talk above for the detail behind each lesson.
