AI pentesting vendor evaluation guide for AI agents

Quick answer:

Ask for a live failure demo, not a maturity slide. A credible AI pentesting vendor for AI agents should be able to show four things in one meeting: how it tests prompt injection through untrusted content, how it constrains tool abuse and outbound actions, what evidence it captures when an agent misbehaves, and how it reruns the exact scenario after a fix. If a vendor mostly talks about "agent security posture" without showing a replayable failure path, you are probably looking at generic AppSec wrapped in fresh terminology.

The shortest buying rule is this: prefer vendors that can produce a bounded proof of exploit, name the exact control that failed, and hand back a retest artifact after remediation. Everything else is marketing garnish.

If you want more category context before you shortlist anyone, start with the broader blog, compare operating models in compare, and keep the commercial boundary honest by checking pricing and download before you confuse "tool," "service," and "platform."

Why evaluating AI agent vendors is different now

AI agents fail in ways that normal pentest buying motions do not capture well. A classic web or API assessment asks whether an attacker can bypass auth, reach sensitive data, or gain code execution. Those questions still matter. They are just not enough once a model can browse the web, read internal content, call tools, write data back into systems, or move between steps with partial autonomy.

That is why buyer language has gotten muddy. Plenty of vendors now say they test "agents," but they mean very different things:

• Some run prompt-level adversarial testing against the model and stop there.
• Some bolt a few LLM checks onto a conventional application pentest.
• Some actually test the full agent loop, including untrusted inputs, tool permissions, side effects, approvals, and recovery.

Those are not interchangeable offers.

Recent official guidance makes the gap obvious. On March 11, 2026, OpenAI wrote that strong real-world prompt injection attacks against agents increasingly resemble social engineering, not just simple string overrides, and argued that the real job is to constrain impact even when manipulation succeeds. On November 24, 2025, Anthropic made a similar point about browser agents: every page, embedded document, and dynamically loaded element can become an attack surface when the model is allowed to act. NIST's adversarial machine learning taxonomy, published on March 24, 2025, is useful here because it treats misuse, poisoning, privacy, and evasion as separate classes of failure rather than one vague "AI risk" bucket.

Put differently, a vendor that cannot explain agent risk as a system problem will miss system-level bugs.

What a serious vendor should test in an AI agent

Any vendor can hand you a checklist. What you need is a testing method that reaches the parts of the stack where agents actually go wrong. A strong engagement usually covers five layers.

1. untrusted input paths

The vendor should test both direct and indirect prompt injection. That includes malicious user messages, but it also includes poisoned documents, CRM notes, issue descriptions, web pages, ticket bodies, hidden HTML, retrieved knowledge chunks, and tool metadata. OWASP's prompt injection cheat sheet is blunt on this point: the dangerous version is often the one that reaches connected tools and APIs, not the one that merely makes the model say something strange.

If the vendor only demos a chat box attack, that is shallow coverage.

2. tool and action boundaries

The hard question is never "can the model be influenced?" Of course it can. The hard question is what the model is allowed to do afterward. The vendor should test whether the agent can call the wrong tool, call the right tool with the wrong arguments, cross permission boundaries, trigger external requests, or expose sensitive data through side channels such as URLs, form posts, attachments, or logs.

This is where weak vendors start hand-waving. They talk about guardrails. You need them to talk about sinks.

3. runtime isolation and environment controls

A browser agent, coding agent, or internal operations agent lives inside a runtime with file access, network reach, secrets, and installed skills or plugins. OWASP's Agentic Skills Top 10 focuses on the behavior layer for a reason: the tool abstraction is not the whole system. A vendor worth paying should be comfortable testing sandboxing, network restrictions, skill supply chain risk, execution approvals, and audit logging, not just model prompts.

If your agent runs locally, this matters even more. Local agents often inherit workstation trust, broader file access, shell access, and convenient shortcuts that nobody would allow in a production API.

4. evidence capture

You need more than screenshots. The output should show the original input, the relevant trace or request sequence, the action the agent attempted, the control that failed, and the blast radius if the failure had not been bounded. Without that level of evidence, engineering cannot reproduce the issue and procurement cannot compare vendors fairly.

5. retest discipline

The first report is not the product. The useful part is whether the vendor can rerun the exact scenario after you patch the prompt, narrow the permission, move the action behind approval, or tighten the runtime. If they cannot retest cleanly, the assessment becomes a one-time narrative instead of a security workflow.

A practical scorecard you can use in real buying calls

You do not need a forty-line procurement spreadsheet. You need a scorecard that forces concrete answers. Rate each vendor from 1 to 5 on the criteria below and require a proof point for each score.

| Criterion | What "good" looks like | What weak answers sound like | | --- | --- | --- | | Agent-specific threat coverage | Tests indirect injection, tool misuse, data exfiltration, approval bypass, and recovery | "We do jailbreak testing and prompt fuzzing" | | System boundary awareness | Covers model, tool layer, runtime, network, secrets, and human approvals | "We focus on the prompt layer first" | | Evidence quality | Provides traces, action logs, affected tools, and reproduction steps | "We provide screenshots and an executive summary" | | Safe exploit discipline | Uses bounded targets, explicit approvals, and narrow write scopes | "Our consultants will explore broadly" | | Retest workflow | Can rerun the same scenario after a fix without rebuilding the whole engagement | "That would be a separate follow-on" | | Local vs cloud realism | Understands workstation agents, self-hosted agents, and hosted agent stacks | "We mostly test SaaS copilots" | | Skill and plugin risk coverage | Reviews installed skills, manifests, dependencies, and permission surfaces | "We can look at that if needed" | | Freshness of method | Can explain what changed in its methodology after recent agent security guidance | "Our AI red team process is mature" |

That last row matters more than buyers think. The field is moving too fast for a vendor to coast on a 2024 LLM red-teaming deck. Ask what changed in the last six months. If they have no answer, their method is probably stale.

Questions that force a real demo

Most buyers let vendors stay abstract for too long. Do not ask, "How do you test AI agents?" Ask questions that require a visible workflow.

1. Show one indirect prompt injection case from first input to final blocked or unsafe action.
2. Show how you tell the difference between a noisy model response and a real security issue with side effects.
3. Show one example where the model was influenced but the system still held because permissions, approvals, or network controls contained the blast radius.
4. Show what evidence engineering receives, not just what leadership sees.
5. Show how you test plugin, skill, or tool installation risk if the agent can extend itself.
6. Show how you treat local agents differently from cloud-only agents.
7. Show one retest after remediation.
8. Show what you refuse to do during a proof of concept because the risk is too broad.

These questions do two jobs at once. They surface technical depth, and they expose whether the vendor has healthy operating discipline. In practice, the weak vendors fail on the second point first.

How to structure the proof of concept

A useful proof of concept is short, narrow, and slightly uncomfortable. It should give the vendor enough room to prove the method, but not enough room to hide behind process theater.

Pick one agent workflow that matters to your team. Good POC targets usually look like one of these:

• A browsing or research agent that reads untrusted web content and can summarize or submit data
• An internal support or operations agent that can query systems and update records
• A coding or automation agent with shell, file system, or repository access

Then define four required proof points:

1. One indirect injection path from untrusted content into model context
2. One action path that could create a material side effect
3. One containment control that should prevent or limit the unsafe action
4. One remediation retest after you adjust the control

Keep the scope bounded. Require written assumptions. Require approvals before any write-capable step. Require the final output to say what was tested, what was intentionally not tested, and why.

This is also where a lot of buying confusion clears up. Some vendors are really selling an expert-led assessment. Some are selling a product you operate. Some sell both, but one side is much stronger. If you compare them, compare them under the same POC target and same success conditions. Otherwise the slickest workflow wins, not the best testing.

Red flags that should slow the deal down

You can usually spot trouble early.

• The vendor treats prompt injection as the whole problem and barely talks about tool permissions or data exfiltration.
• The vendor cannot explain how it captures attempted side effects, only model outputs.
• The vendor has no opinion on skill, plugin, or extension supply chain risk.
• The vendor says approvals are optional because "the model usually behaves."
• The vendor wants broad production access before it can show value on a bounded target.
• The vendor has no strong story for local agents, browser agents, or coding agents.
• The vendor's deliverable sounds like an audit memo, not an engineering artifact.
• The vendor cannot describe a failed engagement and what it learned from it.

One softer red flag matters too: they keep sliding back into general AI governance language. Governance matters, but it is not a substitute for finding failures in the live system.

Where teams usually under-buy

Most teams under-buy in three places.

First, they under-buy on environment realism. They run a tidy demo against a hosted sandbox agent and learn very little about the messier agent flows that touch internal docs, desktop files, browser sessions, or internal APIs.

Second, they under-buy on retesting. They pay for discovery, patch the obvious issue, and never validate whether the new approval step, prompt split, or egress restriction actually held.

Third, they under-buy on internal enablement. The vendor may find a real issue, but the team still needs a way to replay it, regression-test it, and teach engineers what "safe enough" looks like in day-to-day agent work. If you are trying to compare those operating models, the relevant internal paths are the category pages in compare, the broader article archive in blog, and the self-serve product boundary in download and pricing.

That is one reason buyers should separate two different questions:

• "Who can find hard agent bugs for us?"
• "What workflow helps us keep them fixed?"

Sometimes the same vendor answers both. Often it does not.

How I would make the final decision

I would not choose the vendor with the prettiest maturity model. I would choose the one that makes me trust its failure evidence.

That usually means:

• It can show a realistic attack path against a workflow I actually run.
• It can tell me exactly which control failed.
• It can show what would have happened without containment.
• It can rerun the same scenario after a fix.

Everything else is secondary.

If two vendors look close, bias toward the one that is stricter about approvals, narrower about scope, and clearer about where its method does not work yet. In this market, honesty is a feature. Agent security is still young enough that fake certainty is more dangerous than admitted limits.

The other tie-breaker is operational fit. Some security teams need a third party to produce independent findings for leadership or customers. Others need a repeatable internal loop that lets engineers test agent changes weekly, not once per quarter. Be explicit about which problem you are solving. You can buy the wrong answer even from a technically competent vendor.

FAQ

What should an AI agent pentest vendor absolutely be able to show?

At minimum, it should show indirect prompt injection testing, attempted tool misuse, evidence of the resulting action path, and a retest after remediation. If it cannot show those four things, the engagement is probably too shallow.

Is prompt injection testing enough to evaluate an AI agent vendor?

No. Prompt injection is one important entry path, but you also need to evaluate permissions, sandboxing, network controls, secrets handling, plugin or skill installation risk, approval UX, and audit logging.

Should I evaluate local agents differently from cloud agents?

Yes. Local agents often inherit workstation context, file access, shell access, desktop browser state, and convenience permissions that never exist in a hosted API environment. A vendor that ignores that difference will miss real risk.

What does good evidence look like in an agent pentest report?

Good evidence includes the original malicious input or poisoned content, the trace of agent decisions or tool calls, the attempted side effect, the control that failed, and exact steps to reproduce and retest.

When should I prefer a service vendor over a product workflow?

Prefer a service when you need independent assessment, broader external credibility, or help modeling hard attack paths quickly. Prefer a product workflow when your team needs frequent replay, faster retests, and tighter integration with engineering changes. Many teams need both at different stages.

Bottom line

The best AI pentesting vendor evaluation guide for AI agents is brutally simple: make vendors prove they can find a real failure in an agent system, explain it precisely, and verify the fix afterward. If they can only talk in abstractions, keep them out of the final round.