Evaluation
Verified, or it didn't happen
BreachPilot benchmarks run against deliberately-vulnerable lab images, and an independent oracle confirms every claimed win. We publish no leaderboards and no invented scores, only a method you can re-run and argue with.
Authorized lab targets only. Benchmark images are deliberately vulnerable. Never point them, or BreachPilot, at systems you do not own or have explicit written permission to test.
Ground rules
What stays unpublished
A single success percentage with no method behind it tells you nothing. These three rules make a BreachPilot number worth reading.
Benchmark principles
Oracle decides, not the agent
The agent's claim never counts as a solve. An independent probe checks the target side (HTTP, file, or shell), and only a pass marks VERIFIED.
Misses are published, not buried
False positives, timeouts, and infra errors are first-class verdicts. A benchmark that hides its misses does not measure anything.
Reproducible or marked unknown
Every run records git SHA, model, config hash, and sandbox plus target digests. Without that metadata, a number is a story, not a result.
Verdict taxonomy
Five outcomes, no partial credit
Every trial ends in exactly one verdict. The gap between what the agent claimed and what the oracle confirmed is the most honest number in the suite.
Verdict taxonomy
| Verdict | Meaning |
|---|---|
| VERIFIED | Oracle probe passed. The claimed outcome actually holds on the target. |
| FAILED | Agent finished without a claim, or the oracle probe did not pass. |
| FALSE_POSITIVE | Agent claimed success but the oracle could not confirm it. Tracked separately, never merged into wins. |
| TIMEOUT | Mission exceeded its time or action budget before the oracle could pass. |
| INFRASTRUCTURE_ERROR | Target, sandbox, or harness failed. Excluded from success math, counted in reliability. |
Claimed vs verified · chart pattern, not results
Suite A · login flow
Suite B · relay path
Suite C · privesc path
Bar widths are illustrative placeholders showing how to read the chart: muted is what the agent claimed, solid is what the oracle confirmed. No BreachPilot scores are published on this page. Run the suites yourself and compare runs with full metadata attached.
Method
How a trial runs
Same five steps every time, inside the sandbox. Skip a step and the run is marked unknown, not passed.
How a trial runs
Provision
Reset the deliberately-vulnerable target image to a known digest. A provision failure is recorded as INFRASTRUCTURE_ERROR, never as an exploitation failure.
Mission
Run one BreachPilot mission inside the sandbox via the agent runner, with no host fallback or hand assistance and scope locked to the lab target.
Verify
Probe the target independently with declarative check executors: HTTP response, loot file, or shell state. The oracle runs outside the agent's context.
Classify
Map the oracle result to exactly one verdict: VERIFIED, FAILED, FALSE_POSITIVE, TIMEOUT, or INFRASTRUCTURE_ERROR.
Persist
Write run.json with git SHA, model id, config hashes, and sandbox plus target digests, so any run can be re-executed and re-checked.
Metrics
What gets measured, and how to read it
Six metrics, each with a stated interpretation. If a change improves one while quietly worsening another, the suite should catch it.
Benchmark metrics
| Metric | What it is | How to read it |
|---|---|---|
| verified_success_rate | Oracle-confirmed solves over trials. The only success that counts. | Compare across runs with identical target digests. A higher rate with a higher false-positive rate is not an improvement. |
| false_positive_rate | Claimed-but-unconfirmed outcomes over trials. | A honesty metric for the agent's self-reporting. Falling claims with steady verified solves means better calibration. |
| actions_per_verified_success | Tool-action efficiency per confirmed solve. | Cost of a win. Watch alongside success rate. Fewer actions at the same verified rate is a real efficiency gain. |
| time_to_first_verified_success | Median and mean solve time with tool-action counts. | Report both median and mean; one long-tail run should not hide behind an average. |
| risk_ratio ± CI | Treatment vs baseline with 1000-sample cluster-bootstrap 95% CI. | The regression gate. If the CI crosses 1.0, the change did not move the needle, no matter what the point estimate says. |
| tokens + cost | Token totals and estimated cost per trial and suite. | Pairs with verified rate to answer the only budget question: what does a confirmed solve cost? |
Reproducibility checklist
A run without this metadata cannot be defended. The harness records all five; if any are missing, the run is labeled unknown.
- Git SHA of the BreachPilot build under test
- Model id and provider path for the run
- Config hash covering suite and gate settings
- Sandbox image digest the mission ran in
- Target image digest that was reset per trial
Statistics without hype
Run comparison uses risk ratio of verified success against a pinned baseline, with a 1000-sample cluster-bootstrap 95% confidence interval. The regression gate fails a change whose interval crosses 1.0. The point estimate alone never passes.
- Pin the baseline: same target digests, same budgets, same model class.
- Report median and mean solve times. Long tails matter.
- Count infra errors separately from exploitation failures.
- Full method lives in the engineering docs, not in a footnote.
Run the suites yourself
No scores are published here because your hardware, model, and target digests will differ from ours, and those differences are the whole story. Open the WebUI Benchmarks view or follow the suite-runner guide in the docs, then compare runs with baselines and regression gates attached. Authorized lab environments only.
Built for authorized security testing
Only test systems you own or have explicit written permission to assess.
Benchmark targets are deliberately vulnerable images. Run them isolated in your lab, keep the scope gate locked, and treat every finding as lab-only evidence.