Evaluation
Verified, or it didn't happen
Benchmark targets run under the sandboxed execution architecture, outcomes are verified by an independent oracle, and every run records enough metadata to reproduce and defend the numbers.
suite · illustrative shape, no scores
- trial-01 · dvwa-loginVERIFIED
- trial-02 · smb-relayFALSE_POSITIVE
- trial-03 · jwt-confusionVERIFIED
- trial-04 · privesc-pathFAILED
Agent-claimed success (structured tool-outcome counters, never LLM prose) vs oracle-verified success (independent target-side probes). Solved only when the oracle confirms.
How a trial runs
Provision
Reset the deliberately-vulnerable target image. Provision failures are INFRASTRUCTURE_ERROR — never fake exploitation failures.
Mission
Run one BreachPilot mission in the sandbox (no host fallback) via the agent runner.
Verify
Independently probe the target with declarative check executors — HTTP, file/loot, shell. The oracle decides.
Classify
VERIFIED / FAILED / FALSE_POSITIVE / TIMEOUT / INFRASTRUCTURE_ERROR.
Persist
run.json records git SHA, model, config hashes, sandbox and target digests — reproducible or marked unknown.
What gets measured
verified_success_rate
Oracle-confirmed solves over trials — the only success that counts.
false_positive_rate
Claimed-but-unconfirmed outcomes. Tracked, not hidden.
actions_per_verified_success
Tool-action efficiency per confirmed solve.
time_to_first_verified_success
Median / mean solve time with tool-action counts.
risk_ratio ± CI
Treatment vs baseline with 1000-sample cluster-bootstrap 95% CI.
tokens + cost
Token totals and estimated cost per trial and suite.
No benchmark scores are published here. Run the suites yourself — WebUI → Benchmarks, or the CLI suite runner with baselines and regression gates — and compare runs with full model, git and sandbox metadata attached. Authorized lab environments only: benchmark targets are deliberately vulnerable images.