Database & Mission Persistence

Overview

Persistence is SQLite, split across two independent databases plus the filesystem:

Per AGENTS.md rule 2, the two flows (Flow A exploit engine, Flow B legacy research loop) share only the db.py and mission.py schemas. Flow A's own state lives in exploit_workspace/<ip>/exploit_audit.jsonl (tamper-evident JSONL, not SQLite) and is bridged into the shared evidence table read-only (see Evidence bridge).

db.py owns the schema, migrations, ID generation, and the thread-safe DatabaseManager wrapper. All IDs are {prefix}-{seq:05d}-{8hex} (db.py:31-34); timestamps are ISO8601 UTC (db.py:27-28); JSON fields are stored as TEXT and deserialized on read.

Where the DB files live

• research_workspace/research.db — one DB per workspace root, not per mission. Missions are rows inside it. agent_loop.py:96-97 opens workspace_root / "research.db"; cli.py:48 does the same. The default singleton get_default_db() uses RESEARCH_WORKSPACE env or research_workspace/ (db.py:803-817).
• research_workspace/<mission_id>/ — per-mission filesystem workspace (evidence/, reports/, logs/, tasks/) created by MissionController._init_workspace (mission.py:386-402). Evidence files live here; their metadata rows live in research.db.
• reports/api_runtime.db — API daemon state, ApiPersistence(reports_dir) (tools/api/persistence.py:90-95). Deliberately separate so the daemon never touches Flow B's schema (tools/api/persistence.py:3).
• exploit_workspace/<ip>/exploit_audit.jsonl — Flow A's append-only audit log (not SQLite); promoted into evidence via promote_exploit_audit (evidence.py:288-340).
• No top-level sandbox/ directory exists — sandbox code lives at tools/sandbox/ (manager, docker backend/lifecycle, network) with the worker image at docker/sandbox/ (tools/sandbox/manager.py:1-14; docker/sandbox/Dockerfile:1-3).
• Canonical audit helpers live at tools/kernel/audit.py (credential redaction + audit decorators), re-exported by tools/mcp_shared.py for backwards compat (tools/kernel/audit.py:1-11; tools/mcp_shared.py:33-46).

Schema (Flow B research DB)

Schema version: 10 (db.py:23). DDL in db.py:39-312; migrations in db.py:390-863.

• v6 — graph_nodes_v2 / graph_edges_v2 typed intelligence-graph pair (db.py:697-746); legacy graph_nodes / graph_edges stay untouched.
• v7 — belief-state columns on hypotheses + belief_transitions log (db.py:749-778).
• v8 — evidence.provenance_json + evidence_references links (db.py:781-803).
• v9 — decision_telemetry log (db.py:806-838).
• v10 — attempt_fingerprints dedup table (db.py:841-863).

ER-style diagram

missions 1 ──< scope_rules
   │
   ├──< tasks 1 ──< observations
   │     │
   │     └──< outcome_assessments >── hypotheses (1 per mission+key)
   │                                     └──< belief_transitions (v7)
   │
   ├──< graph_nodes 1 ──< graph_edges (from/to node FKs)
   ├──< graph_nodes_v2 1 ──< graph_edges_v2 (v6, scope-based)
   ├──< evidence (task_id FK nullable, finding_id loose ref)
   │     └──< evidence_references (v8, source_table/source_id loose ref)
   ├──< findings
   ├──< audit_logs
   ├──< memories
   ├──< decision_telemetry (v9)
   ├──< attempt_fingerprints (v10)
   └──< embeddings (source_table/source_id loose ref)

lessons (global, no mission FK — cross-mission learning)
_migrations (version ledger)

Table reference

Status enums

missions.status — active, paused, completed (mission.py:416).

tasks.status — pending, running, blocked, complete, failed, needs_approval (db.py:89-91). tasks.phase — recon, analysis, test, validate, exploit, post_exploit, report (db.py:77). Legacy aliases (validation, exploitation, post-exploit, service_enumeration, …) are normalized on insert (task_queue.py:31-42) and remapped by the v2 migration (db.py:406-488).

hypotheses.status — open, confirmed, refuted, inconclusive, exhausted (db.py:108-110). Terminal: confirmed/refuted/exhausted (outcome_judge.py:39-45).

outcome_assessments.execution_outcome — succeeded, failed, blocked (db.py:130-132). outcome_assessments.hypothesis_status — same enum as hypotheses.

findings.status — candidate, needs_validation, rejected, duplicate_suspected, validated, report_ready (db.py:233-235). Transition map in finding_verifier.py:33-40:

candidate ──> needs_validation / rejected / duplicate_suspected
needs_validation ──> validated / rejected / duplicate_suspected
duplicate_suspected ──> rejected / validated / needs_validation
validated ──> report_ready / rejected
report_ready ──> rejected
rejected ──> (terminal)

evidence.type — raw_output, http_response, screenshot, note, diff, file, http_request, structured_json (db.py:210-212).

memory_type — working, episodic, semantic, target, hypothesis, dead_end, finding_note (db.py:259); legacy aliases normalized in memory.py:36-53.

lessons.outcome — success, failure, partial, unknown (db.py:283).

Mission lifecycle in the DB

create_from_config (mission.py:312-383)
  → validate (mission.py:197-234)
  → INSERT missions (status='active') + scope_rules (allow/deny/action)
  → audit_logs 'mission_created'
  → workspace dirs (research_workspace/<id>/…)

run loop (legacy/agent_loop.py)
  → tasks created (pending) → picked by priority → running
  → observation + evidence + outcome assessment per task
  → hypothesis status transitions (open → confirmed/refuted/inconclusive/exhausted)
  → terminal hypothesis blocks sibling pending tasks (outcome_judge.py:540-552)

update_status (mission.py:415-424)  → active / paused / completed
resume (agent_loop.py:100-128)      → load_mission by id; DB row is source of truth

Resume is "load the mission row and re-point every manager at it" — the DB holds all resumable state (agent_loop.py:100-109). reset_stale_running re-queues tasks left running by a crash back to pending (task_queue.py:181-201).

Evidence → finding → outcome pipeline

1. Tool runs → ToolRouter saves raw output via EvidenceStore.save (tool_router.py:200): file written under research_workspace/<mission_id>/evidence/<subdir>/, sha256 hash + summary + metadata row in evidence (evidence.py:55-125).
2. Observer parses output into a structured Observation (facts, new assets/endpoints/parameters/technologies, interesting signals, possible findings, dead ends, hypothesis evidence) — persisted by agent_loop._save_observation (agent_loop.py:1287-1327).
3. OutcomeJudge (outcome_judge.py:190-307) — deterministic, evidence-grounded. Separates whether a tool ran from whether evidence resolved the hypothesis:
   • execution_outcome from the execution result (_execution_outcome, outcome_judge.py:622-632): succeeded/failed/blocked (blocked = scope/approval gate, consumes no attempt).
   • hypothesis_status from structured observation fields only — raw-output words like "success" are not confirmation (_structured_criterion_met, outcome_judge.py:690-724; stopword list outcome_judge.py:925-959).
   • Terminal judgment requires >= min_evidence_references (default 1) persisted evidence refs; support/refutation scores from hypothesis_evidence polarity entries and facts (_explicit_evidence_scores, outcome_judge.py:789-815); contradiction detection (_texts_contradict, outcome_judge.py:832-862).
   • confirmed at support ≥ 0.75, refuted at refutation ≥ 0.75, exhausted after max_inconclusive_attempts (default 3) materially different checks (outcome_judge.py:248-262).
   • information_value (0-1) weights usefulness/structural/evidence/resolution (_information_value, outcome_judge.py:865-895).
4. HypothesisRepository.persist_assessment (outcome_judge.py:428-568) atomically inserts the outcome_assessments row, appends to check_history_json, updates hypothesis status/confidence/counters, blocks sibling pending tasks when terminal, and writes an outcome_judgment audit row.
5. Dedup guards: build_hypothesis_key (sha256 of normalized target+statement, outcome_judge.py:579-581) and build_check_fingerprint (sha256 of tool+args/method, outcome_judge.py:584-602). prepare_task raises ClosedHypothesisError / DuplicateInvestigationError (outcome_judge.py:374-399).
6. FindingVerifier (finding_verifier.py): create_candidate inserts a candidate finding with heuristic impact_score (_score_impact, finding_verifier.py:306-329). validate_finding runs the 10-point checklist (scope, evidence on disk, summary, impact, vuln class, reproduction steps) and auto-transitions to validated when complete (finding_verifier.py:146-226). mark_report_ready gates on validated (finding_verifier.py:238-246). Missing items can be turned back into validate-phase tasks (generate_validation_tasks, finding_verifier.py:340-377).

Evidence bridge (Flow A → Flow B)

promote_exploit_audit (evidence.py:288-340) reads exploit_workspace/<ip>/exploit_audit.jsonl and writes one structured_json evidence row per MCP/Flow-A record, stamped with audit_hash + row_attempt_id in metadata so consumers can join back. Purely additive — the audit JSONL is never mutated (evidence.py:258-262). task_id is intentionally left empty because audit attempt ids are not Flow B task ids (evidence.py:368-371).

Memory & embeddings

• memories table: MemoryManager.remember (memory.py:71-113) normalizes legacy type aliases, then optionally stores an embedding via SemanticMemoryManager.store_embedding (memory.py:106-112).
• embeddings table: vector JSON per (source_table, source_id) — memories, observations, etc. Generated by Ollama /api/embeddings (tools/semantic_memory.py:48-106), default model nomic-embed-text, host from ollama.embed_host (falls back to ollama.host; cloud path sends OLLAMA_API_KEY bearer, tools/semantic_memory.py:73-77). Similarity is numpy cosine in Python (find_similar, tools/semantic_memory.py:175-241); zero-length embeddings are skipped.
• lessons table: cross-mission learning. store_lesson (tools/semantic_memory.py:131-173) persists text + real embedding; ExperienceStore.record_outcome (tools/experience_store.py:96) writes Bayesian outcome rows with embedding_json='[]' (tracked, not recallable). find_similar_lessons (tools/semantic_memory.py:243-333) feeds payload_crafter and exploit reflection. The text column was added in migration v5 (db.py:695-711) — previously the lesson text was embedded but never persisted.
• retrieve_relevant (memory.py:146-192): exact target match first, semantic fallback across missions embedding the context (not the bare IP, which embeds to a meaningless vector).

Task queue mechanics

TaskQueue (task_queue.py) is a persistent priority queue over the tasks table:

• create_task (task_queue.py:55-117) resolves hypothesis identity + check fingerprint first (HypothesisRepository.prepare_task), normalizes phase/risk/status, and scores priority if not supplied.
• get_next_task (task_queue.py:119-139) — the core query:

SELECT t.* FROM tasks t
LEFT JOIN hypotheses h ON h.id=t.hypothesis_id
WHERE t.mission_id=? AND t.status='pending'
  AND (h.id IS NULL OR h.status IN ('open','inconclusive'))
ORDER BY t.priority DESC, t.created_at ASC LIMIT 1

• Priority scoring (_score_priority, task_queue.py:304-380): phase bonus (exploit 30 > validate 25 > test 20), auth-boundary/object-reference/sensitive-data keyword bonuses, hypothesis novelty, information value, minus attempt/duplicate/scope-risk/noise penalties, clamped 0-100.
• deduplicate removes same (target, objective, phase) pending tasks (task_queue.py:269-283); reprioritize re-scores all pending (task_queue.py:285-300); reset_stale_running is the crash-resume primitive (task_queue.py:181-201).

Target graph

TargetGraph (target_graph.py) models the attack surface in graph_nodes/graph_edges. Node types: program, asset, host, domain, ip, service, web_app, api, endpoint, parameter, identity, role, session, object, permission_boundary, technology, evidence, finding (target_graph.py:25-29). Edge relations: owns, exposes, resolves_to, serves, requires_auth, accepts_parameter, returns_object, belongs_to_user, accessible_by, tested_by, produced_evidence, indicates, blocked_by_scope, related_to (target_graph.py:31-36). Notable queries: find_untested_assets (no tested_by edge, target_graph.py:129-141), find_permission_boundaries, find_object_id_candidates (endpoint/parameter values matching id/uuid/object/user/order/account, target_graph.py:152-162), summarize_graph for LLM context.

Report generation

ReportGenerator (report_generator.py) requires report_ready status (report_generator.py:165-170), renders the strict template (REPORT_TEMPLATE, report_generator.py:53-135) with severity from impact_score (_severity_label, report_generator.py:39-48), and writes research_workspace/<mission_id>/reports/<finding_id>.md (report_generator.py:254-255). generate_summary_report aggregates all report_ready/validated/candidate/rejected findings into summary_report.md (report_generator.py:282-335). export_report returns structured JSON (report_generator.py:259-280). Evidence refs are listed by id; content is fetched via EvidenceStore.get.

API runtime DB (reports/api_runtime.db)

Separate schema, version 5 (tools/api/persistence.py:18), thread-safe via a single threading.Lock (tools/api/persistence.py:164):

• runs — id, created_at, updated_at, state, request_json, preview_json, result_json, resumed_from, error, cancelled_at, title, is_demo (tools/api/persistence.py:27-40). Live states (draft, preparing, awaiting_confirmation, running, awaiting_input, queued, cancelling) are marked interrupted with pending decisions expired on daemon startup via recover_interrupted (tools/api/persistence.py:492-510).
• decisions — id, run_id (FK cascade), kind, prompt_text, required_text, options_json, status (pending/answered/expired), answer, created_at, answered_at (tools/api/persistence.py:42-54). Written by ApiDecisionProvider.request (tools/run_service/providers.py:210), answered via answer_decision (tools/api/persistence.py:537-558).
• users — id, username (UNIQUE), password_hash, password_salt, created_at, last_login (v3, tools/api/persistence.py:65-72).
• annotations — id, run_id (FK cascade), user_id, username, body, finding_ref, created_at (v3, tools/api/persistence.py:74-83).
• custom_goals — id, name (UNIQUE, NOCASE), objective, created_at, updated_at (v5, tools/api/persistence.py:87-95).
• app_state — key, value tombstone/flag store (v4, tools/api/persistence.py:121-125); runs.is_demo flags demo sessions (tools/api/persistence.py:39).
• Migrations: v2 runs.title (tools/api/persistence.py:101-103); v3 users + annotations (tools/api/persistence.py:108-119); v4 is_demo + app_state (tools/api/persistence.py:122-125); v5 custom_goals (tools/api/persistence.py:128-134). title holds AI-generated session titles from tools/api/session_titler.py (default TITLE_MODEL = "gemma4:31b-cloud", persisted via update_run_title, tools/api/persistence.py:355-370).

Migration & back-compat

• ensure_schema (db.py:377-387) runs DDL (idempotent CREATE TABLE IF NOT EXISTS) then applies pending migrations 1.._SCHEMA_VERSION, recording each in _migrations.
• v2 — tasks phase enum widened to include exploit/post_exploit; table rebuilt with FK off, legacy phase values remapped (db.py:415-496).
• v3 — created_at indexes for high-volume tables (db.py:498-517).
• v4 — adds hypotheses + outcome_assessments tables, tasks.hypothesis_id/check_fingerprint, observations.hypothesis_evidence_json; backfills hypothesis identity for historical tasks without inferring success from execution status (db.py:520-679).
• v5 — lessons.text column (db.py:682-694).
• v6 — graph_nodes_v2 / graph_edges_v2 typed intelligence-graph tables; legacy graph tables untouched (db.py:697-746).
• v7 — belief-state columns on hypotheses + belief_transitions log (db.py:749-778).
• v8 — evidence.provenance_json + evidence_references links (db.py:781-803).
• v9 — decision_telemetry log (db.py:806-838).
• v10 — attempt_fingerprints dedup table with unique (mission_id, fingerprint) index (db.py:841-863).
• API DB _init_db (tools/api/persistence.py:191-267) runs _DDL then applies _MIGRATION_V2.._MIGRATION_V5 idempotently; v2 adds runs.title (tools/api/persistence.py:101-103), v3 adds users + annotations (tools/api/persistence.py:108-119), v4 adds runs.is_demo + app_state (tools/api/persistence.py:122-125), v5 adds custom_goals (tools/api/persistence.py:128-134).
• Extension rule: schema changes go in DDL + _SCHEMA_VERSION + a _migrate_vN_* helper (docs/extension-guide.md:224-225).
• Back-compat notes: get_default_db ensures the schema (incl. lessons) exists on fresh Flow A installs (db.py:955-969); the legacy resume path calls ensure_schema before reading (legacy/agent_loop.py:118); _row_to_* helpers tolerate missing JSON fields; _coerce_state accepts both HypothesisState and mappings (outcome_judge.py:1009-1036).
• Root Flow B entry points (cli.py, agent_loop.py, mission.py, evidence.py, task_queue.py, memory.py, tool_router.py, finding_verifier.py, report_generator.py, observer.py) are DeprecationWarning shims re-exporting legacy.* (cli.py:1-9; legacy/README.md:13-17).

Related documentation

• Runtime flows — Database-Backed Research Loop vs Exploit Session Flow.
• Architecture — Flow A vs Flow B split and shared schemas.
• Outcome Judgment and Evidence Handling — evidential versus execution status and the Flow A audit trail.
• Disposable Execution Sandbox — sandbox execution funnel and worker image.
• Extension Guide — adding persistent data (DDL + _SCHEMA_VERSION + _run_migration).
• API Persistence — the api_runtime.db schema, migrations, runs, and decisions.

Source map

• db.py
• tools/api/persistence.py
• tools/api/session_titler.py
• tools/run_service/providers.py
• tools/kernel/audit.py
• tools/mcp_shared.py
• tools/sandbox/manager.py
• docker/sandbox/Dockerfile
• cli.py
• agent_loop.py
• mission.py
• legacy/README.md
• legacy/cli.py
• legacy/agent_loop.py
• docs/extension-guide.md