4.7 KiB
CASAN Phase 3 — Wave 2 Results
Date: 2026-06-30
Executed by: Claude (Opus 4.8), in CASAN5, against the live local model (ollama:ornith:9b).
Integrity: real numbers only; blocked items recorded as BLOCKED, not pass. No score >90 claimed.
Evidence: docs/output/casan/phase3-evidence/wave2/.
Implemented (real, verified)
WP-A + WP-S2 — Quantitative H4 (semantic injection, measured)
Built a labeled red-team corpus (.specify/security/redteam-corpus.jsonl: 10 novel injection
paraphrases that are NOT in the regex blocklist + 6 benign) and a runner
(.specify/tests/phase3-redteam-metrics.sh) that scores both layers against the live model.
Real result (phase3-evidence/wave2/redteam-metrics.log):
| Layer | precision | recall |
|---|---|---|
| regex-only | 1.00 | 0.00 (caught 0/10 paraphrases) |
| model layer | 1.00 | 0.90 (caught 9/10, 0 benign false-positives) |
GATE model_recall>=0.8 AND model_recall>regex_recall → PASS. This quantifies the model
layer's value: the regex blocklist is blind to novel paraphrasing (recall 0), the model recovers
90% recall at 100% precision. One paraphrase still slipped the local 9B (recall 0.9) — recorded
honestly; cloud or a larger model would raise coverage.
Honest caveat: corpus is 16 samples; the security DoD targets ≥30 — this is a real but
not-yet-DoD-sized corpus.
WP-C — H6 real per-step billing + spike detection
.specify/scripts/bash/cost-spike-detect.sh(median-based, flags steps > 3× median, exits non-zero).provider-usage.jsonlnow has 18 real records with 9 distinct token values (e.g. 218, 219, 220, 216) from real router calls — not the recycled 2778 sample.- Fail-able proof: injecting a 4× step →
COST_SPIKE_DETECTED(exit 2). The detector also flags the leftover 2778 sample records as anomalies vs. the real ~220-token calls — working as intended. cost_usd=0.0,cost_source=ollama_local_real_tokens— local inference is genuinely free; not faked pricing.
WP-S8 — One-command security gate
.specify/scripts/bash/security-gate.sh aggregates the security-relevant checks into one verdict.
Real run (phase3-evidence/wave2/security-gate.log): PASS=6 FAIL=0 SKIP=0 —
run-casan4 suite, adversarial suite, audit hash-chain (signed), tool-call audit (signed),
model-router tests, red-team H4 metrics. Live-model gates SKIP (not fail) if the tunnel is down.
Status of the other work packages
| WP | Status |
|---|---|
| WP-D (H7 fallback/drift/undo) | Largely done across Wave 1 + earlier Phase 3: failure-driven fallback via real router (Wave 1 test 7), real drift between different artifacts, exception-style undo via rollback-manager checkpoint. Drift against the in-pipeline attempt1-vs-attempt2 artifacts is wired but the full pipeline run is blocked by baseline. |
| WP-E (H1 context) | Done — context-validate.sh red-able; on the real context it correctly flags 12 missing trace files (a real retention gap to fix, not a validator defect). |
| WP-F (H2 rate-limit + schema) | Done in Phase 3 push (rate_limit_per_run, validate-tool-input.sh), adversarially tested. |
| WP-G (H5 governance) | Offline done (signed re-forge-detection, key off-repo). KMS/WORM blocked (no AWS). |
| WP-B (H3 judge + frontend tests) | Partial/blocked. The router --role judge gives a real single-model judge usable now; wiring it into casan-step.mjs review gates and the frontend Vitest/RTL tests need the app baseline (npm install → network) which is blocked. True 2-of-3 consensus needs a 2nd independent provider (no cloud key). → Wave 3. |
Blockers (recorded, NOT passed)
- App baseline —
node_modulesabsent,npm installneeds network (sandbox-blocked). Blocks WP-B frontend tests,casan-step.mjsjudge wiring,npm audit(WP-S3), full pipeline run. - Cloud provider — keys unset → no multi-provider judge diversity (WP-B), no cloud-grade classify coverage.
- AWS/KMS/Object Lock — not configured → WP-G KMS/WORM not claimed.
- Corpus size — red-team corpus is 16, DoD wants ≥30 (expand in Wave 3).
Files added/changed (Wave 2)
- new:
.specify/scripts/bash/cost-spike-detect.sh,security-gate.sh;.specify/security/redteam-corpus.jsonl;.specify/tests/phase3-redteam-metrics.sh - (Wave 1 carried in:
model-router.sh,model-call.py,phase3-model-router-tests.sh, semantic escalation insecurity-check.sh)
Score statement
No average >90 claimed. Wave 2 added measured H4 (recall 0→0.90), real per-step H6 telemetry + spike detection, and a one-command security gate (6/6 green). App-dependent harnesses (WP-B frontend, judge wiring) and KMS/WORM remain blocked. Scores update only after a full independent audit on a green baseline.