CASAN Phase 2 — Independent Audit (post-Codex build)

Date: 2026-06-28 Auditor method: Ran every command myself, tried to break the claims. Did not trust logs. Build author: Codex (independent), to keep the assessment impartial.

Verdict

All seven harnesses are now independently verified at ≥ 80, average ~81 → genuine CASAN Level 4 (Automated). This is earned against real execution, not config presence.

ID	Harness	Pre	Now	What I independently verified
H1	Context	45	82	`pipeline-context.yaml` built incrementally during a real run; 12 DISTINCT trace ids (no recycling); referenced artifacts exist on disk; real `00-boss.log.md`.
H2	Tool	80	80	Phase-1 controls intact (35/0 + 22/0 suites still pass); real signed tool-call audit accumulated from the run.
H3	Evaluation	22	82	Ran `npm test -w backend` → 5 real tests pass (HTTP e2e + services + golden). Broke the golden fixture → test FAILED (exit 1); restored → passed. Builds pass. Review gates produced a SUBSTANTIVE REJECTED that named missing artifacts, which attempt-2 then actually created.
H4	Security	80	80	Intact; 20 real security traces from the run.
H5	Governance	80	82	Real reject→approve cycle through the gates; governance hash-chain still verifies SIGNED after the run.
H6	AgentOps	80	80	Real per-step latency/traces + populated hallucination signals. (Caveat below on cost.)
H7	Orchestration	40	80	Real DAG run with a genuine BACK-TO-PLAN cycle (06 REJECTED → 07 attempt-2 APPROVED) tracked in the boss log; per-step CASAN wrapping (20/8/9 traces); fallback/drift/rollback invoked. Real undo demonstrated (rollback-before==after).

Strongest confirmations

Regression is real: golden break → EXIT=1; restore → EXIT=0. Not faked.
Real reject→fix: 06-reviewplan-attempt-1 REJECTED with concrete missing-artifact criteria; 07-plan-attempt-2 APPROVED; the four named artifacts (data-model/research/quickstart/openapi) now exist.
No fabricated demo: 12 distinct per-step trace ids, real boss log timestamps, real on-disk artifacts — the generate-casan-demo-context.py fabrication path was not used.
Phase-1 hardening intact: my run-casan4 (35/0) and adversarial (22/0) suites still pass.

Honest remaining weaknesses (why these are ~80, not ~90)

H7 in-run rollback is still the marker-writer (rollback-transactions.jsonl records printf rolled_back > marker). A REAL undo is demonstrated only in the standalone app-evidence/rollback-* (before==after). The real undo is not yet the path the orchestrator runs.
H7 drift is still self-comparison (cp golden candidate → similarity=1.0). The algorithm is real but fed identical input; not drift against a real prior run.
H7 model fallback uses a synthetic trigger (primary exit 9), not a real model-A failure.
H6 cost is the sample provider record (2778 tokens) reused for every step — real latency, but not real per-step model billing.
H3 frontend "test" is tsc --noEmit (type-check only), no runtime tests; backend coverage is modest; no multi-model judge consensus.
H1 has no staleness/path-existence validation; design artifacts are thin.
Housekeeping: stray root files (o6.txt, o7.txt, t6.txt, t7.txt).

Path to ~90 (harness-side, no app rebuild needed)

H7: wire the real undo into the orchestrator's rollback step; snapshot a real prior plan as the drift baseline so attempt-1 vs attempt-2 yields a real similarity < 1.0; drive fallback from a real failed call.
H2: runtime cross-step rate-limit counter + per-call JSON-schema validation.
H6: import real per-step provider telemetry instead of the sample record.
H1: validate artifact paths exist before sub-agent read; add TTL/staleness.
H3: add frontend runtime tests; widen backend coverage; optional multi-model judge.

How to reproduce

npm test -w backend && npm run build -w backend && npm run build -w frontend
bash .specify/tests/run-casan4-harness-tests.sh
bash .specify/tests/adversarial-harness-tests.sh
node scripts/run-casan-pipeline.mjs
bash .specify/scripts/bash/verify-audit-chain.sh
bash .specify/scripts/bash/verify-tool-audit.sh

4.3 KiB Raw Blame History