52 lines
4.3 KiB
Markdown
52 lines
4.3 KiB
Markdown
# CASAN Phase 2 — Independent Audit (post-Codex build)
|
|
|
|
**Date:** 2026-06-28
|
|
**Auditor method:** Ran every command myself, tried to break the claims. Did not trust logs.
|
|
**Build author:** Codex (independent), to keep the assessment impartial.
|
|
|
|
## Verdict
|
|
|
|
All seven harnesses are now independently verified at **≥ 80**, average **~81** → genuine **CASAN Level 4 (Automated)**. This is earned against real execution, not config presence.
|
|
|
|
| ID | Harness | Pre | Now | What I independently verified |
|
|
|----|---------|:--:|:--:|---|
|
|
| H1 | Context | 45 | **82** | `pipeline-context.yaml` built incrementally during a real run; 12 DISTINCT trace ids (no recycling); referenced artifacts exist on disk; real `00-boss.log.md`. |
|
|
| H2 | Tool | 80 | **80** | Phase-1 controls intact (35/0 + 22/0 suites still pass); real signed tool-call audit accumulated from the run. |
|
|
| H3 | Evaluation | 22 | **82** | Ran `npm test -w backend` → 5 real tests pass (HTTP e2e + services + golden). Broke the golden fixture → test FAILED (exit 1); restored → passed. Builds pass. Review gates produced a SUBSTANTIVE REJECTED that named missing artifacts, which attempt-2 then actually created. |
|
|
| H4 | Security | 80 | **80** | Intact; 20 real security traces from the run. |
|
|
| H5 | Governance | 80 | **82** | Real reject→approve cycle through the gates; governance hash-chain still verifies SIGNED after the run. |
|
|
| H6 | AgentOps | 80 | **80** | Real per-step latency/traces + populated hallucination signals. (Caveat below on cost.) |
|
|
| H7 | Orchestration | 40 | **80** | Real DAG run with a genuine BACK-TO-PLAN cycle (06 REJECTED → 07 attempt-2 APPROVED) tracked in the boss log; per-step CASAN wrapping (20/8/9 traces); fallback/drift/rollback invoked. Real undo demonstrated (rollback-before==after). |
|
|
|
|
## Strongest confirmations
|
|
- **Regression is real:** golden break → `EXIT=1`; restore → `EXIT=0`. Not faked.
|
|
- **Real reject→fix:** `06-reviewplan-attempt-1` REJECTED with concrete missing-artifact criteria; `07-plan-attempt-2` APPROVED; the four named artifacts (data-model/research/quickstart/openapi) now exist.
|
|
- **No fabricated demo:** 12 distinct per-step trace ids, real boss log timestamps, real on-disk artifacts — the `generate-casan-demo-context.py` fabrication path was not used.
|
|
- **Phase-1 hardening intact:** my `run-casan4` (35/0) and `adversarial` (22/0) suites still pass.
|
|
|
|
## Honest remaining weaknesses (why these are ~80, not ~90)
|
|
1. **H7 in-run rollback is still the marker-writer** (`rollback-transactions.jsonl` records `printf rolled_back > marker`). A REAL undo is demonstrated only in the standalone `app-evidence/rollback-*` (before==after). The real undo is not yet the path the orchestrator runs.
|
|
2. **H7 drift is still self-comparison** (`cp golden candidate` → `similarity=1.0`). The algorithm is real but fed identical input; not drift against a real prior run.
|
|
3. **H7 model fallback uses a synthetic trigger** (`primary exit 9`), not a real model-A failure.
|
|
4. **H6 cost is the sample provider record** (2778 tokens) reused for every step — real latency, but not real per-step model billing.
|
|
5. **H3 frontend "test" is `tsc --noEmit`** (type-check only), no runtime tests; backend coverage is modest; no multi-model judge consensus.
|
|
6. **H1** has no staleness/path-existence validation; design artifacts are thin.
|
|
7. Housekeeping: stray root files (`o6.txt`, `o7.txt`, `t6.txt`, `t7.txt`).
|
|
|
|
## Path to ~90 (harness-side, no app rebuild needed)
|
|
- H7: wire the real undo into the orchestrator's rollback step; snapshot a real prior plan as the drift baseline so attempt-1 vs attempt-2 yields a real similarity < 1.0; drive fallback from a real failed call.
|
|
- H2: runtime cross-step rate-limit counter + per-call JSON-schema validation.
|
|
- H6: import real per-step provider telemetry instead of the sample record.
|
|
- H1: validate artifact paths exist before sub-agent read; add TTL/staleness.
|
|
- H3: add frontend runtime tests; widen backend coverage; optional multi-model judge.
|
|
|
|
## How to reproduce
|
|
```bash
|
|
npm test -w backend && npm run build -w backend && npm run build -w frontend
|
|
bash .specify/tests/run-casan4-harness-tests.sh
|
|
bash .specify/tests/adversarial-harness-tests.sh
|
|
node scripts/run-casan-pipeline.mjs
|
|
bash .specify/scripts/bash/verify-audit-chain.sh
|
|
bash .specify/scripts/bash/verify-tool-audit.sh
|
|
```
|