9.2 KiB
9.2 KiB
CASAN — Team Handoff & Push-to-90 Plan
Repo: Output_CASAN5_REFINED/AINative_OKR_CASAN5 (the active package — GHCP is deprecated)
Status date: 2026-06-28
Goal: every harness H1–H7 above 80, ideally ~90, earned against real execution (no faked evidence).
PART 1 — Where we are now (status report)
Current independent scores (verified, not self-claimed)
| ID | Harness | Score | State |
|---|---|---|---|
| H1 | Context | 82 | Real incremental pipeline-context.yaml from a real run; 12 distinct traces; artifacts on disk |
| H2 | Tool | 80 | Per-agent permission enforced + in execution path; signed tamper-evident tool audit; rollback required |
| H3 | Evaluation | 82 | Real app + real unit/e2e tests that run and can fail; real golden regression; real LLM-judge REJECTED→fix cycle |
| H4 | Security | 80 | Input normalization defeats bypasses; pii-rules wired; output fail-closed on secrets |
| H5 | Governance | 82 | RSA-anchored audit chain (re-forge detected); approver+output_hash hashed; separation of duties |
| H6 | AgentOps | 80 | Real latency/alerts; hallucination detector populated; provider-telemetry cost source |
| H7 | Orchestration | 80 | Real DAG run with real BACK-TO-PLAN; per-step CASAN wrapping; fallback/drift/rollback invoked |
| Average | ~81 | CASAN Level 4 (Automated), genuine |
How this was reached
- Phase 1 (harness hardening, by Claude): lifted H2/H4/H5/H6 from ~50s to ~80 with adversarial-verified controls. See phase1-hardening-reassessment.md.
- Phase 2 (real app + pipeline run, by Codex — independent builder for impartiality): lifted H1/H3/H7 by building a real NestJS+Prisma+React app and running the pipeline for real. Audited independently. See phase2-independent-audit.md.
Verify the current state (run these — all must pass)
cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
npm test -w backend && npm run build -w backend && npm run build -w frontend
bash .specify/tests/run-casan4-harness-tests.sh # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh # 22 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh # TOOL_AUDIT_VALID anchor=signed
Honest residual gaps keeping us at ~81 not ~90
- H7 — in-run rollback still writes a marker (real undo exists only as a standalone demo); drift still compares a file to a copy of itself; model-fallback uses a synthetic trigger.
- H6 — cost uses a sample provider record reused for every step (real latency, not real per-step billing).
- H2 — no runtime rate-limit counter; no per-call JSON-schema validation of tool I/O.
- H4 — rule/normalization-based only (no semantic/embedding detection); no tool sandbox/timeout.
- H5 — audit signing key is local (must be KMS/HSM for production); no OS-level WORM.
- H1 — no staleness / path-existence validation; design artifacts are thin.
- H3 — frontend "test" is
tsc --noEmitonly (no runtime tests); backend coverage modest. - Housekeeping — stray root files
o6.txt,o7.txt,t6.txt,t7.txt.
PART 2 — Model assignment guide (which model for which task)
Use the exact model ID strings. Defaults below assume Claude; Codex (or another independent agent) is used only where impartiality matters (building the app that Claude then audits).
| Model | ID | Cost (in/out per 1M) | Use it for |
|---|---|---|---|
| Claude Opus 4.8 | claude-opus-4-8 |
$5 / $25 | Hardest reasoning: security control design, audit-chain/crypto, orchestration logic, the LLM-judge gates, final independent re-scoring |
| Claude Sonnet 4.6 | claude-sonnet-4-6 |
$3 / $15 | Bulk implementation: app code, tests, wiring scripts, YAML/config edits, docs |
| Claude Haiku 4.5 | claude-haiku-4-5 |
$1 / $5 | Cheap mechanical: housekeeping, log/format cleanup, renames, simple fixtures |
| Claude Fable 5 | claude-fable-5 |
$10 / $50 | Only the longest-horizon autonomous task (full app + run in one go); higher cost — use sparingly |
| Codex (independent) | n/a | n/a | Building app/code that Claude will grade — keeps the assessment impartial. NOT for grading its own output |
Rule of impartiality: whoever builds a thing must not be the one who scores it. App build → Codex; harness/security + scoring → Claude (Opus 4.8). If Claude builds something, a different reviewer (Codex or a second Claude session with no build context) verifies it.
Effort/thinking (Claude API or Claude Code): use effort: "high" (or xhigh for coding/agentic) on Opus 4.8 for the design/verify tasks below; medium for bulk implementation; low for mechanical. Adaptive thinking on for anything non-trivial.
PART 3 — Push-to-90 tasks (assign these)
Each task: owner model, files, acceptance test, and the integrity rule (must be verifiable, no faking).
| # | Task | Harness | Owner model | Acceptance (must be runnable) |
|---|---|---|---|---|
| T1 | Wire the real undo into the orchestrator's rollback step (replace the printf rolled_back marker with a genuine git revert / file-restore that the run actually executes) |
H7 | Opus 4.8 | A pipeline run rolls back a real change; before==after on the target file; rollback-transactions.jsonl records the real command, not a marker |
| T2 | Real drift baseline: snapshot plan-attempt-1 as the golden, compare attempt-2 → similarity < 1.0 and a meaningful threshold decision | H7 | Sonnet 4.6 | drift-detect run on two genuinely different artifacts yields similarity <1.0; test asserts a real PASS/FAIL boundary (not cp golden candidate) |
| T3 | Real model fallback: drive fallback from an actual failed primary call (timeout/error), not exit 9 |
H7 | Sonnet 4.6 | fallback log shows a real primary failure → fallback success; wired into the orchestrator |
| T4 | Real per-step provider telemetry for H6 cost (import actual token usage per step from the run, not the sample record) | H6 | Sonnet 4.6 | metrics.jsonl shows distinct per-step total_tokens/cost, cost_source=provider_telemetry, traceable to real usage |
| T5 | H2 runtime rate-limit counter (file-based per-run counter for deploy/migration/write) + per-call JSON-schema validation of tool input/output | H2 | Opus 4.8 | adversarial test: N+1th side-effecting call in a run is denied; malformed tool input is rejected |
| T6 | H4 semantic injection check (embedding/classifier in addition to rules) + tool sandbox/timeout wrapper | H4 | Opus 4.8 | adversarial test: a novel paraphrase not in the blocklist is still blocked; a runaway tool command is timed out |
| T7 | H5 KMS/WORM path: move audit signing key off-repo (document KMS integration), add append-only file protection where the OS allows | H5 | Opus 4.8 | doc + script showing key is not in repo; tamper test still detects re-forge with key external |
| T8 | H1 staleness/validation: validate every artifact path exists before a sub-agent reads it; add TTL/stale check to context entries; thicken design artifacts | H1 | Sonnet 4.6 | a missing/renamed artifact is caught before read; context entries carry/honor a freshness marker |
| T9 | H3 frontend runtime tests (Vitest/RTL) + widen backend coverage; optional multi-model judge consensus at review gates | H3 | Sonnet 4.6 (tests), Opus 4.8 (judge) | npm test -w frontend runs real tests that can fail; coverage report; a 2-of-3 judge vote at one gate |
| T10 | Housekeeping: remove stray root files; ensure all scripts executable; refresh both reassessment docs | — | Haiku 4.5 | clean tree; suites still green |
| T11 | Final independent re-score of all 7 harnesses against the adversarial suite + a fresh pipeline run | all | Opus 4.8 (must NOT be a task owner above) | a re-assessment doc with per-harness evidence; every harness >80 |
Sequencing: T1–T9 can run in parallel (different files); T10 after; T11 last, by a reviewer who did not own T1–T9.
PART 4 — Non-negotiable integrity rules (put in every task brief)
- Tests must actually run and be able to FAIL. Never hardcode/fake PASS.
- Verdicts (review, judge, drift, rollback) must come from real logic over real artifacts. No hardcoded
approved/PASS. - No recycled trace IDs;
pipeline-context.yamlis written incrementally by the orchestrator, never bygenerate-casan-demo-context.py. - Do not weaken
.specify/scripts/bash/*or the two test suites; add new tests as new files. - The builder of a thing never scores it (see impartiality rule, Part 2).
- Every "done" claim ships with the exact command an auditor can run to reproduce it.
PART 5 — Reference docs in this repo
- phase1-hardening-reassessment.md — H2/H4/H5/H6 hardening + evidence
- phase2-independent-audit.md — H1/H3/H7 app+run audit + residual gaps
- casan-refined-reassessment-2026-06-28.md — original baseline self-assessment
.specify/tests/run-casan4-harness-tests.sh/adversarial-harness-tests.sh— the verification suitesdocs/output/casan/app-evidence/MANIFEST.md— app + pipeline evidence map