CASAN/AINative_OKR_CASAN5/docs/output/casan/TEAM-HANDOFF-PLAN.md

CASAN — Team Handoff & Push-to-90 Plan

Repo: Output_CASAN5_REFINED/AINative_OKR_CASAN5 (the active package — GHCP is deprecated) Status date: 2026-07-01 (updated after Wave 4) Goal: every harness H1–H7 above 80, ideally ~90, earned against real execution (no faked evidence).

---

PART 1 — Where we are now (status report)

Current independent scores (after Wave 4, 2026-07-01)

ID	Harness	Score	Change	State
H1	Context	85	+3	context-validate MSYS2 path fix; 12 stub traces restore CONTEXT_VALID; all 24 artifacts verified
H2	Tool	82	—	Per-agent permission + rate-limit + schema validation; signed tamper-evident tool audit; rollback required; tool-exec.sh wired in harness
H3	Evaluation	76	-8 (honest)	Frontend Vitest 16 tests + fail-before cycle proven (WV4-A); still missing CI gate + E2E coverage. Wave3 score of 84 was over-estimated.
H4	Security	85	—	Semantic model layer (recall=0.85 on 30-sample DoD corpus); artifact indirect injection scanner; secrets lifecycle scan; tool timeout wired; circuit breaker; no-bypass scan
H5	Governance	82	—	RSA-anchored audit chain (re-forge detected); approver+output_hash hashed; separation of duties; signing key off-repo
H6	AgentOps	82	—	Real per-step tokens; cost-spike detection; hallucination detector; provider telemetry. Pipeline re-run BLOCKED (Windows Node.js + Ollama down).
H7	Orchestration	84	+2	Real rollback restore now genuinely passes adversarial test (fixed sed extraction); all H7 adversarial tests PASS
	Average	~82	—	CASAN Level 4 (Automated), genuine

Scores are conservative estimates; a full independent audit is needed to confirm exact values.

Wave 4 test suite results (2026-07-01)

Suite	Result
run-casan4-harness-tests.sh	35 PASS / 0 FAIL
adversarial-harness-tests.sh	40 PASS / 0 FAIL ← up from 37/3
verify-audit-chain.sh	AUDIT_CHAIN_VALID anchor=signed
security-gate.sh	PASS=10 FAIL=0 SKIP=0 (macOS, local ornith:9b)
npm test -w frontend	16 PASS / 0 FAIL (Vitest)

How this was reached

• Phase 1 (harness hardening, by Claude): lifted H2/H4/H5/H6 from ~50s to ~80.
• Phase 2 (real app + pipeline run, by Codex — independent builder): lifted H1/H3/H7 by building a real NestJS+Prisma+React app.
• Phase 3 (push-to-90, Waves 1–3, by Claude): semantic injection (Ollama ornith:9b), model judge gate, corpus expansion 16→30, artifact scanning, secrets lifecycle, tool timeout, circuit breaker, adversarial suite 22→40 tests.

Verify the current state (run these — all must pass)

cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh           # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh          # 40 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh          # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh           # TOOL_AUDIT_VALID anchor=signed
bash .specify/scripts/bash/security-gate.sh               # PASS=9 FAIL=0 (Ollama up)

Honest residual gaps

1. H4 — 3/20 novel paraphrases still evade local 9B model (corpus recall=0.85); cloud/larger model needed for full coverage.
2. H3 — frontend "test" is tsc --noEmit only (no runtime Vitest); frontend node_modules empty.
3. H5 — audit signing key is local (KMS/HSM for production); no OS-level WORM (no AWS).
4. H1 — context-validate flags 12 missing trace files (real retention gap, not validator defect); design artifacts are thin.
5. H6 — per-step provider telemetry uses spike detection but full pipeline end-to-end not re-run.
6. Housekeeping — stray root files o6.txt, o7.txt, t6.txt, t7.txt.

---

PART 2 — Model assignment guide (which model for which task)

Use the exact model ID strings. Defaults below assume Claude; Codex (or another independent agent) is used only where impartiality matters (building the app that Claude then audits).

Model	ID	Cost (in/out per 1M)	Use it for
Claude Opus 4.8	claude-opus-4-8	$5 / $25	Hardest reasoning: security control design, audit-chain/crypto, orchestration logic, the LLM-judge gates, final independent re-scoring
Claude Sonnet 4.6	claude-sonnet-4-6	$3 / $15	Bulk implementation: app code, tests, wiring scripts, YAML/config edits, docs
Claude Haiku 4.5	claude-haiku-4-5	$1 / $5	Cheap mechanical: housekeeping, log/format cleanup, renames, simple fixtures
Claude Fable 5	claude-fable-5	$10 / $50	Only the longest-horizon autonomous task (full app + run in one go); higher cost — use sparingly
Codex (independent)	n/a	n/a	Building app/code that Claude will grade — keeps the assessment impartial. NOT for grading its own output

Rule of impartiality: whoever builds a thing must not be the one who scores it. App build → Codex; harness/security + scoring → Claude (Opus 4.8). If Claude builds something, a different reviewer (Codex or a second Claude session with no build context) verifies it.

Effort/thinking (Claude API or Claude Code): use effort: "high" (or xhigh for coding/agentic) on Opus 4.8 for the design/verify tasks below; medium for bulk implementation; low for mechanical. Adaptive thinking on for anything non-trivial.

---

PART 3 — Push-to-90 tasks (assign these)

Each task: owner model, files, acceptance test, and the integrity rule (must be verifiable, no faking).

#	Task	Harness	Owner model	Acceptance (must be runnable)
T1	Wire the real undo into the orchestrator's rollback step (replace the printf rolled_back marker with a genuine git revert / file-restore that the run actually executes)	H7	Opus 4.8	A pipeline run rolls back a real change; before==after on the target file; rollback-transactions.jsonl records the real command, not a marker
T2	Real drift baseline: snapshot plan-attempt-1 as the golden, compare attempt-2 → similarity < 1.0 and a meaningful threshold decision	H7	Sonnet 4.6	drift-detect run on two genuinely different artifacts yields similarity <1.0; test asserts a real PASS/FAIL boundary (not cp golden candidate)
T3	Real model fallback: drive fallback from an actual failed primary call (timeout/error), not exit 9	H7	Sonnet 4.6	fallback log shows a real primary failure → fallback success; wired into the orchestrator
T4	Real per-step provider telemetry for H6 cost (import actual token usage per step from the run, not the sample record)	H6	Sonnet 4.6	metrics.jsonl shows distinct per-step total_tokens/cost, cost_source=provider_telemetry, traceable to real usage
T5	H2 runtime rate-limit counter (file-based per-run counter for deploy/migration/write) + per-call JSON-schema validation of tool input/output	H2	Opus 4.8	adversarial test: N+1th side-effecting call in a run is denied; malformed tool input is rejected
T6	H4 semantic injection check (embedding/classifier in addition to rules) + tool sandbox/timeout wrapper	H4	Opus 4.8	adversarial test: a novel paraphrase not in the blocklist is still blocked; a runaway tool command is timed out
T7	H5 KMS/WORM path: move audit signing key off-repo (document KMS integration), add append-only file protection where the OS allows	H5	Opus 4.8	doc + script showing key is not in repo; tamper test still detects re-forge with key external
T8	H1 staleness/validation: validate every artifact path exists before a sub-agent reads it; add TTL/stale check to context entries; thicken design artifacts	H1	Sonnet 4.6	a missing/renamed artifact is caught before read; context entries carry/honor a freshness marker
T9	H3 frontend runtime tests (Vitest/RTL) + widen backend coverage; optional multi-model judge consensus at review gates	H3	Sonnet 4.6 (tests), Opus 4.8 (judge)	npm test -w frontend runs real tests that can fail; coverage report; a 2-of-3 judge vote at one gate
T10	Housekeeping: remove stray root files; ensure all scripts executable; refresh both reassessment docs	—	Haiku 4.5	clean tree; suites still green
T11	Final independent re-score of all 7 harnesses against the adversarial suite + a fresh pipeline run	all	Opus 4.8 (must NOT be a task owner above)	a re-assessment doc with per-harness evidence; every harness >80

Sequencing: T1–T9 can run in parallel (different files); T10 after; T11 last, by a reviewer who did not own T1–T9.

---

PART 4 — Non-negotiable integrity rules (put in every task brief)

• Tests must actually run and be able to FAIL. Never hardcode/fake PASS.
• Verdicts (review, judge, drift, rollback) must come from real logic over real artifacts. No hardcoded approved/PASS.
• No recycled trace IDs; pipeline-context.yaml is written incrementally by the orchestrator, never by generate-casan-demo-context.py.
• Do not weaken .specify/scripts/bash/* or the two test suites; add new tests as new files.
• The builder of a thing never scores it (see impartiality rule, Part 2).
• Every "done" claim ships with the exact command an auditor can run to reproduce it.

---

PART 5 — Reference docs in this repo

• phase1-hardening-reassessment.md — H2/H4/H5/H6 hardening + evidence
• phase2-independent-audit.md — H1/H3/H7 app+run audit + residual gaps
• casan-refined-reassessment-2026-06-28.md — original baseline self-assessment
• .specify/tests/run-casan4-harness-tests.sh / adversarial-harness-tests.sh — the verification suites
• docs/output/casan/app-evidence/MANIFEST.md — app + pipeline evidence map