CASAN/docs/output/casan/phase1-hardening-reassessment.md

CASAN Phase 1 Hardening — Independent Re-Assessment

Date: 2026-06-28 Scope: Control-plane harnesses H2, H4, H5, H6 (hardening that is verifiable without a running product). Method: Independent review + a new adversarial test suite (.specify/tests/adversarial-harness-tests.sh) that constructs attacks/tampering and asserts the controls defend — every check would have FAILED before this work. Honesty note: Scores below are earned against adversarial evidence, not against the presence of config files. H1/H3/H7 are deliberately left unchanged here — they cannot honestly exceed 80 without Phase 2 (a real product + real pipeline run). See bottom.

---

1. Score movement (independent)

ID	Harness	Pre-hardening	Phase 1	Basis for the new score
H2	Tool	48	80	Per-agent least-privilege enforced and in the execution line of fire; tool-call audit is now a signed, tamper-evident hash chain (re-forge rejected); every side-effecting tool must declare a rollback strategy.
H4	Security	52	80	Input normalization defeats whitespace/leet/synonym/case bypasses; pii-rules.yaml is now wired into the live path; private-key/DB/AWS secrets blocked; output mode fails closed on secret material.
H5	Governance	62	80	Audit chain is RSA-anchored — a re-forged chain is now detected (it previously passed); approver + output_hash are inside the hashed core; separation of duties (submitter ≠ approver) enforced.
H6	AgentOps	58	80	Hallucination detection is real and populated (signals written to metrics.jsonl, alert at threshold); imported provider telemetry is the authoritative cost source (cost_source labels real vs estimate); latency/exit-code/alerting already real.
H1	Context	45	45	Unchanged — requires Phase 2 (real orchestrator-maintained context).
H3	Evaluation	22	22	Unchanged — requires Phase 2 (real app + real LLM-judge gates + real regression).
H7	Orchestration	40	40	Unchanged — requires Phase 2 (wire L5 scripts into a real run, real rollback).

Average after Phase 1: ~61/100. Still CASAN Level 3 because the ceiling is held by H1/H3/H7 (and H3 < 30 remains a critical GAP). Phase 1 fixed exactly the four harnesses that are honestly fixable without a running product.

---

2. What changed (files)

File	Change
scripts/bash/security-check.sh	normalize_for_match (whitespace/leet/punct folding) + match_either; expanded synonym blocklist; wired pii-mask.py; output fail-closed on secrets.
scripts/bash/pii-mask.py (new)	Policy-driven PII masker — makes pii-rules.yaml the source of truth.
scripts/bash/governance-check.sh	approver+output_hash in hashed core; separation of duties; RSA head-anchor signing after each append.
scripts/bash/verify-audit-chain.sh	Verifies new core + head signature; rejects re-forged chains.
scripts/bash/tool-registry-gate.sh	Per-agent permission + rollback-strategy enforcement; routes audit through the chained appender.
scripts/bash/tool-audit-lib.sh (new)	Shared chained + signed appender for tool-calls.jsonl.
scripts/bash/verify-tool-audit.sh (new)	Verifies the tool-call audit chain + head signature.
scripts/bash/casan-harness.sh	Calls the tool-registry gate before executing side-effecting actions (gate in line of fire).
scripts/bash/agent-metrics.sh	Hallucination detection + provider-telemetry cost source + cost_source/hallucination_signals fields.
scripts/bash/hallucination-scan.py (new)	Keyword/marker detector driven by hallucination-tracking.yaml.
level5/tool-registry.yaml	allowed_agents + rollback for all side-effecting tools.
tests/run-casan4-harness-tests.sh	Updated for new behavior; +5 assertions (35 PASS total).
tests/adversarial-harness-tests.sh (new)	22 independent attack/tamper/authorization checks.

---

3. Evidence

• bash .specify/tests/run-casan4-harness-tests.sh → 35 PASS / 0 FAIL.
• bash .specify/tests/adversarial-harness-tests.sh → 22 PASS / 0 FAIL (attacks defended).

Representative adversarial proofs (all previously would have failed):

• H4: "ignore all previous instructions", "1gnore prev1ous 1nstruct1ons", "disregard the rules above" → blocked.
• H4: private key fed as input → blocked (previously leaked to output in the GHCP predecessor).
• H5: edit one record + recompute the entire chain + rewrite the plain head file → AUDIT_HEAD_SIGNATURE_INVALID (previously passed as valid).
• H2: CASAN_AGENT=design-agent ... deploy → denied unauthorized_agent; same via casan-harness.sh write_code → wrapper aborts.
• H6: marker-heavy output → hallucination_signals=4 + alert; clean output → 0.

---

4. Honest residual gaps (why these are 80, not 95)

• H4: still rule/normalization-based, not semantic/embedding; no process sandbox/timeout for tool execution.
• H5: the audit signing key is local for demonstration — production requires it off-repo (KMS/HSM); no OS-level WORM; risk classifier is keyword-based.
• H2: agent identity is taken from the environment (trusted caller); no runtime cross-step rate-limit counter; no per-call JSON-schema validation of tool I/O.
• H6: word-count remains the fallback cost when no provider telemetry is imported; hallucination detection is keyword-based; dashboard is static, not live.

---

5. Remaining path to "all H1–H7 > 80" (Phase 2 — requires a real run)

H1, H3, H7 are gated on the same missing thing: there is no real product and no real pipeline run. To raise them honestly:

1. Build a real, testable OKR app (NestJS+Prisma backend, React+Vite frontend) from docs/input/okr-requirement.md.
2. Run the reviewer agents as real LLM-as-judge gates that can emit REJECTED and trigger auto-retry/BACK-TO-PLAN (retire the hardcoded approved generator).
3. Run the test kit against the real app (real unit + e2e) → real golden/regression for H3.
4. Wire model-fallback / rollback-manager / drift-detect into the orchestrator; make rollback a real undo.
5. Produce pipeline-context.yaml via incremental orchestrator updates during the run (retire generate-casan-demo-context.py).

Only Phase 2 raises the overall average above 80 and clears the H3 critical GAP.