Files

T

thanhnvandClaude Opus 4.8 2af67ef6a5 docs: re-score from a real full run (155/0) after H5 hardening

Ran all 7 suites sequentially on 2026-07-04 @ 00aabfa with Vault dev live so KMS
runs (not skips): run-casan4 35 · adversarial 44 · phase1-track-a 25 ·
phase2-track-c 29 · phase3-evidence 7 · phase-h5-approval 8 · phase-h5-infra 7
= 155 PASS / 0 FAIL; security-gate PASS=11 FAIL=0.

- scoring-run-report.md: fair re-score — H5 76→80 (approval-identity + KMS live
  rotate/non-exportable + WORM), lowest harness now H6=79, avg ~80.7/100, Level 4.
  Evidence lists the live KMS + WORM results.
- CASAN_HARDENING_STATUS.md: new Phase 4 (C4 approval-identity, B3 KMS, C5 WORM =
  implemented+tested); test inventory 140→155 (7 suites); planned→partial for
  KMS/approval/WORM with honest remaining gaps (live IdP, S3 WORM store, KMS default).
- Plan-07 §2: key-mgmt 2.5→4, policy-approval 2.5→4, external-audit 1.5→3.5;
  header now H5 76→80, lowest harness H6.
- INDEX row, README claim boundary, video-guide Q&A: 155 checks, H5=80, lowest H6=79.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-07-04 23:51:50 +09:00

7.2 KiB

Raw Blame History

CASAN Production Hardening — Implementation Status

Live record of what has been implemented and tested from Plan-07 (Production Hardening) and Plan-09 (Evidence Pack), versus what is still planned. Honest by design: a control is only listed as done when it has an executable test that would fail if the control were removed.

Labels: [implemented+tested] · [scaffold+tested] (works, but a stronger production form is documented as TODO) · [planned] (not built).

1. What is implemented now

Phase 1 — Plan-07 Track A (low-risk hardening) — [implemented+tested]

ID	Control	Where	Test
A1	H4 semantic strict fail-closed (`CASAN_SECURITY_STRICT=1`): model unavailable ⇒ BLOCK, never silent SKIP; non-strict logs `SEMANTIC_SKIPPED` loudly	`security-check.sh`	phase1 A1
A2	H4 unicode/encoding normalization: NFKC + zero-width strip + homoglyph fold + base64/hex decode & rescan	`security-check.sh`, `unicode-normalize.py`, `decode-suspicious.py`	phase1 A2
A3	H4 tool-output injection scan before output re-enters model context (mode off/warn/block; strict⇒block)	`tool-output-scan.sh`, `casan-harness.sh`	phase1 A3
A4	H5 telemetry integrity: provider-usage + cost metrics bound to a signed manifest head (tamper ⇒ mismatch; re-forge ⇒ signature invalid)	`telemetry-integrity.sh`, `sign-audit-head.sh`	phase1 A4
A5	H6 cost controls: absolute per-call cap + cumulative budget + cold-start protection, keeping median×mult spike	`cost-spike-detect.sh`	phase1 A5
A6	Benign corpus (EN/VI/JA, 95 samples) + FP budget gate: FP ≤ 3%, adversarial block ≥ 95%, CRITICAL = 100%	`benign-fp-report.sh`, `benign-corpus/`, `redteam-vectors.jsonl`	phase1 A6

Phase 2 — Plan-07 Track C-MVP (production minimum bar) — mixed

ID	Control	Status	Where	Test
C1	Tool authorization / action gating (ALLOW/WARN/REQUIRE_APPROVAL/BLOCK): sensitive-file writes, destructive/remote-exec commands, dep-install & network egress	[implemented+tested]	`action-gate.sh`	phase2 C1
C2	Supply-chain gate: manifest diff, denylist + typosquat + dangerous lifecycle-script block, new-dep approval, dep-diff report	[implemented+tested]	`supply-chain-gate.sh`, `supply-chain-scan.py`	phase2 C2
C3	Data-exfil guard: secret-to-cloud & artifact-leaks-env ⇒ BLOCK; PII-in-audit ⇒ mask	[implemented+tested]	`data-exfil-guard.sh`	phase2 C3
C6	Runtime sandbox: static policy block (read ~/.ssh, net egress, fork bomb, write-outside-workspace, huge-file) + ulimit/timeout backstops	[scaffold+tested]	`sandbox-run.sh`	phase2 C6

Phase 3 — Plan-09 Evidence Pack (MVP) — [implemented+tested]

Control	Where	Test
`casan pack` / `casan verify-pack` (mapped to `evidence-pack.sh`): standard 12-file pack, hash manifest, signed head, tamper-evident verify, certified-run gate (no false certification, no silent skip)	`evidence-pack.sh`, `evidence-pack-build.py`, `evidence-pack-verify.py`	phase3-evidence-pack

Phase 4 — H5 governance hardening (raises the lowest harness) — mixed

ID	Control	Status	Where	Test
C4	Approval-identity: high-risk approval trusted only when a REGISTERED reviewer cryptographically SIGNS the request and their role is authorized — env-var approver no longer enough (SoD still enforced)	[implemented+tested]	`approval-sign.sh`, `approval-verify.sh`, `reviewers.registry`, `governance-check.sh` (`CASAN_APPROVAL_STRICT=1`)	phase-h5-approval (8)
B3	KMS key management: sign audit/telemetry head via Vault Transit (key `exportable:false` → never leaves KMS) + key rotation; validated live	[implemented+tested] (live when Vault present; skip-aware otherwise)	`vault-kms.sh` (`rotate`, `assert-nonexportable`), `sign-audit-head.sh`	phase-h5-infra (KMS)
C5	External WORM audit: ship audit head to a hash-linked append-only ledger; detect local rollback (`AUDIT_GAP_DETECTED`) and ledger tamper (`AUDIT_LEDGER_TAMPERED`)	[implemented+tested] (local ledger MVP)	`worm-ledger.py`, `audit-ship.sh`, `verify-audit-gap.sh`	phase-h5-infra (WORM)

2. Test inventory (all suites)

Suite	Checks	Purpose
`run-casan4-harness-tests.sh`	35	Baseline happy-path + Level-5 evidence (unchanged)
`adversarial-harness-tests.sh`	44	Original adversarial battery (unchanged)
`phase1-track-a-tests.sh`	25	Track A hardening
`phase2-track-c-tests.sh`	29	Track C-MVP
`phase3-evidence-pack-tests.sh`	7	Evidence Pack MVP
`phase-h5-approval-tests.sh`	8	New — approval-identity (C4)
`phase-h5-infra-tests.sh`	7	New — KMS (B3, live/skip-aware) + WORM (C5)
Total	155	Baseline 79 preserved; +76 new hardening checks. Last full run 2026-07-04 @ `00aabfa`, 0 fail (KMS live via Vault dev).

Run order note: run-casan4-harness-tests.sh does rm -rf .specify/logs, so run it first and never concurrently with the other suites.

3. What is NOT done (still planned — do not claim as production-ready)

Area	Status	Plan ref
H4 multilingual detection (VI/JA injection block-patterns)	[planned]	Plan-07 B1 (V2)
Classifier-inject / split-injection resistance	[planned]	Plan-07 B2 (V5,V6)
Model-digest pinning, sliding-window circuit breaker	[planned]	Plan-07 B4 (V15,V16)
True runtime isolation (container `--network=none --read-only --pids-limit`, nsjail)	[planned] — C6 is a static+ulimit scaffold only	Plan-07 C6 (V22)
Incident severity/kill-switch/runbook	[planned]	Plan-07 C7 (V23)
KMS key management (rotation, non-exportable)	[partial] — Vault Transit path implemented + validated live; not yet the default (local-key fallback), no HSM/short-lived IdP tokens	Plan-07 B3
Reviewer approval workflow	[partial] — cryptographic approval-identity done (signed reviewer + role); live IdP (OIDC/JWT) + policy versioning/diff still planned	Plan-07 C4 (V20)
External append-only (WORM) audit	[partial] — hash-linked local ledger + rollback/tamper detection done; true WORM store (S3 Object Lock/QLDB) + trusted timestamp planned	Plan-07 C5 (V21)
Live CVE/OSV scanning wired in	[partial] — availability detected; local denylist authoritative offline	Plan-07 C2 follow-up

4. Honest claim

Track A + Track C-MVP + Evidence Pack + H5 governance-hardening raise H4/H5/H6 from "PoC/demo (~3.0/5)" to early internal-production hardening, with executable adversarial tests for every control (155 checks, 0 fail — last full run 2026-07-04, KMS validated live via Vault). Fair maturity score (00_SUBMISSION_PACKAGE/evidence/ scoring-run-report.md): per-harness ~80/100, H5 76→80 (approval-identity + KMS live + WORM), so the lowest harness is now H6=79 (was H5=76) — CASAN Level 4, proven by attack. This is not full production readiness: serious production still needs live IdP (OIDC/JWT), a true WORM store (S3 Object Lock), KMS-by-default + HSM, true sandbox isolation, multilingual detection, and hosted telemetry — the [partial]/ [planned] rows above and in CASAN_PLAN_07_PRODUCTION_HARDENING.md.

7.2 KiB Raw Blame History Unescape Escape