docs: honest hardening status + claim boundary (Plan-07 A/C-MVP, Plan-09)

- CASAN_HARDENING_STATUS.md: canonical implemented/scaffold/planned record with per-control test mapping and test inventory (baseline 79 + 61 new = 140 checks). - README claim boundary: separate implemented+tested controls from planned; explicitly does NOT claim full production-readiness (Track B, C-Gov/Ops, true sandbox isolation, IdP/WORM still planned). - INDEX status table: Plan-07 Track A + C-MVP done, Plan-09 MVP done. - video guide: core demo battery counts unchanged (hardening lives in separate suites); added deep-dive commands + 3 Q&A rows + sandbox honesty note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-03 23:07:31 +09:00
parent 62004c8725
commit 4cc78f74eb
4 changed files with 125 additions and 2 deletions
@@ -50,3 +50,30 @@ Expected: all PASS, including H4/H5/H6 and Level 5 evidence checks.
 - CASAN Level 4: achieved.
 - CASAN Level 5: demonstrated locally with signed policy, provider telemetry import, shared harness registry, fallback, drift detection, rollback, KPI feedback, and central dashboard.
 - Enterprise production Level 5 still needs live IdP, WORM storage, live provider API integration, and hosted dashboard.
+
+### Production hardening — implemented vs planned
+
+Following Plan-07/Plan-09, the following hardening is **implemented and tested**
+(each control has an executable adversarial test that fails if the control is
+removed). Full status: `casan-next-plans/CASAN_HARDENING_STATUS.md`.
+
+- **Implemented + tested (Plan-07 Track A):** H4 strict semantic fail-closed
+  (`CASAN_SECURITY_STRICT`), unicode/encoding normalization (homoglyph, zero-width,
+  fullwidth, base64/hex), tool-output injection scan, H5 telemetry-integrity signing,
+  H6 absolute + cumulative + cold-start cost controls, benign/false-positive budget
+  gate (FP ≤ 3%, adversarial block ≥ 95%, CRITICAL = 100%).
+- **Implemented + tested (Plan-07 Track C-MVP):** tool-authorization / action gating,
+  supply-chain gate, data-exfiltration guard.
+- **Scaffold + tested (Track C-MVP):** runtime sandbox — static policy + `ulimit`
+  backstops. **Not** kernel isolation; production needs container `--network=none
+  --read-only --pids-limit` / nsjail.
+- **Implemented + tested (Plan-09 MVP):** Evidence Pack — `casan pack` / `casan
+  verify-pack` (tamper-evident manifest + signed head + certified-run gate).
+- **Planned (NOT done — do not claim as production-ready):** multilingual H4,
+  classifier/split-injection resistance, HSM key management, policy-approval workflow,
+  external WORM audit, incident kill-switch, true sandbox isolation.
+
+Test totals: baseline 79 (run-casan4 35 + adversarial 44) preserved, **+61 new**
+hardening checks (Track A 25, Track C-MVP 29, Evidence Pack 7) = **140** — see
+`CASAN_HARDENING_STATUS.md`. Because these live in **separate** suites, the demo
+attack battery counts in `video/01_video_recording_guide.md` are unchanged.
@@ -95,6 +95,28 @@ CASAN_ROOT="$PWD" bash ../../optimize-docs/video-steps/scorecard.sh
 | "Recall 0.85 của 9B ở đâu ra?" | "Chạy `phase3-redteam-metrics.sh` LIVE trên camera → số thật của 9B. 0.85 là số dự án tự báo; regex = 0.00 trên paraphrase mới (tự chạy)." |
 | "Đây có phải Level 5?" | "Không — tôi tuyên bố Level 4 chứng minh được. Level 5 cần IdP/WORM/telemetry API/dashboard hosted — đó là roadmap." |
 | "Chạy được không cần Ollama?" | "Có — hầu hết vector offline. Model chỉ chạm H4 recall + H6 nguồn telemetry. H5 hoàn toàn không cần model. Cloud (OpenAI/Anthropic) đã hiện thực, chỉ cần key (chưa test key thật)." |
+| "Production hardening tới đâu rồi?" | "Đã hiện thực + test: Plan-07 **Track A** (strict fail-closed, unicode/base64, tool-output scan, telemetry-integrity, cost cap, FP budget) và **Track C-MVP** (tool-authz, supply-chain, data-exfil, sandbox scaffold), cùng **Evidence Pack** (Plan-09). Còn Track B + C-Governance/Ops + sandbox isolation thật là roadmap — nói rõ trong `CASAN_HARDENING_STATUS.md`." |
+| "Sandbox có phải cô lập thật không?" | "Chưa — C6 là static-policy + ulimit scaffold (chặn ~/.ssh, net egress, fork bomb, ghi ngoài workspace, huge-file). Cô lập thật cần container --network=none/--read-only/--pids-limit hoặc nsjail; đã ghi TODO(C6-prod)." |
+
+---
+
+## 5b. Production hardening (deep-dive tuỳ chọn — KHÔNG thay đổi battery chính)
+
+Bộ hardening Plan-07/Plan-09 nằm ở **các suite RIÊNG**, nên số liệu battery demo ở trên **không đổi**. Nếu giám khảo muốn xem sâu:
+
+```bash
+# Track A (25 checks): strict/unicode/tool-output/telemetry/cost/FP-budget
+bash .specify/tests/phase1-track-a-tests.sh
+# Track C-MVP (29 checks): tool-authz/supply-chain/data-exfil/sandbox
+bash .specify/tests/phase2-track-c-tests.sh
+# Evidence Pack MVP (7 checks): casan pack / verify-pack + tamper-evident
+bash .specify/tests/phase3-evidence-pack-tests.sh
+# Sinh 1 proof pack cho run bất kỳ:
+bash .specify/scripts/bash/evidence-pack.sh pack <run-id>
+bash .specify/scripts/bash/evidence-pack.sh verify-pack <run-id>
+```
+
+Tổng: baseline 79 (giữ nguyên) + 61 hardening = **140 checks**. Trạng thái trung thực (implemented / scaffold / planned): `casan-next-plans/CASAN_HARDENING_STATUS.md`.

 ---

@@ -0,0 +1,74 @@
+# CASAN Production Hardening — Implementation Status
+
+> Live record of what has been **implemented and tested** from Plan-07
+> (Production Hardening) and Plan-09 (Evidence Pack), versus what is still
+> **planned**. Honest by design: a control is only listed as done when it has an
+> executable test that would fail if the control were removed.
+>
+> Labels: **[implemented+tested]** · **[scaffold+tested]** (works, but a stronger
+> production form is documented as TODO) · **[planned]** (not built).
+
+## 1. What is implemented now
+
+### Phase 1 — Plan-07 Track A (low-risk hardening) — [implemented+tested]
+
+| ID | Control | Where | Test |
+|---|---|---|---|
+| A1 | H4 semantic **strict fail-closed** (`CASAN_SECURITY_STRICT=1`): model unavailable ⇒ BLOCK, never silent SKIP; non-strict logs `SEMANTIC_SKIPPED` loudly | `security-check.sh` | phase1 A1 |
+| A2 | H4 **unicode/encoding normalization**: NFKC + zero-width strip + homoglyph fold + base64/hex decode & rescan | `security-check.sh`, `unicode-normalize.py`, `decode-suspicious.py` | phase1 A2 |
+| A3 | H4 **tool-output injection scan** before output re-enters model context (mode off/warn/block; strict⇒block) | `tool-output-scan.sh`, `casan-harness.sh` | phase1 A3 |
+| A4 | H5 **telemetry integrity**: provider-usage + cost metrics bound to a signed manifest head (tamper ⇒ mismatch; re-forge ⇒ signature invalid) | `telemetry-integrity.sh`, `sign-audit-head.sh` | phase1 A4 |
+| A5 | H6 **cost controls**: absolute per-call cap + cumulative budget + cold-start protection, keeping median×mult spike | `cost-spike-detect.sh` | phase1 A5 |
+| A6 | **Benign corpus (EN/VI/JA, 95 samples) + FP budget gate**: FP ≤ 3%, adversarial block ≥ 95%, CRITICAL = 100% | `benign-fp-report.sh`, `benign-corpus/`, `redteam-vectors.jsonl` | phase1 A6 |
+
+### Phase 2 — Plan-07 Track C-MVP (production minimum bar) — mixed
+
+| ID | Control | Status | Where | Test |
+|---|---|---|---|---|
+| C1 | **Tool authorization / action gating** (ALLOW/WARN/REQUIRE_APPROVAL/BLOCK): sensitive-file writes, destructive/remote-exec commands, dep-install & network egress | [implemented+tested] | `action-gate.sh` | phase2 C1 |
+| C2 | **Supply-chain gate**: manifest diff, denylist + typosquat + dangerous lifecycle-script block, new-dep approval, dep-diff report | [implemented+tested] | `supply-chain-gate.sh`, `supply-chain-scan.py` | phase2 C2 |
+| C3 | **Data-exfil guard**: secret-to-cloud & artifact-leaks-env ⇒ BLOCK; PII-in-audit ⇒ mask | [implemented+tested] | `data-exfil-guard.sh` | phase2 C3 |
+| C6 | **Runtime sandbox**: static policy block (read ~/.ssh, net egress, fork bomb, write-outside-workspace, huge-file) + ulimit/timeout backstops | [scaffold+tested] | `sandbox-run.sh` | phase2 C6 |
+
+### Phase 3 — Plan-09 Evidence Pack (MVP) — [implemented+tested]
+
+| Control | Where | Test |
+|---|---|---|
+| `casan pack` / `casan verify-pack` (mapped to `evidence-pack.sh`): standard 12-file pack, hash manifest, signed head, tamper-evident verify, certified-run gate (no false certification, no silent skip) | `evidence-pack.sh`, `evidence-pack-build.py`, `evidence-pack-verify.py` | phase3-evidence-pack |
+
+## 2. Test inventory (all suites)
+
+| Suite | Checks | Purpose |
+|---|---|---|
+| `run-casan4-harness-tests.sh` | 35 | Baseline happy-path + Level-5 evidence (unchanged) |
+| `adversarial-harness-tests.sh` | 44 | Original adversarial battery (unchanged) |
+| `phase1-track-a-tests.sh` | 25 | **New** — Track A hardening |
+| `phase2-track-c-tests.sh` | 29 | **New** — Track C-MVP |
+| `phase3-evidence-pack-tests.sh` | 7 | **New** — Evidence Pack MVP |
+| **Total** | **140** | Baseline 79 preserved; +61 new hardening checks |
+
+Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so run it
+**first** and never concurrently with the other suites.
+
+## 3. What is NOT done (still planned — do not claim as production-ready)
+
+| Area | Status | Plan ref |
+|---|---|---|
+| H4 multilingual detection (VI/JA injection block-patterns) | [planned] | Plan-07 B1 (V2) |
+| Classifier-inject / split-injection resistance | [planned] | Plan-07 B2 (V5,V6) |
+| HSM/rotation key management, TOCTOU re-verify | [planned] | Plan-07 B3 (V10,V11) |
+| Model-digest pinning, sliding-window circuit breaker | [planned] | Plan-07 B4 (V15,V16) |
+| **True runtime isolation** (container `--network=none --read-only --pids-limit`, nsjail) | [planned] — C6 is a static+ulimit scaffold only | Plan-07 C6 (V22) |
+| Policy versioning + mandatory reviewer approval + rollback | [planned] | Plan-07 C4 (V20) |
+| External append-only (WORM) audit | [planned] | Plan-07 C5 (V21) |
+| Incident severity/kill-switch/runbook | [planned] | Plan-07 C7 (V23) |
+| Telemetry signing under Vault KMS path | [partial] — local-key path done; KMS path signs unsigned | Plan-07 A4 follow-up |
+| Live CVE/OSV scanning wired in | [partial] — availability detected; local denylist authoritative offline | Plan-07 C2 follow-up |
+
+## 4. Honest claim
+
+Track A + Track C-MVP + Evidence Pack raise H4/H5/H6 from "PoC/demo (~3.0/5)" toward
+**early internal-production hardening (~3.8–4.0)**, with executable adversarial tests
+for every control. This is **not** full production readiness: serious production
+still needs Track B, Track C-Governance/Ops, true sandbox isolation, live IdP/WORM,
+and hosted telemetry — enumerated above and in `CASAN_PLAN_07_PRODUCTION_HARDENING.md`.
@@ -64,9 +64,9 @@ flowchart LR
 | 04 Self-improve | ⬜ chưa bắt đầu | |
 | 05 CI/CD | ⬜ chưa bắt đầu | |
 | 06 Onboard | ⬜ chưa bắt đầu | |
-| 07 Production hardening | ⬜ chưa bắt đầu | Track A → **Track C-MVP (C1+C2+C3+C6)** là minimum bar trước khi cho agent ghi code/chạy test production-like |
+| 07 Production hardening | 🟡 Track A ✅ + Track C-MVP ✅ (B + C-Gov/Ops chưa) | **Track A (A1–A6) DONE + tested (25/25)**; **Track C-MVP: C1/C2/C3 DONE, C6 scaffold DONE + tested (29/29)**. Track B + C-Governance/Ops còn [planned]. Chi tiết: `CASAN_HARDENING_STATUS.md` |
 | 08 Context compression | ⬜ chưa bắt đầu | H1.5 cross-cutting; làm **sau** Plan-07 core (nén thêm bề mặt rủi ro) |
-| 09 Evidence Pack | ⬜ chưa bắt đầu | **Đáng làm sớm** — rẻ, đúng phương châm, ăn điểm thi |
+| 09 Evidence Pack | 🟢 MVP ✅ (7/7) | `casan pack`/`verify-pack` → `evidence-pack.sh`: 12-file pack + manifest + signed head + certified-run gate, tamper-evident. Chi tiết: `CASAN_HARDENING_STATUS.md` |
 | 10 Traceability + H3 Eval | ⬜ chưa bắt đầu | Khác biệt nhất; lõi = ma trận REQ→code→test |
 | 12 Domain Pack SDK | ⬜ chưa bắt đầu | Cần Plan-01 xong trước |
 | Future phases (B1–B6) | 💤 vision | Backlog `CASAN_PLAN_FUTURE_PHASES.md` — chưa xây |