diff --git a/00_SUBMISSION_PACKAGE/README.md b/00_SUBMISSION_PACKAGE/README.md index e02845f..b877396 100644 --- a/00_SUBMISSION_PACKAGE/README.md +++ b/00_SUBMISSION_PACKAGE/README.md @@ -50,3 +50,30 @@ Expected: all PASS, including H4/H5/H6 and Level 5 evidence checks. - CASAN Level 4: achieved. - CASAN Level 5: demonstrated locally with signed policy, provider telemetry import, shared harness registry, fallback, drift detection, rollback, KPI feedback, and central dashboard. - Enterprise production Level 5 still needs live IdP, WORM storage, live provider API integration, and hosted dashboard. + +### Production hardening — implemented vs planned + +Following Plan-07/Plan-09, the following hardening is **implemented and tested** +(each control has an executable adversarial test that fails if the control is +removed). Full status: `casan-next-plans/CASAN_HARDENING_STATUS.md`. + +- **Implemented + tested (Plan-07 Track A):** H4 strict semantic fail-closed + (`CASAN_SECURITY_STRICT`), unicode/encoding normalization (homoglyph, zero-width, + fullwidth, base64/hex), tool-output injection scan, H5 telemetry-integrity signing, + H6 absolute + cumulative + cold-start cost controls, benign/false-positive budget + gate (FP ≤ 3%, adversarial block ≥ 95%, CRITICAL = 100%). +- **Implemented + tested (Plan-07 Track C-MVP):** tool-authorization / action gating, + supply-chain gate, data-exfiltration guard. +- **Scaffold + tested (Track C-MVP):** runtime sandbox — static policy + `ulimit` + backstops. **Not** kernel isolation; production needs container `--network=none + --read-only --pids-limit` / nsjail. +- **Implemented + tested (Plan-09 MVP):** Evidence Pack — `casan pack` / `casan + verify-pack` (tamper-evident manifest + signed head + certified-run gate). +- **Planned (NOT done — do not claim as production-ready):** multilingual H4, + classifier/split-injection resistance, HSM key management, policy-approval workflow, + external WORM audit, incident kill-switch, true sandbox isolation. + +Test totals: baseline 79 (run-casan4 35 + adversarial 44) preserved, **+61 new** +hardening checks (Track A 25, Track C-MVP 29, Evidence Pack 7) = **140** — see +`CASAN_HARDENING_STATUS.md`. Because these live in **separate** suites, the demo +attack battery counts in `video/01_video_recording_guide.md` are unchanged. diff --git a/00_SUBMISSION_PACKAGE/video/01_video_recording_guide.md b/00_SUBMISSION_PACKAGE/video/01_video_recording_guide.md index 4465ed2..fd2b178 100644 --- a/00_SUBMISSION_PACKAGE/video/01_video_recording_guide.md +++ b/00_SUBMISSION_PACKAGE/video/01_video_recording_guide.md @@ -95,6 +95,28 @@ CASAN_ROOT="$PWD" bash ../../optimize-docs/video-steps/scorecard.sh | "Recall 0.85 của 9B ở đâu ra?" | "Chạy `phase3-redteam-metrics.sh` LIVE trên camera → số thật của 9B. 0.85 là số dự án tự báo; regex = 0.00 trên paraphrase mới (tự chạy)." | | "Đây có phải Level 5?" | "Không — tôi tuyên bố Level 4 chứng minh được. Level 5 cần IdP/WORM/telemetry API/dashboard hosted — đó là roadmap." | | "Chạy được không cần Ollama?" | "Có — hầu hết vector offline. Model chỉ chạm H4 recall + H6 nguồn telemetry. H5 hoàn toàn không cần model. Cloud (OpenAI/Anthropic) đã hiện thực, chỉ cần key (chưa test key thật)." | +| "Production hardening tới đâu rồi?" | "Đã hiện thực + test: Plan-07 **Track A** (strict fail-closed, unicode/base64, tool-output scan, telemetry-integrity, cost cap, FP budget) và **Track C-MVP** (tool-authz, supply-chain, data-exfil, sandbox scaffold), cùng **Evidence Pack** (Plan-09). Còn Track B + C-Governance/Ops + sandbox isolation thật là roadmap — nói rõ trong `CASAN_HARDENING_STATUS.md`." | +| "Sandbox có phải cô lập thật không?" | "Chưa — C6 là static-policy + ulimit scaffold (chặn ~/.ssh, net egress, fork bomb, ghi ngoài workspace, huge-file). Cô lập thật cần container --network=none/--read-only/--pids-limit hoặc nsjail; đã ghi TODO(C6-prod)." | + +--- + +## 5b. Production hardening (deep-dive tuỳ chọn — KHÔNG thay đổi battery chính) + +Bộ hardening Plan-07/Plan-09 nằm ở **các suite RIÊNG**, nên số liệu battery demo ở trên **không đổi**. Nếu giám khảo muốn xem sâu: + +```bash +# Track A (25 checks): strict/unicode/tool-output/telemetry/cost/FP-budget +bash .specify/tests/phase1-track-a-tests.sh +# Track C-MVP (29 checks): tool-authz/supply-chain/data-exfil/sandbox +bash .specify/tests/phase2-track-c-tests.sh +# Evidence Pack MVP (7 checks): casan pack / verify-pack + tamper-evident +bash .specify/tests/phase3-evidence-pack-tests.sh +# Sinh 1 proof pack cho run bất kỳ: +bash .specify/scripts/bash/evidence-pack.sh pack +bash .specify/scripts/bash/evidence-pack.sh verify-pack +``` + +Tổng: baseline 79 (giữ nguyên) + 61 hardening = **140 checks**. Trạng thái trung thực (implemented / scaffold / planned): `casan-next-plans/CASAN_HARDENING_STATUS.md`. --- diff --git a/casan-next-plans/CASAN_HARDENING_STATUS.md b/casan-next-plans/CASAN_HARDENING_STATUS.md new file mode 100644 index 0000000..8bf00fe --- /dev/null +++ b/casan-next-plans/CASAN_HARDENING_STATUS.md @@ -0,0 +1,74 @@ +# CASAN Production Hardening — Implementation Status + +> Live record of what has been **implemented and tested** from Plan-07 +> (Production Hardening) and Plan-09 (Evidence Pack), versus what is still +> **planned**. Honest by design: a control is only listed as done when it has an +> executable test that would fail if the control were removed. +> +> Labels: **[implemented+tested]** · **[scaffold+tested]** (works, but a stronger +> production form is documented as TODO) · **[planned]** (not built). + +## 1. What is implemented now + +### Phase 1 — Plan-07 Track A (low-risk hardening) — [implemented+tested] + +| ID | Control | Where | Test | +|---|---|---|---| +| A1 | H4 semantic **strict fail-closed** (`CASAN_SECURITY_STRICT=1`): model unavailable ⇒ BLOCK, never silent SKIP; non-strict logs `SEMANTIC_SKIPPED` loudly | `security-check.sh` | phase1 A1 | +| A2 | H4 **unicode/encoding normalization**: NFKC + zero-width strip + homoglyph fold + base64/hex decode & rescan | `security-check.sh`, `unicode-normalize.py`, `decode-suspicious.py` | phase1 A2 | +| A3 | H4 **tool-output injection scan** before output re-enters model context (mode off/warn/block; strict⇒block) | `tool-output-scan.sh`, `casan-harness.sh` | phase1 A3 | +| A4 | H5 **telemetry integrity**: provider-usage + cost metrics bound to a signed manifest head (tamper ⇒ mismatch; re-forge ⇒ signature invalid) | `telemetry-integrity.sh`, `sign-audit-head.sh` | phase1 A4 | +| A5 | H6 **cost controls**: absolute per-call cap + cumulative budget + cold-start protection, keeping median×mult spike | `cost-spike-detect.sh` | phase1 A5 | +| A6 | **Benign corpus (EN/VI/JA, 95 samples) + FP budget gate**: FP ≤ 3%, adversarial block ≥ 95%, CRITICAL = 100% | `benign-fp-report.sh`, `benign-corpus/`, `redteam-vectors.jsonl` | phase1 A6 | + +### Phase 2 — Plan-07 Track C-MVP (production minimum bar) — mixed + +| ID | Control | Status | Where | Test | +|---|---|---|---|---| +| C1 | **Tool authorization / action gating** (ALLOW/WARN/REQUIRE_APPROVAL/BLOCK): sensitive-file writes, destructive/remote-exec commands, dep-install & network egress | [implemented+tested] | `action-gate.sh` | phase2 C1 | +| C2 | **Supply-chain gate**: manifest diff, denylist + typosquat + dangerous lifecycle-script block, new-dep approval, dep-diff report | [implemented+tested] | `supply-chain-gate.sh`, `supply-chain-scan.py` | phase2 C2 | +| C3 | **Data-exfil guard**: secret-to-cloud & artifact-leaks-env ⇒ BLOCK; PII-in-audit ⇒ mask | [implemented+tested] | `data-exfil-guard.sh` | phase2 C3 | +| C6 | **Runtime sandbox**: static policy block (read ~/.ssh, net egress, fork bomb, write-outside-workspace, huge-file) + ulimit/timeout backstops | [scaffold+tested] | `sandbox-run.sh` | phase2 C6 | + +### Phase 3 — Plan-09 Evidence Pack (MVP) — [implemented+tested] + +| Control | Where | Test | +|---|---|---| +| `casan pack` / `casan verify-pack` (mapped to `evidence-pack.sh`): standard 12-file pack, hash manifest, signed head, tamper-evident verify, certified-run gate (no false certification, no silent skip) | `evidence-pack.sh`, `evidence-pack-build.py`, `evidence-pack-verify.py` | phase3-evidence-pack | + +## 2. Test inventory (all suites) + +| Suite | Checks | Purpose | +|---|---|---| +| `run-casan4-harness-tests.sh` | 35 | Baseline happy-path + Level-5 evidence (unchanged) | +| `adversarial-harness-tests.sh` | 44 | Original adversarial battery (unchanged) | +| `phase1-track-a-tests.sh` | 25 | **New** — Track A hardening | +| `phase2-track-c-tests.sh` | 29 | **New** — Track C-MVP | +| `phase3-evidence-pack-tests.sh` | 7 | **New** — Evidence Pack MVP | +| **Total** | **140** | Baseline 79 preserved; +61 new hardening checks | + +Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so run it +**first** and never concurrently with the other suites. + +## 3. What is NOT done (still planned — do not claim as production-ready) + +| Area | Status | Plan ref | +|---|---|---| +| H4 multilingual detection (VI/JA injection block-patterns) | [planned] | Plan-07 B1 (V2) | +| Classifier-inject / split-injection resistance | [planned] | Plan-07 B2 (V5,V6) | +| HSM/rotation key management, TOCTOU re-verify | [planned] | Plan-07 B3 (V10,V11) | +| Model-digest pinning, sliding-window circuit breaker | [planned] | Plan-07 B4 (V15,V16) | +| **True runtime isolation** (container `--network=none --read-only --pids-limit`, nsjail) | [planned] — C6 is a static+ulimit scaffold only | Plan-07 C6 (V22) | +| Policy versioning + mandatory reviewer approval + rollback | [planned] | Plan-07 C4 (V20) | +| External append-only (WORM) audit | [planned] | Plan-07 C5 (V21) | +| Incident severity/kill-switch/runbook | [planned] | Plan-07 C7 (V23) | +| Telemetry signing under Vault KMS path | [partial] — local-key path done; KMS path signs unsigned | Plan-07 A4 follow-up | +| Live CVE/OSV scanning wired in | [partial] — availability detected; local denylist authoritative offline | Plan-07 C2 follow-up | + +## 4. Honest claim + +Track A + Track C-MVP + Evidence Pack raise H4/H5/H6 from "PoC/demo (~3.0/5)" toward +**early internal-production hardening (~3.8–4.0)**, with executable adversarial tests +for every control. This is **not** full production readiness: serious production +still needs Track B, Track C-Governance/Ops, true sandbox isolation, live IdP/WORM, +and hosted telemetry — enumerated above and in `CASAN_PLAN_07_PRODUCTION_HARDENING.md`. diff --git a/casan-next-plans/CASAN_PLAN_00_INDEX.md b/casan-next-plans/CASAN_PLAN_00_INDEX.md index 8c53495..550b174 100644 --- a/casan-next-plans/CASAN_PLAN_00_INDEX.md +++ b/casan-next-plans/CASAN_PLAN_00_INDEX.md @@ -64,9 +64,9 @@ flowchart LR | 04 Self-improve | ⬜ chưa bắt đầu | | | 05 CI/CD | ⬜ chưa bắt đầu | | | 06 Onboard | ⬜ chưa bắt đầu | | -| 07 Production hardening | ⬜ chưa bắt đầu | Track A → **Track C-MVP (C1+C2+C3+C6)** là minimum bar trước khi cho agent ghi code/chạy test production-like | +| 07 Production hardening | 🟡 Track A ✅ + Track C-MVP ✅ (B + C-Gov/Ops chưa) | **Track A (A1–A6) DONE + tested (25/25)**; **Track C-MVP: C1/C2/C3 DONE, C6 scaffold DONE + tested (29/29)**. Track B + C-Governance/Ops còn [planned]. Chi tiết: `CASAN_HARDENING_STATUS.md` | | 08 Context compression | ⬜ chưa bắt đầu | H1.5 cross-cutting; làm **sau** Plan-07 core (nén thêm bề mặt rủi ro) | -| 09 Evidence Pack | ⬜ chưa bắt đầu | **Đáng làm sớm** — rẻ, đúng phương châm, ăn điểm thi | +| 09 Evidence Pack | 🟢 MVP ✅ (7/7) | `casan pack`/`verify-pack` → `evidence-pack.sh`: 12-file pack + manifest + signed head + certified-run gate, tamper-evident. Chi tiết: `CASAN_HARDENING_STATUS.md` | | 10 Traceability + H3 Eval | ⬜ chưa bắt đầu | Khác biệt nhất; lõi = ma trận REQ→code→test | | 12 Domain Pack SDK | ⬜ chưa bắt đầu | Cần Plan-01 xong trước | | Future phases (B1–B6) | 💤 vision | Backlog `CASAN_PLAN_FUTURE_PHASES.md` — chưa xây |