gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: re-score from a real full run (155/0) after H5 hardening

Browse Source 
Ran all 7 suites sequentially on 2026-07-04 @ 00aabfa with Vault dev live so KMS
runs (not skips): run-casan4 35 · adversarial 44 · phase1-track-a 25 ·
phase2-track-c 29 · phase3-evidence 7 · phase-h5-approval 8 · phase-h5-infra 7
= 155 PASS / 0 FAIL; security-gate PASS=11 FAIL=0.

- scoring-run-report.md: fair re-score — H5 76→80 (approval-identity + KMS live
  rotate/non-exportable + WORM), lowest harness now H6=79, avg ~80.7/100, Level 4.
  Evidence lists the live KMS + WORM results.
- CASAN_HARDENING_STATUS.md: new Phase 4 (C4 approval-identity, B3 KMS, C5 WORM =
  implemented+tested); test inventory 140→155 (7 suites); planned→partial for
  KMS/approval/WORM with honest remaining gaps (live IdP, S3 WORM store, KMS default).
- Plan-07 §2: key-mgmt 2.5→4, policy-approval 2.5→4, external-audit 1.5→3.5;
  header now H5 76→80, lowest harness H6.
- INDEX row, README claim boundary, video-guide Q&A: 155 checks, H5=80, lowest H6=79.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-04 23:51:50 +09:00
co-authored by Claude Opus 4.8
parent 00aabfac4d
commit 2af67ef6a5
6 changed files with 117 additions and 28 deletions
Download Patch File Download Diff File Expand all files Collapse all files
casan-next-plans/CASAN_HARDENING_STATUS.md
+27 -13
View File
@@ -36,16 +36,26 @@
|---|---|---|
| `casan pack` / `casan verify-pack` (mapped to `evidence-pack.sh`): standard 12-file pack, hash manifest, signed head, tamper-evident verify, certified-run gate (no false certification, no silent skip) | `evidence-pack.sh`, `evidence-pack-build.py`, `evidence-pack-verify.py` | phase3-evidence-pack |
### Phase 4 — H5 governance hardening (raises the lowest harness) — mixed
| ID | Control | Status | Where | Test |
|---|---|---|---|---|
| C4 | **Approval-identity**: high-risk approval trusted only when a REGISTERED reviewer cryptographically SIGNS the request and their role is authorized — env-var approver no longer enough (SoD still enforced) | [implemented+tested] | `approval-sign.sh`, `approval-verify.sh`, `reviewers.registry`, `governance-check.sh` (`CASAN_APPROVAL_STRICT=1`) | phase-h5-approval (8) |
| B3 | **KMS key management**: sign audit/telemetry head via Vault Transit (key `exportable:false` → never leaves KMS) + key **rotation**; validated live | [implemented+tested] (live when Vault present; skip-aware otherwise) | `vault-kms.sh` (`rotate`, `assert-nonexportable`), `sign-audit-head.sh` | phase-h5-infra (KMS) |
| C5 | **External WORM audit**: ship audit head to a hash-linked append-only ledger; detect local rollback (`AUDIT_GAP_DETECTED`) and ledger tamper (`AUDIT_LEDGER_TAMPERED`) | [implemented+tested] (local ledger MVP) | `worm-ledger.py`, `audit-ship.sh`, `verify-audit-gap.sh` | phase-h5-infra (WORM) |
## 2. Test inventory (all suites)
| Suite | Checks | Purpose |
|---|---|---|
| `run-casan4-harness-tests.sh` | 35 | Baseline happy-path + Level-5 evidence (unchanged) |
| `adversarial-harness-tests.sh` | 44 | Original adversarial battery (unchanged) |
| `phase1-track-a-tests.sh` | 25 | **New** — Track A hardening |
| `phase2-track-c-tests.sh` | 29 | **New** — Track C-MVP |
| `phase3-evidence-pack-tests.sh` | 7 | **New** — Evidence Pack MVP |
| **Total** | **140** | Baseline 79 preserved; +61 new hardening checks |
| `phase1-track-a-tests.sh` | 25 | Track A hardening |
| `phase2-track-c-tests.sh` | 29 | Track C-MVP |
| `phase3-evidence-pack-tests.sh` | 7 | Evidence Pack MVP |
| `phase-h5-approval-tests.sh` | 8 | **New** — approval-identity (C4) |
| `phase-h5-infra-tests.sh` | 7 | **New** — KMS (B3, live/skip-aware) + WORM (C5) |
| **Total** | **155** | Baseline 79 preserved; +76 new hardening checks. Last full run 2026-07-04 @ `00aabfa`, 0 fail (KMS live via Vault dev). |
Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so run it
**first** and never concurrently with the other suites.
@@ -56,19 +66,23 @@ Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so ru
|---|---|---|
| H4 multilingual detection (VI/JA injection block-patterns) | [planned] | Plan-07 B1 (V2) |
| Classifier-inject / split-injection resistance | [planned] | Plan-07 B2 (V5,V6) |
| HSM/rotation key management, TOCTOU re-verify | [planned] | Plan-07 B3 (V10,V11) |
| Model-digest pinning, sliding-window circuit breaker | [planned] | Plan-07 B4 (V15,V16) |
| **True runtime isolation** (container `--network=none --read-only --pids-limit`, nsjail) | [planned] — C6 is a static+ulimit scaffold only | Plan-07 C6 (V22) |
| Policy versioning + mandatory reviewer approval + rollback | [planned] | Plan-07 C4 (V20) |
| External append-only (WORM) audit | [planned] | Plan-07 C5 (V21) |
| Incident severity/kill-switch/runbook | [planned] | Plan-07 C7 (V23) |
| Telemetry signing under Vault KMS path | [partial] — local-key path done; KMS path signs unsigned | Plan-07 A4 follow-up |
| KMS key management (rotation, non-exportable) | [partial] — Vault Transit path implemented + validated live; not yet the default (local-key fallback), no HSM/short-lived IdP tokens | Plan-07 B3 |
| Reviewer approval workflow | [partial] — cryptographic **approval-identity** done (signed reviewer + role); live **IdP (OIDC/JWT)** + policy versioning/diff still planned | Plan-07 C4 (V20) |
| External append-only (WORM) audit | [partial] — hash-linked local ledger + rollback/tamper detection done; true WORM store (S3 Object Lock/QLDB) + trusted timestamp planned | Plan-07 C5 (V21) |
| Live CVE/OSV scanning wired in | [partial] — availability detected; local denylist authoritative offline | Plan-07 C2 follow-up |
## 4. Honest claim
Track A + Track C-MVP + Evidence Pack raise H4/H5/H6 from "PoC/demo (~3.0/5)" toward
**early internal-production hardening (~3.8–4.0)**, with executable adversarial tests
for every control. This is **not** full production readiness: serious production
still needs Track B, Track C-Governance/Ops, true sandbox isolation, live IdP/WORM,
and hosted telemetry — enumerated above and in `CASAN_PLAN_07_PRODUCTION_HARDENING.md`.
Track A + Track C-MVP + Evidence Pack + H5 governance-hardening raise H4/H5/H6 from
"PoC/demo (~3.0/5)" to **early internal-production hardening**, with executable
adversarial tests for every control (155 checks, 0 fail — last full run 2026-07-04,
KMS validated live via Vault). Fair maturity score (`00_SUBMISSION_PACKAGE/evidence/
scoring-run-report.md`): per-harness **~80/100**, **H5 76→80** (approval-identity +
KMS live + WORM), so the **lowest harness is now H6=79** (was H5=76) — CASAN **Level 4**,
proven by attack. This is **not** full production readiness: serious production still
needs live IdP (OIDC/JWT), a true WORM store (S3 Object Lock), KMS-by-default + HSM,
true sandbox isolation, multilingual detection, and hosted telemetry — the [partial]/
[planned] rows above and in `CASAN_PLAN_07_PRODUCTION_HARDENING.md`.
casan-next-plans/CASAN_PLAN_00_INDEX.md
+1 -1
View File
@@ -64,7 +64,7 @@ flowchart LR
| 04 Self-improve | ⬜ chưa bắt đầu | |
| 05 CI/CD | ⬜ chưa bắt đầu | |
| 06 Onboard | ⬜ chưa bắt đầu | |
| 07 Production hardening | 🟡 Track A ✅ + Track C-MVP ✅ (B + C-Gov/Ops chưa) | **Track A (A1–A6) DONE + tested (25/25)**; **Track C-MVP: C1/C2/C3 DONE, C6 scaffold DONE + tested (29/29)**. Track B + C-Governance/Ops còn [planned]. Chi tiết: `CASAN_HARDENING_STATUS.md` |
| 07 Production hardening | 🟡 Track A ✅ + C-MVP ✅ + H5-hardening ✅ (B + C7 + IdP/WORM-store chưa) | **Track A 25/25 · Track C-MVP 29/29 · H5-hardening (C4 approval-identity 8/8 + KMS live + C5 WORM = infra 7/7)**. Điểm công tâm: H5 76→80, harness thấp nhất giờ H6=79, ~80.7/100. Track B + C7 + live IdP/WORM-store còn [planned]. Chi tiết: `CASAN_HARDENING_STATUS.md` · `evidence/scoring-run-report.md` |
| 08 Context compression | ⬜ chưa bắt đầu | H1.5 cross-cutting; làm **sau** Plan-07 core (nén thêm bề mặt rủi ro) |
| 09 Evidence Pack | 🟢 MVP ✅ (7/7) | `casan pack`/`verify-pack` → `evidence-pack.sh`: 12-file pack + manifest + signed head + certified-run gate, tamper-evident. Chi tiết: `CASAN_HARDENING_STATUS.md` |
| 10 Traceability + H3 Eval | ⬜ chưa bắt đầu | Khác biệt nhất; lõi = ma trận REQ→code→test |
casan-next-plans/CASAN_PLAN_07_PRODUCTION_HARDENING.md
+9 -9
View File
@@ -19,23 +19,23 @@
## 2. Thang điểm sẵn sàng production (0–5, cao = tốt)
> ✅ **CẬP NHẬT 2026-07-04 — Track A + C-MVP + Evidence Pack ĐÃ LÀM + TEST (140 checks, 0 fail).**
> ✅ **CẬP NHẬT 2026-07-04 — Track A + C-MVP + Evidence Pack + H5-hardening ĐÃ LÀM + TEST (155 checks, 0 fail; KMS chạy LIVE qua Vault).**
> Bảng dưới có cột **Baseline → Nay**. Điểm chấm CÔNG TÂM (0–100, theo `casan_harness_assessment.md`):
> **H4 = 80 · H5 = 76 · H6 = 79 · trung bình 7 harness ~80/100 · harness thấp nhất H5=76 → CASAN Level 4 (chạm ngưỡng)**.
> **H4 = 80 · H5 = 76→80 ⬆ · H6 = 79 · trung bình 7 harness ~80.7/100 · harness thấp nhất giờ H6=79 (trước H5=76) → CASAN Level 4 (vững ngưỡng)**.
> Nguồn: `00_SUBMISSION_PACKAGE/evidence/scoring-run-report.md`. Chi tiết implemented-vs-planned: `CASAN_HARDENING_STATUS.md`.
| Chiều | Baseline | **Nay** | Đã đóng (đã test) | Còn hở |
|---|:--:|:--:|---|---|
| Phủ phát hiện (detection) | 3 | **4** | homoglyph · zero-width · fullwidth · base64/hex · tool-output scan | đa ngôn ngữ VI/JA · split-injection (Track B) |
| Fail-safe | 4 | **4.5** | `CASAN_SECURITY_STRICT` fail-closed — hết "semantic SKIP âm thầm" | — |
| Toàn vẹn/chống giả mạo (H5) | 3 | **4** | telemetry vào manifest **ký RSA** (sửa 1 token → MISMATCH) | HSM/KMS · WORM ngoài |
| Toàn vẹn/chống giả mạo (H5) | 3 | **4** | telemetry vào manifest **ký RSA** (sửa 1 token → MISMATCH) | — |
| Kiểm soát chi phí (H6) | 3 | **4** | trần tuyệt đối/call + ngân sách tích luỹ + cold-start (ngoài median×mult) | hard-cap per-provider live |
| Quan sát (observability) | 3 | **3.5** | telemetry toàn vẹn (ký) | alerting realtime · dashboard hosted |
| Đa domain/i18n | 2 | **2.5** | benign corpus VI/JA/EN đo được (FP 0%) | detection vẫn chủ yếu EN (Track B) |
| Quản lý khóa | 2 | **2.5** | khoá ký **off-repo** + ký telemetry | HSM/rotation/tách quyền ký (Track B) |
| Phủ kiểm thử | 4 | **4.5** | **140 test** (35+44+25+29+7) đối kháng | thêm ca đa ngôn ngữ khi làm Track B |
| Quản lý khóa | 2 | **4** | **KMS live** (Vault Transit) — ký qua KMS, **rotate**, khoá **non-exportable** (đã chạy thật) | KMS chưa mặc định (fallback local) · HSM · IdP token ngắn hạn |
| Phủ kiểm thử | 4 | **4.5** | **155 test** (35+44+25+29+7+8+7) đối kháng, 0 fail | thêm ca đa ngôn ngữ khi làm Track B |
**Điểm trung bình (H4/H5/H6 mở rộng) ~3.0 → ~3.7/5; chấm công tâm per-harness ~3.8–4.0/5 (76–80/100).**
**Điểm trung bình (H4/H5/H6 mở rộng) ~3.0 → ~3.8/5; chấm công tâm per-harness H4=80 · H5=80 · H6=79 (~4.0/5). Harness thấp nhất giờ là H6.**
> ⚠️ **Phạm vi rộng hơn — các chiều NGOÀI 3 harness lõi (Track C):**
@@ -45,12 +45,12 @@
| Supply-chain (dependency sinh ra) | 1 | **3.5** | `supply-chain-gate.sh` (V18) — typosquat/postinstall/denylist + dep-diff | CVE/OSV scanner live chưa nối |
| Data-governance / anti-exfil | 2 | **4** | `data-exfil-guard.sh` (V19) — secret→cloud BLOCK, PII→audit mask | — |
| Evidence Pack (Plan-09) | — | **4** | `casan pack/verify-pack` — manifest ký, tamper-evident, certified-gate | KMS sign · hosted store |
| Policy governance / approval | 2 | **2.5** | có outcome `REQUIRE_APPROVAL` + approver định danh | versioning + reviewer workflow (C4) [planned] |
| Policy governance / approval | 2 | **4** | **approval-identity** (C4) — reviewer KÝ request + role authz, hết env-var; SoD giữ (đã test 8/8) | live IdP (OIDC/JWT) · policy versioning/diff [planned] |
| Runtime sandbox | 1 | **2.5** | `sandbox-run.sh` scaffold (V22) — chặn ssh/egress/forkbomb/write-outside + ulimit | **cô lập kernel thật** (container/nsjail) [planned] |
| External append-only audit | 1 | **1.5** | ký chain local | WORM ngoài runtime (C5) [planned] |
| External append-only audit | 1 | **3.5** | **WORM ledger** (C5) — ship head hash-link ngoài, bắt rollback (`AUDIT_GAP_DETECTED`) + tamper | WORM store thật (S3 Object Lock) · trusted timestamp [planned] |
| Incident response | 1 | **1** | — | severity/owner/kill-switch (C7) [planned] |
**→ C-MVP (C1+C2+C3 + Evidence Pack) ~3.8/5 [đã làm]; sandbox mới scaffold; C-Governance/Ops (C4/C5/C7) vẫn ~1.5–2.5 [planned].** Track A + C-MVP đưa trio H4/H5/H6 lên **~3.8–4.0/5 (76–80/100)**; **Track B + C-Governance/Ops + sandbox isolation thật** mới đủ production toàn diện.
**→ C-MVP (C1+C2+C3 + Evidence Pack) ~3.8/5 + H5-hardening (C4 approval-identity, KMS live, C5 WORM) [đã làm + test thật]. Còn: sandbox isolation thật, C7 incident, live IdP/WORM-store [planned].** Trio H4/H5/H6 nay **~4.0/5 (H4=80·H5=80·H6=79)**; harness thấp nhất nhích **76 (H5) → 79 (H6)**. Production toàn diện vẫn cần các mục [planned] ở trên.
## 3. Bảng đường lọt (tóm tắt từ threat-model)