gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs+demo: deep-gap closers — 211/0 re-score, HD1-HD4 scenes, status/README/scoring

Browse Source 
Full authoritative run 2026-07-06: all 12 suites 211 PASS / 0 FAIL (KMS + container
isolation live via Vault dev + Docker; security-gate 11/0).
- run-hardening.sh: new "Vá đường lọt sâu" section (HD1 incident/kill-switch,
  HD2 multilingual VI/JA, HD3 true container isolation, HD4 split+classifier),
  closer updated to 211 checks.
- CASAN_HARDENING_STATUS.md: Phase 6 deep-gap closers table; test inventory
  175→211 (12 suites); C7/multilingual moved out of planned; C6 planned→partial
  (real isolation done); honest claim → H4 83, H2 82, H5/H6 stay 80 (infra-bound).
- scoring-report-02-after-competition.md: current state — 211/0, H4 80→83,
  H2 80→82, avg 80.9→81.6, lowest harness still 80 (H5/H6), 3-milestone table.
- README claim boundary: deep-gap closers listed; totals 175→211; H4/H2 bumps.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-06 09:51:39 +09:00
co-authored by Claude Opus 4.8
parent 5ec1a0cc82
commit 42115e3361
4 changed files with 169 additions and 30 deletions
Download Patch File Download Diff File Expand all files Collapse all files
casan-next-plans/CASAN_HARDENING_STATUS.md
+29 -17
View File
@@ -53,6 +53,15 @@
| D3 | **Hosted dashboard**: dashboard served over HTTP with a stale-aware `/healthz` probe (fresh ⇒ 200 ok; telemetry silent-death ⇒ 503 stale — page-able by any uptime monitor) | [implemented+tested] (local HTTP daemon; production host = nginx/container, same routes) | `dashboard-serve.sh`, `dashboard-server.py` | phase-h6-agentops (③) |
| D4 | **Sliding-window circuit breaker (V15)**: failure **rate** over the last N calls trips `CIRCUIT_OPEN_WINDOW` — interleaving successes no longer evades the consecutive-failure breaker | [implemented+tested] | `circuit-breaker-check.sh` | phase-h6-agentops (④) |
### Phase 6 — Deep-gap closers (Track B + C6/C7, post-competition) — mixed
| ID | Control | Status | Where | Test |
|---|---|---|---|---|
| C7 | **Incident response + kill-switch (V23)**: `incident.sh raise` grades severity (LOW/MED/HIGH/CRIT via `incident-severity.map`), records a routed entry (owner), and for HIGH/CRIT auto-engages the scoped `kill-switch.sh` (project/model/provider/global) + fires an alert; `casan-harness.sh` refuses to run under an engaged switch (opt-in) | [implemented+tested] | `incident.sh`, `kill-switch.sh`, `incident-runbook.md`, `incident-severity.map` | phase-c7-incident (15) |
| B1 | **Multilingual VI/JA injection (V2)**: VI/JA block-patterns (matched on raw UTF-8, anchored on the injection object) catch injections English regex missed, with 0 false positives on the benign VI/JA corpus | [implemented+tested] | `prompt-filter.yaml` (PI-VI-*, PI-JA-*) | phase-h4-multilingual (7) |
| C6 | **TRUE runtime isolation (V22)**: container sandbox (`--network=none --read-only --pids-limit --cap-drop=ALL`, workspace-only mount) — the kernel neutralises host-file reads / egress / out-of-workspace writes; upgrades the static scaffold | [implemented+tested] (live via Docker; skip-aware) | `sandbox-container.sh`, `sandbox-run.sh` (`CASAN_SANDBOX_MODE=container`) | phase-c6-sandbox (6) |
| B2 | **Split + classifier injection (V5,V6)**: `context-assemble-scan.sh` scans the concatenated context so a payload split across benign pieces is caught on assembly; verdict-steering patterns (PI-CLS-*) block content that tries to hijack the evaluator | [implemented+tested] | `context-assemble-scan.sh`, `prompt-filter.yaml` (PI-CLS-*) | phase-h4-split-inject (8) |
## 2. Test inventory (all suites)
| Suite | Checks | Purpose |
@@ -64,8 +73,12 @@
| `phase3-evidence-pack-tests.sh` | 7 | Evidence Pack MVP |
| `phase-h5-approval-tests.sh` | 8 | Approval-identity (C4) |
| `phase-h5-infra-tests.sh` | 7 | KMS (B3, live/skip-aware) + WORM (C5) |
| `phase-h6-agentops-tests.sh` | 20 | **New** — live alerting (D1) + provider-API/reconcile (D2) + hosted dashboard (D3) + window breaker (D4); all against live local HTTP endpoints |
| **Total** | **175** | Baseline 79 preserved; +96 new hardening checks. Last full run 2026-07-05 @ head of `feat/plan07-track-a-hardening`, 0 fail (KMS SKIP this run — validated live 2026-07-04 via Vault dev). |
| `phase-h6-agentops-tests.sh` | 20 | live alerting (D1) + provider-API/reconcile (D2) + hosted dashboard (D3) + window breaker (D4); against live local HTTP endpoints |
| `phase-c7-incident-tests.sh` | 15 | **New** — incident severity + scoped kill-switch (C7) + wrapper enforcement |
| `phase-h4-multilingual-tests.sh` | 7 | **New** — VI/JA injection block + benign VI/JA 0-FP (B1) |
| `phase-c6-sandbox-tests.sh` | 6 | **New** — TRUE container isolation (C6, live via Docker / skip-aware) |
| `phase-h4-split-inject-tests.sh` | 8 | **New** — split-injection assembly scan + classifier-inject (B2) |
| **Total** | **211** | Baseline 79 preserved; +132 new hardening checks. Last full run 2026-07-05 @ head of `feat/plan07-track-a-hardening`, 0 fail (KMS + container isolation validated live via Vault dev + Docker). |
Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so run it
**first** and never concurrently with the other suites.
@@ -74,14 +87,11 @@ Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so ru
| Area | Status | Plan ref |
|---|---|---|
| H4 multilingual detection (VI/JA injection block-patterns) | [planned] | Plan-07 B1 (V2) |
| Classifier-inject / split-injection resistance | [planned] | Plan-07 B2 (V5,V6) |
| Model-digest pinning | [planned] — sliding-window circuit breaker (V15) is now done (Phase 5 D4) | Plan-07 B4 (V16) |
| Live alerting to a managed channel (Slack/PagerDuty + on-call rota) | [partial] — webhook dispatch + dedup + dead-letter done; managed channel & escalation are config away, incident workflow is C7 | Plan-07 C7 / Phase 5 D1 |
| Hosted telemetry dashboard | [partial] — HTTP-served dashboard + stale-aware `/healthz` done locally; deployed host (nginx/container, auth) planned | Phase 5 D3 |
| Provider billing-API telemetry | [partial] — API fetch + schema gate + local-vs-provider reconciliation done against a live local endpoint; real OpenAI/Anthropic usage-API calls (needs keys) planned | Phase 5 D2 |
| **True runtime isolation** (container `--network=none --read-only --pids-limit`, nsjail) | [planned] — C6 is a static+ulimit scaffold only | Plan-07 C6 (V22) |
| Incident severity/kill-switch/runbook | [planned] | Plan-07 C7 (V23) |
| True runtime isolation | [partial] — real container isolation done + validated live via Docker (C6 phase-6); nsjail/rootless + a hardened base image for CI still planned | Plan-07 C6 (V22) |
| KMS key management (rotation, non-exportable) | [partial] — Vault Transit path implemented + validated live; not yet the default (local-key fallback), no HSM/short-lived IdP tokens | Plan-07 B3 |
| Reviewer approval workflow | [partial] — cryptographic **approval-identity** done (signed reviewer + role); live **IdP (OIDC/JWT)** + policy versioning/diff still planned | Plan-07 C4 (V20) |
| External append-only (WORM) audit | [partial] — hash-linked local ledger + rollback/tamper detection done; true WORM store (S3 Object Lock/QLDB) + trusted timestamp planned | Plan-07 C5 (V21) |
@@ -89,16 +99,18 @@ Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so ru
## 4. Honest claim
Track A + Track C-MVP + Evidence Pack + H5 governance-hardening + H6 AgentOps-hardening
raise H4/H5/H6 from "PoC/demo (~3.0/5)" to **early internal-production hardening**, with
executable adversarial tests for every control (175 checks, 0 fail — last full run
2026-07-05; KMS validated live via Vault on 2026-07-04). Fair maturity score
(`00_SUBMISSION_PACKAGE/evidence/scoring-run-report.md`): per-harness **~80/100**,
**H5 76→80** (approval-identity + KMS live + WORM) and **H6 79→80** (live alert dispatch
+ provider-API reconciliation + hosted dashboard + window breaker), so the **lowest
harness is now 80** (H2/H4/H5/H6/H7 level) — CASAN **Level 4**, proven by attack. This is
**not** full production readiness: serious production still needs live IdP (OIDC/JWT), a
true WORM store (S3 Object Lock), KMS-by-default + HSM, true sandbox isolation,
multilingual detection, a deployed dashboard host + managed alert channel/on-call, and
Track A + Track C-MVP + Evidence Pack + H5/H6 hardening + the deep-gap closers
(C7 incident/kill-switch, VI/JA multilingual, true container isolation, split &
classifier injection) raise H4/H5/H6 from "PoC/demo (~3.0/5)" to **early
internal-production hardening**, with executable adversarial tests for every
control (**211 checks, 0 fail** — last full run 2026-07-05; KMS + container
isolation validated live via Vault dev + Docker). Fair maturity score
(`00_SUBMISSION_PACKAGE/evidence/scoring-run-report.md`): **H4 80→83** (multilingual
+ split/classifier closed), **H2 80→82** (real sandbox isolation), C7 incident
dimension closed; **H5 and H6 remain at 80** (their remaining gaps are infra), so the
**lowest harness stays 80** — CASAN **Level 4**, proven by attack. This is **not** full
production readiness: crossing the whole pipeline into "Strong (81+)" still needs the
H5/H6 infra items — live IdP (OIDC/JWT), a true WORM store (S3 Object Lock),
KMS-by-default + HSM, a deployed dashboard host + managed alert channel/on-call, and
real billing-API telemetry — the [partial]/[planned] rows above and in
`CASAN_PLAN_07_PRODUCTION_HARDENING.md`.