feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):

- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
  dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
  Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
  usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
  provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
  a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
  successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).

New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.

Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.

Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-05 01:14:46 +09:00
co-authored by Claude Fable 5
parent 2af67ef6a5
commit da66a36f97
55 changed files with 1306 additions and 455 deletions
+13 -7
View File
@@ -73,14 +73,20 @@ removed). Full status: `casan-next-plans/CASAN_HARDENING_STATUS.md`.
cryptographically signs the request + role authorization — env-var approver no
longer enough); KMS key management (Vault Transit sign + rotation + non-exportable,
**validated live**); external WORM audit ledger (rollback + tamper detection).
- **Implemented + tested (H6 AgentOps hardening):** live alert dispatch (webhook +
dedup + dead-letter, fail-loud, end-to-end from a failing step); provider-telemetry
API fetch + local-vs-provider reconciliation (catches token under-reporting);
hosted dashboard with stale-aware `/healthz`; sliding-window circuit breaker (V15).
- **Planned (NOT done — do not claim as production-ready):** multilingual H4,
classifier/split-injection resistance, HSM + KMS-by-default, live IdP (OIDC/JWT),
true WORM store (S3 Object Lock), incident kill-switch, true sandbox isolation.
true WORM store (S3 Object Lock), incident kill-switch, true sandbox isolation,
deployed dashboard host + managed alert channel, real billing-API telemetry.
Test totals: baseline 79 (run-casan4 35 + adversarial 44) preserved, **+76 new**
Test totals: baseline 79 (run-casan4 35 + adversarial 44) preserved, **+96 new**
hardening checks (Track A 25, Track C-MVP 29, Evidence Pack 7, H5-approval 8,
H5-infra KMS+WORM 7) = **155**, 0 fail — last full run 2026-07-04 with KMS live via
Vault (`evidence/scoring-run-report.md`). Fair maturity ~80/100 per harness; H5 rose
76→80 so the **lowest harness is now H6=79** (CASAN Level 4, proven by attack). See
`CASAN_HARDENING_STATUS.md`. Because these live in **separate** suites, the demo
attack battery counts in `video/01_video_recording_guide.md` are unchanged.
H5-infra KMS+WORM 7, H6-agentops 20) = **175**, 0 fail — last full run 2026-07-05
(KMS validated live 2026-07-04 via Vault; `evidence/scoring-run-report.md`). Fair
maturity ~80/100 per harness; H5 rose 76→80 and H6 rose 79→80 so **no harness is
below 80** (CASAN Level 4, proven by attack). See `CASAN_HARDENING_STATUS.md`.
Because these live in **separate** suites, the demo attack battery counts in
`video/01_video_recording_guide.md` are unchanged.