Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):
- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).
New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.
Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.
Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add REAL=1 to the video-steps demo so attack vectors flow through the real
production entry-point instead of calling sub-scripts directly.
- run-all.sh: REAL=1 feeds each H4 vector (A1/A2/A4/A6/A7 + cross-layer step
1) as the INPUT of an agent step run through casan-harness.sh, so the
BLOCK/PASS verdict is produced by the wrapper itself (H4-in -> H5 -> H6 ->
exec -> H4-out) exactly as when the real pipeline meets malicious input.
After the battery it runs a real pipeline slice (STEP1 okr.srs via
casan-harness.sh -- node casan-step.mjs) and shows audit.jsonl growing by a
real record. An inline inventory documents which vectors intentionally keep
calling a single control directly (artifact-scan, audit tamper/re-forge,
detectors on synthetic telemetry) and why. Default mode (no REAL) unchanged.
- map-live.sh: show the PIPELINE (STEP1) row only under REAL=1, driven by a
mode sidecar file written by run-all.sh.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Hướng A — scorecard.sh (video demo):
- h5_1 approval workflow: hardcode 0 → governance-check deploy live (approval_required)
- h6_2 hallucination rate: hardcode 0 → hallucination-scan phân biệt dirty>clean live
- "N/5 mục" chuyển từ text cứng sang đếm động
- H4/H5/H6 → 100/100 (5/5 gate live), Average 57.9 → 90.0
Hướng B — run-casan-pipeline.mjs:
- fallback: stub 'exit 9' → 'cat /nonexistent' (real failure, nhất quán adversarial T3)
- drift: giữ so fallback-output vs golden (clean run=1.0); năng lực phát hiện
drift thật chứng minh ở adversarial suite
- Full 12-step run verify: H1 CONTEXT_VALID=24, H2 tool-audit records=25 signed,
H5 audit-chain records=22 signed, H6 provider_telemetry per-step thật, H7 rollback real
phase3-real-run-scoring.md: giải thích vì sao scorecard cũ cho H5=60/H6=80
(hardcode), phân biệt scorecard-90 vs re-score-84 (2 mục đích khác nhau).
Verify: adversarial 44/0, security-gate 11/0/0, pipeline 12 steps OK.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
