Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):
- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).
New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.
Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.
Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Hướng A — scorecard.sh (video demo):
- h5_1 approval workflow: hardcode 0 → governance-check deploy live (approval_required)
- h6_2 hallucination rate: hardcode 0 → hallucination-scan phân biệt dirty>clean live
- "N/5 mục" chuyển từ text cứng sang đếm động
- H4/H5/H6 → 100/100 (5/5 gate live), Average 57.9 → 90.0
Hướng B — run-casan-pipeline.mjs:
- fallback: stub 'exit 9' → 'cat /nonexistent' (real failure, nhất quán adversarial T3)
- drift: giữ so fallback-output vs golden (clean run=1.0); năng lực phát hiện
drift thật chứng minh ở adversarial suite
- Full 12-step run verify: H1 CONTEXT_VALID=24, H2 tool-audit records=25 signed,
H5 audit-chain records=22 signed, H6 provider_telemetry per-step thật, H7 rollback real
phase3-real-run-scoring.md: giải thích vì sao scorecard cũ cho H5=60/H6=80
(hardcode), phân biệt scorecard-90 vs re-score-84 (2 mục đích khác nhau).
Verify: adversarial 44/0, security-gate 11/0/0, pipeline 12 steps OK.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
T1 (H7): casan-step.mjs now calls rollback-manager.sh checkpoint before
overwriting plan.md at attempt-2, writes tx-id to plan.checkpoint.txid
sidecar, and executes rollback on REJECTED verdict. rollback-transactions.jsonl
records a real cp restore command. Adversarial test: checkpoint exists,
real cp command recorded, plan hash matches pre-overwrite content.
T4 (H6): casan-harness.sh exports CASAN_STEP_NAME=$ACTION_NAME before
agent-metrics.sh so nested model calls (model-call.py) and the provider-
cost-lookup.py query share the same step label. metrics.jsonl now writes
cost_source=provider_telemetry instead of word_count_estimate when a real
Ollama call is made within the same step. Adversarial test: verified with
CASAN_STEP_NAME=t4-telemetry-test end-to-end.
adversarial-harness-tests.sh: 40 → 44 PASS / 0 FAIL (+3 T1, +1 T4)
security-gate.sh: PASS=10 FAIL=0 SKIP=0 (verified, local ornith:9b)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- frontend/package.json: add @testing-library/dom ^10.0.0 (missing peer
dep of @testing-library/react that caused test failure on macOS)
- docs: update security gate result to PASS=10 FAIL=0 SKIP=0 (macOS
with local ornith:9b) vs PASS=7 SKIP=1 on Windows (no Ollama)
- audit logs: real evidence from running all 10 gates (adversarial suite,
model router, red-team 30-sample, judge gate, frontend Vitest)
- remove 10 timestamp-named trace stubs (not referenced by
pipeline-context.yaml; UUID stubs in place and validated)
Verified: security-gate.sh PASS=10 FAIL=0 SKIP=0
adversarial-harness-tests.sh PASS=40 FAIL=0
npm test -w frontend: 16 PASS / 0 FAIL
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
thanhnv