thanhnvandClaude Fable 5
da66a36f97
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
7 lines
2.0 KiB
Plaintext
7 lines
2.0 KiB
Plaintext
{"timestamp":"2026-07-04T16:03:36Z","trace_id":"432d6880-9240-461b-9a81-77cdcc30da75","severity":"WARN","resource":{"service.name":"demo.agent","service.version":"1.0.0"},"body":{"message":"Alert triggered: hallucination-suspected","alert.type":"hallucination-suspected","step.name":"step-1-srs"},"attributes":{"latency_ms":226,"status":"success"}}
|
|
{"timestamp":"2026-07-04T16:03:37Z","trace_id":"fc76dac7-c2aa-4d24-b2fc-3a9405172db2","severity":"WARN","resource":{"service.name":"demo.agent","service.version":"1.0.0"},"body":{"message":"Alert triggered: execution-failed","alert.type":"execution-failed","step.name":"failing-step"},"attributes":{"latency_ms":212,"status":"failed"}}
|
|
{"timestamp":"2026-07-04T16:04:13Z","trace_id":"48ec5c2e-8a95-406e-b5c7-e1043db5d7ba","severity":"WARN","resource":{"service.name":"unknown-agent","service.version":"1.0.0"},"body":{"message":"Alert triggered: execution-failed","alert.type":"execution-failed","step.name":"write_code"},"attributes":{"latency_ms":226,"status":"failed"}}
|
|
{"timestamp":"2026-07-04T16:04:15Z","trace_id":"00b1ef47-479b-4fdb-b1f5-6a627e3efcc9","severity":"WARN","resource":{"service.name":"adv","service.version":"1.0.0"},"body":{"message":"Alert triggered: hallucination-suspected","alert.type":"hallucination-suspected","step.name":"step-1-srs"},"attributes":{"latency_ms":208,"status":"success"}}
|
|
{"timestamp":"2026-07-04T16:04:22Z","trace_id":"796d2566-21c7-4775-82ac-f247700974cd","severity":"WARN","resource":{"service.name":"unknown-agent","service.version":"1.0.0"},"body":{"message":"Alert triggered: execution-failed","alert.type":"execution-failed","step.name":"test_timeout"},"attributes":{"latency_ms":2245,"status":"failed"}}
|
|
{"timestamp":"2026-07-04T16:11:57Z","trace_id":"692c1c0b-a905-4d81-8f4d-ea1a0509ce47","severity":"WARN","resource":{"service.name":"h6.e2e","service.version":"1.0.0"},"body":{"message":"Alert triggered: execution-failed","alert.type":"execution-failed","step.name":"e2e_fail_step"},"attributes":{"latency_ms":255,"status":"failed"}}
|