thanhnvandClaude Fable 5
da66a36f97
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
15 lines
7.4 KiB
JSON
15 lines
7.4 KiB
JSON
{"timestamp":"2026-07-04T16:03:36Z","trace_id":"da53987e-b91b-49de-a42a-6af44c685638","harness":"H6-agentops","agent":"demo.agent","step":"demo-step","status":"success","exit_code":0,"latency_ms":60,"retry_count":0,"input_tokens":6,"output_tokens":6,"total_tokens":12,"cost_estimate":0.00002400,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac"}
|
|
{"timestamp":"2026-07-04T16:03:36Z","trace_id":"432d6880-9240-461b-9a81-77cdcc30da75","harness":"H6-agentops","agent":"demo.agent","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":226,"retry_count":0,"input_tokens":13,"output_tokens":13,"total_tokens":26,"cost_estimate":0.00005200,"cost_source":"word_count_estimate","hallucination_signals":4,"alerts":["hallucination-suspected"],"input_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57","output_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57"}
|
|
{"timestamp":"2026-07-04T16:03:37Z","trace_id":"fa952bbb-64a1-408a-bdb2-aa1f74c90866","harness":"H6-agentops","agent":"demo.agent","step":"speckit.implement","status":"success","exit_code":0,"latency_ms":57,"retry_count":0,"input_tokens":6,"output_tokens":6,"total_tokens":2778,"cost_estimate":0.08334,"cost_source":"provider_telemetry","hallucination_signals":0,"alerts":[],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac"}
|
|
{"timestamp":"2026-07-04T16:03:37Z","trace_id":"fc76dac7-c2aa-4d24-b2fc-3a9405172db2","harness":"H6-agentops","agent":"demo.agent","step":"failing-step","status":"failed","exit_code":7,"latency_ms":212,"retry_count":0,"input_tokens":6,"output_tokens":0,"total_tokens":6,"cost_estimate":0.00001200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
|
|
{"timestamp":"2026-07-04T16:03:39Z","trace_id":"f4cea8da-c6e6-4aa9-887f-4fcfd5a8770f","harness":"H6-agentops","agent":"wrapper.demo","step":"wrapper-step","status":"success","exit_code":0,"latency_ms":60,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116","output_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116"}
|
|
{"timestamp":"2026-07-04T16:03:42Z","trace_id":"5092d8b7-62b1-470d-b474-5f7340c9701c","harness":"H6-agentops","agent":"wrapper.demo","step":"wrapper-step","status":"success","exit_code":0,"latency_ms":212,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116","output_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116"}
|
|
{"timestamp":"2026-07-04T16:04:13Z","trace_id":"48ec5c2e-8a95-406e-b5c7-e1043db5d7ba","harness":"H6-agentops","agent":"unknown-agent","step":"write_code","status":"failed","exit_code":0,"latency_ms":226,"retry_count":0,"input_tokens":3,"output_tokens":0,"total_tokens":3,"cost_estimate":0.00000600,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"cae0b3e41bdd7bf9fc5ab7f73d4422a65c3766f8097c90d952d07dd371605290","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
|
|
{"timestamp":"2026-07-04T16:04:15Z","trace_id":"00b1ef47-479b-4fdb-b1f5-6a627e3efcc9","harness":"H6-agentops","agent":"adv","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":208,"retry_count":0,"input_tokens":13,"output_tokens":13,"total_tokens":26,"cost_estimate":0.00005200,"cost_source":"word_count_estimate","hallucination_signals":4,"alerts":["hallucination-suspected"],"input_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57","output_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57"}
|
|
{"timestamp":"2026-07-04T16:04:16Z","trace_id":"f4da12ad-d5d5-45d9-9201-fce23c45d419","harness":"H6-agentops","agent":"adv","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":222,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"a97a0aba66ab15562e4d271fbcb2071c92d3662d99f2e2faac7f263ea35cfa0b","output_hash":"a97a0aba66ab15562e4d271fbcb2071c92d3662d99f2e2faac7f263ea35cfa0b"}
|
|
{"timestamp":"2026-07-04T16:04:22Z","trace_id":"796d2566-21c7-4775-82ac-f247700974cd","harness":"H6-agentops","agent":"unknown-agent","step":"test_timeout","status":"failed","exit_code":124,"latency_ms":2245,"retry_count":0,"input_tokens":1,"output_tokens":0,"total_tokens":1,"cost_estimate":0.00000200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"7d3f9b6284c6f36e77b425cac882e8fbbcc97a4727ec20790853076d0f463453","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
|
|
{"timestamp":"2026-07-04T16:04:48Z","trace_id":"3d906f4a-8577-4451-9619-01ace6fdba77","harness":"H6-agentops","agent":"unknown-agent","step":"t4-telemetry-test","status":"success","exit_code":0,"latency_ms":246,"retry_count":0,"input_tokens":2,"output_tokens":2,"total_tokens":210,"cost_estimate":0.0,"cost_source":"provider_telemetry","hallucination_signals":0,"alerts":[],"input_hash":"19f254d48ce072ae3832d0271898d40921875a1a870bf4389e67ef226ffc1cc6","output_hash":"19f254d48ce072ae3832d0271898d40921875a1a870bf4389e67ef226ffc1cc6"}
|
|
{"timestamp":"2026-07-04T16:05:00Z","trace_id":"a0e25f2c-ad80-43b9-9e7f-cd8beec50366","harness":"H6-agentops","agent":"unknown-agent","step":"fetch_step","status":"success","exit_code":0,"latency_ms":251,"retry_count":0,"input_tokens":3,"output_tokens":9,"total_tokens":12,"cost_estimate":0.00002400,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"7df5d60319af2916ff4d99c086c43b3989be3420fffff872e1077a650ca528f4","output_hash":"48433d000381574392125115f788a7cd1c03749df5b3dba658b2abb69756685c"}
|
|
{"timestamp":"2026-07-04T16:05:03Z","trace_id":"0b95c529-dea4-440f-a7d2-f0d696dcfbd8","harness":"H6-agentops","agent":"unknown-agent","step":"fetch_step","status":"success","exit_code":0,"latency_ms":223,"retry_count":0,"input_tokens":3,"output_tokens":9,"total_tokens":12,"cost_estimate":0.00002400,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"7df5d60319af2916ff4d99c086c43b3989be3420fffff872e1077a650ca528f4","output_hash":"48433d000381574392125115f788a7cd1c03749df5b3dba658b2abb69756685c"}
|
|
{"timestamp":"2026-07-04T16:11:57Z","trace_id":"692c1c0b-a905-4d81-8f4d-ea1a0509ce47","harness":"H6-agentops","agent":"h6.e2e","step":"e2e_fail_step","status":"failed","exit_code":3,"latency_ms":255,"retry_count":0,"input_tokens":1,"output_tokens":0,"total_tokens":1,"cost_estimate":0.00000200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"7d3f9b6284c6f36e77b425cac882e8fbbcc97a4727ec20790853076d0f463453","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
|