feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
co-authored by
Claude Fable 5
Claude Fable 5
parent
2af67ef6a5
commit
da66a36f97
@@ -15,35 +15,35 @@ th { background: #f1f5f9; }
|
||||
</head>
|
||||
<body>
|
||||
<h1>CASAN Level 5 Central AgentOps Dashboard</h1>
|
||||
<p>Generated: 2026-07-03T15:03:39Z</p>
|
||||
<p>Generated: 2026-07-04T16:11:59Z</p>
|
||||
<div class="grid">
|
||||
<div class="card"><div>Total Runs</div><div class="value">6</div></div>
|
||||
<div class="card"><div>Average Latency</div><div class="value">137.17ms</div></div>
|
||||
<div class="card"><div>Estimated Cost</div><div class="value">$0.083484</div></div>
|
||||
<div class="card"><div>Failures</div><div class="value">1</div></div>
|
||||
<div class="card"><div>Fallback Routes</div><div class="value">1</div></div>
|
||||
<div class="card"><div>Total Runs</div><div class="value">14</div></div>
|
||||
<div class="card"><div>Average Latency</div><div class="value">335.93ms</div></div>
|
||||
<div class="card"><div>Estimated Cost</div><div class="value">$0.083622</div></div>
|
||||
<div class="card"><div>Failures</div><div class="value">4</div></div>
|
||||
<div class="card"><div>Fallback Routes</div><div class="value">2</div></div>
|
||||
</div>
|
||||
<h2>Provider Usage Telemetry</h2>
|
||||
<div class="grid">
|
||||
<div class="card"><div>Provider Runs</div><div class="value">2</div></div>
|
||||
<div class="card"><div>Provider Tokens</div><div class="value">5556</div></div>
|
||||
<div class="card"><div>Provider Runs</div><div class="value">4</div></div>
|
||||
<div class="card"><div>Provider Tokens</div><div class="value">5937</div></div>
|
||||
<div class="card"><div>Provider Cost</div><div class="value">$0.16668</div></div>
|
||||
<div class="card"><div>Registered Projects</div><div class="value">3</div></div>
|
||||
<div class="card"><div>Tool Denials</div><div class="value">2</div></div>
|
||||
<div class="card"><div>Tool Denials</div><div class="value">6</div></div>
|
||||
</div>
|
||||
<h2>Governance Signals</h2>
|
||||
<table>
|
||||
<tr><th>Signal</th><th>Value</th></tr>
|
||||
<tr><td>Tool registry denials</td><td>2</td></tr>
|
||||
<tr><td>Fallback records</td><td>1</td></tr>
|
||||
<tr><td>Tool registry records</td><td>3</td></tr>
|
||||
<tr><td>Provider telemetry records</td><td>2</td></tr>
|
||||
<tr><td>Tool registry denials</td><td>6</td></tr>
|
||||
<tr><td>Fallback records</td><td>2</td></tr>
|
||||
<tr><td>Tool registry records</td><td>11</td></tr>
|
||||
<tr><td>Provider telemetry records</td><td>4</td></tr>
|
||||
<tr><td>Registered harness projects</td><td>3</td></tr>
|
||||
</table>
|
||||
<h2>Recent AgentOps Metrics</h2>
|
||||
<table>
|
||||
<tr><th>Trace</th><th>Agent</th><th>Step</th><th>Status</th><th>Latency</th><th>Tokens</th><th>Cost</th></tr>
|
||||
<tr><td>cb926abb-0d31-45d0-b03d-1d2afdf7c622</td><td>demo.agent</td><td>demo-step</td><td>success</td><td>59</td><td>12</td><td>2.4e-05</td></tr><tr><td>95b088e2-f7ce-403d-bcca-1cf06186360d</td><td>demo.agent</td><td>step-1-srs</td><td>success</td><td>215</td><td>26</td><td>5.2e-05</td></tr><tr><td>082ad921-01f5-42ac-ac4c-52c232dce0e4</td><td>demo.agent</td><td>speckit.implement</td><td>success</td><td>65</td><td>2778</td><td>0.08334</td></tr><tr><td>4e0fdde7-238d-4c20-9edc-fc4bae3c3213</td><td>demo.agent</td><td>failing-step</td><td>failed</td><td>196</td><td>6</td><td>1.2e-05</td></tr><tr><td>02b57916-9f25-4bcb-b1c2-f4bf58adfc52</td><td>wrapper.demo</td><td>wrapper-step</td><td>success</td><td>70</td><td>14</td><td>2.8e-05</td></tr><tr><td>a607903d-5d84-40fc-80ea-139c96c133ae</td><td>wrapper.demo</td><td>wrapper-step</td><td>success</td><td>218</td><td>14</td><td>2.8e-05</td></tr>
|
||||
<tr><td>f4cea8da-c6e6-4aa9-887f-4fcfd5a8770f</td><td>wrapper.demo</td><td>wrapper-step</td><td>success</td><td>60</td><td>14</td><td>2.8e-05</td></tr><tr><td>5092d8b7-62b1-470d-b474-5f7340c9701c</td><td>wrapper.demo</td><td>wrapper-step</td><td>success</td><td>212</td><td>14</td><td>2.8e-05</td></tr><tr><td>48ec5c2e-8a95-406e-b5c7-e1043db5d7ba</td><td>unknown-agent</td><td>write_code</td><td>failed</td><td>226</td><td>3</td><td>6e-06</td></tr><tr><td>00b1ef47-479b-4fdb-b1f5-6a627e3efcc9</td><td>adv</td><td>step-1-srs</td><td>success</td><td>208</td><td>26</td><td>5.2e-05</td></tr><tr><td>f4da12ad-d5d5-45d9-9201-fce23c45d419</td><td>adv</td><td>step-1-srs</td><td>success</td><td>222</td><td>14</td><td>2.8e-05</td></tr><tr><td>796d2566-21c7-4775-82ac-f247700974cd</td><td>unknown-agent</td><td>test_timeout</td><td>failed</td><td>2245</td><td>1</td><td>2e-06</td></tr><tr><td>3d906f4a-8577-4451-9619-01ace6fdba77</td><td>unknown-agent</td><td>t4-telemetry-test</td><td>success</td><td>246</td><td>210</td><td>0.0</td></tr><tr><td>a0e25f2c-ad80-43b9-9e7f-cd8beec50366</td><td>unknown-agent</td><td>fetch_step</td><td>success</td><td>251</td><td>12</td><td>2.4e-05</td></tr><tr><td>0b95c529-dea4-440f-a7d2-f0d696dcfbd8</td><td>unknown-agent</td><td>fetch_step</td><td>success</td><td>223</td><td>12</td><td>2.4e-05</td></tr><tr><td>692c1c0b-a905-4d81-8f4d-ea1a0509ce47</td><td>h6.e2e</td><td>e2e_fail_step</td><td>failed</td><td>255</td><td>1</td><td>2e-06</td></tr>
|
||||
</table>
|
||||
</body>
|
||||
</html>
|
||||
|
||||
Reference in New Issue
Block a user