thanhnvandClaude Fable 5
da66a36f97
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
50 lines
3.5 KiB
HTML
50 lines
3.5 KiB
HTML
<!doctype html>
|
|
<html lang="en">
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<title>CASAN Central AgentOps Dashboard</title>
|
|
<style>
|
|
body { font-family: Arial, sans-serif; margin: 32px; color: #172033; }
|
|
.grid { display: grid; grid-template-columns: repeat(5, 1fr); gap: 12px; }
|
|
.card { border: 1px solid #d8dee9; border-radius: 8px; padding: 16px; }
|
|
.value { font-size: 28px; font-weight: 700; color: #0f766e; }
|
|
table { border-collapse: collapse; width: 100%; margin-top: 20px; }
|
|
td, th { border: 1px solid #d8dee9; padding: 8px; text-align: left; }
|
|
th { background: #f1f5f9; }
|
|
</style>
|
|
</head>
|
|
<body>
|
|
<h1>CASAN Level 5 Central AgentOps Dashboard</h1>
|
|
<p>Generated: 2026-07-04T16:11:59Z</p>
|
|
<div class="grid">
|
|
<div class="card"><div>Total Runs</div><div class="value">14</div></div>
|
|
<div class="card"><div>Average Latency</div><div class="value">335.93ms</div></div>
|
|
<div class="card"><div>Estimated Cost</div><div class="value">$0.083622</div></div>
|
|
<div class="card"><div>Failures</div><div class="value">4</div></div>
|
|
<div class="card"><div>Fallback Routes</div><div class="value">2</div></div>
|
|
</div>
|
|
<h2>Provider Usage Telemetry</h2>
|
|
<div class="grid">
|
|
<div class="card"><div>Provider Runs</div><div class="value">4</div></div>
|
|
<div class="card"><div>Provider Tokens</div><div class="value">5937</div></div>
|
|
<div class="card"><div>Provider Cost</div><div class="value">$0.16668</div></div>
|
|
<div class="card"><div>Registered Projects</div><div class="value">3</div></div>
|
|
<div class="card"><div>Tool Denials</div><div class="value">6</div></div>
|
|
</div>
|
|
<h2>Governance Signals</h2>
|
|
<table>
|
|
<tr><th>Signal</th><th>Value</th></tr>
|
|
<tr><td>Tool registry denials</td><td>6</td></tr>
|
|
<tr><td>Fallback records</td><td>2</td></tr>
|
|
<tr><td>Tool registry records</td><td>11</td></tr>
|
|
<tr><td>Provider telemetry records</td><td>4</td></tr>
|
|
<tr><td>Registered harness projects</td><td>3</td></tr>
|
|
</table>
|
|
<h2>Recent AgentOps Metrics</h2>
|
|
<table>
|
|
<tr><th>Trace</th><th>Agent</th><th>Step</th><th>Status</th><th>Latency</th><th>Tokens</th><th>Cost</th></tr>
|
|
<tr><td>f4cea8da-c6e6-4aa9-887f-4fcfd5a8770f</td><td>wrapper.demo</td><td>wrapper-step</td><td>success</td><td>60</td><td>14</td><td>2.8e-05</td></tr><tr><td>5092d8b7-62b1-470d-b474-5f7340c9701c</td><td>wrapper.demo</td><td>wrapper-step</td><td>success</td><td>212</td><td>14</td><td>2.8e-05</td></tr><tr><td>48ec5c2e-8a95-406e-b5c7-e1043db5d7ba</td><td>unknown-agent</td><td>write_code</td><td>failed</td><td>226</td><td>3</td><td>6e-06</td></tr><tr><td>00b1ef47-479b-4fdb-b1f5-6a627e3efcc9</td><td>adv</td><td>step-1-srs</td><td>success</td><td>208</td><td>26</td><td>5.2e-05</td></tr><tr><td>f4da12ad-d5d5-45d9-9201-fce23c45d419</td><td>adv</td><td>step-1-srs</td><td>success</td><td>222</td><td>14</td><td>2.8e-05</td></tr><tr><td>796d2566-21c7-4775-82ac-f247700974cd</td><td>unknown-agent</td><td>test_timeout</td><td>failed</td><td>2245</td><td>1</td><td>2e-06</td></tr><tr><td>3d906f4a-8577-4451-9619-01ace6fdba77</td><td>unknown-agent</td><td>t4-telemetry-test</td><td>success</td><td>246</td><td>210</td><td>0.0</td></tr><tr><td>a0e25f2c-ad80-43b9-9e7f-cd8beec50366</td><td>unknown-agent</td><td>fetch_step</td><td>success</td><td>251</td><td>12</td><td>2.4e-05</td></tr><tr><td>0b95c529-dea4-440f-a7d2-f0d696dcfbd8</td><td>unknown-agent</td><td>fetch_step</td><td>success</td><td>223</td><td>12</td><td>2.4e-05</td></tr><tr><td>692c1c0b-a905-4d81-8f4d-ea1a0509ce47</td><td>h6.e2e</td><td>e2e_fail_step</td><td>failed</td><td>255</td><td>1</td><td>2e-06</td></tr>
|
|
</table>
|
|
</body>
|
|
</html>
|