feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-05 01:14:46 +09:00
parent 2af67ef6a5
commit da66a36f97
55 changed files with 1306 additions and 455 deletions
@@ -0,0 +1,78 @@
+#!/usr/bin/env python3
+"""CASAN H6 — hosted AgentOps dashboard server (D3).
+
+Serves the generated dashboard over HTTP with a stale-aware /healthz probe so
+an external monitor (uptime check / load-balancer) can page when telemetry
+stops flowing — not just when the process dies.
+
+Routes:
+  GET /            -> dashboard HTML (also /dashboard)
+  GET /healthz     -> 200 {"status":"ok",...} while metrics are fresh,
+                      503 {"status":"stale",...} when metrics are older than
+                      CASAN_DASHBOARD_STALE_S (default 3600s) or missing.
+
+Usage: dashboard-server.py <port>
+"""
+import json
+import os
+import pathlib
+import sys
+import time
+from http.server import BaseHTTPRequestHandler, HTTPServer
+
+ROOT = pathlib.Path(__file__).resolve().parents[3]
+PORT = int(sys.argv[1]) if len(sys.argv) > 1 else 8787
+DASH = pathlib.Path(os.environ.get(
+    "CASAN_DASHBOARD_HTML", ROOT / "docs" / "output" / "casan" / "central-agentops-dashboard.html"))
+METRICS = pathlib.Path(os.environ.get(
+    "CASAN_DASHBOARD_METRICS", ROOT / ".specify" / "logs" / "cost" / "metrics.jsonl"))
+ALERTS = pathlib.Path(os.environ.get(
+    "CASAN_DASHBOARD_ALERTS", ROOT / ".specify" / "agentops" / "alerts.log"))
+STALE_S = int(os.environ.get("CASAN_DASHBOARD_STALE_S", "3600"))
+
+
+def count_lines(path: pathlib.Path) -> int:
+    if not path.exists():
+        return 0
+    return sum(1 for line in path.read_text(encoding="utf-8").splitlines() if line.strip())
+
+
+class Handler(BaseHTTPRequestHandler):
+    def _send(self, code: int, ctype: str, body: str) -> None:
+        data = body.encode("utf-8")
+        self.send_response(code)
+        self.send_header("Content-Type", ctype)
+        self.send_header("Content-Length", str(len(data)))
+        self.end_headers()
+        self.wfile.write(data)
+
+    def do_GET(self):  # noqa: N802 (http.server API)
+        if self.path == "/healthz":
+            if METRICS.exists():
+                age = int(time.time() - METRICS.stat().st_mtime)
+                stale = age > STALE_S
+            else:
+                age = -1
+                stale = True
+            body = json.dumps({
+                "status": "stale" if stale else "ok",
+                "metrics_age_s": age,
+                "stale_after_s": STALE_S,
+                "runs": count_lines(METRICS),
+                "alerts": count_lines(ALERTS),
+            })
+            self._send(503 if stale else 200, "application/json", body)
+        elif self.path in ("/", "/dashboard"):
+            if DASH.exists():
+                self._send(200, "text/html; charset=utf-8", DASH.read_text(encoding="utf-8"))
+            else:
+                self._send(404, "text/plain", "dashboard not generated")
+        else:
+            self._send(404, "text/plain", "not found")
+
+    def log_message(self, *args):  # silence per-request stderr noise
+        pass
+
+
+if __name__ == "__main__":
+    HTTPServer(("127.0.0.1", PORT), Handler).serve_forever()