feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
2af67ef6a5
commit
da66a36f97
@@ -0,0 +1,70 @@
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
|
||||
# CASAN H6 — provider usage telemetry API fetch (D2).
|
||||
# Pulls usage records from a provider usage HTTP API (production: the
|
||||
# OpenAI/Anthropic usage endpoints) and imports them into provider-usage.jsonl
|
||||
# through the same schema gate as import-provider-telemetry.sh.
|
||||
# Fail-loud by design: unreachable API or invalid schema => non-zero exit and
|
||||
# NOTHING is imported (all-or-nothing, no partial/dirty telemetry).
|
||||
#
|
||||
# Usage: provider-usage-fetch.sh <api-url> [out-jsonl]
|
||||
#
|
||||
# Greppable outputs:
|
||||
# PROVIDER_TELEMETRY_FETCHED | PROVIDER_USAGE_INVALID | PROVIDER_API_UNREACHABLE
|
||||
|
||||
API_URL="${1:-}"
|
||||
if [[ -z "$API_URL" ]]; then
|
||||
echo "Usage: provider-usage-fetch.sh <api-url> [out-jsonl]" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
OUT="${2:-$PROJECT_ROOT/.specify/logs/level5/provider-usage.jsonl}"
|
||||
mkdir -p "$(dirname "$OUT")"
|
||||
|
||||
BODY="$(mktemp)"
|
||||
trap 'rm -f "$BODY"' EXIT
|
||||
|
||||
if ! curl -sS -m 10 --retry 2 --retry-delay 1 -f "$API_URL" -o "$BODY" 2>/dev/null; then
|
||||
echo "PROVIDER_API_UNREACHABLE url=$API_URL (telemetry NOT imported)" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
python - "$BODY" "$OUT" "$API_URL" <<'PY'
|
||||
import json
|
||||
import sys
|
||||
from datetime import datetime, timezone
|
||||
|
||||
src, out, url = sys.argv[1], sys.argv[2], sys.argv[3]
|
||||
try:
|
||||
data = json.load(open(src, encoding="utf-8"))
|
||||
except (json.JSONDecodeError, UnicodeDecodeError):
|
||||
raise SystemExit("PROVIDER_USAGE_INVALID response is not JSON")
|
||||
records = data if isinstance(data, list) else [data]
|
||||
required = ["provider", "model", "run_id", "step", "input_tokens", "output_tokens",
|
||||
"total_tokens", "cost_usd", "latency_ms", "status"]
|
||||
for idx, rec in enumerate(records):
|
||||
if not isinstance(rec, dict):
|
||||
raise SystemExit(f"PROVIDER_USAGE_INVALID record[{idx}] is not an object")
|
||||
missing = [key for key in required if key not in rec]
|
||||
if missing:
|
||||
raise SystemExit(f"PROVIDER_USAGE_INVALID record[{idx}] missing={missing}")
|
||||
now = datetime.now(timezone.utc).strftime("%Y-%m-%dT%H:%M:%SZ")
|
||||
with open(out, "a", encoding="utf-8") as fh:
|
||||
for rec in records:
|
||||
fh.write(json.dumps({
|
||||
"timestamp": now,
|
||||
"harness": "L5-provider-telemetry",
|
||||
"telemetry_source": "provider_api",
|
||||
"api_endpoint": url,
|
||||
**rec,
|
||||
}) + "\n")
|
||||
print(f"PROVIDER_TELEMETRY_FETCHED count={len(records)} url={url} output={out}")
|
||||
PY
|
||||
RC=$?
|
||||
if [[ "$RC" -ne 0 ]]; then
|
||||
echo "PROVIDER_USAGE_INVALID import rejected (nothing written)" >&2
|
||||
exit 1
|
||||
fi
|
||||
Reference in New Issue
Block a user