feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Fable 5
parent
2af67ef6a5
commit
da66a36f97
@@ -11,14 +11,22 @@ set -uo pipefail
|
||||
# consecutive recent failures. If ≥ CIRCUIT_BREAKER_THRESHOLD consecutive
|
||||
# model calls failed, prints CIRCUIT_OPEN and exits non-zero so the caller
|
||||
# can stop invoking the model (prevents cascading failures / cost runaway).
|
||||
# Also runs a SLIDING-WINDOW breaker (V15): a failure RATE ≥
|
||||
# CIRCUIT_WINDOW_FAIL_PCT over the last CIRCUIT_WINDOW records trips
|
||||
# CIRCUIT_OPEN_WINDOW — interleaving successes between failures no longer
|
||||
# evades the breaker.
|
||||
#
|
||||
# Usage: circuit-breaker-check.sh [--no-bypass-only | --breaker-only]
|
||||
# Env: CASAN_PROVIDER_LOG (log override), CIRCUIT_BREAKER_THRESHOLD,
|
||||
# CIRCUIT_WINDOW (default 10), CIRCUIT_WINDOW_FAIL_PCT (default 50)
|
||||
# Exit: 0 all OK, 1 bypass found or circuit open.
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
PROVIDER_LOG="$ROOT/.specify/logs/level5/provider-usage.jsonl"
|
||||
PROVIDER_LOG="${CASAN_PROVIDER_LOG:-$ROOT/.specify/logs/level5/provider-usage.jsonl}"
|
||||
CIRCUIT_BREAKER_THRESHOLD="${CIRCUIT_BREAKER_THRESHOLD:-5}"
|
||||
CIRCUIT_WINDOW="${CIRCUIT_WINDOW:-10}"
|
||||
CIRCUIT_WINDOW_FAIL_PCT="${CIRCUIT_WINDOW_FAIL_PCT:-50}"
|
||||
MODE="${1:-both}"
|
||||
FAIL=0
|
||||
|
||||
@@ -110,6 +118,43 @@ PY
|
||||
else
|
||||
ok "Circuit breaker closed: consecutive_failures=$consecutive_fails (threshold=$CIRCUIT_BREAKER_THRESHOLD)"
|
||||
fi
|
||||
|
||||
# Sliding-window failure RATE (V15): interleaved successes reset the
|
||||
# consecutive counter but do not hide a failing provider from the rate.
|
||||
window_stats="$(python - "$PROVIDER_LOG" "$CIRCUIT_WINDOW" << 'PY'
|
||||
import json, sys
|
||||
|
||||
log_file, window = sys.argv[1], int(sys.argv[2])
|
||||
try:
|
||||
lines = [l for l in open(log_file) if l.strip()]
|
||||
recent = lines[-window:]
|
||||
fails = 0
|
||||
total = 0
|
||||
for line in recent:
|
||||
try:
|
||||
r = json.loads(line)
|
||||
except json.JSONDecodeError:
|
||||
continue
|
||||
total += 1
|
||||
if r.get("status") in ("error", "fail", "failed"):
|
||||
fails += 1
|
||||
print(f"{fails} {total}")
|
||||
except Exception:
|
||||
print("0 0")
|
||||
PY
|
||||
)"
|
||||
window_fails="${window_stats%% *}"
|
||||
window_total="${window_stats##* }"
|
||||
if [[ "$window_total" -ge "$CIRCUIT_WINDOW" ]]; then
|
||||
window_pct=$(( window_fails * 100 / window_total ))
|
||||
if [[ "$window_pct" -ge "$CIRCUIT_WINDOW_FAIL_PCT" ]]; then
|
||||
fail "CIRCUIT_OPEN_WINDOW: failure rate ${window_pct}% over last $window_total calls (threshold=${CIRCUIT_WINDOW_FAIL_PCT}%) — interleaved successes do not close the circuit"
|
||||
else
|
||||
ok "Window breaker closed: failure rate ${window_pct}% over last $window_total calls (threshold=${CIRCUIT_WINDOW_FAIL_PCT}%)"
|
||||
fi
|
||||
else
|
||||
ok "Window breaker closed: only $window_total records (< window=$CIRCUIT_WINDOW), rate not evaluated"
|
||||
fi
|
||||
fi
|
||||
fi
|
||||
|
||||
|
||||
Reference in New Issue
Block a user