feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
co-authored by
Claude Fable 5
Claude Fable 5
parent
2af67ef6a5
commit
da66a36f97
@@ -0,0 +1,145 @@
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
|
||||
# CASAN H6 — live alert dispatch (D1).
|
||||
# Pushes AgentOps alerts to a real HTTP webhook (Slack/Teams/PagerDuty-style
|
||||
# endpoint) instead of only appending to a local log file. Undelivered alerts
|
||||
# are queued to a dead-letter file so no alert is silently lost.
|
||||
#
|
||||
# Usage:
|
||||
# alert-dispatch.sh <alert-json-file> dispatch one alert (JSON object)
|
||||
# alert-dispatch.sh --flush-deadletter retry alerts that failed delivery
|
||||
#
|
||||
# Env:
|
||||
# CASAN_ALERT_WEBHOOK webhook URL (required to dispatch)
|
||||
# CASAN_ALERT_STRICT=1 delivery failure => exit 1 (fail-loud); default warn
|
||||
# CASAN_ALERT_DEDUP_WINDOW_S suppress same service/step/type within N s (default 300)
|
||||
# CASAN_AGENTOPS_DIR state dir override (default .specify/agentops)
|
||||
#
|
||||
# Greppable outputs:
|
||||
# ALERT_DISPATCHED | ALERT_DEDUP_SUPPRESSED | ALERT_DELIVERY_FAILED |
|
||||
# ALERT_DEADLETTER_FLUSHED | ALERT_WEBHOOK_UNSET
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
||||
AGENTOPS_DIR="${CASAN_AGENTOPS_DIR:-$PROJECT_ROOT/.specify/agentops}"
|
||||
STATE="$AGENTOPS_DIR/alert-dispatch-state.jsonl"
|
||||
DEADLETTER="$AGENTOPS_DIR/alert-deadletter.jsonl"
|
||||
WEBHOOK="${CASAN_ALERT_WEBHOOK:-}"
|
||||
STRICT="${CASAN_ALERT_STRICT:-0}"
|
||||
DEDUP_S="${CASAN_ALERT_DEDUP_WINDOW_S:-300}"
|
||||
mkdir -p "$AGENTOPS_DIR"
|
||||
|
||||
post_payload() { # <json-payload> — 0 = delivered
|
||||
curl -sS -m 5 --retry 2 --retry-delay 1 \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d "$1" "$WEBHOOK" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
if [[ "${1:-}" == "--flush-deadletter" ]]; then
|
||||
if [[ -z "$WEBHOOK" ]]; then
|
||||
echo "ALERT_WEBHOOK_UNSET cannot flush dead-letter queue" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -s "$DEADLETTER" ]]; then
|
||||
echo "ALERT_DEADLETTER_FLUSHED redelivered=0 remaining=0"
|
||||
exit 0
|
||||
fi
|
||||
TMP="$DEADLETTER.tmp"
|
||||
: > "$TMP"
|
||||
sent=0; kept=0
|
||||
while IFS= read -r line; do
|
||||
[[ -z "$line" ]] && continue
|
||||
if post_payload "$line"; then
|
||||
sent=$((sent + 1))
|
||||
else
|
||||
printf '%s\n' "$line" >> "$TMP"
|
||||
kept=$((kept + 1))
|
||||
fi
|
||||
done < "$DEADLETTER"
|
||||
mv "$TMP" "$DEADLETTER"
|
||||
echo "ALERT_DEADLETTER_FLUSHED redelivered=$sent remaining=$kept"
|
||||
[[ "$kept" -eq 0 ]] || exit 1
|
||||
exit 0
|
||||
fi
|
||||
|
||||
ALERT_FILE="${1:-}"
|
||||
if [[ -z "$ALERT_FILE" || ! -f "$ALERT_FILE" ]]; then
|
||||
echo "Usage: alert-dispatch.sh <alert-json-file> | --flush-deadletter" >&2
|
||||
exit 64
|
||||
fi
|
||||
|
||||
if [[ -z "$WEBHOOK" ]]; then
|
||||
echo "ALERT_WEBHOOK_UNSET alert not dispatched (set CASAN_ALERT_WEBHOOK)" >&2
|
||||
[[ "$STRICT" == "1" ]] && exit 1
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Normalize the alert, decide severity, and apply the dedup window.
|
||||
DECISION="$(python - "$ALERT_FILE" "$STATE" "$DEDUP_S" <<'PY'
|
||||
import json, sys, time
|
||||
|
||||
alert_path, state_path, window = sys.argv[1], sys.argv[2], int(sys.argv[3])
|
||||
alert = json.load(open(alert_path, encoding="utf-8"))
|
||||
body = alert.get("body", {}) if isinstance(alert.get("body"), dict) else {}
|
||||
resource = alert.get("resource", {}) if isinstance(alert.get("resource"), dict) else {}
|
||||
atype = body.get("alert.type") or alert.get("alert_type") or "unknown"
|
||||
step = body.get("step.name") or alert.get("step") or "unknown"
|
||||
service = resource.get("service.name") or alert.get("agent") or "unknown"
|
||||
critical = {"execution-failed", "circuit-open", "cost-spike", "token-overuse", "audit-gap"}
|
||||
severity = "CRITICAL" if atype in critical else "WARN"
|
||||
key = f"{service}/{step}/{atype}"
|
||||
now = int(time.time())
|
||||
last = None
|
||||
try:
|
||||
with open(state_path, encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
line = line.strip()
|
||||
if not line:
|
||||
continue
|
||||
rec = json.loads(line)
|
||||
if rec.get("key") == key:
|
||||
last = rec.get("ts")
|
||||
except OSError:
|
||||
pass
|
||||
if last is not None and now - last < window:
|
||||
print("SUPPRESS " + key)
|
||||
raise SystemExit(0)
|
||||
payload = json.dumps({
|
||||
"source": "casan-agentops",
|
||||
"severity": severity,
|
||||
"alert_type": atype,
|
||||
"step": step,
|
||||
"service": service,
|
||||
"dedup_key": key,
|
||||
"alert": alert,
|
||||
})
|
||||
with open(state_path, "a", encoding="utf-8") as fh:
|
||||
fh.write(json.dumps({"key": key, "ts": now}) + "\n")
|
||||
print("SEND " + payload)
|
||||
PY
|
||||
)"
|
||||
|
||||
case "$DECISION" in
|
||||
SUPPRESS*)
|
||||
echo "ALERT_DEDUP_SUPPRESSED key=${DECISION#SUPPRESS } window_s=$DEDUP_S"
|
||||
exit 0
|
||||
;;
|
||||
SEND*)
|
||||
PAYLOAD="${DECISION#SEND }"
|
||||
;;
|
||||
*)
|
||||
echo "ALERT_DISPATCH_ERROR unparsable alert file: $ALERT_FILE" >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
if post_payload "$PAYLOAD"; then
|
||||
echo "ALERT_DISPATCHED webhook=$WEBHOOK"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
printf '%s\n' "$PAYLOAD" >> "$DEADLETTER"
|
||||
echo "ALERT_DELIVERY_FAILED queued=dead-letter webhook=$WEBHOOK" >&2
|
||||
[[ "$STRICT" == "1" ]] && exit 1
|
||||
exit 0
|
||||
Reference in New Issue
Block a user