thanhnvandClaude Fable 5
da66a36f97
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
72 lines
2.4 KiB
Bash
72 lines
2.4 KiB
Bash
#!/usr/bin/env bash
|
|
set -uo pipefail
|
|
|
|
# CASAN H6 — hosted AgentOps dashboard (D3).
|
|
# Regenerates the dashboard then serves it over HTTP (dashboard-server.py)
|
|
# with a stale-aware /healthz probe. MVP hosting: local HTTP daemon with a
|
|
# pid file; production swaps in a real host (nginx/container) same routes.
|
|
#
|
|
# Usage:
|
|
# dashboard-serve.sh start [port] regenerate + serve (default port 8787)
|
|
# dashboard-serve.sh stop stop the running server
|
|
# dashboard-serve.sh status curl /healthz of the running server
|
|
#
|
|
# Env: CASAN_DASHBOARD_STALE_S, CASAN_DASHBOARD_METRICS, CASAN_DASHBOARD_HTML,
|
|
# CASAN_AGENTOPS_DIR (pid-file location)
|
|
#
|
|
# Greppable outputs: DASHBOARD_HOSTED | DASHBOARD_STOPPED | DASHBOARD_NOT_RUNNING
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
|
AGENTOPS_DIR="${CASAN_AGENTOPS_DIR:-$PROJECT_ROOT/.specify/agentops}"
|
|
PID_FILE="$AGENTOPS_DIR/dashboard.pid"
|
|
PORT_FILE="$AGENTOPS_DIR/dashboard.port"
|
|
GENERATOR="$PROJECT_ROOT/.specify/tests/generate-agentops-dashboard.py"
|
|
CMD="${1:-start}"
|
|
mkdir -p "$AGENTOPS_DIR"
|
|
|
|
case "$CMD" in
|
|
start)
|
|
PORT="${2:-8787}"
|
|
if [[ -f "$GENERATOR" ]]; then
|
|
python "$GENERATOR" >/dev/null 2>&1 || true
|
|
fi
|
|
python "$SCRIPT_DIR/dashboard-server.py" "$PORT" &
|
|
SERVER_PID=$!
|
|
echo "$SERVER_PID" > "$PID_FILE"
|
|
echo "$PORT" > "$PORT_FILE"
|
|
for _ in 1 2 3 4 5 6 7 8 9 10; do
|
|
if curl -sS -m 2 -o /dev/null "http://127.0.0.1:$PORT/healthz" 2>/dev/null; then
|
|
echo "DASHBOARD_HOSTED url=http://127.0.0.1:$PORT/ healthz=http://127.0.0.1:$PORT/healthz pid=$SERVER_PID"
|
|
exit 0
|
|
fi
|
|
sleep 0.3
|
|
done
|
|
echo "DASHBOARD_START_FAILED port=$PORT (server did not come up)" >&2
|
|
kill "$SERVER_PID" 2>/dev/null
|
|
rm -f "$PID_FILE" "$PORT_FILE"
|
|
exit 1
|
|
;;
|
|
stop)
|
|
if [[ -f "$PID_FILE" ]] && kill "$(cat "$PID_FILE")" 2>/dev/null; then
|
|
echo "DASHBOARD_STOPPED pid=$(cat "$PID_FILE")"
|
|
rm -f "$PID_FILE" "$PORT_FILE"
|
|
exit 0
|
|
fi
|
|
echo "DASHBOARD_NOT_RUNNING" >&2
|
|
rm -f "$PID_FILE" "$PORT_FILE"
|
|
exit 1
|
|
;;
|
|
status)
|
|
if [[ -f "$PORT_FILE" ]]; then
|
|
curl -sS -m 3 "http://127.0.0.1:$(cat "$PORT_FILE")/healthz" && echo "" && exit 0
|
|
fi
|
|
echo "DASHBOARD_NOT_RUNNING" >&2
|
|
exit 1
|
|
;;
|
|
*)
|
|
echo "Usage: dashboard-serve.sh start [port] | stop | status" >&2
|
|
exit 64
|
|
;;
|
|
esac
|