thanhnvandClaude Fable 5
da66a36f97
Close the three gaps the scoring report itself flagged for H6 plus V15, each as a real MVP + fail-able adversarial test (same pattern that lifted H5): - D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing, dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict. Wired into agent-metrics.sh so a failing step pages live end-to-end. - D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY. - D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale). - D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW). New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints (webhook sink, mock provider API, dashboard server) — deterministic, no model needed. Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing — same fix class as tool-audit-lib.sh / governance-check.sh. Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites (KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced: scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS (Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
69 lines
2.3 KiB
Bash
69 lines
2.3 KiB
Bash
#!/usr/bin/env bash
|
|
set -uo pipefail
|
|
|
|
# CASAN H6 — local-vs-provider telemetry reconciliation (D2).
|
|
# Provider-API usage records are the billing ground truth; local metrics must
|
|
# not under-report tokens (the cost-hiding attack: trim local metrics so a
|
|
# runaway/exfil step looks cheap). Per step, local claimed tokens must cover
|
|
# provider-reported tokens within a tolerance; a provider step entirely absent
|
|
# from local metrics is also a discrepancy (hidden run).
|
|
#
|
|
# Usage: telemetry-reconcile.sh <local-metrics.jsonl> <provider-usage.jsonl> [tolerance_pct]
|
|
#
|
|
# Greppable outputs: TELEMETRY_RECONCILED | TELEMETRY_DISCREPANCY
|
|
|
|
LOCAL_LOG="${1:-}"
|
|
PROVIDER_LOG="${2:-}"
|
|
TOLERANCE_PCT="${3:-10}"
|
|
|
|
if [[ -z "$LOCAL_LOG" || -z "$PROVIDER_LOG" || ! -f "$PROVIDER_LOG" ]]; then
|
|
echo "Usage: telemetry-reconcile.sh <local-metrics.jsonl> <provider-usage.jsonl> [tolerance_pct]" >&2
|
|
exit 64
|
|
fi
|
|
|
|
python - "$LOCAL_LOG" "$PROVIDER_LOG" "$TOLERANCE_PCT" <<'PY'
|
|
import json
|
|
import sys
|
|
|
|
local_path, provider_path, tol_pct = sys.argv[1], sys.argv[2], float(sys.argv[3])
|
|
|
|
def sums_by_step(path):
|
|
totals = {}
|
|
try:
|
|
with open(path, encoding="utf-8") as fh:
|
|
for line in fh:
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
rec = json.loads(line)
|
|
step = rec.get("step")
|
|
if step is None:
|
|
continue
|
|
totals[step] = totals.get(step, 0) + int(rec.get("total_tokens", 0))
|
|
except OSError:
|
|
pass
|
|
return totals
|
|
|
|
local = sums_by_step(local_path)
|
|
provider = sums_by_step(provider_path)
|
|
if not provider:
|
|
raise SystemExit("TELEMETRY_DISCREPANCY provider log empty — nothing to reconcile against")
|
|
|
|
issues = []
|
|
for step, prov_tokens in sorted(provider.items()):
|
|
loc_tokens = local.get(step)
|
|
if loc_tokens is None:
|
|
issues.append(f"step={step} local=MISSING provider={prov_tokens}")
|
|
continue
|
|
floor = prov_tokens * (1 - tol_pct / 100.0)
|
|
if loc_tokens < floor:
|
|
issues.append(f"step={step} local={loc_tokens} provider={prov_tokens} (under-reported beyond {tol_pct}%)")
|
|
|
|
if issues:
|
|
for issue in issues:
|
|
print(f"TELEMETRY_DISCREPANCY {issue}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
print(f"TELEMETRY_RECONCILED steps={len(provider)} tolerance_pct={tol_pct}")
|
|
PY
|
|
exit $?
|