thanhnvandClaude Opus 4.8
664bd1f00c
Physically move the pure-code subtrees out of .specify into the package, leaving compat symlinks at the old .specify/<dir> paths so every existing reference (internal CASAN_HARNESS_ROOT + external CI/docker/mjs) keeps resolving. Runtime state stays put. Moved (git mv): scripts/ tests/ security/ templates/ config/ governance/ memory/ .specify/<dir> -> packages/casan-harness/<dir> (+ .specify/<dir> symlink) Stays in .specify (state/governance/domain, handled later): logs/ agentops/ level5/ init-options.json traceability-map.json Python `.resolve()` self-location followed the compat symlink into packages and lost the app root; generate-casan-demo-context.py, generate-agentops-dashboard.py and dashboard-server.py now walk UP for the `.specify` state marker instead of a fixed parent depth (fixes "missing trace files" in run-casan4). Full gate: PASS=64 FAIL=0 SKIP=3 (CASAN_CI_STEP_TIMEOUT_SEC=1200 — track-a ~450s runs close to the 600s default and can tip over under load; this is timing variance, not a regression — it passed cleanly with headroom). Runtime log/audit artifacts kept unstaged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
69 lines
2.3 KiB
Bash
69 lines
2.3 KiB
Bash
#!/usr/bin/env bash
|
|
set -uo pipefail
|
|
|
|
# CASAN H6 — local-vs-provider telemetry reconciliation (D2).
|
|
# Provider-API usage records are the billing ground truth; local metrics must
|
|
# not under-report tokens (the cost-hiding attack: trim local metrics so a
|
|
# runaway/exfil step looks cheap). Per step, local claimed tokens must cover
|
|
# provider-reported tokens within a tolerance; a provider step entirely absent
|
|
# from local metrics is also a discrepancy (hidden run).
|
|
#
|
|
# Usage: telemetry-reconcile.sh <local-metrics.jsonl> <provider-usage.jsonl> [tolerance_pct]
|
|
#
|
|
# Greppable outputs: TELEMETRY_RECONCILED | TELEMETRY_DISCREPANCY
|
|
|
|
LOCAL_LOG="${1:-}"
|
|
PROVIDER_LOG="${2:-}"
|
|
TOLERANCE_PCT="${3:-10}"
|
|
|
|
if [[ -z "$LOCAL_LOG" || -z "$PROVIDER_LOG" || ! -f "$PROVIDER_LOG" ]]; then
|
|
echo "Usage: telemetry-reconcile.sh <local-metrics.jsonl> <provider-usage.jsonl> [tolerance_pct]" >&2
|
|
exit 64
|
|
fi
|
|
|
|
python - "$LOCAL_LOG" "$PROVIDER_LOG" "$TOLERANCE_PCT" <<'PY'
|
|
import json
|
|
import sys
|
|
|
|
local_path, provider_path, tol_pct = sys.argv[1], sys.argv[2], float(sys.argv[3])
|
|
|
|
def sums_by_step(path):
|
|
totals = {}
|
|
try:
|
|
with open(path, encoding="utf-8") as fh:
|
|
for line in fh:
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
rec = json.loads(line)
|
|
step = rec.get("step")
|
|
if step is None:
|
|
continue
|
|
totals[step] = totals.get(step, 0) + int(rec.get("total_tokens", 0))
|
|
except OSError:
|
|
pass
|
|
return totals
|
|
|
|
local = sums_by_step(local_path)
|
|
provider = sums_by_step(provider_path)
|
|
if not provider:
|
|
raise SystemExit("TELEMETRY_DISCREPANCY provider log empty — nothing to reconcile against")
|
|
|
|
issues = []
|
|
for step, prov_tokens in sorted(provider.items()):
|
|
loc_tokens = local.get(step)
|
|
if loc_tokens is None:
|
|
issues.append(f"step={step} local=MISSING provider={prov_tokens}")
|
|
continue
|
|
floor = prov_tokens * (1 - tol_pct / 100.0)
|
|
if loc_tokens < floor:
|
|
issues.append(f"step={step} local={loc_tokens} provider={prov_tokens} (under-reported beyond {tol_pct}%)")
|
|
|
|
if issues:
|
|
for issue in issues:
|
|
print(f"TELEMETRY_DISCREPANCY {issue}", file=sys.stderr)
|
|
raise SystemExit(1)
|
|
print(f"TELEMETRY_RECONCILED steps={len(provider)} tolerance_pct={tol_pct}")
|
|
PY
|
|
exit $?
|