Nam Pham Dinh ThanhandClaude Sonnet 4.6
838b2473b6
WV4-A: Added 16 Vitest/RTL tests to frontend (jsdom env, fail-before proof verified) WV4-B: Created 12 stub traces for pipeline retention gap; fixed MSYS2/Python path mismatch in context-validate.sh; run-casan4-harness-tests.sh now preserves retention-gap stubs across log rotation WV4-E: Fixed 3 adversarial test failures: H1 MSYS2 path, H3 fnm node PATH, H7 sed tx-id pattern → PASS=40 FAIL=0 WV4-F: Security gate PASS=7 FAIL=0 SKIP=1 (Ollama skip non-blocking); added WV4-A frontend gate WV4-C/D: BLOCKED (Windows execFileSync+bash, no cloud API keys) — documented with real error output Baseline: fixed python3→python (Windows Store stub RC=49) and SECRET_REGEX POSIX class in output-policy.yaml Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
50 lines
1.7 KiB
Bash
Executable File
50 lines
1.7 KiB
Bash
Executable File
#!/usr/bin/env bash
|
|
set -uo pipefail
|
|
|
|
# CASAN H6 cost-spike detector (WP-C).
|
|
# Answers the H6 key question: "if a step suddenly costs 3x the tokens, does
|
|
# anyone know?". Reads real per-step usage from provider-usage.jsonl, computes
|
|
# the median total_tokens across steps, and flags any step exceeding
|
|
# MULTIPLIER x median. Exits non-zero if a spike is found (so a pipeline/CI gate
|
|
# goes red).
|
|
#
|
|
# Usage: cost-spike-detect.sh [provider-usage.jsonl] [multiplier]
|
|
# Exit: 0 no spike, 2 spike detected, 64 usage, 3 not enough data.
|
|
|
|
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
|
PROJECT_ROOT="$(cd "$SCRIPT_DIR/../../.." && pwd)"
|
|
LOG="${1:-$PROJECT_ROOT/.specify/logs/level5/provider-usage.jsonl}"
|
|
MULT="${2:-3.0}"
|
|
|
|
[[ -f "$LOG" ]] || { echo "COST_SPIKE_NO_DATA file=$LOG" >&2; exit 3; }
|
|
|
|
python - "$LOG" "$MULT" <<'PY'
|
|
import json, sys, statistics
|
|
path, mult = sys.argv[1], float(sys.argv[2])
|
|
rows = []
|
|
for line in open(path, encoding="utf-8"):
|
|
line = line.strip()
|
|
if not line:
|
|
continue
|
|
try:
|
|
r = json.loads(line)
|
|
except ValueError:
|
|
continue
|
|
if "total_tokens" in r:
|
|
rows.append((r.get("step", "?"), int(r["total_tokens"])))
|
|
if len(rows) < 3:
|
|
sys.stderr.write(f"COST_SPIKE_NO_DATA records={len(rows)} (need >=3)\n")
|
|
raise SystemExit(3)
|
|
tokens = [t for _, t in rows]
|
|
median = statistics.median(tokens)
|
|
threshold = median * mult
|
|
spikes = [(s, t) for s, t in rows if t > threshold]
|
|
print(f"records={len(rows)} median_tokens={median} threshold={threshold:.0f} (x{mult})")
|
|
for s, t in spikes:
|
|
print(f"SPIKE step={s} tokens={t} (> {threshold:.0f})")
|
|
if spikes:
|
|
sys.stderr.write(f"COST_SPIKE_DETECTED count={len(spikes)}\n")
|
|
raise SystemExit(2)
|
|
print("COST_SPIKE_NONE")
|
|
PY
|