thanhnvandClaude Sonnet 4.6
a8edbea534
T1 (H7): casan-step.mjs now calls rollback-manager.sh checkpoint before overwriting plan.md at attempt-2, writes tx-id to plan.checkpoint.txid sidecar, and executes rollback on REJECTED verdict. rollback-transactions.jsonl records a real cp restore command. Adversarial test: checkpoint exists, real cp command recorded, plan hash matches pre-overwrite content. T4 (H6): casan-harness.sh exports CASAN_STEP_NAME=$ACTION_NAME before agent-metrics.sh so nested model calls (model-call.py) and the provider- cost-lookup.py query share the same step label. metrics.jsonl now writes cost_source=provider_telemetry instead of word_count_estimate when a real Ollama call is made within the same step. Adversarial test: verified with CASAN_STEP_NAME=t4-telemetry-test end-to-end. adversarial-harness-tests.sh: 40 → 44 PASS / 0 FAIL (+3 T1, +1 T4) security-gate.sh: PASS=10 FAIL=0 SKIP=0 (verified, local ornith:9b) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
39 lines
13 KiB
JSON
39 lines
13 KiB
JSON
{"timestamp": "2026-07-01T02:23:35Z", "harness": "L5-provider-telemetry", "provider": "sample-provider", "model": "sample-model-large", "run_id": "provider-run-001", "step": "speckit.implement", "input_tokens": 1842, "output_tokens": 936, "total_tokens": 2778, "cost_usd": 0.08334, "latency_ms": 4210, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:23:42Z", "harness": "L5-provider-telemetry", "provider": "sample-provider", "model": "sample-model-large", "run_id": "provider-run-001", "step": "speckit.implement", "input_tokens": 1842, "output_tokens": 936, "total_tokens": 2778, "cost_usd": 0.08334, "latency_ms": 4210, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:01Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "judge", "role": "judge", "input_tokens": 168, "output_tokens": 3, "total_tokens": 171, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1245, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:05Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "t4-telemetry-test", "role": "classify", "input_tokens": 208, "output_tokens": 2, "total_tokens": 210, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 406, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:08Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "generate", "role": "generate", "input_tokens": 72, "output_tokens": 26, "total_tokens": 98, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1909, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:09Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 218, "output_tokens": 3, "total_tokens": 221, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1383, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:12Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "generate", "role": "generate", "input_tokens": 72, "output_tokens": 26, "total_tokens": 98, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1911, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:14Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 220, "output_tokens": 3, "total_tokens": 223, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1401, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:17Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 219, "output_tokens": 3, "total_tokens": 222, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1388, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:19Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 218, "output_tokens": 3, "total_tokens": 221, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1413, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:21Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 220, "output_tokens": 3, "total_tokens": 223, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1385, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:23Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 219, "output_tokens": 3, "total_tokens": 222, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1408, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:26Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 217, "output_tokens": 3, "total_tokens": 220, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1515, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:28Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 217, "output_tokens": 3, "total_tokens": 220, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1422, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:30Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 215, "output_tokens": 2, "total_tokens": 217, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1342, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:32Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 219, "output_tokens": 3, "total_tokens": 222, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1396, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:35Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 219, "output_tokens": 3, "total_tokens": 222, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1426, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:37Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 216, "output_tokens": 2, "total_tokens": 218, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1348, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:39Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 217, "output_tokens": 2, "total_tokens": 219, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1359, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:41Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 218, "output_tokens": 2, "total_tokens": 220, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1341, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:43Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 216, "output_tokens": 2, "total_tokens": 218, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1342, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:46Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 218, "output_tokens": 2, "total_tokens": 220, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1372, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:48Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 214, "output_tokens": 2, "total_tokens": 216, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1359, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:50Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 232, "output_tokens": 3, "total_tokens": 235, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1507, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:53Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 231, "output_tokens": 3, "total_tokens": 234, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1518, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:55Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 231, "output_tokens": 3, "total_tokens": 234, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1526, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:57Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 232, "output_tokens": 3, "total_tokens": 235, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1520, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:24:59Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 226, "output_tokens": 2, "total_tokens": 228, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1346, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:02Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 234, "output_tokens": 3, "total_tokens": 237, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1558, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:04Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 229, "output_tokens": 3, "total_tokens": 232, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1518, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:07Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 237, "output_tokens": 3, "total_tokens": 240, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1515, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:09Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 231, "output_tokens": 2, "total_tokens": 233, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1482, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:11Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 228, "output_tokens": 3, "total_tokens": 231, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1384, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:13Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 226, "output_tokens": 2, "total_tokens": 228, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1358, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:16Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 229, "output_tokens": 2, "total_tokens": 231, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1467, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:18Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 227, "output_tokens": 2, "total_tokens": 229, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1372, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:20Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "classify", "role": "classify", "input_tokens": 228, "output_tokens": 2, "total_tokens": 230, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1340, "status": "success"}
|
|
{"timestamp": "2026-07-01T02:25:23Z", "harness": "L5-provider-telemetry", "provider": "ollama", "model": "ornith:9b", "run_id": "adhoc", "step": "judge", "role": "judge", "input_tokens": 168, "output_tokens": 3, "total_tokens": 171, "cost_usd": 0.0, "cost_source": "ollama_local_real_tokens", "latency_ms": 1254, "status": "success"}
|