CASAN Plan-19 — Harness Reporting Contract

Purpose

Turn harness evidence into reviewable, portable reports. H6 is the first implemented report; H1–H7 share one contract so later reports do not invent incompatible envelopes, filter semantics or verdict fields.

Source-of-truth boundary

H6 is a read-only projection over three canonical files. The report API and the existing Control Panel resolve these paths through the same app-root.ts contract; harness writers resolve them through casan-paths.sh.

Source	Canonical environment variable	Default path	Required
Runtime metrics	`CASAN_TELEMETRY_METRICS_LOG`	`.specify/logs/cost/metrics.jsonl`	yes
Provider usage	`CASAN_TELEMETRY_PROVIDER_LOG`	`.specify/logs/level5/provider-usage.jsonl`	no
Alerts	`CASAN_TELEMETRY_ALERTS_LOG`	`.specify/agentops/alerts.log`	no

Legacy names (CASAN_DASHBOARD_METRICS, CASAN_CP_PROVIDER_USAGE, CASAN_PROVIDER_LOG, CASAN_DASHBOARD_ALERTS, and chat-specific metrics aliases) are compatibility fallbacks only. New deployments should set the canonical variables.

Freshness is source-specific and based on filesystem modification time. A source becomes stale when its age exceeds CASAN_DASHBOARD_STALE_S (default 3600 seconds). Missing required metrics produce a missing freshness status and an insufficient-data verdict.

Common H1–H7 contract

packages/casan-control-panel/backend/src/reports/report.contract.ts defines schema version 1. Every harness report has:

identity: report_id, harness, title, generated_at;
scope: project, from, to, run;
decision: verdict, verdict_reasons, and explicit thresholds;
evidence: source-level freshness, record counts and findings;
quality: complete, partial or insufficient with warnings;
presentation-neutral summary and details data.

The catalog endpoint marks H6 as implemented; H1–H5 and H7 are contract_ready. This is deliberate: report presence must not be confused with measured evidence or a passing score.

H6 API and UI

Capability	Route
Report catalog	`GET /api/v1/reports`
H6 JSON API	`GET /api/v1/reports/h6`
Independent export	`GET /api/v1/reports/h6/export?format=json
Browser report	`/reports/h6`

Optional filters are project, from, to, and run; limit only controls recent-row detail and does not change summary aggregation. Project/run identifiers are allowlisted to safe identifier characters, dates must parse as ISO-compatible values, and inverted ranges fail with 400.

The H6 summary includes runtime counts and outcomes, failure rate, retries, latency distribution, local/provider tokens, estimated/provider-reported cost, provider calls and alert signals. Verdicts are evidence-derived:

no_data: no runtime or provider record matches the scope;
fail: required metrics are missing or failure rate exceeds its threshold;
attention: stale telemetry, failures, degraded runs, high P95 latency or alerts exist;
pass: data is present and no finding crosses a configured threshold.

Defaults are 5% maximum failure rate and 5000 ms maximum P95 latency, configurable through CASAN_H6_FAILURE_RATE_THRESHOLD_PCT and CASAN_H6_P95_LATENCY_THRESHOLD_MS.

Export integrity

HTML and JSON exports are generated from the exact report object returned by the H6 service. The HTML artifact is standalone and print-ready; JSON keeps schema version 1 for automated consumers. Neither export reads a second data source or contains a hard-coded maturity score. Source paths outside the app root are represented as [external]/<filename> so an export does not disclose host directory structure.

Verification

npm run build -w @casan/control-panel-backend
npm run build -w @casan/control-panel-frontend
cd packages/casan-control-panel/backend
node --import tsx --test test/reports.test.ts

The report tests cover the common catalog, filtered aggregation, exact-run/no-data behavior, query validation, HTML escaping and absence of hard-coded scoring.

4.0 KiB Raw Blame History Unescape Escape