91 lines
4.0 KiB
Markdown
91 lines
4.0 KiB
Markdown
# CASAN Plan-19 — Harness Reporting Contract
|
||
|
||
## Purpose
|
||
|
||
Turn harness evidence into reviewable, portable reports. H6 is the first implemented
|
||
report; H1–H7 share one contract so later reports do not invent incompatible envelopes,
|
||
filter semantics or verdict fields.
|
||
|
||
## Source-of-truth boundary
|
||
|
||
H6 is a read-only projection over three canonical files. The report API and the existing
|
||
Control Panel resolve these paths through the same `app-root.ts` contract; harness writers
|
||
resolve them through `casan-paths.sh`.
|
||
|
||
| Source | Canonical environment variable | Default path | Required |
|
||
|---|---|---|---|
|
||
| Runtime metrics | `CASAN_TELEMETRY_METRICS_LOG` | `.specify/logs/cost/metrics.jsonl` | yes |
|
||
| Provider usage | `CASAN_TELEMETRY_PROVIDER_LOG` | `.specify/logs/level5/provider-usage.jsonl` | no |
|
||
| Alerts | `CASAN_TELEMETRY_ALERTS_LOG` | `.specify/agentops/alerts.log` | no |
|
||
|
||
Legacy names (`CASAN_DASHBOARD_METRICS`, `CASAN_CP_PROVIDER_USAGE`,
|
||
`CASAN_PROVIDER_LOG`, `CASAN_DASHBOARD_ALERTS`, and chat-specific metrics aliases) are
|
||
compatibility fallbacks only. New deployments should set the canonical variables.
|
||
|
||
Freshness is source-specific and based on filesystem modification time. A source becomes
|
||
stale when its age exceeds `CASAN_DASHBOARD_STALE_S` (default `3600` seconds). Missing
|
||
required metrics produce a `missing` freshness status and an insufficient-data verdict.
|
||
|
||
## Common H1–H7 contract
|
||
|
||
`packages/casan-control-panel/backend/src/reports/report.contract.ts` defines schema
|
||
version `1`. Every harness report has:
|
||
|
||
- identity: `report_id`, `harness`, `title`, `generated_at`;
|
||
- scope: `project`, `from`, `to`, `run`;
|
||
- decision: `verdict`, `verdict_reasons`, and explicit `thresholds`;
|
||
- evidence: source-level freshness, record counts and findings;
|
||
- quality: `complete`, `partial` or `insufficient` with warnings;
|
||
- presentation-neutral `summary` and `details` data.
|
||
|
||
The catalog endpoint marks H6 as `implemented`; H1–H5 and H7 are `contract_ready`. This
|
||
is deliberate: report presence must not be confused with measured evidence or a passing
|
||
score.
|
||
|
||
## H6 API and UI
|
||
|
||
| Capability | Route |
|
||
|---|---|
|
||
| Report catalog | `GET /api/v1/reports` |
|
||
| H6 JSON API | `GET /api/v1/reports/h6` |
|
||
| Independent export | `GET /api/v1/reports/h6/export?format=json|html` |
|
||
| Browser report | `/reports/h6` |
|
||
|
||
Optional filters are `project`, `from`, `to`, and `run`; `limit` only controls recent-row
|
||
detail and does not change summary aggregation. Project/run identifiers are allowlisted to
|
||
safe identifier characters, dates must parse as ISO-compatible values, and inverted ranges
|
||
fail with `400`.
|
||
|
||
The H6 summary includes runtime counts and outcomes, failure rate, retries, latency
|
||
distribution, local/provider tokens, estimated/provider-reported cost, provider calls and
|
||
alert signals. Verdicts are evidence-derived:
|
||
|
||
- `no_data`: no runtime or provider record matches the scope;
|
||
- `fail`: required metrics are missing or failure rate exceeds its threshold;
|
||
- `attention`: stale telemetry, failures, degraded runs, high P95 latency or alerts exist;
|
||
- `pass`: data is present and no finding crosses a configured threshold.
|
||
|
||
Defaults are `5%` maximum failure rate and `5000 ms` maximum P95 latency, configurable
|
||
through `CASAN_H6_FAILURE_RATE_THRESHOLD_PCT` and
|
||
`CASAN_H6_P95_LATENCY_THRESHOLD_MS`.
|
||
|
||
## Export integrity
|
||
|
||
HTML and JSON exports are generated from the exact report object returned by the H6 service.
|
||
The HTML artifact is standalone and print-ready; JSON keeps schema version `1` for automated
|
||
consumers. Neither export reads a second data source or contains a hard-coded maturity
|
||
score. Source paths outside the app root are represented as `[external]/<filename>` so an
|
||
export does not disclose host directory structure.
|
||
|
||
## Verification
|
||
|
||
```bash
|
||
npm run build -w @casan/control-panel-backend
|
||
npm run build -w @casan/control-panel-frontend
|
||
cd packages/casan-control-panel/backend
|
||
node --import tsx --test test/reports.test.ts
|
||
```
|
||
|
||
The report tests cover the common catalog, filtered aggregation, exact-run/no-data behavior,
|
||
query validation, HTML escaping and absence of hard-coded scoring.
|