update report h6

2026-07-20 23:47:09 +07:00
parent f462079435
commit 4b6819f578
38 changed files with 1244 additions and 62 deletions
@@ -0,0 +1,90 @@
+# CASAN Plan-19 — Harness Reporting Contract
+
+## Purpose
+
+Turn harness evidence into reviewable, portable reports. H6 is the first implemented
+report; H1–H7 share one contract so later reports do not invent incompatible envelopes,
+filter semantics or verdict fields.
+
+## Source-of-truth boundary
+
+H6 is a read-only projection over three canonical files. The report API and the existing
+Control Panel resolve these paths through the same `app-root.ts` contract; harness writers
+resolve them through `casan-paths.sh`.
+
+| Source | Canonical environment variable | Default path | Required |
+|---|---|---|---|
+| Runtime metrics | `CASAN_TELEMETRY_METRICS_LOG` | `.specify/logs/cost/metrics.jsonl` | yes |
+| Provider usage | `CASAN_TELEMETRY_PROVIDER_LOG` | `.specify/logs/level5/provider-usage.jsonl` | no |
+| Alerts | `CASAN_TELEMETRY_ALERTS_LOG` | `.specify/agentops/alerts.log` | no |
+
+Legacy names (`CASAN_DASHBOARD_METRICS`, `CASAN_CP_PROVIDER_USAGE`,
+`CASAN_PROVIDER_LOG`, `CASAN_DASHBOARD_ALERTS`, and chat-specific metrics aliases) are
+compatibility fallbacks only. New deployments should set the canonical variables.
+
+Freshness is source-specific and based on filesystem modification time. A source becomes
+stale when its age exceeds `CASAN_DASHBOARD_STALE_S` (default `3600` seconds). Missing
+required metrics produce a `missing` freshness status and an insufficient-data verdict.
+
+## Common H1–H7 contract
+
+`packages/casan-control-panel/backend/src/reports/report.contract.ts` defines schema
+version `1`. Every harness report has:
+
+- identity: `report_id`, `harness`, `title`, `generated_at`;
+- scope: `project`, `from`, `to`, `run`;
+- decision: `verdict`, `verdict_reasons`, and explicit `thresholds`;
+- evidence: source-level freshness, record counts and findings;
+- quality: `complete`, `partial` or `insufficient` with warnings;
+- presentation-neutral `summary` and `details` data.
+
+The catalog endpoint marks H6 as `implemented`; H1–H5 and H7 are `contract_ready`. This
+is deliberate: report presence must not be confused with measured evidence or a passing
+score.
+
+## H6 API and UI
+
+| Capability | Route |
+|---|---|
+| Report catalog | `GET /api/v1/reports` |
+| H6 JSON API | `GET /api/v1/reports/h6` |
+| Independent export | `GET /api/v1/reports/h6/export?format=json|html` |
+| Browser report | `/reports/h6` |
+
+Optional filters are `project`, `from`, `to`, and `run`; `limit` only controls recent-row
+detail and does not change summary aggregation. Project/run identifiers are allowlisted to
+safe identifier characters, dates must parse as ISO-compatible values, and inverted ranges
+fail with `400`.
+
+The H6 summary includes runtime counts and outcomes, failure rate, retries, latency
+distribution, local/provider tokens, estimated/provider-reported cost, provider calls and
+alert signals. Verdicts are evidence-derived:
+
+- `no_data`: no runtime or provider record matches the scope;
+- `fail`: required metrics are missing or failure rate exceeds its threshold;
+- `attention`: stale telemetry, failures, degraded runs, high P95 latency or alerts exist;
+- `pass`: data is present and no finding crosses a configured threshold.
+
+Defaults are `5%` maximum failure rate and `5000 ms` maximum P95 latency, configurable
+through `CASAN_H6_FAILURE_RATE_THRESHOLD_PCT` and
+`CASAN_H6_P95_LATENCY_THRESHOLD_MS`.
+
+## Export integrity
+
+HTML and JSON exports are generated from the exact report object returned by the H6 service.
+The HTML artifact is standalone and print-ready; JSON keeps schema version `1` for automated
+consumers. Neither export reads a second data source or contains a hard-coded maturity
+score. Source paths outside the app root are represented as `[external]/<filename>` so an
+export does not disclose host directory structure.
+
+## Verification
+
+```bash
+npm run build -w @casan/control-panel-backend
+npm run build -w @casan/control-panel-frontend
+cd packages/casan-control-panel/backend
+node --import tsx --test test/reports.test.ts
+```
+
+The report tests cover the common catalog, filtered aggregation, exact-run/no-data behavior,
+query validation, HTML escaping and absence of hard-coded scoring.