CASAN/casan-next-plans/CASAN_PLAN_10_TRACEABILITY_EVAL.md

CASAN PLAN 10 — Traceability REQ→Code→Test + H3 Eval

Status 2026-07-06: MVP implemented + tested. Scope is deterministic traceability for the OKR sample app; broader H3 eval-set expansion remains a platform follow-up.

MVP delivered

Capability	Where	Verification
Parse FR-* from docs/input/okr-requirement.md	.specify/scripts/bash/traceability-matrix.py	phase10-traceability-tests.sh
Declarative FR→code→test map	.specify/traceability-map.json	gate checks every mapped file exists
Gate every FR has >=1 code file and >=1 test file	traceability-matrix.py --gate	missing FR test coverage fails
Symbol/line-level refs (object form {file, symbols, lines}, backward compatible with plain path)	traceability-matrix.py (resolve_files, symbol_present)	phase10 symbol/line checks (missing symbol/line ⇒ gate FAIL)
Evidence Pack includes traceability matrix	evidence-pack-build.py	phase3-evidence-pack-tests.sh expects 13 files
CI enforcement	ci-harness-gate.sh runs phase10-traceability	CI fails a new FR without code+test (+symbol) coverage

Verify môi trường: chạy trong WSL/Linux (msys + Python Windows lệch dịch path → fail giả). phase10 = 6/0 trong WSL.

Current result

python3 .specify/scripts/bash/traceability-matrix.py --gate
# TRACEABILITY_MATRIX requirements=5 pass=5 fail=0 symbols=2/2 ...

bash .specify/tests/phase10-traceability-tests.sh   # chạy trong WSL
# TRACEABILITY SUMMARY: PASS=6 FAIL=0

Generated artifact:

• AINative_OKR_CASAN5/docs/output/casan/traceability-matrix.json (nạp thêm symbol_refs, missing_symbols, missing_lines)

Next steps

Priority	Work	Done when
P1 ✅	Symbol/line-level refs đã làm	matrix trỏ tới symbol/line chính xác; missing ⇒ gate FAIL
P3 ✅	Enforce traceability in release CI đã có	ci-harness-gate.sh chạy phase10
P2	Expand H3 eval-set beyond the OKR sample app	independent eval corpus runs through judge gates
P4	Enrich symbol refs cho các FR còn lại (FR-02→FR-05)	mỗi FR có ≥1 symbol ref được verify