Hướng A — scorecard.sh (video demo):
- h5_1 approval workflow: hardcode 0 → governance-check deploy live (approval_required)
- h6_2 hallucination rate: hardcode 0 → hallucination-scan phân biệt dirty>clean live
- "N/5 mục" chuyển từ text cứng sang đếm động
- H4/H5/H6 → 100/100 (5/5 gate live), Average 57.9 → 90.0
Hướng B — run-casan-pipeline.mjs:
- fallback: stub 'exit 9' → 'cat /nonexistent' (real failure, nhất quán adversarial T3)
- drift: giữ so fallback-output vs golden (clean run=1.0); năng lực phát hiện
drift thật chứng minh ở adversarial suite
- Full 12-step run verify: H1 CONTEXT_VALID=24, H2 tool-audit records=25 signed,
H5 audit-chain records=22 signed, H6 provider_telemetry per-step thật, H7 rollback real
phase3-real-run-scoring.md: giải thích vì sao scorecard cũ cho H5=60/H6=80
(hardcode), phân biệt scorecard-90 vs re-score-84 (2 mục đích khác nhau).
Verify: adversarial 44/0, security-gate 11/0/0, pipeline 12 steps OK.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
T1 (H7): casan-step.mjs now calls rollback-manager.sh checkpoint before
overwriting plan.md at attempt-2, writes tx-id to plan.checkpoint.txid
sidecar, and executes rollback on REJECTED verdict. rollback-transactions.jsonl
records a real cp restore command. Adversarial test: checkpoint exists,
real cp command recorded, plan hash matches pre-overwrite content.
T4 (H6): casan-harness.sh exports CASAN_STEP_NAME=$ACTION_NAME before
agent-metrics.sh so nested model calls (model-call.py) and the provider-
cost-lookup.py query share the same step label. metrics.jsonl now writes
cost_source=provider_telemetry instead of word_count_estimate when a real
Ollama call is made within the same step. Adversarial test: verified with
CASAN_STEP_NAME=t4-telemetry-test end-to-end.
adversarial-harness-tests.sh: 40 → 44 PASS / 0 FAIL (+3 T1, +1 T4)
security-gate.sh: PASS=10 FAIL=0 SKIP=0 (verified, local ornith:9b)
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- frontend/package.json: add @testing-library/dom ^10.0.0 (missing peer
dep of @testing-library/react that caused test failure on macOS)
- docs: update security gate result to PASS=10 FAIL=0 SKIP=0 (macOS
with local ornith:9b) vs PASS=7 SKIP=1 on Windows (no Ollama)
- audit logs: real evidence from running all 10 gates (adversarial suite,
model router, red-team 30-sample, judge gate, frontend Vitest)
- remove 10 timestamp-named trace stubs (not referenced by
pipeline-context.yaml; UUID stubs in place and validated)
Verified: security-gate.sh PASS=10 FAIL=0 SKIP=0
adversarial-harness-tests.sh PASS=40 FAIL=0
npm test -w frontend: 16 PASS / 0 FAIL
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
