CASAN/docs/output/casan/phase3-wave5-prompt.md

CASAN Phase 3 — Wave 5 Prompt (New Session)

Copy toàn bộ nội dung bên dưới vào session mới.

---

Context

Dự án: Output_CASAN5_REFINED/AINative_OKR_CASAN5 — OKR web app (NestJS + Prisma + SQLite + React + Vite + Tailwind), được đánh giá theo framework FPT CASAN gồm 7 Harness H1–H7 (thang điểm 0–100).

Mục tiêu ban đầu: tất cả harness > 80, ideally ~90. Hiện tại ~84 average, tất cả > 80 — CASAN Level 4 genuine.

---

Trạng thái sau Wave 4 + T1/T4 fixes (commit a0df8fd, 2026-07-01)

Điểm

Harness	Score
H1 Context	85
H2 Tool	84
H3 Evaluation	78
H4 Security	86
H5 Governance	83
H6 AgentOps	84
H7 Orchestration	87
Average	~84

Verify commands (tất cả phải PASS trước khi làm gì)

cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh           # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh          # 44 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh          # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh           # TOOL_AUDIT_VALID anchor=signed
bash .specify/scripts/bash/security-gate.sh               # PASS=10 FAIL=0 SKIP=0
npm test -w frontend                                       # 16 PASS / 0 FAIL

Environment

• macOS, shell: zsh
• Ollama local: ornith:9b (qwen3.5 family) tại 127.0.0.1:11434 — UP
• Node 20+, Python 3.x (lệnh: python không phải python3)
• Không có ANTHROPIC_API_KEY / OPENAI_API_KEY
• Không có AWS/KMS/Object Lock

---

Những gì đã xong (Wave 1–4 + T1/T4)

• H4: Semantic injection via Ollama ornith:9b, 30-sample corpus (recall=0.85), artifact indirect injection scanner, secrets scan, tool timeout, circuit breaker, no-bypass scan, SSRF guard
• H5: RSA-anchored audit chain (SHA-256 + RSA anchor), signed policy manifest, private key NOT in repo (gitignored *.pem)
• H2: Per-agent least privilege (tool-registry-gate.sh), runtime rate-limit counter, per-call JSON-schema validation (validate-tool-input.sh), tool-exec.sh timeout wired into harness
• H3: 16 Vitest frontend tests (Badge, ProgressBar, Zod schema, progress calc, API error), fail-before cycle proven
• H6: cost_source=provider_telemetry (real Ollama tokens via CASAN_STEP_NAME propagation), cost-spike detection, hallucination scanner
• H7: Rollback wired into pipeline (casan-step.mjs → rollback-manager.sh checkpoint/execute), drift-detect real difflib, model fallback with real failure (not exit 9 stub)
• H1: context-validate CONTEXT_VALID checked=24, TTL/staleness logic implemented
• Adversarial suite: 44 PASS / 0 FAIL (up from 22 at start of Phase 3)
• Security gate: PASS=10 FAIL=0 SKIP=0 (with local Ollama)

---

Residual gaps (limiting ~84 → 90)

Những gap này cần infra hoặc scope lớn hơn:

Task	Harness	Gap	Cần gì
CI gate integration	H3	Không có GitHub Actions / local CI chạy tests tự động	Cần CI pipeline config
E2E / browser tests	H3	Không có Playwright/Cypress	Cần browser test infra
Cloud model recall	H4	Recall=0.85, 15% novel paraphrases vẫn evade 9B model	Cần ANTHROPIC_API_KEY hoặc model 70B+
KMS/HSM signing	H5	Private key local, không có cloud key management	Cần AWS KMS hoặc tương đương
Full pipeline re-run	H6	Pipeline chưa được re-run end-to-end trong Phase 3	Cần run-casan-pipeline.mjs chạy thành công với real app
Design artifact depth	H1	12/24 references là stub traces (retention_gap)	Cần re-run pipeline để tạo real traces

Nếu có ANTHROPIC_API_KEY: Có thể nâng H4 recall từ 0.85 → ~0.95 bằng cách thêm Anthropic/GPT làm secondary judge.

---

Stack kỹ thuật quan trọng

Model call

• Script: .specify/scripts/bash/model-router.sh → .specify/scripts/bash/model-call.py
• Ollama endpoint: http://127.0.0.1:11434/api/generate
• think: false required cho ornith:9b (thinking model)
• num_predict=16, temperature=0 cho classify/judge
• SSRF guard: chỉ cho phép 127.0.0.1:11434

Pipeline

• Orchestrator: scripts/casan-step.mjs (Node.js ESM)
• Harness wrapper: .specify/scripts/bash/casan-harness.sh
• Steps: 01-srs → 02-bd → 03-spec → 04-reviewspec → 05-plan → 06-reviewplan → 07-dd → 08-testkit → 09-tasks → 10-reviewcode
• Model judge gate (WP-B): wired vào steps 04, 06, 10 — AND(rule, model), SKIP non-blocking

Audit chain

• .specify/logs/audit/audit.jsonl — append-only, SHA-256 hash chain
• .specify/logs/audit/audit-head.sig — RSA signature của head hash
• Private key: .specify/level5/central-governance/policy-private.pem (gitignored, local only)
• Public key: .specify/level5/central-governance/policy-public.pem (in repo, cho verify)
• Verify: bash .specify/scripts/bash/verify-audit-chain.sh

Rollback (mới wired - T1)

• rollback-manager.sh checkpoint <file> → backup + ghi restore command thật vào rollback-transactions.jsonl
• rollback-manager.sh execute <tx-id> → chạy bash -c "cp backup original" thật
• Sidecar: docs/output/specs/001-okr-web-app/plan.checkpoint.txid
• Wired trong casan-step.mjs step 05-plan (attempt≠1) và step 06-reviewplan (REJECTED)

Provider telemetry (mới fix - T4)

• casan-harness.sh export CASAN_STEP_NAME=$ACTION_NAME
• model-call.py đọc CASAN_STEP_NAME từ env → ghi vào provider-usage.jsonl
• agent-metrics.sh → provider-cost-lookup.py match by step name → cost_source=provider_telemetry

---

Integrity rules (KHÔNG được vi phạm)

1. Không fake PASS/APPROVED/verdict/token/cost/rollback/audit — mọi kết quả phải từ execution thật
2. Không hardcode expected PASS — mọi test phải có thể FAIL
3. Không dùng .specify/tests/generate-casan-demo-context.py
4. Không gọi regex-only logic là semantic hoặc model-based
5. Không claim cloud/hybrid judging nếu không có real API key
6. Không claim H5 WORM/KMS — AWS không được cấu hình
7. Không claim target scores đạt được nếu chưa có real audit
8. Nếu infra thiếu → viết BLOCKED hoặc SKIPPED với lý do rõ ràng
9. Mọi fix phải có fail-before proof (test phải FAIL trước fix, PASS sau fix)
10. Người build không tự score (builder ≠ reviewer)

---

Key files

File	Vai trò
scripts/casan-step.mjs	Pipeline orchestrator (246+18 dòng, có checkpoint/rollback T1)
.specify/scripts/bash/casan-harness.sh	Unified harness (H4→H5→H2→H6→H4)
.specify/scripts/bash/model-call.py	Model router workhorse (Ollama, SSRF guard, fail-closed)
.specify/scripts/bash/security-check.sh	H4 input/output filter + semantic injection
.specify/scripts/bash/governance-check.sh	H5 policy enforcement
.specify/scripts/bash/tool-registry-gate.sh	H2 per-agent permission + rate-limit
.specify/scripts/bash/rollback-manager.sh	H7 checkpoint/execute real restore
.specify/scripts/bash/agent-metrics.sh	H6 token/cost/latency/hallucination
.specify/scripts/bash/drift-detect.sh	H7 difflib similarity
.specify/scripts/bash/model-fallback.sh	H7 primary/fallback runner
.specify/scripts/bash/security-gate.sh	One-command gate (10 gates)
.specify/tests/adversarial-harness-tests.sh	44-test adversarial suite
.specify/tests/run-casan4-harness-tests.sh	35-test harness suite
.specify/security/redteam-corpus.jsonl	30-sample red team (20 injection + 10 benign)
frontend/src/__tests__/okr.test.tsx	16 Vitest frontend tests
docs/output/casan/phase3-final-rescore.md	Final re-score với evidence
docs/output/casan/TEAM-HANDOFF-PLAN.md	Task list T1–T11, model assignment guide

---

Gợi ý việc tiếp theo (nếu muốn tiếp tục push → 90)

Theo thứ tự ROI cao nhất:

1. CI gate (H3 +3~5): Thêm .github/workflows/ci.yml chạy npm test -w frontend + bash adversarial-harness-tests.sh tự động trên mỗi PR
2. Pipeline re-run (H6 +2~3): Chạy run-casan-pipeline.mjs đầy đủ (macOS, Ollama lên, node available) để lấy real per-step telemetry
3. E2E tests (H3 +2~3): Viết Playwright tests cho login, create objective, update progress (cần npm run dev -w backend, dev -w frontend)
4. Cloud recall (H4 +3): Nếu có ANTHROPIC_API_KEY → thêm Anthropic làm secondary judge, consensus voting 2/3

Mỗi task phải có fail-before test → fix → pass-after test cycle, không hardcode.