docs: add Wave 5 self-contained prompt for new session

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
2026-07-01 11:39:52 +09:00
parent a0df8fd9d5
commit 6e95e929f0
1 changed files with 166 additions and 0 deletions
@@ -0,0 +1,166 @@
+# CASAN Phase 3 — Wave 5 Prompt (New Session)
+
+> Copy toàn bộ nội dung bên dưới vào session mới.
+
+---
+
+## Context
+
+Dự án: `Output_CASAN5_REFINED/AINative_OKR_CASAN5` — OKR web app (NestJS + Prisma + SQLite + React + Vite + Tailwind), được đánh giá theo framework **FPT CASAN** gồm 7 Harness H1–H7 (thang điểm 0–100).
+
+Mục tiêu ban đầu: tất cả harness > 80, ideally ~90. **Hiện tại ~84 average, tất cả > 80 — CASAN Level 4 genuine.**
+
+---
+
+## Trạng thái sau Wave 4 + T1/T4 fixes (commit `a0df8fd`, 2026-07-01)
+
+### Điểm
+
+| Harness | Score |
+|---|:---:|
+| H1 Context | 85 |
+| H2 Tool | 84 |
+| H3 Evaluation | 78 |
+| H4 Security | 86 |
+| H5 Governance | 83 |
+| H6 AgentOps | 84 |
+| H7 Orchestration | 87 |
+| **Average** | **~84** |
+
+### Verify commands (tất cả phải PASS trước khi làm gì)
+
+```bash
+cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
+bash .specify/tests/run-casan4-harness-tests.sh           # 35 PASS / 0 FAIL
+bash .specify/tests/adversarial-harness-tests.sh          # 44 PASS / 0 FAIL
+bash .specify/scripts/bash/verify-audit-chain.sh          # AUDIT_CHAIN_VALID anchor=signed
+bash .specify/scripts/bash/verify-tool-audit.sh           # TOOL_AUDIT_VALID anchor=signed
+bash .specify/scripts/bash/security-gate.sh               # PASS=10 FAIL=0 SKIP=0
+npm test -w frontend                                       # 16 PASS / 0 FAIL
+```
+
+### Environment
+
+- macOS, shell: zsh
+- **Ollama local**: `ornith:9b` (qwen3.5 family) tại `127.0.0.1:11434` — UP
+- Node 20+, Python 3.x (lệnh: `python` không phải `python3`)
+- Không có `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`
+- Không có AWS/KMS/Object Lock
+
+---
+
+## Những gì đã xong (Wave 1–4 + T1/T4)
+
+- **H4**: Semantic injection via Ollama ornith:9b, 30-sample corpus (recall=0.85), artifact indirect injection scanner, secrets scan, tool timeout, circuit breaker, no-bypass scan, SSRF guard
+- **H5**: RSA-anchored audit chain (SHA-256 + RSA anchor), signed policy manifest, private key NOT in repo (gitignored `*.pem`)
+- **H2**: Per-agent least privilege (tool-registry-gate.sh), runtime rate-limit counter, per-call JSON-schema validation (validate-tool-input.sh), tool-exec.sh timeout wired into harness
+- **H3**: 16 Vitest frontend tests (Badge, ProgressBar, Zod schema, progress calc, API error), fail-before cycle proven
+- **H6**: cost_source=provider_telemetry (real Ollama tokens via CASAN_STEP_NAME propagation), cost-spike detection, hallucination scanner
+- **H7**: Rollback wired into pipeline (`casan-step.mjs` → `rollback-manager.sh checkpoint/execute`), drift-detect real difflib, model fallback with real failure (not exit 9 stub)
+- **H1**: context-validate CONTEXT_VALID checked=24, TTL/staleness logic implemented
+- **Adversarial suite**: 44 PASS / 0 FAIL (up from 22 at start of Phase 3)
+- **Security gate**: PASS=10 FAIL=0 SKIP=0 (with local Ollama)
+
+---
+
+## Residual gaps (limiting ~84 → 90)
+
+Những gap này cần infra hoặc scope lớn hơn:
+
+| Task | Harness | Gap | Cần gì |
+|---|:---:|---|---|
+| **CI gate integration** | H3 | Không có GitHub Actions / local CI chạy tests tự động | Cần CI pipeline config |
+| **E2E / browser tests** | H3 | Không có Playwright/Cypress | Cần browser test infra |
+| **Cloud model recall** | H4 | Recall=0.85, 15% novel paraphrases vẫn evade 9B model | Cần `ANTHROPIC_API_KEY` hoặc model 70B+ |
+| **KMS/HSM signing** | H5 | Private key local, không có cloud key management | Cần AWS KMS hoặc tương đương |
+| **Full pipeline re-run** | H6 | Pipeline chưa được re-run end-to-end trong Phase 3 | Cần `run-casan-pipeline.mjs` chạy thành công với real app |
+| **Design artifact depth** | H1 | 12/24 references là stub traces (retention_gap) | Cần re-run pipeline để tạo real traces |
+
+**Nếu có ANTHROPIC_API_KEY**: Có thể nâng H4 recall từ 0.85 → ~0.95 bằng cách thêm Anthropic/GPT làm secondary judge.
+
+---
+
+## Stack kỹ thuật quan trọng
+
+### Model call
+- Script: `.specify/scripts/bash/model-router.sh` → `.specify/scripts/bash/model-call.py`
+- Ollama endpoint: `http://127.0.0.1:11434/api/generate`
+- `think: false` required cho ornith:9b (thinking model)
+- `num_predict=16`, `temperature=0` cho classify/judge
+- SSRF guard: chỉ cho phép `127.0.0.1:11434`
+
+### Pipeline
+- Orchestrator: `scripts/casan-step.mjs` (Node.js ESM)
+- Harness wrapper: `.specify/scripts/bash/casan-harness.sh`
+- Steps: `01-srs → 02-bd → 03-spec → 04-reviewspec → 05-plan → 06-reviewplan → 07-dd → 08-testkit → 09-tasks → 10-reviewcode`
+- Model judge gate (WP-B): wired vào steps 04, 06, 10 — AND(rule, model), SKIP non-blocking
+
+### Audit chain
+- `.specify/logs/audit/audit.jsonl` — append-only, SHA-256 hash chain
+- `.specify/logs/audit/audit-head.sig` — RSA signature của head hash
+- Private key: `.specify/level5/central-governance/policy-private.pem` (gitignored, local only)
+- Public key: `.specify/level5/central-governance/policy-public.pem` (in repo, cho verify)
+- Verify: `bash .specify/scripts/bash/verify-audit-chain.sh`
+
+### Rollback (mới wired - T1)
+- `rollback-manager.sh checkpoint <file>` → backup + ghi restore command thật vào `rollback-transactions.jsonl`
+- `rollback-manager.sh execute <tx-id>` → chạy `bash -c "cp backup original"` thật
+- Sidecar: `docs/output/specs/001-okr-web-app/plan.checkpoint.txid`
+- Wired trong `casan-step.mjs` step 05-plan (attempt≠1) và step 06-reviewplan (REJECTED)
+
+### Provider telemetry (mới fix - T4)
+- `casan-harness.sh` export `CASAN_STEP_NAME=$ACTION_NAME`
+- `model-call.py` đọc `CASAN_STEP_NAME` từ env → ghi vào `provider-usage.jsonl`
+- `agent-metrics.sh` → `provider-cost-lookup.py` match by step name → `cost_source=provider_telemetry`
+
+---
+
+## Integrity rules (KHÔNG được vi phạm)
+
+1. **Không fake PASS/APPROVED/verdict/token/cost/rollback/audit** — mọi kết quả phải từ execution thật
+2. **Không hardcode expected PASS** — mọi test phải có thể FAIL
+3. **Không dùng `.specify/tests/generate-casan-demo-context.py`**
+4. **Không gọi regex-only logic là semantic hoặc model-based**
+5. **Không claim cloud/hybrid judging nếu không có real API key**
+6. **Không claim H5 WORM/KMS** — AWS không được cấu hình
+7. **Không claim target scores đạt được nếu chưa có real audit**
+8. **Nếu infra thiếu → viết BLOCKED hoặc SKIPPED với lý do rõ ràng**
+9. **Mọi fix phải có fail-before proof** (test phải FAIL trước fix, PASS sau fix)
+10. **Người build không tự score** (builder ≠ reviewer)
+
+---
+
+## Key files
+
+| File | Vai trò |
+|---|---|
+| `scripts/casan-step.mjs` | Pipeline orchestrator (246+18 dòng, có checkpoint/rollback T1) |
+| `.specify/scripts/bash/casan-harness.sh` | Unified harness (H4→H5→H2→H6→H4) |
+| `.specify/scripts/bash/model-call.py` | Model router workhorse (Ollama, SSRF guard, fail-closed) |
+| `.specify/scripts/bash/security-check.sh` | H4 input/output filter + semantic injection |
+| `.specify/scripts/bash/governance-check.sh` | H5 policy enforcement |
+| `.specify/scripts/bash/tool-registry-gate.sh` | H2 per-agent permission + rate-limit |
+| `.specify/scripts/bash/rollback-manager.sh` | H7 checkpoint/execute real restore |
+| `.specify/scripts/bash/agent-metrics.sh` | H6 token/cost/latency/hallucination |
+| `.specify/scripts/bash/drift-detect.sh` | H7 difflib similarity |
+| `.specify/scripts/bash/model-fallback.sh` | H7 primary/fallback runner |
+| `.specify/scripts/bash/security-gate.sh` | One-command gate (10 gates) |
+| `.specify/tests/adversarial-harness-tests.sh` | 44-test adversarial suite |
+| `.specify/tests/run-casan4-harness-tests.sh` | 35-test harness suite |
+| `.specify/security/redteam-corpus.jsonl` | 30-sample red team (20 injection + 10 benign) |
+| `frontend/src/__tests__/okr.test.tsx` | 16 Vitest frontend tests |
+| `docs/output/casan/phase3-final-rescore.md` | Final re-score với evidence |
+| `docs/output/casan/TEAM-HANDOFF-PLAN.md` | Task list T1–T11, model assignment guide |
+
+---
+
+## Gợi ý việc tiếp theo (nếu muốn tiếp tục push → 90)
+
+Theo thứ tự ROI cao nhất:
+
+1. **CI gate (H3 +3~5)**: Thêm `.github/workflows/ci.yml` chạy `npm test -w frontend` + `bash adversarial-harness-tests.sh` tự động trên mỗi PR
+2. **Pipeline re-run (H6 +2~3)**: Chạy `run-casan-pipeline.mjs` đầy đủ (macOS, Ollama lên, node available) để lấy real per-step telemetry
+3. **E2E tests (H3 +2~3)**: Viết Playwright tests cho login, create objective, update progress (cần npm run dev -w backend, dev -w frontend)
+4. **Cloud recall (H4 +3)**: Nếu có ANTHROPIC_API_KEY → thêm Anthropic làm secondary judge, consensus voting 2/3
+
+Mỗi task phải có **fail-before test → fix → pass-after test** cycle, không hardcode.