diff --git a/AINative_OKR_CASAN5/docs/output/casan/phase3-wave5-prompt.md b/AINative_OKR_CASAN5/docs/output/casan/phase3-wave5-prompt.md new file mode 100644 index 0000000..886b009 --- /dev/null +++ b/AINative_OKR_CASAN5/docs/output/casan/phase3-wave5-prompt.md @@ -0,0 +1,166 @@ +# CASAN Phase 3 — Wave 5 Prompt (New Session) + +> Copy toàn bộ nội dung bên dưới vào session mới. + +--- + +## Context + +Dự án: `Output_CASAN5_REFINED/AINative_OKR_CASAN5` — OKR web app (NestJS + Prisma + SQLite + React + Vite + Tailwind), được đánh giá theo framework **FPT CASAN** gồm 7 Harness H1–H7 (thang điểm 0–100). + +Mục tiêu ban đầu: tất cả harness > 80, ideally ~90. **Hiện tại ~84 average, tất cả > 80 — CASAN Level 4 genuine.** + +--- + +## Trạng thái sau Wave 4 + T1/T4 fixes (commit `a0df8fd`, 2026-07-01) + +### Điểm + +| Harness | Score | +|---|:---:| +| H1 Context | 85 | +| H2 Tool | 84 | +| H3 Evaluation | 78 | +| H4 Security | 86 | +| H5 Governance | 83 | +| H6 AgentOps | 84 | +| H7 Orchestration | 87 | +| **Average** | **~84** | + +### Verify commands (tất cả phải PASS trước khi làm gì) + +```bash +cd Output_CASAN5_REFINED/AINative_OKR_CASAN5 +bash .specify/tests/run-casan4-harness-tests.sh # 35 PASS / 0 FAIL +bash .specify/tests/adversarial-harness-tests.sh # 44 PASS / 0 FAIL +bash .specify/scripts/bash/verify-audit-chain.sh # AUDIT_CHAIN_VALID anchor=signed +bash .specify/scripts/bash/verify-tool-audit.sh # TOOL_AUDIT_VALID anchor=signed +bash .specify/scripts/bash/security-gate.sh # PASS=10 FAIL=0 SKIP=0 +npm test -w frontend # 16 PASS / 0 FAIL +``` + +### Environment + +- macOS, shell: zsh +- **Ollama local**: `ornith:9b` (qwen3.5 family) tại `127.0.0.1:11434` — UP +- Node 20+, Python 3.x (lệnh: `python` không phải `python3`) +- Không có `ANTHROPIC_API_KEY` / `OPENAI_API_KEY` +- Không có AWS/KMS/Object Lock + +--- + +## Những gì đã xong (Wave 1–4 + T1/T4) + +- **H4**: Semantic injection via Ollama ornith:9b, 30-sample corpus (recall=0.85), artifact indirect injection scanner, secrets scan, tool timeout, circuit breaker, no-bypass scan, SSRF guard +- **H5**: RSA-anchored audit chain (SHA-256 + RSA anchor), signed policy manifest, private key NOT in repo (gitignored `*.pem`) +- **H2**: Per-agent least privilege (tool-registry-gate.sh), runtime rate-limit counter, per-call JSON-schema validation (validate-tool-input.sh), tool-exec.sh timeout wired into harness +- **H3**: 16 Vitest frontend tests (Badge, ProgressBar, Zod schema, progress calc, API error), fail-before cycle proven +- **H6**: cost_source=provider_telemetry (real Ollama tokens via CASAN_STEP_NAME propagation), cost-spike detection, hallucination scanner +- **H7**: Rollback wired into pipeline (`casan-step.mjs` → `rollback-manager.sh checkpoint/execute`), drift-detect real difflib, model fallback with real failure (not exit 9 stub) +- **H1**: context-validate CONTEXT_VALID checked=24, TTL/staleness logic implemented +- **Adversarial suite**: 44 PASS / 0 FAIL (up from 22 at start of Phase 3) +- **Security gate**: PASS=10 FAIL=0 SKIP=0 (with local Ollama) + +--- + +## Residual gaps (limiting ~84 → 90) + +Những gap này cần infra hoặc scope lớn hơn: + +| Task | Harness | Gap | Cần gì | +|---|:---:|---|---| +| **CI gate integration** | H3 | Không có GitHub Actions / local CI chạy tests tự động | Cần CI pipeline config | +| **E2E / browser tests** | H3 | Không có Playwright/Cypress | Cần browser test infra | +| **Cloud model recall** | H4 | Recall=0.85, 15% novel paraphrases vẫn evade 9B model | Cần `ANTHROPIC_API_KEY` hoặc model 70B+ | +| **KMS/HSM signing** | H5 | Private key local, không có cloud key management | Cần AWS KMS hoặc tương đương | +| **Full pipeline re-run** | H6 | Pipeline chưa được re-run end-to-end trong Phase 3 | Cần `run-casan-pipeline.mjs` chạy thành công với real app | +| **Design artifact depth** | H1 | 12/24 references là stub traces (retention_gap) | Cần re-run pipeline để tạo real traces | + +**Nếu có ANTHROPIC_API_KEY**: Có thể nâng H4 recall từ 0.85 → ~0.95 bằng cách thêm Anthropic/GPT làm secondary judge. + +--- + +## Stack kỹ thuật quan trọng + +### Model call +- Script: `.specify/scripts/bash/model-router.sh` → `.specify/scripts/bash/model-call.py` +- Ollama endpoint: `http://127.0.0.1:11434/api/generate` +- `think: false` required cho ornith:9b (thinking model) +- `num_predict=16`, `temperature=0` cho classify/judge +- SSRF guard: chỉ cho phép `127.0.0.1:11434` + +### Pipeline +- Orchestrator: `scripts/casan-step.mjs` (Node.js ESM) +- Harness wrapper: `.specify/scripts/bash/casan-harness.sh` +- Steps: `01-srs → 02-bd → 03-spec → 04-reviewspec → 05-plan → 06-reviewplan → 07-dd → 08-testkit → 09-tasks → 10-reviewcode` +- Model judge gate (WP-B): wired vào steps 04, 06, 10 — AND(rule, model), SKIP non-blocking + +### Audit chain +- `.specify/logs/audit/audit.jsonl` — append-only, SHA-256 hash chain +- `.specify/logs/audit/audit-head.sig` — RSA signature của head hash +- Private key: `.specify/level5/central-governance/policy-private.pem` (gitignored, local only) +- Public key: `.specify/level5/central-governance/policy-public.pem` (in repo, cho verify) +- Verify: `bash .specify/scripts/bash/verify-audit-chain.sh` + +### Rollback (mới wired - T1) +- `rollback-manager.sh checkpoint ` → backup + ghi restore command thật vào `rollback-transactions.jsonl` +- `rollback-manager.sh execute ` → chạy `bash -c "cp backup original"` thật +- Sidecar: `docs/output/specs/001-okr-web-app/plan.checkpoint.txid` +- Wired trong `casan-step.mjs` step 05-plan (attempt≠1) và step 06-reviewplan (REJECTED) + +### Provider telemetry (mới fix - T4) +- `casan-harness.sh` export `CASAN_STEP_NAME=$ACTION_NAME` +- `model-call.py` đọc `CASAN_STEP_NAME` từ env → ghi vào `provider-usage.jsonl` +- `agent-metrics.sh` → `provider-cost-lookup.py` match by step name → `cost_source=provider_telemetry` + +--- + +## Integrity rules (KHÔNG được vi phạm) + +1. **Không fake PASS/APPROVED/verdict/token/cost/rollback/audit** — mọi kết quả phải từ execution thật +2. **Không hardcode expected PASS** — mọi test phải có thể FAIL +3. **Không dùng `.specify/tests/generate-casan-demo-context.py`** +4. **Không gọi regex-only logic là semantic hoặc model-based** +5. **Không claim cloud/hybrid judging nếu không có real API key** +6. **Không claim H5 WORM/KMS** — AWS không được cấu hình +7. **Không claim target scores đạt được nếu chưa có real audit** +8. **Nếu infra thiếu → viết BLOCKED hoặc SKIPPED với lý do rõ ràng** +9. **Mọi fix phải có fail-before proof** (test phải FAIL trước fix, PASS sau fix) +10. **Người build không tự score** (builder ≠ reviewer) + +--- + +## Key files + +| File | Vai trò | +|---|---| +| `scripts/casan-step.mjs` | Pipeline orchestrator (246+18 dòng, có checkpoint/rollback T1) | +| `.specify/scripts/bash/casan-harness.sh` | Unified harness (H4→H5→H2→H6→H4) | +| `.specify/scripts/bash/model-call.py` | Model router workhorse (Ollama, SSRF guard, fail-closed) | +| `.specify/scripts/bash/security-check.sh` | H4 input/output filter + semantic injection | +| `.specify/scripts/bash/governance-check.sh` | H5 policy enforcement | +| `.specify/scripts/bash/tool-registry-gate.sh` | H2 per-agent permission + rate-limit | +| `.specify/scripts/bash/rollback-manager.sh` | H7 checkpoint/execute real restore | +| `.specify/scripts/bash/agent-metrics.sh` | H6 token/cost/latency/hallucination | +| `.specify/scripts/bash/drift-detect.sh` | H7 difflib similarity | +| `.specify/scripts/bash/model-fallback.sh` | H7 primary/fallback runner | +| `.specify/scripts/bash/security-gate.sh` | One-command gate (10 gates) | +| `.specify/tests/adversarial-harness-tests.sh` | 44-test adversarial suite | +| `.specify/tests/run-casan4-harness-tests.sh` | 35-test harness suite | +| `.specify/security/redteam-corpus.jsonl` | 30-sample red team (20 injection + 10 benign) | +| `frontend/src/__tests__/okr.test.tsx` | 16 Vitest frontend tests | +| `docs/output/casan/phase3-final-rescore.md` | Final re-score với evidence | +| `docs/output/casan/TEAM-HANDOFF-PLAN.md` | Task list T1–T11, model assignment guide | + +--- + +## Gợi ý việc tiếp theo (nếu muốn tiếp tục push → 90) + +Theo thứ tự ROI cao nhất: + +1. **CI gate (H3 +3~5)**: Thêm `.github/workflows/ci.yml` chạy `npm test -w frontend` + `bash adversarial-harness-tests.sh` tự động trên mỗi PR +2. **Pipeline re-run (H6 +2~3)**: Chạy `run-casan-pipeline.mjs` đầy đủ (macOS, Ollama lên, node available) để lấy real per-step telemetry +3. **E2E tests (H3 +2~3)**: Viết Playwright tests cho login, create objective, update progress (cần npm run dev -w backend, dev -w frontend) +4. **Cloud recall (H4 +3)**: Nếu có ANTHROPIC_API_KEY → thêm Anthropic làm secondary judge, consensus voting 2/3 + +Mỗi task phải có **fail-before test → fix → pass-after test** cycle, không hardcode.