thanhnvandClaude Opus 4.8
36a4812ef3
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.
- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
.specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
- .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
-> packages/casan-harness/... (.specify/logs state kept)
- .claude/launch.json, .gitea/*-runbook.md: path prefixes
- CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
- policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.
Full gate from the new root: PASS=64 FAIL=0 SKIP=3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
167 lines
8.7 KiB
Markdown
167 lines
8.7 KiB
Markdown
# CASAN Phase 3 — Wave 5 Prompt (New Session)
|
||
|
||
> Copy toàn bộ nội dung bên dưới vào session mới.
|
||
|
||
---
|
||
|
||
## Context
|
||
|
||
Dự án: `Output_CASAN5_REFINED/AINative_OKR_CASAN5` — OKR web app (NestJS + Prisma + SQLite + React + Vite + Tailwind), được đánh giá theo framework **FPT CASAN** gồm 7 Harness H1–H7 (thang điểm 0–100).
|
||
|
||
Mục tiêu ban đầu: tất cả harness > 80, ideally ~90. **Hiện tại ~84 average, tất cả > 80 — CASAN Level 4 genuine.**
|
||
|
||
---
|
||
|
||
## Trạng thái sau Wave 4 + T1/T4 fixes (commit `a0df8fd`, 2026-07-01)
|
||
|
||
### Điểm
|
||
|
||
| Harness | Score |
|
||
|---|:---:|
|
||
| H1 Context | 85 |
|
||
| H2 Tool | 84 |
|
||
| H3 Evaluation | 78 |
|
||
| H4 Security | 86 |
|
||
| H5 Governance | 83 |
|
||
| H6 AgentOps | 84 |
|
||
| H7 Orchestration | 87 |
|
||
| **Average** | **~84** |
|
||
|
||
### Verify commands (tất cả phải PASS trước khi làm gì)
|
||
|
||
```bash
|
||
cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
|
||
bash .specify/tests/run-casan4-harness-tests.sh # 35 PASS / 0 FAIL
|
||
bash .specify/tests/adversarial-harness-tests.sh # 44 PASS / 0 FAIL
|
||
bash .specify/scripts/bash/verify-audit-chain.sh # AUDIT_CHAIN_VALID anchor=signed
|
||
bash .specify/scripts/bash/verify-tool-audit.sh # TOOL_AUDIT_VALID anchor=signed
|
||
bash .specify/scripts/bash/security-gate.sh # PASS=10 FAIL=0 SKIP=0
|
||
npm test -w frontend # 16 PASS / 0 FAIL
|
||
```
|
||
|
||
### Environment
|
||
|
||
- macOS, shell: zsh
|
||
- **Ollama local**: `ornith:9b` (qwen3.5 family) tại `127.0.0.1:11434` — UP
|
||
- Node 20+, Python 3.x (lệnh: `python` không phải `python3`)
|
||
- Không có `ANTHROPIC_API_KEY` / `OPENAI_API_KEY`
|
||
- Không có AWS/KMS/Object Lock
|
||
|
||
---
|
||
|
||
## Những gì đã xong (Wave 1–4 + T1/T4)
|
||
|
||
- **H4**: Semantic injection via Ollama ornith:9b, 30-sample corpus (recall=0.85), artifact indirect injection scanner, secrets scan, tool timeout, circuit breaker, no-bypass scan, SSRF guard
|
||
- **H5**: RSA-anchored audit chain (SHA-256 + RSA anchor), signed policy manifest, private key NOT in repo (gitignored `*.pem`)
|
||
- **H2**: Per-agent least privilege (tool-registry-gate.sh), runtime rate-limit counter, per-call JSON-schema validation (validate-tool-input.sh), tool-exec.sh timeout wired into harness
|
||
- **H3**: 16 Vitest frontend tests (Badge, ProgressBar, Zod schema, progress calc, API error), fail-before cycle proven
|
||
- **H6**: cost_source=provider_telemetry (real Ollama tokens via CASAN_STEP_NAME propagation), cost-spike detection, hallucination scanner
|
||
- **H7**: Rollback wired into pipeline (`casan-step.mjs` → `rollback-manager.sh checkpoint/execute`), drift-detect real difflib, model fallback with real failure (not exit 9 stub)
|
||
- **H1**: context-validate CONTEXT_VALID checked=24, TTL/staleness logic implemented
|
||
- **Adversarial suite**: 44 PASS / 0 FAIL (up from 22 at start of Phase 3)
|
||
- **Security gate**: PASS=10 FAIL=0 SKIP=0 (with local Ollama)
|
||
|
||
---
|
||
|
||
## Residual gaps (limiting ~84 → 90)
|
||
|
||
Những gap này cần infra hoặc scope lớn hơn:
|
||
|
||
| Task | Harness | Gap | Cần gì |
|
||
|---|:---:|---|---|
|
||
| **CI gate integration** | H3 | Không có GitHub Actions / local CI chạy tests tự động | Cần CI pipeline config |
|
||
| **E2E / browser tests** | H3 | Không có Playwright/Cypress | Cần browser test infra |
|
||
| **Cloud model recall** | H4 | Recall=0.85, 15% novel paraphrases vẫn evade 9B model | Cần `ANTHROPIC_API_KEY` hoặc model 70B+ |
|
||
| **KMS/HSM signing** | H5 | Private key local, không có cloud key management | Cần AWS KMS hoặc tương đương |
|
||
| **Full pipeline re-run** | H6 | Pipeline chưa được re-run end-to-end trong Phase 3 | Cần `run-casan-pipeline.mjs` chạy thành công với real app |
|
||
| **Design artifact depth** | H1 | 12/24 references là stub traces (retention_gap) | Cần re-run pipeline để tạo real traces |
|
||
|
||
**Nếu có ANTHROPIC_API_KEY**: Có thể nâng H4 recall từ 0.85 → ~0.95 bằng cách thêm Anthropic/GPT làm secondary judge.
|
||
|
||
---
|
||
|
||
## Stack kỹ thuật quan trọng
|
||
|
||
### Model call
|
||
- Script: `.specify/scripts/bash/model-router.sh` → `.specify/scripts/bash/model-call.py`
|
||
- Ollama endpoint: `http://127.0.0.1:11434/api/generate`
|
||
- `think: false` required cho ornith:9b (thinking model)
|
||
- `num_predict=16`, `temperature=0` cho classify/judge
|
||
- SSRF guard: chỉ cho phép `127.0.0.1:11434`
|
||
|
||
### Pipeline
|
||
- Orchestrator: `scripts/casan-step.mjs` (Node.js ESM)
|
||
- Harness wrapper: `.specify/scripts/bash/casan-harness.sh`
|
||
- Steps: `01-srs → 02-bd → 03-spec → 04-reviewspec → 05-plan → 06-reviewplan → 07-dd → 08-testkit → 09-tasks → 10-reviewcode`
|
||
- Model judge gate (WP-B): wired vào steps 04, 06, 10 — AND(rule, model), SKIP non-blocking
|
||
|
||
### Audit chain
|
||
- `.specify/logs/audit/audit.jsonl` — append-only, SHA-256 hash chain
|
||
- `.specify/logs/audit/audit-head.sig` — RSA signature của head hash
|
||
- Private key: `.specify/level5/central-governance/policy-private.pem` (gitignored, local only)
|
||
- Public key: `.specify/level5/central-governance/policy-public.pem` (in repo, cho verify)
|
||
- Verify: `bash .specify/scripts/bash/verify-audit-chain.sh`
|
||
|
||
### Rollback (mới wired - T1)
|
||
- `rollback-manager.sh checkpoint <file>` → backup + ghi restore command thật vào `rollback-transactions.jsonl`
|
||
- `rollback-manager.sh execute <tx-id>` → chạy `bash -c "cp backup original"` thật
|
||
- Sidecar: `docs/output/specs/001-okr-web-app/plan.checkpoint.txid`
|
||
- Wired trong `casan-step.mjs` step 05-plan (attempt≠1) và step 06-reviewplan (REJECTED)
|
||
|
||
### Provider telemetry (mới fix - T4)
|
||
- `casan-harness.sh` export `CASAN_STEP_NAME=$ACTION_NAME`
|
||
- `model-call.py` đọc `CASAN_STEP_NAME` từ env → ghi vào `provider-usage.jsonl`
|
||
- `agent-metrics.sh` → `provider-cost-lookup.py` match by step name → `cost_source=provider_telemetry`
|
||
|
||
---
|
||
|
||
## Integrity rules (KHÔNG được vi phạm)
|
||
|
||
1. **Không fake PASS/APPROVED/verdict/token/cost/rollback/audit** — mọi kết quả phải từ execution thật
|
||
2. **Không hardcode expected PASS** — mọi test phải có thể FAIL
|
||
3. **Không dùng `.specify/tests/generate-casan-demo-context.py`**
|
||
4. **Không gọi regex-only logic là semantic hoặc model-based**
|
||
5. **Không claim cloud/hybrid judging nếu không có real API key**
|
||
6. **Không claim H5 WORM/KMS** — AWS không được cấu hình
|
||
7. **Không claim target scores đạt được nếu chưa có real audit**
|
||
8. **Nếu infra thiếu → viết BLOCKED hoặc SKIPPED với lý do rõ ràng**
|
||
9. **Mọi fix phải có fail-before proof** (test phải FAIL trước fix, PASS sau fix)
|
||
10. **Người build không tự score** (builder ≠ reviewer)
|
||
|
||
---
|
||
|
||
## Key files
|
||
|
||
| File | Vai trò |
|
||
|---|---|
|
||
| `scripts/casan-step.mjs` | Pipeline orchestrator (246+18 dòng, có checkpoint/rollback T1) |
|
||
| `.specify/scripts/bash/casan-harness.sh` | Unified harness (H4→H5→H2→H6→H4) |
|
||
| `.specify/scripts/bash/model-call.py` | Model router workhorse (Ollama, SSRF guard, fail-closed) |
|
||
| `.specify/scripts/bash/security-check.sh` | H4 input/output filter + semantic injection |
|
||
| `.specify/scripts/bash/governance-check.sh` | H5 policy enforcement |
|
||
| `.specify/scripts/bash/tool-registry-gate.sh` | H2 per-agent permission + rate-limit |
|
||
| `.specify/scripts/bash/rollback-manager.sh` | H7 checkpoint/execute real restore |
|
||
| `.specify/scripts/bash/agent-metrics.sh` | H6 token/cost/latency/hallucination |
|
||
| `.specify/scripts/bash/drift-detect.sh` | H7 difflib similarity |
|
||
| `.specify/scripts/bash/model-fallback.sh` | H7 primary/fallback runner |
|
||
| `.specify/scripts/bash/security-gate.sh` | One-command gate (10 gates) |
|
||
| `.specify/tests/adversarial-harness-tests.sh` | 44-test adversarial suite |
|
||
| `.specify/tests/run-casan4-harness-tests.sh` | 35-test harness suite |
|
||
| `.specify/security/redteam-corpus.jsonl` | 30-sample red team (20 injection + 10 benign) |
|
||
| `frontend/src/__tests__/okr.test.tsx` | 16 Vitest frontend tests |
|
||
| `docs/output/casan/phase3-final-rescore.md` | Final re-score với evidence |
|
||
| `docs/output/casan/TEAM-HANDOFF-PLAN.md` | Task list T1–T11, model assignment guide |
|
||
|
||
---
|
||
|
||
## Gợi ý việc tiếp theo (nếu muốn tiếp tục push → 90)
|
||
|
||
Theo thứ tự ROI cao nhất:
|
||
|
||
1. **CI gate (H3 +3~5)**: Thêm `.github/workflows/ci.yml` chạy `npm test -w frontend` + `bash adversarial-harness-tests.sh` tự động trên mỗi PR
|
||
2. **Pipeline re-run (H6 +2~3)**: Chạy `run-casan-pipeline.mjs` đầy đủ (macOS, Ollama lên, node available) để lấy real per-step telemetry
|
||
3. **E2E tests (H3 +2~3)**: Viết Playwright tests cho login, create objective, update progress (cần npm run dev -w backend, dev -w frontend)
|
||
4. **Cloud recall (H4 +3)**: Nếu có ANTHROPIC_API_KEY → thêm Anthropic làm secondary judge, consensus voting 2/3
|
||
|
||
Mỗi task phải có **fail-before test → fix → pass-after test** cycle, không hardcode.
|