Files

T

thanhnvandClaude Sonnet 4.6 6e95e929f0 docs: add Wave 5 self-contained prompt for new session

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

2026-07-01 11:39:52 +09:00

8.7 KiB

Raw Blame History

CASAN Phase 3 — Wave 5 Prompt (New Session)

Copy toàn bộ nội dung bên dưới vào session mới.

Context

Dự án: Output_CASAN5_REFINED/AINative_OKR_CASAN5 — OKR web app (NestJS + Prisma + SQLite + React + Vite + Tailwind), được đánh giá theo framework FPT CASAN gồm 7 Harness H1–H7 (thang điểm 0–100).

Mục tiêu ban đầu: tất cả harness > 80, ideally ~90. Hiện tại ~84 average, tất cả > 80 — CASAN Level 4 genuine.

Trạng thái sau Wave 4 + T1/T4 fixes (commit `a0df8fd`, 2026-07-01)

Điểm

Harness	Score
H1 Context	85
H2 Tool	84
H3 Evaluation	78
H4 Security	86
H5 Governance	83
H6 AgentOps	84
H7 Orchestration	87
Average	~84

Verify commands (tất cả phải PASS trước khi làm gì)

cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh           # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh          # 44 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh          # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh           # TOOL_AUDIT_VALID anchor=signed
bash .specify/scripts/bash/security-gate.sh               # PASS=10 FAIL=0 SKIP=0
npm test -w frontend                                       # 16 PASS / 0 FAIL

Environment

macOS, shell: zsh
Ollama local: ornith:9b (qwen3.5 family) tại 127.0.0.1:11434 — UP
Node 20+, Python 3.x (lệnh: python không phải python3)
Không có ANTHROPIC_API_KEY / OPENAI_API_KEY
Không có AWS/KMS/Object Lock

Những gì đã xong (Wave 1–4 + T1/T4)

H4: Semantic injection via Ollama ornith:9b, 30-sample corpus (recall=0.85), artifact indirect injection scanner, secrets scan, tool timeout, circuit breaker, no-bypass scan, SSRF guard
H5: RSA-anchored audit chain (SHA-256 + RSA anchor), signed policy manifest, private key NOT in repo (gitignored *.pem)
H2: Per-agent least privilege (tool-registry-gate.sh), runtime rate-limit counter, per-call JSON-schema validation (validate-tool-input.sh), tool-exec.sh timeout wired into harness
H3: 16 Vitest frontend tests (Badge, ProgressBar, Zod schema, progress calc, API error), fail-before cycle proven
H6: cost_source=provider_telemetry (real Ollama tokens via CASAN_STEP_NAME propagation), cost-spike detection, hallucination scanner
H7: Rollback wired into pipeline (casan-step.mjs → rollback-manager.sh checkpoint/execute), drift-detect real difflib, model fallback with real failure (not exit 9 stub)
H1: context-validate CONTEXT_VALID checked=24, TTL/staleness logic implemented
Adversarial suite: 44 PASS / 0 FAIL (up from 22 at start of Phase 3)
Security gate: PASS=10 FAIL=0 SKIP=0 (with local Ollama)

Residual gaps (limiting ~84 → 90)

Những gap này cần infra hoặc scope lớn hơn:

Task	Harness	Gap	Cần gì
CI gate integration	H3	Không có GitHub Actions / local CI chạy tests tự động	Cần CI pipeline config
E2E / browser tests	H3	Không có Playwright/Cypress	Cần browser test infra
Cloud model recall	H4	Recall=0.85, 15% novel paraphrases vẫn evade 9B model	Cần `ANTHROPIC_API_KEY` hoặc model 70B+
KMS/HSM signing	H5	Private key local, không có cloud key management	Cần AWS KMS hoặc tương đương
Full pipeline re-run	H6	Pipeline chưa được re-run end-to-end trong Phase 3	Cần `run-casan-pipeline.mjs` chạy thành công với real app
Design artifact depth	H1	12/24 references là stub traces (retention_gap)	Cần re-run pipeline để tạo real traces

Nếu có ANTHROPIC_API_KEY: Có thể nâng H4 recall từ 0.85 → ~0.95 bằng cách thêm Anthropic/GPT làm secondary judge.

Stack kỹ thuật quan trọng

Model call

Script: .specify/scripts/bash/model-router.sh → .specify/scripts/bash/model-call.py
Ollama endpoint: http://127.0.0.1:11434/api/generate
think: false required cho ornith:9b (thinking model)
num_predict=16, temperature=0 cho classify/judge
SSRF guard: chỉ cho phép 127.0.0.1:11434

Pipeline

Orchestrator: scripts/casan-step.mjs (Node.js ESM)
Harness wrapper: .specify/scripts/bash/casan-harness.sh
Steps: 01-srs → 02-bd → 03-spec → 04-reviewspec → 05-plan → 06-reviewplan → 07-dd → 08-testkit → 09-tasks → 10-reviewcode
Model judge gate (WP-B): wired vào steps 04, 06, 10 — AND(rule, model), SKIP non-blocking

Audit chain

.specify/logs/audit/audit.jsonl — append-only, SHA-256 hash chain
.specify/logs/audit/audit-head.sig — RSA signature của head hash
Private key: .specify/level5/central-governance/policy-private.pem (gitignored, local only)
Public key: .specify/level5/central-governance/policy-public.pem (in repo, cho verify)
Verify: bash .specify/scripts/bash/verify-audit-chain.sh

Rollback (mới wired - T1)

rollback-manager.sh checkpoint <file> → backup + ghi restore command thật vào rollback-transactions.jsonl
rollback-manager.sh execute <tx-id> → chạy bash -c "cp backup original" thật
Sidecar: docs/output/specs/001-okr-web-app/plan.checkpoint.txid
Wired trong casan-step.mjs step 05-plan (attempt≠1) và step 06-reviewplan (REJECTED)

Provider telemetry (mới fix - T4)

casan-harness.sh export CASAN_STEP_NAME=$ACTION_NAME
model-call.py đọc CASAN_STEP_NAME từ env → ghi vào provider-usage.jsonl
agent-metrics.sh → provider-cost-lookup.py match by step name → cost_source=provider_telemetry

Integrity rules (KHÔNG được vi phạm)

Không fake PASS/APPROVED/verdict/token/cost/rollback/audit — mọi kết quả phải từ execution thật
Không hardcode expected PASS — mọi test phải có thể FAIL
Không dùng .specify/tests/generate-casan-demo-context.py
Không gọi regex-only logic là semantic hoặc model-based
Không claim cloud/hybrid judging nếu không có real API key
Không claim H5 WORM/KMS — AWS không được cấu hình
Không claim target scores đạt được nếu chưa có real audit
Nếu infra thiếu → viết BLOCKED hoặc SKIPPED với lý do rõ ràng
Mọi fix phải có fail-before proof (test phải FAIL trước fix, PASS sau fix)
Người build không tự score (builder ≠ reviewer)

Key files

File	Vai trò
`scripts/casan-step.mjs`	Pipeline orchestrator (246+18 dòng, có checkpoint/rollback T1)
`.specify/scripts/bash/casan-harness.sh`	Unified harness (H4→H5→H2→H6→H4)
`.specify/scripts/bash/model-call.py`	Model router workhorse (Ollama, SSRF guard, fail-closed)
`.specify/scripts/bash/security-check.sh`	H4 input/output filter + semantic injection
`.specify/scripts/bash/governance-check.sh`	H5 policy enforcement
`.specify/scripts/bash/tool-registry-gate.sh`	H2 per-agent permission + rate-limit
`.specify/scripts/bash/rollback-manager.sh`	H7 checkpoint/execute real restore
`.specify/scripts/bash/agent-metrics.sh`	H6 token/cost/latency/hallucination
`.specify/scripts/bash/drift-detect.sh`	H7 difflib similarity
`.specify/scripts/bash/model-fallback.sh`	H7 primary/fallback runner
`.specify/scripts/bash/security-gate.sh`	One-command gate (10 gates)
`.specify/tests/adversarial-harness-tests.sh`	44-test adversarial suite
`.specify/tests/run-casan4-harness-tests.sh`	35-test harness suite
`.specify/security/redteam-corpus.jsonl`	30-sample red team (20 injection + 10 benign)
`frontend/src/__tests__/okr.test.tsx`	16 Vitest frontend tests
`docs/output/casan/phase3-final-rescore.md`	Final re-score với evidence
`docs/output/casan/TEAM-HANDOFF-PLAN.md`	Task list T1–T11, model assignment guide

Gợi ý việc tiếp theo (nếu muốn tiếp tục push → 90)

Theo thứ tự ROI cao nhất:

CI gate (H3 +3~5): Thêm .github/workflows/ci.yml chạy npm test -w frontend + bash adversarial-harness-tests.sh tự động trên mỗi PR
Pipeline re-run (H6 +2~3): Chạy run-casan-pipeline.mjs đầy đủ (macOS, Ollama lên, node available) để lấy real per-step telemetry
E2E tests (H3 +2~3): Viết Playwright tests cho login, create objective, update progress (cần npm run dev -w backend, dev -w frontend)
Cloud recall (H4 +3): Nếu có ANTHROPIC_API_KEY → thêm Anthropic làm secondary judge, consensus voting 2/3

Mỗi task phải có fail-before test → fix → pass-after test cycle, không hardcode.

8.7 KiB Raw Blame History Unescape Escape