feat: complete llm sourcegen plan
This commit is contained in:
thanhnv
@@ -10,7 +10,7 @@
|
||||
> gì phải làm"**. Control **đã implement+test** → xem `CASAN_HARDENING_STATUS.md`.
|
||||
> Mục lục plan → `CASAN_PLAN_00_INDEX.md`. Ba file không lặp trạng thái chi
|
||||
> tiết; Index chỉ trỏ tới hai file này.
|
||||
> Test hiện tại: **218 PASS / 0 FAIL** trên 13 core harness suite; **Plan-16 security remediation 33 SEC suite = 181 PASS / 0 FAIL** (WSL, đã nối `ci-harness-gate.sh`); local-prod Docker infra lab **2 PASS / 0 FAIL** (`infra-lab verify`: 7/0 internal checks); `phase3-model-router` riêng **11 PASS / 0 FAIL** (3 case cần Ollama live); `phase2-sourcegen` riêng **3 PASS / 0 FAIL**; frontend Vitest **16 PASS / 0 FAIL**. Backend `npm test` còn bị chặn bởi test-infra cũ (`schema.prisma` MySQL nhưng `setup-sqlite.mjs` chạy SQLite).
|
||||
> Test hiện tại: **218 PASS / 0 FAIL** trên 13 core harness suite; **Plan-16 security remediation 33 SEC suite = 181 PASS / 0 FAIL** (WSL, đã nối `ci-harness-gate.sh`); local-prod Docker infra lab **2 PASS / 0 FAIL** (`infra-lab verify`: 7/0 internal checks); `phase3-model-router` riêng **11 PASS / 0 FAIL** (3 case cần Ollama live); `phase2-sourcegen` riêng **10 PASS / 0 FAIL**; frontend Vitest **16 PASS / 0 FAIL**; backend `npm test` **46 PASS / 0 FAIL / 3 SKIP** (LLM judge thiếu Anthropic key).
|
||||
> Điểm công tâm vẫn quanh **~81/100**, harness thấp nhất 80; TIER 2 infra thật vẫn là trần Strong.
|
||||
> Nguồn liên quan: `CASAN_HARDENING_STATUS.md` (chi tiết control) · `evidence/scoring-run-report.md` (điểm).
|
||||
|
||||
@@ -48,7 +48,7 @@
|
||||
|---|:--:|---|
|
||||
| **10 Traceability + H3 Eval** | ✅ MVP done+test | Đã nối traceability vào Evidence Pack + **symbol/line-level refs** (missing symbol/line ⇒ gate FAIL, phase10 6/0 WSL) + enforce trong CI (`ci-harness-gate.sh`). Sau MVP: H3 eval-set độc lập (nhiều model) + enrich symbol refs cho FR-02→FR-05. |
|
||||
| **03 Cloud patch** | 🟡 MVP done+test | `model-call.py` đã hỗ trợ `openai:<model>` và `anthropic:<model>` qua endpoint hard-pin + API key env; parser token usage và malformed payload có deterministic test. Còn live smoke với key thật + billing usage API ground truth. |
|
||||
| **02 LLM source-gen** | 🟡 đợt A MVP done+test | `01-srs` và `02-bd` hỗ trợ `CASAN_GEN_MODE=model`: gọi `model-router.sh --role generate`, scan H4 draft bằng `artifact-scan.sh`, validate token bắt buộc, fallback template nếu lỗi. Test `phase2-sourcegen` 3/0. Còn đợt B/C/D + full live pipeline smoke. |
|
||||
| **02 LLM source-gen** | ✅ **A–D done+test (offline/local)** | `01-srs`, `02-bd`, `03-spec`, `05-plan`, `07-dd`, `08-testkit`, `09-tasks`, `10-implement` hỗ trợ `CASAN_GEN_MODE=model`: gọi `model-router.sh --role generate`, scan H4 draft bằng `artifact-scan.sh`, validate token bắt buộc, ghi H5 sourcegen audit + H6 telemetry, fallback template nếu lỗi. Real pipeline có STEP10 implement draft + STEP12 run-tests gate; retry/escalation loops STEP5→STEP3, STEP7→STEP6, STEP11→STEP10. Test `phase2-sourcegen` **10/0**; full template pipeline smoke trong `/tmp` tới STEP12 PASS; backend **46/0/3skip**, frontend **16/0**. Cloud OpenAI/Anthropic live smoke + billing ground truth thuộc Plan-03 (cần key thật). |
|
||||
| **04 Self-improve** | � core done+test | `packages/casan-harness/scripts/bash/self-improve.py`: `propose` đọc metrics/drift → proposal dry-run (không ghi); `apply` bắt buộc approval, áp qua governed store (audit); sensitive/loosen luôn cần duyệt. `phase-selfimprove-tests.sh` 7/0 (WSL), nối CI. Còn: luật đề xuất phong phú hơn (corpus/model escalation), chạy định kỳ CI (05). |
|
||||
| **05 CI/CD** | 🟡 CI gate MVP done | `packages/casan-harness/scripts/bash/ci-harness-gate.sh` chạy các suite harness/hardening/sourcegen/traceability/frontend theo thứ tự an toàn, có timeout/filter; `.gitea/workflows/harness-ci.yml` gọi gate trên push/PR. Filtered local verify PASS=2/0. Còn: full gate xanh trên runner thật, xử lý A6 nếu còn chậm/treo, bật Docker infra lab nếu runner hỗ trợ, package/release artifact `fpt-casan-sdd-harness`. |
|
||||
| **06 Onboard dự án 2** | 📋 chưa bắt đầu (🔓 **đã mở khoá** — 01 done) | Chứng minh reuse: cắm 1 repo khác + golden/corpus/input, đăng ký qua `verify-harness-reuse.sh`, không sửa gate. Phụ thuộc 01 ✅. App mới chỉ cần `apps/<project>/domain/` + set `CASAN_DOMAIN_ROOT` (đã có `domain_root` per-project trong `project-registry.json`). |
|
||||
|
||||
@@ -20,7 +20,7 @@
|
||||
| # | File | Mảng | Nhãn |
|
||||
|---|---|---|:--:|
|
||||
| 01 | `CASAN_PLAN_01_RESTRUCTURE.md` | Tái cấu trúc thư mục Phase 0→6 + tách package harness | ✅ (2026-07-08, merged main) |
|
||||
| 02 | `CASAN_PLAN_02_LLM_SOURCEGEN.md` | Nối LLM thật vào sinh source thay template | 🟡 (đợt A) |
|
||||
| 02 | `CASAN_PLAN_02_LLM_SOURCEGEN.md` | Nối LLM thật vào sinh source thay template | ✅ A–D done+test |
|
||||
| 03 | `CASAN_PLAN_03_CLOUD_PATCH.md` | Patch cloud OpenAI/Anthropic, bỏ stub | 🟡 / 🔌 live |
|
||||
| 04 | `CASAN_PLAN_04_SELFIMPROVE.md` | Khép vòng `casan improve` | 📋 |
|
||||
| 05 | `CASAN_PLAN_05_CICD.md` | CI/CD + release package | 🟡 |
|
||||
@@ -54,7 +54,7 @@
|
||||
| **P3** | ✅ **Plan-01 restructure DONE** (2026-07-08, merged main; gate 64/0/3) | Đã tách package harness (`packages/casan-harness`) + domain (`apps/okr/domain`) → **06/12 đã mở khoá** |
|
||||
| **P4** | **Plan-06 onboard dự án 2** | Chứng minh reuse core harness (thuyết phục nhất) |
|
||||
| **P5** | **Plan-05 CI/CD** runner green + release package | Tự động hoá bảo vệ core harness |
|
||||
| P6 | Plan-02 source-gen đợt B–D | **Thứ yếu** — chỉ mở rộng sau khi core vững; mọi output vẫn qua H1→H7 |
|
||||
| P6 | ✅ Plan-02 source-gen A–D | Done offline/local; cloud provider live smoke nằm ở Plan-03 vì cần key thật |
|
||||
| P7 | Plan-03 live cloud smoke | Cần key thật (điều kiện gỡ blocker: xem Plan-03) |
|
||||
| P8 | Plan-04 / 08 / 12 | Làm sau các plan nền |
|
||||
|
||||
|
||||
@@ -1,16 +1,17 @@
|
||||
# KẾ HOẠCH 02 — Nối LLM thật vào sinh source (thay template deterministic)
|
||||
|
||||
> Status 2026-07-06: **Đợt A MVP implemented + tested** cho `01-srs` và
|
||||
> `02-bd`. Mặc định vẫn dùng template deterministic; khi bật
|
||||
> `CASAN_GEN_MODE=model`, hai step này gọi `model-router.sh --role generate`,
|
||||
> H4 artifact-scan output nháp, validate tiêu chí tối thiểu, rồi mới ghi
|
||||
> artifact. Lỗi model / output thiếu / H4 block đều fallback template và ghi
|
||||
> `source=template-fallback` trong report.
|
||||
> Status 2026-07-09: **Plan-02 A–D implemented + tested (offline/local)**.
|
||||
> Mặc định vẫn dùng template deterministic; khi bật `CASAN_GEN_MODE=model`,
|
||||
> các step `01-srs`, `02-bd`, `03-spec`, `05-plan`, `07-dd`, `08-testkit`,
|
||||
> `09-tasks`, `10-implement` gọi `model-router.sh --role generate`, H4
|
||||
> `artifact-scan` output nháp, validate token bắt buộc, ghi H5 sourcegen audit
|
||||
> + H6 sourcegen telemetry, rồi mới ghi artifact. Lỗi model / output thiếu /
|
||||
> H4 block đều fallback template và ghi `source=template-fallback` trong report.
|
||||
>
|
||||
> Hiện `casan-step.mjs` vẫn sinh các step sau bằng **template hard-code** trong
|
||||
> `switch(step)` [có, đọc code]; model đã dùng ở H3 judge + H4 semantic và nay
|
||||
> ở source-gen đợt A. Kế hoạch: mở rộng dần để LLM **thật sự sinh artifact**,
|
||||
> nhưng **mọi output vẫn chui qua H1→H7**.
|
||||
> `run-casan-pipeline.mjs` đã có STEP10 implement draft + STEP12 run-tests
|
||||
> gate thật; STEP5/STEP7/STEP11 có retry/escalation loop. Cloud OpenAI/Anthropic
|
||||
> live smoke vẫn thuộc Plan-03 vì cần API key thật; Plan-02 local path dùng
|
||||
> Ollama qua router, timeout sourcegen mặc định 240s cho prompt dài.
|
||||
>
|
||||
> Phụ thuộc: nên làm sau **03 (cloud patch)** để có lựa chọn model mạnh cho bước khó; **01** giúp gọn nhưng không bắt buộc.
|
||||
|
||||
@@ -35,8 +36,8 @@ Thứ tự chuyển ưu tiên step **rõ ràng, ít rủi ro** trước:
|
||||
| Đợt | Step chuyển | Vì sao trước/sau |
|
||||
|---|---|---|
|
||||
| A | `01-srs`, `02-bd` | văn bản có cấu trúc, dễ chấm, rủi ro thấp |
|
||||
| B | `03-spec`, `06-plan` | có review loop (STEP5/7) đỡ lỗi |
|
||||
| C | `08-dd`, `09-tasks` | phụ thuộc spec/plan tốt |
|
||||
| B | `03-spec`, `05-plan` | có review loop (STEP5/7) đỡ lỗi |
|
||||
| C | `07-dd`, `08-testkit`, `09-tasks` | phụ thuộc spec/plan tốt |
|
||||
| D | `10-implement` (code) | rủi ro cao nhất → làm cuối, cần test thật (STEP12) làm lưới |
|
||||
|
||||
---
|
||||
@@ -45,17 +46,17 @@ Thứ tự chuyển ưu tiên step **rõ ràng, ít rủi ro** trước:
|
||||
|
||||
| Task | Việc | File | Verify | Done khi |
|
||||
|---|---|---|---|---|
|
||||
| 2.1 | Trừu tượng hoá: thêm hàm `generate(step, ctx)` chọn **model** hoặc **template** theo cờ `CASAN_GEN_MODE` | `casan-step.mjs` | mode=template → hành vi cũ y hệt | ✅ đợt A |
|
||||
| 2.2 | Viết prompt-template cho mỗi step (đưa requirement + architecture + tiêu chí chấp nhận vào prompt) | mới `prompts/<step>.md` | prompt render đủ ngữ cảnh | có prompt từng step |
|
||||
| 2.3 | Gọi model qua `model-router.sh --role generate` (KHÔNG gọi model-call trực tiếp) | `casan-step.mjs` | output đi qua H4 trước khi ghi | ✅ đợt A (`01-srs`, `02-bd`) |
|
||||
| 2.4 | Chuẩn hoá output model → đúng file artifact + `STEP-RESULT` block | parser | verdict/artifacts hợp lệ | schema đúng |
|
||||
| 2.1 | Trừu tượng hoá: thêm hàm `generate(step, ctx)` chọn **model** hoặc **template** theo cờ `CASAN_GEN_MODE` | `casan-step.mjs` | mode=template → hành vi cũ y hệt | ✅ A–D |
|
||||
| 2.2 | Viết prompt-template cho mỗi step (đưa requirement + architecture + tiêu chí chấp nhận vào prompt) | `packages/casan-harness/prompts/sourcegen/<step>.md` | prompt render đủ ngữ cảnh | ✅ prompt từng step |
|
||||
| 2.3 | Gọi model qua `model-router.sh --role generate` (KHÔNG gọi model-call trực tiếp) | `casan-step.mjs` | output đi qua H4 trước khi ghi | ✅ A–D |
|
||||
| 2.4 | Chuẩn hoá output model → đúng file artifact + `STEP-RESULT` block | parser | verdict/artifacts hợp lệ | ✅ schema đúng |
|
||||
| 2.5 | Nạp **golden + tiêu chí** cho H3 judge từng step | `apps/okr/domain/golden-runs/` | judge chấm được đạt/không | H3 hoạt động |
|
||||
| 2.6 | Fallback: model SKIP/kém → dùng template (đợt A/B), hoặc REJECT → vòng review | `casan-step.mjs` | ép model lỗi → không vỡ | ✅ đợt A |
|
||||
| 2.7 | Chuyển đợt A (srs, bd) sang mode=model | pipeline | chạy full, 2 artifact do model sinh, qua harness | 🟡 MVP done; full live pipeline smoke còn |
|
||||
| 2.8 | Chuyển đợt B (spec, plan) + kiểm vòng REJECT hoạt động | pipeline | ép spec kém → STEP5 REJECT → retry | loop chạy |
|
||||
| 2.9 | Chuyển đợt C (dd, tasks) | pipeline | artifact hợp lệ, drift trong ngưỡng | đợt C xong |
|
||||
| 2.10 | Chuyển đợt D (implement code) — **bắt buộc** STEP12 chạy test thật làm cổng | pipeline | test dự án PASS mới nhận code | đợt D xong |
|
||||
| 2.11 | Ghi prompt + model + token vào audit (H5) & telemetry (H6) | logging | audit có prompt, provider-usage có token | truy vết được |
|
||||
| 2.6 | Fallback: model SKIP/kém → dùng template (đợt A/B), hoặc REJECT → vòng review | `casan-step.mjs`, `run-casan-pipeline.mjs` | ép model lỗi → không vỡ | ✅ A–D |
|
||||
| 2.7 | Chuyển đợt A (srs, bd) sang mode=model | pipeline/test | artifact do model sinh, qua harness | ✅ done+test |
|
||||
| 2.8 | Chuyển đợt B (spec, plan) + kiểm vòng REJECT hoạt động | pipeline/test | ép spec/plan kém → STEP5/7 REJECT → retry | ✅ loop chạy |
|
||||
| 2.9 | Chuyển đợt C (dd, testkit, tasks) | pipeline/test | artifact hợp lệ, drift trong ngưỡng | ✅ đợt C xong |
|
||||
| 2.10 | Chuyển đợt D (implement code) — **bắt buộc** STEP12 chạy test thật làm cổng | pipeline/test | test dự án PASS mới nhận code | ✅ đợt D xong |
|
||||
| 2.11 | Ghi prompt + model + token vào audit (H5) & telemetry (H6) | logging | audit có prompt, provider-usage có token | ✅ truy vết được |
|
||||
|
||||
---
|
||||
|
||||
@@ -88,10 +89,18 @@ flowchart LR
|
||||
| Regression pipeline | `CASAN_GEN_MODE=template` luôn giữ đường cũ |
|
||||
|
||||
## Tiêu chí HOÀN THÀNH
|
||||
- [x] `CASAN_GEN_MODE=template` cho hành vi cũ y hệt (an toàn quay lui) ở đợt A.
|
||||
- [ ] `CASAN_GEN_MODE=model`: đợt A–D artifact do LLM sinh, **đều qua H1→H7**. Đợt A đã có MVP cho `01-srs`/`02-bd`.
|
||||
- [ ] Code (đợt D) chỉ nhận khi STEP12 test PASS.
|
||||
- [ ] Prompt/model/token vào audit (H5) + telemetry (H6).
|
||||
- [ ] Có escalation khi H3 REJECT lặp; local vẫn là mặc định.
|
||||
|
||||
> Sau kế hoạch này, câu "pipeline sinh source bằng AI" mới **đúng nghĩa**. Trước đó phải nói rõ đang dùng **template**.
|
||||
- [x] `CASAN_GEN_MODE=template` cho hành vi cũ y hệt (an toàn quay lui).
|
||||
- [x] `CASAN_GEN_MODE=model`: đợt A–D artifact đi qua `model-router --role generate`, H4 scan, required-token validation, fallback template khi lỗi, và harness H1→H7 ở pipeline.
|
||||
- [x] Code/implement draft (đợt D) chỉ được accept khi STEP12 run-tests PASS (`implementation.accepted.json` chỉ ghi sau PASS).
|
||||
- [x] Prompt/model/token vào H5 sourcegen audit + H6 sourcegen telemetry.
|
||||
- [x] Có escalation khi H3 REJECT lặp; local vẫn là mặc định, cloud/frontier chỉ dùng khi cấu hình `CASAN_REVIEW_ESCALATE_MODEL` + key.
|
||||
|
||||
## Verify 2026-07-09
|
||||
- `phase2-sourcegen-tests.sh`: **10/0** — A–D model-mode mocked router, fallback H4, audit/telemetry, STEP12 pass/fail acceptance.
|
||||
- `run-casan-pipeline.mjs` full template smoke in `/tmp`: STEP1/2/3/5/6/7/8/8b/9/10/11 APPROVED, STEP12 PASS; loops STEP5→STEP3, STEP7→STEP6, STEP11→STEP10 exercised.
|
||||
- App test gate: backend `npm test -w backend` **46 pass / 0 fail / 3 skip**; frontend `npm test -w frontend` **16/0**.
|
||||
- Local Ollama generate path verified via `model-router.sh --role generate` (`ollama:ornith:9b`, real token telemetry); sourcegen prompts use 240s default timeout for long local generations.
|
||||
|
||||
> Sau kế hoạch này, câu "pipeline sinh source bằng AI" đúng với mode model/local.
|
||||
> Nếu cần OpenAI/Anthropic live-provider smoke và billing ground truth, làm tiếp
|
||||
> theo Plan-03 vì cần API key/dịch vụ ngoài.
|
||||
|
||||
@@ -48,10 +48,11 @@ Expected evidence:
|
||||
| P4 | Add model allowlist policy for approved cloud models | Unknown cloud model names require approval or block |
|
||||
| P5 | Decide default failover order for local -> cloud or cloud -> local | `model-fallback.sh` policy is explicit per role |
|
||||
|
||||
## Điều kiện gỡ blocker API key (để P02 đợt B–D không treo vô thời hạn)
|
||||
## Điều kiện gỡ blocker API key (để live provider smoke không treo vô thời hạn)
|
||||
|
||||
Live smoke (P1 ở trên) và Plan-02 đợt B–D **đều chờ cùng một điều kiện**: có API
|
||||
key provider thật. Ghi rõ để bất kỳ AI/người kế tiếp gỡ được:
|
||||
Live smoke (P1 ở trên) cần API key provider thật. Plan-02 A–D đã có đường
|
||||
offline/local qua `model-router`; phần còn lại ở đây chỉ là claim **live cloud
|
||||
provider thật**. Ghi rõ để bất kỳ AI/người kế tiếp gỡ được:
|
||||
|
||||
| Hạng mục | Quyết định cần chốt |
|
||||
|---|---|
|
||||
@@ -61,7 +62,7 @@ key provider thật. Ghi rõ để bất kỳ AI/người kế tiếp gỡ đư
|
||||
| **Evidence phải lưu** | 1 file smoke ghi: exit `0`, token thật > 0 trong `/tmp/out.json`, 1 dòng `provider-usage.jsonl` với source `*_api_real_tokens`, và xác nhận không rò secret. |
|
||||
| **Khi CHƯA có key** | Giữ nguyên trạng thái: deterministic offline test là bằng chứng hợp lệ; **không** claim live provider. Không tạo fake PASS/fake token. |
|
||||
|
||||
> **Chốt:** P02 đợt B–D có thể phát triển & test offline bằng deterministic mock;
|
||||
> **Chốt:** P02 A–D có thể phát triển & test offline bằng deterministic mock/local;
|
||||
> chỉ **bước xác nhận live** mới cần key. Vì vậy thiếu key **không chặn** việc
|
||||
> mở rộng source-gen — chỉ chặn claim "đã chạy live provider thật".
|
||||
|
||||
|
||||
@@ -13,7 +13,7 @@
|
||||
| Safe test order | `run-casan4` runs first because it rewrites `.specify/logs`; hardening suites run after | script order |
|
||||
| Per-suite timeout | `CASAN_CI_STEP_TIMEOUT_SEC` protects CI from hung model/security subprocesses | full gate attempt exposed a long A6 run; timeout wrapper added |
|
||||
| Filtered local verification | `CASAN_CI_SUITE_FILTER='phase2-sourcegen|phase10-traceability'` | `CI_GATE_SUMMARY PASS=2 FAIL=0 SKIP=15` |
|
||||
| Source-gen gate included | `phase2-sourcegen-tests.sh` included | sourcegen 3/0 |
|
||||
| Source-gen gate included | `phase2-sourcegen-tests.sh` included | sourcegen 10/0 |
|
||||
| Traceability gate included | `phase10-traceability-tests.sh` included | traceability 3/0 |
|
||||
| Frontend runtime tests included | `npm test -w frontend` when npm is present | frontend Vitest path |
|
||||
| Optional Docker local-prod lab | `CASAN_CI_RUN_INFRA_LAB=1` runs `phase-prod-infra-lab-tests.sh` only when Docker Compose is available | skip-aware by default |
|
||||
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating a CASAN SRS artifact.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Preserve FR-01 through FR-05 exactly.
|
||||
- Include Purpose, Scope, Functional Requirements, Non Functional Requirements, and Metrics.
|
||||
- Do not include secrets or policy-bypass instructions.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating a CASAN Business Design artifact.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Preserve SCR-00 through SCR-04 exactly.
|
||||
- Include Screen Layout and API Boundary sections.
|
||||
- Keep frontend/backend boundaries concrete.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating a CASAN feature specification.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Preserve FR-01 through FR-05 exactly.
|
||||
- Include Requirements, Acceptance Criteria, role-filtering, validation rules, and Source Trace.
|
||||
- Do not choose implementation details beyond what is required to make behavior testable.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating a CASAN implementation plan.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Include Backend Modules, Tests, Build, and any requested retry-loop criteria.
|
||||
- When asked for the final plan, include Golden regression test and Rollback strategy verbatim.
|
||||
- Keep the plan aligned to the existing NestJS/Prisma/SQLite/React/Vite stack.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating a CASAN detail design artifact.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Include Backend Design, Authorization, and Progress Calculation sections.
|
||||
- Reference concrete services/controllers only when grounded in the architecture.
|
||||
- Preserve role-boundary behavior for employee, manager, and admin.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating CASAN test cases.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Preserve TC-01 through TC-06 exactly.
|
||||
- Cover login, employee isolation, manager visibility, invalid objective payload, progress update, and golden drift.
|
||||
- Keep tests executable by the existing backend/frontend test stack.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,24 @@
|
||||
You are generating CASAN implementation tasks.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Include Backend, Frontend, tests, and Golden regression work.
|
||||
- Keep tasks traceable to the implementation plan and acceptance criteria.
|
||||
- Do not mark unrelated future work as complete.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -0,0 +1,25 @@
|
||||
You are generating a CASAN implementation draft.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
- Include Backend Source, Frontend Source, and Acceptance Gate sections.
|
||||
- Reference concrete source files and tests.
|
||||
- State clearly that this draft is not accepted until STEP12 test PASS.
|
||||
- Do not include shell commands that bypass harness policy.
|
||||
|
||||
Feature: {{featureId}}
|
||||
Module: {{moduleId}}
|
||||
Step: {{stepId}}
|
||||
Artifact: {{title}}
|
||||
|
||||
Extra instructions:
|
||||
{{extraInstructions}}
|
||||
|
||||
Requirement:
|
||||
{{requirement}}
|
||||
|
||||
Architecture:
|
||||
{{architecture}}
|
||||
|
||||
Reference structure:
|
||||
{{templateContent}}
|
||||
@@ -37,15 +37,49 @@ cat > "$WORK/.specify/scripts/bash/model-router.sh" <<'EOF'
|
||||
set -euo pipefail
|
||||
prompt_file="$1"
|
||||
out_json="$2"
|
||||
if grep -q "Business Design" "$prompt_file"; then
|
||||
cat > "$out_json" <<'JSON'
|
||||
{"text":"# Model Generated BD\n\n## Screen Layout\n- SCR-00 Login\n- SCR-01 Dashboard\n- SCR-02 Detail\n- SCR-03 Create Objective\n- SCR-04 Key Result Detail\n\n## API Boundary\nFrontend calls backend through src/lib/api.ts.\n","input_tokens":21,"output_tokens":34,"total_tokens":55}
|
||||
JSON
|
||||
else
|
||||
cat > "$out_json" <<'JSON'
|
||||
{"text":"# Model Generated SRS\n\n## Functional Requirements\n- FR-01 Login\n- FR-02 Create Objective\n- FR-03 Create Key Result\n- FR-04 Update Progress\n- FR-05 Dashboard\n\n## Non Functional Requirements\nAuthentication required.\n","input_tokens":20,"output_tokens":30,"total_tokens":50}
|
||||
JSON
|
||||
fi
|
||||
step="${CASAN_STEP_NAME:-unknown}"
|
||||
case "$step" in
|
||||
02-bd)
|
||||
text="# Model Generated BD\n\n## Screen Layout\n- SCR-00 Login\n- SCR-01 Dashboard\n- SCR-02 Detail\n- SCR-03 Create Objective\n- SCR-04 Key Result Detail\n\n## API Boundary\nFrontend calls backend through src/lib/api.ts.\n"
|
||||
;;
|
||||
03-spec)
|
||||
text="# Model Generated Spec\n\n## Requirements\n- FR-01 Login\n- FR-02 Create Objective\n- FR-03 Create Key Result\n- FR-04 Update Progress\n- FR-05 Dashboard\n\n## Acceptance Criteria\nEmployee isolation and manager visibility are testable.\n"
|
||||
;;
|
||||
05-plan)
|
||||
text="# Model Generated Plan\n\n## Backend Modules\nauth, users, objectives, key-results.\n\n## Tests\nBackend service tests and e2e tests.\n\n## Golden regression test\nCompare manager objective list with golden fixture.\n\n## Rollback strategy\nUse rollback-manager.sh checkpoints before overwrite.\n\n## Build\nRun npm test and npm run build.\n"
|
||||
;;
|
||||
07-dd)
|
||||
text="# Model Generated DD\n\n## Backend Design\nControllers call services and PrismaService.\n\n## Authorization\nEmployees only access owned objectives; managers/admins read all.\n\n## Progress Calculation\nProgress updates write ProgressUpdate and recalculate objective status.\n"
|
||||
;;
|
||||
08-testkit)
|
||||
text="# Model Generated Test Cases\n\n- TC-01 login rejects wrong password.\n- TC-02 employee list returns only own objectives.\n- TC-03 manager list returns all seeded objectives.\n- TC-04 invalid objective payload returns 400.\n- TC-05 progress patch updates a key result.\n- TC-06 golden manager objective response fails on drift.\n"
|
||||
;;
|
||||
09-tasks)
|
||||
text="# Model Generated Tasks\n\n- [X] Backend auth module with JWT and bcrypt.\n- [X] Frontend dashboard and detail pages.\n- [X] Backend and frontend tests.\n- [X] Golden regression evidence.\n"
|
||||
;;
|
||||
10-implement)
|
||||
text="# Model Generated Implementation Draft\n\n## Backend Source\n- backend/src/auth/auth.service.ts\n- backend/src/objectives/objectives.service.ts\n\n## Frontend Source\n- frontend/src/lib/api.ts\n- frontend/src/pages/DashboardPage.tsx\n\n## Acceptance Gate\nThis draft is not accepted until STEP12 test PASS.\n"
|
||||
;;
|
||||
*)
|
||||
text="# Model Generated SRS\n\n## Functional Requirements\n- FR-01 Login\n- FR-02 Create Objective\n- FR-03 Create Key Result\n- FR-04 Update Progress\n- FR-05 Dashboard\n\n## Non Functional Requirements\nAuthentication required.\n"
|
||||
;;
|
||||
esac
|
||||
python3 - "$out_json" "$text" <<'PY'
|
||||
import json, sys
|
||||
out, text = sys.argv[1], sys.argv[2]
|
||||
open(out, "w", encoding="utf-8").write(json.dumps({
|
||||
"timestamp": "2026-07-09T00:00:00Z",
|
||||
"text": text,
|
||||
"model_id": "mock:sourcegen",
|
||||
"role": "generate",
|
||||
"route": "mock:primary",
|
||||
"input_tokens": 21,
|
||||
"output_tokens": 34,
|
||||
"total_tokens": 55,
|
||||
"latency_ms": 1,
|
||||
"temperature": 0.2
|
||||
}) + "\n")
|
||||
PY
|
||||
EOF
|
||||
chmod +x "$WORK/.specify/scripts/bash/model-router.sh"
|
||||
|
||||
@@ -77,6 +111,89 @@ else
|
||||
fail "model generated BD was not accepted"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$WORK" || exit 1
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-spec.md" node scripts/casan-step.mjs 03-spec 1 >/dev/null
|
||||
)
|
||||
if grep -q "Model Generated Spec" "$WORK/docs/output/specs/001-okr-web-app/spec.md" \
|
||||
&& grep -q "source=model" "$WORK/docs/output/output_logs/001-okr-web-app/reports/03-spec-report.md"; then
|
||||
pass "source-gen wave B accepts model generated spec"
|
||||
else
|
||||
fail "model generated spec was not accepted"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$WORK" || exit 1
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-plan1.md" node scripts/casan-step.mjs 05-plan 1 >/dev/null
|
||||
CASAN_OUTPUT="$WORK/out-reviewplan1.md" node scripts/casan-step.mjs 06-reviewplan 1 >/dev/null
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-plan2.md" node scripts/casan-step.mjs 05-plan 2 >/dev/null
|
||||
CASAN_OUTPUT="$WORK/out-reviewplan2.md" node scripts/casan-step.mjs 06-reviewplan 2 >/dev/null
|
||||
)
|
||||
if grep -q "verdict: REJECTED" "$WORK/out-reviewplan1.md" \
|
||||
&& grep -q "verdict: APPROVED" "$WORK/out-reviewplan2.md" \
|
||||
&& grep -q "source=model" "$WORK/docs/output/output_logs/001-okr-web-app/reports/05-plan-report-attempt-2.md"; then
|
||||
pass "source-gen wave B preserves review-plan reject then retry loop"
|
||||
else
|
||||
fail "plan review loop did not reject then approve"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$WORK" || exit 1
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-dd.md" node scripts/casan-step.mjs 07-dd 1 >/dev/null
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-testkit.md" node scripts/casan-step.mjs 08-testkit 1 >/dev/null
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-tasks.md" node scripts/casan-step.mjs 09-tasks 1 >/dev/null
|
||||
)
|
||||
if grep -q "Model Generated DD" "$WORK/docs/output/ipa-docs/dd/dd-mod01-okr-management.md" \
|
||||
&& grep -q "Model Generated Test Cases" "$WORK/docs/output/ipa-docs/testcase/testcase-mod01-okr-management.md" \
|
||||
&& grep -q "Model Generated Tasks" "$WORK/docs/output/specs/001-okr-web-app/tasks.md"; then
|
||||
pass "source-gen wave C accepts model generated dd testkit and tasks"
|
||||
else
|
||||
fail "wave C model artifacts were not accepted"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$WORK" || exit 1
|
||||
CASAN_GEN_MODE=model CASAN_OUTPUT="$WORK/out-impl.md" node scripts/casan-step.mjs 10-implement 1 >/dev/null
|
||||
)
|
||||
if grep -q "Model Generated Implementation Draft" "$WORK/docs/output/specs/001-okr-web-app/implementation.md" \
|
||||
&& [[ ! -f "$WORK/docs/output/specs/001-okr-web-app/implementation.accepted.json" ]]; then
|
||||
pass "source-gen wave D creates implementation draft without accepting before tests"
|
||||
else
|
||||
fail "implementation draft acceptance gate was bypassed"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$WORK" || exit 1
|
||||
CASAN_SOURCEGEN_TEST_CMD='printf "unit tests pass\n"' CASAN_OUTPUT="$WORK/out-runtests-pass.md" node scripts/casan-step.mjs 12-runtests 1 >/dev/null
|
||||
)
|
||||
if grep -q "verdict: PASS" "$WORK/out-runtests-pass.md" \
|
||||
&& grep -q '"accepted": true' "$WORK/docs/output/specs/001-okr-web-app/implementation.accepted.json"; then
|
||||
pass "source-gen wave D accepts implementation only after STEP12 test pass"
|
||||
else
|
||||
fail "STEP12 did not accept passing implementation"
|
||||
fi
|
||||
|
||||
(
|
||||
cd "$WORK" || exit 1
|
||||
CASAN_SOURCEGEN_TEST_CMD='exit 9' CASAN_OUTPUT="$WORK/out-runtests-fail.md" node scripts/casan-step.mjs 12-runtests 1 >/dev/null
|
||||
)
|
||||
if grep -q "verdict: FAIL" "$WORK/out-runtests-fail.md" \
|
||||
&& [[ ! -f "$WORK/docs/output/specs/001-okr-web-app/implementation.accepted.json" ]]; then
|
||||
pass "source-gen wave D refuses implementation when STEP12 test fails"
|
||||
else
|
||||
fail "STEP12 accepted failing implementation"
|
||||
fi
|
||||
|
||||
if grep -q '"harness":"H5-sourcegen-audit"' "$WORK/.specify/logs/audit/sourcegen.jsonl" \
|
||||
&& grep -q '"prompt_sha256"' "$WORK/.specify/logs/audit/sourcegen.jsonl" \
|
||||
&& grep -q '"model_id":"mock:sourcegen"' "$WORK/.specify/logs/audit/sourcegen.jsonl" \
|
||||
&& grep -q '"harness":"H6-sourcegen-telemetry"' "$WORK/.specify/logs/level5/sourcegen-provider-usage.jsonl" \
|
||||
&& grep -q '"total_tokens":55' "$WORK/.specify/logs/level5/sourcegen-provider-usage.jsonl"; then
|
||||
pass "source-gen records prompt model and tokens into audit and telemetry"
|
||||
else
|
||||
fail "source-gen audit or telemetry is missing prompt/model/token evidence"
|
||||
fi
|
||||
|
||||
cat > "$WORK/.specify/scripts/bash/artifact-scan.sh" <<'EOF'
|
||||
#!/usr/bin/env bash
|
||||
exit 2
|
||||
|
||||
+269
-16
@@ -1,5 +1,6 @@
|
||||
import { spawnSync } from 'node:child_process';
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync, unlinkSync } from 'node:fs';
|
||||
import { createHash } from 'node:crypto';
|
||||
import { existsSync, mkdirSync, readFileSync, writeFileSync, unlinkSync, appendFileSync } from 'node:fs';
|
||||
import { tmpdir } from 'node:os';
|
||||
import { dirname, join } from 'node:path';
|
||||
import { fileURLToPath } from 'node:url';
|
||||
@@ -69,7 +70,8 @@ function judgeArtifact(filePath, criteria) {
|
||||
}
|
||||
|
||||
function renderSourcePrompt(stepId, title, templateContent, extraInstructions = '') {
|
||||
return `You are generating a CASAN SDLC artifact.
|
||||
const promptTemplatePath = join(__appRoot, 'packages', 'casan-harness', 'prompts', 'sourcegen', `${stepId}.md`);
|
||||
const defaultTemplate = `You are generating a CASAN SDLC artifact.
|
||||
|
||||
Rules:
|
||||
- Output markdown only.
|
||||
@@ -93,6 +95,75 @@ ${architecture.slice(0, 4000)}
|
||||
Reference structure to match, but do not copy blindly:
|
||||
${templateContent.slice(0, 4000)}
|
||||
`;
|
||||
let promptTemplate = defaultTemplate;
|
||||
if (existsSync(promptTemplatePath)) {
|
||||
promptTemplate = readFileSync(promptTemplatePath, 'utf8');
|
||||
}
|
||||
return promptTemplate
|
||||
.replaceAll('{{featureId}}', featureId)
|
||||
.replaceAll('{{moduleId}}', moduleId)
|
||||
.replaceAll('{{stepId}}', stepId)
|
||||
.replaceAll('{{title}}', title)
|
||||
.replaceAll('{{extraInstructions}}', extraInstructions)
|
||||
.replaceAll('{{requirement}}', requirement.slice(0, 6000))
|
||||
.replaceAll('{{architecture}}', architecture.slice(0, 4000))
|
||||
.replaceAll('{{templateContent}}', templateContent.slice(0, 4000));
|
||||
}
|
||||
|
||||
function sha256(text) {
|
||||
return createHash('sha256').update(text).digest('hex');
|
||||
}
|
||||
|
||||
function redactForAudit(text) {
|
||||
return String(text)
|
||||
.replace(/(sk-[A-Za-z0-9_-]{12,})/g, '***REDACTED_KEY***')
|
||||
.replace(/(Bearer\s+)[A-Za-z0-9._-]+/gi, '$1***REDACTED_TOKEN***')
|
||||
.slice(0, 2000);
|
||||
}
|
||||
|
||||
function sourcegenAuditPath() {
|
||||
return process.env.CASAN_SOURCEGEN_AUDIT_LOG || '.specify/logs/audit/sourcegen.jsonl';
|
||||
}
|
||||
|
||||
function sourcegenTelemetryPath() {
|
||||
return process.env.CASAN_SOURCEGEN_TELEMETRY_LOG || '.specify/logs/level5/sourcegen-provider-usage.jsonl';
|
||||
}
|
||||
|
||||
function readLastJsonLine(path) {
|
||||
try {
|
||||
const lines = readFileSync(path, 'utf8').trim().split('\n').filter(Boolean);
|
||||
return lines.length ? JSON.parse(lines[lines.length - 1]) : null;
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
function appendSourcegenAudit(record) {
|
||||
const path = sourcegenAuditPath();
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
const prev = readLastJsonLine(path);
|
||||
const base = {
|
||||
timestamp: new Date().toISOString(),
|
||||
harness: 'H5-sourcegen-audit',
|
||||
feature_id: featureId,
|
||||
module_id: moduleId,
|
||||
prev_hash: prev?.record_hash || 'GENESIS',
|
||||
...record,
|
||||
};
|
||||
const recordHash = sha256(JSON.stringify(base));
|
||||
appendFileSync(path, `${JSON.stringify({ ...base, record_hash: recordHash })}\n`, 'utf8');
|
||||
}
|
||||
|
||||
function appendSourcegenTelemetry(record) {
|
||||
const path = sourcegenTelemetryPath();
|
||||
mkdirSync(dirname(path), { recursive: true });
|
||||
appendFileSync(path, `${JSON.stringify({
|
||||
timestamp: new Date().toISOString(),
|
||||
harness: 'H6-sourcegen-telemetry',
|
||||
feature_id: featureId,
|
||||
module_id: moduleId,
|
||||
...record,
|
||||
})}\n`, 'utf8');
|
||||
}
|
||||
|
||||
function generateArtifact({ stepId, title, templateContent, required = [], extraInstructions = '' }) {
|
||||
@@ -105,24 +176,68 @@ function generateArtifact({ stepId, title, templateContent, required = [], extra
|
||||
const tmpOut = join(tmpdir(), `casan-generate-out-${uid}.json`);
|
||||
const tmpDraft = join(tmpdir(), `casan-generate-draft-${uid}.md`);
|
||||
try {
|
||||
writeFileSync(tmpPrompt, renderSourcePrompt(stepId, title, templateContent, extraInstructions), 'utf8');
|
||||
const prompt = renderSourcePrompt(stepId, title, templateContent, extraInstructions);
|
||||
const promptHash = sha256(prompt);
|
||||
writeFileSync(tmpPrompt, prompt, 'utf8');
|
||||
logDebug(`model call role=generate step=${stepId}`);
|
||||
const r = spawnSync(
|
||||
'bash',
|
||||
[join(SCRIPTS_DIR, 'model-router.sh'), tmpPrompt, tmpOut, '--role', 'generate'],
|
||||
{ timeout: 120000, encoding: 'utf8', env: { ...process.env, CASAN_STEP_NAME: stepId } },
|
||||
{
|
||||
timeout: Number(process.env.CASAN_SOURCEGEN_MODEL_TIMEOUT_MS || 300000),
|
||||
encoding: 'utf8',
|
||||
env: {
|
||||
...process.env,
|
||||
CASAN_STEP_NAME: stepId,
|
||||
CASAN_MODEL_TIMEOUT_SEC: process.env.CASAN_MODEL_TIMEOUT_SEC || process.env.CASAN_SOURCEGEN_MODEL_TIMEOUT_SEC || '240',
|
||||
},
|
||||
},
|
||||
);
|
||||
if (r.status !== 0) {
|
||||
appendSourcegenAudit({
|
||||
step_id: stepId,
|
||||
event: 'generate_fallback',
|
||||
reason: `model_generate_rc=${r.status}`,
|
||||
prompt_sha256: promptHash,
|
||||
prompt_text: redactForAudit(prompt),
|
||||
source: 'template-fallback',
|
||||
});
|
||||
return { content: templateContent, source: 'template-fallback', note: `model_generate_rc=${r.status}` };
|
||||
}
|
||||
|
||||
const d = JSON.parse(readFileSync(tmpOut, 'utf8'));
|
||||
const generated = String(d.text || '').trim();
|
||||
if (generated.length < 80) {
|
||||
appendSourcegenAudit({
|
||||
step_id: stepId,
|
||||
event: 'generate_fallback',
|
||||
reason: 'model_output_too_short',
|
||||
prompt_sha256: promptHash,
|
||||
prompt_text: redactForAudit(prompt),
|
||||
model_id: d.model_id,
|
||||
route: d.route,
|
||||
input_tokens: d.input_tokens,
|
||||
output_tokens: d.output_tokens,
|
||||
total_tokens: d.total_tokens,
|
||||
source: 'template-fallback',
|
||||
});
|
||||
return { content: templateContent, source: 'template-fallback', note: 'model_output_too_short' };
|
||||
}
|
||||
const missing = required.filter((token) => !generated.includes(token));
|
||||
if (missing.length > 0) {
|
||||
appendSourcegenAudit({
|
||||
step_id: stepId,
|
||||
event: 'generate_fallback',
|
||||
reason: `missing_required=${missing.join(',')}`,
|
||||
prompt_sha256: promptHash,
|
||||
prompt_text: redactForAudit(prompt),
|
||||
model_id: d.model_id,
|
||||
route: d.route,
|
||||
input_tokens: d.input_tokens,
|
||||
output_tokens: d.output_tokens,
|
||||
total_tokens: d.total_tokens,
|
||||
source: 'template-fallback',
|
||||
});
|
||||
return { content: templateContent, source: 'template-fallback', note: `missing_required=${missing.join(',')}` };
|
||||
}
|
||||
|
||||
@@ -133,8 +248,44 @@ function generateArtifact({ stepId, title, templateContent, required = [], extra
|
||||
{ timeout: 30000, encoding: 'utf8' },
|
||||
);
|
||||
if (scan.status !== 0) {
|
||||
appendSourcegenAudit({
|
||||
step_id: stepId,
|
||||
event: 'generate_fallback',
|
||||
reason: `artifact_scan_rc=${scan.status}`,
|
||||
prompt_sha256: promptHash,
|
||||
prompt_text: redactForAudit(prompt),
|
||||
model_id: d.model_id,
|
||||
route: d.route,
|
||||
input_tokens: d.input_tokens,
|
||||
output_tokens: d.output_tokens,
|
||||
total_tokens: d.total_tokens,
|
||||
source: 'template-fallback',
|
||||
});
|
||||
return { content: templateContent, source: 'template-fallback', note: `artifact_scan_rc=${scan.status}` };
|
||||
}
|
||||
appendSourcegenAudit({
|
||||
step_id: stepId,
|
||||
event: 'generate_accept',
|
||||
prompt_sha256: promptHash,
|
||||
prompt_text: redactForAudit(prompt),
|
||||
output_sha256: sha256(generated),
|
||||
model_id: d.model_id,
|
||||
route: d.route,
|
||||
input_tokens: d.input_tokens,
|
||||
output_tokens: d.output_tokens,
|
||||
total_tokens: d.total_tokens,
|
||||
source: 'model',
|
||||
});
|
||||
appendSourcegenTelemetry({
|
||||
provider: String(d.route || '').split(':')[0] || 'unknown',
|
||||
model: d.model_id || 'unknown',
|
||||
step: stepId,
|
||||
role: 'generate',
|
||||
input_tokens: Number(d.input_tokens || 0),
|
||||
output_tokens: Number(d.output_tokens || 0),
|
||||
total_tokens: Number(d.total_tokens || 0),
|
||||
status: 'success',
|
||||
});
|
||||
return { content: `${generated}\n`, source: 'model', note: `tokens=${d.total_tokens ?? '?'}` };
|
||||
} catch (err) {
|
||||
return { content: templateContent, source: 'template-fallback', note: `generate_error=${err?.name || 'Error'}` };
|
||||
@@ -222,6 +373,9 @@ const contractPath = `docs/output/specs/${featureId}/contracts/openapi.md`;
|
||||
const ddPath = `docs/output/ipa-docs/dd/dd-mod01-okr-management.md`;
|
||||
const testcasePath = `docs/output/ipa-docs/testcase/testcase-mod01-okr-management.md`;
|
||||
const tasksPath = `docs/output/specs/${featureId}/tasks.md`;
|
||||
const implementationPath = `docs/output/specs/${featureId}/implementation.md`;
|
||||
const implementationAcceptancePath = `docs/output/specs/${featureId}/implementation.accepted.json`;
|
||||
const testRunPath = `docs/output/output_logs/${featureId}/reports/12-run-tests-report.md`;
|
||||
const codeReviewPath = `docs/output/output_logs/${featureId}/reports/11-review-code-report.md`;
|
||||
let finalTitle = `# ${step}`;
|
||||
let finalBody = '';
|
||||
@@ -269,8 +423,16 @@ switch (step) {
|
||||
break;
|
||||
}
|
||||
case '03-spec': {
|
||||
write(specPath, `# Feature Specification: OKR Web App\n\n## Feature ID\n${featureId}\n\n## Requirements\n- FR-01: Login authenticates username/email plus password and returns standard envelope.\n- FR-02: Objective creation validates title, owner, and quarter.\n- FR-03: Key Result creation validates objective, title, values, deadline, and progress.\n- FR-04: Progress update accepts 0-100 and stores a ProgressUpdate record.\n- FR-05: Dashboard list filters by role: ADMIN and MANAGER see all; EMPLOYEE sees own objectives.\n\n## Acceptance Criteria\n- Employee cannot read or update another employee objective or key result.\n- Manager can read all seeded objectives.\n- Invalid quarter format returns validation error.\n- Golden objective response fails on drift.\n\n## Source Trace\nRequirement characters read: ${requirement.length}. Architecture characters read: ${architecture.length}.\n`);
|
||||
report(`docs/output/output_logs/${featureId}/reports/03-spec-report.md`, '# STEP 3: Specify Report', `Generated ${specPath}.`, 'APPROVED', [specPath]);
|
||||
const templateContent = `# Feature Specification: OKR Web App\n\n## Feature ID\n${featureId}\n\n## Requirements\n- FR-01: Login authenticates username/email plus password and returns standard envelope.\n- FR-02: Objective creation validates title, owner, and quarter.\n- FR-03: Key Result creation validates objective, title, values, deadline, and progress.\n- FR-04: Progress update accepts 0-100 and stores a ProgressUpdate record.\n- FR-05: Dashboard list filters by role: ADMIN and MANAGER see all; EMPLOYEE sees own objectives.\n\n## Acceptance Criteria\n- Employee cannot read or update another employee objective or key result.\n- Manager can read all seeded objectives.\n- Invalid quarter format returns validation error.\n- Golden objective response fails on drift.\n\n## Role-Based Filtering\n- ADMIN can read all objectives.\n- MANAGER can read all seeded objectives.\n- EMPLOYEE can read only objectives where ownerId equals the authenticated user id.\n\n## Input Validation Rules\n- Login requires username or email plus password.\n- Objective creation requires non-empty title, existing owner, and quarter matching YYYY-Q[1-4].\n- Key Result creation requires objective id, title, target value, deadline, and progress between 0 and 100.\n- Progress update rejects values below 0 or above 100.\n\n## Source Trace\nRequirement characters read: ${requirement.length}. Architecture characters read: ${architecture.length}.\n`;
|
||||
const generated = generateArtifact({
|
||||
stepId: '03-spec',
|
||||
title: 'Feature Specification',
|
||||
templateContent,
|
||||
required: ['FR-01', 'FR-02', 'FR-03', 'FR-04', 'FR-05', 'Acceptance Criteria'],
|
||||
extraInstructions: 'Create a feature specification with FR coverage, role filtering, validation rules, acceptance criteria, and source trace.',
|
||||
});
|
||||
write(specPath, generated.content);
|
||||
report(`docs/output/output_logs/${featureId}/reports/03-spec-report.md`, '# STEP 3: Specify Report', `Generated ${specPath}. source=${generated.source} note=${generated.note}.`, 'APPROVED', [specPath]);
|
||||
break;
|
||||
}
|
||||
case '04-reviewspec': {
|
||||
@@ -302,14 +464,24 @@ switch (step) {
|
||||
try { writeFileSync(CHECKPOINT_SIDECAR, txId, 'utf8'); } catch {}
|
||||
}
|
||||
}
|
||||
write(planPath, `# Implementation Plan: OKR Web App\n\n## Stack\nNestJS, Prisma Client, SQLite, React, Vite, Tailwind, Zod, TanStack Query.\n\n## Backend Modules\n- auth: JWT login and cookie issuance.\n- users: admin/manager user list.\n- objectives: role-filtered list, detail, create.\n- key-results: detail, create, progress update.\n\n## Tests\n- Backend service tests.\n- Backend HTTP e2e tests.\n${incomplete ? '- TODO: define golden regression and rollback strategy.\n' : '- Golden regression test compares seeded manager objectives with backend/test/golden/objectives.manager.json.\n- Rollback strategy restores changed artifacts from backups through rollback-manager.sh.\n'}\n## Build\nRun npm test and npm run build for backend and frontend.\n`);
|
||||
const templateContent = `# Implementation Plan: OKR Web App\n\n## Stack\nNestJS, Prisma Client, SQLite, React, Vite, Tailwind, Zod, TanStack Query.\n\n## Backend Modules\n- auth: JWT login and cookie issuance.\n- users: admin/manager user list.\n- objectives: role-filtered list, detail, create.\n- key-results: detail, create, progress update.\n\n## Tests\n- Backend service tests.\n- Backend HTTP e2e tests.\n${incomplete ? '- TODO: define golden regression and rollback strategy.\n' : '- Golden regression test compares seeded manager objectives with backend/test/golden/objectives.manager.json.\n- Rollback strategy restores changed artifacts from backups through rollback-manager.sh.\n'}\n## Build\nRun npm test and npm run build for backend and frontend.\n`;
|
||||
const generated = generateArtifact({
|
||||
stepId: '05-plan',
|
||||
title: 'Implementation Plan',
|
||||
templateContent,
|
||||
required: ['Backend Modules', 'Tests', ...(incomplete ? [] : ['Golden regression test', 'Rollback strategy'])],
|
||||
extraInstructions: incomplete
|
||||
? 'Create the first implementation plan draft. Leave companion artifacts for the retry loop so review-plan can reject missing artifacts.'
|
||||
: 'Create the final implementation plan with Backend Modules, Tests, Golden regression test, Rollback strategy, and build commands.',
|
||||
});
|
||||
write(planPath, generated.content);
|
||||
if (!incomplete) {
|
||||
write(dataModelPath, '# Data Model\n\nUser 1:N Objective. Objective 1:N KeyResult. KeyResult 1:N ProgressUpdate. Role/status are SQLite strings constrained in service/types.\n');
|
||||
write(researchPath, '# Research\n\nSQLite selected to satisfy no Docker/Postgres e2e. Prisma Client remains application ORM. Node built-in sqlite applies migration SQL because Prisma schema-engine push fails in this Node 24 local environment.\n');
|
||||
write(quickstartPath, '# Quickstart\n\n1. npm install\n2. npm run db:setup -w backend\n3. npm run seed -w backend\n4. npm run dev -w backend\n5. npm run dev -w frontend\n');
|
||||
write(contractPath, '# API Contract\n\nPOST /auth/login\nGET /objectives\nGET /objectives/:id\nPOST /objectives\nGET /key-results/:id\nPOST /key-results\nPATCH /key-results/:id/progress\n');
|
||||
}
|
||||
report(`docs/output/output_logs/${featureId}/reports/05-plan-report-attempt-${attempt}.md`, `# STEP 5: Plan Report Attempt ${attempt}`, `Generated ${planPath}.`, 'APPROVED', [planPath]);
|
||||
report(`docs/output/output_logs/${featureId}/reports/05-plan-report-attempt-${attempt}.md`, `# STEP 5: Plan Report Attempt ${attempt}`, `Generated ${planPath}. source=${generated.source} note=${generated.note}.`, 'APPROVED', [planPath]);
|
||||
break;
|
||||
}
|
||||
case '06-reviewplan': {
|
||||
@@ -355,21 +527,66 @@ switch (step) {
|
||||
break;
|
||||
}
|
||||
case '07-dd': {
|
||||
write(ddPath, `# DD-MOD-01 OKR Management\n\n## Backend Design\nControllers are thin and call AuthService, UsersService, ObjectivesService, KeyResultsService. PrismaService is the only database access layer.\n\n## Authorization\nJwtAuthGuard verifies Bearer/cookie token. Employees are constrained to ownerId == user.sub. Managers/Admins read all objectives.\n\n## Progress Calculation\nKeyResultsService updates progress in a transaction, writes ProgressUpdate, then recalculates objective status.\n`);
|
||||
report(`docs/output/output_logs/${featureId}/reports/07-dd-report.md`, '# STEP 7: Detail Design Report', `Generated ${ddPath}.`, 'APPROVED', [ddPath]);
|
||||
const templateContent = `# DD-MOD-01 OKR Management\n\n## Backend Design\nControllers are thin and call AuthService, UsersService, ObjectivesService, KeyResultsService. PrismaService is the only database access layer.\n\n## Authorization\nJwtAuthGuard verifies Bearer/cookie token. Employees are constrained to ownerId == user.sub. Managers/Admins read all objectives.\n\n## Progress Calculation\nKeyResultsService updates progress in a transaction, writes ProgressUpdate, then recalculates objective status.\n`;
|
||||
const generated = generateArtifact({
|
||||
stepId: '07-dd',
|
||||
title: 'Detail Design',
|
||||
templateContent,
|
||||
required: ['Backend Design', 'Authorization', 'Progress Calculation'],
|
||||
extraInstructions: 'Create a detailed design for backend modules, authorization, database access, and progress calculation.',
|
||||
});
|
||||
write(ddPath, generated.content);
|
||||
report(`docs/output/output_logs/${featureId}/reports/07-dd-report.md`, '# STEP 7: Detail Design Report', `Generated ${ddPath}. source=${generated.source} note=${generated.note}.`, 'APPROVED', [ddPath]);
|
||||
break;
|
||||
}
|
||||
case '08-testkit': {
|
||||
write(testcasePath, `# Test Cases MOD-01\n\n- TC-01 login rejects wrong password.\n- TC-02 employee list returns only own objectives.\n- TC-03 manager list returns all seeded objectives.\n- TC-04 invalid objective payload returns 400.\n- TC-05 progress patch updates a key result and stores progress.\n- TC-06 golden manager objective response fails on drift.\n`);
|
||||
report(`docs/output/output_logs/${featureId}/reports/08-testkit-report.md`, '# STEP 8: Testkit Report', `Generated ${testcasePath}.`, 'APPROVED', [testcasePath]);
|
||||
const templateContent = `# Test Cases MOD-01\n\n- TC-01 login rejects wrong password.\n- TC-02 employee list returns only own objectives.\n- TC-03 manager list returns all seeded objectives.\n- TC-04 invalid objective payload returns 400.\n- TC-05 progress patch updates a key result and stores progress.\n- TC-06 golden manager objective response fails on drift.\n`;
|
||||
const generated = generateArtifact({
|
||||
stepId: '08-testkit',
|
||||
title: 'Test Case Design',
|
||||
templateContent,
|
||||
required: ['TC-01', 'TC-02', 'TC-03', 'TC-04', 'TC-05', 'TC-06'],
|
||||
extraInstructions: 'Create test cases with TC IDs covering login, role-filtering, validation, progress update, and golden drift.',
|
||||
});
|
||||
write(testcasePath, generated.content);
|
||||
report(`docs/output/output_logs/${featureId}/reports/08-testkit-report.md`, '# STEP 8: Testkit Report', `Generated ${testcasePath}. source=${generated.source} note=${generated.note}.`, 'APPROVED', [testcasePath]);
|
||||
break;
|
||||
}
|
||||
case '09-tasks': {
|
||||
write(tasksPath, `# Tasks\n\n- [X] Backend auth module with JWT and bcrypt.\n- [X] Backend users/objectives/key-results modules.\n- [X] Prisma schema, SQLite migration SQL, idempotent seed.\n- [X] Frontend login, dashboard, objective detail, create objective, key result detail.\n- [X] Backend service and e2e tests.\n- [X] Golden regression fixture and deliberate failure evidence.\n- [X] Build/test logs captured.\n`);
|
||||
report(`docs/output/output_logs/${featureId}/reports/09-tasks-report.md`, '# STEP 9: Tasks Report', `Generated ${tasksPath}.`, 'APPROVED', [tasksPath]);
|
||||
const templateContent = `# Tasks\n\n- [X] Backend auth module with JWT and bcrypt.\n- [X] Backend users/objectives/key-results modules.\n- [X] Prisma schema, SQLite migration SQL, idempotent seed.\n- [X] Frontend login, dashboard, objective detail, create objective, key result detail.\n- [X] Backend service and e2e tests.\n- [X] Golden regression fixture and deliberate failure evidence.\n- [X] Build/test logs captured.\n`;
|
||||
const generated = generateArtifact({
|
||||
stepId: '09-tasks',
|
||||
title: 'Implementation Tasks',
|
||||
templateContent,
|
||||
required: ['Backend', 'Frontend', 'tests', 'Golden'],
|
||||
extraInstructions: 'Create a task list that keeps test work and implementation work traceable to the plan and acceptance criteria.',
|
||||
});
|
||||
write(tasksPath, generated.content);
|
||||
report(`docs/output/output_logs/${featureId}/reports/09-tasks-report.md`, '# STEP 9: Tasks Report', `Generated ${tasksPath}. source=${generated.source} note=${generated.note}.`, 'APPROVED', [tasksPath]);
|
||||
break;
|
||||
}
|
||||
case '10-reviewcode': {
|
||||
case '10-implement': {
|
||||
const templateContent = `# Implementation Draft: OKR Web App\n\n## Scope\nThe OKR implementation is represented by the existing backend and frontend source files. This step records the implementation candidate that must be accepted only after STEP12 run-tests passes.\n\n## Backend Source\n- backend/src/auth/auth.service.ts\n- backend/src/objectives/objectives.service.ts\n- backend/src/key-results/key-results.service.ts\n- backend/test/services.test.ts\n- backend/test/e2e.test.ts\n\n## Frontend Source\n- frontend/src/lib/api.ts\n- frontend/src/pages/DashboardPage.tsx\n- frontend/src/pages/ObjectiveDetailPage.tsx\n\n## Acceptance Gate\nThis draft is not accepted until STEP12 writes ${implementationAcceptancePath} after the real test command exits 0.\n`;
|
||||
const generated = generateArtifact({
|
||||
stepId: '10-implement',
|
||||
title: 'Implementation Draft',
|
||||
templateContent,
|
||||
required: ['Backend Source', 'Frontend Source', 'Acceptance Gate', 'STEP12'],
|
||||
extraInstructions: 'Create a code implementation draft/manifest. Reference concrete source files and state that acceptance requires STEP12 test PASS.',
|
||||
});
|
||||
write(implementationPath, generated.content);
|
||||
try { unlinkSync(implementationAcceptancePath); } catch {}
|
||||
report(
|
||||
`docs/output/output_logs/${featureId}/reports/10-implement-report.md`,
|
||||
'# STEP 10: Implementation Draft Report',
|
||||
`Generated ${implementationPath}. source=${generated.source} note=${generated.note}. acceptance=pending STEP12.`,
|
||||
'APPROVED',
|
||||
[implementationPath],
|
||||
);
|
||||
break;
|
||||
}
|
||||
case '10-reviewcode':
|
||||
case '11-reviewcode': {
|
||||
const missingFiles = backendFiles.filter((path) => {
|
||||
try {
|
||||
readFileSync(path, 'utf8');
|
||||
@@ -382,7 +599,7 @@ switch (step) {
|
||||
const issues = [];
|
||||
if (!serviceText.includes('PrismaService')) issues.push('PrismaService not used');
|
||||
if (!serviceText.includes('ForbiddenException')) issues.push('authorization exception not found');
|
||||
if (!serviceText.includes('golden objective list response does not drift')) issues.push('golden test not found');
|
||||
if (!/golden:.*objective list.*does not drift/i.test(serviceText)) issues.push('golden test not found');
|
||||
issues.push(...missingFiles.map((path) => `missing file: ${path}`));
|
||||
let verdict = issues.length === 0 ? 'APPROVED' : 'REJECTED';
|
||||
// WP-B: model judge gate
|
||||
@@ -402,6 +619,42 @@ switch (step) {
|
||||
);
|
||||
break;
|
||||
}
|
||||
case '12-runtests': {
|
||||
let implementation = '';
|
||||
try { implementation = readFileSync(implementationPath, 'utf8'); } catch {}
|
||||
const testCmd = process.env.CASAN_SOURCEGEN_TEST_CMD || 'npm test';
|
||||
const r = spawnSync('bash', ['-lc', testCmd], {
|
||||
cwd: __appRoot,
|
||||
encoding: 'utf8',
|
||||
timeout: Number(process.env.CASAN_SOURCEGEN_TEST_TIMEOUT_MS || 180000),
|
||||
env: { ...process.env },
|
||||
});
|
||||
const output = `${r.stdout || ''}\n${r.stderr || ''}`.trim().slice(0, 6000);
|
||||
const passed = r.status === 0;
|
||||
const acceptance = {
|
||||
timestamp: new Date().toISOString(),
|
||||
feature_id: featureId,
|
||||
implementation_artifact: implementationPath,
|
||||
implementation_sha256: sha256(implementation),
|
||||
test_command: testCmd,
|
||||
test_exit_code: r.status,
|
||||
accepted: passed,
|
||||
};
|
||||
if (passed) {
|
||||
write(implementationAcceptancePath, `${JSON.stringify(acceptance, null, 2)}\n`);
|
||||
} else {
|
||||
try { unlinkSync(implementationAcceptancePath); } catch {}
|
||||
}
|
||||
report(
|
||||
testRunPath,
|
||||
'# STEP 12: Run Tests Report',
|
||||
`Command: ${testCmd}\nExit code: ${r.status}\nAccepted implementation: ${passed ? 'yes' : 'no'}\n\n## Output\n${output || '(no output)'}`,
|
||||
passed ? 'PASS' : 'FAIL',
|
||||
passed ? [implementationPath, implementationAcceptancePath] : [implementationPath],
|
||||
passed ? [] : [`test command failed rc=${r.status}`],
|
||||
);
|
||||
break;
|
||||
}
|
||||
default:
|
||||
throw new Error(`Unknown step: ${step}`);
|
||||
}
|
||||
|
||||
@@ -40,7 +40,10 @@ const DIAGRAM_STEP = {
|
||||
'07-dd': 'STEP8',
|
||||
'08-testkit': 'STEP8b',
|
||||
'09-tasks': 'STEP9',
|
||||
'10-implement': 'STEP10',
|
||||
'10-reviewcode': 'STEP11',
|
||||
'11-reviewcode': 'STEP11',
|
||||
'12-runtests': 'STEP12',
|
||||
};
|
||||
|
||||
const FULL_DIAGRAM = [
|
||||
@@ -168,7 +171,7 @@ function logLoop(fromStep, verdict, toStep, note) {
|
||||
log('warn', 'boss', line);
|
||||
}
|
||||
|
||||
function runHarness({ id, agent, step, attempt = '1' }) {
|
||||
function runHarness({ id, agent, step, attempt = '1', extraEnv = {} }) {
|
||||
const diagram = DIAGRAM_STEP[step] ?? step;
|
||||
const input = `${casanDir}/${id}-input.txt`;
|
||||
const output = `${casanDir}/${id}-output.md`;
|
||||
@@ -192,6 +195,7 @@ function runHarness({ id, agent, step, attempt = '1' }) {
|
||||
CASAN_AGENT_NAME: agent,
|
||||
CASAN_STEP_NAME: id,
|
||||
CASAN_PHASE_REPORT: phaseReport,
|
||||
...extraEnv,
|
||||
},
|
||||
},
|
||||
);
|
||||
@@ -321,19 +325,37 @@ const sequence = [
|
||||
{ id: '01-srs', agent: 'okr.srs', step: '01-srs' },
|
||||
{ id: '02-bd', agent: 'okr.bd', step: '02-bd' },
|
||||
{ id: '03-spec', agent: 'speckit.specify', step: '03-spec' },
|
||||
{ id: '04-reviewspec', agent: 'okr.reviewspec', step: '04-reviewspec' },
|
||||
{ id: '05-plan-attempt-1', agent: 'speckit.plan', step: '05-plan', attempt: '1' },
|
||||
];
|
||||
|
||||
for (const item of sequence) {
|
||||
const { verdict } = runHarness(item);
|
||||
if (item.step === '04-reviewspec' && verdict === 'REJECTED') {
|
||||
// Diagram loop STEP5 -> STEP3. The current sequence expects APPROVED here;
|
||||
// if a rejection ever happens we surface the loop instead of hiding it.
|
||||
logLoop('STEP5', verdict, 'STEP3', 'review-spec rejected; sequence continues but needs attention');
|
||||
runHarness(item);
|
||||
}
|
||||
|
||||
let reviewSpec = runHarness({ id: '04-reviewspec', agent: 'okr.reviewspec', step: '04-reviewspec' });
|
||||
if (reviewSpec.verdict === 'REJECTED') {
|
||||
logLoop('STEP5', reviewSpec.verdict, 'STEP3', 'review-spec rejected; retrying spec with template fallback');
|
||||
runHarness({
|
||||
id: '04b-spec-attempt-2',
|
||||
agent: 'speckit.specify',
|
||||
step: '03-spec',
|
||||
attempt: '2',
|
||||
extraEnv: { CASAN_GEN_MODE: 'template' },
|
||||
});
|
||||
reviewSpec = runHarness({
|
||||
id: '04c-reviewspec-attempt-2',
|
||||
agent: 'okr.reviewspec',
|
||||
step: '04-reviewspec',
|
||||
attempt: '2',
|
||||
extraEnv: { CASAN_MODEL_PRIMARY: process.env.CASAN_REVIEW_ESCALATE_MODEL || 'openai:gpt-4o-mini' },
|
||||
});
|
||||
if (reviewSpec.verdict === 'REJECTED') {
|
||||
logLoop('STEP5', reviewSpec.verdict, 'STOP', 'review-spec rejected after fallback retry');
|
||||
throw new Error('review-spec rejected after fallback retry');
|
||||
}
|
||||
}
|
||||
|
||||
runHarness({ id: '05-plan-attempt-1', agent: 'speckit.plan', step: '05-plan', attempt: '1' });
|
||||
|
||||
const reviewPlan1 = runHarness({ id: '06-reviewplan-attempt-1', agent: 'okr.reviewplan', step: '06-reviewplan', attempt: '1' });
|
||||
if (reviewPlan1.verdict === 'REJECTED') {
|
||||
logLoop('STEP7', reviewPlan1.verdict, 'STEP6', 'BACK-TO-PLAN: retrying plan with missing criteria fixed');
|
||||
@@ -345,7 +367,7 @@ runHarness({ id: '07-plan-attempt-2', agent: 'speckit.plan', step: '05-plan', at
|
||||
const fallbackOut = `${casanDir}/model-fallback-output.txt`;
|
||||
log('debug', 'boss', `model-fallback invoked (real primary failure) → ${fallbackOut}`);
|
||||
execFileSync(
|
||||
`/model-fallback.sh`,
|
||||
`${HARNESS_BASH}/model-fallback.sh`,
|
||||
[
|
||||
fallbackOut,
|
||||
// Real primary failure: reading a nonexistent path exits non-zero (not a
|
||||
@@ -366,46 +388,66 @@ appendBoss(`Model fallback invoked; output ${fallbackOut}`);
|
||||
const driftCandidate = `${casanDir}/drift-plan-candidate.txt`;
|
||||
copyFileSync(fallbackOut, driftCandidate);
|
||||
log('debug', 'boss', 'drift-detect: fallback output vs golden baseline');
|
||||
execFileSync(`/drift-detect.sh`, [
|
||||
execFileSync(`${HARNESS_BASH}/drift-detect.sh`, [
|
||||
GOLDEN_PLAN,
|
||||
driftCandidate,
|
||||
'.specify/logs/level5/okr-plan-drift-report.json',
|
||||
], { cwd: root, stdio: 'inherit' });
|
||||
appendBoss('Drift detection invoked: fallback output vs golden baseline.');
|
||||
|
||||
const reviewPlan2 = runHarness({ id: '08-reviewplan-attempt-2', agent: 'okr.reviewplan', step: '06-reviewplan', attempt: '2' });
|
||||
const reviewPlan2 = runHarness({
|
||||
id: '08-reviewplan-attempt-2',
|
||||
agent: 'okr.reviewplan',
|
||||
step: '06-reviewplan',
|
||||
attempt: '2',
|
||||
extraEnv: { CASAN_MODEL_PRIMARY: process.env.CASAN_REVIEW_ESCALATE_MODEL || 'openai:gpt-4o-mini' },
|
||||
});
|
||||
if (reviewPlan2.verdict === 'REJECTED') {
|
||||
logLoop('STEP7', reviewPlan2.verdict, 'STEP6', 'BACK-TO-PLAN attempt 2 still rejected');
|
||||
throw new Error('review-plan rejected after retry');
|
||||
}
|
||||
runHarness({ id: '09-dd', agent: 'okr.dd', step: '07-dd' });
|
||||
runHarness({ id: '10-testkit', agent: 'okr.testkit', step: '08-testkit' });
|
||||
runHarness({ id: '11-tasks', agent: 'speckit.tasks', step: '09-tasks' });
|
||||
const reviewCode = runHarness({ id: '12-reviewcode', agent: 'okr.reviewcode', step: '10-reviewcode' });
|
||||
runHarness({ id: '12-implement', agent: 'speckit.implement', step: '10-implement' });
|
||||
const reviewCode = runHarness({ id: '13-reviewcode', agent: 'okr.reviewcode', step: '11-reviewcode' });
|
||||
if (reviewCode.verdict === 'REJECTED') {
|
||||
// Diagram loop STEP11 -> STEP10 (implement is not an agent step yet).
|
||||
logLoop('STEP11', reviewCode.verdict, 'STEP10', 'review-code rejected; implement step must be re-run');
|
||||
runHarness({ id: '13b-implement-attempt-2', agent: 'speckit.implement', step: '10-implement', attempt: '2' });
|
||||
const reviewCode2 = runHarness({
|
||||
id: '13c-reviewcode-attempt-2',
|
||||
agent: 'okr.reviewcode',
|
||||
step: '11-reviewcode',
|
||||
attempt: '2',
|
||||
extraEnv: { CASAN_MODEL_PRIMARY: process.env.CASAN_REVIEW_ESCALATE_MODEL || 'openai:gpt-4o-mini' },
|
||||
});
|
||||
if (reviewCode2.verdict === 'REJECTED') {
|
||||
logLoop('STEP11', reviewCode2.verdict, 'STOP', 'review-code rejected after implement retry');
|
||||
throw new Error('review-code rejected after implement retry');
|
||||
}
|
||||
}
|
||||
const runTests = runHarness({ id: '14-runtests', agent: 'okr.testkit run-tests', step: '12-runtests' });
|
||||
if (runTests.verdict === 'FAIL') {
|
||||
logLoop('STEP12', runTests.verdict, 'STEP6', 'tests FAIL: re-plan/re-implement before acceptance');
|
||||
}
|
||||
|
||||
const rollbackDir = 'docs/output/casan/app-evidence';
|
||||
mkdirSync(rollbackDir, { recursive: true });
|
||||
const rollbackTarget = `${rollbackDir}/rollback-target.txt`;
|
||||
const rollbackBackup = `${rollbackDir}/rollback-target.backup.txt`;
|
||||
writeFileSync(rollbackTarget, 'original pipeline rollback content\n', 'utf8');
|
||||
copyFileSync(rollbackTarget, rollbackBackup);
|
||||
writeFileSync(`${rollbackDir}/rollback-before.txt`, readFileSync(rollbackTarget, 'utf8'), 'utf8');
|
||||
writeFileSync(rollbackTarget, 'changed content that must be undone\n', 'utf8');
|
||||
writeFileSync(`${rollbackDir}/rollback-changed.txt`, readFileSync(rollbackTarget, 'utf8'), 'utf8');
|
||||
const record = execFileSync(`/rollback-manager.sh`, [
|
||||
'record',
|
||||
'restore rollback target evidence file',
|
||||
`cp ${rollbackBackup} ${rollbackTarget}`,
|
||||
const record = execFileSync(`${HARNESS_BASH}/rollback-manager.sh`, [
|
||||
'checkpoint',
|
||||
rollbackTarget,
|
||||
], { cwd: root, encoding: 'utf8' });
|
||||
writeFileSync(`${rollbackDir}/rollback-record.stdout`, record, 'utf8');
|
||||
const tx = record.match(/transaction_id=([^\s]+)/)?.[1];
|
||||
if (!tx) {
|
||||
throw new Error('rollback transaction id not found');
|
||||
}
|
||||
const execute = execFileSync(`/rollback-manager.sh`, ['execute', tx], {
|
||||
writeFileSync(rollbackTarget, 'changed content that must be undone\n', 'utf8');
|
||||
writeFileSync(`${rollbackDir}/rollback-changed.txt`, readFileSync(rollbackTarget, 'utf8'), 'utf8');
|
||||
const execute = execFileSync(`${HARNESS_BASH}/rollback-manager.sh`, ['execute', tx], {
|
||||
cwd: root,
|
||||
encoding: 'utf8',
|
||||
});
|
||||
|
||||
Reference in New Issue
Block a user