feat: certify codegen chat drafts through loop
This commit is contained in:
thanhnv
@@ -61,7 +61,7 @@
|
||||
| Future B1–B6 | 💤 vision | `CASAN_PLAN_FUTURE_PHASES.md` — approval workflow nâng cao · state machine · model benchmark · governed memory · auto-remediation · platform KPI. |
|
||||
| **17 Loop Engineering** | � T1–T6 done+test (offline) | **Agentic Loop Governance** — đủ 5 primitive + orchestrator (97/0 WSL, nối CI). T1 **Governor** (`loop-governor.py`; deny-by-default, no/corrupt policy→strict/HALT, on_exceed halt/escalate) 15/0; T2 **Convergence** (`loop-convergence.py`; repeat/thrash→OSCILLATING, flat→STALLED, fail-closed) 15/0; T3 **Verify Contract** (`loop-gate.py`; H4→DENY, unmet→FAIL, correction bounded→ESCALATE, no self-declared DONE) 20/0; T4 **Trace/Replay** (`loop-trace.py`; append-only hash-linked, edited→BREAK, tampered artifact→replay DRIFT) 16/0; T5 **Meta-loop** (`loop-metaloop.py`; propose≠apply, SoD, loosen>org_ceiling refused, apply qua governed CP store→đổi thật ceiling + rollback) 15/0; T6 **Orchestrator** (`loop-run.sh`; gate→governor→convergence→trace/turn, secure-by-default opt-out, nén giữa vòng) 16/0. State qua `CASAN_LOOP_STATE_ROOT` (repo `.specify/state` sạch). **Còn (infra):** T4 KMS-anchor head (A7 Vault), T6 widget Command Center (17.22, C5), live H3-judge. Chi tiết: `CASAN_PLAN_17_LOOP_ENGINEERING.md`. |
|
||||
| **16 Security audit remediation** | � P0/P1/P2 phần lớn done+test | **Remediation đã thực thi:** 28 SEC suite (151/0 WSL, nối `ci-harness-gate.sh`). Done: SEC-01..10, 12, 13, **14** (model-digest bỏ env-override ở prod/strict), 15, 16..21, **22** (trusted-time JWT `exp` ARCH-06 + tag proposal nguồn-không-tin ARCH-08), **26** (stored/second-order injection scan), 27..30, **23 Phase 1–5 offline** (multi-tenant: tenant-store+guard · per-tenant CP/audit/telemetry · RBAC data-boundary · tenant kill-switch/quota · ký registry · crypt at-rest per-tenant), **24 offline** (image digest-pin + ký workflow), **25 offline** (artifact attestation tested==deployed); **SEC-11 gộp vào SEC-17** (`CASAN_PROFILE=prod` enforce-by-default). **Còn 📋 planned (hạ tầng/process):** SEC-22 ARCH-10 (attestation ngoài) · SEC-23 23.11 (crypt qua Vault Transit) · **SEC-24 còn** (live CVE/OSV + scan image thật — offline image-pin/ký-workflow đã done) · **SEC-25 còn** (signed-commit enrollment + SLSA chain — offline artifact-attestation đã done). Chi tiết: `CASAN_PLAN_16` §0a/§2d. |
|
||||
| **18 Chat Console** | 🟡 **MVP-0 + MVP-1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 + Track 8.1/8.2/8.3/8.4 done** · target arch còn planned | **Governed Chat Console** (cắt lát MVP chống lan man). **MVP-0 Ask CASAN read-only DONE**: `prompt-mode-router.py`, `chat-readonly.py`, `chat-session.schema.json`, H4 input/output scan, H5 chat audit hash-chain, H6 token telemetry, answer kèm evidence sources. **MVP-1 Operator DONE**: registered actions through `action-gate`, no free-command, action artifacts with provenance. **MVP-2 Track 4 DONE**: `agent-registry.yaml`, `chat-agent-resolver.py`, `chat:select_agent` RBAC, delegation hold, tool allowlist BLOCK, Control Panel agent picker. **MVP-2 OPERATOR Track 5/6 DONE**: `chat-turn.py` certifies UNCERTIFIED draft through Plan-17 `loop-run.sh` + trace verify/replay before side-effect release. **Track 8.1/8.2 DONE**: `chat-replay.py` verifies chat chain, evidence artifact hash, and OPERATOR loop replay; Control Panel `GET /api/v1/chat/replay`. **Track 8.3 DONE**: Command Center `chat_loop` widget reads chat audit/replay evidence, loop ticker, budget gauge, and click-through evidence drawer. **Track 8.4 DONE**: `REQUIRES_APPROVAL` chat turns become pending `chat.escalate` approvals, SoD/reason enforced, strict/fake JWT denied before mutation. Test: chat suites **46/0** (`phase-chat-prompt-router` 8/0, `phase-chat-readonly` 5/0, `phase-chat-session-audit` 3/0, `phase-chat-operator` 8/0, `phase-chat-agent-select` 8/0, `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0, `phase-chat-replay` 4/0, `phase-chat-approval` 4/0), Control Panel **28/0** + build xanh. Next: CODEGEN-as-loop → MVP-3 multi-tenant. |
|
||||
| **18 Chat Console** | 🟡 **MVP-0 + MVP-1 + MVP-2 done+test; MVP-3 multi-tenant planned** · target arch còn planned | **Governed Chat Console** (cắt lát MVP chống lan man). **MVP-0 Ask CASAN read-only DONE**: `prompt-mode-router.py`, `chat-readonly.py`, `chat-session.schema.json`, H4 input/output scan, H5 chat audit hash-chain, H6 token telemetry, answer kèm evidence sources. **MVP-1 Operator DONE**: registered actions through `action-gate`, no free-command, action artifacts with provenance. **MVP-2 Track 4 DONE**: `agent-registry.yaml`, `chat-agent-resolver.py`, `chat:select_agent` RBAC, delegation hold, tool allowlist BLOCK, Control Panel agent picker, CODEGEN draft-only through `artifact-scan` + Plan-17 loop certification. **MVP-2 OPERATOR Track 5/6 DONE**: `chat-turn.py` certifies UNCERTIFIED draft through Plan-17 `loop-run.sh` + trace verify/replay before side-effect release. **Track 8.1/8.2 DONE**: `chat-replay.py` verifies chat chain, evidence artifact hash, and OPERATOR/CODEGEN loop replay; Control Panel `GET /api/v1/chat/replay`. **Track 8.3 DONE**: Command Center `chat_loop` widget reads chat audit/replay evidence, loop ticker, budget gauge, and click-through evidence drawer. **Track 8.4 DONE**: `REQUIRES_APPROVAL` chat turns become pending `chat.escalate` approvals, SoD/reason enforced, strict/fake JWT denied before mutation. Test: chat suites **51/0** (`phase-chat-prompt-router` 9/0, `phase-chat-readonly` 5/0, `phase-chat-session-audit` 3/0, `phase-chat-operator` 8/0, `phase-chat-agent-select` 8/0, `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0, `phase-chat-replay` 4/0, `phase-chat-approval` 4/0, `phase-chat-codegen` 4/0), Control Panel **28/0** + build xanh. Next: MVP-3 multi-tenant. |
|
||||
---
|
||||
|
||||
## Trần điểm & điều kiện lên "Strong (81+)"
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
| 15 | `CASAN_PLAN_15_RESPONSIBLE_AI_DATA_GOV.md` | Responsible AI & Data Governance (FPT §14.3–14.4) | 📋 |
|
||||
| 16 | `CASAN_PLAN_16_SECURITY_AUDIT_REMEDIATION.md` | Security audit core harness + kế hoạch vá (tamper-evidence/injection/fail-open) | � P0/P1/P2 done |
|
||||
| 17 | `CASAN_PLAN_17_LOOP_ENGINEERING.md` | Loop Engineering / Agentic Loop Governance (budget governor, convergence, verify-contract, loop trace/replay, meta-loop) | 📋 |
|
||||
| 18 | `CASAN_PLAN_18_CHAT_CONSOLE.md` | Governed Chat Console (target arch; **MVP-0 Ask CASAN + MVP-1 Operator done; MVP-2 Track 4/5/6 + Track 8.1/8.2/8.3/8.4 done** → CODEGEN → multi-tenant) | 🟡 MVP-0/1 + MVP-2 partial done |
|
||||
| 18 | `CASAN_PLAN_18_CHAT_CONSOLE.md` | Governed Chat Console (target arch; **MVP-0 Ask CASAN + MVP-1 Operator + MVP-2 Chat-as-Loop/Agent/Replay done** → multi-tenant) | 🟡 MVP-0/1/2 done |
|
||||
| Future | `CASAN_PLAN_FUTURE_PHASES.md` | Approval workflow, state machine, benchmark, memory, remediation, KPI | 💤 vision |
|
||||
|
||||
> **Không có plan số 11:** số 11 được bỏ trống có chủ ý — nhánh eval/traceability đã
|
||||
|
||||
@@ -23,11 +23,12 @@ package "DONE - Plan 13 Control Panel" #E8F5E9 {
|
||||
component "Command Center baseline\n9 evidence-backed widgets\nbriefing + ticker + evidence drawer" as command_done #D5F5E3
|
||||
component "Loop/Chat widget\nbudget + replay drawer + chat ticker\nextends Command Center contract" as loop_widgets_done #D5F5E3
|
||||
component "Approvals escalation\nturn ESCALATED -> inbox\nJWT + SoD + reason" as approvals_done #D5F5E3
|
||||
component "CODEGEN-as-loop\nH4 guarded codegen\nloop-certified artifacts" as codegen_done #D5F5E3
|
||||
}
|
||||
|
||||
package "NEXT - offline platform follow-ups" #FFF8E1 {
|
||||
component "RAI / Data Gov view\nretention + model-card + PII report" as rai #F9E79F
|
||||
component "CODEGEN-as-loop\nH4 guarded codegen\nloop-certified artifacts" as codegen_loop #F9E79F
|
||||
component "Chat multi-tenant hardening\nper-tenant state + quotas + crypto" as chat_mt #F9E79F
|
||||
}
|
||||
|
||||
package "EXTERNAL / MANAGED PROD" #FDEDEC {
|
||||
@@ -44,10 +45,11 @@ api_done --> command_done : GET /api/v1/command
|
||||
command_done --> gov_core : provenance + audit state
|
||||
loop_widgets_done --> command_done : specialized widget
|
||||
approvals_done --> gov_core : proposals + SoD
|
||||
codegen_done --> gov_core : guarded loop-run
|
||||
|
||||
lead --> rai : next if staying offline
|
||||
rai --> gov_core : read RAI + policy artifacts
|
||||
codegen_loop --> gov_core : guarded loop-run
|
||||
chat_mt --> gov_core : tenant boundaries
|
||||
|
||||
managed_oidc ..> local_prod : replace mock IdP/cert
|
||||
alerts ..> gov_core : route real incidents
|
||||
@@ -55,7 +57,7 @@ billing ..> command_done : ground-truth FinOps
|
||||
|
||||
note right of rai
|
||||
Recommended next:
|
||||
1. If no external infra: build RAI view or Plan-18 CODEGEN slice.
|
||||
1. If no external infra: build RAI view or Plan-18 tenant hardening.
|
||||
2. If infra is available: promote local-prod OIDC to managed prod.
|
||||
3. Keep chat widgets on the existing Command envelope.
|
||||
4. Keep every metric evidence-backed with provenance.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# KẾ HOẠCH 18 — Governed Chat Console (Chat-as-Loop qua Control Plane)
|
||||
|
||||
> Status 2026-07-08: **🟡 MVP-0 + MVP-1 done+test; MVP-2 Track 4 + Operator Track 5/6 + Track 8.1/8.2/8.3/8.4 done — target architecture còn planned.**
|
||||
> Status 2026-07-08: **🟡 MVP-0 + MVP-1 + MVP-2 done+test; MVP-3 multi-tenant còn planned.**
|
||||
> Đã implement **Ask CASAN — Read-only Evidence Assistant** qua harness + Control
|
||||
> Panel (`/api/v1/chat/ask`, `/chat` UI): Prompt Router deterministic
|
||||
> `READ_ONLY/OPERATOR/BLOCK/NOT_SUPPORTED`, context whitelist, H4 scan in/out, H5
|
||||
@@ -11,7 +11,8 @@
|
||||
> gate chọn agent, delegation hold, tool allowlist block, Control Panel agent picker.
|
||||
> **Operator Track 5/6** đã bind mỗi side-effect turn vào Plan-17 `loop-run.sh`,
|
||||
> verify trace/replay, H4 preflight/context/tool-output scan, và chỉ release action
|
||||
> sau khi draft được certify. **Track 8.1/8.2/8.3/8.4** đã có replay/verify-chain
|
||||
> sau khi draft được certify. **CODEGEN-as-loop** đã có draft-only artifact scan +
|
||||
> loop certification. **Track 8.1/8.2/8.3/8.4** đã có replay/verify-chain
|
||||
> foundation, Command Center Chat/Loop widget đọc chat audit/replay evidence thật,
|
||||
> và escalation vào approvals inbox.
|
||||
> Mục tiêu
|
||||
@@ -261,7 +262,7 @@ flowchart TD
|
||||
| 18.4.1 | ✅ `agent-registry.yaml` versioned (tool_allowlist, max_delegation, roles_allowed, model_role) + resolver bind theo turn | `packages/casan-harness/config/agent-registry.yaml` + `chat-agent-resolver.py` | chọn agent → allowlist đúng |
|
||||
| 18.4.2 | ✅ RBAC gate chọn agent (14): role không đủ → DENY; vượt `max_delegation` → security-sensitive → approval hold | nối `rbac-check.py` (`chat:select_agent`) | role thấp chọn agent tự chủ → DENY |
|
||||
| 18.4.3 | ✅ Tool-call runtime chỉ tool **trong allowlist**; ngoài → BLOCK trước runtime | `chat-agent-resolver.py` | tool ngoài allowlist → BLOCK |
|
||||
| 18.4.4 | ⏳ Skill code-gen chạy trong allowlist + H4 artifact-scan (Plan-02) trước output | nối `artifact-scan.sh` | skill sinh mã có injection → BLOCK |
|
||||
| 18.4.4 | ✅ Skill code-gen chạy trong allowlist + H4 artifact-scan (Plan-02) trước output | `chat-turn.py` + `artifact-scan.sh` | skill sinh mã có injection → BLOCK |
|
||||
|
||||
### Track 5 — Chat-as-loop pipeline `[MVP-2]`
|
||||
| Task | Việc | File | Verify (WSL) |
|
||||
@@ -282,7 +283,7 @@ flowchart TD
|
||||
|---|---|---|---|
|
||||
| 18.7.1 | ✅ Chat API (NestJS) **bọc** harness (MVP-0/1: read-only + operator; single-source, không verdict riêng) | `packages/casan-control-panel/backend/src/chat/*` | API gọi `chat-turn.py`; `console:test` xanh |
|
||||
| 18.7.2 | ✅ Chat panel React + hiển thị **evidence sources**, registered actions, action-gate + badge `mode/risk` | `packages/casan-control-panel/frontend/src/pages/Chat.tsx` | `console:build` xanh |
|
||||
| 18.7.3 | 🟡 (MVP-2 partial) agent/skill picker theo RBAC + delegation-level + loop certification badges đã có; còn nút **loop-breaker** và replay drawer khi Track 8 mở | Control Plane UI | picker hiện locked agent, backend DENY agent ngoài quyền; OPERATOR hiện `loop_run` |
|
||||
| 18.7.3 | 🟡 agent/skill picker theo RBAC + delegation-level + loop certification badges + replay/evidence surfaces đã có; nút **loop-breaker** trực tiếp vẫn là target UI follow-up | Control Plane UI | picker hiện locked agent, backend DENY agent ngoài quyền; OPERATOR/CODEGEN hiện `loop_run` |
|
||||
| 18.7.4 | Fail-loud: API/telemetry chết → UI `STALE/503` (tái dùng D3 `/healthz`) | nối Plan-07 D3 | ngắt backend → UI báo stale |
|
||||
|
||||
### Track 8 — Replay / Evidence / Command Center widgets `[MVP-2 → MVP-3]`
|
||||
@@ -323,7 +324,8 @@ flowchart TD
|
||||
`phase-chat-readonly-tests.sh`, `phase-chat-session-audit-tests.sh`; MVP-1 —
|
||||
`phase-chat-operator-tests.sh`; MVP-2 — `phase-chat-agent-select-tests.sh`,
|
||||
`phase-chat-pipeline-tests.sh`, `phase-chat-stream-hold-tests.sh`,
|
||||
`phase-chat-replay-tests.sh`, `phase-chat-approval-tests.sh`; MVP-3 —
|
||||
`phase-chat-replay-tests.sh`, `phase-chat-approval-tests.sh`,
|
||||
`phase-chat-codegen-tests.sh`; MVP-3 —
|
||||
`phase-chat-tenant-tests.sh`. Chạy **WSL**
|
||||
(deterministic, mock model; nhánh live cần Ollama SKIP-aware). Nối `ci-harness-gate.sh`;
|
||||
cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`.
|
||||
@@ -351,6 +353,7 @@ cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`.
|
||||
- [x] RBAC gate `chat:select_agent`; role thấp chọn agent tự chủ → DENY; cross-tenant → DENY.
|
||||
- [x] Delegation vượt agent max → `REQUIRES_APPROVAL`; tool ngoài allowlist → BLOCK trước runtime.
|
||||
- [x] Control Panel `GET /api/v1/chat/agents`, `/chat` agent/skill/delegation picker, response có `agent_binding`. Verify: `phase-chat-agent-select` 8/0, RBAC 12/0, SEC-23 RBAC 6/0, `console:test` 25/0, `console:build` xanh.
|
||||
- [x] CODEGEN draft path binds `codegen-draft`/`sourcegen-draft`, proves `artifact-scan` + `tool-output-scan` in allowlist, blocks injected generated artifacts, and certifies draft-only artifact through Plan-17 loop-run. Verify: `phase-chat-codegen` 4/0, `phase-chat-prompt-router` 9/0.
|
||||
|
||||
**MVP-2 Track 5/6 OPERATOR slice (Chat-as-loop + draft-hold):**
|
||||
- [x] OPERATOR turns create an `UNCERTIFIED` draft artifact, run `harness-preflight`, `context-assemble-scan`, `loop-run.sh`, `loop-trace verify-chain`, and `loop-trace replay` before action release.
|
||||
@@ -389,11 +392,11 @@ cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`.
|
||||
---
|
||||
|
||||
## 7. Ghi chú trung thực & Non-goals (không lan man)
|
||||
- **[MVP-0 + MVP-1 + MVP-2 Track 4 + Operator Track 5/6 + Track 8.1/8.2/8.3/8.4] — 🟡 done+test.**
|
||||
- **[MVP-0 + MVP-1 + MVP-2] — 🟡 done+test.**
|
||||
Ask CASAN read-only, Operator registered actions, agent/skill governance foundation,
|
||||
OPERATOR chat-as-loop/draft-hold, replay/verify-chain foundation, và Command Center
|
||||
Chat/Loop widget, approvals escalation đã có harness CLI + Control Panel API/UI. Các
|
||||
phase CODEGEN-as-loop / multi-tenant production vẫn chưa mở.
|
||||
CODEGEN draft-as-loop, OPERATOR chat-as-loop/draft-hold, replay/verify-chain foundation,
|
||||
Command Center Chat/Loop widget, approvals escalation đã có harness CLI + Control Panel
|
||||
API/UI. Phase multi-tenant production vẫn chưa mở.
|
||||
- **MVP-first (chống lan man):** **MVP-0 (Ask CASAN read-only) KHÔNG phụ thuộc Plan-17/
|
||||
14/SEC-23** — làm được ngay trên nền H4/H5/H6 hiện có. Chỉ **MVP-2 trở đi** (chat-as-
|
||||
loop + agent) mới cần Plan-17 (T1–T3) + Plan-14 RBAC; **MVP-3** mới cần SEC-23 tenant.
|
||||
|
||||
@@ -77,7 +77,7 @@ approval JWT thật ([16 SEC-07](CASAN_PLAN_16_SECURITY_AUDIT_REMEDIATION.md)),
|
||||
|---|---|---|:--:|:--:|---|
|
||||
| B9 | ✅ **done** — **MVP-0 Ask CASAN read-only**: Prompt Router (Track 0) + Read-only Ask CASAN (Track 1) + session/audit (Track 2) + Control Panel API/UI (Track 7). H4 in/out, H5 audit hash-chain, H6 token, evidence sources; chat suites 14/0, Control Panel 23/0 + build xanh. Real model provider binding remains Track M follow-up when a provider is enabled. | [18 §1b, Track 0/1/2/7/M](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | ✅ | done |
|
||||
| B10 | ✅ **done** — **MVP-1 Operator mode**: registered actions `run tests` / `build pack` / `verify pack`, no free-command, all through `action-gate`, action artifacts with provenance, Control Panel quick actions. `phase-chat-operator` 8/0; total chat suites 24/0; Control Panel 24/0 + build xanh. | [18 Track 3](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | ✅ | done |
|
||||
| B11 | 🟡 **partial done** — Track 4 Agent/Skill governance + OPERATOR Track 5/6 chat-as-loop/draft-hold + Track 8.1/8.2 replay foundation + Track 8.3 Command Center Chat/Loop widget + Track 8.4 approvals escalation. `chat-turn.py` certifies OPERATOR draft through Plan-17 `loop-run.sh`; `chat-replay.py` detects evidence artifact drift and chat-chain tamper; `/command` shows chat loop ticker/budget/evidence; delegation escalation creates pending `chat.escalate` approvals. Chat suites 46/0; Control Panel 28/0 + build xanh. **Remaining MVP-2:** CODEGEN-as-loop. | [18 Track 4/5/6/8](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | 🔗 | B6, C7(RBAC), B10 |
|
||||
| B11 | ✅ **MVP-2 done** — Track 4 Agent/Skill governance + CODEGEN draft-as-loop + OPERATOR Track 5/6 chat-as-loop/draft-hold + Track 8.1/8.2 replay foundation + Track 8.3 Command Center Chat/Loop widget + Track 8.4 approvals escalation. `chat-turn.py` certifies OPERATOR/CODEGEN drafts through Plan-17 `loop-run.sh`; `chat-replay.py` detects evidence artifact drift and chat-chain tamper; `/command` shows chat loop ticker/budget/evidence; delegation escalation creates pending `chat.escalate` approvals. Chat suites 51/0; Control Panel 28/0 + build xanh. | [18 Track 4/5/6/8](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | ✅ | B6, C7(RBAC), B10 |
|
||||
| B12 | ✅ **done** — Widget Loop/Chat trên Command Center (`chat_loop` widget, loop ticker, budget gauge, replay evidence drawer) | [17 (17.22)](CASAN_PLAN_17_LOOP_ENGINEERING.md) · [18 Track 8](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | ✅ | C5 ✅ baseline |
|
||||
| B13 | **MVP-3** multi-tenant chat hardening (Track 9) | [18 Track 9](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | 🔗 | A3(SEC-23) |
|
||||
|
||||
|
||||
Reference in New Issue
Block a user