diff --git a/docs/plans/CASAN_BACKLOG_STATUS.md b/docs/plans/CASAN_BACKLOG_STATUS.md index 7b2a34f..a11bfa7 100644 --- a/docs/plans/CASAN_BACKLOG_STATUS.md +++ b/docs/plans/CASAN_BACKLOG_STATUS.md @@ -61,7 +61,7 @@ | Future B1–B6 | 💤 vision | `CASAN_PLAN_FUTURE_PHASES.md` — approval workflow nâng cao · state machine · model benchmark · governed memory · auto-remediation · platform KPI. | | **17 Loop Engineering** | � T1–T6 done+test (offline) | **Agentic Loop Governance** — đủ 5 primitive + orchestrator (97/0 WSL, nối CI). T1 **Governor** (`loop-governor.py`; deny-by-default, no/corrupt policy→strict/HALT, on_exceed halt/escalate) 15/0; T2 **Convergence** (`loop-convergence.py`; repeat/thrash→OSCILLATING, flat→STALLED, fail-closed) 15/0; T3 **Verify Contract** (`loop-gate.py`; H4→DENY, unmet→FAIL, correction bounded→ESCALATE, no self-declared DONE) 20/0; T4 **Trace/Replay** (`loop-trace.py`; append-only hash-linked, edited→BREAK, tampered artifact→replay DRIFT) 16/0; T5 **Meta-loop** (`loop-metaloop.py`; propose≠apply, SoD, loosen>org_ceiling refused, apply qua governed CP store→đổi thật ceiling + rollback) 15/0; T6 **Orchestrator** (`loop-run.sh`; gate→governor→convergence→trace/turn, secure-by-default opt-out, nén giữa vòng) 16/0. State qua `CASAN_LOOP_STATE_ROOT` (repo `.specify/state` sạch). **Còn (infra):** T4 KMS-anchor head (A7 Vault), T6 widget Command Center (17.22, C5), live H3-judge. Chi tiết: `CASAN_PLAN_17_LOOP_ENGINEERING.md`. | | **16 Security audit remediation** | � P0/P1/P2 phần lớn done+test | **Remediation đã thực thi:** 28 SEC suite (151/0 WSL, nối `ci-harness-gate.sh`). Done: SEC-01..10, 12, 13, **14** (model-digest bỏ env-override ở prod/strict), 15, 16..21, **22** (trusted-time JWT `exp` ARCH-06 + tag proposal nguồn-không-tin ARCH-08), **26** (stored/second-order injection scan), 27..30, **23 Phase 1–5 offline** (multi-tenant: tenant-store+guard · per-tenant CP/audit/telemetry · RBAC data-boundary · tenant kill-switch/quota · ký registry · crypt at-rest per-tenant), **24 offline** (image digest-pin + ký workflow), **25 offline** (artifact attestation tested==deployed); **SEC-11 gộp vào SEC-17** (`CASAN_PROFILE=prod` enforce-by-default). **Còn 📋 planned (hạ tầng/process):** SEC-22 ARCH-10 (attestation ngoài) · SEC-23 23.11 (crypt qua Vault Transit) · **SEC-24 còn** (live CVE/OSV + scan image thật — offline image-pin/ký-workflow đã done) · **SEC-25 còn** (signed-commit enrollment + SLSA chain — offline artifact-attestation đã done). Chi tiết: `CASAN_PLAN_16` §0a/§2d. | -| **18 Chat Console** | 🟡 **MVP-0 + MVP-1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 done** · target arch còn planned | **Governed Chat Console** (cắt lát MVP chống lan man). **MVP-0 Ask CASAN read-only DONE**: `prompt-mode-router.py`, `chat-readonly.py`, `chat-session.schema.json`, H4 input/output scan, H5 chat audit hash-chain, H6 token telemetry, answer kèm evidence sources. **MVP-1 Operator DONE**: `operator-actions.yaml`, `chat-operator.py`, `chat-turn.py`, registered actions `run tests`/`build pack`/`verify pack`, all through `action-gate`, no free-command, action artifacts with provenance. **MVP-2 Track 4 DONE**: `agent-registry.yaml`, `chat-agent-resolver.py`, `chat:select_agent` RBAC, delegation hold, tool allowlist BLOCK, Control Panel agent picker. **MVP-2 OPERATOR Track 5/6 DONE**: `chat-turn.py` creates UNCERTIFIED draft, runs `harness-preflight` + `context-assemble-scan` + Plan-17 `loop-run.sh` + trace verify/replay, then releases side-effect only after certification; denied draft does not execute action. Test: chat suites **38/0** (`phase-chat-prompt-router` 8/0, `phase-chat-readonly` 5/0, `phase-chat-session-audit` 3/0, `phase-chat-operator` 8/0, `phase-chat-agent-select` 8/0, `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0), Control Panel **25/0** + build xanh. Next: CODEGEN-as-loop + Track 8 replay/widget → MVP-3 multi-tenant. | +| **18 Chat Console** | 🟡 **MVP-0 + MVP-1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 + Track 8.1/8.2 done** · target arch còn planned | **Governed Chat Console** (cắt lát MVP chống lan man). **MVP-0 Ask CASAN read-only DONE**: `prompt-mode-router.py`, `chat-readonly.py`, `chat-session.schema.json`, H4 input/output scan, H5 chat audit hash-chain, H6 token telemetry, answer kèm evidence sources. **MVP-1 Operator DONE**: registered actions through `action-gate`, no free-command, action artifacts with provenance. **MVP-2 Track 4 DONE**: `agent-registry.yaml`, `chat-agent-resolver.py`, `chat:select_agent` RBAC, delegation hold, tool allowlist BLOCK, Control Panel agent picker. **MVP-2 OPERATOR Track 5/6 DONE**: `chat-turn.py` certifies UNCERTIFIED draft through Plan-17 `loop-run.sh` + trace verify/replay before side-effect release. **Track 8.1/8.2 DONE**: `chat-replay.py` verifies chat chain, evidence artifact hash, and OPERATOR loop replay; Control Panel `GET /api/v1/chat/replay`. Test: chat suites **42/0** (`phase-chat-prompt-router` 8/0, `phase-chat-readonly` 5/0, `phase-chat-session-audit` 3/0, `phase-chat-operator` 8/0, `phase-chat-agent-select` 8/0, `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0, `phase-chat-replay` 4/0), Control Panel **25/0** + build xanh. Next: CODEGEN-as-loop + Track 8.3/8.4 replay widget/approvals escalation → MVP-3 multi-tenant. | --- ## Trần điểm & điều kiện lên "Strong (81+)" diff --git a/docs/plans/CASAN_PLAN_00_INDEX.md b/docs/plans/CASAN_PLAN_00_INDEX.md index 88441cc..6ea5e37 100644 --- a/docs/plans/CASAN_PLAN_00_INDEX.md +++ b/docs/plans/CASAN_PLAN_00_INDEX.md @@ -35,7 +35,7 @@ | 15 | `CASAN_PLAN_15_RESPONSIBLE_AI_DATA_GOV.md` | Responsible AI & Data Governance (FPT §14.3–14.4) | 📋 | | 16 | `CASAN_PLAN_16_SECURITY_AUDIT_REMEDIATION.md` | Security audit core harness + kế hoạch vá (tamper-evidence/injection/fail-open) | � P0/P1/P2 done | | 17 | `CASAN_PLAN_17_LOOP_ENGINEERING.md` | Loop Engineering / Agentic Loop Governance (budget governor, convergence, verify-contract, loop trace/replay, meta-loop) | 📋 | -| 18 | `CASAN_PLAN_18_CHAT_CONSOLE.md` | Governed Chat Console (target arch; **MVP-0 Ask CASAN + MVP-1 Operator done; MVP-2 Track 4 + Operator Track 5/6 done** → replay/widget → multi-tenant) | 🟡 MVP-0/1 + Track 4/5/6 slice done | +| 18 | `CASAN_PLAN_18_CHAT_CONSOLE.md` | Governed Chat Console (target arch; **MVP-0 Ask CASAN + MVP-1 Operator done; MVP-2 Track 4/5/6 + Track 8.1/8.2 done** → widget/approvals → multi-tenant) | 🟡 MVP-0/1 + MVP-2 partial done | | Future | `CASAN_PLAN_FUTURE_PHASES.md` | Approval workflow, state machine, benchmark, memory, remediation, KPI | 💤 vision | > **Không có plan số 11:** số 11 được bỏ trống có chủ ý — nhánh eval/traceability đã diff --git a/docs/plans/CASAN_PLAN_18_CHAT_CONSOLE.md b/docs/plans/CASAN_PLAN_18_CHAT_CONSOLE.md index c1e99e9..be1a74b 100644 --- a/docs/plans/CASAN_PLAN_18_CHAT_CONSOLE.md +++ b/docs/plans/CASAN_PLAN_18_CHAT_CONSOLE.md @@ -286,8 +286,8 @@ flowchart TD ### Track 8 — Replay / Evidence / Command Center widgets `[MVP-2 → MVP-3]` | Task | Việc | File | Verify (WSL) | |---|---|---|---| -| 18.8.1 | **Replay** một turn (verify lại artifact đã ghi) phát hiện tamper/non-determinism | nối `loop-trace.py replay` | sửa artifact turn → replay lệch | -| 18.8.2 | `verify-chain` lịch sử chat (tamper-evidence, SEC-01/02, KMS khi TIER-2) | nối `loop-trace verify-chain` | sửa 1 turn → chain BREAK | +| 18.8.1 | ✅ **Replay** một turn (verify lại artifact đã ghi) phát hiện tamper/non-determinism | `chat-replay.py` + `loop-trace.py replay` | sửa artifact turn → replay lệch | +| 18.8.2 | ✅ `verify-chain` lịch sử chat (tamper-evidence, SEC-01/02; KMS khi TIER-2 là follow-up) | `chat-replay.py verify-chain` | sửa 1 turn → chain BREAK | | 18.8.3 | Widget **Chat/Loop** trên Command Center (13 §8.6): ticker per-iteration, budget gauge, click → Evidence drawer | nối Plan-13 §8.6 | số khớp fixture; click → evidence | | 18.8.4 | **Approvals/Escalation panel**: turn `ESCALATED` vào approvals inbox (13 §3.4), duyệt/từ chối (JWT + SoD + lý do) | nối Plan-13 §3.4 | escalate → pending; JWT giả → DENY | @@ -355,6 +355,12 @@ cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`. - [x] Tool output is scanned with `tool-output-scan.sh` before returning the operator result to chat. - [x] Control Panel response includes `loop_run`; `/chat` displays loop certification/replay status. Verify: `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0, `phase-chat-operator` 8/0, `console:test` 25/0, `console:build` xanh. +**MVP-2 Track 8.1/8.2 (Replay / verify-chain foundation):** +- [x] `chat-replay.py verify-chain` recomputes chat audit hash-chain and fails closed on edited records. +- [x] `chat-replay.py replay` verifies recorded evidence artifact hashes and replays OPERATOR loop traces via `loop-trace.py replay`. +- [x] Control Plane `GET /api/v1/chat/replay` wraps harness replay; no verdict reimplementation in NestJS. +- [x] Tampered operator evidence artifact → `DRIFT`; tampered chat audit → `BREAK`. Verify: `phase-chat-replay` 4/0, `console:test` 25/0, `console:build` xanh. + **Full (target architecture) — DoD tổng:** - [ ] Turn chạy **đúng như một loop-run Plan-17** (không định nghĩa vòng lặp riêng). - [ ] **Không đường vòng:** test chứng minh bỏ bất kỳ gate nào (preflight/verify) → FAIL. @@ -368,10 +374,11 @@ cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`. --- ## 7. Ghi chú trung thực & Non-goals (không lan man) -- **[MVP-0 + MVP-1 + MVP-2 Track 4 + Operator Track 5/6] — 🟡 done+test.** Ask CASAN - read-only, Operator registered actions, agent/skill governance foundation, và - OPERATOR chat-as-loop/draft-hold đã có harness CLI + Control Panel API/UI. Các - phase CODEGEN-as-loop / replay widget / multi-tenant production vẫn chưa mở. +- **[MVP-0 + MVP-1 + MVP-2 Track 4 + Operator Track 5/6 + Track 8.1/8.2] — 🟡 done+test.** + Ask CASAN read-only, Operator registered actions, agent/skill governance foundation, + OPERATOR chat-as-loop/draft-hold, và replay/verify-chain foundation đã có harness + CLI + Control Panel API. Các phase CODEGEN-as-loop / replay widget / approvals + escalation / multi-tenant production vẫn chưa mở. - **MVP-first (chống lan man):** **MVP-0 (Ask CASAN read-only) KHÔNG phụ thuộc Plan-17/ 14/SEC-23** — làm được ngay trên nền H4/H5/H6 hiện có. Chỉ **MVP-2 trở đi** (chat-as- loop + agent) mới cần Plan-17 (T1–T3) + Plan-14 RBAC; **MVP-3** mới cần SEC-23 tenant. diff --git a/docs/plans/CASAN_TEAM_ALLOCATION_3DEV.md b/docs/plans/CASAN_TEAM_ALLOCATION_3DEV.md index 90f343c..e075fce 100644 --- a/docs/plans/CASAN_TEAM_ALLOCATION_3DEV.md +++ b/docs/plans/CASAN_TEAM_ALLOCATION_3DEV.md @@ -77,7 +77,7 @@ approval JWT thật ([16 SEC-07](CASAN_PLAN_16_SECURITY_AUDIT_REMEDIATION.md)), |---|---|---|:--:|:--:|---| | B9 | ✅ **done** — **MVP-0 Ask CASAN read-only**: Prompt Router (Track 0) + Read-only Ask CASAN (Track 1) + session/audit (Track 2) + Control Panel API/UI (Track 7). H4 in/out, H5 audit hash-chain, H6 token, evidence sources; chat suites 14/0, Control Panel 23/0 + build xanh. Real model provider binding remains Track M follow-up when a provider is enabled. | [18 §1b, Track 0/1/2/7/M](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | ✅ | done | | B10 | ✅ **done** — **MVP-1 Operator mode**: registered actions `run tests` / `build pack` / `verify pack`, no free-command, all through `action-gate`, action artifacts with provenance, Control Panel quick actions. `phase-chat-operator` 8/0; total chat suites 24/0; Control Panel 24/0 + build xanh. | [18 Track 3](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | ✅ | done | -| B11 | 🟡 **partial done** — Track 4 Agent/Skill governance foundation + OPERATOR Track 5/6 chat-as-loop/draft-hold. `chat-turn.py` now certifies OPERATOR draft through Plan-17 `loop-run.sh` + trace verify/replay before releasing registered actions; denied drafts do not execute. Chat suites 38/0; Control Panel 25/0 + build xanh. **Remaining MVP-2:** CODEGEN-as-loop and Track 8 replay/widget. | [18 Track 4/5/6](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | 🔗 | B6, C7(RBAC), B10 | +| B11 | 🟡 **partial done** — Track 4 Agent/Skill governance + OPERATOR Track 5/6 chat-as-loop/draft-hold + Track 8.1/8.2 replay foundation. `chat-turn.py` certifies OPERATOR draft through Plan-17 `loop-run.sh`; `chat-replay.py` detects evidence artifact drift and chat-chain tamper. Chat suites 42/0; Control Panel 25/0 + build xanh. **Remaining MVP-2:** CODEGEN-as-loop and Track 8.3/8.4 replay widget/approvals escalation. | [18 Track 4/5/6/8](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | 🔗 | B6, C7(RBAC), B10 | | B12 | **Widget Loop/Chat** trên Command Center (loop ticker/budget/replay drawer) | [17 (17.22)](CASAN_PLAN_17_LOOP_ENGINEERING.md) · [18 Track 8](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | 🔗 | C5 ✅ baseline | | B13 | **MVP-3** multi-tenant chat hardening (Track 9) | [18 Track 9](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | 🔗 | A3(SEC-23) | diff --git a/packages/casan-control-panel/README.md b/packages/casan-control-panel/README.md index bfa8978..093bb69 100644 --- a/packages/casan-control-panel/README.md +++ b/packages/casan-control-panel/README.md @@ -68,6 +68,9 @@ Governed Chat (Plan-18 MVP-0/1 + MVP-2 Track 4 and Operator Track 5/6): with role visibility; selected agent/skill/delegation are bound by harness `chat-agent-resolver.py`. - `GET /api/v1/chat/audit/verify` — verifies the chat audit hash chain. +- `GET /api/v1/chat/replay?chatId=` — verifies chat-chain integrity, + evidence artifact hashes, and OPERATOR loop trace replay through harness + `chat-replay.py`. - `/chat` UI shows actor/role scope, `mode/risk/decision` badges, certified answer, evidence sources, registered operator actions, agent binding, loop certification, action-gate status, router details, and audit hash. Side-effect requests outside diff --git a/packages/casan-control-panel/backend/src/chat/chat.controller.ts b/packages/casan-control-panel/backend/src/chat/chat.controller.ts index 3fbe252..1cbfcd9 100644 --- a/packages/casan-control-panel/backend/src/chat/chat.controller.ts +++ b/packages/casan-control-panel/backend/src/chat/chat.controller.ts @@ -1,4 +1,4 @@ -import { Body, Controller, Get, Headers, Inject, Post } from '@nestjs/common'; +import { Body, Controller, Get, Headers, Inject, Post, Query } from '@nestjs/common'; import { ok } from '../common/api-response.js'; import { actorFromHeaders } from '../common/auth-context.js'; import { ChatAskInput, ChatService } from './chat.service.js'; @@ -17,6 +17,11 @@ export class ChatController { return ok(this.svc.verifyAudit()); } + @Get('replay') + replay(@Query('chatId') chatId?: string, @Query('turnId') turnId?: string) { + return ok(this.svc.replay(chatId || '', turnId || '')); + } + @Get('actions') actions(@Headers() headers: Record) { return ok(this.svc.listActions(actorFromHeaders(headers))); diff --git a/packages/casan-control-panel/backend/src/chat/chat.service.ts b/packages/casan-control-panel/backend/src/chat/chat.service.ts index 74a15e6..3155bbf 100644 --- a/packages/casan-control-panel/backend/src/chat/chat.service.ts +++ b/packages/casan-control-panel/backend/src/chat/chat.service.ts @@ -22,6 +22,7 @@ const HARNESS_BIN = join(APP_ROOT, 'packages', 'casan-harness', 'scripts', 'bash const CHAT_CLI = join(HARNESS_BIN, 'chat-turn.py'); const OPERATOR_CLI = join(HARNESS_BIN, 'chat-operator.py'); const AGENT_CLI = join(HARNESS_BIN, 'chat-agent-resolver.py'); +const REPLAY_CLI = join(HARNESS_BIN, 'chat-replay.py'); const RBAC_CLI = join(HARNESS_BIN, 'rbac-check.py'); function runPython(script: string, args: string[]): CommandResult { @@ -93,6 +94,16 @@ export class ChatService { return { ok: res.status === 0, output: res.stdout || res.stderr }; } + replay(chatId = '', turnId = '') { + const args = ['replay']; + if (chatId) args.push('--chat-id', chatId); + if (turnId) args.push('--turn-id', turnId); + const res = runPython(REPLAY_CLI, args); + const parsed = parseJson>(res.stdout); + if (parsed) return { ok: res.status === 0, ...parsed }; + throw new InternalServerErrorException(res.stderr || res.stdout || 'CHAT_REPLAY_FAILED'); + } + listActions(actor: SettingsActor) { this.requireRead(actor); const res = runPython(OPERATOR_CLI, ['list-actions']); diff --git a/packages/casan-control-panel/backend/test/chat.test.ts b/packages/casan-control-panel/backend/test/chat.test.ts index e66cc45..700d407 100644 --- a/packages/casan-control-panel/backend/test/chat.test.ts +++ b/packages/casan-control-panel/backend/test/chat.test.ts @@ -89,6 +89,10 @@ test('chat ask executes registered operator action through action-gate', () => { assert.equal(res.loop_run.side_effect_released, true); assert.equal(res.loop_run.trace_verify.ok, true); assert.equal(res.loop_run.replay.ok, true); + const replay = svc.replay('operator-chat') as any; + assert.equal(replay.ok, true); + assert.equal(replay.decision, 'MATCH'); + assert.equal(replay.loop_replayed, 1); assert.equal(res.audit_verify.ok, true); }); }); diff --git a/packages/casan-control-panel/frontend/src/lib/api.ts b/packages/casan-control-panel/frontend/src/lib/api.ts index e9aae33..089d775 100644 --- a/packages/casan-control-panel/frontend/src/lib/api.ts +++ b/packages/casan-control-panel/frontend/src/lib/api.ts @@ -157,6 +157,15 @@ export interface ChatAgent { allowed_for_role?: boolean; } +export interface ChatReplay { + ok: boolean; + decision: 'MATCH' | 'DRIFT' | 'BREAK' | string; + records: number; + loop_replayed?: number; + diffs?: any[]; + audit_path?: string; +} + export interface SettingsState { actor: SettingsActor; capabilities: { @@ -206,6 +215,7 @@ export const api = { askChat: (actor: SettingsActor, body: { message: string; chatId?: string; agentId?: string; skillId?: string; delegationLevel?: number }) => post('chat/ask', body, actorHeaders(actor)), verifyChatAudit: () => get<{ ok: boolean; output: string }>('chat/audit/verify'), + replayChat: (chatId = '', turnId = '') => get(`chat/replay?chatId=${encodeURIComponent(chatId)}&turnId=${encodeURIComponent(turnId)}`), chatActions: (actor: SettingsActor) => getWithHeaders<{ success: boolean; actions: ChatAction[] }>('chat/actions', actorHeaders(actor)), chatAgents: (actor: SettingsActor) => getWithHeaders<{ success: boolean; agents: ChatAgent[] }>('chat/agents', actorHeaders(actor)), }; diff --git a/packages/casan-harness/scripts/bash/chat-operator.py b/packages/casan-harness/scripts/bash/chat-operator.py index 488e238..3fbf262 100755 --- a/packages/casan-harness/scripts/bash/chat-operator.py +++ b/packages/casan-harness/scripts/bash/chat-operator.py @@ -69,6 +69,14 @@ def sha(text: str) -> str: return hashlib.sha256(text.encode("utf-8")).hexdigest() +def sha_file(path: str) -> str: + h = hashlib.sha256() + with open(path, "rb") as fh: + for chunk in iter(lambda: fh.read(65536), b""): + h.update(chunk) + return h.hexdigest() + + def git_commit() -> str: try: r = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, capture_output=True, text=True, timeout=5) @@ -204,6 +212,12 @@ def record_metrics(trace_id: str, message: str, answer: str, status: str, latenc def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, router, decision, answer, action=None, action_gate=None, artifact_path="", safe_message=""): elapsed = int((datetime.now(timezone.utc) - started).total_seconds() * 1000) + loop_run = {} + if os.environ.get("CASAN_CHAT_LOOP_RUN_JSON"): + try: + loop_run = json.loads(os.environ["CASAN_CHAT_LOOP_RUN_JSON"]) + except ValueError: + loop_run = {"success": False, "decision": "DENIED", "reason": "loop_run_json_invalid"} source = [] if artifact_path: source.append({ @@ -211,6 +225,7 @@ def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, route "line": 1, "excerpt": answer[:260], "score": 10 if decision == "ACTION_COMPLETED" else 1, + "hash": sha_file(artifact_path), "envelope": provenance("chat-operator-artifact", artifact_path, decision == "ACTION_COMPLETED"), }) rec = record_turn({ @@ -229,7 +244,8 @@ def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, route "user_msg_ref": sha(message), "safe_preview": (safe_message or "")[:180], "answer_ref": sha(answer), - "sources": [{"path": s.get("path"), "line": s.get("line")} for s in source], + "sources": [{"path": s.get("path"), "line": s.get("line"), "hash": s.get("hash")} for s in source], + "loop_run": loop_run, }) record_metrics(trace_id, safe_message or message, answer, "success" if decision == "ACTION_COMPLETED" else "failed", elapsed, (action or {}).get("id", "none")) return { @@ -247,6 +263,7 @@ def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, route "router": router, "action": {k: action.get(k) for k in ("id", "label", "description")} if action else None, "action_gate": action_gate or {}, + "loop_run": loop_run, } diff --git a/packages/casan-harness/scripts/bash/chat-replay.py b/packages/casan-harness/scripts/bash/chat-replay.py new file mode 100644 index 0000000..b39964f --- /dev/null +++ b/packages/casan-harness/scripts/bash/chat-replay.py @@ -0,0 +1,187 @@ +#!/usr/bin/env python3 +"""Plan-18 Track 8 chat replay / verify-chain. + +Verifies chat history after the fact: + * H5 chat turn hash-chain is intact, + * recorded evidence artifact hashes still match, + * OPERATOR loop-run traces still verify and replay through Plan-17 loop-trace. + +This complements chat-readonly.py verify-audit by validating artifact state, not +only the append-only JSONL chain. +""" +import argparse +import hashlib +import json +import os +import subprocess +import sys + + +GENESIS_HASH = "0" * 64 + + +def project_root() -> str: + d = os.path.abspath(os.path.dirname(__file__)) + p = d + while p != os.path.dirname(p): + if os.path.isdir(os.path.join(p, ".specify")) or os.path.isdir(os.path.join(p, "packages/casan-harness")): + return p + p = os.path.dirname(p) + return os.path.abspath(os.path.join(d, "..", "..", "..")) + + +ROOT = project_root() +BIN = os.path.join(ROOT, "packages", "casan-harness", "scripts", "bash") +LOOP_TRACE = os.path.join(BIN, "loop-trace.py") + + +def state_root() -> str: + return os.environ.get("CASAN_STATE_ROOT") or os.path.join(ROOT, ".specify") + + +def audit_path() -> str: + return os.environ.get("CASAN_CHAT_AUDIT_LOG") or os.path.join(state_root(), "logs", "chat", "chat-turns.jsonl") + + +def sha_text(text: str) -> str: + return hashlib.sha256(text.encode("utf-8")).hexdigest() + + +def sha_file(path: str) -> str: + h = hashlib.sha256() + with open(path, "rb") as fh: + for chunk in iter(lambda: fh.read(65536), b""): + h.update(chunk) + return h.hexdigest() + + +def load_records(): + records = [] + try: + with open(audit_path(), encoding="utf-8") as fh: + for line in fh: + if line.strip(): + records.append(json.loads(line)) + except OSError: + return [] + return records + + +def verify_chain(records): + prev = GENESIS_HASH + for idx, rec in enumerate(records, start=1): + got = rec.get("record_hash") + rest = {k: v for k, v in rec.items() if k != "record_hash"} + if rest.get("prev_hash") != prev or sha_text(json.dumps(rest, sort_keys=True, ensure_ascii=False)) != got: + return False, idx + prev = got + return True, None + + +def resolve_path(path: str) -> str: + if os.path.isabs(path): + return path + return os.path.join(ROOT, path) + + +def loop_state_root(loop_run): + explicit = os.environ.get("CASAN_LOOP_STATE_ROOT") + if explicit: + return explicit + artifact = loop_run.get("artifact") or "" + marker = os.sep + "logs" + os.sep + "chat" + os.sep + "loop-runs" + os.sep + if marker in artifact: + return artifact.split(marker, 1)[0] + os.sep + "logs" + os.sep + "chat" + os.sep + "loop-state" + return os.path.join(state_root(), "logs", "chat", "loop-state") + + +def run_loop_trace(loop_run, cmd: str): + env = {**os.environ, "CASAN_LOOP_STATE_ROOT": loop_state_root(loop_run)} + run_id = loop_run.get("run_id") + if not run_id: + return False, "missing_loop_run_id" + args = ["python3", LOOP_TRACE, cmd, "--run-id", run_id] + if cmd == "replay": + args += ["--profile", os.environ.get("CASAN_PROFILE", "dev")] + r = subprocess.run(args, cwd=ROOT, capture_output=True, text=True, env=env) + return r.returncode == 0, (r.stdout + r.stderr).strip() + + +def filter_records(records, chat_id="", turn_id=""): + out = records + if chat_id: + out = [r for r in out if r.get("chat_id") == chat_id] + if turn_id: + out = [r for r in out if r.get("turn_id") == turn_id] + return out + + +def replay(args) -> int: + records = load_records() + ok, broken_at = verify_chain(records) + if not ok: + print(json.dumps({"decision": "BREAK", "reason": "chat_chain_broken", "broken_at": broken_at}, ensure_ascii=False)) + return 3 + + selected = filter_records(records, args.chat_id, args.turn_id) + diffs = [] + replayed = 0 + for rec in selected: + for src in rec.get("sources", []): + expected = src.get("hash") + if not expected: + continue + path = resolve_path(src.get("path", "")) + if not os.path.isfile(path): + diffs.append({"seq": rec.get("seq"), "kind": "artifact_missing", "path": src.get("path")}) + continue + got = sha_file(path) + if got != expected: + diffs.append({"seq": rec.get("seq"), "kind": "artifact_hash_mismatch", "path": src.get("path"), "expected": expected, "got": got}) + + loop_run = rec.get("loop_run") or {} + if loop_run.get("run_id"): + replayed += 1 + trace_ok, trace_out = run_loop_trace(loop_run, "verify-chain") + replay_ok, replay_out = run_loop_trace(loop_run, "replay") + if not trace_ok: + diffs.append({"seq": rec.get("seq"), "kind": "loop_trace_break", "run_id": loop_run.get("run_id"), "output": trace_out[:400]}) + if not replay_ok: + diffs.append({"seq": rec.get("seq"), "kind": "loop_replay_drift", "run_id": loop_run.get("run_id"), "output": replay_out[:400]}) + + payload = { + "decision": "DRIFT" if diffs else "MATCH", + "records": len(selected), + "loop_replayed": replayed, + "diffs": diffs, + "audit_path": audit_path(), + } + print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True)) + return 3 if diffs else 0 + + +def verify(args) -> int: + records = load_records() + ok, broken_at = verify_chain(records) + payload = {"decision": "OK" if ok else "BREAK", "records": len(records), "audit_path": audit_path()} + if broken_at: + payload["broken_at"] = broken_at + print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True)) + return 0 if ok else 3 + + +def main() -> int: + ap = argparse.ArgumentParser() + sub = ap.add_subparsers(dest="cmd", required=True) + r = sub.add_parser("replay") + r.add_argument("--chat-id", default="") + r.add_argument("--turn-id", default="") + r.set_defaults(func=replay) + v = sub.add_parser("verify-chain") + v.set_defaults(func=verify) + args = ap.parse_args() + return args.func(args) + + +if __name__ == "__main__": + raise SystemExit(main()) diff --git a/packages/casan-harness/scripts/bash/chat-turn.py b/packages/casan-harness/scripts/bash/chat-turn.py index eacfa0e..008c62e 100755 --- a/packages/casan-harness/scripts/bash/chat-turn.py +++ b/packages/casan-harness/scripts/bash/chat-turn.py @@ -63,7 +63,10 @@ def scan_tool_output(text: str, label: str): def run_mode(args, binding, loop_run=None, scan_output_label=""): - r = subprocess.run(args, cwd=ROOT, capture_output=True, text=True) + env = os.environ.copy() + if loop_run is not None: + env["CASAN_CHAT_LOOP_RUN_JSON"] = json.dumps(loop_run, ensure_ascii=False, sort_keys=True) + r = subprocess.run(args, cwd=ROOT, capture_output=True, text=True, env=env) if scan_output_label: scan_rc, scan_msg = scan_tool_output(r.stdout + "\n" + r.stderr, scan_output_label) if scan_rc != 0: diff --git a/packages/casan-harness/scripts/bash/ci-harness-gate.sh b/packages/casan-harness/scripts/bash/ci-harness-gate.sh index ff91f6a..f5837d3 100755 --- a/packages/casan-harness/scripts/bash/ci-harness-gate.sh +++ b/packages/casan-harness/scripts/bash/ci-harness-gate.sh @@ -152,6 +152,7 @@ run "phase-chat-operator" bash "$TESTS/phase-chat-operator-tests.sh" run "phase-chat-agent-select" bash "$TESTS/phase-chat-agent-select-tests.sh" run "phase-chat-pipeline" bash "$TESTS/phase-chat-pipeline-tests.sh" run "phase-chat-stream-hold" bash "$TESTS/phase-chat-stream-hold-tests.sh" +run "phase-chat-replay" bash "$TESTS/phase-chat-replay-tests.sh" # ARCH-02: coverage cannot silently drop; ARCH-01: harness/policy cannot silently # drift. Both SKIP cleanly when no manifest is provisioned (non-strict dev/CI). diff --git a/packages/casan-harness/tests/phase-chat-replay-tests.sh b/packages/casan-harness/tests/phase-chat-replay-tests.sh new file mode 100644 index 0000000..4463ead --- /dev/null +++ b/packages/casan-harness/tests/phase-chat-replay-tests.sh @@ -0,0 +1,88 @@ +#!/usr/bin/env bash +set -uo pipefail + +# Plan-18 MVP-2 Track 8: chat replay verifies chat chain, evidence artifact hash, +# and Plan-17 loop trace replay for OPERATOR turns. + +SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +source "$SCRIPT_DIR/../scripts/bash/casan-paths.sh" +TURN="$CASAN_HARNESS_ROOT/scripts/bash/chat-turn.py" +REPLAY="$CASAN_HARNESS_ROOT/scripts/bash/chat-replay.py" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT +export CASAN_STATE_ROOT="$WORK/state" +export CASAN_LOOP_STATE_ROOT="$CASAN_STATE_ROOT/logs/chat/loop-state" + +PASS=0; FAIL=0 +pass() { echo "PASS: $1"; PASS=$((PASS + 1)); } +fail() { echo "FAIL: $1"; FAIL=$((FAIL + 1)); } + +echo "===== Plan-18 MVP-2 chat replay =====" + +python3 "$TURN" ask --message "run tests" --actor bob --role operator --chat-id replay-chat > "$WORK/turn.json" + +python3 "$REPLAY" verify-chain > "$WORK/verify.json" \ + && grep -q '"decision": "OK"' "$WORK/verify.json" \ + && pass "chat replay verifies chat audit chain" || fail "chat replay chain verify failed" + +python3 "$REPLAY" replay --chat-id replay-chat > "$WORK/replay.json" +python3 - "$WORK/replay.json" <<'PY' \ + && pass "chat replay matches clean operator turn" || fail "chat replay did not match clean turn" +import json, sys +d = json.load(open(sys.argv[1])) +assert d["decision"] == "MATCH" +assert d["records"] == 1 +assert d["loop_replayed"] == 1 +assert d["diffs"] == [] +PY + +ARTIFACT="$(python3 - "$WORK/turn.json" <<'PY' +import json, os, sys +d = json.load(open(sys.argv[1])) +print(d["sources"][0]["path"]) +PY +)" +case "$ARTIFACT" in + /*) ARTIFACT_PATH="$ARTIFACT" ;; + *) ARTIFACT_PATH="$CASAN_APP_ROOT/$ARTIFACT" ;; +esac +printf '\nTAMPERED\n' >> "$ARTIFACT_PATH" +set +e +python3 "$REPLAY" replay --chat-id replay-chat > "$WORK/tamper-artifact.json" +RC=$? +set -e 2>/dev/null || true +python3 - "$WORK/tamper-artifact.json" "$RC" <<'PY' \ + && pass "chat replay detects evidence artifact tamper" || fail "chat replay missed artifact tamper" +import json, sys +d = json.load(open(sys.argv[1])) +assert int(sys.argv[2]) == 3 +assert d["decision"] == "DRIFT" +assert any(x["kind"] == "artifact_hash_mismatch" for x in d["diffs"]) +PY + +# Fresh state for chain tamper so artifact drift above does not mask the chain test. +export CASAN_STATE_ROOT="$WORK/state-chain" +export CASAN_LOOP_STATE_ROOT="$CASAN_STATE_ROOT/logs/chat/loop-state" +python3 "$TURN" ask --message "run tests" --actor bob --role operator --chat-id chain-chat >/dev/null +AUDIT="$CASAN_STATE_ROOT/logs/chat/chat-turns.jsonl" +python3 - "$AUDIT" <<'PY' +import json, sys +rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()] +rows[0]["decision"] = "TAMPERED" +open(sys.argv[1], "w").write("\n".join(json.dumps(r) for r in rows) + "\n") +PY +set +e +python3 "$REPLAY" verify-chain > "$WORK/tamper-chain.json" +RC=$? +set -e 2>/dev/null || true +python3 - "$WORK/tamper-chain.json" "$RC" <<'PY' \ + && pass "chat replay verify-chain detects audit tamper" || fail "chat replay missed audit tamper" +import json, sys +d = json.load(open(sys.argv[1])) +assert int(sys.argv[2]) == 3 +assert d["decision"] == "BREAK" +PY + +echo "" +echo "===== CHAT REPLAY SUMMARY: PASS=$PASS FAIL=$FAIL =====" +[[ "$FAIL" -eq 0 ]] || exit 1 diff --git a/packages/casan-platform/README.md b/packages/casan-platform/README.md index 90234c9..3451c71 100644 --- a/packages/casan-platform/README.md +++ b/packages/casan-platform/README.md @@ -19,7 +19,7 @@ Optional layer for teams that want UI / dashboard / visibility. Packages: `casan | RBAC + approval inbox | ✅ local-prod done | Settings + kill-switch API RBAC enforcement, approval inbox/delegation/oversight, SoD, governed setting proposal apply, and local OIDC claim→role mapping smoke are done. Enterprise IdP rollout remains production follow-up. | | Evidence Pack Viewer | 📋 planned | reads `docs/output/casan/evidence-packs/` | | Attack Battery Viewer | 📋 planned | reads red-team corpus + H4 recall results | -| Read-only Ask CASAN + Operator + agent selection + operator loop-hold | 🟡 MVP-0/1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 done | Plan-18: `POST /api/v1/chat/ask`, `GET /api/v1/chat/actions`, `GET /api/v1/chat/agents` + `/chat`, backed by harness `chat-turn.py`, `chat-agent-resolver.py`, and Plan-17 `loop-run.sh` | +| Read-only Ask CASAN + Operator + agent selection + operator loop-hold + replay | 🟡 MVP-0/1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 + Track 8.1/8.2 done | Plan-18: `POST /api/v1/chat/ask`, `GET /api/v1/chat/actions`, `GET /api/v1/chat/agents`, `GET /api/v1/chat/replay` + `/chat`, backed by harness `chat-turn.py`, `chat-agent-resolver.py`, `chat-replay.py`, and Plan-17 `loop-run.sh` | | Gitea webhook integration | 📋 planned | trigger gate / publish evidence on push | ## Build (preview) @@ -29,6 +29,6 @@ scripts/package-release.sh platform # → dist/casan-platform-preview-vX.Y.Z The bundle includes a `PREVIEW-INCOMPLETE.txt` marker. Do not treat it as a finished product. ## To implement later -Start from Plan-15 RAI view or Plan-18 MVP-2 Track 8 (replay/widget) or CODEGEN-as-loop. Keep new numbers +Start from Plan-15 RAI view or Plan-18 MVP-2 Track 8.3/8.4 (replay widget / approvals escalation) or CODEGEN-as-loop. Keep new numbers evidence-backed with provenance; any write/governed action must continue to route through harness RBAC, approval, and audit primitives.