feat: add chat replay verification
This commit is contained in:
thanhnv
@@ -61,7 +61,7 @@
|
||||
| Future B1–B6 | 💤 vision | `CASAN_PLAN_FUTURE_PHASES.md` — approval workflow nâng cao · state machine · model benchmark · governed memory · auto-remediation · platform KPI. |
|
||||
| **17 Loop Engineering** | � T1–T6 done+test (offline) | **Agentic Loop Governance** — đủ 5 primitive + orchestrator (97/0 WSL, nối CI). T1 **Governor** (`loop-governor.py`; deny-by-default, no/corrupt policy→strict/HALT, on_exceed halt/escalate) 15/0; T2 **Convergence** (`loop-convergence.py`; repeat/thrash→OSCILLATING, flat→STALLED, fail-closed) 15/0; T3 **Verify Contract** (`loop-gate.py`; H4→DENY, unmet→FAIL, correction bounded→ESCALATE, no self-declared DONE) 20/0; T4 **Trace/Replay** (`loop-trace.py`; append-only hash-linked, edited→BREAK, tampered artifact→replay DRIFT) 16/0; T5 **Meta-loop** (`loop-metaloop.py`; propose≠apply, SoD, loosen>org_ceiling refused, apply qua governed CP store→đổi thật ceiling + rollback) 15/0; T6 **Orchestrator** (`loop-run.sh`; gate→governor→convergence→trace/turn, secure-by-default opt-out, nén giữa vòng) 16/0. State qua `CASAN_LOOP_STATE_ROOT` (repo `.specify/state` sạch). **Còn (infra):** T4 KMS-anchor head (A7 Vault), T6 widget Command Center (17.22, C5), live H3-judge. Chi tiết: `CASAN_PLAN_17_LOOP_ENGINEERING.md`. |
|
||||
| **16 Security audit remediation** | � P0/P1/P2 phần lớn done+test | **Remediation đã thực thi:** 28 SEC suite (151/0 WSL, nối `ci-harness-gate.sh`). Done: SEC-01..10, 12, 13, **14** (model-digest bỏ env-override ở prod/strict), 15, 16..21, **22** (trusted-time JWT `exp` ARCH-06 + tag proposal nguồn-không-tin ARCH-08), **26** (stored/second-order injection scan), 27..30, **23 Phase 1–5 offline** (multi-tenant: tenant-store+guard · per-tenant CP/audit/telemetry · RBAC data-boundary · tenant kill-switch/quota · ký registry · crypt at-rest per-tenant), **24 offline** (image digest-pin + ký workflow), **25 offline** (artifact attestation tested==deployed); **SEC-11 gộp vào SEC-17** (`CASAN_PROFILE=prod` enforce-by-default). **Còn 📋 planned (hạ tầng/process):** SEC-22 ARCH-10 (attestation ngoài) · SEC-23 23.11 (crypt qua Vault Transit) · **SEC-24 còn** (live CVE/OSV + scan image thật — offline image-pin/ký-workflow đã done) · **SEC-25 còn** (signed-commit enrollment + SLSA chain — offline artifact-attestation đã done). Chi tiết: `CASAN_PLAN_16` §0a/§2d. |
|
||||
| **18 Chat Console** | 🟡 **MVP-0 + MVP-1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 done** · target arch còn planned | **Governed Chat Console** (cắt lát MVP chống lan man). **MVP-0 Ask CASAN read-only DONE**: `prompt-mode-router.py`, `chat-readonly.py`, `chat-session.schema.json`, H4 input/output scan, H5 chat audit hash-chain, H6 token telemetry, answer kèm evidence sources. **MVP-1 Operator DONE**: `operator-actions.yaml`, `chat-operator.py`, `chat-turn.py`, registered actions `run tests`/`build pack`/`verify pack`, all through `action-gate`, no free-command, action artifacts with provenance. **MVP-2 Track 4 DONE**: `agent-registry.yaml`, `chat-agent-resolver.py`, `chat:select_agent` RBAC, delegation hold, tool allowlist BLOCK, Control Panel agent picker. **MVP-2 OPERATOR Track 5/6 DONE**: `chat-turn.py` creates UNCERTIFIED draft, runs `harness-preflight` + `context-assemble-scan` + Plan-17 `loop-run.sh` + trace verify/replay, then releases side-effect only after certification; denied draft does not execute action. Test: chat suites **38/0** (`phase-chat-prompt-router` 8/0, `phase-chat-readonly` 5/0, `phase-chat-session-audit` 3/0, `phase-chat-operator` 8/0, `phase-chat-agent-select` 8/0, `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0), Control Panel **25/0** + build xanh. Next: CODEGEN-as-loop + Track 8 replay/widget → MVP-3 multi-tenant. |
|
||||
| **18 Chat Console** | 🟡 **MVP-0 + MVP-1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 + Track 8.1/8.2 done** · target arch còn planned | **Governed Chat Console** (cắt lát MVP chống lan man). **MVP-0 Ask CASAN read-only DONE**: `prompt-mode-router.py`, `chat-readonly.py`, `chat-session.schema.json`, H4 input/output scan, H5 chat audit hash-chain, H6 token telemetry, answer kèm evidence sources. **MVP-1 Operator DONE**: registered actions through `action-gate`, no free-command, action artifacts with provenance. **MVP-2 Track 4 DONE**: `agent-registry.yaml`, `chat-agent-resolver.py`, `chat:select_agent` RBAC, delegation hold, tool allowlist BLOCK, Control Panel agent picker. **MVP-2 OPERATOR Track 5/6 DONE**: `chat-turn.py` certifies UNCERTIFIED draft through Plan-17 `loop-run.sh` + trace verify/replay before side-effect release. **Track 8.1/8.2 DONE**: `chat-replay.py` verifies chat chain, evidence artifact hash, and OPERATOR loop replay; Control Panel `GET /api/v1/chat/replay`. Test: chat suites **42/0** (`phase-chat-prompt-router` 8/0, `phase-chat-readonly` 5/0, `phase-chat-session-audit` 3/0, `phase-chat-operator` 8/0, `phase-chat-agent-select` 8/0, `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0, `phase-chat-replay` 4/0), Control Panel **25/0** + build xanh. Next: CODEGEN-as-loop + Track 8.3/8.4 replay widget/approvals escalation → MVP-3 multi-tenant. |
|
||||
---
|
||||
|
||||
## Trần điểm & điều kiện lên "Strong (81+)"
|
||||
|
||||
@@ -35,7 +35,7 @@
|
||||
| 15 | `CASAN_PLAN_15_RESPONSIBLE_AI_DATA_GOV.md` | Responsible AI & Data Governance (FPT §14.3–14.4) | 📋 |
|
||||
| 16 | `CASAN_PLAN_16_SECURITY_AUDIT_REMEDIATION.md` | Security audit core harness + kế hoạch vá (tamper-evidence/injection/fail-open) | � P0/P1/P2 done |
|
||||
| 17 | `CASAN_PLAN_17_LOOP_ENGINEERING.md` | Loop Engineering / Agentic Loop Governance (budget governor, convergence, verify-contract, loop trace/replay, meta-loop) | 📋 |
|
||||
| 18 | `CASAN_PLAN_18_CHAT_CONSOLE.md` | Governed Chat Console (target arch; **MVP-0 Ask CASAN + MVP-1 Operator done; MVP-2 Track 4 + Operator Track 5/6 done** → replay/widget → multi-tenant) | 🟡 MVP-0/1 + Track 4/5/6 slice done |
|
||||
| 18 | `CASAN_PLAN_18_CHAT_CONSOLE.md` | Governed Chat Console (target arch; **MVP-0 Ask CASAN + MVP-1 Operator done; MVP-2 Track 4/5/6 + Track 8.1/8.2 done** → widget/approvals → multi-tenant) | 🟡 MVP-0/1 + MVP-2 partial done |
|
||||
| Future | `CASAN_PLAN_FUTURE_PHASES.md` | Approval workflow, state machine, benchmark, memory, remediation, KPI | 💤 vision |
|
||||
|
||||
> **Không có plan số 11:** số 11 được bỏ trống có chủ ý — nhánh eval/traceability đã
|
||||
|
||||
@@ -286,8 +286,8 @@ flowchart TD
|
||||
### Track 8 — Replay / Evidence / Command Center widgets `[MVP-2 → MVP-3]`
|
||||
| Task | Việc | File | Verify (WSL) |
|
||||
|---|---|---|---|
|
||||
| 18.8.1 | **Replay** một turn (verify lại artifact đã ghi) phát hiện tamper/non-determinism | nối `loop-trace.py replay` | sửa artifact turn → replay lệch |
|
||||
| 18.8.2 | `verify-chain` lịch sử chat (tamper-evidence, SEC-01/02, KMS khi TIER-2) | nối `loop-trace verify-chain` | sửa 1 turn → chain BREAK |
|
||||
| 18.8.1 | ✅ **Replay** một turn (verify lại artifact đã ghi) phát hiện tamper/non-determinism | `chat-replay.py` + `loop-trace.py replay` | sửa artifact turn → replay lệch |
|
||||
| 18.8.2 | ✅ `verify-chain` lịch sử chat (tamper-evidence, SEC-01/02; KMS khi TIER-2 là follow-up) | `chat-replay.py verify-chain` | sửa 1 turn → chain BREAK |
|
||||
| 18.8.3 | Widget **Chat/Loop** trên Command Center (13 §8.6): ticker per-iteration, budget gauge, click → Evidence drawer | nối Plan-13 §8.6 | số khớp fixture; click → evidence |
|
||||
| 18.8.4 | **Approvals/Escalation panel**: turn `ESCALATED` vào approvals inbox (13 §3.4), duyệt/từ chối (JWT + SoD + lý do) | nối Plan-13 §3.4 | escalate → pending; JWT giả → DENY |
|
||||
|
||||
@@ -355,6 +355,12 @@ cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`.
|
||||
- [x] Tool output is scanned with `tool-output-scan.sh` before returning the operator result to chat.
|
||||
- [x] Control Panel response includes `loop_run`; `/chat` displays loop certification/replay status. Verify: `phase-chat-pipeline` 4/0, `phase-chat-stream-hold` 2/0, `phase-chat-operator` 8/0, `console:test` 25/0, `console:build` xanh.
|
||||
|
||||
**MVP-2 Track 8.1/8.2 (Replay / verify-chain foundation):**
|
||||
- [x] `chat-replay.py verify-chain` recomputes chat audit hash-chain and fails closed on edited records.
|
||||
- [x] `chat-replay.py replay` verifies recorded evidence artifact hashes and replays OPERATOR loop traces via `loop-trace.py replay`.
|
||||
- [x] Control Plane `GET /api/v1/chat/replay` wraps harness replay; no verdict reimplementation in NestJS.
|
||||
- [x] Tampered operator evidence artifact → `DRIFT`; tampered chat audit → `BREAK`. Verify: `phase-chat-replay` 4/0, `console:test` 25/0, `console:build` xanh.
|
||||
|
||||
**Full (target architecture) — DoD tổng:**
|
||||
- [ ] Turn chạy **đúng như một loop-run Plan-17** (không định nghĩa vòng lặp riêng).
|
||||
- [ ] **Không đường vòng:** test chứng minh bỏ bất kỳ gate nào (preflight/verify) → FAIL.
|
||||
@@ -368,10 +374,11 @@ cập nhật tổng test ở `CASAN_HARDENING_STATUS.md`.
|
||||
---
|
||||
|
||||
## 7. Ghi chú trung thực & Non-goals (không lan man)
|
||||
- **[MVP-0 + MVP-1 + MVP-2 Track 4 + Operator Track 5/6] — 🟡 done+test.** Ask CASAN
|
||||
read-only, Operator registered actions, agent/skill governance foundation, và
|
||||
OPERATOR chat-as-loop/draft-hold đã có harness CLI + Control Panel API/UI. Các
|
||||
phase CODEGEN-as-loop / replay widget / multi-tenant production vẫn chưa mở.
|
||||
- **[MVP-0 + MVP-1 + MVP-2 Track 4 + Operator Track 5/6 + Track 8.1/8.2] — 🟡 done+test.**
|
||||
Ask CASAN read-only, Operator registered actions, agent/skill governance foundation,
|
||||
OPERATOR chat-as-loop/draft-hold, và replay/verify-chain foundation đã có harness
|
||||
CLI + Control Panel API. Các phase CODEGEN-as-loop / replay widget / approvals
|
||||
escalation / multi-tenant production vẫn chưa mở.
|
||||
- **MVP-first (chống lan man):** **MVP-0 (Ask CASAN read-only) KHÔNG phụ thuộc Plan-17/
|
||||
14/SEC-23** — làm được ngay trên nền H4/H5/H6 hiện có. Chỉ **MVP-2 trở đi** (chat-as-
|
||||
loop + agent) mới cần Plan-17 (T1–T3) + Plan-14 RBAC; **MVP-3** mới cần SEC-23 tenant.
|
||||
|
||||
@@ -77,7 +77,7 @@ approval JWT thật ([16 SEC-07](CASAN_PLAN_16_SECURITY_AUDIT_REMEDIATION.md)),
|
||||
|---|---|---|:--:|:--:|---|
|
||||
| B9 | ✅ **done** — **MVP-0 Ask CASAN read-only**: Prompt Router (Track 0) + Read-only Ask CASAN (Track 1) + session/audit (Track 2) + Control Panel API/UI (Track 7). H4 in/out, H5 audit hash-chain, H6 token, evidence sources; chat suites 14/0, Control Panel 23/0 + build xanh. Real model provider binding remains Track M follow-up when a provider is enabled. | [18 §1b, Track 0/1/2/7/M](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | ✅ | done |
|
||||
| B10 | ✅ **done** — **MVP-1 Operator mode**: registered actions `run tests` / `build pack` / `verify pack`, no free-command, all through `action-gate`, action artifacts with provenance, Control Panel quick actions. `phase-chat-operator` 8/0; total chat suites 24/0; Control Panel 24/0 + build xanh. | [18 Track 3](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | ✅ | done |
|
||||
| B11 | 🟡 **partial done** — Track 4 Agent/Skill governance foundation + OPERATOR Track 5/6 chat-as-loop/draft-hold. `chat-turn.py` now certifies OPERATOR draft through Plan-17 `loop-run.sh` + trace verify/replay before releasing registered actions; denied drafts do not execute. Chat suites 38/0; Control Panel 25/0 + build xanh. **Remaining MVP-2:** CODEGEN-as-loop and Track 8 replay/widget. | [18 Track 4/5/6](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | 🔗 | B6, C7(RBAC), B10 |
|
||||
| B11 | 🟡 **partial done** — Track 4 Agent/Skill governance + OPERATOR Track 5/6 chat-as-loop/draft-hold + Track 8.1/8.2 replay foundation. `chat-turn.py` certifies OPERATOR draft through Plan-17 `loop-run.sh`; `chat-replay.py` detects evidence artifact drift and chat-chain tamper. Chat suites 42/0; Control Panel 25/0 + build xanh. **Remaining MVP-2:** CODEGEN-as-loop and Track 8.3/8.4 replay widget/approvals escalation. | [18 Track 4/5/6/8](CASAN_PLAN_18_CHAT_CONSOLE.md) | L | 🔗 | B6, C7(RBAC), B10 |
|
||||
| B12 | **Widget Loop/Chat** trên Command Center (loop ticker/budget/replay drawer) | [17 (17.22)](CASAN_PLAN_17_LOOP_ENGINEERING.md) · [18 Track 8](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | 🔗 | C5 ✅ baseline |
|
||||
| B13 | **MVP-3** multi-tenant chat hardening (Track 9) | [18 Track 9](CASAN_PLAN_18_CHAT_CONSOLE.md) | M | 🔗 | A3(SEC-23) |
|
||||
|
||||
|
||||
@@ -68,6 +68,9 @@ Governed Chat (Plan-18 MVP-0/1 + MVP-2 Track 4 and Operator Track 5/6):
|
||||
with role visibility; selected agent/skill/delegation are bound by harness
|
||||
`chat-agent-resolver.py`.
|
||||
- `GET /api/v1/chat/audit/verify` — verifies the chat audit hash chain.
|
||||
- `GET /api/v1/chat/replay?chatId=<id>` — verifies chat-chain integrity,
|
||||
evidence artifact hashes, and OPERATOR loop trace replay through harness
|
||||
`chat-replay.py`.
|
||||
- `/chat` UI shows actor/role scope, `mode/risk/decision` badges, certified answer,
|
||||
evidence sources, registered operator actions, agent binding, loop certification,
|
||||
action-gate status, router details, and audit hash. Side-effect requests outside
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
import { Body, Controller, Get, Headers, Inject, Post } from '@nestjs/common';
|
||||
import { Body, Controller, Get, Headers, Inject, Post, Query } from '@nestjs/common';
|
||||
import { ok } from '../common/api-response.js';
|
||||
import { actorFromHeaders } from '../common/auth-context.js';
|
||||
import { ChatAskInput, ChatService } from './chat.service.js';
|
||||
@@ -17,6 +17,11 @@ export class ChatController {
|
||||
return ok(this.svc.verifyAudit());
|
||||
}
|
||||
|
||||
@Get('replay')
|
||||
replay(@Query('chatId') chatId?: string, @Query('turnId') turnId?: string) {
|
||||
return ok(this.svc.replay(chatId || '', turnId || ''));
|
||||
}
|
||||
|
||||
@Get('actions')
|
||||
actions(@Headers() headers: Record<string, string | string[] | undefined>) {
|
||||
return ok(this.svc.listActions(actorFromHeaders(headers)));
|
||||
|
||||
@@ -22,6 +22,7 @@ const HARNESS_BIN = join(APP_ROOT, 'packages', 'casan-harness', 'scripts', 'bash
|
||||
const CHAT_CLI = join(HARNESS_BIN, 'chat-turn.py');
|
||||
const OPERATOR_CLI = join(HARNESS_BIN, 'chat-operator.py');
|
||||
const AGENT_CLI = join(HARNESS_BIN, 'chat-agent-resolver.py');
|
||||
const REPLAY_CLI = join(HARNESS_BIN, 'chat-replay.py');
|
||||
const RBAC_CLI = join(HARNESS_BIN, 'rbac-check.py');
|
||||
|
||||
function runPython(script: string, args: string[]): CommandResult {
|
||||
@@ -93,6 +94,16 @@ export class ChatService {
|
||||
return { ok: res.status === 0, output: res.stdout || res.stderr };
|
||||
}
|
||||
|
||||
replay(chatId = '', turnId = '') {
|
||||
const args = ['replay'];
|
||||
if (chatId) args.push('--chat-id', chatId);
|
||||
if (turnId) args.push('--turn-id', turnId);
|
||||
const res = runPython(REPLAY_CLI, args);
|
||||
const parsed = parseJson<Record<string, any>>(res.stdout);
|
||||
if (parsed) return { ok: res.status === 0, ...parsed };
|
||||
throw new InternalServerErrorException(res.stderr || res.stdout || 'CHAT_REPLAY_FAILED');
|
||||
}
|
||||
|
||||
listActions(actor: SettingsActor) {
|
||||
this.requireRead(actor);
|
||||
const res = runPython(OPERATOR_CLI, ['list-actions']);
|
||||
|
||||
@@ -89,6 +89,10 @@ test('chat ask executes registered operator action through action-gate', () => {
|
||||
assert.equal(res.loop_run.side_effect_released, true);
|
||||
assert.equal(res.loop_run.trace_verify.ok, true);
|
||||
assert.equal(res.loop_run.replay.ok, true);
|
||||
const replay = svc.replay('operator-chat') as any;
|
||||
assert.equal(replay.ok, true);
|
||||
assert.equal(replay.decision, 'MATCH');
|
||||
assert.equal(replay.loop_replayed, 1);
|
||||
assert.equal(res.audit_verify.ok, true);
|
||||
});
|
||||
});
|
||||
|
||||
@@ -157,6 +157,15 @@ export interface ChatAgent {
|
||||
allowed_for_role?: boolean;
|
||||
}
|
||||
|
||||
export interface ChatReplay {
|
||||
ok: boolean;
|
||||
decision: 'MATCH' | 'DRIFT' | 'BREAK' | string;
|
||||
records: number;
|
||||
loop_replayed?: number;
|
||||
diffs?: any[];
|
||||
audit_path?: string;
|
||||
}
|
||||
|
||||
export interface SettingsState {
|
||||
actor: SettingsActor;
|
||||
capabilities: {
|
||||
@@ -206,6 +215,7 @@ export const api = {
|
||||
askChat: (actor: SettingsActor, body: { message: string; chatId?: string; agentId?: string; skillId?: string; delegationLevel?: number }) =>
|
||||
post<ChatAnswer>('chat/ask', body, actorHeaders(actor)),
|
||||
verifyChatAudit: () => get<{ ok: boolean; output: string }>('chat/audit/verify'),
|
||||
replayChat: (chatId = '', turnId = '') => get<ChatReplay>(`chat/replay?chatId=${encodeURIComponent(chatId)}&turnId=${encodeURIComponent(turnId)}`),
|
||||
chatActions: (actor: SettingsActor) => getWithHeaders<{ success: boolean; actions: ChatAction[] }>('chat/actions', actorHeaders(actor)),
|
||||
chatAgents: (actor: SettingsActor) => getWithHeaders<{ success: boolean; agents: ChatAgent[] }>('chat/agents', actorHeaders(actor)),
|
||||
};
|
||||
|
||||
@@ -69,6 +69,14 @@ def sha(text: str) -> str:
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def sha_file(path: str) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(65536), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def git_commit() -> str:
|
||||
try:
|
||||
r = subprocess.run(["git", "rev-parse", "HEAD"], cwd=ROOT, capture_output=True, text=True, timeout=5)
|
||||
@@ -204,6 +212,12 @@ def record_metrics(trace_id: str, message: str, answer: str, status: str, latenc
|
||||
def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, router, decision, answer,
|
||||
action=None, action_gate=None, artifact_path="", safe_message=""):
|
||||
elapsed = int((datetime.now(timezone.utc) - started).total_seconds() * 1000)
|
||||
loop_run = {}
|
||||
if os.environ.get("CASAN_CHAT_LOOP_RUN_JSON"):
|
||||
try:
|
||||
loop_run = json.loads(os.environ["CASAN_CHAT_LOOP_RUN_JSON"])
|
||||
except ValueError:
|
||||
loop_run = {"success": False, "decision": "DENIED", "reason": "loop_run_json_invalid"}
|
||||
source = []
|
||||
if artifact_path:
|
||||
source.append({
|
||||
@@ -211,6 +225,7 @@ def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, route
|
||||
"line": 1,
|
||||
"excerpt": answer[:260],
|
||||
"score": 10 if decision == "ACTION_COMPLETED" else 1,
|
||||
"hash": sha_file(artifact_path),
|
||||
"envelope": provenance("chat-operator-artifact", artifact_path, decision == "ACTION_COMPLETED"),
|
||||
})
|
||||
rec = record_turn({
|
||||
@@ -229,7 +244,8 @@ def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, route
|
||||
"user_msg_ref": sha(message),
|
||||
"safe_preview": (safe_message or "")[:180],
|
||||
"answer_ref": sha(answer),
|
||||
"sources": [{"path": s.get("path"), "line": s.get("line")} for s in source],
|
||||
"sources": [{"path": s.get("path"), "line": s.get("line"), "hash": s.get("hash")} for s in source],
|
||||
"loop_run": loop_run,
|
||||
})
|
||||
record_metrics(trace_id, safe_message or message, answer, "success" if decision == "ACTION_COMPLETED" else "failed", elapsed, (action or {}).get("id", "none"))
|
||||
return {
|
||||
@@ -247,6 +263,7 @@ def finish(started, trace_id, chat_id, turn_id, tenant_id, actor, message, route
|
||||
"router": router,
|
||||
"action": {k: action.get(k) for k in ("id", "label", "description")} if action else None,
|
||||
"action_gate": action_gate or {},
|
||||
"loop_run": loop_run,
|
||||
}
|
||||
|
||||
|
||||
|
||||
@@ -0,0 +1,187 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Plan-18 Track 8 chat replay / verify-chain.
|
||||
|
||||
Verifies chat history after the fact:
|
||||
* H5 chat turn hash-chain is intact,
|
||||
* recorded evidence artifact hashes still match,
|
||||
* OPERATOR loop-run traces still verify and replay through Plan-17 loop-trace.
|
||||
|
||||
This complements chat-readonly.py verify-audit by validating artifact state, not
|
||||
only the append-only JSONL chain.
|
||||
"""
|
||||
import argparse
|
||||
import hashlib
|
||||
import json
|
||||
import os
|
||||
import subprocess
|
||||
import sys
|
||||
|
||||
|
||||
GENESIS_HASH = "0" * 64
|
||||
|
||||
|
||||
def project_root() -> str:
|
||||
d = os.path.abspath(os.path.dirname(__file__))
|
||||
p = d
|
||||
while p != os.path.dirname(p):
|
||||
if os.path.isdir(os.path.join(p, ".specify")) or os.path.isdir(os.path.join(p, "packages/casan-harness")):
|
||||
return p
|
||||
p = os.path.dirname(p)
|
||||
return os.path.abspath(os.path.join(d, "..", "..", ".."))
|
||||
|
||||
|
||||
ROOT = project_root()
|
||||
BIN = os.path.join(ROOT, "packages", "casan-harness", "scripts", "bash")
|
||||
LOOP_TRACE = os.path.join(BIN, "loop-trace.py")
|
||||
|
||||
|
||||
def state_root() -> str:
|
||||
return os.environ.get("CASAN_STATE_ROOT") or os.path.join(ROOT, ".specify")
|
||||
|
||||
|
||||
def audit_path() -> str:
|
||||
return os.environ.get("CASAN_CHAT_AUDIT_LOG") or os.path.join(state_root(), "logs", "chat", "chat-turns.jsonl")
|
||||
|
||||
|
||||
def sha_text(text: str) -> str:
|
||||
return hashlib.sha256(text.encode("utf-8")).hexdigest()
|
||||
|
||||
|
||||
def sha_file(path: str) -> str:
|
||||
h = hashlib.sha256()
|
||||
with open(path, "rb") as fh:
|
||||
for chunk in iter(lambda: fh.read(65536), b""):
|
||||
h.update(chunk)
|
||||
return h.hexdigest()
|
||||
|
||||
|
||||
def load_records():
|
||||
records = []
|
||||
try:
|
||||
with open(audit_path(), encoding="utf-8") as fh:
|
||||
for line in fh:
|
||||
if line.strip():
|
||||
records.append(json.loads(line))
|
||||
except OSError:
|
||||
return []
|
||||
return records
|
||||
|
||||
|
||||
def verify_chain(records):
|
||||
prev = GENESIS_HASH
|
||||
for idx, rec in enumerate(records, start=1):
|
||||
got = rec.get("record_hash")
|
||||
rest = {k: v for k, v in rec.items() if k != "record_hash"}
|
||||
if rest.get("prev_hash") != prev or sha_text(json.dumps(rest, sort_keys=True, ensure_ascii=False)) != got:
|
||||
return False, idx
|
||||
prev = got
|
||||
return True, None
|
||||
|
||||
|
||||
def resolve_path(path: str) -> str:
|
||||
if os.path.isabs(path):
|
||||
return path
|
||||
return os.path.join(ROOT, path)
|
||||
|
||||
|
||||
def loop_state_root(loop_run):
|
||||
explicit = os.environ.get("CASAN_LOOP_STATE_ROOT")
|
||||
if explicit:
|
||||
return explicit
|
||||
artifact = loop_run.get("artifact") or ""
|
||||
marker = os.sep + "logs" + os.sep + "chat" + os.sep + "loop-runs" + os.sep
|
||||
if marker in artifact:
|
||||
return artifact.split(marker, 1)[0] + os.sep + "logs" + os.sep + "chat" + os.sep + "loop-state"
|
||||
return os.path.join(state_root(), "logs", "chat", "loop-state")
|
||||
|
||||
|
||||
def run_loop_trace(loop_run, cmd: str):
|
||||
env = {**os.environ, "CASAN_LOOP_STATE_ROOT": loop_state_root(loop_run)}
|
||||
run_id = loop_run.get("run_id")
|
||||
if not run_id:
|
||||
return False, "missing_loop_run_id"
|
||||
args = ["python3", LOOP_TRACE, cmd, "--run-id", run_id]
|
||||
if cmd == "replay":
|
||||
args += ["--profile", os.environ.get("CASAN_PROFILE", "dev")]
|
||||
r = subprocess.run(args, cwd=ROOT, capture_output=True, text=True, env=env)
|
||||
return r.returncode == 0, (r.stdout + r.stderr).strip()
|
||||
|
||||
|
||||
def filter_records(records, chat_id="", turn_id=""):
|
||||
out = records
|
||||
if chat_id:
|
||||
out = [r for r in out if r.get("chat_id") == chat_id]
|
||||
if turn_id:
|
||||
out = [r for r in out if r.get("turn_id") == turn_id]
|
||||
return out
|
||||
|
||||
|
||||
def replay(args) -> int:
|
||||
records = load_records()
|
||||
ok, broken_at = verify_chain(records)
|
||||
if not ok:
|
||||
print(json.dumps({"decision": "BREAK", "reason": "chat_chain_broken", "broken_at": broken_at}, ensure_ascii=False))
|
||||
return 3
|
||||
|
||||
selected = filter_records(records, args.chat_id, args.turn_id)
|
||||
diffs = []
|
||||
replayed = 0
|
||||
for rec in selected:
|
||||
for src in rec.get("sources", []):
|
||||
expected = src.get("hash")
|
||||
if not expected:
|
||||
continue
|
||||
path = resolve_path(src.get("path", ""))
|
||||
if not os.path.isfile(path):
|
||||
diffs.append({"seq": rec.get("seq"), "kind": "artifact_missing", "path": src.get("path")})
|
||||
continue
|
||||
got = sha_file(path)
|
||||
if got != expected:
|
||||
diffs.append({"seq": rec.get("seq"), "kind": "artifact_hash_mismatch", "path": src.get("path"), "expected": expected, "got": got})
|
||||
|
||||
loop_run = rec.get("loop_run") or {}
|
||||
if loop_run.get("run_id"):
|
||||
replayed += 1
|
||||
trace_ok, trace_out = run_loop_trace(loop_run, "verify-chain")
|
||||
replay_ok, replay_out = run_loop_trace(loop_run, "replay")
|
||||
if not trace_ok:
|
||||
diffs.append({"seq": rec.get("seq"), "kind": "loop_trace_break", "run_id": loop_run.get("run_id"), "output": trace_out[:400]})
|
||||
if not replay_ok:
|
||||
diffs.append({"seq": rec.get("seq"), "kind": "loop_replay_drift", "run_id": loop_run.get("run_id"), "output": replay_out[:400]})
|
||||
|
||||
payload = {
|
||||
"decision": "DRIFT" if diffs else "MATCH",
|
||||
"records": len(selected),
|
||||
"loop_replayed": replayed,
|
||||
"diffs": diffs,
|
||||
"audit_path": audit_path(),
|
||||
}
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 3 if diffs else 0
|
||||
|
||||
|
||||
def verify(args) -> int:
|
||||
records = load_records()
|
||||
ok, broken_at = verify_chain(records)
|
||||
payload = {"decision": "OK" if ok else "BREAK", "records": len(records), "audit_path": audit_path()}
|
||||
if broken_at:
|
||||
payload["broken_at"] = broken_at
|
||||
print(json.dumps(payload, ensure_ascii=False, indent=2, sort_keys=True))
|
||||
return 0 if ok else 3
|
||||
|
||||
|
||||
def main() -> int:
|
||||
ap = argparse.ArgumentParser()
|
||||
sub = ap.add_subparsers(dest="cmd", required=True)
|
||||
r = sub.add_parser("replay")
|
||||
r.add_argument("--chat-id", default="")
|
||||
r.add_argument("--turn-id", default="")
|
||||
r.set_defaults(func=replay)
|
||||
v = sub.add_parser("verify-chain")
|
||||
v.set_defaults(func=verify)
|
||||
args = ap.parse_args()
|
||||
return args.func(args)
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
raise SystemExit(main())
|
||||
@@ -63,7 +63,10 @@ def scan_tool_output(text: str, label: str):
|
||||
|
||||
|
||||
def run_mode(args, binding, loop_run=None, scan_output_label=""):
|
||||
r = subprocess.run(args, cwd=ROOT, capture_output=True, text=True)
|
||||
env = os.environ.copy()
|
||||
if loop_run is not None:
|
||||
env["CASAN_CHAT_LOOP_RUN_JSON"] = json.dumps(loop_run, ensure_ascii=False, sort_keys=True)
|
||||
r = subprocess.run(args, cwd=ROOT, capture_output=True, text=True, env=env)
|
||||
if scan_output_label:
|
||||
scan_rc, scan_msg = scan_tool_output(r.stdout + "\n" + r.stderr, scan_output_label)
|
||||
if scan_rc != 0:
|
||||
|
||||
@@ -152,6 +152,7 @@ run "phase-chat-operator" bash "$TESTS/phase-chat-operator-tests.sh"
|
||||
run "phase-chat-agent-select" bash "$TESTS/phase-chat-agent-select-tests.sh"
|
||||
run "phase-chat-pipeline" bash "$TESTS/phase-chat-pipeline-tests.sh"
|
||||
run "phase-chat-stream-hold" bash "$TESTS/phase-chat-stream-hold-tests.sh"
|
||||
run "phase-chat-replay" bash "$TESTS/phase-chat-replay-tests.sh"
|
||||
|
||||
# ARCH-02: coverage cannot silently drop; ARCH-01: harness/policy cannot silently
|
||||
# drift. Both SKIP cleanly when no manifest is provisioned (non-strict dev/CI).
|
||||
|
||||
@@ -0,0 +1,88 @@
|
||||
#!/usr/bin/env bash
|
||||
set -uo pipefail
|
||||
|
||||
# Plan-18 MVP-2 Track 8: chat replay verifies chat chain, evidence artifact hash,
|
||||
# and Plan-17 loop trace replay for OPERATOR turns.
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/../scripts/bash/casan-paths.sh"
|
||||
TURN="$CASAN_HARNESS_ROOT/scripts/bash/chat-turn.py"
|
||||
REPLAY="$CASAN_HARNESS_ROOT/scripts/bash/chat-replay.py"
|
||||
WORK="$(mktemp -d)"
|
||||
trap 'rm -rf "$WORK"' EXIT
|
||||
export CASAN_STATE_ROOT="$WORK/state"
|
||||
export CASAN_LOOP_STATE_ROOT="$CASAN_STATE_ROOT/logs/chat/loop-state"
|
||||
|
||||
PASS=0; FAIL=0
|
||||
pass() { echo "PASS: $1"; PASS=$((PASS + 1)); }
|
||||
fail() { echo "FAIL: $1"; FAIL=$((FAIL + 1)); }
|
||||
|
||||
echo "===== Plan-18 MVP-2 chat replay ====="
|
||||
|
||||
python3 "$TURN" ask --message "run tests" --actor bob --role operator --chat-id replay-chat > "$WORK/turn.json"
|
||||
|
||||
python3 "$REPLAY" verify-chain > "$WORK/verify.json" \
|
||||
&& grep -q '"decision": "OK"' "$WORK/verify.json" \
|
||||
&& pass "chat replay verifies chat audit chain" || fail "chat replay chain verify failed"
|
||||
|
||||
python3 "$REPLAY" replay --chat-id replay-chat > "$WORK/replay.json"
|
||||
python3 - "$WORK/replay.json" <<'PY' \
|
||||
&& pass "chat replay matches clean operator turn" || fail "chat replay did not match clean turn"
|
||||
import json, sys
|
||||
d = json.load(open(sys.argv[1]))
|
||||
assert d["decision"] == "MATCH"
|
||||
assert d["records"] == 1
|
||||
assert d["loop_replayed"] == 1
|
||||
assert d["diffs"] == []
|
||||
PY
|
||||
|
||||
ARTIFACT="$(python3 - "$WORK/turn.json" <<'PY'
|
||||
import json, os, sys
|
||||
d = json.load(open(sys.argv[1]))
|
||||
print(d["sources"][0]["path"])
|
||||
PY
|
||||
)"
|
||||
case "$ARTIFACT" in
|
||||
/*) ARTIFACT_PATH="$ARTIFACT" ;;
|
||||
*) ARTIFACT_PATH="$CASAN_APP_ROOT/$ARTIFACT" ;;
|
||||
esac
|
||||
printf '\nTAMPERED\n' >> "$ARTIFACT_PATH"
|
||||
set +e
|
||||
python3 "$REPLAY" replay --chat-id replay-chat > "$WORK/tamper-artifact.json"
|
||||
RC=$?
|
||||
set -e 2>/dev/null || true
|
||||
python3 - "$WORK/tamper-artifact.json" "$RC" <<'PY' \
|
||||
&& pass "chat replay detects evidence artifact tamper" || fail "chat replay missed artifact tamper"
|
||||
import json, sys
|
||||
d = json.load(open(sys.argv[1]))
|
||||
assert int(sys.argv[2]) == 3
|
||||
assert d["decision"] == "DRIFT"
|
||||
assert any(x["kind"] == "artifact_hash_mismatch" for x in d["diffs"])
|
||||
PY
|
||||
|
||||
# Fresh state for chain tamper so artifact drift above does not mask the chain test.
|
||||
export CASAN_STATE_ROOT="$WORK/state-chain"
|
||||
export CASAN_LOOP_STATE_ROOT="$CASAN_STATE_ROOT/logs/chat/loop-state"
|
||||
python3 "$TURN" ask --message "run tests" --actor bob --role operator --chat-id chain-chat >/dev/null
|
||||
AUDIT="$CASAN_STATE_ROOT/logs/chat/chat-turns.jsonl"
|
||||
python3 - "$AUDIT" <<'PY'
|
||||
import json, sys
|
||||
rows = [json.loads(l) for l in open(sys.argv[1]) if l.strip()]
|
||||
rows[0]["decision"] = "TAMPERED"
|
||||
open(sys.argv[1], "w").write("\n".join(json.dumps(r) for r in rows) + "\n")
|
||||
PY
|
||||
set +e
|
||||
python3 "$REPLAY" verify-chain > "$WORK/tamper-chain.json"
|
||||
RC=$?
|
||||
set -e 2>/dev/null || true
|
||||
python3 - "$WORK/tamper-chain.json" "$RC" <<'PY' \
|
||||
&& pass "chat replay verify-chain detects audit tamper" || fail "chat replay missed audit tamper"
|
||||
import json, sys
|
||||
d = json.load(open(sys.argv[1]))
|
||||
assert int(sys.argv[2]) == 3
|
||||
assert d["decision"] == "BREAK"
|
||||
PY
|
||||
|
||||
echo ""
|
||||
echo "===== CHAT REPLAY SUMMARY: PASS=$PASS FAIL=$FAIL ====="
|
||||
[[ "$FAIL" -eq 0 ]] || exit 1
|
||||
@@ -19,7 +19,7 @@ Optional layer for teams that want UI / dashboard / visibility. Packages: `casan
|
||||
| RBAC + approval inbox | ✅ local-prod done | Settings + kill-switch API RBAC enforcement, approval inbox/delegation/oversight, SoD, governed setting proposal apply, and local OIDC claim→role mapping smoke are done. Enterprise IdP rollout remains production follow-up. |
|
||||
| Evidence Pack Viewer | 📋 planned | reads `docs/output/casan/evidence-packs/` |
|
||||
| Attack Battery Viewer | 📋 planned | reads red-team corpus + H4 recall results |
|
||||
| Read-only Ask CASAN + Operator + agent selection + operator loop-hold | 🟡 MVP-0/1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 done | Plan-18: `POST /api/v1/chat/ask`, `GET /api/v1/chat/actions`, `GET /api/v1/chat/agents` + `/chat`, backed by harness `chat-turn.py`, `chat-agent-resolver.py`, and Plan-17 `loop-run.sh` |
|
||||
| Read-only Ask CASAN + Operator + agent selection + operator loop-hold + replay | 🟡 MVP-0/1 done+test; MVP-2 Track 4 + OPERATOR Track 5/6 + Track 8.1/8.2 done | Plan-18: `POST /api/v1/chat/ask`, `GET /api/v1/chat/actions`, `GET /api/v1/chat/agents`, `GET /api/v1/chat/replay` + `/chat`, backed by harness `chat-turn.py`, `chat-agent-resolver.py`, `chat-replay.py`, and Plan-17 `loop-run.sh` |
|
||||
| Gitea webhook integration | 📋 planned | trigger gate / publish evidence on push |
|
||||
|
||||
## Build (preview)
|
||||
@@ -29,6 +29,6 @@ scripts/package-release.sh platform # → dist/casan-platform-preview-vX.Y.Z
|
||||
The bundle includes a `PREVIEW-INCOMPLETE.txt` marker. Do not treat it as a finished product.
|
||||
|
||||
## To implement later
|
||||
Start from Plan-15 RAI view or Plan-18 MVP-2 Track 8 (replay/widget) or CODEGEN-as-loop. Keep new numbers
|
||||
Start from Plan-15 RAI view or Plan-18 MVP-2 Track 8.3/8.4 (replay widget / approvals escalation) or CODEGEN-as-loop. Keep new numbers
|
||||
evidence-backed with provenance; any write/governed action must continue to route through
|
||||
harness RBAC, approval, and audit primitives.
|
||||
|
||||
Reference in New Issue
Block a user