Complete cloud provider patch MVP

00_SUBMISSION_PACKAGE/evidence/scoring-run-report.md

@@ -17,7 +17,7 @@
   run-casan4 **35** · adversarial **44** · phase1-track-a **25** · phase2-track-c **29** ·
   phase3-evidence-pack **7** · phase-h5-approval **12** · phase-h5-infra **7** · phase-h6-agentops **20** ·
   phase-c7-incident **15** · phase-h4-multilingual **7** · phase-c6-sandbox **6** · phase-h4-split-inject **8** · phase10-traceability **3**.
-- Direct model-router suite: **10 PASS / 0 FAIL** (includes model-digest pin OK, mismatch BLOCK, mismatch WARN rollout mode).
+- Direct model-router suite: **11 PASS / 0 FAIL** (includes model-digest pin OK, mismatch BLOCK, mismatch WARN rollout mode, deterministic OpenAI/Anthropic usage parser coverage).
 - Local production-like Docker infra lab: **2 PASS / 0 FAIL**, with `infra-lab verify` internal **7 PASS / 0 FAIL** (Vault Transit, OIDC/JWKS, MinIO Object Lock, alert webhook, billing API mock, dashboard nginx auth).
 - Frontend Vitest: **16 PASS / 0 FAIL**. Backend `npm test` is blocked by pre-existing app-test infra mismatch (`schema.prisma` provider MySQL but `setup-sqlite.mjs` applies the MySQL migration to SQLite).
 - `security-gate` aggregate: **verdict PASS=11 FAIL=0 SKIP=0** (run 2026-07-04).
@@ -81,7 +81,7 @@ bash .specify/tests/phase-h4-multilingual-tests.sh     # 7/0
 bash .specify/tests/phase-c6-sandbox-tests.sh          # 6/0
 bash .specify/tests/phase-h4-split-inject-tests.sh     # 8/0
 bash .specify/tests/phase10-traceability-tests.sh      # 3/0
-bash .specify/tests/phase3-model-router-tests.sh       # 10/0 (direct model-router/digest suite)
+bash .specify/tests/phase3-model-router-tests.sh       # 11/0 (direct model-router/digest/cloud parser suite)
 bash .specify/tests/phase-prod-infra-lab-tests.sh      # 2/0, starts Docker infra lab
 bash .specify/scripts/bash/security-gate.sh            # verdict PASS=11 FAIL=0
 ```

AINative_OKR_CASAN5/.specify/scripts/bash/model-call.py

@@ -156,15 +156,42 @@ def call_openai(model_name, prompt, role):
     except Exception as exc:  # honest non-zero, no fake success
         fail(f"backend_unreachable {type(exc).__name__}: {str(exc)[:120]}")
     latency_ms = int((time.time() - t0) * 1000)
-    usage = payload.get("usage", {})
+    text, input_tokens, output_tokens = parse_openai_payload(payload)
     return {
-        "text": (payload["choices"][0]["message"]["content"] or "").strip(),
-        "input_tokens": int(usage.get("prompt_tokens", 0)),
-        "output_tokens": int(usage.get("completion_tokens", 0)),
+        "text": text,
+        "input_tokens": input_tokens,
+        "output_tokens": output_tokens,
         "latency_ms": latency_ms,
     }
 
 
+def parse_openai_payload(payload):
+    try:
+        text = (payload["choices"][0]["message"]["content"] or "").strip()
+        usage = payload["usage"]
+        input_tokens = int(usage["prompt_tokens"])
+        output_tokens = int(usage["completion_tokens"])
+    except (KeyError, IndexError, TypeError, ValueError) as exc:
+        fail(f"provider_usage_invalid openai {type(exc).__name__}: {str(exc)[:80]}")
+    return text, input_tokens, output_tokens
+
+
+def parse_anthropic_payload(payload):
+    try:
+        content = payload["content"]
+        if not isinstance(content, list):
+            raise TypeError("content is not a list")
+        text = "".join(
+            b.get("text", "") for b in content if isinstance(b, dict) and b.get("type") == "text"
+        ).strip()
+        usage = payload["usage"]
+        input_tokens = int(usage["input_tokens"])
+        output_tokens = int(usage["output_tokens"])
+    except (KeyError, TypeError, ValueError) as exc:
+        fail(f"provider_usage_invalid anthropic {type(exc).__name__}: {str(exc)[:80]}")
+    return text, input_tokens, output_tokens
+
+
 def call_anthropic(model_name, prompt, role):
     # Endpoint hard-pinned to the allowlisted host (no env override). NOTE: on
     # current Claude models (Opus 4.8/4.7, Sonnet 5, ...) `temperature`/`top_p`
@@ -197,20 +224,14 @@ def call_anthropic(model_name, prompt, role):
     except Exception as exc:  # honest non-zero, no fake success
         fail(f"backend_unreachable {type(exc).__name__}: {str(exc)[:120]}")
     latency_ms = int((time.time() - t0) * 1000)
-    # content is a list of blocks; concatenate text blocks. A safety refusal
-    # (stop_reason=="refusal") yields empty text -> extract_verdict fails closed.
-    text = "".join(
-        b.get("text", "") for b in payload.get("content", []) if b.get("type") == "text"
-    ).strip()
-    usage = payload.get("usage", {})
+    text, input_tokens, output_tokens = parse_anthropic_payload(payload)
     return {
         "text": text,
-        "input_tokens": int(usage.get("input_tokens", 0)),
-        "output_tokens": int(usage.get("output_tokens", 0)),
+        "input_tokens": input_tokens,
+        "output_tokens": output_tokens,
         "latency_ms": latency_ms,
     }
 
-
 def main():
     ap = argparse.ArgumentParser()
     ap.add_argument("prompt_file")

AINative_OKR_CASAN5/.specify/tests/phase3-model-router-tests.sh

@@ -97,7 +97,74 @@ else
   skip "cloud-unavailable test (ANTHROPIC_API_KEY is set)"
 fi
 
-# 8: deliberate failing primary route -> fallback through the REAL router (not exit 9).
+# 8: cloud response parsers use provider token usage and fail closed on malformed
+# payloads. This is deterministic: urllib is monkeypatched, so no API key or
+# network call is needed.
+python - "$SCRIPTS/model-call.py" <<'PY'
+import importlib.util
+import contextlib
+import io
+import json
+import os
+import sys
+
+spec = importlib.util.spec_from_file_location("mc", sys.argv[1])
+mc = importlib.util.module_from_spec(spec); spec.loader.exec_module(mc)
+os.environ["OPENAI_API_KEY"] = "test-openai-key"
+os.environ["ANTHROPIC_API_KEY"] = "test-anthropic-key"
+
+seen = []
+
+class FakeResp:
+    def __init__(self, payload):
+        self.payload = payload
+    def __enter__(self):
+        return self
+    def __exit__(self, exc_type, exc, tb):
+        return False
+    def read(self):
+        return json.dumps(self.payload).encode()
+
+def fake_urlopen(req, timeout):
+    seen.append((req.full_url, dict(req.header_items()), json.loads(req.data.decode())))
+    if "openai.com" in req.full_url:
+        return FakeResp({
+            "choices": [{"message": {"content": "SAFE"}}],
+            "usage": {"prompt_tokens": 11, "completion_tokens": 3},
+        })
+    if "anthropic.com" in req.full_url:
+        return FakeResp({
+            "content": [{"type": "text", "text": "APPROVED"}],
+            "usage": {"input_tokens": 17, "output_tokens": 5},
+        })
+    raise AssertionError(req.full_url)
+
+mc.urllib.request.urlopen = fake_urlopen
+op = mc.call_openai("gpt-test", "hello", "classify")
+an = mc.call_anthropic("claude-test", "hello", "judge")
+assert op["text"] == "SAFE" and op["input_tokens"] == 11 and op["output_tokens"] == 3, op
+assert an["text"] == "APPROVED" and an["input_tokens"] == 17 and an["output_tokens"] == 5, an
+assert seen[0][0] == "https://api.openai.com/v1/chat/completions", seen[0]
+assert seen[1][0] == "https://api.anthropic.com/v1/messages", seen[1]
+assert seen[0][2]["temperature"] == 0 and seen[0][2]["max_tokens"] == 16, seen[0][2]
+assert "temperature" not in seen[1][2] and seen[1][2]["max_tokens"] == 16, seen[1][2]
+
+for fn, bad in (
+    (mc.parse_openai_payload, {"choices": [{"message": {"content": "SAFE"}}]}),
+    (mc.parse_anthropic_payload, {"content": [{"type": "text", "text": "APPROVED"}]}),
+):
+    try:
+        with contextlib.redirect_stderr(io.StringIO()):
+            fn(bad)
+    except SystemExit as exc:
+        assert exc.code == 2, exc.code
+    else:
+        raise AssertionError(f"{fn.__name__} accepted malformed provider payload")
+print("ok")
+PY
+[[ $? -eq 0 ]] && pass "cloud provider responses parse real usage and reject malformed payloads" || fail "cloud provider parser coverage failed"
+
+# 9: deliberate failing primary route -> fallback through the REAL router (not exit 9).
 if [[ "$TUNNEL_UP" -eq 1 ]]; then
   printf 'Return exactly: OK\n' > "$WORK/f.txt"
   set +e

casan-next-plans/CASAN_BACKLOG_STATUS.md

@@ -5,7 +5,7 @@
 > cũng làm tiếp được ngay**. Cập nhật mỗi khi hoàn thành một mục.
 >
 > Cập nhật lần cuối: 2026-07-06 · Nhánh làm tiếp từ handoff Claude.
-> Test hiện tại: **218 PASS / 0 FAIL** trên 13 core harness suite; local-prod Docker infra lab **2 PASS / 0 FAIL** (`infra-lab verify`: 7/0 internal checks); `phase3-model-router` riêng **10 PASS / 0 FAIL**; frontend Vitest **16 PASS / 0 FAIL**. Backend `npm test` còn bị chặn bởi test-infra cũ (`schema.prisma` MySQL nhưng `setup-sqlite.mjs` chạy SQLite).
+> Test hiện tại: **218 PASS / 0 FAIL** trên 13 core harness suite; local-prod Docker infra lab **2 PASS / 0 FAIL** (`infra-lab verify`: 7/0 internal checks); `phase3-model-router` riêng **11 PASS / 0 FAIL**; frontend Vitest **16 PASS / 0 FAIL**. Backend `npm test` còn bị chặn bởi test-infra cũ (`schema.prisma` MySQL nhưng `setup-sqlite.mjs` chạy SQLite).
 > Điểm công tâm vẫn quanh **~81/100**, harness thấp nhất 80; TIER 2 infra thật vẫn là trần Strong.
 > Nguồn liên quan: `CASAN_HARDENING_STATUS.md` (chi tiết control) · `evidence/scoring-run-report.md` (điểm).
 
@@ -40,8 +40,8 @@
 | Plan | Trạng thái | Lõi cần làm (bước tiếp theo cho AI kế) |
 |---|:--:|---|
 | **10 Traceability + H3 Eval** | ✅ MVP done+test | Đã nối traceability vào Evidence Pack. Sau MVP: line/symbol-level traceability + H3 eval-set độc lập (nhiều model). |
+| **03 Cloud patch** | 🟡 MVP done+test | `model-call.py` đã hỗ trợ `openai:<model>` và `anthropic:<model>` qua endpoint hard-pin + API key env; parser token usage và malformed payload có deterministic test. Còn live smoke với key thật + billing usage API ground truth. |
 | **02 LLM source-gen** | 📋 chưa bắt đầu | Thay template bằng LLM thật sinh source qua `model-router.sh`; đi qua wrapper H4→H7. Phụ thuộc 03. Bước 1: định contract prompt→file cho 1 module (objectives), gate bằng H3 judge + traceability. |
-| **03 Cloud patch** | 📋 chưa bắt đầu | Bỏ stub trong `model-call.py` cho OpenAI/Anthropic; test bằng key thật (🔌). Bước 1: env `CASAN_MODEL_PRIMARY=openai:…`, xác thực round-trip + ghi provider-usage thật. |
 | **04 Self-improve** | 📋 chưa bắt đầu | Khép vòng `casan improve`: đọc metrics/drift/hallucination → đề xuất vá → chạy lại gate. Phụ thuộc 02, 05. Bước 1: script đọc `metrics.jsonl` + `drift-report.json` → sinh backlog vá tự động. |
 | **05 CI/CD** | 📋 một phần (act_runner/deploy có) | Chuẩn hoá pipeline phát hành package `fpt-casan-sdd-harness` + chạy 12 suite trong CI (Vault+Docker service). Bước 1: `.gitea/workflows/harness-ci.yml` chạy toàn bộ suite + security-gate. |
 | **06 Onboard dự án 2** | 📋 chưa bắt đầu | Chứng minh reuse: cắm 1 repo khác + golden/corpus/input, đăng ký qua `verify-harness-reuse.sh`, không sửa gate. Phụ thuộc 01. |

casan-next-plans/CASAN_HARDENING_STATUS.md

@@ -88,7 +88,7 @@
 | `phase-h4-split-inject-tests.sh` | 8 | **New** — split-injection assembly scan + classifier-inject (B2) |
 | `phase10-traceability-tests.sh` | 3 | **New** — Plan-10 FR→code→test matrix + fail-able missing-test gate |
 | `phase-prod-infra-lab-tests.sh` | 2 | **New optional/local-prod** — Docker Compose infra lab starts + verifies Vault/IdP/MinIO/dashboard/alert/billing |
-| **Total** | **218 core + 2 local-prod infra lab** | Baseline 79 preserved; +139 new hardening/traceability checks. Last full harness run 2026-07-06, 0 fail. Direct `phase3-model-router-tests.sh` adds 10/0; `infra-lab verify` adds 7 internal infra checks. |
+| **Total** | **218 core + 2 local-prod infra lab** | Baseline 79 preserved; +139 new hardening/traceability checks. Last full harness run 2026-07-06, 0 fail. Direct `phase3-model-router-tests.sh` adds 11/0 including deterministic cloud provider parser coverage; `infra-lab verify` adds 7 internal infra checks. |
 
 Run order note: `run-casan4-harness-tests.sh` does `rm -rf .specify/logs`, so run it
 **first** and never concurrently with the other suites.

casan-next-plans/CASAN_PLAN_00_INDEX.md

@@ -21,7 +21,7 @@
 |---|---|---|---|---|
 | 01 | `CASAN_PLAN_01_RESTRUCTURE.md` | Tai cau truc thu muc Phase 0-6 | **File chi tiet dang thieu**; cong viec chua lam | Cao, nhung nen lam tren nhanh rieng |
 | 02 | `CASAN_PLAN_02_LLM_SOURCEGEN.md` | Noi LLM that vao sinh source thay template | Chua bat dau | Cao sau 03 |
-| 03 | `CASAN_PLAN_03_CLOUD_PATCH.md` | Patch cloud/OpenAI/Anthropic, bo stub | **File chi tiet dang thieu**; cong viec chua lam | Cao, can key/provider |
+| 03 | `CASAN_PLAN_03_CLOUD_PATCH.md` | Patch cloud/OpenAI/Anthropic, bo stub | MVP da co + deterministic test; live smoke can key that | Cao cho live provider |
 | 04 | `CASAN_PLAN_04_SELFIMPROVE.md` | Khep vong `casan improve` | Chua bat dau | Trung-Cao sau 02/05 |
 | 05 | `CASAN_PLAN_05_CICD.md` | CI/CD + release package | **File chi tiet dang thieu**; moi co mot phan runner/deploy | Cao cho release gate |
 | 06 | `CASAN_PLAN_06_ONBOARD.md` | Onboard du an that thu 2 | Chua bat dau | Cao de chung minh reuse |
@@ -36,7 +36,7 @@
 
 | Muc | Ket qua | Verify |
 |---|---|---|
-| B4 model-digest pinning | Pin digest `ornith:9b`; mismatch BLOCK mac dinh, WARN rollout | `phase3-model-router-tests.sh` 10/10 |
+| B4 model-digest pinning + cloud parser coverage | Pin digest `ornith:9b`; mismatch BLOCK mac dinh, WARN rollout; OpenAI/Anthropic parser deterministic test | `phase3-model-router-tests.sh` 11/11 |
 | C4 approval IdP/OIDC MVP | `approval-verify.sh` verify RS256 JWT, role, expiry, request binding; co mock IdP/JWKS | `phase-h5-approval-tests.sh` 12/12 |
 | Plan-10 traceability MVP | Parse `FR-*`, map requirement -> code -> test, gate thieu coverage | `phase10-traceability-tests.sh` 3/3 |
 | Docker local-prod infra lab | Vault, IdP/JWKS, MinIO Object Lock, dashboard nginx auth, alert webhook, billing API mock | `phase-prod-infra-lab-tests.sh` 2/2; `infra-lab verify` 7/7 |
@@ -45,8 +45,8 @@
 
 | Uu tien | Viec con lai | Ly do | Buoc dau tien de lam tiep |
 |---|---|---|---|
-| P1 | Plan-03 Cloud patch/OpenAI/Anthropic | De LLM source-gen va billing telemetry that co y nghia | Tao/bo sung `CASAN_PLAN_03_CLOUD_PATCH.md`; wire `CASAN_MODEL_PRIMARY=openai/anthropic` qua `model-call.py`, test round-trip co key |
-| P2 | Plan-02 LLM source-gen dot A | Hien pipeline van sinh artifact bang template deterministic | Dinh contract `generate(step, ctx)` cho `01-srs`/`02-bd`, goi `model-router.sh --role generate`, fallback template |
+| P1 | Plan-02 LLM source-gen dot A | Hien pipeline van sinh artifact bang template deterministic | Dinh contract `generate(step, ctx)` cho `01-srs`/`02-bd`, goi `model-router.sh --role generate`, fallback template |
+| P2 | Plan-03 live cloud smoke | MVP cloud patch da test offline; chua co bang chung key/provider that tren may nay | Khi co `OPENAI_API_KEY`/`ANTHROPIC_API_KEY`, chay smoke va luu evidence token usage that |
 | P3 | Managed prod infra T2 | Docker lab da chay, nhung chua the claim production Strong | Cau hinh enterprise IdP/JWKS, S3 Object Lock/QLDB, KMS default/HSM, dashboard TLS/OIDC, Slack/PagerDuty, billing API that |
 | P4 | Plan-05 CI/CD release gate | Can CI chay tat ca suite va goi package | Tao/bo sung `CASAN_PLAN_05_CICD.md`; them workflow chay 218 core + optional Docker infra lab |
 | P5 | Plan-06 onboard project 2 | Chung minh harness reuse that | Chon domain thu 2, tao app/domain data/golden/corpus, dang ky registry, chay `verify-harness-reuse.sh` |
@@ -109,6 +109,7 @@ bash .specify/scripts/bash/infra-lab.sh verify
 
 - `CASAN_BACKLOG_STATUS.md` la handoff chi tiet nhat ve viec con lai.
 - `CASAN_HARDENING_STATUS.md` la nguon cho control nao da implemented+tested.
+- `CASAN_PLAN_03_CLOUD_PATCH.md` la nguon cho cloud provider patch va cach chay live khi co key.
 - Khong stage cac file audit log runtime neu chi thay doi do chay verify.
 - Backend `npm test` toan repo con blocker cu: Prisma schema MySQL nhung `setup-sqlite.mjs`
   ap migration vao SQLite. Day khong phai loi cua cac plan hardening vua lam.

casan-next-plans/CASAN_PLAN_03_CLOUD_PATCH.md

@@ -0,0 +1,55 @@
+# CASAN PLAN 03 — Cloud Provider Patch (OpenAI/Anthropic)
+
+> Status 2026-07-06: **MVP implemented + deterministic tests added**. The router
+> can call OpenAI/Anthropic when real API keys are present, while local tests
+> verify endpoint pinning, provider usage parsing, and fail-closed malformed
+> payload handling without requiring paid keys.
+
+## Delivered
+
+| Capability | Where | Verification |
+|---|---|---|
+| Model spec prefixes `openai:<model>` and `anthropic:<model>` | `.specify/scripts/bash/model-call.py` | `phase3-model-router-tests.sh` |
+| Endpoint allowlist/SSRF posture | Cloud hosts are hard-pinned to `api.openai.com` and `api.anthropic.com`; Ollama remains `127.0.0.1:11434` only | router SSRF test |
+| Honest unavailable state | Missing `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` exits non-zero; no fake PASS or fake token usage | router cloud-unavailable test |
+| Real provider token usage path | OpenAI `usage.prompt_tokens/completion_tokens`; Anthropic `usage.input_tokens/output_tokens` | deterministic monkeypatched cloud parser test |
+| Malformed provider payload fail-closed | Missing usage/schema exits non-zero via `provider_usage_invalid` | deterministic parser test |
+| Provider telemetry source tags | Usage rows write `openai_api_real_tokens` or `anthropic_api_real_tokens` | `model-call.py` provider log path |
+
+## How to run live when keys exist
+
+```bash
+cd AINative_OKR_CASAN5
+export CASAN_MODEL_PRIMARY=openai:gpt-4o-mini
+export OPENAI_API_KEY=...
+bash .specify/scripts/bash/model-router.sh /tmp/prompt.txt /tmp/out.json --role classify
+
+export CASAN_MODEL_PRIMARY=anthropic:claude-sonnet-4-5
+export ANTHROPIC_API_KEY=...
+bash .specify/scripts/bash/model-router.sh /tmp/prompt.txt /tmp/out.json --role judge
+```
+
+Expected evidence:
+
+- command exits `0` for well-formed provider output;
+- `/tmp/out.json` has non-zero `input_tokens` and `output_tokens`;
+- `.specify/logs/level5/provider-usage.jsonl` gets a row with provider
+  `openai` or `anthropic` and cost source `*_api_real_tokens`;
+- malformed or unreachable provider exits non-zero and does not fabricate a
+  successful verdict.
+
+## Remaining production work
+
+| Priority | Work | Done when |
+|---|---|---|
+| P1 | Run live smoke with real org keys on the target Mac/CI | Evidence file records real provider response, token counts, and no secret leakage |
+| P2 | Add provider cost lookup table for selected models | `cost_usd` is computed from current provider pricing instead of `0.0` |
+| P3 | Wire real provider billing usage API, not only per-call response usage | H6 reconcile uses OpenAI/Anthropic ground truth APIs with schema-versioned fetchers |
+| P4 | Add model allowlist policy for approved cloud models | Unknown cloud model names require approval or block |
+| P5 | Decide default failover order for local -> cloud or cloud -> local | `model-fallback.sh` policy is explicit per role |
+
+## Notes for Plan-02
+
+Plan-02 source generation should call `model-router.sh --role generate`, not
+provider SDKs directly. This keeps H4 prompt handling, H6 telemetry, endpoint
+allowlisting, and provider usage logging in one path.

casan-next-plans/CASAN_PLAN_07_PRODUCTION_HARDENING.md

@@ -19,7 +19,7 @@
 
 ## 2. Thang điểm sẵn sàng production (0–5, cao = tốt)
 
-> ✅ **CẬP NHẬT 2026-07-06 — Track A + C-MVP + Evidence Pack + H5/H6/deep-gap hardening + Plan-10 traceability ĐÃ LÀM + TEST (218 core checks, 0 fail; model-router riêng 10/0). Local production-like Docker infra lab cũng đã có (2/0, verify nội bộ 7/0).**
+> ✅ **CẬP NHẬT 2026-07-06 — Track A + C-MVP + Evidence Pack + H5/H6/deep-gap hardening + Plan-10 traceability ĐÃ LÀM + TEST (218 core checks, 0 fail; model-router riêng 11/0). Local production-like Docker infra lab cũng đã có (2/0, verify nội bộ 7/0).**
 > Bảng dưới có cột **Baseline → Nay**. Điểm chấm CÔNG TÂM (0–100, theo `casan_harness_assessment.md`):
 > **H4 = 80 · H5 = 76→80 ⬆ · H6 = 79→80 ⬆ · trung bình 7 harness ~80.9/100 · không còn harness nào dưới 80 → CASAN Level 4 (vững ngưỡng)**.
 > Nguồn: `00_SUBMISSION_PACKAGE/evidence/scoring-run-report.md`. Chi tiết implemented-vs-planned: `CASAN_HARDENING_STATUS.md`.