80 lines
6.3 KiB
Markdown
80 lines
6.3 KiB
Markdown
# CASAN Phase 1 Hardening — Independent Re-Assessment
|
||
|
||
**Date:** 2026-06-28
|
||
**Scope:** Control-plane harnesses H2, H4, H5, H6 (hardening that is verifiable without a running product).
|
||
**Method:** Independent review + a new **adversarial** test suite (`.specify/tests/adversarial-harness-tests.sh`) that constructs attacks/tampering and asserts the controls defend — every check would have FAILED before this work.
|
||
**Honesty note:** Scores below are earned against adversarial evidence, not against the presence of config files. H1/H3/H7 are deliberately left unchanged here — they cannot honestly exceed 80 without Phase 2 (a real product + real pipeline run). See bottom.
|
||
|
||
---
|
||
|
||
## 1. Score movement (independent)
|
||
|
||
| ID | Harness | Pre-hardening | Phase 1 | Basis for the new score |
|
||
|----|---------|:---:|:---:|---|
|
||
| H2 | Tool | 48 | **80** | Per-agent least-privilege enforced **and in the execution line of fire**; tool-call audit is now a signed, tamper-evident hash chain (re-forge rejected); every side-effecting tool must declare a rollback strategy. |
|
||
| H4 | Security | 52 | **80** | Input normalization defeats whitespace/leet/synonym/case bypasses; `pii-rules.yaml` is now wired into the live path; private-key/DB/AWS secrets blocked; output mode fails closed on secret material. |
|
||
| H5 | Governance | 62 | **80** | Audit chain is RSA-anchored — a re-forged chain is now detected (it previously passed); `approver` + `output_hash` are inside the hashed core; separation of duties (submitter ≠ approver) enforced. |
|
||
| H6 | AgentOps | 58 | **80** | Hallucination detection is real and populated (signals written to `metrics.jsonl`, alert at threshold); imported provider telemetry is the authoritative cost source (`cost_source` labels real vs estimate); latency/exit-code/alerting already real. |
|
||
| H1 | Context | 45 | 45 | Unchanged — requires Phase 2 (real orchestrator-maintained context). |
|
||
| H3 | Evaluation | 22 | 22 | Unchanged — requires Phase 2 (real app + real LLM-judge gates + real regression). |
|
||
| H7 | Orchestration | 40 | 40 | Unchanged — requires Phase 2 (wire L5 scripts into a real run, real rollback). |
|
||
|
||
**Average after Phase 1:** ~61/100. Still **CASAN Level 3** because the ceiling is held by H1/H3/H7 (and H3 < 30 remains a critical GAP). Phase 1 fixed exactly the four harnesses that are honestly fixable without a running product.
|
||
|
||
---
|
||
|
||
## 2. What changed (files)
|
||
|
||
| File | Change |
|
||
|---|---|
|
||
| `scripts/bash/security-check.sh` | `normalize_for_match` (whitespace/leet/punct folding) + `match_either`; expanded synonym blocklist; wired `pii-mask.py`; output fail-closed on secrets. |
|
||
| `scripts/bash/pii-mask.py` (new) | Policy-driven PII masker — makes `pii-rules.yaml` the source of truth. |
|
||
| `scripts/bash/governance-check.sh` | `approver`+`output_hash` in hashed core; separation of duties; RSA head-anchor signing after each append. |
|
||
| `scripts/bash/verify-audit-chain.sh` | Verifies new core + head signature; rejects re-forged chains. |
|
||
| `scripts/bash/tool-registry-gate.sh` | Per-agent permission + rollback-strategy enforcement; routes audit through the chained appender. |
|
||
| `scripts/bash/tool-audit-lib.sh` (new) | Shared chained + signed appender for `tool-calls.jsonl`. |
|
||
| `scripts/bash/verify-tool-audit.sh` (new) | Verifies the tool-call audit chain + head signature. |
|
||
| `scripts/bash/casan-harness.sh` | Calls the tool-registry gate before executing side-effecting actions (gate in line of fire). |
|
||
| `scripts/bash/agent-metrics.sh` | Hallucination detection + provider-telemetry cost source + `cost_source`/`hallucination_signals` fields. |
|
||
| `scripts/bash/hallucination-scan.py` (new) | Keyword/marker detector driven by `hallucination-tracking.yaml`. |
|
||
| `level5/tool-registry.yaml` | `allowed_agents` + rollback for all side-effecting tools. |
|
||
| `tests/run-casan4-harness-tests.sh` | Updated for new behavior; +5 assertions (35 PASS total). |
|
||
| `tests/adversarial-harness-tests.sh` (new) | 22 independent attack/tamper/authorization checks. |
|
||
|
||
---
|
||
|
||
## 3. Evidence
|
||
|
||
- `bash .specify/tests/run-casan4-harness-tests.sh` → **35 PASS / 0 FAIL**.
|
||
- `bash .specify/tests/adversarial-harness-tests.sh` → **22 PASS / 0 FAIL** (attacks defended).
|
||
|
||
Representative adversarial proofs (all previously would have failed):
|
||
- H4: `"ignore all previous instructions"`, `"1gnore prev1ous 1nstruct1ons"`, `"disregard the rules above"` → blocked.
|
||
- H4: private key fed as input → blocked (previously leaked to output in the GHCP predecessor).
|
||
- H5: edit one record + recompute the entire chain + rewrite the plain head file → `AUDIT_HEAD_SIGNATURE_INVALID` (previously passed as valid).
|
||
- H2: `CASAN_AGENT=design-agent ... deploy` → denied `unauthorized_agent`; same via `casan-harness.sh write_code` → wrapper aborts.
|
||
- H6: marker-heavy output → `hallucination_signals=4` + alert; clean output → `0`.
|
||
|
||
---
|
||
|
||
## 4. Honest residual gaps (why these are 80, not 95)
|
||
|
||
- **H4:** still rule/normalization-based, not semantic/embedding; no process sandbox/timeout for tool execution.
|
||
- **H5:** the audit signing key is local for demonstration — production requires it off-repo (KMS/HSM); no OS-level WORM; risk classifier is keyword-based.
|
||
- **H2:** agent identity is taken from the environment (trusted caller); no runtime cross-step rate-limit counter; no per-call JSON-schema validation of tool I/O.
|
||
- **H6:** word-count remains the *fallback* cost when no provider telemetry is imported; hallucination detection is keyword-based; dashboard is static, not live.
|
||
|
||
---
|
||
|
||
## 5. Remaining path to "all H1–H7 > 80" (Phase 2 — requires a real run)
|
||
|
||
H1, H3, H7 are gated on the same missing thing: **there is no real product and no real pipeline run.** To raise them honestly:
|
||
|
||
1. Build a real, testable OKR app (NestJS+Prisma backend, React+Vite frontend) from `docs/input/okr-requirement.md`.
|
||
2. Run the reviewer agents as real LLM-as-judge gates that can emit REJECTED and trigger auto-retry/BACK-TO-PLAN (retire the hardcoded `approved` generator).
|
||
3. Run the test kit against the real app (real unit + e2e) → real golden/regression for H3.
|
||
4. Wire `model-fallback` / `rollback-manager` / `drift-detect` into the orchestrator; make rollback a real undo.
|
||
5. Produce `pipeline-context.yaml` via incremental orchestrator updates during the run (retire `generate-casan-demo-context.py`).
|
||
|
||
Only Phase 2 raises the overall average above 80 and clears the H3 critical GAP.
|