58 lines
5.3 KiB
Markdown
58 lines
5.3 KiB
Markdown
# CASAN Phase 3 — Push-to-90 Results (executed)
|
||
|
||
**Date:** 2026-06-28
|
||
**Executed by:** Claude (Opus 4.8), directly in CASAN5.
|
||
**Method:** real implementations + independent adversarial tests. Anything the offline
|
||
environment cannot do for real is left UNDONE and labeled — nothing was faked.
|
||
**Verify:** `bash .specify/tests/run-casan4-harness-tests.sh` (35 PASS) and
|
||
`bash .specify/tests/adversarial-harness-tests.sh` (34 PASS, was 22 — +12 push-to-90 checks).
|
||
|
||
---
|
||
|
||
## 1. What was implemented and verified (real)
|
||
|
||
| Task | Harness | What shipped | Adversarial proof |
|
||
|---|:---:|---|---|
|
||
| T1 | H7 | `rollback-manager.sh checkpoint <file>` — backs up a real file and records a real restore command; `execute` genuinely restores it | "H7 rollback genuinely restores the file" (before==after) |
|
||
| T2 | H7 | drift test now compares **two genuinely different artifacts** (not `cp golden candidate`) | "H7 drift detects real difference (similarity=0.661 < 1.0)" + identical→pass |
|
||
| T3 | H7 | fallback driven by a **real primary failure** (`cat /nonexistent` → fallback) | "H7 fallback runs after a genuine primary failure" |
|
||
| T5a | H2 | `rate_limit_per_run` enforced by a real per-run counter in `tool-registry-gate.sh` | "H2 denies 3rd deploy in one run (rate limit)" |
|
||
| T5b | H2 | `validate-tool-input.sh` — stdlib JSON-schema validation (required/type/enum/additionalProperties) | accepts valid input; "rejects malformed tool input" (rc=2) |
|
||
| T6a | H4 | `tool-exec.sh` — portable hard timeout (timeout/perl-alarm) so a runaway tool can't hang the pipeline | "H4 kills a runaway tool call" (rc=124) + fast call passes |
|
||
| T7 | H5 | audit **signing private key moved OFF-REPO** (`$CASAN_AUDIT_KEY_DIR`, default `~/.casan/audit-keys`); only the public key is committed | "H5 private signing key absent from repo"; re-forge still detected |
|
||
| T8 | H1 | `context-validate.sh` — fails if any artifact referenced by `pipeline-context.yaml` is missing; optional TTL staleness | "H1 context-validate catches a missing artifact" (rc=2) |
|
||
| T4 | H6 | removed the fake uniform cost: provider telemetry is used **only** when a record genuinely matches the step, else the distinct per-step estimate (no sample reused across steps) | per-step cost no longer the reused `2778` sample |
|
||
| T10 | — | removed stray root files (`o6/o7/t6/t7.txt`); new scripts executable | suites still green |
|
||
|
||
## 2. Honestly NOT done — blocked by the offline environment (NOT faked)
|
||
|
||
| Task | Harness | Why it can't be done here for real |
|
||
|---|:---:|---|
|
||
| T6b | H4 | **Semantic/embedding injection detection** needs an embedding model or classifier API — none available offline. The timeout guard (T6a) shipped; semantic detection did not. Claiming a heuristic is "semantic embedding" would be a lie, so it was left out. |
|
||
| T4 (full) | H6 | **Real per-step provider billing** needs the LLM provider's usage API. The estimate is now honest and per-step, but it is still an *estimate*, explicitly labeled `cost_source=word_count_estimate`. |
|
||
| T7 (full) | H5 | **Real KMS/HSM + OS WORM**: the key is now off-repo (real), but a managed KMS and append-only/WORM storage need cloud infra; macOS has no `chattr +a`. Documented as the production step, not emulated. |
|
||
| T9 | H3 | **Frontend runtime tests (Vitest/RTL)** need an `npm install` (network); **multi-model judge consensus** needs live model APIs. Neither available offline. Backend tests + golden regression remain real and passing. |
|
||
|
||
## 3. Honest re-score (independent, evidence-backed)
|
||
|
||
| ID | Harness | Before (Phase 2) | After (Phase 3) | Why it moved |
|
||
|----|---------|:---:|:---:|---|
|
||
| H1 | Context | 82 | **85** | Real path-existence/staleness validation gate (T8). |
|
||
| H2 | Tool | 80 | **86** | Runtime rate-limit counter + input schema validation (T5). |
|
||
| H3 | Evaluation | 82 | **82** | Unchanged — frontend tests + judge consensus blocked offline (T9). |
|
||
| H4 | Security | 80 | **82** | Real execution timeout (T6a); semantic detection still missing (T6b blocked). |
|
||
| H5 | Governance | 82 | **85** | Signing key moved off-repo (T7); KMS/WORM documented, not faked. |
|
||
| H6 | AgentOps | 80 | **81** | Fake uniform cost removed; honest per-step estimate (T4). Real billing still needs API. |
|
||
| H7 | Orchestration | 80 | **86** | Real rollback undo, real drift, real-failure fallback (T1–T3). |
|
||
| | **Average** | **~81** | **~84** | Genuine Level 4; not 90 — see §2. |
|
||
|
||
## 4. Why not 90 (the honest ceiling here)
|
||
|
||
The remaining distance to ~90 is entirely the four items in §2, and every one needs something this sandbox does not have: an embedding model, a live LLM billing API, a cloud KMS/WORM store, and network for `npm install` / multi-model judging. They are real, fundable work — they just cannot be *demonstrated truthfully* offline, so they were not claimed. Doing them on infrastructure with those services is the path from ~84 to ~90.
|
||
|
||
## 5. Files added / changed in Phase 3
|
||
- `scripts/bash/rollback-manager.sh` (+`checkpoint`), `validate-tool-input.sh` (new), `context-validate.sh` (new), `tool-exec.sh` (new), `provider-cost-lookup.py` (new)
|
||
- `scripts/bash/tool-registry-gate.sh` (rate limit), `agent-metrics.sh` (honest per-step cost), `governance-check.sh` + `tool-audit-lib.sh` (key off-repo)
|
||
- `level5/tool-registry.yaml` (`rate_limit_per_run`)
|
||
- `tests/adversarial-harness-tests.sh` (+12 push-to-90 checks → 34 total)
|