312 lines
11 KiB
Markdown
312 lines
11 KiB
Markdown
# Prompt For Claude — CASAN Phase 3 Wave 1 Real Core Improvements
|
|
|
|
You are working inside this repo:
|
|
|
|
`/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5`
|
|
|
|
Read these files first:
|
|
|
|
- `docs/output/casan/phase3-push-to-90-plan.md`
|
|
- `docs/output/casan/phase3-security-hardening-plan.md`
|
|
- `docs/output/casan/phase3-preflight.md`
|
|
- `docs/output/casan/app-evidence/MANIFEST.md`
|
|
- `docs/output/output_logs/001-okr-web-app/pipeline-context.yaml`
|
|
|
|
Execute Phase 3 Wave 1 as real engineering work. Do not produce demo-grade controls. Do not claim Phase 3 is complete.
|
|
|
|
## Preflight Facts To Respect
|
|
|
|
- This is a git repo on branch `main`.
|
|
- Current preflight has untracked Phase 3 plan/evidence files.
|
|
- Mac runtime: Node v24.12.0, npm 11.7.0, Python 3.9.0.
|
|
- Linux server runtime: Ubuntu 26.04, Node v22.23.1, npm 11.17.0, Python 3.14.4, Ollama 0.30.11.
|
|
- Ollama runs on Linux server `192.168.1.5`, not directly on Mac.
|
|
- Mac accesses Linux Ollama through SSH tunnel: `http://127.0.0.1:11434`.
|
|
- Primary local model is `ollama:ornith:9b`.
|
|
- `ornith:9b` digest: `a75697c145891910e312c95e4a9fc1ccb8653e5ef543b23b0403a4665b82fd91`.
|
|
- `gemma4:12b` is also available but is not the declared primary.
|
|
- Cloud env is unavailable: `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` are unset.
|
|
- Current captured `npm test` and `npm run build` failed because `prisma: command not found`.
|
|
- Current `npm audit --workspaces --audit-level=high` failed with 5 high and 7 moderate vulnerabilities.
|
|
|
|
Before implementing, resolve or clearly document the local dependency issue:
|
|
|
|
```bash
|
|
npm install
|
|
npm test
|
|
npm run build
|
|
```
|
|
|
|
Capture refreshed logs under:
|
|
|
|
`docs/output/casan/phase3-evidence/wave1/`
|
|
|
|
If tests/build still fail, stop implementation and write a blocker report. Do not build Phase 3 on a broken baseline unless the failure is explicitly scoped and accepted.
|
|
|
|
## Absolute Integrity Rules
|
|
|
|
- Do not fake PASS, APPROVED, token usage, model verdicts, cost, drift, rollback, or audit results.
|
|
- Do not hardcode expected PASS results.
|
|
- Do not recycle sample token/cost numbers.
|
|
- Every test you add must be able to fail.
|
|
- Do not use `.specify/tests/generate-casan-demo-context.py`.
|
|
- Do not call regex-only logic semantic or model-based.
|
|
- Do not claim cloud/hybrid judging; current run is local-only unless real cloud keys are present.
|
|
- Do not claim H5 WORM/KMS; AWS/KMS/Object Lock is not configured.
|
|
- Do not claim target scores as achieved. Only write achieved scores after a real audit.
|
|
- Keep evidence under `docs/output/casan/phase3-evidence/wave1/`.
|
|
- If infrastructure is missing, write `BLOCKED` or `SKIPPED` with a reason. Do not mark it pass.
|
|
|
|
## Core Modification Policy
|
|
|
|
For Phase 3, you are explicitly allowed and expected to modify core CASAN harness files when that is the correct engineering path.
|
|
|
|
This includes files under:
|
|
|
|
- `.specify/scripts/bash/*`
|
|
- `.specify/tests/*`
|
|
- `scripts/run-casan-pipeline.mjs`
|
|
- `scripts/casan-step.mjs`
|
|
|
|
Do not avoid core changes by adding superficial wrappers if the real control belongs in the core harness.
|
|
|
|
This is not a demo-grade improvement task. Implement real controls in the actual execution path. If a claimed H1-H7 improvement is not wired into the path that the harness/pipeline actually uses, it does not count.
|
|
|
|
Rules for core changes:
|
|
|
|
- Do not weaken existing security, governance, tool-audit, rollback, drift, or agent-metrics checks.
|
|
- Do not edit tests to hide failures.
|
|
- Do not loosen adversarial tests.
|
|
- Do not replace real command execution with canned output.
|
|
- Do not hardcode PASS/APPROVED/verdict/token/cost values.
|
|
- Every core behavior change must include fail-able evidence:
|
|
1. demonstrate the old weakness or failing case,
|
|
2. implement the fix,
|
|
3. demonstrate the case now fails closed or passes correctly.
|
|
- If a core script changes, run relevant existing harness checks afterward:
|
|
- `bash .specify/tests/run-casan4-harness-tests.sh`
|
|
- `bash .specify/tests/adversarial-harness-tests.sh`
|
|
- `bash .specify/scripts/bash/verify-audit-chain.sh`
|
|
- `bash .specify/scripts/bash/verify-tool-audit.sh`
|
|
- If a test cannot run because of environment constraints, record that as a blocker or limitation, not a pass.
|
|
|
|
For every core harness modification, document in `docs/output/casan/phase3-wave1-results.md`:
|
|
|
|
- file changed,
|
|
- behavior changed,
|
|
- threat/control improved,
|
|
- tests/evidence proving the change,
|
|
- compatibility result with existing harness tests.
|
|
|
|
## Required Environment
|
|
|
|
Assume operator has opened this tunnel in another terminal:
|
|
|
|
```bash
|
|
ssh -N -L 11434:127.0.0.1:11434 thanhnv@192.168.1.5
|
|
```
|
|
|
|
Use:
|
|
|
|
```bash
|
|
export CASAN_MODEL_BACKEND=local
|
|
export CASAN_MODEL_PRIMARY=ollama:ornith:9b
|
|
export CASAN_MODEL_SECONDARY=ollama:ornith:9b
|
|
```
|
|
|
|
Verify before model work:
|
|
|
|
```bash
|
|
curl -sS http://127.0.0.1:11434/api/tags
|
|
curl -sS http://127.0.0.1:11434/api/generate \
|
|
-H 'Content-Type: application/json' \
|
|
-d '{"model":"ornith:9b","prompt":"Return exactly: OK","stream":false}'
|
|
```
|
|
|
|
## Wave 1 Scope
|
|
|
|
Implement the foundation and minimum defensible security layer, wired into real execution paths.
|
|
|
|
### A. Model Router Foundation
|
|
|
|
Create or modify as needed:
|
|
|
|
- `.specify/scripts/bash/model-router.sh`
|
|
- `.specify/scripts/bash/model-call.py`
|
|
- any core script that must call the router for the control to be real
|
|
|
|
Interface:
|
|
|
|
```bash
|
|
.specify/scripts/bash/model-router.sh <prompt-file> <out-json> [--role classify|judge|generate]
|
|
```
|
|
|
|
Required behavior:
|
|
|
|
- Support `ollama:<model>` via `POST http://127.0.0.1:11434/api/generate`.
|
|
- Use `ornith:9b` as the primary local model.
|
|
- Cloud backends may be implemented only if real env keys exist; otherwise report unavailable honestly.
|
|
- Output JSON must include:
|
|
- `text`
|
|
- `model_id`
|
|
- `role`
|
|
- `input_tokens`
|
|
- `output_tokens`
|
|
- `latency_ms`
|
|
- `temperature`
|
|
- `route`
|
|
- `verdict` when role is `classify` or `judge`
|
|
- For Ollama, use real `prompt_eval_count` and `eval_count`.
|
|
- Append real usage to `.specify/logs/level5/provider-usage.jsonl`.
|
|
- Do not recycle sample values.
|
|
- On backend/model failure, return nonzero with a clear error. Do not emit a fake fallback success.
|
|
|
|
### B. WP-S1 Router Hardening
|
|
|
|
Implement hardening in the real model path:
|
|
|
|
- Delimit untrusted content:
|
|
`<<<UNTRUSTED>>> ... <<<END_UNTRUSTED>>>`
|
|
- `classify` output must be exactly `INJECTION` or `SAFE`.
|
|
- `judge` output must be exactly `APPROVED` or `REJECTED`.
|
|
- Any malformed model output must fail closed.
|
|
- Never log API keys, secrets, raw Authorization headers, or `.env` contents.
|
|
- For future cloud egress, mask PII before sending. Since cloud keys are unset now, test this path as unavailable or with a local dry-run payload capture.
|
|
- Enforce endpoint allowlist:
|
|
- `api.anthropic.com`
|
|
- `api.openai.com`
|
|
- `127.0.0.1:11434`
|
|
- Reject arbitrary model URLs and metadata IP attempts.
|
|
- Use `temperature=0` for classify/judge.
|
|
- Write router audit/usage records without leaking prompt secrets.
|
|
|
|
### C. Wire The Router Into Real Harness Paths
|
|
|
|
Do not leave the router as an unused utility.
|
|
|
|
For Wave 1, wire at least one real path:
|
|
|
|
1. H6 usage path:
|
|
- Ensure a real model-router call can feed `.specify/logs/level5/provider-usage.jsonl`.
|
|
- Ensure `agent-metrics.sh` can consume real provider telemetry for a matching step where available.
|
|
|
|
2. H7 fallback path:
|
|
- Replace fake fallback triggers like `bash -c "exit 9"` in `scripts/run-casan-pipeline.mjs` with a real failing model-router primary route, then fallback to `ollama:ornith:9b` only if the model is available.
|
|
- If the model is unavailable, record H7 fallback as blocked, not passed.
|
|
|
|
3. Optional H4/H3 path if feasible in Wave 1:
|
|
- Wire classify/judge into `security-check.sh` or `casan-step.mjs` only if you can prove fail-before/fix/pass in this turn.
|
|
- If not feasible, document it as Wave 2 work. Do not claim H4/H3 model-based improvement.
|
|
|
|
### D. Fail-able Router Tests
|
|
|
|
Create or modify:
|
|
|
|
- `.specify/tests/phase3-model-router-tests.sh`
|
|
|
|
Tests must cover:
|
|
|
|
1. `ollama:ornith:9b` generate works through tunnel, or fails with clear reason if tunnel unavailable.
|
|
2. Router records real `input_tokens` and `output_tokens` from Ollama.
|
|
3. Classifier/judge malformed output fails closed.
|
|
4. SSRF-like endpoint/model config is rejected.
|
|
5. Fake API key pattern does not appear in `.specify/logs`.
|
|
6. Cloud backend reports unavailable honestly while keys are unset.
|
|
7. A deliberate failing primary route causes fallback only through the real router path, not `exit 9`.
|
|
|
|
No hardcoded PASS.
|
|
|
|
### E. Context Validation
|
|
|
|
Create or modify:
|
|
|
|
- `.specify/scripts/bash/context-validate.sh`
|
|
- `scripts/run-casan-pipeline.mjs` if needed to add `generated_at` or invoke validation in the real pipeline
|
|
|
|
Behavior:
|
|
|
|
- Default context:
|
|
`docs/output/output_logs/001-okr-web-app/pipeline-context.yaml`
|
|
- Accept optional context path arg.
|
|
- Fail if any `artifact:` path is missing.
|
|
- Fail if any `trace_file:` path is missing.
|
|
- Warn if entries have no `generated_at`.
|
|
- If adding `generated_at`, do it in the real `appendContext` path.
|
|
|
|
Create fail/pass evidence:
|
|
|
|
- copy context to temp,
|
|
- inject one missing artifact path,
|
|
- show validator exits nonzero,
|
|
- run validator on real context and show pass.
|
|
|
|
### F. Supply Chain Baseline
|
|
|
|
Current `npm audit --workspaces --audit-level=high` fails with 5 high vulnerabilities.
|
|
|
|
For Wave 1:
|
|
|
|
- Do not claim supply-chain gate green.
|
|
- Either fix high vulnerabilities without breaking app/test/build, or record them as a blocker for WP-S3.
|
|
- If you run `npm audit fix` or upgrade major dependencies, run full app tests/build afterward and capture logs.
|
|
|
|
### G. Wave 1 Results
|
|
|
|
Create:
|
|
|
|
- `docs/output/casan/phase3-wave1-results.md`
|
|
|
|
Include:
|
|
|
|
- what was implemented,
|
|
- what was not implemented,
|
|
- infrastructure used,
|
|
- exact commands run,
|
|
- exact evidence files,
|
|
- real test results,
|
|
- core files modified and why,
|
|
- current blockers,
|
|
- remaining work for WP-A/B/C/D/E/F/G and WP-S2..S8,
|
|
- explicit statement that no average >90 score is claimed yet.
|
|
|
|
## Required Commands And Evidence
|
|
|
|
Capture logs under:
|
|
|
|
`docs/output/casan/phase3-evidence/wave1/`
|
|
|
|
Run and capture:
|
|
|
|
```bash
|
|
npm install
|
|
npm test
|
|
npm run build
|
|
bash .specify/tests/phase3-model-router-tests.sh
|
|
bash .specify/scripts/bash/context-validate.sh docs/output/output_logs/001-okr-web-app/pipeline-context.yaml
|
|
```
|
|
|
|
If any core harness script changed, also run and capture:
|
|
|
|
```bash
|
|
bash .specify/tests/run-casan4-harness-tests.sh
|
|
bash .specify/tests/adversarial-harness-tests.sh
|
|
bash .specify/scripts/bash/verify-audit-chain.sh
|
|
bash .specify/scripts/bash/verify-tool-audit.sh
|
|
```
|
|
|
|
If baseline is green and model tunnel is available, run and capture:
|
|
|
|
```bash
|
|
node scripts/run-casan-pipeline.mjs
|
|
```
|
|
|
|
## Deliverables
|
|
|
|
- Working Wave 1 code.
|
|
- Real evidence logs under `docs/output/casan/phase3-evidence/wave1/`.
|
|
- `docs/output/casan/phase3-wave1-results.md`.
|
|
- No fabricated evidence.
|
|
- No target score claimed as achieved.
|
|
|
|
If blocked, stop and write a precise blocker report instead of manufacturing evidence.
|