gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update optimize wave3 (need update wave 4 to wave 8)

Browse Source
This commit is contained in:
thanhnv
2026-07-01 00:02:30 +09:00
parent 07ac1bdcdd
commit eaf919e744
140 changed files with 3780 additions and 844 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/docs/output/casan/phase3-claude-wave1-prompt.md
+311
View File
@@ -0,0 +1,311 @@
# Prompt For Claude — CASAN Phase 3 Wave 1 Real Core Improvements
You are working inside this repo:
`/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5`
Read these files first:
- `docs/output/casan/phase3-push-to-90-plan.md`
- `docs/output/casan/phase3-security-hardening-plan.md`
- `docs/output/casan/phase3-preflight.md`
- `docs/output/casan/app-evidence/MANIFEST.md`
- `docs/output/output_logs/001-okr-web-app/pipeline-context.yaml`
Execute Phase 3 Wave 1 as real engineering work. Do not produce demo-grade controls. Do not claim Phase 3 is complete.
## Preflight Facts To Respect
- This is a git repo on branch `main`.
- Current preflight has untracked Phase 3 plan/evidence files.
- Mac runtime: Node v24.12.0, npm 11.7.0, Python 3.9.0.
- Linux server runtime: Ubuntu 26.04, Node v22.23.1, npm 11.17.0, Python 3.14.4, Ollama 0.30.11.
- Ollama runs on Linux server `192.168.1.5`, not directly on Mac.
- Mac accesses Linux Ollama through SSH tunnel: `http://127.0.0.1:11434`.
- Primary local model is `ollama:ornith:9b`.
- `ornith:9b` digest: `a75697c145891910e312c95e4a9fc1ccb8653e5ef543b23b0403a4665b82fd91`.
- `gemma4:12b` is also available but is not the declared primary.
- Cloud env is unavailable: `ANTHROPIC_API_KEY` and `OPENAI_API_KEY` are unset.
- Current captured `npm test` and `npm run build` failed because `prisma: command not found`.
- Current `npm audit --workspaces --audit-level=high` failed with 5 high and 7 moderate vulnerabilities.
Before implementing, resolve or clearly document the local dependency issue:
```bash
npm install
npm test
npm run build
```
Capture refreshed logs under:
`docs/output/casan/phase3-evidence/wave1/`
If tests/build still fail, stop implementation and write a blocker report. Do not build Phase 3 on a broken baseline unless the failure is explicitly scoped and accepted.
## Absolute Integrity Rules
- Do not fake PASS, APPROVED, token usage, model verdicts, cost, drift, rollback, or audit results.
- Do not hardcode expected PASS results.
- Do not recycle sample token/cost numbers.
- Every test you add must be able to fail.
- Do not use `.specify/tests/generate-casan-demo-context.py`.
- Do not call regex-only logic semantic or model-based.
- Do not claim cloud/hybrid judging; current run is local-only unless real cloud keys are present.
- Do not claim H5 WORM/KMS; AWS/KMS/Object Lock is not configured.
- Do not claim target scores as achieved. Only write achieved scores after a real audit.
- Keep evidence under `docs/output/casan/phase3-evidence/wave1/`.
- If infrastructure is missing, write `BLOCKED` or `SKIPPED` with a reason. Do not mark it pass.
## Core Modification Policy
For Phase 3, you are explicitly allowed and expected to modify core CASAN harness files when that is the correct engineering path.
This includes files under:
- `.specify/scripts/bash/*`
- `.specify/tests/*`
- `scripts/run-casan-pipeline.mjs`
- `scripts/casan-step.mjs`
Do not avoid core changes by adding superficial wrappers if the real control belongs in the core harness.
This is not a demo-grade improvement task. Implement real controls in the actual execution path. If a claimed H1-H7 improvement is not wired into the path that the harness/pipeline actually uses, it does not count.
Rules for core changes:
- Do not weaken existing security, governance, tool-audit, rollback, drift, or agent-metrics checks.
- Do not edit tests to hide failures.
- Do not loosen adversarial tests.
- Do not replace real command execution with canned output.
- Do not hardcode PASS/APPROVED/verdict/token/cost values.
- Every core behavior change must include fail-able evidence:
1. demonstrate the old weakness or failing case,
2. implement the fix,
3. demonstrate the case now fails closed or passes correctly.
- If a core script changes, run relevant existing harness checks afterward:
- `bash .specify/tests/run-casan4-harness-tests.sh`
- `bash .specify/tests/adversarial-harness-tests.sh`
- `bash .specify/scripts/bash/verify-audit-chain.sh`
- `bash .specify/scripts/bash/verify-tool-audit.sh`
- If a test cannot run because of environment constraints, record that as a blocker or limitation, not a pass.
For every core harness modification, document in `docs/output/casan/phase3-wave1-results.md`:
- file changed,
- behavior changed,
- threat/control improved,
- tests/evidence proving the change,
- compatibility result with existing harness tests.
## Required Environment
Assume operator has opened this tunnel in another terminal:
```bash
ssh -N -L 11434:127.0.0.1:11434 thanhnv@192.168.1.5
```
Use:
```bash
export CASAN_MODEL_BACKEND=local
export CASAN_MODEL_PRIMARY=ollama:ornith:9b
export CASAN_MODEL_SECONDARY=ollama:ornith:9b
```
Verify before model work:
```bash
curl -sS http://127.0.0.1:11434/api/tags
curl -sS http://127.0.0.1:11434/api/generate \
-H 'Content-Type: application/json' \
-d '{"model":"ornith:9b","prompt":"Return exactly: OK","stream":false}'
```
## Wave 1 Scope
Implement the foundation and minimum defensible security layer, wired into real execution paths.
### A. Model Router Foundation
Create or modify as needed:
- `.specify/scripts/bash/model-router.sh`
- `.specify/scripts/bash/model-call.py`
- any core script that must call the router for the control to be real
Interface:
```bash
.specify/scripts/bash/model-router.sh <prompt-file> <out-json> [--role classify|judge|generate]
```
Required behavior:
- Support `ollama:<model>` via `POST http://127.0.0.1:11434/api/generate`.
- Use `ornith:9b` as the primary local model.
- Cloud backends may be implemented only if real env keys exist; otherwise report unavailable honestly.
- Output JSON must include:
- `text`
- `model_id`
- `role`
- `input_tokens`
- `output_tokens`
- `latency_ms`
- `temperature`
- `route`
- `verdict` when role is `classify` or `judge`
- For Ollama, use real `prompt_eval_count` and `eval_count`.
- Append real usage to `.specify/logs/level5/provider-usage.jsonl`.
- Do not recycle sample values.
- On backend/model failure, return nonzero with a clear error. Do not emit a fake fallback success.
### B. WP-S1 Router Hardening
Implement hardening in the real model path:
- Delimit untrusted content:
`<<<UNTRUSTED>>> ... <<<END_UNTRUSTED>>>`
- `classify` output must be exactly `INJECTION` or `SAFE`.
- `judge` output must be exactly `APPROVED` or `REJECTED`.
- Any malformed model output must fail closed.
- Never log API keys, secrets, raw Authorization headers, or `.env` contents.
- For future cloud egress, mask PII before sending. Since cloud keys are unset now, test this path as unavailable or with a local dry-run payload capture.
- Enforce endpoint allowlist:
- `api.anthropic.com`
- `api.openai.com`
- `127.0.0.1:11434`
- Reject arbitrary model URLs and metadata IP attempts.
- Use `temperature=0` for classify/judge.
- Write router audit/usage records without leaking prompt secrets.
### C. Wire The Router Into Real Harness Paths
Do not leave the router as an unused utility.
For Wave 1, wire at least one real path:
1. H6 usage path:
- Ensure a real model-router call can feed `.specify/logs/level5/provider-usage.jsonl`.
- Ensure `agent-metrics.sh` can consume real provider telemetry for a matching step where available.
2. H7 fallback path:
- Replace fake fallback triggers like `bash -c "exit 9"` in `scripts/run-casan-pipeline.mjs` with a real failing model-router primary route, then fallback to `ollama:ornith:9b` only if the model is available.
- If the model is unavailable, record H7 fallback as blocked, not passed.
3. Optional H4/H3 path if feasible in Wave 1:
- Wire classify/judge into `security-check.sh` or `casan-step.mjs` only if you can prove fail-before/fix/pass in this turn.
- If not feasible, document it as Wave 2 work. Do not claim H4/H3 model-based improvement.
### D. Fail-able Router Tests
Create or modify:
- `.specify/tests/phase3-model-router-tests.sh`
Tests must cover:
1. `ollama:ornith:9b` generate works through tunnel, or fails with clear reason if tunnel unavailable.
2. Router records real `input_tokens` and `output_tokens` from Ollama.
3. Classifier/judge malformed output fails closed.
4. SSRF-like endpoint/model config is rejected.
5. Fake API key pattern does not appear in `.specify/logs`.
6. Cloud backend reports unavailable honestly while keys are unset.
7. A deliberate failing primary route causes fallback only through the real router path, not `exit 9`.
No hardcoded PASS.
### E. Context Validation
Create or modify:
- `.specify/scripts/bash/context-validate.sh`
- `scripts/run-casan-pipeline.mjs` if needed to add `generated_at` or invoke validation in the real pipeline
Behavior:
- Default context:
`docs/output/output_logs/001-okr-web-app/pipeline-context.yaml`
- Accept optional context path arg.
- Fail if any `artifact:` path is missing.
- Fail if any `trace_file:` path is missing.
- Warn if entries have no `generated_at`.
- If adding `generated_at`, do it in the real `appendContext` path.
Create fail/pass evidence:
- copy context to temp,
- inject one missing artifact path,
- show validator exits nonzero,
- run validator on real context and show pass.
### F. Supply Chain Baseline
Current `npm audit --workspaces --audit-level=high` fails with 5 high vulnerabilities.
For Wave 1:
- Do not claim supply-chain gate green.
- Either fix high vulnerabilities without breaking app/test/build, or record them as a blocker for WP-S3.
- If you run `npm audit fix` or upgrade major dependencies, run full app tests/build afterward and capture logs.
### G. Wave 1 Results
Create:
- `docs/output/casan/phase3-wave1-results.md`
Include:
- what was implemented,
- what was not implemented,
- infrastructure used,
- exact commands run,
- exact evidence files,
- real test results,
- core files modified and why,
- current blockers,
- remaining work for WP-A/B/C/D/E/F/G and WP-S2..S8,
- explicit statement that no average >90 score is claimed yet.
## Required Commands And Evidence
Capture logs under:
`docs/output/casan/phase3-evidence/wave1/`
Run and capture:
```bash
npm install
npm test
npm run build
bash .specify/tests/phase3-model-router-tests.sh
bash .specify/scripts/bash/context-validate.sh docs/output/output_logs/001-okr-web-app/pipeline-context.yaml
```
If any core harness script changed, also run and capture:
```bash
bash .specify/tests/run-casan4-harness-tests.sh
bash .specify/tests/adversarial-harness-tests.sh
bash .specify/scripts/bash/verify-audit-chain.sh
bash .specify/scripts/bash/verify-tool-audit.sh
```
If baseline is green and model tunnel is available, run and capture:
```bash
node scripts/run-casan-pipeline.mjs
```
## Deliverables
- Working Wave 1 code.
- Real evidence logs under `docs/output/casan/phase3-evidence/wave1/`.
- `docs/output/casan/phase3-wave1-results.md`.
- No fabricated evidence.
- No target score claimed as achieved.
If blocked, stop and write a precise blocker report instead of manufacturing evidence.