gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
a832c40bdb0c56d3db3eafb24416e8ff89f7a3d8
CASAN/AINative_OKR_CASAN5/docs/output/casan/phase3-claude-wave1-prompt.md
T

11 KiB
Raw Blame History

Prompt For Claude — CASAN Phase 3 Wave 1 Real Core Improvements

You are working inside this repo:

/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5

Read these files first:

  • docs/output/casan/phase3-push-to-90-plan.md
  • docs/output/casan/phase3-security-hardening-plan.md
  • docs/output/casan/phase3-preflight.md
  • docs/output/casan/app-evidence/MANIFEST.md
  • docs/output/output_logs/001-okr-web-app/pipeline-context.yaml

Execute Phase 3 Wave 1 as real engineering work. Do not produce demo-grade controls. Do not claim Phase 3 is complete.

Preflight Facts To Respect

  • This is a git repo on branch main.
  • Current preflight has untracked Phase 3 plan/evidence files.
  • Mac runtime: Node v24.12.0, npm 11.7.0, Python 3.9.0.
  • Linux server runtime: Ubuntu 26.04, Node v22.23.1, npm 11.17.0, Python 3.14.4, Ollama 0.30.11.
  • Ollama runs on Linux server 192.168.1.5, not directly on Mac.
  • Mac accesses Linux Ollama through SSH tunnel: http://127.0.0.1:11434.
  • Primary local model is ollama:ornith:9b.
  • ornith:9b digest: a75697c145891910e312c95e4a9fc1ccb8653e5ef543b23b0403a4665b82fd91.
  • gemma4:12b is also available but is not the declared primary.
  • Cloud env is unavailable: ANTHROPIC_API_KEY and OPENAI_API_KEY are unset.
  • Current captured npm test and npm run build failed because prisma: command not found.
  • Current npm audit --workspaces --audit-level=high failed with 5 high and 7 moderate vulnerabilities.

Before implementing, resolve or clearly document the local dependency issue:

npm install
npm test
npm run build

Capture refreshed logs under:

docs/output/casan/phase3-evidence/wave1/

If tests/build still fail, stop implementation and write a blocker report. Do not build Phase 3 on a broken baseline unless the failure is explicitly scoped and accepted.

Absolute Integrity Rules

  • Do not fake PASS, APPROVED, token usage, model verdicts, cost, drift, rollback, or audit results.
  • Do not hardcode expected PASS results.
  • Do not recycle sample token/cost numbers.
  • Every test you add must be able to fail.
  • Do not use .specify/tests/generate-casan-demo-context.py.
  • Do not call regex-only logic semantic or model-based.
  • Do not claim cloud/hybrid judging; current run is local-only unless real cloud keys are present.
  • Do not claim H5 WORM/KMS; AWS/KMS/Object Lock is not configured.
  • Do not claim target scores as achieved. Only write achieved scores after a real audit.
  • Keep evidence under docs/output/casan/phase3-evidence/wave1/.
  • If infrastructure is missing, write BLOCKED or SKIPPED with a reason. Do not mark it pass.

Core Modification Policy

For Phase 3, you are explicitly allowed and expected to modify core CASAN harness files when that is the correct engineering path.

This includes files under:

  • .specify/scripts/bash/*
  • .specify/tests/*
  • scripts/run-casan-pipeline.mjs
  • scripts/casan-step.mjs

Do not avoid core changes by adding superficial wrappers if the real control belongs in the core harness.

This is not a demo-grade improvement task. Implement real controls in the actual execution path. If a claimed H1-H7 improvement is not wired into the path that the harness/pipeline actually uses, it does not count.

Rules for core changes:

  • Do not weaken existing security, governance, tool-audit, rollback, drift, or agent-metrics checks.
  • Do not edit tests to hide failures.
  • Do not loosen adversarial tests.
  • Do not replace real command execution with canned output.
  • Do not hardcode PASS/APPROVED/verdict/token/cost values.
  • Every core behavior change must include fail-able evidence:
    1. demonstrate the old weakness or failing case,
    2. implement the fix,
    3. demonstrate the case now fails closed or passes correctly.
  • If a core script changes, run relevant existing harness checks afterward:
    • bash .specify/tests/run-casan4-harness-tests.sh
    • bash .specify/tests/adversarial-harness-tests.sh
    • bash .specify/scripts/bash/verify-audit-chain.sh
    • bash .specify/scripts/bash/verify-tool-audit.sh
  • If a test cannot run because of environment constraints, record that as a blocker or limitation, not a pass.

For every core harness modification, document in docs/output/casan/phase3-wave1-results.md:

  • file changed,
  • behavior changed,
  • threat/control improved,
  • tests/evidence proving the change,
  • compatibility result with existing harness tests.

Required Environment

Assume operator has opened this tunnel in another terminal:

ssh -N -L 11434:127.0.0.1:11434 thanhnv@192.168.1.5

Use:

export CASAN_MODEL_BACKEND=local
export CASAN_MODEL_PRIMARY=ollama:ornith:9b
export CASAN_MODEL_SECONDARY=ollama:ornith:9b

Verify before model work:

curl -sS http://127.0.0.1:11434/api/tags
curl -sS http://127.0.0.1:11434/api/generate \
  -H 'Content-Type: application/json' \
  -d '{"model":"ornith:9b","prompt":"Return exactly: OK","stream":false}'

Wave 1 Scope

Implement the foundation and minimum defensible security layer, wired into real execution paths.

A. Model Router Foundation

Create or modify as needed:

  • .specify/scripts/bash/model-router.sh
  • .specify/scripts/bash/model-call.py
  • any core script that must call the router for the control to be real

Interface:

.specify/scripts/bash/model-router.sh <prompt-file> <out-json> [--role classify|judge|generate]

Required behavior:

  • Support ollama:<model> via POST http://127.0.0.1:11434/api/generate.
  • Use ornith:9b as the primary local model.
  • Cloud backends may be implemented only if real env keys exist; otherwise report unavailable honestly.
  • Output JSON must include:
    • text
    • model_id
    • role
    • input_tokens
    • output_tokens
    • latency_ms
    • temperature
    • route
    • verdict when role is classify or judge
  • For Ollama, use real prompt_eval_count and eval_count.
  • Append real usage to .specify/logs/level5/provider-usage.jsonl.
  • Do not recycle sample values.
  • On backend/model failure, return nonzero with a clear error. Do not emit a fake fallback success.

B. WP-S1 Router Hardening

Implement hardening in the real model path:

  • Delimit untrusted content: <<<UNTRUSTED>>> ... <<<END_UNTRUSTED>>>
  • classify output must be exactly INJECTION or SAFE.
  • judge output must be exactly APPROVED or REJECTED.
  • Any malformed model output must fail closed.
  • Never log API keys, secrets, raw Authorization headers, or .env contents.
  • For future cloud egress, mask PII before sending. Since cloud keys are unset now, test this path as unavailable or with a local dry-run payload capture.
  • Enforce endpoint allowlist:
    • api.anthropic.com
    • api.openai.com
    • 127.0.0.1:11434
  • Reject arbitrary model URLs and metadata IP attempts.
  • Use temperature=0 for classify/judge.
  • Write router audit/usage records without leaking prompt secrets.

C. Wire The Router Into Real Harness Paths

Do not leave the router as an unused utility.

For Wave 1, wire at least one real path:

  1. H6 usage path:

    • Ensure a real model-router call can feed .specify/logs/level5/provider-usage.jsonl.
    • Ensure agent-metrics.sh can consume real provider telemetry for a matching step where available.

  2. H7 fallback path:

    • Replace fake fallback triggers like bash -c "exit 9" in scripts/run-casan-pipeline.mjs with a real failing model-router primary route, then fallback to ollama:ornith:9b only if the model is available.
    • If the model is unavailable, record H7 fallback as blocked, not passed.

  3. Optional H4/H3 path if feasible in Wave 1:

    • Wire classify/judge into security-check.sh or casan-step.mjs only if you can prove fail-before/fix/pass in this turn.
    • If not feasible, document it as Wave 2 work. Do not claim H4/H3 model-based improvement.

D. Fail-able Router Tests

Create or modify:

  • .specify/tests/phase3-model-router-tests.sh

Tests must cover:

  1. ollama:ornith:9b generate works through tunnel, or fails with clear reason if tunnel unavailable.
  2. Router records real input_tokens and output_tokens from Ollama.
  3. Classifier/judge malformed output fails closed.
  4. SSRF-like endpoint/model config is rejected.
  5. Fake API key pattern does not appear in .specify/logs.
  6. Cloud backend reports unavailable honestly while keys are unset.
  7. A deliberate failing primary route causes fallback only through the real router path, not exit 9.

No hardcoded PASS.

E. Context Validation

Create or modify:

  • .specify/scripts/bash/context-validate.sh
  • scripts/run-casan-pipeline.mjs if needed to add generated_at or invoke validation in the real pipeline

Behavior:

  • Default context: docs/output/output_logs/001-okr-web-app/pipeline-context.yaml
  • Accept optional context path arg.
  • Fail if any artifact: path is missing.
  • Fail if any trace_file: path is missing.
  • Warn if entries have no generated_at.
  • If adding generated_at, do it in the real appendContext path.

Create fail/pass evidence:

  • copy context to temp,
  • inject one missing artifact path,
  • show validator exits nonzero,
  • run validator on real context and show pass.

F. Supply Chain Baseline

Current npm audit --workspaces --audit-level=high fails with 5 high vulnerabilities.

For Wave 1:

  • Do not claim supply-chain gate green.
  • Either fix high vulnerabilities without breaking app/test/build, or record them as a blocker for WP-S3.
  • If you run npm audit fix or upgrade major dependencies, run full app tests/build afterward and capture logs.

G. Wave 1 Results

Create:

  • docs/output/casan/phase3-wave1-results.md

Include:

  • what was implemented,
  • what was not implemented,
  • infrastructure used,
  • exact commands run,
  • exact evidence files,
  • real test results,
  • core files modified and why,
  • current blockers,
  • remaining work for WP-A/B/C/D/E/F/G and WP-S2..S8,
  • explicit statement that no average >90 score is claimed yet.

Required Commands And Evidence

Capture logs under:

docs/output/casan/phase3-evidence/wave1/

Run and capture:

npm install
npm test
npm run build
bash .specify/tests/phase3-model-router-tests.sh
bash .specify/scripts/bash/context-validate.sh docs/output/output_logs/001-okr-web-app/pipeline-context.yaml

If any core harness script changed, also run and capture:

bash .specify/tests/run-casan4-harness-tests.sh
bash .specify/tests/adversarial-harness-tests.sh
bash .specify/scripts/bash/verify-audit-chain.sh
bash .specify/scripts/bash/verify-tool-audit.sh

If baseline is green and model tunnel is available, run and capture:

node scripts/run-casan-pipeline.mjs

Deliverables

  • Working Wave 1 code.
  • Real evidence logs under docs/output/casan/phase3-evidence/wave1/.
  • docs/output/casan/phase3-wave1-results.md.
  • No fabricated evidence.
  • No target score claimed as achieved.

If blocked, stop and write a precise blocker report instead of manufacturing evidence.

Reference in New Issue View Git Blame Copy Permalink