gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update optimize wave3 (need update wave 4 to wave 8)

Browse Source
This commit is contained in:
thanhnv
2026-07-01 00:02:30 +09:00
parent 07ac1bdcdd
commit eaf919e744
140 changed files with 3780 additions and 844 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/docs/output/casan/TEAM-HANDOFF-PLAN.md
+30 -26
View File
@@ -1,49 +1,53 @@
# CASAN — Team Handoff & Push-to-90 Plan
**Repo:** `Output_CASAN5_REFINED/AINative_OKR_CASAN5` (the active package — GHCP is deprecated)
**Status date:** 2026-06-28
**Status date:** 2026-06-30 (updated after Wave 3)
**Goal:** every harness H1–H7 **above 80**, ideally ~90, earned against real execution (no faked evidence).
---
## PART 1 — Where we are now (status report)
### Current independent scores (verified, not self-claimed)
### Current independent scores (after Wave 3, 2026-06-30)
| ID | Harness | Score | State |
|----|---------|:---:|---|
| H1 | Context | **82** | Real incremental `pipeline-context.yaml` from a real run; 12 distinct traces; artifacts on disk |
| H2 | Tool | **80** | Per-agent permission enforced + in execution path; signed tamper-evident tool audit; rollback required |
| H3 | Evaluation | **82** | Real app + real unit/e2e tests that run and can fail; real golden regression; real LLM-judge REJECTED→fix cycle |
| H4 | Security | **80** | Input normalization defeats bypasses; pii-rules wired; output fail-closed on secrets |
| H5 | Governance | **82** | RSA-anchored audit chain (re-forge detected); approver+output_hash hashed; separation of duties |
| H6 | AgentOps | **80** | Real latency/alerts; hallucination detector populated; provider-telemetry cost source |
| H7 | Orchestration | **80** | Real DAG run with real BACK-TO-PLAN; per-step CASAN wrapping; fallback/drift/rollback invoked |
| | **Average** | **~81** | **CASAN Level 4 (Automated), genuine** |
| H1 | Context | **82** | Real incremental `pipeline-context.yaml` from a real run; 12 distinct traces; artifacts on disk; context-validate.sh catches missing artifacts |
| H2 | Tool | **82** | Per-agent permission + rate-limit + schema validation; signed tamper-evident tool audit; rollback required; tool-exec.sh wired in harness |
| H3 | Evaluation | **84** | Real app + real unit/e2e tests; real golden regression; LLM-judge gate wired into review steps 04/06/10 with fail-before proof |
| H4 | Security | **85** | Semantic model layer (recall=0.85 on 30-sample DoD corpus); artifact indirect injection scanner; secrets lifecycle scan; tool timeout wired; circuit breaker; no-bypass scan |
| H5 | Governance | **82** | RSA-anchored audit chain (re-forge detected); approver+output_hash hashed; separation of duties; signing key off-repo |
| H6 | AgentOps | **82** | Real per-step tokens; cost-spike detection; hallucination detector; provider telemetry |
| H7 | Orchestration | **82** | Real DAG run; real rollback restore; real drift (similarity<1.0); failure-driven fallback |
| | **Average** | **~83** | **CASAN Level 4 (Automated), genuine — approaching Level 5** |
Scores are conservative estimates; a full independent audit is needed to confirm exact values.
### How this was reached
- **Phase 1 (harness hardening, by Claude):** lifted H2/H4/H5/H6 from ~50s to ~80 with adversarial-verified controls. See [phase1-hardening-reassessment.md](phase1-hardening-reassessment.md).
- **Phase 2 (real app + pipeline run, by Codex — independent builder for impartiality):** lifted H1/H3/H7 by building a real NestJS+Prisma+React app and running the pipeline for real. Audited independently. See [phase2-independent-audit.md](phase2-independent-audit.md).
- **Phase 1 (harness hardening, by Claude):** lifted H2/H4/H5/H6 from ~50s to ~80.
- **Phase 2 (real app + pipeline run, by Codex — independent builder):** lifted H1/H3/H7 by building a real NestJS+Prisma+React app.
- **Phase 3 (push-to-90, Waves 1–3, by Claude):** semantic injection (Ollama ornith:9b), model judge gate, corpus expansion 16→30, artifact scanning, secrets lifecycle, tool timeout, circuit breaker, adversarial suite 22→40 tests.
### Verify the current state (run these — all must pass)
```bash
cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
npm test -w backend && npm run build -w backend && npm run build -w frontend
bash .specify/tests/run-casan4-harness-tests.sh # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh # 22 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh # TOOL_AUDIT_VALID anchor=signed
bash .specify/tests/run-casan4-harness-tests.sh # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh # 40 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh # TOOL_AUDIT_VALID anchor=signed
bash .specify/scripts/bash/security-gate.sh # PASS=9 FAIL=0 (Ollama up)
```
### Honest residual gaps keeping us at ~81 not ~90
1. **H7** — in-run rollback still writes a marker (real undo exists only as a standalone demo); drift still compares a file to a copy of itself; model-fallback uses a synthetic trigger.
2. **H6** — cost uses a sample provider record reused for every step (real latency, not real per-step billing).
3. **H2** — no runtime rate-limit counter; no per-call JSON-schema validation of tool I/O.
4. **H4** — rule/normalization-based only (no semantic/embedding detection); no tool sandbox/timeout.
5. **H5** — audit signing key is local (must be KMS/HSM for production); no OS-level WORM.
6. **H1** — no staleness / path-existence validation; design artifacts are thin.
7. **H3** — frontend "test" is `tsc --noEmit` only (no runtime tests); backend coverage modest.
8. Housekeeping — stray root files `o6.txt`, `o7.txt`, `t6.txt`, `t7.txt`.
### Honest residual gaps
1. **H4** — 3/20 novel paraphrases still evade local 9B model (corpus recall=0.85); cloud/larger model needed for full coverage.
2. **H3** — frontend "test" is `tsc --noEmit` only (no runtime Vitest); frontend node_modules empty.
3. **H5** — audit signing key is local (KMS/HSM for production); no OS-level WORM (no AWS).
4. **H1** — context-validate flags 12 missing trace files (real retention gap, not validator defect); design artifacts are thin.
5. **H6** — per-step provider telemetry uses spike detection but full pipeline end-to-end not re-run.
6. Housekeeping — stray root files `o6.txt`, `o7.txt`, `t6.txt`, `t7.txt`.
---