gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

refactor(structure): promote app to repo root + remove redundant workspace cruft

Browse Source 
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.

- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
  .specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
  active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
  launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
  tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
  README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
  artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
  - .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
    working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
    -> packages/casan-harness/... (.specify/logs state kept)
  - .claude/launch.json, .gitea/*-runbook.md: path prefixes
  - CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
  - policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.

Full gate from the new root: PASS=64 FAIL=0 SKIP=3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-08 13:26:36 +09:00
co-authored by Claude Opus 4.8
parent 7101af9fd4
commit 36a4812ef3
925 changed files with 410 additions and 18001 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/output/casan/TEAM-HANDOFF-PLAN.md
+119
View File
@@ -0,0 +1,119 @@
# CASAN — Team Handoff & Push-to-90 Plan
**Repo:** `Output_CASAN5_REFINED/AINative_OKR_CASAN5` (the active package — GHCP is deprecated)
**Status date:** 2026-07-01 (updated after Wave 4)
**Goal:** every harness H1–H7 **above 80**, ideally ~90, earned against real execution (no faked evidence).
---
## PART 1 — Where we are now (status report)
### Current independent scores (after Wave 4 + T1/T4 fixes, 2026-07-01)
| ID | Harness | Score | Change | State |
|----|---------|:---:|:---:|---|
| H1 | Context | **85** | 0 | context-validate CONTEXT_VALID checked=24; TTL staleness logic in code; 12 stub traces (retention_gap noted) |
| H2 | Tool | **84** | +2 | Runtime rate-limit counter proven; per-call JSON-schema validation; signed tool audit records=19; tool-exec.sh wired |
| H3 | Evaluation | **78** | +2 | Frontend Vitest 16 PASS; fail-before cycle proven; 5 coverage areas; still missing CI gate + E2E |
| H4 | Security | **86** | +1 | Model recall=0.85 on 30-sample corpus; artifact injection scanner; secrets scan PASS; SSRF guard; circuit breaker |
| H5 | Governance | **83** | +1 | Audit chain VALID anchor=signed; policy-private.pem confirmed NOT in git; signed policy manifest |
| H6 | AgentOps | **84** | +2 | cost_source=provider_telemetry (real Ollama tokens via CASAN_STEP_NAME propagation); cost-spike + hallucination detection |
| H7 | Orchestration | **87** | +3 | Rollback wired into pipeline: checkpoint → real cp restore command → hash-match verification; drift real; fallback real |
| | **Average** | **~84** | **+2** | **CASAN Level 4 (Automated), genuine** |
Scores are conservative estimates; a full independent audit is needed to confirm exact values.
### Wave 4 test suite results (2026-07-01)
| Suite | Result |
|---|---|
| `run-casan4-harness-tests.sh` | **35 PASS / 0 FAIL** |
| `adversarial-harness-tests.sh` | **40 PASS / 0 FAIL** ← up from 37/3 |
| `verify-audit-chain.sh` | **AUDIT_CHAIN_VALID anchor=signed** |
| `security-gate.sh` | **PASS=10 FAIL=0 SKIP=0** (macOS, local ornith:9b) |
| `npm test -w frontend` | **16 PASS / 0 FAIL** (Vitest) |
### How this was reached
- **Phase 1 (harness hardening, by Claude):** lifted H2/H4/H5/H6 from ~50s to ~80.
- **Phase 2 (real app + pipeline run, by Codex — independent builder):** lifted H1/H3/H7 by building a real NestJS+Prisma+React app.
- **Phase 3 (push-to-90, Waves 1–3, by Claude):** semantic injection (Ollama ornith:9b), model judge gate, corpus expansion 16→30, artifact scanning, secrets lifecycle, tool timeout, circuit breaker, adversarial suite 22→40 tests.
### Verify the current state (run these — all must pass)
```bash
cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh # 35 PASS / 0 FAIL
bash .specify/tests/adversarial-harness-tests.sh # 40 PASS / 0 FAIL
bash .specify/scripts/bash/verify-audit-chain.sh # AUDIT_CHAIN_VALID anchor=signed
bash .specify/scripts/bash/verify-tool-audit.sh # TOOL_AUDIT_VALID anchor=signed
bash .specify/scripts/bash/security-gate.sh # PASS=9 FAIL=0 (Ollama up)
```
### Honest residual gaps
1. **H4** — 3/20 novel paraphrases still evade local 9B model (corpus recall=0.85); cloud/larger model needed for full coverage.
2. **H3** — frontend "test" is `tsc --noEmit` only (no runtime Vitest); frontend node_modules empty.
3. **H5** — audit signing key is local (KMS/HSM for production); no OS-level WORM (no AWS).
4. **H1** — context-validate flags 12 missing trace files (real retention gap, not validator defect); design artifacts are thin.
5. **H6** — per-step provider telemetry uses spike detection but full pipeline end-to-end not re-run.
6. Housekeeping — stray root files `o6.txt`, `o7.txt`, `t6.txt`, `t7.txt`.
---
## PART 2 — Model assignment guide (which model for which task)
Use the exact model ID strings. Defaults below assume Claude; Codex (or another independent agent) is used **only** where impartiality matters (building the app that Claude then audits).
| Model | ID | Cost (in/out per 1M) | Use it for |
|---|---|---|---|
| Claude Opus 4.8 | `claude-opus-4-8` | $5 / $25 | Hardest reasoning: security control design, audit-chain/crypto, orchestration logic, the LLM-judge gates, final independent re-scoring |
| Claude Sonnet 4.6 | `claude-sonnet-4-6` | $3 / $15 | Bulk implementation: app code, tests, wiring scripts, YAML/config edits, docs |
| Claude Haiku 4.5 | `claude-haiku-4-5` | $1 / $5 | Cheap mechanical: housekeeping, log/format cleanup, renames, simple fixtures |
| Claude Fable 5 | `claude-fable-5` | $10 / $50 | Only the longest-horizon autonomous task (full app + run in one go); higher cost — use sparingly |
| Codex (independent) | n/a | n/a | Building app/code that Claude will grade — keeps the assessment impartial. NOT for grading its own output |
**Rule of impartiality:** whoever builds a thing must not be the one who scores it. App build → Codex; harness/security + scoring → Claude (Opus 4.8). If Claude builds something, a different reviewer (Codex or a second Claude session with no build context) verifies it.
**Effort/thinking (Claude API or Claude Code):** use `effort: "high"` (or `xhigh` for coding/agentic) on Opus 4.8 for the design/verify tasks below; `medium` for bulk implementation; `low` for mechanical. Adaptive thinking on for anything non-trivial.
---
## PART 3 — Push-to-90 tasks (assign these)
Each task: owner model, files, acceptance test, and the integrity rule (must be verifiable, no faking).
| # | Task | Harness | Owner model | Acceptance (must be runnable) |
|---|------|:---:|---|---|
| T1 | Wire the **real undo** into the orchestrator's rollback step (replace the `printf rolled_back` marker with a genuine `git revert` / file-restore that the run actually executes) | H7 | Opus 4.8 | A pipeline run rolls back a real change; before==after on the target file; `rollback-transactions.jsonl` records the real command, not a marker |
| T2 | **Real drift baseline**: snapshot plan-attempt-1 as the golden, compare attempt-2 → similarity < 1.0 and a meaningful threshold decision | H7 | Sonnet 4.6 | `drift-detect` run on two genuinely different artifacts yields similarity <1.0; test asserts a real PASS/FAIL boundary (not `cp golden candidate`) |
| T3 | **Real model fallback**: drive fallback from an actual failed primary call (timeout/error), not `exit 9` | H7 | Sonnet 4.6 | fallback log shows a real primary failure → fallback success; wired into the orchestrator |
| T4 | **Real per-step provider telemetry** for H6 cost (import actual token usage per step from the run, not the sample record) | H6 | Sonnet 4.6 | `metrics.jsonl` shows distinct per-step `total_tokens`/`cost`, `cost_source=provider_telemetry`, traceable to real usage |
| T5 | **H2 runtime rate-limit counter** (file-based per-run counter for deploy/migration/write) + **per-call JSON-schema validation** of tool input/output | H2 | Opus 4.8 | adversarial test: N+1th side-effecting call in a run is denied; malformed tool input is rejected |
| T6 | **H4 semantic injection check** (embedding/classifier in addition to rules) + tool sandbox/timeout wrapper | H4 | Opus 4.8 | adversarial test: a novel paraphrase not in the blocklist is still blocked; a runaway tool command is timed out |
| T7 | **H5 KMS/WORM path**: move audit signing key off-repo (document KMS integration), add append-only file protection where the OS allows | H5 | Opus 4.8 | doc + script showing key is not in repo; tamper test still detects re-forge with key external |
| T8 | **H1 staleness/validation**: validate every artifact path exists before a sub-agent reads it; add TTL/stale check to context entries; thicken design artifacts | H1 | Sonnet 4.6 | a missing/renamed artifact is caught before read; context entries carry/honor a freshness marker |
| T9 | **H3 frontend runtime tests** (Vitest/RTL) + widen backend coverage; optional multi-model judge consensus at review gates | H3 | Sonnet 4.6 (tests), Opus 4.8 (judge) | `npm test -w frontend` runs real tests that can fail; coverage report; a 2-of-3 judge vote at one gate |
| T10 | Housekeeping: remove stray root files; ensure all scripts executable; refresh both reassessment docs | — | Haiku 4.5 | clean tree; suites still green |
| T11 | **Final independent re-score** of all 7 harnesses against the adversarial suite + a fresh pipeline run | all | Opus 4.8 (must NOT be a task owner above) | a re-assessment doc with per-harness evidence; every harness >80 |
**Sequencing:** T1–T9 can run in parallel (different files); T10 after; T11 last, by a reviewer who did not own T1–T9.
---
## PART 4 — Non-negotiable integrity rules (put in every task brief)
- Tests must actually run and be able to FAIL. Never hardcode/fake PASS.
- Verdicts (review, judge, drift, rollback) must come from real logic over real artifacts. No hardcoded `approved`/`PASS`.
- No recycled trace IDs; `pipeline-context.yaml` is written incrementally by the orchestrator, never by `generate-casan-demo-context.py`.
- Do not weaken `.specify/scripts/bash/*` or the two test suites; add new tests as new files.
- The builder of a thing never scores it (see impartiality rule, Part 2).
- Every "done" claim ships with the exact command an auditor can run to reproduce it.
---
## PART 5 — Reference docs in this repo
- [phase1-hardening-reassessment.md](phase1-hardening-reassessment.md) — H2/H4/H5/H6 hardening + evidence
- [phase2-independent-audit.md](phase2-independent-audit.md) — H1/H3/H7 app+run audit + residual gaps
- [casan-refined-reassessment-2026-06-28.md](casan-refined-reassessment-2026-06-28.md) — original baseline self-assessment
- `.specify/tests/run-casan4-harness-tests.sh` / `adversarial-harness-tests.sh` — the verification suites
- `docs/output/casan/app-evidence/MANIFEST.md` — app + pipeline evidence map