refactor(structure): promote app to repo root + remove redundant workspace cruft
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.
- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
.specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
- .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
-> packages/casan-harness/... (.specify/logs state kept)
- .claude/launch.json, .gitea/*-runbook.md: path prefixes
- CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
- policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.
Full gate from the new root: PASS=64 FAIL=0 SKIP=3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 4.8
parent
7101af9fd4
commit
36a4812ef3
@@ -0,0 +1,51 @@
|
||||
# CASAN Phase 2 — Independent Audit (post-Codex build)
|
||||
|
||||
**Date:** 2026-06-28
|
||||
**Auditor method:** Ran every command myself, tried to break the claims. Did not trust logs.
|
||||
**Build author:** Codex (independent), to keep the assessment impartial.
|
||||
|
||||
## Verdict
|
||||
|
||||
All seven harnesses are now independently verified at **≥ 80**, average **~81** → genuine **CASAN Level 4 (Automated)**. This is earned against real execution, not config presence.
|
||||
|
||||
| ID | Harness | Pre | Now | What I independently verified |
|
||||
|----|---------|:--:|:--:|---|
|
||||
| H1 | Context | 45 | **82** | `pipeline-context.yaml` built incrementally during a real run; 12 DISTINCT trace ids (no recycling); referenced artifacts exist on disk; real `00-boss.log.md`. |
|
||||
| H2 | Tool | 80 | **80** | Phase-1 controls intact (35/0 + 22/0 suites still pass); real signed tool-call audit accumulated from the run. |
|
||||
| H3 | Evaluation | 22 | **82** | Ran `npm test -w backend` → 5 real tests pass (HTTP e2e + services + golden). Broke the golden fixture → test FAILED (exit 1); restored → passed. Builds pass. Review gates produced a SUBSTANTIVE REJECTED that named missing artifacts, which attempt-2 then actually created. |
|
||||
| H4 | Security | 80 | **80** | Intact; 20 real security traces from the run. |
|
||||
| H5 | Governance | 80 | **82** | Real reject→approve cycle through the gates; governance hash-chain still verifies SIGNED after the run. |
|
||||
| H6 | AgentOps | 80 | **80** | Real per-step latency/traces + populated hallucination signals. (Caveat below on cost.) |
|
||||
| H7 | Orchestration | 40 | **80** | Real DAG run with a genuine BACK-TO-PLAN cycle (06 REJECTED → 07 attempt-2 APPROVED) tracked in the boss log; per-step CASAN wrapping (20/8/9 traces); fallback/drift/rollback invoked. Real undo demonstrated (rollback-before==after). |
|
||||
|
||||
## Strongest confirmations
|
||||
- **Regression is real:** golden break → `EXIT=1`; restore → `EXIT=0`. Not faked.
|
||||
- **Real reject→fix:** `06-reviewplan-attempt-1` REJECTED with concrete missing-artifact criteria; `07-plan-attempt-2` APPROVED; the four named artifacts (data-model/research/quickstart/openapi) now exist.
|
||||
- **No fabricated demo:** 12 distinct per-step trace ids, real boss log timestamps, real on-disk artifacts — the `generate-casan-demo-context.py` fabrication path was not used.
|
||||
- **Phase-1 hardening intact:** my `run-casan4` (35/0) and `adversarial` (22/0) suites still pass.
|
||||
|
||||
## Honest remaining weaknesses (why these are ~80, not ~90)
|
||||
1. **H7 in-run rollback is still the marker-writer** (`rollback-transactions.jsonl` records `printf rolled_back > marker`). A REAL undo is demonstrated only in the standalone `app-evidence/rollback-*` (before==after). The real undo is not yet the path the orchestrator runs.
|
||||
2. **H7 drift is still self-comparison** (`cp golden candidate` → `similarity=1.0`). The algorithm is real but fed identical input; not drift against a real prior run.
|
||||
3. **H7 model fallback uses a synthetic trigger** (`primary exit 9`), not a real model-A failure.
|
||||
4. **H6 cost is the sample provider record** (2778 tokens) reused for every step — real latency, but not real per-step model billing.
|
||||
5. **H3 frontend "test" is `tsc --noEmit`** (type-check only), no runtime tests; backend coverage is modest; no multi-model judge consensus.
|
||||
6. **H1** has no staleness/path-existence validation; design artifacts are thin.
|
||||
7. Housekeeping: stray root files (`o6.txt`, `o7.txt`, `t6.txt`, `t7.txt`).
|
||||
|
||||
## Path to ~90 (harness-side, no app rebuild needed)
|
||||
- H7: wire the real undo into the orchestrator's rollback step; snapshot a real prior plan as the drift baseline so attempt-1 vs attempt-2 yields a real similarity < 1.0; drive fallback from a real failed call.
|
||||
- H2: runtime cross-step rate-limit counter + per-call JSON-schema validation.
|
||||
- H6: import real per-step provider telemetry instead of the sample record.
|
||||
- H1: validate artifact paths exist before sub-agent read; add TTL/staleness.
|
||||
- H3: add frontend runtime tests; widen backend coverage; optional multi-model judge.
|
||||
|
||||
## How to reproduce
|
||||
```bash
|
||||
npm test -w backend && npm run build -w backend && npm run build -w frontend
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
bash .specify/tests/adversarial-harness-tests.sh
|
||||
node scripts/run-casan-pipeline.mjs
|
||||
bash .specify/scripts/bash/verify-audit-chain.sh
|
||||
bash .specify/scripts/bash/verify-tool-audit.sh
|
||||
```
|
||||
Reference in New Issue
Block a user