thanhnvandClaude Opus 4.8
36a4812ef3
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.
- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
.specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
- .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
-> packages/casan-harness/... (.specify/logs state kept)
- .claude/launch.json, .gitea/*-runbook.md: path prefixes
- CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
- policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.
Full gate from the new root: PASS=64 FAIL=0 SKIP=3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
5.3 KiB
5.3 KiB
CASAN Phase 3 — Push-to-90 Results (executed)
Date: 2026-06-28
Executed by: Claude (Opus 4.8), directly in CASAN5.
Method: real implementations + independent adversarial tests. Anything the offline
environment cannot do for real is left UNDONE and labeled — nothing was faked.
Verify: bash .specify/tests/run-casan4-harness-tests.sh (35 PASS) and
bash .specify/tests/adversarial-harness-tests.sh (34 PASS, was 22 — +12 push-to-90 checks).
1. What was implemented and verified (real)
| Task | Harness | What shipped | Adversarial proof |
|---|---|---|---|
| T1 | H7 | rollback-manager.sh checkpoint <file> — backs up a real file and records a real restore command; execute genuinely restores it |
"H7 rollback genuinely restores the file" (before==after) |
| T2 | H7 | drift test now compares two genuinely different artifacts (not cp golden candidate) |
"H7 drift detects real difference (similarity=0.661 < 1.0)" + identical→pass |
| T3 | H7 | fallback driven by a real primary failure (cat /nonexistent → fallback) |
"H7 fallback runs after a genuine primary failure" |
| T5a | H2 | rate_limit_per_run enforced by a real per-run counter in tool-registry-gate.sh |
"H2 denies 3rd deploy in one run (rate limit)" |
| T5b | H2 | validate-tool-input.sh — stdlib JSON-schema validation (required/type/enum/additionalProperties) |
accepts valid input; "rejects malformed tool input" (rc=2) |
| T6a | H4 | tool-exec.sh — portable hard timeout (timeout/perl-alarm) so a runaway tool can't hang the pipeline |
"H4 kills a runaway tool call" (rc=124) + fast call passes |
| T7 | H5 | audit signing private key moved OFF-REPO ($CASAN_AUDIT_KEY_DIR, default ~/.casan/audit-keys); only the public key is committed |
"H5 private signing key absent from repo"; re-forge still detected |
| T8 | H1 | context-validate.sh — fails if any artifact referenced by pipeline-context.yaml is missing; optional TTL staleness |
"H1 context-validate catches a missing artifact" (rc=2) |
| T4 | H6 | removed the fake uniform cost: provider telemetry is used only when a record genuinely matches the step, else the distinct per-step estimate (no sample reused across steps) | per-step cost no longer the reused 2778 sample |
| T10 | — | removed stray root files (o6/o7/t6/t7.txt); new scripts executable |
suites still green |
2. Honestly NOT done — blocked by the offline environment (NOT faked)
| Task | Harness | Why it can't be done here for real |
|---|---|---|
| T6b | H4 | Semantic/embedding injection detection needs an embedding model or classifier API — none available offline. The timeout guard (T6a) shipped; semantic detection did not. Claiming a heuristic is "semantic embedding" would be a lie, so it was left out. |
| T4 (full) | H6 | Real per-step provider billing needs the LLM provider's usage API. The estimate is now honest and per-step, but it is still an estimate, explicitly labeled cost_source=word_count_estimate. |
| T7 (full) | H5 | Real KMS/HSM + OS WORM: the key is now off-repo (real), but a managed KMS and append-only/WORM storage need cloud infra; macOS has no chattr +a. Documented as the production step, not emulated. |
| T9 | H3 | Frontend runtime tests (Vitest/RTL) need an npm install (network); multi-model judge consensus needs live model APIs. Neither available offline. Backend tests + golden regression remain real and passing. |
3. Honest re-score (independent, evidence-backed)
| ID | Harness | Before (Phase 2) | After (Phase 3) | Why it moved |
|---|---|---|---|---|
| H1 | Context | 82 | 85 | Real path-existence/staleness validation gate (T8). |
| H2 | Tool | 80 | 86 | Runtime rate-limit counter + input schema validation (T5). |
| H3 | Evaluation | 82 | 82 | Unchanged — frontend tests + judge consensus blocked offline (T9). |
| H4 | Security | 80 | 82 | Real execution timeout (T6a); semantic detection still missing (T6b blocked). |
| H5 | Governance | 82 | 85 | Signing key moved off-repo (T7); KMS/WORM documented, not faked. |
| H6 | AgentOps | 80 | 81 | Fake uniform cost removed; honest per-step estimate (T4). Real billing still needs API. |
| H7 | Orchestration | 80 | 86 | Real rollback undo, real drift, real-failure fallback (T1–T3). |
| Average | ~81 | ~84 | Genuine Level 4; not 90 — see §2. |
4. Why not 90 (the honest ceiling here)
The remaining distance to ~90 is entirely the four items in §2, and every one needs something this sandbox does not have: an embedding model, a live LLM billing API, a cloud KMS/WORM store, and network for npm install / multi-model judging. They are real, fundable work — they just cannot be demonstrated truthfully offline, so they were not claimed. Doing them on infrastructure with those services is the path from ~84 to ~90.
5. Files added / changed in Phase 3
scripts/bash/rollback-manager.sh(+checkpoint),validate-tool-input.sh(new),context-validate.sh(new),tool-exec.sh(new),provider-cost-lookup.py(new)scripts/bash/tool-registry-gate.sh(rate limit),agent-metrics.sh(honest per-step cost),governance-check.sh+tool-audit-lib.sh(key off-repo)level5/tool-registry.yaml(rate_limit_per_run)tests/adversarial-harness-tests.sh(+12 push-to-90 checks → 34 total)