Files

T

thanhnvandClaude Opus 4.8 36a4812ef3 refactor(structure): promote app to repo root + remove redundant workspace cruft

Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.

- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
  .specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
  active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
  launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
  tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
  README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
  artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
  - .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
    working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
    -> packages/casan-harness/... (.specify/logs state kept)
  - .claude/launch.json, .gitea/*-runbook.md: path prefixes
  - CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
  - policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.

Full gate from the new root: PASS=64 FAIL=0 SKIP=3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

2026-07-08 13:26:36 +09:00

4.3 KiB

Raw Permalink Blame History

CASAN Phase 2 — Independent Audit (post-Codex build)

Date: 2026-06-28 Auditor method: Ran every command myself, tried to break the claims. Did not trust logs. Build author: Codex (independent), to keep the assessment impartial.

Verdict

All seven harnesses are now independently verified at ≥ 80, average ~81 → genuine CASAN Level 4 (Automated). This is earned against real execution, not config presence.

ID	Harness	Pre	Now	What I independently verified
H1	Context	45	82	`pipeline-context.yaml` built incrementally during a real run; 12 DISTINCT trace ids (no recycling); referenced artifacts exist on disk; real `00-boss.log.md`.
H2	Tool	80	80	Phase-1 controls intact (35/0 + 22/0 suites still pass); real signed tool-call audit accumulated from the run.
H3	Evaluation	22	82	Ran `npm test -w backend` → 5 real tests pass (HTTP e2e + services + golden). Broke the golden fixture → test FAILED (exit 1); restored → passed. Builds pass. Review gates produced a SUBSTANTIVE REJECTED that named missing artifacts, which attempt-2 then actually created.
H4	Security	80	80	Intact; 20 real security traces from the run.
H5	Governance	80	82	Real reject→approve cycle through the gates; governance hash-chain still verifies SIGNED after the run.
H6	AgentOps	80	80	Real per-step latency/traces + populated hallucination signals. (Caveat below on cost.)
H7	Orchestration	40	80	Real DAG run with a genuine BACK-TO-PLAN cycle (06 REJECTED → 07 attempt-2 APPROVED) tracked in the boss log; per-step CASAN wrapping (20/8/9 traces); fallback/drift/rollback invoked. Real undo demonstrated (rollback-before==after).

Strongest confirmations

Regression is real: golden break → EXIT=1; restore → EXIT=0. Not faked.
Real reject→fix: 06-reviewplan-attempt-1 REJECTED with concrete missing-artifact criteria; 07-plan-attempt-2 APPROVED; the four named artifacts (data-model/research/quickstart/openapi) now exist.
No fabricated demo: 12 distinct per-step trace ids, real boss log timestamps, real on-disk artifacts — the generate-casan-demo-context.py fabrication path was not used.
Phase-1 hardening intact: my run-casan4 (35/0) and adversarial (22/0) suites still pass.

Honest remaining weaknesses (why these are ~80, not ~90)

H7 in-run rollback is still the marker-writer (rollback-transactions.jsonl records printf rolled_back > marker). A REAL undo is demonstrated only in the standalone app-evidence/rollback-* (before==after). The real undo is not yet the path the orchestrator runs.
H7 drift is still self-comparison (cp golden candidate → similarity=1.0). The algorithm is real but fed identical input; not drift against a real prior run.
H7 model fallback uses a synthetic trigger (primary exit 9), not a real model-A failure.
H6 cost is the sample provider record (2778 tokens) reused for every step — real latency, but not real per-step model billing.
H3 frontend "test" is tsc --noEmit (type-check only), no runtime tests; backend coverage is modest; no multi-model judge consensus.
H1 has no staleness/path-existence validation; design artifacts are thin.
Housekeeping: stray root files (o6.txt, o7.txt, t6.txt, t7.txt).

Path to ~90 (harness-side, no app rebuild needed)

H7: wire the real undo into the orchestrator's rollback step; snapshot a real prior plan as the drift baseline so attempt-1 vs attempt-2 yields a real similarity < 1.0; drive fallback from a real failed call.
H2: runtime cross-step rate-limit counter + per-call JSON-schema validation.
H6: import real per-step provider telemetry instead of the sample record.
H1: validate artifact paths exist before sub-agent read; add TTL/staleness.
H3: add frontend runtime tests; widen backend coverage; optional multi-model judge.

How to reproduce

npm test -w backend && npm run build -w backend && npm run build -w frontend
bash .specify/tests/run-casan4-harness-tests.sh
bash .specify/tests/adversarial-harness-tests.sh
node scripts/run-casan-pipeline.mjs
bash .specify/scripts/bash/verify-audit-chain.sh
bash .specify/scripts/bash/verify-tool-audit.sh

4.3 KiB Raw Permalink Blame History