thanhnvandClaude Opus 4.8
36a4812ef3
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.
- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
.specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
- .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
-> packages/casan-harness/... (.specify/logs state kept)
- .claude/launch.json, .gitea/*-runbook.md: path prefixes
- CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
- policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.
Full gate from the new root: PASS=64 FAIL=0 SKIP=3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
67 lines
3.1 KiB
Markdown
67 lines
3.1 KiB
Markdown
# CASAN Refined Re-Assessment - AINative_OKR_CASAN5
|
|
|
|
Generated: 2026-06-28
|
|
|
|
## Summary
|
|
|
|
This package merges the stable CASAN4 submission with the strongest additions from `AINative_OKR_Claude_GHCP`, then fixes the regressions found during review.
|
|
|
|
Positioning:
|
|
|
|
- CASAN Level 4: achieved.
|
|
- Level 5: stronger local demonstration, not full enterprise production.
|
|
- Evidence score: 88.7 / 100.
|
|
|
|
## Scorecard
|
|
|
|
| Harness | CASAN4 | Refined | Delta | Reason |
|
|
|---|---:|---:|---:|---|
|
|
| H1 Context | 90 | 91 | +1 | Context schema now exposes Level 5 paths for tool registry, drift, rollback, and AgentOps evidence. |
|
|
| H2 Tool | 84 | 88 | +4 | Tool registry expanded, side-effect idempotency tested, per-call audit added, cache hit no longer bypasses CASAN trace generation. |
|
|
| H3 Evaluation | 85 | 86 | +1 | Hallucination checklist added for review gates. |
|
|
| H4 Security | 88 | 90 | +2 | Jailbreak and private key tests added; private key leakage bug fixed with explicit regex and `grep --`. |
|
|
| H5 Governance | 87 | 88 | +1 | Risk policy and registry auto-update guidance added; audit chain remains verified. |
|
|
| H6 AgentOps | 89 | 90 | +1 | JSON alert format and tool-call audit evidence added. |
|
|
| H7 Orchestration | 87 | 88 | +1 | Refined wrapper preserves evidence on cache hit and keeps rollback/drift/fallback evidence. |
|
|
| **Average** | **87.1** | **88.7** | **+1.6** | Stronger Level 5 evidence path with verified regression fixes. |
|
|
|
|
## Verified Improvements
|
|
|
|
The test harness now checks:
|
|
|
|
- Prompt injection block.
|
|
- PII masking.
|
|
- Jailbreak block.
|
|
- Private key material block.
|
|
- High-risk deny-by-default.
|
|
- Explicit approval path.
|
|
- Tool-call audit log.
|
|
- Cache hit still records fresh CASAN traces.
|
|
- Audit hash-chain verification.
|
|
- Drift, fallback, tool registry, rollback, KPI, signed policy, provider telemetry, harness reuse, and central dashboard evidence.
|
|
|
|
## Regressions Fixed from AINative_OKR_Claude_GHCP
|
|
|
|
| Issue | Fix |
|
|
|---|---|
|
|
| Bash scripts were not executable in the submitted folder | New folder is copied from CASAN4 baseline with executable bits preserved. |
|
|
| Private key regex caused `grep` option parsing and leaked private key text | `security-check.sh` now uses `grep --` and explicit private key, DB connection string, and AWS key regexes. |
|
|
| Idempotency cache bypassed H4/H5/H6 on cache hit | `casan-harness.sh` now runs H4 input and H5 before cache use, records H6 cache-copy metrics, and runs H4 output. |
|
|
| Tool registry did not write central tool-call audit | `tool-registry-gate.sh` now writes `.specify/logs/audit/tool-calls.jsonl`. |
|
|
| Submission package still referenced CASAN4 paths | Paths were updated to `AINative_OKR_CASAN5`. |
|
|
|
|
## Claim Boundary
|
|
|
|
This package is stronger than CASAN4 for hackathon evidence. It should still be described as:
|
|
|
|
> Level 4 achieved; Level 5 demonstrated locally with verifiable controls.
|
|
|
|
Do not claim full production Level 5 until these are connected to enterprise services:
|
|
|
|
- IdP-backed approval and separation of duties.
|
|
- WORM or external signed log storage.
|
|
- Live provider telemetry APIs.
|
|
- Hosted dashboard with real-time ingestion.
|
|
- Shared harness package reused across independent production projects.
|
|
|