thanhnvandClaude Opus 4.8
36a4812ef3
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.
- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
.specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
- .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
-> packages/casan-harness/... (.specify/logs state kept)
- .claude/launch.json, .gitea/*-runbook.md: path prefixes
- CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
- policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.
Full gate from the new root: PASS=64 FAIL=0 SKIP=3.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
101 lines
5.1 KiB
Markdown
101 lines
5.1 KiB
Markdown
# CASAN Higher-Level Criteria
|
|
|
|
## Level 1 - Curious
|
|
|
|
Minimum signs:
|
|
|
|
- Individual AI usage.
|
|
- No shared prompt library.
|
|
- No audit log.
|
|
- No official governance.
|
|
|
|
## Level 2 - Augmented
|
|
|
|
Minimum signs:
|
|
|
|
- Approved AI tools are used by teams.
|
|
- Some workflow productivity gain.
|
|
- Basic acceptable-use policy.
|
|
- Vendor-provided safety controls.
|
|
|
|
## Level 3 - Standard
|
|
|
|
Minimum signs:
|
|
|
|
- Shared AI-SDLC process.
|
|
- Reusable templates and agent definitions.
|
|
- Review gates.
|
|
- Basic context management.
|
|
- Repeatable test and review flow.
|
|
|
|
## Level 4 - Automated
|
|
|
|
Minimum signs:
|
|
|
|
- Multi-step workflow is agent-orchestrated.
|
|
- H4 Security, H5 Governance, and H6 AgentOps are automated runtime gates.
|
|
- Side-effecting actions have policy decisions and audit evidence.
|
|
- Prompt injection, PII leakage, and secret exposure are tested.
|
|
- Trace, metrics, cost, latency, retry, and failure alerts are captured.
|
|
- Human approval exists for high-risk actions but is non-interactive and auditable.
|
|
|
|
## Level 5 - Native
|
|
|
|
Minimum signs:
|
|
|
|
- Harness is reusable across projects and business domains.
|
|
- Agent workflows improve based on measured outcomes.
|
|
- Model fallback, drift detection, rollback, and self-healing are operational.
|
|
- Governance, security, and AgentOps are centralized across teams.
|
|
- Business KPIs are tied to agent decisions and quality gates.
|
|
- Human-led, AI-first delegation is explicit by risk tier.
|
|
|
|
## Concrete Level 5 Backlog
|
|
|
|
| Area | Level 5 requirement | Suggested implementation |
|
|
|---|---|---|
|
|
| Drift Detection | Detect behavior changes across model/prompt versions | Store golden run outputs and compare semantic/structural deltas on every run |
|
|
| Model Fallback | Route failed or low-confidence steps to another model/tool path | Add policy-based fallback matrix: primary model, fallback model, max retry, risk tier |
|
|
| Tool Registry | Central registry for all side-effecting tools | Add `tool-registry.yaml` with schema, owner, risk level, idempotency key, timeout, rollback |
|
|
| Rollback | Recover from failed write/deploy/db steps | Add transaction boundary records and rollback scripts for `write_code`, `migration`, `deploy` |
|
|
| Business Feedback | Optimize based on business outcomes, not only technical metrics | Track cycle time, defect leakage, review rejection rate, manual rework hours |
|
|
| Harness Registry | Reuse harness modules across projects | Package H4/H5/H6 as versioned shared assets with changelog and compatibility contract |
|
|
| Central AgentOps | Aggregate traces across teams/projects | Send trace/metrics JSONL to a central dashboard or log platform |
|
|
| Governance Federation | Cross-project policy enforcement | Use signed policy bundles and a centralized approval identity provider |
|
|
| Self-Healing | Agent can choose bounded recovery path | Add failure classifier, remediation playbooks, and max retry/backoff policy |
|
|
| Continuous Evaluation | Evaluation improves with production feedback | Convert incidents and reviewer comments into golden tests automatically |
|
|
|
|
## Level 5 Exit Criteria
|
|
|
|
- At least two projects use the same versioned Harness package.
|
|
- A golden dataset detects drift before production execution.
|
|
- A failed model/tool path successfully falls back without bypassing governance.
|
|
- A side-effecting failure demonstrates rollback.
|
|
- Business KPI report shows before/after improvement.
|
|
- Central dashboard shows cost, latency, failure, rejection, and drift trends.
|
|
- Policies are signed or centrally controlled, not locally editable only.
|
|
|
|
## Level 5 Evidence Added in This Package
|
|
|
|
| Capability | Evidence |
|
|
|---|---|
|
|
| Drift detection | `.specify/scripts/bash/drift-detect.sh`, `docs/output/casan/level5-evidence/09-drift-report.json` |
|
|
| Model fallback | `.specify/scripts/bash/model-fallback.sh`, `.specify/logs/level5/fallback.jsonl` |
|
|
| Tool registry + idempotency | `.specify/level5/tool-registry.yaml`, `.specify/scripts/bash/tool-registry-gate.sh` |
|
|
| Rollback transaction | `.specify/scripts/bash/rollback-manager.sh`, `.specify/logs/level5/rollback-transactions.jsonl` |
|
|
| Business KPI feedback | `.specify/scripts/bash/business-kpi-report.sh`, `docs/output/casan/level5-evidence/14-business-kpi-report.json` |
|
|
| Central governance | `.specify/level5/central-governance/policy-bundle.yaml` |
|
|
| Signed policy bundle | `.specify/scripts/bash/sign-policy-bundle.sh`, `.specify/level5/central-governance/policy-manifest.sig` |
|
|
| Provider usage telemetry | `.specify/scripts/bash/import-provider-telemetry.sh`, `.specify/logs/level5/provider-usage.jsonl` |
|
|
| Shared harness reuse | `.specify/level5/harness-package.json`, `.specify/level5/project-registry.json` |
|
|
| Central dashboard | `docs/output/casan/central-agentops-dashboard.html` |
|
|
|
|
## Level 5 Status
|
|
|
|
This package now provides **Level 5 demonstration evidence** and working local implementations for the core Level 5 mechanisms. It should be presented as:
|
|
|
|
- Level 4: achieved.
|
|
- Level 5: demonstrated in a local/file-based environment.
|
|
|
|
To claim full enterprise production Level 5 outside this local package, connect these mechanisms to enterprise services: organization identity provider, WORM/signed log storage, real LLM provider usage APIs, and a deployed shared dashboard.
|