67 lines
3.1 KiB
Markdown
67 lines
3.1 KiB
Markdown
# CASAN Refined Re-Assessment - AINative_OKR_CASAN5
|
|
|
|
Generated: 2026-06-28
|
|
|
|
## Summary
|
|
|
|
This package merges the stable CASAN4 submission with the strongest additions from `AINative_OKR_Claude_GHCP`, then fixes the regressions found during review.
|
|
|
|
Positioning:
|
|
|
|
- CASAN Level 4: achieved.
|
|
- Level 5: stronger local demonstration, not full enterprise production.
|
|
- Evidence score: 88.7 / 100.
|
|
|
|
## Scorecard
|
|
|
|
| Harness | CASAN4 | Refined | Delta | Reason |
|
|
|---|---:|---:|---:|---|
|
|
| H1 Context | 90 | 91 | +1 | Context schema now exposes Level 5 paths for tool registry, drift, rollback, and AgentOps evidence. |
|
|
| H2 Tool | 84 | 88 | +4 | Tool registry expanded, side-effect idempotency tested, per-call audit added, cache hit no longer bypasses CASAN trace generation. |
|
|
| H3 Evaluation | 85 | 86 | +1 | Hallucination checklist added for review gates. |
|
|
| H4 Security | 88 | 90 | +2 | Jailbreak and private key tests added; private key leakage bug fixed with explicit regex and `grep --`. |
|
|
| H5 Governance | 87 | 88 | +1 | Risk policy and registry auto-update guidance added; audit chain remains verified. |
|
|
| H6 AgentOps | 89 | 90 | +1 | JSON alert format and tool-call audit evidence added. |
|
|
| H7 Orchestration | 87 | 88 | +1 | Refined wrapper preserves evidence on cache hit and keeps rollback/drift/fallback evidence. |
|
|
| **Average** | **87.1** | **88.7** | **+1.6** | Stronger Level 5 evidence path with verified regression fixes. |
|
|
|
|
## Verified Improvements
|
|
|
|
The test harness now checks:
|
|
|
|
- Prompt injection block.
|
|
- PII masking.
|
|
- Jailbreak block.
|
|
- Private key material block.
|
|
- High-risk deny-by-default.
|
|
- Explicit approval path.
|
|
- Tool-call audit log.
|
|
- Cache hit still records fresh CASAN traces.
|
|
- Audit hash-chain verification.
|
|
- Drift, fallback, tool registry, rollback, KPI, signed policy, provider telemetry, harness reuse, and central dashboard evidence.
|
|
|
|
## Regressions Fixed from AINative_OKR_Claude_GHCP
|
|
|
|
| Issue | Fix |
|
|
|---|---|
|
|
| Bash scripts were not executable in the submitted folder | New folder is copied from CASAN4 baseline with executable bits preserved. |
|
|
| Private key regex caused `grep` option parsing and leaked private key text | `security-check.sh` now uses `grep --` and explicit private key, DB connection string, and AWS key regexes. |
|
|
| Idempotency cache bypassed H4/H5/H6 on cache hit | `casan-harness.sh` now runs H4 input and H5 before cache use, records H6 cache-copy metrics, and runs H4 output. |
|
|
| Tool registry did not write central tool-call audit | `tool-registry-gate.sh` now writes `.specify/logs/audit/tool-calls.jsonl`. |
|
|
| Submission package still referenced CASAN4 paths | Paths were updated to `AINative_OKR_CASAN5`. |
|
|
|
|
## Claim Boundary
|
|
|
|
This package is stronger than CASAN4 for hackathon evidence. It should still be described as:
|
|
|
|
> Level 4 achieved; Level 5 demonstrated locally with verifiable controls.
|
|
|
|
Do not claim full production Level 5 until these are connected to enterprise services:
|
|
|
|
- IdP-backed approval and separation of duties.
|
|
- WORM or external signed log storage.
|
|
- Live provider telemetry APIs.
|
|
- Hosted dashboard with real-time ingestion.
|
|
- Shared harness package reused across independent production projects.
|
|
|