183 lines
7.2 KiB
Markdown
183 lines
7.2 KiB
Markdown
# CASAN Level 4 Assessment - AINATIVE_OKR_CASAN5
|
|
|
|
## Executive Summary
|
|
|
|
This deliverable upgrades the SDD Speckit OKR pipeline from a Level 3-oriented scaffold to a CASAN Level 4-ready pipeline by making H4 Security, H5 Governance, and H6 AgentOps mandatory runtime gates.
|
|
|
|
The target state is not just documentation. The project includes executable scripts, policies, orchestrator instructions, tests, and evidence paths.
|
|
|
|
## Scorecard
|
|
|
|
| ID | Harness | Baseline | CASAN4 target | Evidence |
|
|
|---|---:|---:|---:|---|
|
|
| H1 | Context Harness | 90 | 90 | `pipeline-context.yaml` protocol, templates, memory |
|
|
| H2 | Tool Harness | 75 | 78 | deterministic scripts, explicit action names, side-effect gates |
|
|
| H3 | Evaluation Harness | 85 | 85 | existing review gates and testkit |
|
|
| H4 | Security Harness | 20 | 84 | policy-backed prompt injection block, PII masking, secret redaction, output filter |
|
|
| H5 | Governance Harness | 25 | 82 | risk scoring, non-interactive approval, hash-chain audit log + verifier |
|
|
| H6 | AgentOps Harness | 30 | 83 | trace JSON, token/cost/latency/retry/status, alerts |
|
|
| H7 | Orchestration Harness | 80 | 81 | Boss requires CASAN gate protocol around every step; demo context covers Steps 0-13 |
|
|
|
|
Average target score: 88.7 / 100.
|
|
|
|
CASAN result: Level 4 - Automated, provided the full pipeline is executed with the included harness gates and evidence is retained.
|
|
|
|
## What Changed
|
|
|
|
### H4 - Security Harness
|
|
|
|
Implemented:
|
|
|
|
- Prompt injection block for override, role hijack, hidden/system prompt reveal attempts.
|
|
- PII masking for email, phone, personal ID.
|
|
- Secret detection and redaction.
|
|
- Output filtering mode after agent/tool execution.
|
|
- Security audit JSONL and trace JSON.
|
|
- Policy values are read from `.specify/security/prompt-filter.yaml` and `.specify/security/output-policy.yaml` where applicable, with safe shell defaults retained.
|
|
|
|
Key files:
|
|
|
|
- `.specify/scripts/bash/security-check.sh`
|
|
- `.specify/security/prompt-filter.yaml`
|
|
- `.specify/security/pii-rules.yaml`
|
|
- `.specify/security/output-policy.yaml`
|
|
- `.specify/logs/audit/security.jsonl`
|
|
- `.specify/logs/trace/security-*.json`
|
|
|
|
### H5 - Governance Harness
|
|
|
|
Implemented:
|
|
|
|
- Risk scoring by action type and content.
|
|
- Low risk auto-approval with audit.
|
|
- Medium risk auto-approval with audit marker.
|
|
- High risk deny-by-default.
|
|
- Explicit high-risk override using `CASAN_APPROVAL_DECISION=approve` and `CASAN_APPROVER`.
|
|
- Append-only audit JSONL with `previous_record_hash` and `record_hash`.
|
|
- Audit-chain verifier script: `.specify/scripts/bash/verify-audit-chain.sh`.
|
|
|
|
Key files:
|
|
|
|
- `.specify/scripts/bash/governance-check.sh`
|
|
- `.specify/governance/approval-flow.yaml`
|
|
- `.specify/governance/audit-log.yaml`
|
|
- `.specify/governance/risk-registry.yaml`
|
|
- `.specify/logs/audit/audit.jsonl`
|
|
- `.specify/logs/trace/governance-*.json`
|
|
|
|
### H6 - AgentOps Harness
|
|
|
|
Implemented:
|
|
|
|
- Per-step latency.
|
|
- Token estimate.
|
|
- Cost estimate.
|
|
- Retry count capture.
|
|
- Exit code and status.
|
|
- Failure, retry, latency, token alert hooks.
|
|
- Valid JSON traces and JSONL metrics.
|
|
- Demo pipeline context proving the 13-step orchestration shape: `docs/output/output_logs/casan-demo/pipeline-context.yaml`.
|
|
|
|
Key files:
|
|
|
|
- `.specify/scripts/bash/agent-metrics.sh`
|
|
- `.specify/agentops/metrics.yaml`
|
|
- `.specify/agentops/alerts.yaml`
|
|
- `.specify/agentops/tracking.yaml`
|
|
- `.specify/logs/cost/metrics.jsonl`
|
|
- `.specify/logs/trace/agentops-*.json`
|
|
- `.specify/agentops/alerts.log`
|
|
|
|
## Orchestration Integration
|
|
|
|
Boss orchestration now requires `protocols/casan-harness-protocol.md`.
|
|
|
|
Every delegated step follows:
|
|
|
|
1. H4 input security.
|
|
2. H5 governance.
|
|
3. H6 metrics around real execution.
|
|
4. H4 output filtering.
|
|
5. Pipeline context update with evidence paths.
|
|
|
|
Key files:
|
|
|
|
- `.claude/agents/okr.bossbuiltin.md`
|
|
- `.github/agents/okr.bossbuiltin.agent.md`
|
|
- `.claude/agents/protocols/casan-harness-protocol.md`
|
|
- `.github/agents/protocols/casan-harness-protocol.md`
|
|
- `.claude/agents/protocols/pipeline-context.md`
|
|
- `.github/agents/protocols/pipeline-context.md`
|
|
|
|
## Level 4 Acceptance Criteria
|
|
|
|
The submission is Level 4-ready only when all criteria below pass:
|
|
|
|
| Criterion | Required evidence |
|
|
|---|---|
|
|
| Prompt injection is blocked | `01-security-attack.stderr`, `security-*.json` |
|
|
| PII is masked before LLM input | `02-pii-output.txt` |
|
|
| Secrets are blocked or redacted | `security.jsonl`, output policy |
|
|
| High-risk action is denied by default | `03-governance-deny.stderr`, `governance-*.json` |
|
|
| Approved high-risk action has approver identity | `04-governance-approve.stdout`, `audit.jsonl` |
|
|
| Audit is tamper-evident | `previous_record_hash`, `record_hash` in `audit.jsonl` |
|
|
| Audit hash chain validates | `06b-audit-chain.stdout` |
|
|
| AgentOps metrics are recorded | `metrics.jsonl` |
|
|
| Failed execution produces alert | `alerts.log` |
|
|
| Trace files are valid JSON | harness test report |
|
|
| Boss mandates H4/H5/H6 gates | updated boss files and CASAN protocol |
|
|
| Demo context maps Steps 0-13 to CASAN evidence | `docs/output/output_logs/casan-demo/pipeline-context.yaml` |
|
|
|
|
## Test Command
|
|
|
|
Run from project root:
|
|
|
|
```bash
|
|
bash .specify/tests/run-casan4-harness-tests.sh
|
|
```
|
|
|
|
Expected result:
|
|
|
|
- Prompt injection test exits with security block.
|
|
- PII test outputs masked email and phone.
|
|
- High-risk governance test denies by default.
|
|
- Explicit approver test passes.
|
|
- AgentOps writes metrics and failure alert.
|
|
- Wrapper completes end-to-end.
|
|
- All trace JSON files validate.
|
|
|
|
## Higher CASAN Level Criteria
|
|
|
|
### To sustain Level 4
|
|
|
|
- Keep all gates mandatory in Boss.
|
|
- Fail closed on security and governance errors.
|
|
- Keep audit and metrics logs immutable or externally backed.
|
|
- Keep evidence for every full pipeline run.
|
|
- Review risk registry after every new tool/action type.
|
|
|
|
### To move toward Level 5 - Native
|
|
|
|
- Add drift detection by comparing current agent output with golden runs.
|
|
- Add model fallback routing with policy constraints.
|
|
- Add reusable enterprise tool registry with schema and idempotency keys.
|
|
- Add business KPI feedback loop, not only technical metrics.
|
|
- Add centralized dashboard for cross-project AgentOps.
|
|
- Add automatic harness self-tuning based on failure patterns.
|
|
- Add transaction rollback for failed side-effecting steps.
|
|
- Add multi-agent dependency graph with dynamic re-planning and bounded retries.
|
|
|
|
## Level 5 Production-Style Add-ons
|
|
|
|
The package also includes local/file-based implementations for the final Level 5 blockers:
|
|
|
|
| Production Level 5 item | Implemented evidence |
|
|
|---|---|
|
|
| Central governance | `.specify/level5/central-governance/policy-bundle.yaml` |
|
|
| Signed policy bundle | `.specify/level5/central-governance/policy-manifest.json`, `.specify/level5/central-governance/policy-manifest.sig`, `.specify/level5/central-governance/policy-public.pem` |
|
|
| Real provider telemetry path | `.specify/scripts/bash/import-provider-telemetry.sh`, `.specify/logs/level5/provider-usage.jsonl` |
|
|
| Shared harness package across projects | `.specify/level5/harness-package.json`, `.specify/level5/project-registry.json` |
|
|
| Central dashboard | `docs/output/casan/central-agentops-dashboard.html` |
|
|
|
|
Level 5 status: demonstrated locally with verifiable evidence. Enterprise production rollout still requires connecting the same controls to external services such as IdP approvals, WORM log storage, live provider usage APIs, and a hosted dashboard.
|