gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
CASAN/docs/output/casan/casan-level4-assessment.md
T
thanhnvandClaude Opus 4.8 36a4812ef3 refactor(structure): promote app to repo root + remove redundant workspace cruft 
Standard production layout: the OKR app (was nested under AINative_OKR_CASAN5/) is now
the repository root. No more wrapper directory.

- Promote AINative_OKR_CASAN5/* -> repo root (backend/ frontend/ packages/ apps/
  .specify/ docs/ infra/ nginx/ scripts/ + configs). Merge tool dirs: .gitea (kept the
  active deploy ci.yml, added harness-ci.yml + runbooks), .claude (agents/commands +
  launch.json), .github moved up.
- Remove redundant: 00_SUBMISSION_PACKAGE, scattered root notes (FPT_CASAN_Full.md,
  tu-tuong-casan.md, casan-tu-sinh..., casan_harness_assessment.md, source-review...,
  README_CASAN5_REFINED.md), casan-next-plans/ and optimize-docs/ (competition/planning
  artifacts — roadmap + design history preserved in git log / commit messages).
- Update all references to the old layout:
  - .gitea/workflows/{ci,harness-ci}.yml, .github/workflows/{ci,deploy}.yml:
    working-directory .; drop AINative_OKR_CASAN5/ prefix; .specify/{tests,scripts}
    -> packages/casan-harness/... (.specify/logs state kept)
  - .claude/launch.json, .gitea/*-runbook.md: path prefixes
  - CLAUDE.md, README.md: docs/input -> apps/okr/domain/input
  - policy-bundle.yaml: 8 policy paths -> packages/casan-harness/...; manifest re-signed
- secrets-scan.sh: fixture excludes -> new package/domain paths.

Full gate from the new root: PASS=64 FAIL=0 SKIP=3.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-08 13:26:36 +09:00

7.2 KiB
Raw Permalink Blame History

CASAN Level 4 Assessment - AINATIVE_OKR_CASAN5

Executive Summary

This deliverable upgrades the SDD Speckit OKR pipeline from a Level 3-oriented scaffold to a CASAN Level 4-ready pipeline by making H4 Security, H5 Governance, and H6 AgentOps mandatory runtime gates.

The target state is not just documentation. The project includes executable scripts, policies, orchestrator instructions, tests, and evidence paths.

Scorecard

ID Harness Baseline CASAN4 target Evidence
H1 Context Harness 90 90 pipeline-context.yaml protocol, templates, memory
H2 Tool Harness 75 78 deterministic scripts, explicit action names, side-effect gates
H3 Evaluation Harness 85 85 existing review gates and testkit
H4 Security Harness 20 84 policy-backed prompt injection block, PII masking, secret redaction, output filter
H5 Governance Harness 25 82 risk scoring, non-interactive approval, hash-chain audit log + verifier
H6 AgentOps Harness 30 83 trace JSON, token/cost/latency/retry/status, alerts
H7 Orchestration Harness 80 81 Boss requires CASAN gate protocol around every step; demo context covers Steps 0-13

Average target score: 88.7 / 100.

CASAN result: Level 4 - Automated, provided the full pipeline is executed with the included harness gates and evidence is retained.

What Changed

H4 - Security Harness

Implemented:

  • Prompt injection block for override, role hijack, hidden/system prompt reveal attempts.
  • PII masking for email, phone, personal ID.
  • Secret detection and redaction.
  • Output filtering mode after agent/tool execution.
  • Security audit JSONL and trace JSON.
  • Policy values are read from .specify/security/prompt-filter.yaml and .specify/security/output-policy.yaml where applicable, with safe shell defaults retained.

Key files:

  • .specify/scripts/bash/security-check.sh
  • .specify/security/prompt-filter.yaml
  • .specify/security/pii-rules.yaml
  • .specify/security/output-policy.yaml
  • .specify/logs/audit/security.jsonl
  • .specify/logs/trace/security-*.json

H5 - Governance Harness

Implemented:

  • Risk scoring by action type and content.
  • Low risk auto-approval with audit.
  • Medium risk auto-approval with audit marker.
  • High risk deny-by-default.
  • Explicit high-risk override using CASAN_APPROVAL_DECISION=approve and CASAN_APPROVER.
  • Append-only audit JSONL with previous_record_hash and record_hash.
  • Audit-chain verifier script: .specify/scripts/bash/verify-audit-chain.sh.

Key files:

  • .specify/scripts/bash/governance-check.sh
  • .specify/governance/approval-flow.yaml
  • .specify/governance/audit-log.yaml
  • .specify/governance/risk-registry.yaml
  • .specify/logs/audit/audit.jsonl
  • .specify/logs/trace/governance-*.json

H6 - AgentOps Harness

Implemented:

  • Per-step latency.
  • Token estimate.
  • Cost estimate.
  • Retry count capture.
  • Exit code and status.
  • Failure, retry, latency, token alert hooks.
  • Valid JSON traces and JSONL metrics.
  • Demo pipeline context proving the 13-step orchestration shape: docs/output/output_logs/casan-demo/pipeline-context.yaml.

Key files:

  • .specify/scripts/bash/agent-metrics.sh
  • .specify/agentops/metrics.yaml
  • .specify/agentops/alerts.yaml
  • .specify/agentops/tracking.yaml
  • .specify/logs/cost/metrics.jsonl
  • .specify/logs/trace/agentops-*.json
  • .specify/agentops/alerts.log

Orchestration Integration

Boss orchestration now requires protocols/casan-harness-protocol.md.

Every delegated step follows:

  1. H4 input security.
  2. H5 governance.
  3. H6 metrics around real execution.
  4. H4 output filtering.
  5. Pipeline context update with evidence paths.

Key files:

  • .claude/agents/okr.bossbuiltin.md
  • .github/agents/okr.bossbuiltin.agent.md
  • .claude/agents/protocols/casan-harness-protocol.md
  • .github/agents/protocols/casan-harness-protocol.md
  • .claude/agents/protocols/pipeline-context.md
  • .github/agents/protocols/pipeline-context.md

Level 4 Acceptance Criteria

The submission is Level 4-ready only when all criteria below pass:

Criterion Required evidence
Prompt injection is blocked 01-security-attack.stderr, security-*.json
PII is masked before LLM input 02-pii-output.txt
Secrets are blocked or redacted security.jsonl, output policy
High-risk action is denied by default 03-governance-deny.stderr, governance-*.json
Approved high-risk action has approver identity 04-governance-approve.stdout, audit.jsonl
Audit is tamper-evident previous_record_hash, record_hash in audit.jsonl
Audit hash chain validates 06b-audit-chain.stdout
AgentOps metrics are recorded metrics.jsonl
Failed execution produces alert alerts.log
Trace files are valid JSON harness test report
Boss mandates H4/H5/H6 gates updated boss files and CASAN protocol
Demo context maps Steps 0-13 to CASAN evidence docs/output/output_logs/casan-demo/pipeline-context.yaml

Test Command

Run from project root:

bash .specify/tests/run-casan4-harness-tests.sh

Expected result:

  • Prompt injection test exits with security block.
  • PII test outputs masked email and phone.
  • High-risk governance test denies by default.
  • Explicit approver test passes.
  • AgentOps writes metrics and failure alert.
  • Wrapper completes end-to-end.
  • All trace JSON files validate.

Higher CASAN Level Criteria

To sustain Level 4

  • Keep all gates mandatory in Boss.
  • Fail closed on security and governance errors.
  • Keep audit and metrics logs immutable or externally backed.
  • Keep evidence for every full pipeline run.
  • Review risk registry after every new tool/action type.

To move toward Level 5 - Native

  • Add drift detection by comparing current agent output with golden runs.
  • Add model fallback routing with policy constraints.
  • Add reusable enterprise tool registry with schema and idempotency keys.
  • Add business KPI feedback loop, not only technical metrics.
  • Add centralized dashboard for cross-project AgentOps.
  • Add automatic harness self-tuning based on failure patterns.
  • Add transaction rollback for failed side-effecting steps.
  • Add multi-agent dependency graph with dynamic re-planning and bounded retries.

Level 5 Production-Style Add-ons

The package also includes local/file-based implementations for the final Level 5 blockers:

Production Level 5 item Implemented evidence
Central governance .specify/level5/central-governance/policy-bundle.yaml
Signed policy bundle .specify/level5/central-governance/policy-manifest.json, .specify/level5/central-governance/policy-manifest.sig, .specify/level5/central-governance/policy-public.pem
Real provider telemetry path .specify/scripts/bash/import-provider-telemetry.sh, .specify/logs/level5/provider-usage.jsonl
Shared harness package across projects .specify/level5/harness-package.json, .specify/level5/project-registry.json
Central dashboard docs/output/casan/central-agentops-dashboard.html

Level 5 status: demonstrated locally with verifiable evidence. Enterprise production rollout still requires connecting the same controls to external services such as IdP approvals, WORM log storage, live provider usage APIs, and a hosted dashboard.

Reference in New Issue View Git Blame Copy Permalink