gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

update first - 84

Browse Source
This commit is contained in:
thanhnv
2026-06-30 02:21:39 +09:00
commit 07ac1bdcdd
561 changed files with 88164 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
00_SUBMISSION_PACKAGE/docs/00_task_breakdown.md
+94
View File
@@ -0,0 +1,94 @@
# Task Breakdown
## Objective
Prepare a complete HarnessAthon submission package with source code, evidence, PPT, video instructions, judge Q&A, and AI-friendly review context.
## Task 1 - Verify Technical Pipeline
Status: done
Deliverables:
- `../AINative_OKR_CASAN5`
- `.specify/tests/run-casan4-harness-tests.sh`
- `../AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md`
Acceptance criteria:
- H4 security tests pass.
- H5 governance tests pass.
- H6 AgentOps tests pass.
- Level 5 evidence checks pass.
- No private signing key is included in the final zip.
## Task 2 - Package Evidence
Status: done
Deliverables:
- `04_evidence_map.md`
- `../AINative_OKR_CASAN5/docs/output/casan/`
- `../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml`
Acceptance criteria:
- Every major claim maps to a concrete file path.
- Evidence includes runtime logs, audit chain, metrics, policy signature, provider telemetry, and dashboard.
## Task 3 - Prepare Presentation
Status: done
Deliverables:
- `../presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
- `../presentation/slides/compile.js`
Acceptance criteria:
- Deck explains baseline, architecture, H4/H5/H6, Level 5 mechanisms, evidence map, score, and claim boundary.
- No placeholder text remains.
## Task 4 - Prepare Video Guidance
Status: done
Deliverables:
- `../video/01_video_recording_guide.md`
Acceptance criteria:
- Video script covers opening, baseline, live verification, evidence files, Level 5 boundary, and closing.
- Recommended length is 5-7 minutes.
## Task 5 - Prepare Defense Material
Status: done
Deliverables:
- `03_judge_qna.md`
- `02_pitch_script.md`
Acceptance criteria:
- Answers distinguish Level 4 achieved from Level 5 locally demonstrated.
- Answers explain security, governance, AgentOps, telemetry, dashboard, rollback, and production gaps.
## Task 6 - Optimize for AI Review
Status: done
Deliverables:
- `../ai_context/AI_README.md`
- `05_ai_optimized_structure.md`
Acceptance criteria:
- AI reviewers can identify goal, verification command, important paths, and claim boundary in under one minute.
- Markdown uses stable headings, short sections, file paths, and explicit acceptance criteria.
00_SUBMISSION_PACKAGE/docs/01_submission_checklist.md
+46
View File
@@ -0,0 +1,46 @@
# Submission Checklist
## Required Package
- [x] Source package: `../AINative_OKR_CASAN5`
- [x] Zip package: `../AINative_OKR_CASAN5.zip`
- [x] PPT deck: `presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
- [x] Video guide: `video/01_video_recording_guide.md`
- [x] Judge Q&A: `docs/03_judge_qna.md`
- [x] AI context: `ai_context/AI_README.md`
## Demo Flow
1. Open the deck.
2. Explain baseline: H4=20, H5=25, H6=30.
3. Show the new runtime controls:
- H4 security
- H5 governance
- H6 AgentOps
4. Run the verification command.
5. Open `central-agentops-dashboard.html`.
6. Explain Level 5 evidence and production boundary.
## Verification
```bash
cd ../AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh
```
Must show:
- H4 blocks prompt injection.
- H4 masks PII.
- H5 denies high-risk action by default.
- H5 audit-chain validates.
- H6 records latency/cost/status and alerts.
- Level 5 drift/fallback/tool registry/rollback/KPI/policy signature/provider telemetry/reuse/dashboard checks pass.
## Recommended Submission Files
- `AINative_OKR_CASAN5.zip`
- `00_SUBMISSION_PACKAGE/presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
- `00_SUBMISSION_PACKAGE/docs/03_judge_qna.md`
- Video recording made using `video/01_video_recording_guide.md`
00_SUBMISSION_PACKAGE/docs/02_pitch_script.md
+34
View File
@@ -0,0 +1,34 @@
# Pitch Script
## 30-Second Opening
We started from an SDD Speckit OKR pipeline that was already strong in context, evaluation, and orchestration, but weak in Security, Governance, and AgentOps. The original assessment placed it around CASAN Level 3 to 4 transition. Our work turns H4/H5/H6 into runtime controls with verifiable evidence.
## 3-Minute Story
1. **Problem**: The original pipeline could generate documents and code, but it lacked production controls. Prompt injection, unclear approval, missing telemetry, and weak auditability prevented a solid Level 4 claim.
2. **Approach**: We added a CASAN harness layer around the pipeline:
- H4 filters input and output.
- H5 evaluates risk, denies high-risk actions by default, and writes tamper-evident audit logs.
- H6 records latency, cost, tokens, retry, status, and alerts.
3. **Evidence**: We added automated tests and evidence files. The test suite proves injection blocking, PII masking, governance denial/approval, audit-chain validation, metrics, alerting, and JSON trace validity.
4. **Level 5 Demonstration**: We added drift detection, model fallback, tool registry with idempotency, rollback transaction, business KPI feedback, signed central policy bundle, provider telemetry import, shared harness registry, and central dashboard.
5. **Conclusion**: Level 4 is achieved. Level 5 is demonstrated locally with verifiable controls. For enterprise production, the same interfaces can be connected to IdP, WORM storage, live provider telemetry APIs, and a hosted dashboard.
## 20-Minute Presentation Timing
| Time | Topic |
|---:|---|
| 0:00-1:30 | Problem and baseline score |
| 1:30-4:30 | CASAN architecture |
| 4:30-8:00 | H4/H5/H6 runtime controls |
| 8:00-11:00 | Live verification output |
| 11:00-14:00 | Level 5 mechanisms |
| 14:00-16:00 | Dashboard and evidence map |
| 16:00-18:00 | Scorecard and claim boundary |
| 18:00-20:00 | Q&A setup |
00_SUBMISSION_PACKAGE/docs/03_judge_qna.md
+67
View File
@@ -0,0 +1,67 @@
# Judge Q&A
## Q1. What exactly changed from the original package?
The original package had H4/H5/H6 scaffold files but did not enforce them in the orchestrator and did not provide runtime evidence. We added executable harness scripts, Boss protocol integration, tests, trace logs, audit logs, metrics, Level 5 mechanisms, and a submission package.
## Q2. Why do you claim Level 4 is achieved?
Because the pipeline now has automated runtime gates for:
- H4 Security: prompt injection block, PII masking, secret redaction, output filtering.
- H5 Governance: risk scoring, deny-by-default, explicit approval, audit hash-chain.
- H6 AgentOps: latency, token, cost, retry, status, alerts, trace JSON.
These are tested and evidenced in `harness-test-report.md`.
## Q3. Are you claiming full production Level 5?
We claim Level 5 is demonstrated locally with verifiable controls. Full enterprise production Level 5 requires external integrations: IdP-backed approvals, WORM/signed log storage, live provider telemetry APIs, and a hosted dashboard.
## Q4. What is the strongest evidence?
Run:
```bash
cd AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh
```
It validates H4/H5/H6 and Level 5 evidence end-to-end.
## Q5. How do you prevent prompt injection?
`security-check.sh` blocks known prompt injection patterns, reads policy rules, masks PII, blocks critical secrets in input, and filters output. Evidence: `01-security-attack.stderr`, `security-*.json`.
## Q6. How is governance enforced?
`governance-check.sh` scores risk by action and content. High-risk actions are denied unless explicit approval variables are present. Every decision writes an audit record with `previous_record_hash` and `record_hash`; `verify-audit-chain.sh` validates the chain.
## Q7. How is AgentOps measured?
`agent-metrics.sh` records status, exit code, latency, retry count, token estimate, cost estimate, alerts, input hash, and output hash. Provider usage can be imported through `import-provider-telemetry.sh`.
## Q8. What does the central dashboard show?
`central-agentops-dashboard.html` aggregates local metrics, provider telemetry, fallback records, tool registry decisions, and registered harness projects.
## Q9. What happens if a side-effecting tool runs twice?
The Level 5 tool registry requires idempotency keys for side-effecting tools such as deploy, migration, write_code, and external_api. Without an idempotency key, deploy is denied in the test.
## Q10. What is still not production-grade?
Local files replace enterprise services. To productionize, connect the same contracts to IdP, WORM storage, hosted dashboard, live provider APIs, and real deployment rollback hooks.
## Q11. How should we score it?
Evidence-based hackathon score:
- Level 4: achieved.
- Level 5: demonstrated locally.
- Overall score: around 88.7/100.
Production-strict score:
- Level 4: strong.
- Level 5: ready for enterprise integration, not fully production-hosted.
00_SUBMISSION_PACKAGE/docs/04_evidence_map.md
+22
View File
@@ -0,0 +1,22 @@
# Evidence Map
| Claim | Evidence |
|---|---|
| H4 blocks prompt injection | `../AINative_OKR_CASAN5/docs/output/casan/evidence/01-security-attack.stderr` |
| H4 masks PII | `../AINative_OKR_CASAN5/docs/output/casan/evidence/02-pii-output.txt` |
| H5 denies high risk | `../AINative_OKR_CASAN5/docs/output/casan/evidence/03-governance-deny.stderr` |
| H5 explicit approval works | `../AINative_OKR_CASAN5/docs/output/casan/evidence/04-high-risk-approved-output.txt` |
| H5 audit chain valid | `../AINative_OKR_CASAN5/docs/output/casan/evidence/06b-audit-chain.stdout` |
| H6 metrics recorded | `../AINative_OKR_CASAN5/.specify/logs/cost/metrics.jsonl` |
| H6 failure alert | `../AINative_OKR_CASAN5/.specify/agentops/alerts.log` |
| 13-step context evidence | `../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml` |
| Drift detection | `../AINative_OKR_CASAN5/docs/output/casan/level5-evidence/09-drift-report.json` |
| Fallback routing | `../AINative_OKR_CASAN5/.specify/logs/level5/fallback.jsonl` |
| Tool registry/idempotency | `../AINative_OKR_CASAN5/.specify/logs/level5/tool-registry.jsonl` |
| Rollback | `../AINative_OKR_CASAN5/.specify/logs/level5/rollback-transactions.jsonl` |
| KPI feedback | `../AINative_OKR_CASAN5/docs/output/casan/level5-evidence/14-business-kpi-report.json` |
| Signed policy | `../AINative_OKR_CASAN5/.specify/level5/central-governance/policy-manifest.sig` |
| Provider telemetry | `../AINative_OKR_CASAN5/.specify/logs/level5/provider-usage.jsonl` |
| Shared harness reuse | `../AINative_OKR_CASAN5/.specify/level5/project-registry.json` |
| Central dashboard | `../AINative_OKR_CASAN5/docs/output/casan/central-agentops-dashboard.html` |
00_SUBMISSION_PACKAGE/docs/05_ai_optimized_structure.md
+56
View File
@@ -0,0 +1,56 @@
# AI-Optimized Structure
## Why Optimize
The project now contains source files, policy files, generated evidence, dashboards, and reports. For humans and AI reviewers, the risk is context overload. The optimized structure should separate:
- Entry points
- Runtime harness
- Evidence
- Presentation materials
- Generated logs
## Recommended Top-Level Structure
```text
Output_CASAN5_REFINED/
├── README_CASAN4_DELIVERABLE.md
├── AINative_OKR_CASAN5.zip
├── AINative_OKR_CASAN5/
│ ├── .claude/
│ ├── .github/
│ ├── .specify/
│ │ ├── scripts/bash/ # executable harness controls
│ │ ├── security/ # H4 policies
│ │ ├── governance/ # H5 policies
│ │ ├── agentops/ # H6 policies
│ │ ├── level5/ # Level 5 configs
│ │ └── tests/ # verification suite
│ └── docs/output/casan/ # reports and evidence
└── 00_SUBMISSION_PACKAGE/
├── README.md
├── docs/
├── presentation/
├── video/
├── evidence/
└── ai_context/
```
## AI Reading Order
1. `00_SUBMISSION_PACKAGE/README.md`
2. `00_SUBMISSION_PACKAGE/ai_context/AI_README.md`
3. `AINative_OKR_CASAN5/docs/output/casan/before-after-scorecard.md`
4. `AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md`
5. `AINative_OKR_CASAN5/.claude/agents/protocols/casan-harness-protocol.md`
6. `AINative_OKR_CASAN5/.specify/tests/run-casan4-harness-tests.sh`
## Markdown Optimization Rules
- Keep one claim per section.
- Link each claim to one evidence path.
- Separate Level 4 achieved from Level 5 demonstrated.
- Put commands in fenced code blocks.
- Keep Q&A concise and adversarial.
- Avoid long raw logs in narrative files; link to logs instead.