update first - 84
This commit is contained in:
@@ -0,0 +1,52 @@
|
||||
# HarnessAthon Submission Package
|
||||
|
||||
## One-Line Positioning
|
||||
|
||||
We upgraded the SDD Speckit OKR pipeline from CASAN Level 3/4 transition to **CASAN Level 4 achieved** and **Level 5 demonstrated locally with verifiable controls**.
|
||||
|
||||
## Open These First
|
||||
|
||||
1. `presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
|
||||
2. `docs/00_task_breakdown.md`
|
||||
3. `docs/01_submission_checklist.md`
|
||||
4. `docs/02_pitch_script.md`
|
||||
5. `docs/03_judge_qna.md`
|
||||
6. `docs/04_evidence_map.md`
|
||||
7. `video/01_video_recording_guide.md`
|
||||
8. `ai_context/AI_README.md`
|
||||
|
||||
## Main Evidence
|
||||
|
||||
| Evidence | Source |
|
||||
|---|---|
|
||||
| Harness test report | `../AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md` |
|
||||
| CASAN assessment | `../AINative_OKR_CASAN5/docs/output/casan/casan-level4-assessment.md` |
|
||||
| Before/after scorecard | `../AINative_OKR_CASAN5/docs/output/casan/before-after-scorecard.md` |
|
||||
| Demo pipeline context | `../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml` |
|
||||
| Central dashboard | `../AINative_OKR_CASAN5/docs/output/casan/central-agentops-dashboard.html` |
|
||||
| Final zip | `../AINative_OKR_CASAN5.zip` |
|
||||
|
||||
## Folder Purpose
|
||||
|
||||
| Folder | Purpose |
|
||||
|---|---|
|
||||
| `docs/` | Checklist, pitch script, judge Q&A, evidence map, AI-optimized structure notes |
|
||||
| `presentation/` | PPT deck and deterministic generation source |
|
||||
| `video/` | Screen-recording guide and narration outline |
|
||||
| `evidence/` | Evidence index pointing to canonical generated evidence |
|
||||
| `ai_context/` | One-page context for AI/teammate review |
|
||||
|
||||
## Verification Command
|
||||
|
||||
```bash
|
||||
cd ../AINative_OKR_CASAN5
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
```
|
||||
|
||||
Expected: all PASS, including H4/H5/H6 and Level 5 evidence checks.
|
||||
|
||||
## Claim Boundary
|
||||
|
||||
- CASAN Level 4: achieved.
|
||||
- CASAN Level 5: demonstrated locally with signed policy, provider telemetry import, shared harness registry, fallback, drift detection, rollback, KPI feedback, and central dashboard.
|
||||
- Enterprise production Level 5 still needs live IdP, WORM storage, live provider API integration, and hosted dashboard.
|
||||
@@ -0,0 +1,47 @@
|
||||
# AI Context README
|
||||
|
||||
Use this file when asking another AI or teammate to inspect the submission.
|
||||
|
||||
## Goal
|
||||
|
||||
Prepare a HarnessAthon submission that proves:
|
||||
|
||||
- CASAN Level 4 achieved.
|
||||
- CASAN Level 5 demonstrated locally with verifiable evidence.
|
||||
|
||||
## Main Project
|
||||
|
||||
`../AINative_OKR_CASAN5`
|
||||
|
||||
## Fast Verification
|
||||
|
||||
```bash
|
||||
cd ../AINative_OKR_CASAN5
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
```
|
||||
|
||||
## Important Paths
|
||||
|
||||
| Area | Path |
|
||||
|---|---|
|
||||
| H4/H5/H6 scripts | `.specify/scripts/bash/` |
|
||||
| Level 5 configs | `.specify/level5/` |
|
||||
| CASAN reports | `docs/output/casan/` |
|
||||
| Demo pipeline context | `docs/output/output_logs/casan-demo/pipeline-context.yaml` |
|
||||
| Test runner | `.specify/tests/run-casan4-harness-tests.sh` |
|
||||
| Submission package | `../00_SUBMISSION_PACKAGE` |
|
||||
|
||||
## Review Priority
|
||||
|
||||
1. Verify tests pass.
|
||||
2. Check `policy-private.pem` is not in the zip.
|
||||
3. Check evidence map.
|
||||
4. Check deck and video guide.
|
||||
5. Challenge the Level 5 claim boundary.
|
||||
|
||||
## Claim Boundary
|
||||
|
||||
Do not overclaim full enterprise production Level 5. Say:
|
||||
|
||||
> Level 4 achieved. Level 5 demonstrated locally with verifiable controls. Enterprise production requires IdP, WORM storage, live provider telemetry, and hosted dashboard integration.
|
||||
|
||||
@@ -0,0 +1,94 @@
|
||||
# Task Breakdown
|
||||
|
||||
## Objective
|
||||
|
||||
Prepare a complete HarnessAthon submission package with source code, evidence, PPT, video instructions, judge Q&A, and AI-friendly review context.
|
||||
|
||||
## Task 1 - Verify Technical Pipeline
|
||||
|
||||
Status: done
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `../AINative_OKR_CASAN5`
|
||||
- `.specify/tests/run-casan4-harness-tests.sh`
|
||||
- `../AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- H4 security tests pass.
|
||||
- H5 governance tests pass.
|
||||
- H6 AgentOps tests pass.
|
||||
- Level 5 evidence checks pass.
|
||||
- No private signing key is included in the final zip.
|
||||
|
||||
## Task 2 - Package Evidence
|
||||
|
||||
Status: done
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `04_evidence_map.md`
|
||||
- `../AINative_OKR_CASAN5/docs/output/casan/`
|
||||
- `../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Every major claim maps to a concrete file path.
|
||||
- Evidence includes runtime logs, audit chain, metrics, policy signature, provider telemetry, and dashboard.
|
||||
|
||||
## Task 3 - Prepare Presentation
|
||||
|
||||
Status: done
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `../presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
|
||||
- `../presentation/slides/compile.js`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Deck explains baseline, architecture, H4/H5/H6, Level 5 mechanisms, evidence map, score, and claim boundary.
|
||||
- No placeholder text remains.
|
||||
|
||||
## Task 4 - Prepare Video Guidance
|
||||
|
||||
Status: done
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `../video/01_video_recording_guide.md`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Video script covers opening, baseline, live verification, evidence files, Level 5 boundary, and closing.
|
||||
- Recommended length is 5-7 minutes.
|
||||
|
||||
## Task 5 - Prepare Defense Material
|
||||
|
||||
Status: done
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `03_judge_qna.md`
|
||||
- `02_pitch_script.md`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- Answers distinguish Level 4 achieved from Level 5 locally demonstrated.
|
||||
- Answers explain security, governance, AgentOps, telemetry, dashboard, rollback, and production gaps.
|
||||
|
||||
## Task 6 - Optimize for AI Review
|
||||
|
||||
Status: done
|
||||
|
||||
Deliverables:
|
||||
|
||||
- `../ai_context/AI_README.md`
|
||||
- `05_ai_optimized_structure.md`
|
||||
|
||||
Acceptance criteria:
|
||||
|
||||
- AI reviewers can identify goal, verification command, important paths, and claim boundary in under one minute.
|
||||
- Markdown uses stable headings, short sections, file paths, and explicit acceptance criteria.
|
||||
|
||||
@@ -0,0 +1,46 @@
|
||||
# Submission Checklist
|
||||
|
||||
## Required Package
|
||||
|
||||
- [x] Source package: `../AINative_OKR_CASAN5`
|
||||
- [x] Zip package: `../AINative_OKR_CASAN5.zip`
|
||||
- [x] PPT deck: `presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
|
||||
- [x] Video guide: `video/01_video_recording_guide.md`
|
||||
- [x] Judge Q&A: `docs/03_judge_qna.md`
|
||||
- [x] AI context: `ai_context/AI_README.md`
|
||||
|
||||
## Demo Flow
|
||||
|
||||
1. Open the deck.
|
||||
2. Explain baseline: H4=20, H5=25, H6=30.
|
||||
3. Show the new runtime controls:
|
||||
- H4 security
|
||||
- H5 governance
|
||||
- H6 AgentOps
|
||||
4. Run the verification command.
|
||||
5. Open `central-agentops-dashboard.html`.
|
||||
6. Explain Level 5 evidence and production boundary.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
cd ../AINative_OKR_CASAN5
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
```
|
||||
|
||||
Must show:
|
||||
|
||||
- H4 blocks prompt injection.
|
||||
- H4 masks PII.
|
||||
- H5 denies high-risk action by default.
|
||||
- H5 audit-chain validates.
|
||||
- H6 records latency/cost/status and alerts.
|
||||
- Level 5 drift/fallback/tool registry/rollback/KPI/policy signature/provider telemetry/reuse/dashboard checks pass.
|
||||
|
||||
## Recommended Submission Files
|
||||
|
||||
- `AINative_OKR_CASAN5.zip`
|
||||
- `00_SUBMISSION_PACKAGE/presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
|
||||
- `00_SUBMISSION_PACKAGE/docs/03_judge_qna.md`
|
||||
- Video recording made using `video/01_video_recording_guide.md`
|
||||
|
||||
@@ -0,0 +1,34 @@
|
||||
# Pitch Script
|
||||
|
||||
## 30-Second Opening
|
||||
|
||||
We started from an SDD Speckit OKR pipeline that was already strong in context, evaluation, and orchestration, but weak in Security, Governance, and AgentOps. The original assessment placed it around CASAN Level 3 to 4 transition. Our work turns H4/H5/H6 into runtime controls with verifiable evidence.
|
||||
|
||||
## 3-Minute Story
|
||||
|
||||
1. **Problem**: The original pipeline could generate documents and code, but it lacked production controls. Prompt injection, unclear approval, missing telemetry, and weak auditability prevented a solid Level 4 claim.
|
||||
|
||||
2. **Approach**: We added a CASAN harness layer around the pipeline:
|
||||
- H4 filters input and output.
|
||||
- H5 evaluates risk, denies high-risk actions by default, and writes tamper-evident audit logs.
|
||||
- H6 records latency, cost, tokens, retry, status, and alerts.
|
||||
|
||||
3. **Evidence**: We added automated tests and evidence files. The test suite proves injection blocking, PII masking, governance denial/approval, audit-chain validation, metrics, alerting, and JSON trace validity.
|
||||
|
||||
4. **Level 5 Demonstration**: We added drift detection, model fallback, tool registry with idempotency, rollback transaction, business KPI feedback, signed central policy bundle, provider telemetry import, shared harness registry, and central dashboard.
|
||||
|
||||
5. **Conclusion**: Level 4 is achieved. Level 5 is demonstrated locally with verifiable controls. For enterprise production, the same interfaces can be connected to IdP, WORM storage, live provider telemetry APIs, and a hosted dashboard.
|
||||
|
||||
## 20-Minute Presentation Timing
|
||||
|
||||
| Time | Topic |
|
||||
|---:|---|
|
||||
| 0:00-1:30 | Problem and baseline score |
|
||||
| 1:30-4:30 | CASAN architecture |
|
||||
| 4:30-8:00 | H4/H5/H6 runtime controls |
|
||||
| 8:00-11:00 | Live verification output |
|
||||
| 11:00-14:00 | Level 5 mechanisms |
|
||||
| 14:00-16:00 | Dashboard and evidence map |
|
||||
| 16:00-18:00 | Scorecard and claim boundary |
|
||||
| 18:00-20:00 | Q&A setup |
|
||||
|
||||
@@ -0,0 +1,67 @@
|
||||
# Judge Q&A
|
||||
|
||||
## Q1. What exactly changed from the original package?
|
||||
|
||||
The original package had H4/H5/H6 scaffold files but did not enforce them in the orchestrator and did not provide runtime evidence. We added executable harness scripts, Boss protocol integration, tests, trace logs, audit logs, metrics, Level 5 mechanisms, and a submission package.
|
||||
|
||||
## Q2. Why do you claim Level 4 is achieved?
|
||||
|
||||
Because the pipeline now has automated runtime gates for:
|
||||
|
||||
- H4 Security: prompt injection block, PII masking, secret redaction, output filtering.
|
||||
- H5 Governance: risk scoring, deny-by-default, explicit approval, audit hash-chain.
|
||||
- H6 AgentOps: latency, token, cost, retry, status, alerts, trace JSON.
|
||||
|
||||
These are tested and evidenced in `harness-test-report.md`.
|
||||
|
||||
## Q3. Are you claiming full production Level 5?
|
||||
|
||||
We claim Level 5 is demonstrated locally with verifiable controls. Full enterprise production Level 5 requires external integrations: IdP-backed approvals, WORM/signed log storage, live provider telemetry APIs, and a hosted dashboard.
|
||||
|
||||
## Q4. What is the strongest evidence?
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd AINative_OKR_CASAN5
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
```
|
||||
|
||||
It validates H4/H5/H6 and Level 5 evidence end-to-end.
|
||||
|
||||
## Q5. How do you prevent prompt injection?
|
||||
|
||||
`security-check.sh` blocks known prompt injection patterns, reads policy rules, masks PII, blocks critical secrets in input, and filters output. Evidence: `01-security-attack.stderr`, `security-*.json`.
|
||||
|
||||
## Q6. How is governance enforced?
|
||||
|
||||
`governance-check.sh` scores risk by action and content. High-risk actions are denied unless explicit approval variables are present. Every decision writes an audit record with `previous_record_hash` and `record_hash`; `verify-audit-chain.sh` validates the chain.
|
||||
|
||||
## Q7. How is AgentOps measured?
|
||||
|
||||
`agent-metrics.sh` records status, exit code, latency, retry count, token estimate, cost estimate, alerts, input hash, and output hash. Provider usage can be imported through `import-provider-telemetry.sh`.
|
||||
|
||||
## Q8. What does the central dashboard show?
|
||||
|
||||
`central-agentops-dashboard.html` aggregates local metrics, provider telemetry, fallback records, tool registry decisions, and registered harness projects.
|
||||
|
||||
## Q9. What happens if a side-effecting tool runs twice?
|
||||
|
||||
The Level 5 tool registry requires idempotency keys for side-effecting tools such as deploy, migration, write_code, and external_api. Without an idempotency key, deploy is denied in the test.
|
||||
|
||||
## Q10. What is still not production-grade?
|
||||
|
||||
Local files replace enterprise services. To productionize, connect the same contracts to IdP, WORM storage, hosted dashboard, live provider APIs, and real deployment rollback hooks.
|
||||
|
||||
## Q11. How should we score it?
|
||||
|
||||
Evidence-based hackathon score:
|
||||
|
||||
- Level 4: achieved.
|
||||
- Level 5: demonstrated locally.
|
||||
- Overall score: around 88.7/100.
|
||||
|
||||
Production-strict score:
|
||||
|
||||
- Level 4: strong.
|
||||
- Level 5: ready for enterprise integration, not fully production-hosted.
|
||||
@@ -0,0 +1,22 @@
|
||||
# Evidence Map
|
||||
|
||||
| Claim | Evidence |
|
||||
|---|---|
|
||||
| H4 blocks prompt injection | `../AINative_OKR_CASAN5/docs/output/casan/evidence/01-security-attack.stderr` |
|
||||
| H4 masks PII | `../AINative_OKR_CASAN5/docs/output/casan/evidence/02-pii-output.txt` |
|
||||
| H5 denies high risk | `../AINative_OKR_CASAN5/docs/output/casan/evidence/03-governance-deny.stderr` |
|
||||
| H5 explicit approval works | `../AINative_OKR_CASAN5/docs/output/casan/evidence/04-high-risk-approved-output.txt` |
|
||||
| H5 audit chain valid | `../AINative_OKR_CASAN5/docs/output/casan/evidence/06b-audit-chain.stdout` |
|
||||
| H6 metrics recorded | `../AINative_OKR_CASAN5/.specify/logs/cost/metrics.jsonl` |
|
||||
| H6 failure alert | `../AINative_OKR_CASAN5/.specify/agentops/alerts.log` |
|
||||
| 13-step context evidence | `../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml` |
|
||||
| Drift detection | `../AINative_OKR_CASAN5/docs/output/casan/level5-evidence/09-drift-report.json` |
|
||||
| Fallback routing | `../AINative_OKR_CASAN5/.specify/logs/level5/fallback.jsonl` |
|
||||
| Tool registry/idempotency | `../AINative_OKR_CASAN5/.specify/logs/level5/tool-registry.jsonl` |
|
||||
| Rollback | `../AINative_OKR_CASAN5/.specify/logs/level5/rollback-transactions.jsonl` |
|
||||
| KPI feedback | `../AINative_OKR_CASAN5/docs/output/casan/level5-evidence/14-business-kpi-report.json` |
|
||||
| Signed policy | `../AINative_OKR_CASAN5/.specify/level5/central-governance/policy-manifest.sig` |
|
||||
| Provider telemetry | `../AINative_OKR_CASAN5/.specify/logs/level5/provider-usage.jsonl` |
|
||||
| Shared harness reuse | `../AINative_OKR_CASAN5/.specify/level5/project-registry.json` |
|
||||
| Central dashboard | `../AINative_OKR_CASAN5/docs/output/casan/central-agentops-dashboard.html` |
|
||||
|
||||
@@ -0,0 +1,56 @@
|
||||
# AI-Optimized Structure
|
||||
|
||||
## Why Optimize
|
||||
|
||||
The project now contains source files, policy files, generated evidence, dashboards, and reports. For humans and AI reviewers, the risk is context overload. The optimized structure should separate:
|
||||
|
||||
- Entry points
|
||||
- Runtime harness
|
||||
- Evidence
|
||||
- Presentation materials
|
||||
- Generated logs
|
||||
|
||||
## Recommended Top-Level Structure
|
||||
|
||||
```text
|
||||
Output_CASAN5_REFINED/
|
||||
├── README_CASAN4_DELIVERABLE.md
|
||||
├── AINative_OKR_CASAN5.zip
|
||||
├── AINative_OKR_CASAN5/
|
||||
│ ├── .claude/
|
||||
│ ├── .github/
|
||||
│ ├── .specify/
|
||||
│ │ ├── scripts/bash/ # executable harness controls
|
||||
│ │ ├── security/ # H4 policies
|
||||
│ │ ├── governance/ # H5 policies
|
||||
│ │ ├── agentops/ # H6 policies
|
||||
│ │ ├── level5/ # Level 5 configs
|
||||
│ │ └── tests/ # verification suite
|
||||
│ └── docs/output/casan/ # reports and evidence
|
||||
└── 00_SUBMISSION_PACKAGE/
|
||||
├── README.md
|
||||
├── docs/
|
||||
├── presentation/
|
||||
├── video/
|
||||
├── evidence/
|
||||
└── ai_context/
|
||||
```
|
||||
|
||||
## AI Reading Order
|
||||
|
||||
1. `00_SUBMISSION_PACKAGE/README.md`
|
||||
2. `00_SUBMISSION_PACKAGE/ai_context/AI_README.md`
|
||||
3. `AINative_OKR_CASAN5/docs/output/casan/before-after-scorecard.md`
|
||||
4. `AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md`
|
||||
5. `AINative_OKR_CASAN5/.claude/agents/protocols/casan-harness-protocol.md`
|
||||
6. `AINative_OKR_CASAN5/.specify/tests/run-casan4-harness-tests.sh`
|
||||
|
||||
## Markdown Optimization Rules
|
||||
|
||||
- Keep one claim per section.
|
||||
- Link each claim to one evidence path.
|
||||
- Separate Level 4 achieved from Level 5 demonstrated.
|
||||
- Put commands in fenced code blocks.
|
||||
- Keep Q&A concise and adversarial.
|
||||
- Avoid long raw logs in narrative files; link to logs instead.
|
||||
|
||||
@@ -0,0 +1,37 @@
|
||||
# Evidence Index
|
||||
|
||||
This folder is an index only. The canonical evidence remains inside `../AINative_OKR_CASAN5` so the source package, logs, reports, and scripts stay together.
|
||||
|
||||
## Fastest Verification
|
||||
|
||||
```bash
|
||||
cd ../AINative_OKR_CASAN5
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
```
|
||||
|
||||
Expected result: all checks pass.
|
||||
|
||||
## High-Value Evidence Paths
|
||||
|
||||
| Evidence Type | Path |
|
||||
|---|---|
|
||||
| Full test report | `../AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md` |
|
||||
| Security attack block | `../AINative_OKR_CASAN5/docs/output/casan/evidence/01-security-attack.stderr` |
|
||||
| PII masking output | `../AINative_OKR_CASAN5/docs/output/casan/evidence/02-pii-output.txt` |
|
||||
| Governance deny evidence | `../AINative_OKR_CASAN5/docs/output/casan/evidence/03-governance-deny.stderr` |
|
||||
| Approved high-risk evidence | `../AINative_OKR_CASAN5/docs/output/casan/evidence/04-high-risk-approved-output.txt` |
|
||||
| Audit-chain verification | `../AINative_OKR_CASAN5/docs/output/casan/evidence/06b-audit-chain.stdout` |
|
||||
| Demo pipeline context | `../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml` |
|
||||
| Level 5 evidence folder | `../AINative_OKR_CASAN5/docs/output/casan/level5-evidence/` |
|
||||
| Central dashboard | `../AINative_OKR_CASAN5/docs/output/casan/central-agentops-dashboard.html` |
|
||||
| Policy signature | `../AINative_OKR_CASAN5/.specify/level5/central-governance/policy-manifest.sig` |
|
||||
| Public verification key | `../AINative_OKR_CASAN5/.specify/level5/central-governance/policy-public.pem` |
|
||||
|
||||
## Evidence Review Order
|
||||
|
||||
1. Run the test harness.
|
||||
2. Open `harness-test-report.md`.
|
||||
3. Open `pipeline-context.yaml`.
|
||||
4. Open `central-agentops-dashboard.html`.
|
||||
5. Inspect signed policy files and confirm `policy-private.pem` is not packaged.
|
||||
|
||||
Binary file not shown.
@@ -0,0 +1,29 @@
|
||||
# Presentation Package
|
||||
|
||||
## Main Deck
|
||||
|
||||
`HarnessAthon_CASAN_Level5_Demo.pptx`
|
||||
|
||||
## Source
|
||||
|
||||
`slides/compile.js`
|
||||
|
||||
The deck is generated with PptxGenJS so it can be regenerated deterministically.
|
||||
|
||||
## Regenerate
|
||||
|
||||
```bash
|
||||
cd slides
|
||||
NODE_PATH=/Users/thanhnguyen/.cache/codex-runtimes/codex-primary-runtime/dependencies/node/node_modules \
|
||||
/Users/thanhnguyen/.cache/codex-runtimes/codex-primary-runtime/dependencies/node/bin/node compile.js
|
||||
```
|
||||
|
||||
## Speaking Flow
|
||||
|
||||
1. Level 4 achieved, Level 5 demonstrated locally.
|
||||
2. Baseline gaps: H4/H5/H6.
|
||||
3. Runtime harness architecture.
|
||||
4. Evidence for H4, H5, H6.
|
||||
5. Level 5 mechanisms.
|
||||
6. Claim boundary for enterprise production.
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
const pptxgen = require("pptxgenjs");
|
||||
const pptx = new pptxgen();
|
||||
|
||||
pptx.defineLayout({ name: "CASAN_WIDE", width: 10, height: 5.625 });
|
||||
pptx.layout = "CASAN_WIDE";
|
||||
pptx.author = "FPT Software AI CoE";
|
||||
pptx.company = "FPT Software";
|
||||
pptx.subject = "HarnessAthon CASAN Level 5 Demo";
|
||||
pptx.title = "HarnessAthon CASAN Level 5 Demo";
|
||||
pptx.lang = "en-US";
|
||||
pptx.theme = {
|
||||
headFontFace: "Aptos Display",
|
||||
bodyFontFace: "Aptos",
|
||||
lang: "en-US"
|
||||
};
|
||||
|
||||
const theme = {
|
||||
primary: "0A0A0A",
|
||||
secondary: "404040",
|
||||
accent: "0070F3",
|
||||
gold: "D4AF37",
|
||||
light: "F5F5F5",
|
||||
bg: "FFFFFF",
|
||||
dark: "141414",
|
||||
danger: "D90429",
|
||||
success: "0F766E"
|
||||
};
|
||||
|
||||
const W = 10;
|
||||
const H = 5.625;
|
||||
|
||||
function addBadge(slide, n) {
|
||||
slide.addShape(pptx.ShapeType.ellipse, {
|
||||
x: 9.35, y: 5.13, w: 0.34, h: 0.34,
|
||||
fill: { color: theme.primary },
|
||||
line: { color: theme.primary }
|
||||
});
|
||||
slide.addText(String(n).padStart(2, "0"), {
|
||||
x: 9.35, y: 5.18, w: 0.34, h: 0.18,
|
||||
fontSize: 8, color: "FFFFFF", align: "center",
|
||||
fontFace: "Aptos", margin: 0
|
||||
});
|
||||
}
|
||||
|
||||
function title(slide, text, subtitle) {
|
||||
slide.addText(text, {
|
||||
x: 0.52, y: 0.35, w: 7.8, h: 0.42,
|
||||
fontSize: 25, bold: true, color: theme.primary,
|
||||
fontFace: "Aptos Display", margin: 0, fit: "shrink"
|
||||
});
|
||||
if (subtitle) {
|
||||
slide.addText(subtitle, {
|
||||
x: 0.54, y: 0.82, w: 8.4, h: 0.28,
|
||||
fontSize: 10.5, color: theme.secondary,
|
||||
fontFace: "Aptos", margin: 0, fit: "shrink"
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function footer(slide) {
|
||||
slide.addText("HarnessAthon | CASAN Evidence Package", {
|
||||
x: 0.52, y: 5.18, w: 4.8, h: 0.16,
|
||||
fontSize: 7.5, color: "737373", margin: 0
|
||||
});
|
||||
}
|
||||
|
||||
function card(slide, x, y, w, h, header, body, accent = theme.accent) {
|
||||
slide.addShape(pptx.ShapeType.roundRect, {
|
||||
x, y, w, h,
|
||||
rectRadius: 0.08,
|
||||
fill: { color: "FFFFFF" },
|
||||
line: { color: "D4D4D4", transparency: 10 }
|
||||
});
|
||||
slide.addShape(pptx.ShapeType.rect, {
|
||||
x, y, w: 0.07, h,
|
||||
fill: { color: accent },
|
||||
line: { color: accent }
|
||||
});
|
||||
slide.addText(header, {
|
||||
x: x + 0.18, y: y + 0.14, w: w - 0.32, h: 0.26,
|
||||
fontSize: 12.5, bold: true, color: theme.primary, margin: 0, fit: "shrink"
|
||||
});
|
||||
slide.addText(body, {
|
||||
x: x + 0.18, y: y + 0.48, w: w - 0.32, h: h - 0.58,
|
||||
fontSize: 9.2, color: theme.secondary, margin: 0.02,
|
||||
breakLine: false, fit: "shrink"
|
||||
});
|
||||
}
|
||||
|
||||
function metric(slide, x, y, label, value, color = theme.accent) {
|
||||
slide.addText(value, {
|
||||
x, y, w: 1.35, h: 0.42,
|
||||
fontSize: 25, bold: true, color, margin: 0,
|
||||
align: "center", fit: "shrink"
|
||||
});
|
||||
slide.addText(label, {
|
||||
x: x - 0.1, y: y + 0.48, w: 1.55, h: 0.28,
|
||||
fontSize: 8.2, color: theme.secondary, margin: 0,
|
||||
align: "center", fit: "shrink"
|
||||
});
|
||||
}
|
||||
|
||||
function addScoreBar(slide, x, y, label, from, to, color) {
|
||||
slide.addText(label, { x, y, w: 1.55, h: 0.18, fontSize: 8.5, color: theme.primary, margin: 0 });
|
||||
slide.addShape(pptx.ShapeType.rect, { x: x + 1.7, y: y + 0.03, w: 2.4, h: 0.11, fill: { color: "E5E5E5" }, line: { color: "E5E5E5" } });
|
||||
slide.addShape(pptx.ShapeType.rect, { x: x + 1.7, y: y + 0.03, w: 2.4 * to / 100, h: 0.11, fill: { color }, line: { color } });
|
||||
slide.addText(`${from} -> ${to}`, { x: x + 4.25, y: y - 0.02, w: 0.75, h: 0.18, fontSize: 8.2, color: theme.secondary, margin: 0 });
|
||||
}
|
||||
|
||||
function baseSlide(n) {
|
||||
const slide = pptx.addSlide();
|
||||
slide.background = { color: theme.bg };
|
||||
slide.addShape(pptx.ShapeType.rect, { x: 0, y: 0, w: W, h: 0.08, fill: { color: theme.gold }, line: { color: theme.gold } });
|
||||
if (n > 1) {
|
||||
addBadge(slide, n);
|
||||
footer(slide);
|
||||
}
|
||||
return slide;
|
||||
}
|
||||
|
||||
// 1 Cover
|
||||
{
|
||||
const slide = baseSlide(1);
|
||||
slide.background = { color: theme.dark };
|
||||
slide.addShape(pptx.ShapeType.rect, { x: 0, y: 0, w: W, h: H, fill: { color: theme.dark }, line: { color: theme.dark } });
|
||||
slide.addShape(pptx.ShapeType.rect, { x: 6.6, y: 0, w: 3.4, h: H, fill: { color: theme.accent, transparency: 4 }, line: { color: theme.accent, transparency: 100 } });
|
||||
slide.addText("HARNESSATHON", { x: 0.55, y: 0.55, w: 3.6, h: 0.28, fontSize: 12, color: theme.gold, bold: true, margin: 0 });
|
||||
slide.addText("CASAN Level 5 Demo", { x: 0.55, y: 1.25, w: 5.9, h: 0.75, fontSize: 34, bold: true, color: "FFFFFF", margin: 0, fit: "shrink" });
|
||||
slide.addText("From Level 4 achieved to Level 5 demonstrated with executable, auditable, measurable harness controls.", {
|
||||
x: 0.58, y: 2.15, w: 5.6, h: 0.72, fontSize: 15.5, color: "E5E5E5", margin: 0.02, fit: "shrink"
|
||||
});
|
||||
card(slide, 6.95, 1.03, 2.25, 0.82, "H4 Security", "Prompt injection block\nPII + secret filtering", theme.gold);
|
||||
card(slide, 6.95, 2.08, 2.25, 0.82, "H5 Governance", "Risk approval\nSigned policy bundle", theme.gold);
|
||||
card(slide, 6.95, 3.13, 2.25, 0.82, "H6 AgentOps", "Cost, latency, alerts\nCentral dashboard", theme.gold);
|
||||
slide.addText("FPT Software AI CoE", { x: 0.58, y: 4.95, w: 2.6, h: 0.18, fontSize: 9, color: "BDBDBD", margin: 0 });
|
||||
}
|
||||
|
||||
// 2 Agenda
|
||||
{
|
||||
const slide = baseSlide(2);
|
||||
title(slide, "What the Judges Should See", "A compact map from baseline gaps to executable evidence.");
|
||||
const items = [
|
||||
["01", "Baseline", "Why Level 3 -> 4 was the starting point"],
|
||||
["02", "Runtime Harness", "How H4/H5/H6 now gate every step"],
|
||||
["03", "Evidence", "What can be verified from logs and tests"],
|
||||
["04", "Level 5", "What is demonstrated and what remains external"]
|
||||
];
|
||||
items.forEach((it, i) => {
|
||||
const x = 0.7 + (i % 2) * 4.45;
|
||||
const y = 1.45 + Math.floor(i / 2) * 1.55;
|
||||
slide.addText(it[0], { x, y, w: 0.72, h: 0.48, fontSize: 22, bold: true, color: theme.gold, margin: 0 });
|
||||
slide.addText(it[1], { x: x + 0.82, y: y + 0.03, w: 2.2, h: 0.28, fontSize: 16, bold: true, color: theme.primary, margin: 0 });
|
||||
slide.addText(it[2], { x: x + 0.82, y: y + 0.45, w: 3.2, h: 0.28, fontSize: 9.5, color: theme.secondary, margin: 0 });
|
||||
});
|
||||
}
|
||||
|
||||
// 3 Baseline
|
||||
{
|
||||
const slide = baseSlide(3);
|
||||
title(slide, "Baseline: Strong Engine, Weak Control Layer", "Original assessment had H1/H3/H7 strong, but H4/H5/H6 blocked a confident Level 4 claim.");
|
||||
card(slide, 0.6, 1.25, 2.2, 1.05, "Security Gap", "No prompt injection scan\nNo secret/PII gate\nNo output filtering", theme.danger);
|
||||
card(slide, 3.1, 1.25, 2.2, 1.05, "Governance Gap", "No approval workflow\nAudit was not tamper-evident\nNo enforced risk registry", theme.danger);
|
||||
card(slide, 5.6, 1.25, 2.2, 1.05, "AgentOps Gap", "No cost/token tracking\nNo failure alert\nNo drift signal", theme.danger);
|
||||
metric(slide, 7.98, 1.28, "Baseline Avg", "57.9", theme.danger);
|
||||
addScoreBar(slide, 0.72, 3.1, "H4 Security", 20, 84, theme.accent);
|
||||
addScoreBar(slide, 0.72, 3.55, "H5 Governance", 25, 82, theme.accent);
|
||||
addScoreBar(slide, 0.72, 4.0, "H6 AgentOps", 30, 83, theme.accent);
|
||||
addScoreBar(slide, 0.72, 4.45, "Average", 58, 87, theme.gold);
|
||||
}
|
||||
|
||||
// 4 Architecture
|
||||
{
|
||||
const slide = baseSlide(4);
|
||||
title(slide, "New Architecture: Harness Around Every Step", "H4/H5/H6 are no longer documents; they are mandatory runtime gates.");
|
||||
const stages = [
|
||||
["Input", "User or agent payload"],
|
||||
["H4", "Security scan"],
|
||||
["H5", "Risk + approval"],
|
||||
["Agent", "Real execution"],
|
||||
["H6", "Metrics + alert"],
|
||||
["H4", "Output filter"],
|
||||
["Evidence", "Trace + audit"]
|
||||
];
|
||||
stages.forEach((s, i) => {
|
||||
const x = 0.45 + i * 1.32;
|
||||
const color = i === 0 || i === 6 ? theme.primary : (i === 3 ? theme.gold : theme.accent);
|
||||
slide.addShape(pptx.ShapeType.roundRect, { x, y: 2.05, w: 1.02, h: 0.75, rectRadius: 0.08, fill: { color }, line: { color } });
|
||||
slide.addText(s[0], { x: x + 0.05, y: 2.24, w: 0.92, h: 0.18, fontSize: 12, bold: true, color: "FFFFFF", align: "center", margin: 0 });
|
||||
slide.addText(s[1], { x: x - 0.04, y: 2.93, w: 1.1, h: 0.36, fontSize: 7.5, color: theme.secondary, align: "center", margin: 0 });
|
||||
if (i < stages.length - 1) {
|
||||
slide.addText(">", { x: x + 1.1, y: 2.25, w: 0.18, h: 0.2, fontSize: 13, bold: true, color: theme.secondary, margin: 0 });
|
||||
}
|
||||
});
|
||||
card(slide, 0.75, 3.85, 2.55, 0.75, "Boss Integration", "okr.bossbuiltin mandates CASAN protocol before every delegated step.", theme.gold);
|
||||
card(slide, 3.75, 3.85, 2.55, 0.75, "Pipeline Context", "Demo context maps Steps 0-13 to H4/H5/H6 traces.", theme.gold);
|
||||
card(slide, 6.75, 3.85, 2.55, 0.75, "Evidence First", "Every claim links to executable tests and generated artifacts.", theme.gold);
|
||||
}
|
||||
|
||||
// 5 H4
|
||||
{
|
||||
const slide = baseSlide(5);
|
||||
title(slide, "H4 Security Harness", "Fail closed before LLM/tool execution and filter generated output afterwards.");
|
||||
card(slide, 0.65, 1.25, 2.35, 1.05, "Input Gate", "Blocks prompt injection\nBlocks critical secrets\nMasks email, phone, ID", theme.accent);
|
||||
card(slide, 3.25, 1.25, 2.35, 1.05, "Output Gate", "Redacts secrets\nMasks PII again\nFlags uncertainty language", theme.accent);
|
||||
card(slide, 5.85, 1.25, 2.35, 1.05, "Evidence", "security-*.json\nsecurity.jsonl\nattack + PII test files", theme.gold);
|
||||
slide.addText("Verification highlights", { x: 0.7, y: 3.05, w: 2.2, h: 0.28, fontSize: 16, bold: true, color: theme.primary, margin: 0 });
|
||||
slide.addText("PASS: H4 blocks prompt injection\nPASS: PII output contains ***MASKED_EMAIL***\nPASS: PII output contains ***MASKED_PHONE***", {
|
||||
x: 0.7, y: 3.55, w: 7.9, h: 0.72, fontSize: 11, color: theme.secondary, margin: 0.02, fit: "shrink"
|
||||
});
|
||||
}
|
||||
|
||||
// 6 H5
|
||||
{
|
||||
const slide = baseSlide(6);
|
||||
title(slide, "H5 Governance Harness", "High-risk actions are denied by default and every decision is tamper-evident.");
|
||||
card(slide, 0.6, 1.22, 2.1, 1.1, "Risk Scoring", "Action type + content\nLow / medium / high\nDeploy and secrets are high", theme.accent);
|
||||
card(slide, 2.95, 1.22, 2.1, 1.1, "Approval", "Non-interactive\nApprover identity required\nDeny by default", theme.accent);
|
||||
card(slide, 5.3, 1.22, 2.1, 1.1, "Audit Chain", "previous_record_hash\nrecord_hash\nVerifier script", theme.gold);
|
||||
metric(slide, 7.75, 1.35, "Governance Score", "87", theme.gold);
|
||||
slide.addText("Proof points", { x: 0.72, y: 3.08, w: 1.8, h: 0.3, fontSize: 15, bold: true, color: theme.primary, margin: 0 });
|
||||
slide.addText("High-risk deploy without idempotency: denied\nHigh-risk deploy with approver: approved\nAudit chain verification: POLICY + AUDIT valid", {
|
||||
x: 0.72, y: 3.55, w: 8.2, h: 0.7, fontSize: 11, color: theme.secondary, margin: 0.02
|
||||
});
|
||||
}
|
||||
|
||||
// 7 H6
|
||||
{
|
||||
const slide = baseSlide(7);
|
||||
title(slide, "H6 AgentOps Harness", "Every run becomes observable: status, latency, token, cost, retry, alerts.");
|
||||
metric(slide, 0.9, 1.35, "Trace JSON", "10", theme.accent);
|
||||
metric(slide, 2.55, 1.35, "Failure Alerts", "1+", theme.danger);
|
||||
metric(slide, 4.2, 1.35, "Provider Telemetry", "on", theme.gold);
|
||||
metric(slide, 5.85, 1.35, "Dashboard", "on", theme.success);
|
||||
card(slide, 0.72, 3.0, 2.6, 0.92, "Metrics JSONL", "latency_ms, cost_estimate, total_tokens, status, alerts", theme.accent);
|
||||
card(slide, 3.65, 3.0, 2.6, 0.92, "Provider Usage", "sample-provider record imported into provider-usage.jsonl", theme.gold);
|
||||
card(slide, 6.58, 3.0, 2.6, 0.92, "Central Dashboard", "Aggregates metrics, fallback, tools, provider usage, projects", theme.success);
|
||||
}
|
||||
|
||||
// 8 Level 5
|
||||
{
|
||||
const slide = baseSlide(8);
|
||||
title(slide, "Level 5 Mechanisms Demonstrated", "The package includes local, verifiable implementations for the main Level 5 controls.");
|
||||
const rows = [
|
||||
["Drift", "Golden run comparison"],
|
||||
["Fallback", "Primary fail -> fallback route"],
|
||||
["Tool Registry", "Owner, risk, idempotency"],
|
||||
["Rollback", "Transaction record + execute"],
|
||||
["KPI Feedback", "Business outcome report"],
|
||||
["Signed Policy", "Manifest + public verify"],
|
||||
["Reuse", "Shared harness registry"],
|
||||
["Dashboard", "Central AgentOps HTML"]
|
||||
];
|
||||
rows.forEach((r, i) => {
|
||||
const x = 0.55 + (i % 4) * 2.35;
|
||||
const y = 1.25 + Math.floor(i / 4) * 1.42;
|
||||
card(slide, x, y, 2.05, 0.82, r[0], r[1], i % 2 ? theme.gold : theme.accent);
|
||||
});
|
||||
slide.addText("Positioning: Level 4 achieved; Level 5 demonstrated locally with clear production integration boundary.", {
|
||||
x: 0.7, y: 4.45, w: 8.5, h: 0.28, fontSize: 11, color: theme.primary, bold: true, margin: 0
|
||||
});
|
||||
}
|
||||
|
||||
// 9 Evidence
|
||||
{
|
||||
const slide = baseSlide(9);
|
||||
title(slide, "Evidence Map", "The strongest part of the submission is that each claim has a file path.");
|
||||
const evidence = [
|
||||
["Injection blocked", "01-security-attack.stderr"],
|
||||
["Audit valid", "06b-audit-chain.stdout"],
|
||||
["13-step context", "pipeline-context.yaml"],
|
||||
["Policy signed", "policy-manifest.sig"],
|
||||
["Provider usage", "provider-usage.jsonl"],
|
||||
["Harness reuse", "project-registry.json"]
|
||||
];
|
||||
evidence.forEach((e, i) => {
|
||||
const y = 1.2 + i * 0.55;
|
||||
slide.addShape(pptx.ShapeType.rect, { x: 0.75, y: y + 0.04, w: 0.12, h: 0.12, fill: { color: theme.gold }, line: { color: theme.gold } });
|
||||
slide.addText(e[0], { x: 1.0, y, w: 2.4, h: 0.2, fontSize: 10.5, bold: true, color: theme.primary, margin: 0 });
|
||||
slide.addText(e[1], { x: 3.05, y, w: 5.5, h: 0.2, fontSize: 9.5, color: theme.secondary, margin: 0 });
|
||||
});
|
||||
card(slide, 6.55, 3.85, 2.55, 0.72, "Run It Live", "bash .specify/tests/run-casan4-harness-tests.sh", theme.success);
|
||||
}
|
||||
|
||||
// 10 Dashboard
|
||||
{
|
||||
const slide = baseSlide(10);
|
||||
title(slide, "Central AgentOps Dashboard", "A static dashboard proves the aggregation shape for Level 5 operations.");
|
||||
card(slide, 0.65, 1.18, 2.35, 1.02, "Aggregates", "H6 run metrics\nProvider telemetry\nFallback route records", theme.accent);
|
||||
card(slide, 3.3, 1.18, 2.35, 1.02, "Governance Signals", "Tool denials\nRegistered projects\nPolicy verification", theme.gold);
|
||||
card(slide, 5.95, 1.18, 2.35, 1.02, "Production Path", "Replace static HTML with hosted dashboard and live data ingestion", theme.success);
|
||||
slide.addShape(pptx.ShapeType.rect, { x: 0.85, y: 3.0, w: 7.95, h: 0.55, fill: { color: "F5F5F5" }, line: { color: "D4D4D4" } });
|
||||
slide.addText("central-agentops-dashboard.html", { x: 1.05, y: 3.18, w: 3.2, h: 0.18, fontSize: 10.5, bold: true, color: theme.primary, margin: 0 });
|
||||
slide.addText("Cost | Latency | Failures | Fallback | Provider Tokens | Registered Projects", {
|
||||
x: 1.05, y: 3.68, w: 7.2, h: 0.3, fontSize: 12, color: theme.secondary, margin: 0
|
||||
});
|
||||
}
|
||||
|
||||
// 11 Score
|
||||
{
|
||||
const slide = baseSlide(11);
|
||||
title(slide, "Score and Claim Boundary", "Strong enough for competition; honest enough for production review.");
|
||||
metric(slide, 0.88, 1.35, "Level 4 Score", "88.7", theme.accent);
|
||||
metric(slide, 2.55, 1.35, "Level 5 Demo", "86", theme.gold);
|
||||
metric(slide, 4.2, 1.35, "Tests", "PASS", theme.success);
|
||||
card(slide, 0.75, 3.0, 2.65, 0.95, "Claim", "Level 4 achieved.\nLevel 5 demonstrated locally.", theme.success);
|
||||
card(slide, 3.65, 3.0, 2.65, 0.95, "Boundary", "Enterprise production requires IdP, WORM logs, live provider APIs, hosted dashboard.", theme.gold);
|
||||
card(slide, 6.55, 3.0, 2.65, 0.95, "Why It Matters", "Harness is executable, auditable, measurable, and testable.", theme.accent);
|
||||
}
|
||||
|
||||
// 12 Close
|
||||
{
|
||||
const slide = baseSlide(12);
|
||||
slide.background = { color: theme.dark };
|
||||
slide.addShape(pptx.ShapeType.rect, { x: 0, y: 0, w: W, h: H, fill: { color: theme.dark }, line: { color: theme.dark } });
|
||||
addBadge(slide, 12);
|
||||
slide.addText("Ready for Review", { x: 0.75, y: 1.0, w: 5.8, h: 0.65, fontSize: 36, bold: true, color: "FFFFFF", margin: 0 });
|
||||
slide.addText("The control layer is now executable. The evidence is reproducible. The boundary is explicit.", {
|
||||
x: 0.78, y: 1.88, w: 6.6, h: 0.55, fontSize: 16, color: "E5E5E5", margin: 0.02
|
||||
});
|
||||
card(slide, 0.9, 3.25, 2.3, 0.72, "Verify", "Run harness tests", theme.gold);
|
||||
card(slide, 3.55, 3.25, 2.3, 0.72, "Inspect", "Open evidence map", theme.gold);
|
||||
card(slide, 6.2, 3.25, 2.3, 0.72, "Challenge", "Use Q&A file", theme.gold);
|
||||
slide.addText("Thank you", { x: 0.78, y: 4.95, w: 1.4, h: 0.18, fontSize: 9, color: "BDBDBD", margin: 0 });
|
||||
}
|
||||
|
||||
pptx.writeFile({ fileName: "../HarnessAthon_CASAN_Level5_Demo.pptx" });
|
||||
@@ -0,0 +1,86 @@
|
||||
# Video Recording Guide
|
||||
|
||||
## Recommended Length
|
||||
|
||||
5-7 minutes for submission video. Keep it focused on evidence.
|
||||
|
||||
## Recording Setup
|
||||
|
||||
Use Zoom, Microsoft Teams, OBS, or QuickTime screen recording.
|
||||
|
||||
Recommended screen layout:
|
||||
|
||||
- Left: terminal
|
||||
- Right: VS Code/Finder/browser with evidence files
|
||||
- Optional: open PPT in presenter mode before recording intro
|
||||
|
||||
## Video Structure
|
||||
|
||||
### 0:00-0:30 - Opening
|
||||
|
||||
Say:
|
||||
|
||||
> This submission upgrades the SDD Speckit OKR pipeline from CASAN Level 3/4 transition to Level 4 achieved and Level 5 demonstrated locally.
|
||||
|
||||
Show:
|
||||
|
||||
- `00_SUBMISSION_PACKAGE/README.md`
|
||||
- final zip
|
||||
|
||||
### 0:30-1:30 - Baseline and Architecture
|
||||
|
||||
Show:
|
||||
|
||||
- `before-after-scorecard.md`
|
||||
- `casan-level4-assessment.md`
|
||||
|
||||
Explain the original gaps: H4 Security, H5 Governance, H6 AgentOps.
|
||||
|
||||
### 1:30-3:30 - Live Verification
|
||||
|
||||
Run:
|
||||
|
||||
```bash
|
||||
cd Output_CASAN5_REFINED/AINative_OKR_CASAN5
|
||||
bash .specify/tests/run-casan4-harness-tests.sh
|
||||
```
|
||||
|
||||
Pause on PASS lines:
|
||||
|
||||
- H4 injection block
|
||||
- H5 audit-chain valid
|
||||
- Level 5 drift/fallback/tool registry/rollback/KPI/policy/provider telemetry/reuse/dashboard
|
||||
|
||||
### 3:30-4:30 - Evidence Files
|
||||
|
||||
Open:
|
||||
|
||||
- `docs/output/casan/evidence/harness-test-report.md`
|
||||
- `docs/output/output_logs/casan-demo/pipeline-context.yaml`
|
||||
- `docs/output/casan/central-agentops-dashboard.html`
|
||||
|
||||
### 4:30-5:30 - Level 5 Boundary
|
||||
|
||||
Say:
|
||||
|
||||
> Level 4 is achieved. Level 5 is demonstrated locally with verifiable controls. For enterprise production, these contracts should be connected to IdP, WORM log storage, live provider telemetry API, and hosted dashboard.
|
||||
|
||||
### 5:30-6:30 - Closing
|
||||
|
||||
Show:
|
||||
|
||||
- `presentation/HarnessAthon_CASAN_Level5_Demo.pptx`
|
||||
- `docs/03_judge_qna.md`
|
||||
|
||||
End with:
|
||||
|
||||
> The key difference is that the harness is no longer a document. It is executable, auditable, measurable, and testable.
|
||||
|
||||
## Recording Tips
|
||||
|
||||
- Do one dry run before recording.
|
||||
- Zoom terminal font to at least 16pt.
|
||||
- Keep the terminal command history clean.
|
||||
- Do not show secrets or personal desktop notifications.
|
||||
- Keep the final video file name simple: `HarnessAthon_CASAN_Level5_Demo.mp4`.
|
||||
|
||||
Reference in New Issue
Block a user