Files

T

thanhnvandClaude Fable 5 da66a36f97 feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):

- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
  dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
  Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
  usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
  provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
  a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
  successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).

New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.

Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.

Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

2026-07-05 01:14:46 +09:00

README.md

update first - 84

2026-06-30 02:21:39 +09:00

scoring-run-report.md

feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

2026-07-05 01:14:46 +09:00

README.md

Evidence Index

This folder is an index only. The canonical evidence remains inside ../AINative_OKR_CASAN5 so the source package, logs, reports, and scripts stay together.

Fastest Verification

cd ../AINative_OKR_CASAN5
bash .specify/tests/run-casan4-harness-tests.sh

Expected result: all checks pass.

High-Value Evidence Paths

Evidence Type	Path
Full test report	`../AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md`
Security attack block	`../AINative_OKR_CASAN5/docs/output/casan/evidence/01-security-attack.stderr`
PII masking output	`../AINative_OKR_CASAN5/docs/output/casan/evidence/02-pii-output.txt`
Governance deny evidence	`../AINative_OKR_CASAN5/docs/output/casan/evidence/03-governance-deny.stderr`
Approved high-risk evidence	`../AINative_OKR_CASAN5/docs/output/casan/evidence/04-high-risk-approved-output.txt`
Audit-chain verification	`../AINative_OKR_CASAN5/docs/output/casan/evidence/06b-audit-chain.stdout`
Demo pipeline context	`../AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml`
Level 5 evidence folder	`../AINative_OKR_CASAN5/docs/output/casan/level5-evidence/`
Central dashboard	`../AINative_OKR_CASAN5/docs/output/casan/central-agentops-dashboard.html`
Policy signature	`../AINative_OKR_CASAN5/.specify/level5/central-governance/policy-manifest.sig`
Public verification key	`../AINative_OKR_CASAN5/.specify/level5/central-governance/policy-public.pem`

Evidence Review Order

Run the test harness.
Open harness-test-report.md.
Open pipeline-context.yaml.
Open central-agentops-dashboard.html.
Inspect signed policy files and confirm policy-private.pem is not packaged.