gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Browse Source 
Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):

- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
  dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
  Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
  usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
  provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
  a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
  successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).

New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.

Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.

Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-05 01:14:46 +09:00
co-authored by Claude Fable 5
parent 2af67ef6a5
commit da66a36f97
55 changed files with 1306 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/docs/output/casan/evidence/07b-wrapper-cache.stdout
+4 -4
View File
@@ -1,5 +1,5 @@
SECURITY_PASS trace_id=e1ec9ef9-0236-4778-986c-770ae163a72f risk=low action=allow output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/tmp/security-input-1783091015-1003.txt
GOVERNANCE_APPROVED trace_id=4d42037d-403d-42a7-a67f-e22ef114c890 risk=low approval_status=auto_approved output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/tmp/governance-approved-1783091015-1003.txt
AGENTOPS_RECORDED trace_id=a607903d-5d84-40fc-80ea-139c96c133ae status=success latency_ms=218 tokens=14 cost=0.00002800 output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/tmp/raw-output-1783091015-1003.txt
SECURITY_PASS trace_id=2d1e3749-25f6-4272-90fb-df2ce7337aaa risk=low action=allow output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/docs/output/casan/evidence/07-wrapper-output.txt
SECURITY_PASS trace_id=f22da2ba-c99b-4335-8895-8e133b927a82 risk=low action=allow output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/tmp/security-input-1783181021-36367.txt
GOVERNANCE_APPROVED trace_id=464bb377-4c1e-489a-b388-7e5c469d1784 risk=low approval_status=auto_approved output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/tmp/governance-approved-1783181021-36367.txt
AGENTOPS_RECORDED trace_id=5092d8b7-62b1-470d-b474-5f7340c9701c status=success latency_ms=212 tokens=14 cost=0.00002800 output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/tmp/raw-output-1783181021-36367.txt
SECURITY_PASS trace_id=10af0fa9-12a3-4e61-82d4-a034fe8c3771 risk=low action=allow output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/docs/output/casan/evidence/07-wrapper-output.txt
CASAN_HARNESS_COMPLETE cache=cached key=a695c82f9d29aba2adace11fc766fd1b0ffae9ef27514df5907bdf66a69e0bba output=/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/docs/output/casan/evidence/07-wrapper-output.txt