gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Browse Source 
Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):

- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
  dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
  Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
  usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
  provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
  a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
  successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).

New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.

Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.

Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-05 01:14:46 +09:00
co-authored by Claude Fable 5
parent 2af67ef6a5
commit da66a36f97
55 changed files with 1306 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md
+21 -21
View File
@@ -1,6 +1,6 @@
# CASAN4 Harness Test Report
Generated: 2026-07-03T15:03:26Z
Generated: 2026-07-04T16:03:32Z
PASS: H4 blocks prompt injection
PASS: /Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/docs/output/casan/evidence/02-pii-output.txt contains ***MASKED_EMAIL***
@@ -82,38 +82,38 @@ PASS: /Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/docs/output/casan/evidence/harness-test-report.md
## Trace Files
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-02b57916-9f25-4bcb-b1c2-f4bf58adfc52.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-082ad921-01f5-42ac-ac4c-52c232dce0e4.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-236cb598-7a82-413d-8817-dde96e58cfa3.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-4e0fdde7-238d-4c20-9edc-fc4bae3c3213.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-432d6880-9240-461b-9a81-77cdcc30da75.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-4e14ae44-8f88-4e0f-89ab-3e52ad7d0987.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-5092d8b7-62b1-470d-b474-5f7340c9701c.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-69c773cb-4c43-4d8f-b933-65e692a59509.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-70219708-1c20-45a5-964e-107a3bcbb4ea.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-790ad863-fef7-418e-9716-ea0ab1c2e5c1.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-812b0adb-8253-4a79-ac4c-0b181f7ded41.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-92cab24c-4988-4996-bc2f-74ae9b1684cf.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-95b088e2-f7ce-403d-bcca-1cf06186360d.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-a607903d-5d84-40fc-80ea-139c96c133ae.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-cb926abb-0d31-45d0-b03d-1d2afdf7c622.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-da53987e-b91b-49de-a42a-6af44c685638.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-ddae00a1-7960-4a99-8741-de089b93e283.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-e75e1165-3a92-4b54-9473-eff24e8a8b60.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-f25ea973-62bb-41ad-8db2-f5c0f6df239c.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-f4cea8da-c6e6-4aa9-887f-4fcfd5a8770f.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-fa952bbb-64a1-408a-bdb2-aa1f74c90866.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-fc199d1f-efc5-407f-917d-b96f0f042975.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-fc76dac7-c2aa-4d24-b2fc-3a9405172db2.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/agentops-fd2a8ee9-d78d-4f41-8fe8-2a6ed988141e.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-4e1cd7d6-4540-4284-a1fb-89db2acf26ec.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-fb004aee-625d-418b-a2a8-7724179cb3d9.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-2d1e3749-25f6-4272-90fb-df2ce7337aaa.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-4a50cd89-a135-4b94-973f-8a4371581340.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-509dba29-c0f7-475c-99ad-d9e2c8f6be5f.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-747fc560-2855-4c60-a57a-9b035766e01b.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-7a9c98de-60d9-4900-baab-b2dd7d87bb8c.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-a6bea7ae-fa7a-454b-8a14-f1d03402696f.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-acfebabd-a8f2-4a0e-b4d2-649e7c5e9140.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-c9801774-4925-42d2-a1d1-a8db6430f8cf.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-e1ec9ef9-0236-4778-986c-770ae163a72f.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-ec348425-cfdd-49c9-a640-d7654918431c.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-a6d0e737-0260-4ad7-8f2a-81da8fd5f44e.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-db13a5e6-573d-4fd3-8b17-33c6f0ec7d33.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-0082cfb9-854e-4806-b115-fe2d34946dab.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-10af0fa9-12a3-4e61-82d4-a034fe8c3771.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-14c11a55-f4c8-4b52-9c35-55a0d2fb9003.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-2589e8af-e541-409f-973b-aa509722641d.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-4069096f-d7f2-41b9-9d3d-b1d3c2bb6f10.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-5afc5ff5-3130-467e-b04f-7a34489da81f.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-60909d76-5ae0-40c5-a433-f1f6a4d5ebc4.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-a3e4f81b-fcf6-43b4-879f-3a840eaaacb5.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-f22da2ba-c99b-4335-8895-8e133b927a82.json
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/trace/security-f495655e-4018-4c54-8e39-d8d4010ea6f8.json
## Audit Files
/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/audit/audit-head.sig