gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Browse Source 
Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):

- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
  dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
  Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
  usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
  provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
  a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
  successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).

New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.

Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.

Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-05 01:14:46 +09:00
co-authored by Claude Fable 5
parent 2af67ef6a5
commit da66a36f97
55 changed files with 1306 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/docs/output/output_logs/casan-demo/pipeline-context.yaml
+41 -41
View File
@@ -1,5 +1,5 @@
# CASAN4 demo pipeline context
generated-at: 2026-07-03T15:03:38Z
generated-at: 2026-07-04T16:03:44Z
feature-id: casan-demo
module-id: mod-casan
module-keyword: harness
@@ -22,14 +22,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-2d1e3749-25f6-4272-90fb-df2ce7337aaa.json
trace: .specify/logs/trace/security-10af0fa9-12a3-4e61-82d4-a034fe8c3771.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-02b57916-9f25-4bcb-b1c2-f4bf58adfc52.json
trace: .specify/logs/trace/agentops-236cb598-7a82-413d-8817-dde96e58cfa3.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-1-srs:
status: COMPLETE
@@ -38,14 +38,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-747fc560-2855-4c60-a57a-9b035766e01b.json
trace: .specify/logs/trace/security-14c11a55-f4c8-4b52-9c35-55a0d2fb9003.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
trace: .specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-082ad921-01f5-42ac-ac4c-52c232dce0e4.json
trace: .specify/logs/trace/agentops-432d6880-9240-461b-9a81-77cdcc30da75.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-2-bd:
status: COMPLETE
@@ -54,14 +54,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-7a9c98de-60d9-4900-baab-b2dd7d87bb8c.json
trace: .specify/logs/trace/security-5afc5ff5-3130-467e-b04f-7a34489da81f.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-236cb598-7a82-413d-8817-dde96e58cfa3.json
trace: .specify/logs/trace/agentops-4e14ae44-8f88-4e0f-89ab-3e52ad7d0987.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-3-spec:
status: COMPLETE
@@ -70,14 +70,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-a6bea7ae-fa7a-454b-8a14-f1d03402696f.json
trace: .specify/logs/trace/security-60909d76-5ae0-40c5-a433-f1f6a4d5ebc4.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
trace: .specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-4e14ae44-8f88-4e0f-89ab-3e52ad7d0987.json
trace: .specify/logs/trace/agentops-5092d8b7-62b1-470d-b474-5f7340c9701c.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-4-clarify:
status: COMPLETE
@@ -86,10 +86,10 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-acfebabd-a8f2-4a0e-b4d2-649e7c5e9140.json
trace: .specify/logs/trace/security-a3e4f81b-fcf6-43b4-879f-3a840eaaacb5.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
@@ -102,10 +102,10 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-c9801774-4925-42d2-a1d1-a8db6430f8cf.json
trace: .specify/logs/trace/security-f22da2ba-c99b-4335-8895-8e133b927a82.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
trace: .specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
@@ -118,10 +118,10 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-e1ec9ef9-0236-4778-986c-770ae163a72f.json
trace: .specify/logs/trace/security-f495655e-4018-4c54-8e39-d8d4010ea6f8.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
@@ -134,10 +134,10 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-2d1e3749-25f6-4272-90fb-df2ce7337aaa.json
trace: .specify/logs/trace/security-10af0fa9-12a3-4e61-82d4-a034fe8c3771.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
trace: .specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
@@ -150,10 +150,10 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-747fc560-2855-4c60-a57a-9b035766e01b.json
trace: .specify/logs/trace/security-14c11a55-f4c8-4b52-9c35-55a0d2fb9003.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
@@ -166,14 +166,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-7a9c98de-60d9-4900-baab-b2dd7d87bb8c.json
trace: .specify/logs/trace/security-5afc5ff5-3130-467e-b04f-7a34489da81f.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
trace: .specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-95b088e2-f7ce-403d-bcca-1cf06186360d.json
trace: .specify/logs/trace/agentops-da53987e-b91b-49de-a42a-6af44c685638.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-9-tasks:
status: COMPLETE
@@ -182,14 +182,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-a6bea7ae-fa7a-454b-8a14-f1d03402696f.json
trace: .specify/logs/trace/security-60909d76-5ae0-40c5-a433-f1f6a4d5ebc4.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-a607903d-5d84-40fc-80ea-139c96c133ae.json
trace: .specify/logs/trace/agentops-ddae00a1-7960-4a99-8741-de089b93e283.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-10-implement:
status: COMPLETE
@@ -198,14 +198,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-acfebabd-a8f2-4a0e-b4d2-649e7c5e9140.json
trace: .specify/logs/trace/security-a3e4f81b-fcf6-43b4-879f-3a840eaaacb5.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4e1cd7d6-4540-4284-a1fb-89db2acf26ec.json
trace: .specify/logs/trace/governance-a6d0e737-0260-4ad7-8f2a-81da8fd5f44e.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-cb926abb-0d31-45d0-b03d-1d2afdf7c622.json
trace: .specify/logs/trace/agentops-e75e1165-3a92-4b54-9473-eff24e8a8b60.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-11-review-code:
status: COMPLETE
@@ -214,14 +214,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-c9801774-4925-42d2-a1d1-a8db6430f8cf.json
trace: .specify/logs/trace/security-f22da2ba-c99b-4335-8895-8e133b927a82.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-5a8297f3-caec-4f5a-affa-3521a8c34d7d.json
trace: .specify/logs/trace/governance-f193b886-e340-4c29-a0f1-9c17d850198a.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-ddae00a1-7960-4a99-8741-de089b93e283.json
trace: .specify/logs/trace/agentops-f25ea973-62bb-41ad-8db2-f5c0f6df239c.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-12-testkit:
status: COMPLETE
@@ -230,14 +230,14 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-e1ec9ef9-0236-4778-986c-770ae163a72f.json
trace: .specify/logs/trace/security-f495655e-4018-4c54-8e39-d8d4010ea6f8.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4d42037d-403d-42a7-a67f-e22ef114c890.json
trace: .specify/logs/trace/governance-464bb377-4c1e-489a-b388-7e5c469d1784.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-e75e1165-3a92-4b54-9473-eff24e8a8b60.json
trace: .specify/logs/trace/agentops-f4cea8da-c6e6-4aa9-887f-4fcfd5a8770f.json
metrics-log: .specify/logs/cost/metrics.jsonl
step-13-launch:
status: COMPLETE
@@ -246,12 +246,12 @@ steps:
casan:
h4-security:
status: PASS
trace: .specify/logs/trace/security-2d1e3749-25f6-4272-90fb-df2ce7337aaa.json
trace: .specify/logs/trace/security-10af0fa9-12a3-4e61-82d4-a034fe8c3771.json
h5-governance:
decision: approved
trace: .specify/logs/trace/governance-4e1cd7d6-4540-4284-a1fb-89db2acf26ec.json
trace: .specify/logs/trace/governance-a6d0e737-0260-4ad7-8f2a-81da8fd5f44e.json
audit-log: .specify/logs/audit/audit.jsonl
h6-agentops:
status: success
trace: .specify/logs/trace/agentops-f25ea973-62bb-41ad-8db2-f5c0f6df239c.json
trace: .specify/logs/trace/agentops-fa952bbb-64a1-408a-bdb2-aa1f74c90866.json
metrics-log: .specify/logs/cost/metrics.jsonl