gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(h6): AgentOps hardening — live alerting, provider-API reconcile, hosted dashboard, window breaker (79→80)

Browse Source 
Close the three gaps the scoring report itself flagged for H6 plus V15,
each as a real MVP + fail-able adversarial test (same pattern that lifted H5):

- D1 alert-dispatch.sh: alerts POST to a real HTTP webhook (severity routing,
  dedup window, retry) + dead-letter queue with redelivery; fail-loud in strict.
  Wired into agent-metrics.sh so a failing step pages live end-to-end.
- D2 provider-usage-fetch.sh + telemetry-reconcile.sh: pull usage from a provider
  usage HTTP API (all-or-nothing schema gate, fail-loud) + reconcile local vs
  provider ground truth — token under-reporting/hidden runs => TELEMETRY_DISCREPANCY.
- D3 dashboard-serve.sh + dashboard-server.py: serve the dashboard over HTTP with
  a stale-aware /healthz probe (fresh=200 ok, telemetry silent-death=503 stale).
- D4 circuit-breaker-check.sh: sliding-window failure-rate breaker (V15) — interleaved
  successes no longer evade the consecutive-failure breaker (CIRCUIT_OPEN_WINDOW).

New suite phase-h6-agentops-tests.sh: 20/20, all live against local HTTP endpoints
(webhook sink, mock provider API, dashboard server) — deterministic, no model needed.

Also fix sign-policy-bundle.sh key-sync invariant: the local-fallback branch only
exported policy-public.pem when generating a NEW key, so a Vault-DOWN run after a
Vault-signed run verified a local-key signature against the Vault pubkey (RSA padding
error, run-casan4 died mid-suite). Now always re-exports the pubkey before signing —
same fix class as tool-audit-lib.sh / governance-check.sh.

Full battery re-run sequentially: 175/175 PASS, 0 FAIL across 8 suites
(KMS SKIP this run — Vault down; validated live 2026-07-04). Docs synced:
scoring-run-report (H6 79→80, no harness below 80, 155→175), CASAN_HARDENING_STATUS
(Phase 5 D1–D4), Plan-07, submission README, and run-hardening.sh (H6+ scenes HO1–HO4).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-05 01:14:46 +09:00
co-authored by Claude Fable 5
parent 2af67ef6a5
commit da66a36f97
55 changed files with 1306 additions and 455 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/.specify/logs/audit/audit.jsonl
+19 -10
View File
@@ -1,10 +1,19 @@
{"timestamp":"2026-07-03T15:03:29Z","trace_id":"fb004aee-625d-418b-a2a8-7724179cb3d9","harness":"H5-governance","action":"deploy","actor":"developer","risk_level":"high","decision":"denied","approval_status":"approval_required","approver":"","input_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","output_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","previous_record_hash":"","record_hash":"5853af1901b2bb9e3a5a4e58a756ffea9ef7362db91bc359976070a28663d43b"}
{"timestamp":"2026-07-03T15:03:29Z","trace_id":"4e1cd7d6-4540-4284-a1fb-89db2acf26ec","harness":"H5-governance","action":"deploy","actor":"developer","risk_level":"high","decision":"approved","approval_status":"human_approved","approver":"architect@example.local","input_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","output_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","previous_record_hash":"5853af1901b2bb9e3a5a4e58a756ffea9ef7362db91bc359976070a28663d43b","record_hash":"ddd6257b7109b854a32fbb2c16a29343d499c936f940554adf0ae0da7c31e491"}
{"timestamp":"2026-07-03T15:03:33Z","trace_id":"5a8297f3-caec-4f5a-affa-3521a8c34d7d","harness":"H5-governance","action":"agent_step","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","output_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","previous_record_hash":"ddd6257b7109b854a32fbb2c16a29343d499c936f940554adf0ae0da7c31e491","record_hash":"8cab8a5bdea3bf95a5c37d1f3ae0e6d8998e6a50f92bd1b2f4cd13989dea4cd5"}
{"timestamp":"2026-07-03T15:03:36Z","trace_id":"4d42037d-403d-42a7-a67f-e22ef114c890","harness":"H5-governance","action":"agent_step","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","output_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","previous_record_hash":"8cab8a5bdea3bf95a5c37d1f3ae0e6d8998e6a50f92bd1b2f4cd13989dea4cd5","record_hash":"55e04abc144ed482ecce99cb3c48870560e05daa1e68df8ae276660c6280ce0c"}
{"timestamp":"2026-07-03T15:03:47Z","trace_id":"0c8cc253-0394-4a72-abb1-1779238e2850","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"separation_of_duties_violation","approver":"alice","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"55e04abc144ed482ecce99cb3c48870560e05daa1e68df8ae276660c6280ce0c","record_hash":"e9fefb4519b2b5467725917a0e266dc2dd11e86b23e643806b2ab61e6caa1eb9"}
{"timestamp":"2026-07-03T15:03:47Z","trace_id":"bfbe30ee-5316-46e4-a39b-18ba34ba2659","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"approved","approval_status":"human_approved","approver":"bob","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"e9fefb4519b2b5467725917a0e266dc2dd11e86b23e643806b2ab61e6caa1eb9","record_hash":"f04be7de3e6f17a516c8e1071f84113b020e5f63544480e3044c06200f686005"}
{"timestamp":"2026-07-03T15:03:49Z","trace_id":"fa7b6a3a-7d4b-4cb1-b4d7-26c599f1410a","harness":"H5-governance","action":"write_code","actor":"developer","risk_level":"medium","decision":"approved","approval_status":"policy_auto_approved_with_audit","approver":"","input_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","output_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","previous_record_hash":"f04be7de3e6f17a516c8e1071f84113b020e5f63544480e3044c06200f686005","record_hash":"18e46529fc164a18393655e11d674aa84f60807e47565ac369fa9e88aecd6321"}
{"timestamp":"2026-07-03T15:03:50Z","trace_id":"b1526d88-6d67-41c5-a5fd-870da5ec92fd","harness":"H5-governance","action":"write_code","actor":"developer","risk_level":"medium","decision":"approved","approval_status":"policy_auto_approved_with_audit","approver":"","input_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","output_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","previous_record_hash":"18e46529fc164a18393655e11d674aa84f60807e47565ac369fa9e88aecd6321","record_hash":"31f0d401cff6b1e90143ffa4592586787894ccf4ab1b654f59435c8047ede9e6"}
{"timestamp":"2026-07-03T15:03:59Z","trace_id":"effd270b-2442-4f7a-85bd-983551e724b8","harness":"H5-governance","action":"test_timeout","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"c96c6d5be8d08a12e7b5cdc1b207fa6b2430974c86803d8891675e76fd992c20","output_hash":"c96c6d5be8d08a12e7b5cdc1b207fa6b2430974c86803d8891675e76fd992c20","previous_record_hash":"31f0d401cff6b1e90143ffa4592586787894ccf4ab1b654f59435c8047ede9e6","record_hash":"82a8daceaef2a75e7c0dd1973a6a6d5c292b7dde144499803272cea62e372b0f"}
{"timestamp":"2026-07-03T15:06:02Z","trace_id":"33114869-297a-440b-ac36-2e631be1413d","harness":"H5-governance","action":"deploy","actor":"developer","risk_level":"high","decision":"denied","approval_status":"approval_required","approver":"","input_hash":"c3a4b726eb38096f511200d9be903d32beec1278a54f2855b9437fce73e7e216","output_hash":"c3a4b726eb38096f511200d9be903d32beec1278a54f2855b9437fce73e7e216","previous_record_hash":"82a8daceaef2a75e7c0dd1973a6a6d5c292b7dde144499803272cea62e372b0f","record_hash":"bf927018b89cafc87b52e2b7add4c7f78bd630cb2ba8db628f361d28c87fd2e4"}
{"timestamp":"2026-07-04T16:03:36Z","trace_id":"db13a5e6-573d-4fd3-8b17-33c6f0ec7d33","harness":"H5-governance","action":"deploy","actor":"developer","risk_level":"high","decision":"denied","approval_status":"approval_required","approver":"","input_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","output_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","previous_record_hash":"","record_hash":"c04a55e782acdc22af5c90a39dee8ec42350f02cd814e4fcfb43badd8048d8ef"}
{"timestamp":"2026-07-04T16:03:36Z","trace_id":"a6d0e737-0260-4ad7-8f2a-81da8fd5f44e","harness":"H5-governance","action":"deploy","actor":"developer","risk_level":"high","decision":"approved","approval_status":"human_approved","approver":"architect@example.local","input_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","output_hash":"3c1cf354bd8e23e98738dd97e726a7c751cf9c7574454d52b9698bfbb9000bff","previous_record_hash":"c04a55e782acdc22af5c90a39dee8ec42350f02cd814e4fcfb43badd8048d8ef","record_hash":"8289d81d54b7e79b734675c609a84bb6f2d800ee383535407c0fa1e48b685218"}
{"timestamp":"2026-07-04T16:03:39Z","trace_id":"f193b886-e340-4c29-a0f1-9c17d850198a","harness":"H5-governance","action":"agent_step","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","output_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","previous_record_hash":"8289d81d54b7e79b734675c609a84bb6f2d800ee383535407c0fa1e48b685218","record_hash":"073ed7580be2e1f1fc29ad88dfdb882a3f971e486586263218f894b81051e7c3"}
{"timestamp":"2026-07-04T16:03:42Z","trace_id":"464bb377-4c1e-489a-b388-7e5c469d1784","harness":"H5-governance","action":"agent_step","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","output_hash":"0649852ef55d103b9b1134daf5324e92a3e2254386dc137787ca9432b2e4b9bf","previous_record_hash":"073ed7580be2e1f1fc29ad88dfdb882a3f971e486586263218f894b81051e7c3","record_hash":"750ed15c130e210e11e3d0542cc09c8042c295c8f50f6ddea883fe5d95ec8dda"}
{"timestamp":"2026-07-04T16:04:09Z","trace_id":"1566abb0-f268-4e25-a38c-5023db8505bf","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"separation_of_duties_violation","approver":"alice","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"750ed15c130e210e11e3d0542cc09c8042c295c8f50f6ddea883fe5d95ec8dda","record_hash":"610751df56d8a4506331d84164b66fac88c15f10e3db2fcf5c3c1ec2d621160d"}
{"timestamp":"2026-07-04T16:04:09Z","trace_id":"56168960-dee1-4b3d-8494-8d6aab231305","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"approved","approval_status":"human_approved","approver":"bob","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"610751df56d8a4506331d84164b66fac88c15f10e3db2fcf5c3c1ec2d621160d","record_hash":"1d840afebd74802c16d3c22179880757460504803806f09468ca114fa3fe23e2"}
{"timestamp":"2026-07-04T16:04:11Z","trace_id":"ee279e21-cc40-40d4-8a04-4180a31972a9","harness":"H5-governance","action":"write_code","actor":"developer","risk_level":"medium","decision":"approved","approval_status":"policy_auto_approved_with_audit","approver":"","input_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","output_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","previous_record_hash":"1d840afebd74802c16d3c22179880757460504803806f09468ca114fa3fe23e2","record_hash":"08f07f8b92edb2757893597515bb4c2f4c6526906ea889de1939a55fe4a5f5d9"}
{"timestamp":"2026-07-04T16:04:12Z","trace_id":"a23bc14d-ea22-48fe-9d87-2324b6faae8a","harness":"H5-governance","action":"write_code","actor":"developer","risk_level":"medium","decision":"approved","approval_status":"policy_auto_approved_with_audit","approver":"","input_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","output_hash":"d62056ac97257e952127e4ee599f12b633725abe0c5a7c8fbaa82d6c29561190","previous_record_hash":"08f07f8b92edb2757893597515bb4c2f4c6526906ea889de1939a55fe4a5f5d9","record_hash":"dbb786fca90f61dec47308e9ed467bd222102232cd8c7cdf1f7e9462fa7b9e9e"}
{"timestamp":"2026-07-04T16:04:22Z","trace_id":"42b9a4cb-bcc5-4323-9ce8-e9f84ddbddac","harness":"H5-governance","action":"test_timeout","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"c96c6d5be8d08a12e7b5cdc1b207fa6b2430974c86803d8891675e76fd992c20","output_hash":"c96c6d5be8d08a12e7b5cdc1b207fa6b2430974c86803d8891675e76fd992c20","previous_record_hash":"dbb786fca90f61dec47308e9ed467bd222102232cd8c7cdf1f7e9462fa7b9e9e","record_hash":"bdd01e83c2ea44e0ed8dd1033532694bc013a003482aa67f15a363686787fd90"}
{"timestamp":"2026-07-04T16:05:00Z","trace_id":"ab7dc071-7d45-46cc-bce1-97d187655539","harness":"H5-governance","action":"fetch_step","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"427aa6ae245aa0f05cc75c0aea61de8129aae1e0e589d5727873d4d7f9baf872","output_hash":"427aa6ae245aa0f05cc75c0aea61de8129aae1e0e589d5727873d4d7f9baf872","previous_record_hash":"bdd01e83c2ea44e0ed8dd1033532694bc013a003482aa67f15a363686787fd90","record_hash":"d0b808796bb0da3690e0f91a56af82738b11e61b0c18a8b39dd9cec4a34de68c"}
{"timestamp":"2026-07-04T16:05:02Z","trace_id":"f54f2254-877a-4638-80a5-4323f10a348c","harness":"H5-governance","action":"fetch_step","actor":"developer","risk_level":"low","decision":"approved","approval_status":"auto_approved","approver":"","input_hash":"427aa6ae245aa0f05cc75c0aea61de8129aae1e0e589d5727873d4d7f9baf872","output_hash":"427aa6ae245aa0f05cc75c0aea61de8129aae1e0e589d5727873d4d7f9baf872","previous_record_hash":"d0b808796bb0da3690e0f91a56af82738b11e61b0c18a8b39dd9cec4a34de68c","record_hash":"4a26bcccdbe6797de50f4d1d33387646666fa6d994e813835bb7872e53b56abf"}
{"timestamp":"2026-07-04T16:11:54Z","trace_id":"42efbb68-73c6-4931-8926-08283c9b1dbd","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"approved","approval_status":"human_approved_signed","approver":"ops-owner","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"4a26bcccdbe6797de50f4d1d33387646666fa6d994e813835bb7872e53b56abf","record_hash":"a90a790e8543d5e0db9ae9d97d8f4fe62eabc1ca9a3eb8c02d2dcc56fc6cae91"}
{"timestamp":"2026-07-04T16:11:54Z","trace_id":"3197d2d8-2d97-42d8-b9ad-9f742987b981","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"approval_required_signed","approver":"ops-owner","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"a90a790e8543d5e0db9ae9d97d8f4fe62eabc1ca9a3eb8c02d2dcc56fc6cae91","record_hash":"dc6c2beb7b960224d78fe0a4c44f750bc3ab0b4536b4b46de3f74d1a19b2db65"}
{"timestamp":"2026-07-04T16:11:54Z","trace_id":"95406ed2-ac23-4e0d-9796-f4af669646e9","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"approval_signature_invalid","approver":"tech-lead","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"dc6c2beb7b960224d78fe0a4c44f750bc3ab0b4536b4b46de3f74d1a19b2db65","record_hash":"360e4ecc50134e16b0d2247edc59442ab3880aa02678565f4b0b26de60513c4b"}
{"timestamp":"2026-07-04T16:11:54Z","trace_id":"8c8a282e-e99b-4106-bb1e-084bfa9ca551","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"approval_signature_invalid","approver":"ops-owner","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"360e4ecc50134e16b0d2247edc59442ab3880aa02678565f4b0b26de60513c4b","record_hash":"113869ff982c0ea98f017266b2039ec3e519ace34979e4124524d70446e73b61"}
{"timestamp":"2026-07-04T16:11:55Z","trace_id":"6d7dc349-00f7-4827-b23a-57773dfc43c9","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"approval_signature_invalid","approver":"ghost","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"113869ff982c0ea98f017266b2039ec3e519ace34979e4124524d70446e73b61","record_hash":"810a8610168b0cf68397107738dce6ae400169bf19dd3e602c6b62c4cd3e5d38"}
{"timestamp":"2026-07-04T16:11:55Z","trace_id":"0c189fad-c117-4663-b41a-a91f2f98f8f8","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"denied","approval_status":"approval_signature_invalid","approver":"ops-owner","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"810a8610168b0cf68397107738dce6ae400169bf19dd3e602c6b62c4cd3e5d38","record_hash":"f135b495b9f024eb7e2eb0657ef0d1711a18c9998ef7084566cc6210e6ee2ed7"}
{"timestamp":"2026-07-04T16:11:55Z","trace_id":"1161c0fb-925f-43df-8a7e-b09f439f105d","harness":"H5-governance","action":"deploy","actor":"ops-owner","risk_level":"high","decision":"denied","approval_status":"separation_of_duties_violation","approver":"ops-owner","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"f135b495b9f024eb7e2eb0657ef0d1711a18c9998ef7084566cc6210e6ee2ed7","record_hash":"e31833b834ce37567cdb9bea2a151af533b2ab0f2a1eae5e26716d104452705d"}
{"timestamp":"2026-07-04T16:11:55Z","trace_id":"46efe000-8e14-4109-9948-9984e4ee1bfa","harness":"H5-governance","action":"deploy","actor":"alice","risk_level":"high","decision":"approved","approval_status":"human_approved","approver":"bob","input_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","output_hash":"ec9c2145d75be1bf0080711b5bd106465a8307147205ea249bfa1faa0bf05bbb","previous_record_hash":"e31833b834ce37567cdb9bea2a151af533b2ab0f2a1eae5e26716d104452705d","record_hash":"b3f71b62c1defd3df9f90e9aa8e62f2e8a28e6b14daebc91531ecc6656c3bcda"}