scorecard: chấm live 2 mục hardcode; pipeline: fallback real + honest scoring doc

Hướng A — scorecard.sh (video demo):
- h5_1 approval workflow: hardcode 0 → governance-check deploy live (approval_required)
- h6_2 hallucination rate: hardcode 0 → hallucination-scan phân biệt dirty>clean live
- "N/5 mục" chuyển từ text cứng sang đếm động
- H4/H5/H6 → 100/100 (5/5 gate live), Average 57.9 → 90.0

Hướng B — run-casan-pipeline.mjs:
- fallback: stub 'exit 9' → 'cat /nonexistent' (real failure, nhất quán adversarial T3)
- drift: giữ so fallback-output vs golden (clean run=1.0); năng lực phát hiện
  drift thật chứng minh ở adversarial suite
- Full 12-step run verify: H1 CONTEXT_VALID=24, H2 tool-audit records=25 signed,
  H5 audit-chain records=22 signed, H6 provider_telemetry per-step thật, H7 rollback real

phase3-real-run-scoring.md: giải thích vì sao scorecard cũ cho H5=60/H6=80
(hardcode), phân biệt scorecard-90 vs re-score-84 (2 mục đích khác nhau).

Verify: adversarial 44/0, security-gate 11/0/0, pipeline 12 steps OK.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-03 00:18:43 +09:00
co-authored by Claude Sonnet 4.6
parent fda0d6447d
commit 2f06662f5d
26 changed files with 469 additions and 189 deletions
@@ -1,19 +1,19 @@
{"timestamp": "2026-07-01T08:22:20Z", "trace_id": "0fffefc3-0482-4965-8a44-9e7c0f1a19b7", "agent": "demo.agent", "step": "step-1-srs", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "", "record_hash": "85fab7e6edaf0fe23f0faab6063d6cc22b622c5705dbb98c3f32ee3648056be9"}
{"timestamp": "2026-07-01T08:22:22Z", "trace_id": "1f6a097b-3dd3-4120-aded-f9dff2b82ecc", "agent": "demo.agent", "step": "failing-step", "tool": "Bash", "command": "bash -c exit 7", "exit_code": 7, "status": "failed", "previous_record_hash": "85fab7e6edaf0fe23f0faab6063d6cc22b622c5705dbb98c3f32ee3648056be9", "record_hash": "e7b09366cb38a46589315483856d93207cf1817693950490a3ecbd618bb3b0ec"}
{"timestamp": "2026-07-01T08:22:34Z", "trace_id": "bcbd5a59-edd5-45bc-93f2-5ef642629f7e", "agent": "wrapper.demo", "step": "wrapper-step", "tool": "Bash", "command": "bash -c cp \"$1\" \"$CASAN_OUTPUT\" _ /Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/idempotency/a695c82f9d29aba2adace11fc766fd1b0ffae9ef27514df5907bdf66a69e0bba.output", "exit_code": 0, "status": "success", "previous_record_hash": "e7b09366cb38a46589315483856d93207cf1817693950490a3ecbd618bb3b0ec", "record_hash": "d1092ac89510705b129b587beda324d055585fabd51d6b9e63c50effcf522079"}
{"timestamp": "2026-07-01T08:22:36Z", "trace_id": "28f618f2-2187-4f95-9c41-a5fa3cdc8bfa", "tool": "deploy", "agent": "release-manager", "idempotency_key": "", "decision": "denied", "reason": "missing_idempotency_key", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "d1092ac89510705b129b587beda324d055585fabd51d6b9e63c50effcf522079", "record_hash": "13e2e6663f041520380054b62b208dcd572044c3fccc816e88fcaf2f8a300cba"}
{"timestamp": "2026-07-01T08:22:36Z", "trace_id": "db4c5806-1365-43e9-a7c6-7c20bec08cd5", "tool": "deploy", "agent": "release-manager", "idempotency_key": "deploy-demo-001", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "13e2e6663f041520380054b62b208dcd572044c3fccc816e88fcaf2f8a300cba", "record_hash": "a2df07e76cbf1d577e4076ddab1d25648759f1d465eb094297b5794039dfed2e"}
{"timestamp": "2026-07-01T08:22:36Z", "trace_id": "05556471-81ba-45fd-ae51-636ece8bda45", "tool": "deploy", "agent": "design-agent", "idempotency_key": "deploy-demo-002", "decision": "denied", "reason": "unauthorized_agent", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "a2df07e76cbf1d577e4076ddab1d25648759f1d465eb094297b5794039dfed2e", "record_hash": "98dc1cb0e79aa1fd99ae2cfe20aeac3536a2b1763fcf1566e51651595446286c"}
{"timestamp": "2026-07-01T08:22:46Z", "trace_id": "8b024ab7-faf4-4030-b32c-02a67b4ed2da", "tool": "deploy", "agent": "design-agent", "idempotency_key": "k1", "decision": "denied", "reason": "unauthorized_agent", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "98dc1cb0e79aa1fd99ae2cfe20aeac3536a2b1763fcf1566e51651595446286c", "record_hash": "3fd71c020749009962f08e7f70695767dc8542a31d8b4bacf6208e35defeeadd"}
{"timestamp": "2026-07-01T08:22:46Z", "trace_id": "df1127cb-401a-47f0-91c7-0ae102d761c8", "tool": "deploy", "agent": "", "idempotency_key": "k2", "decision": "denied", "reason": "missing_agent_identity", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "3fd71c020749009962f08e7f70695767dc8542a31d8b4bacf6208e35defeeadd", "record_hash": "d3edcd8bfa552c0213d61fce5945aab73eff5c51ddd1151b17cda7836e285211"}
{"timestamp": "2026-07-01T08:22:46Z", "trace_id": "a09af213-5ab5-497e-b35c-2ca0ba1ee71d", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k3", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "d3edcd8bfa552c0213d61fce5945aab73eff5c51ddd1151b17cda7836e285211", "record_hash": "664cfd77d8726752625aa8a20870f20068fb94afdc65ea29fdea659cc7084dca"}
{"timestamp": "2026-07-01T08:22:57Z", "trace_id": "0f10a69a-043f-4965-9419-6fc26f623a46", "tool": "write_code", "agent": "design-agent", "idempotency_key": "d6d750d0ee93e4b9f9b917a42e3f7f6e8725c63bf412594778810bd91c175084", "decision": "denied", "reason": "unauthorized_agent", "risk_level": "high", "owner": "engineering", "previous_record_hash": "664cfd77d8726752625aa8a20870f20068fb94afdc65ea29fdea659cc7084dca", "record_hash": "6e7394ddef5fa8775512d4aaf9024a844a9a7d0b78225dd3c1ac821ff0255b13"}
{"timestamp": "2026-07-01T08:22:58Z", "trace_id": "76bb296c-ac6f-43c2-ae09-d17377b1361e", "tool": "write_code", "agent": "implement-agent", "idempotency_key": "d6d750d0ee93e4b9f9b917a42e3f7f6e8725c63bf412594778810bd91c175084", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "engineering", "previous_record_hash": "6e7394ddef5fa8775512d4aaf9024a844a9a7d0b78225dd3c1ac821ff0255b13", "record_hash": "8242e270791e3e3613758f1249fdba7478e6ce52bd62bc7a74c104998a8235ea"}
{"timestamp": "2026-07-01T08:22:58Z", "trace_id": "295fb8ed-bbf8-4e10-b2ef-d44c640a4451", "agent": "unknown-agent", "step": "write_code", "tool": "Bash", "command": "/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/scripts/bash/tool-exec.sh 30 -- bash -c echo wrote", "exit_code": 0, "status": "success", "previous_record_hash": "8242e270791e3e3613758f1249fdba7478e6ce52bd62bc7a74c104998a8235ea", "record_hash": "fdc94a9e91ad13247c354a2bed7459b6a19bb43b93b0459bb817ae71987d902e"}
{"timestamp": "2026-07-01T08:23:00Z", "trace_id": "3a817ed6-f0bf-407e-8c78-c5a96523c0a5", "agent": "adv", "step": "step-1-srs", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "fdc94a9e91ad13247c354a2bed7459b6a19bb43b93b0459bb817ae71987d902e", "record_hash": "bfdbca1b092c309d8139c0fcc769c1aa2e9f4733694e68dbc62e5b191341896b"}
{"timestamp": "2026-07-01T08:23:01Z", "trace_id": "dd89020a-7bea-4329-892c-310cf63ee23c", "agent": "adv", "step": "step-1-srs", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "bfdbca1b092c309d8139c0fcc769c1aa2e9f4733694e68dbc62e5b191341896b", "record_hash": "ca904df210574c5bfefe85a81ec5cae64f305f13aab946e0e702fbed05d83191"}
{"timestamp": "2026-07-01T08:23:02Z", "trace_id": "0f6f5be7-04d4-43c2-968f-22a9832f984f", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k1", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "ca904df210574c5bfefe85a81ec5cae64f305f13aab946e0e702fbed05d83191", "record_hash": "59632091ef7f5c774d92fc0ec33b49ffb244f9f7d34fde0ea3dd6b8142acbb1d"}
{"timestamp": "2026-07-01T08:23:02Z", "trace_id": "4ed85702-36f5-453a-bf0a-3b041da5cd62", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k2", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "59632091ef7f5c774d92fc0ec33b49ffb244f9f7d34fde0ea3dd6b8142acbb1d", "record_hash": "b3ca9da84c15d73280e5d0b048dc5ff99efd83814704fa3a73f0a3577af78238"}
{"timestamp": "2026-07-01T08:23:03Z", "trace_id": "b8baf09c-30d2-41df-92a5-a9da8d517bcc", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k3", "decision": "denied", "reason": "rate_limit_exceeded(limit=2)", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "b3ca9da84c15d73280e5d0b048dc5ff99efd83814704fa3a73f0a3577af78238", "record_hash": "2d6c400adb741b39374f4faae36f5979f4b3c13d342662533e600f04713c2a79"}
{"timestamp": "2026-07-01T08:23:16Z", "trace_id": "1c4aa7e3-c8a0-40b1-b7a1-9432f4dd2f11", "agent": "unknown-agent", "step": "test_timeout", "tool": "Bash", "command": "/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/scripts/bash/tool-exec.sh 2 -- sleep 60", "exit_code": 124, "status": "failed", "previous_record_hash": "2d6c400adb741b39374f4faae36f5979f4b3c13d342662533e600f04713c2a79", "record_hash": "5982e75c063a4e58067ab51a5ebd9750757e1174cf2152412f9267c6225a4f02"}
{"timestamp": "2026-07-01T08:23:35Z", "trace_id": "0eb3483d-7da2-4459-8ad7-d9f2969cc5c3", "agent": "unknown-agent", "step": "t4-telemetry-test", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "5982e75c063a4e58067ab51a5ebd9750757e1174cf2152412f9267c6225a4f02", "record_hash": "53ffb8f5fc653345a8e0cbafad355e218d093ff02ac2de68d98ba37bf3c1e386"}
{"timestamp": "2026-07-02T15:15:41Z", "trace_id": "697fa530-83e0-4516-afb1-721d5a886baf", "agent": "demo.agent", "step": "step-1-srs", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "", "record_hash": "01e18919ae54362e6dbd6140e0e5473724aeb9ab3e1cf75ef52c8f8ec308b355"}
{"timestamp": "2026-07-02T15:15:42Z", "trace_id": "36e863a6-980c-42c9-910b-f803b8c568c9", "agent": "demo.agent", "step": "failing-step", "tool": "Bash", "command": "bash -c exit 7", "exit_code": 7, "status": "failed", "previous_record_hash": "01e18919ae54362e6dbd6140e0e5473724aeb9ab3e1cf75ef52c8f8ec308b355", "record_hash": "dd505818f823f0476c403bdb8192fe5ea74b54ce9f979346f37c91ebf3f52b31"}
{"timestamp": "2026-07-02T15:15:45Z", "trace_id": "e4f014d8-c28e-4f5f-92da-b33ee05658c1", "agent": "wrapper.demo", "step": "wrapper-step", "tool": "Bash", "command": "bash -c cp \"$1\" \"$CASAN_OUTPUT\" _ /Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/logs/idempotency/a695c82f9d29aba2adace11fc766fd1b0ffae9ef27514df5907bdf66a69e0bba.output", "exit_code": 0, "status": "success", "previous_record_hash": "dd505818f823f0476c403bdb8192fe5ea74b54ce9f979346f37c91ebf3f52b31", "record_hash": "4beeb8f536a24a54743e0becb905d20d6ae159d1fddc82b77efc5d28c7685bff"}
{"timestamp": "2026-07-02T15:15:46Z", "trace_id": "3438ed09-602f-453f-9a99-6b2f3664b646", "tool": "deploy", "agent": "release-manager", "idempotency_key": "", "decision": "denied", "reason": "missing_idempotency_key", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "4beeb8f536a24a54743e0becb905d20d6ae159d1fddc82b77efc5d28c7685bff", "record_hash": "a854a786af60fbf8ca30fc1c8ad654016f22ad80e29e69103830b52cd70fa75d"}
{"timestamp": "2026-07-02T15:15:46Z", "trace_id": "8377db5f-e282-42e5-a045-5e842edb1e8f", "tool": "deploy", "agent": "release-manager", "idempotency_key": "deploy-demo-001", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "a854a786af60fbf8ca30fc1c8ad654016f22ad80e29e69103830b52cd70fa75d", "record_hash": "ff9a6647141ca72369999cc4645ec7d5aa598fb1ba048900c5e1318c50cd2cc2"}
{"timestamp": "2026-07-02T15:15:46Z", "trace_id": "d23c175c-e864-45ab-b2cc-1bbf86075e24", "tool": "deploy", "agent": "design-agent", "idempotency_key": "deploy-demo-002", "decision": "denied", "reason": "unauthorized_agent", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "ff9a6647141ca72369999cc4645ec7d5aa598fb1ba048900c5e1318c50cd2cc2", "record_hash": "f1c455dcca9b5d81671084608e423df399f1c01c5e52169a10f67c0b83ad4697"}
{"timestamp": "2026-07-02T15:15:53Z", "trace_id": "c63052ab-0cce-490a-a814-d18fbd49df5e", "tool": "deploy", "agent": "design-agent", "idempotency_key": "k1", "decision": "denied", "reason": "unauthorized_agent", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "f1c455dcca9b5d81671084608e423df399f1c01c5e52169a10f67c0b83ad4697", "record_hash": "dbc86df00d38b2ca11efbd0920875721ecc77ff340a973ad75231ff89b1722ed"}
{"timestamp": "2026-07-02T15:15:53Z", "trace_id": "14fa1288-2d35-49c7-9614-0d2bcb202487", "tool": "deploy", "agent": "", "idempotency_key": "k2", "decision": "denied", "reason": "missing_agent_identity", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "dbc86df00d38b2ca11efbd0920875721ecc77ff340a973ad75231ff89b1722ed", "record_hash": "8134dcf1a83c1a336575e0294a519bb3cc76ed37595ea6927887d222b858cc68"}
{"timestamp": "2026-07-02T15:15:53Z", "trace_id": "c670ece0-312c-432c-b35d-69e17f46c820", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k3", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "8134dcf1a83c1a336575e0294a519bb3cc76ed37595ea6927887d222b858cc68", "record_hash": "65fbda4e5bd502eb57263c0216358fba3d9aaa35bd057f6389376ddf5fc66239"}
{"timestamp": "2026-07-02T15:15:54Z", "trace_id": "cd43f413-dbbf-4519-8eba-7a26dd9a037e", "tool": "write_code", "agent": "design-agent", "idempotency_key": "d6d750d0ee93e4b9f9b917a42e3f7f6e8725c63bf412594778810bd91c175084", "decision": "denied", "reason": "unauthorized_agent", "risk_level": "high", "owner": "engineering", "previous_record_hash": "65fbda4e5bd502eb57263c0216358fba3d9aaa35bd057f6389376ddf5fc66239", "record_hash": "4d949a434a72d038d3c781970d2333d02a93f72c2562622403a0e9d30c8ffa78"}
{"timestamp": "2026-07-02T15:15:55Z", "trace_id": "829faee4-8065-4c52-aa7b-e45fec893085", "tool": "write_code", "agent": "implement-agent", "idempotency_key": "d6d750d0ee93e4b9f9b917a42e3f7f6e8725c63bf412594778810bd91c175084", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "engineering", "previous_record_hash": "4d949a434a72d038d3c781970d2333d02a93f72c2562622403a0e9d30c8ffa78", "record_hash": "b28fa1009eb9b6c2526d57b9bd65f3418db114cae96244fc134e722c954727e7"}
{"timestamp": "2026-07-02T15:15:55Z", "trace_id": "67e350c0-a744-4807-8217-b0ab5ced6601", "agent": "unknown-agent", "step": "write_code", "tool": "Bash", "command": "/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/scripts/bash/tool-exec.sh 30 -- bash -c echo wrote", "exit_code": 0, "status": "success", "previous_record_hash": "b28fa1009eb9b6c2526d57b9bd65f3418db114cae96244fc134e722c954727e7", "record_hash": "f1bbab70d29724cc36702a21e3b416993383041de78c4d529811c2ab07e8cc06"}
{"timestamp": "2026-07-02T15:15:56Z", "trace_id": "dc241bcb-19c7-42f3-85a6-62ec99952582", "agent": "adv", "step": "step-1-srs", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "f1bbab70d29724cc36702a21e3b416993383041de78c4d529811c2ab07e8cc06", "record_hash": "8cef20149d2d410e2fd0b923b2ed48a3b27be97b99382c2aa204946a85e707e8"}
{"timestamp": "2026-07-02T15:15:57Z", "trace_id": "54bb8027-c093-44c9-902b-3a71a4008634", "agent": "adv", "step": "step-1-srs", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "8cef20149d2d410e2fd0b923b2ed48a3b27be97b99382c2aa204946a85e707e8", "record_hash": "5ac1d8c7340aa62e77dc2a81c67a353db8f0e1039dcf3e3697c5ceb656485e32"}
{"timestamp": "2026-07-02T15:15:58Z", "trace_id": "1933731e-9fb0-474c-a80f-e1e35d909510", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k1", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "5ac1d8c7340aa62e77dc2a81c67a353db8f0e1039dcf3e3697c5ceb656485e32", "record_hash": "655460149a3ace331826d1813610f94f9d3855b84e23d5aeaf71e083554150e6"}
{"timestamp": "2026-07-02T15:15:58Z", "trace_id": "4dab312c-eb11-4a23-acfc-026b7a71ba53", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k2", "decision": "approved", "reason": "registered", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "655460149a3ace331826d1813610f94f9d3855b84e23d5aeaf71e083554150e6", "record_hash": "c61caf9bc8b82e4826fc7b5dae33fc4539ca821131074d9cd9354c6b0b4d3f7a"}
{"timestamp": "2026-07-02T15:15:58Z", "trace_id": "04ab4e1e-0b69-47a8-9389-ce8078035743", "tool": "deploy", "agent": "release-manager", "idempotency_key": "k3", "decision": "denied", "reason": "rate_limit_exceeded(limit=2)", "risk_level": "high", "owner": "release-manager", "previous_record_hash": "c61caf9bc8b82e4826fc7b5dae33fc4539ca821131074d9cd9354c6b0b4d3f7a", "record_hash": "3001530ef878f0ba816dad63a5bed9b29c47be3fec81aac68dc3ba9418bdff1a"}
{"timestamp": "2026-07-02T15:16:02Z", "trace_id": "7ad48fea-02b3-431d-b239-8fb4d2708bff", "agent": "unknown-agent", "step": "test_timeout", "tool": "Bash", "command": "/Users/thanhnguyen/Documents/AI/HarnessHkt/Harness_Hakathon/Output_CASAN5_REFINED/AINative_OKR_CASAN5/.specify/scripts/bash/tool-exec.sh 2 -- sleep 60", "exit_code": 124, "status": "failed", "previous_record_hash": "3001530ef878f0ba816dad63a5bed9b29c47be3fec81aac68dc3ba9418bdff1a", "record_hash": "377f787e2530311b8596306da8cae1364b32e0d9244bd29958cff17923706db6"}
{"timestamp": "2026-07-02T15:16:12Z", "trace_id": "2ef3b6d1-4134-4107-b1cd-ce655f9919e6", "agent": "unknown-agent", "step": "t4-telemetry-test", "tool": "Bash", "command": "bash -c cp \"$CASAN_INPUT\" \"$CASAN_OUTPUT\"", "exit_code": 0, "status": "success", "previous_record_hash": "377f787e2530311b8596306da8cae1364b32e0d9244bd29958cff17923706db6", "record_hash": "8dd9a1bcb706c87055d6b073aca3cd052aaafa690252777becd73ce10a87e4a0"}