gitea-admin/CASAN
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

scorecard: chấm live 2 mục hardcode; pipeline: fallback real + honest scoring doc

Browse Source 
Hướng A — scorecard.sh (video demo):
- h5_1 approval workflow: hardcode 0 → governance-check deploy live (approval_required)
- h6_2 hallucination rate: hardcode 0 → hallucination-scan phân biệt dirty>clean live
- "N/5 mục" chuyển từ text cứng sang đếm động
- H4/H5/H6 → 100/100 (5/5 gate live), Average 57.9 → 90.0

Hướng B — run-casan-pipeline.mjs:
- fallback: stub 'exit 9' → 'cat /nonexistent' (real failure, nhất quán adversarial T3)
- drift: giữ so fallback-output vs golden (clean run=1.0); năng lực phát hiện
  drift thật chứng minh ở adversarial suite
- Full 12-step run verify: H1 CONTEXT_VALID=24, H2 tool-audit records=25 signed,
  H5 audit-chain records=22 signed, H6 provider_telemetry per-step thật, H7 rollback real

phase3-real-run-scoring.md: giải thích vì sao scorecard cũ cho H5=60/H6=80
(hardcode), phân biệt scorecard-90 vs re-score-84 (2 mục đích khác nhau).

Verify: adversarial 44/0, security-gate 11/0/0, pipeline 12 steps OK.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
thanhnv
2026-07-03 00:18:43 +09:00
co-authored by Claude Sonnet 4.6
parent fda0d6447d
commit 2f06662f5d
26 changed files with 469 additions and 189 deletions
Download Patch File Download Diff File Expand all files Collapse all files
AINative_OKR_CASAN5/.specify/logs/cost/metrics.jsonl
+11 -11
View File
@@ -1,11 +1,11 @@
{"timestamp":"2026-07-01T08:22:20Z","trace_id":"b0093a64-db8d-47ed-a9b3-eb08f18b0c8a","harness":"H6-agentops","agent":"demo.agent","step":"demo-step","status":"success","exit_code":0,"latency_ms":72,"retry_count":0,"input_tokens":6,"output_tokens":6,"total_tokens":12,"cost_estimate":0.00002400,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac"}
{"timestamp":"2026-07-01T08:22:20Z","trace_id":"0fffefc3-0482-4965-8a44-9e7c0f1a19b7","harness":"H6-agentops","agent":"demo.agent","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":296,"retry_count":0,"input_tokens":13,"output_tokens":13,"total_tokens":26,"cost_estimate":0.00005200,"cost_source":"word_count_estimate","hallucination_signals":4,"alerts":["hallucination-suspected"],"input_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57","output_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57"}
{"timestamp":"2026-07-01T08:22:21Z","trace_id":"679c0d14-39ed-4303-9d9d-3d73b5c7906b","harness":"H6-agentops","agent":"demo.agent","step":"speckit.implement","status":"success","exit_code":0,"latency_ms":74,"retry_count":0,"input_tokens":6,"output_tokens":6,"total_tokens":2778,"cost_estimate":0.08334,"cost_source":"provider_telemetry","hallucination_signals":0,"alerts":[],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac"}
{"timestamp":"2026-07-01T08:22:22Z","trace_id":"1f6a097b-3dd3-4120-aded-f9dff2b82ecc","harness":"H6-agentops","agent":"demo.agent","step":"failing-step","status":"failed","exit_code":7,"latency_ms":358,"retry_count":0,"input_tokens":6,"output_tokens":0,"total_tokens":6,"cost_estimate":0.00001200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
{"timestamp":"2026-07-01T08:22:24Z","trace_id":"51d4f5d0-8640-462e-8440-78e9d901fe05","harness":"H6-agentops","agent":"wrapper.demo","step":"wrapper-step","status":"success","exit_code":0,"latency_ms":112,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116","output_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116"}
{"timestamp":"2026-07-01T08:22:34Z","trace_id":"bcbd5a59-edd5-45bc-93f2-5ef642629f7e","harness":"H6-agentops","agent":"wrapper.demo","step":"wrapper-step","status":"success","exit_code":0,"latency_ms":317,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116","output_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116"}
{"timestamp":"2026-07-01T08:22:58Z","trace_id":"295fb8ed-bbf8-4e10-b2ef-d44c640a4451","harness":"H6-agentops","agent":"unknown-agent","step":"write_code","status":"failed","exit_code":0,"latency_ms":304,"retry_count":0,"input_tokens":3,"output_tokens":0,"total_tokens":3,"cost_estimate":0.00000600,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"cae0b3e41bdd7bf9fc5ab7f73d4422a65c3766f8097c90d952d07dd371605290","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
{"timestamp":"2026-07-01T08:23:00Z","trace_id":"3a817ed6-f0bf-407e-8c78-c5a96523c0a5","harness":"H6-agentops","agent":"adv","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":239,"retry_count":0,"input_tokens":13,"output_tokens":13,"total_tokens":26,"cost_estimate":0.00005200,"cost_source":"word_count_estimate","hallucination_signals":4,"alerts":["hallucination-suspected"],"input_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57","output_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57"}
{"timestamp":"2026-07-01T08:23:01Z","trace_id":"dd89020a-7bea-4329-892c-310cf63ee23c","harness":"H6-agentops","agent":"adv","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":242,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"a97a0aba66ab15562e4d271fbcb2071c92d3662d99f2e2faac7f263ea35cfa0b","output_hash":"a97a0aba66ab15562e4d271fbcb2071c92d3662d99f2e2faac7f263ea35cfa0b"}
{"timestamp":"2026-07-01T08:23:16Z","trace_id":"1c4aa7e3-c8a0-40b1-b7a1-9432f4dd2f11","harness":"H6-agentops","agent":"unknown-agent","step":"test_timeout","status":"failed","exit_code":124,"latency_ms":2328,"retry_count":0,"input_tokens":1,"output_tokens":0,"total_tokens":1,"cost_estimate":0.00000200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"7d3f9b6284c6f36e77b425cac882e8fbbcc97a4727ec20790853076d0f463453","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
{"timestamp":"2026-07-01T08:23:35Z","trace_id":"0eb3483d-7da2-4459-8ad7-d9f2969cc5c3","harness":"H6-agentops","agent":"unknown-agent","step":"t4-telemetry-test","status":"success","exit_code":0,"latency_ms":1252,"retry_count":0,"input_tokens":2,"output_tokens":2,"total_tokens":210,"cost_estimate":0.0,"cost_source":"provider_telemetry","hallucination_signals":0,"alerts":[],"input_hash":"19f254d48ce072ae3832d0271898d40921875a1a870bf4389e67ef226ffc1cc6","output_hash":"19f254d48ce072ae3832d0271898d40921875a1a870bf4389e67ef226ffc1cc6"}
{"timestamp":"2026-07-02T15:15:41Z","trace_id":"1e7758aa-866c-4d65-9a03-f13c58f790c9","harness":"H6-agentops","agent":"demo.agent","step":"demo-step","status":"success","exit_code":0,"latency_ms":57,"retry_count":0,"input_tokens":6,"output_tokens":6,"total_tokens":12,"cost_estimate":0.00002400,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac"}
{"timestamp":"2026-07-02T15:15:41Z","trace_id":"697fa530-83e0-4516-afb1-721d5a886baf","harness":"H6-agentops","agent":"demo.agent","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":218,"retry_count":0,"input_tokens":13,"output_tokens":13,"total_tokens":26,"cost_estimate":0.00005200,"cost_source":"word_count_estimate","hallucination_signals":4,"alerts":["hallucination-suspected"],"input_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57","output_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57"}
{"timestamp":"2026-07-02T15:15:41Z","trace_id":"3df4f172-0e06-42bd-b036-07235e5c88b7","harness":"H6-agentops","agent":"demo.agent","step":"speckit.implement","status":"success","exit_code":0,"latency_ms":56,"retry_count":0,"input_tokens":6,"output_tokens":6,"total_tokens":2778,"cost_estimate":0.08334,"cost_source":"provider_telemetry","hallucination_signals":0,"alerts":[],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac"}
{"timestamp":"2026-07-02T15:15:42Z","trace_id":"36e863a6-980c-42c9-910b-f803b8c568c9","harness":"H6-agentops","agent":"demo.agent","step":"failing-step","status":"failed","exit_code":7,"latency_ms":207,"retry_count":0,"input_tokens":6,"output_tokens":0,"total_tokens":6,"cost_estimate":0.00001200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"2a5a257dc10475c9105ba981755390fb815f3102d3e6d4fc6f7fb161f7351bac","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
{"timestamp":"2026-07-02T15:15:43Z","trace_id":"08f0e50f-993e-431f-92b5-9e2ce169edcb","harness":"H6-agentops","agent":"wrapper.demo","step":"wrapper-step","status":"success","exit_code":0,"latency_ms":66,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116","output_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116"}
{"timestamp":"2026-07-02T15:15:45Z","trace_id":"e4f014d8-c28e-4f5f-92da-b33ee05658c1","harness":"H6-agentops","agent":"wrapper.demo","step":"wrapper-step","status":"success","exit_code":0,"latency_ms":214,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116","output_hash":"054cd5120725af9fdf9a8033f7b0f60d2d8ba3861926b00686d74700668e5116"}
{"timestamp":"2026-07-02T15:15:55Z","trace_id":"67e350c0-a744-4807-8217-b0ab5ced6601","harness":"H6-agentops","agent":"unknown-agent","step":"write_code","status":"failed","exit_code":0,"latency_ms":218,"retry_count":0,"input_tokens":3,"output_tokens":0,"total_tokens":3,"cost_estimate":0.00000600,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"cae0b3e41bdd7bf9fc5ab7f73d4422a65c3766f8097c90d952d07dd371605290","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
{"timestamp":"2026-07-02T15:15:56Z","trace_id":"dc241bcb-19c7-42f3-85a6-62ec99952582","harness":"H6-agentops","agent":"adv","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":207,"retry_count":0,"input_tokens":13,"output_tokens":13,"total_tokens":26,"cost_estimate":0.00005200,"cost_source":"word_count_estimate","hallucination_signals":4,"alerts":["hallucination-suspected"],"input_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57","output_hash":"666dfe86cafeb8150bcc28d0b5cb2c43e1149dddaa126b5fbc8adee318f46d57"}
{"timestamp":"2026-07-02T15:15:57Z","trace_id":"54bb8027-c093-44c9-902b-3a71a4008634","harness":"H6-agentops","agent":"adv","step":"step-1-srs","status":"success","exit_code":0,"latency_ms":221,"retry_count":0,"input_tokens":7,"output_tokens":7,"total_tokens":14,"cost_estimate":0.00002800,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":[],"input_hash":"a97a0aba66ab15562e4d271fbcb2071c92d3662d99f2e2faac7f263ea35cfa0b","output_hash":"a97a0aba66ab15562e4d271fbcb2071c92d3662d99f2e2faac7f263ea35cfa0b"}
{"timestamp":"2026-07-02T15:16:02Z","trace_id":"7ad48fea-02b3-431d-b239-8fb4d2708bff","harness":"H6-agentops","agent":"unknown-agent","step":"test_timeout","status":"failed","exit_code":124,"latency_ms":2228,"retry_count":0,"input_tokens":1,"output_tokens":0,"total_tokens":1,"cost_estimate":0.00000200,"cost_source":"word_count_estimate","hallucination_signals":0,"alerts":["execution-failed"],"input_hash":"7d3f9b6284c6f36e77b425cac882e8fbbcc97a4727ec20790853076d0f463453","output_hash":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"}
{"timestamp":"2026-07-02T15:16:12Z","trace_id":"2ef3b6d1-4134-4107-b1cd-ce655f9919e6","harness":"H6-agentops","agent":"unknown-agent","step":"t4-telemetry-test","status":"success","exit_code":0,"latency_ms":216,"retry_count":0,"input_tokens":2,"output_tokens":2,"total_tokens":210,"cost_estimate":0.0,"cost_source":"provider_telemetry","hallucination_signals":0,"alerts":[],"input_hash":"19f254d48ce072ae3832d0271898d40921875a1a870bf4389e67ef226ffc1cc6","output_hash":"19f254d48ce072ae3832d0271898d40921875a1a870bf4389e67ef226ffc1cc6"}