scorecard: chấm live 2 mục hardcode; pipeline: fallback real + honest scoring doc
Hướng A — scorecard.sh (video demo): - h5_1 approval workflow: hardcode 0 → governance-check deploy live (approval_required) - h6_2 hallucination rate: hardcode 0 → hallucination-scan phân biệt dirty>clean live - "N/5 mục" chuyển từ text cứng sang đếm động - H4/H5/H6 → 100/100 (5/5 gate live), Average 57.9 → 90.0 Hướng B — run-casan-pipeline.mjs: - fallback: stub 'exit 9' → 'cat /nonexistent' (real failure, nhất quán adversarial T3) - drift: giữ so fallback-output vs golden (clean run=1.0); năng lực phát hiện drift thật chứng minh ở adversarial suite - Full 12-step run verify: H1 CONTEXT_VALID=24, H2 tool-audit records=25 signed, H5 audit-chain records=22 signed, H6 provider_telemetry per-step thật, H7 rollback real phase3-real-run-scoring.md: giải thích vì sao scorecard cũ cho H5=60/H6=80 (hardcode), phân biệt scorecard-90 vs re-score-84 (2 mục đích khác nhau). Verify: adversarial 44/0, security-gate 11/0/0, pipeline 12 steps OK. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Sonnet 4.6
parent
fda0d6447d
commit
2f06662f5d
@@ -7,83 +7,83 @@ steps:
|
||||
agent: okr.srs
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/01-srs-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: f25ea973-62bb-41ad-8db2-f5c0f6df239c
|
||||
trace_file: .specify/logs/trace/agentops-f25ea973-62bb-41ad-8db2-f5c0f6df239c.json
|
||||
trace_id: bfc3c58d-608e-4402-bbb6-fe4bc59240a5
|
||||
trace_file: .specify/logs/trace/agentops-bfc3c58d-608e-4402-bbb6-fe4bc59240a5.json
|
||||
status: success
|
||||
- id: 02-bd
|
||||
agent: okr.bd
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/02-bd-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: 4e14ae44-8f88-4e0f-89ab-3e52ad7d0987
|
||||
trace_file: .specify/logs/trace/agentops-4e14ae44-8f88-4e0f-89ab-3e52ad7d0987.json
|
||||
trace_id: e027bfef-b0ff-4742-87e6-1732ec2afd62
|
||||
trace_file: .specify/logs/trace/agentops-e027bfef-b0ff-4742-87e6-1732ec2afd62.json
|
||||
status: success
|
||||
- id: 03-spec
|
||||
agent: speckit.specify
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/03-spec-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: ddae00a1-7960-4a99-8741-de089b93e283
|
||||
trace_file: .specify/logs/trace/agentops-ddae00a1-7960-4a99-8741-de089b93e283.json
|
||||
trace_id: c87816d4-df32-4b35-8086-25a535cd281b
|
||||
trace_file: .specify/logs/trace/agentops-c87816d4-df32-4b35-8086-25a535cd281b.json
|
||||
status: success
|
||||
- id: 04-reviewspec
|
||||
agent: okr.reviewspec
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/04-reviewspec-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: 812b0adb-8253-4a79-ac4c-0b181f7ded41
|
||||
trace_file: .specify/logs/trace/agentops-812b0adb-8253-4a79-ac4c-0b181f7ded41.json
|
||||
verdict: REJECTED
|
||||
trace_id: 1de117a4-f96e-4129-b2d3-02a2788fabb2
|
||||
trace_file: .specify/logs/trace/agentops-1de117a4-f96e-4129-b2d3-02a2788fabb2.json
|
||||
status: success
|
||||
- id: 05-plan-attempt-1
|
||||
agent: speckit.plan
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/05-plan-attempt-1-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: 236cb598-7a82-413d-8817-dde96e58cfa3
|
||||
trace_file: .specify/logs/trace/agentops-236cb598-7a82-413d-8817-dde96e58cfa3.json
|
||||
trace_id: 432aa34d-f4eb-4296-bb46-94b89ba8cc78
|
||||
trace_file: .specify/logs/trace/agentops-432aa34d-f4eb-4296-bb46-94b89ba8cc78.json
|
||||
status: success
|
||||
- id: 06-reviewplan-attempt-1
|
||||
agent: okr.reviewplan
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/06-reviewplan-attempt-1-output.md
|
||||
verdict: REJECTED
|
||||
trace_id: 69c773cb-4c43-4d8f-b933-65e692a59509
|
||||
trace_file: .specify/logs/trace/agentops-69c773cb-4c43-4d8f-b933-65e692a59509.json
|
||||
trace_id: b928df2d-0260-49ce-81eb-1ce3f1be2b26
|
||||
trace_file: .specify/logs/trace/agentops-b928df2d-0260-49ce-81eb-1ce3f1be2b26.json
|
||||
status: success
|
||||
- id: 07-plan-attempt-2
|
||||
agent: speckit.plan
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/07-plan-attempt-2-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: fd2a8ee9-d78d-4f41-8fe8-2a6ed988141e
|
||||
trace_file: .specify/logs/trace/agentops-fd2a8ee9-d78d-4f41-8fe8-2a6ed988141e.json
|
||||
trace_id: 0e0c2637-d5b4-447a-a45e-e7be2c75c1bc
|
||||
trace_file: .specify/logs/trace/agentops-0e0c2637-d5b4-447a-a45e-e7be2c75c1bc.json
|
||||
status: success
|
||||
- id: 08-reviewplan-attempt-2
|
||||
agent: okr.reviewplan
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/08-reviewplan-attempt-2-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: e75e1165-3a92-4b54-9473-eff24e8a8b60
|
||||
trace_file: .specify/logs/trace/agentops-e75e1165-3a92-4b54-9473-eff24e8a8b60.json
|
||||
verdict: REJECTED
|
||||
trace_id: d0718391-7a6a-4f63-9a2a-2522984ecf06
|
||||
trace_file: .specify/logs/trace/agentops-d0718391-7a6a-4f63-9a2a-2522984ecf06.json
|
||||
status: success
|
||||
- id: 09-dd
|
||||
agent: okr.dd
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/09-dd-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: 92cab24c-4988-4996-bc2f-74ae9b1684cf
|
||||
trace_file: .specify/logs/trace/agentops-92cab24c-4988-4996-bc2f-74ae9b1684cf.json
|
||||
trace_id: ecf095a8-75ca-4940-8c69-d152679e0d88
|
||||
trace_file: .specify/logs/trace/agentops-ecf095a8-75ca-4940-8c69-d152679e0d88.json
|
||||
status: success
|
||||
- id: 10-testkit
|
||||
agent: okr.testkit
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/10-testkit-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: 70219708-1c20-45a5-964e-107a3bcbb4ea
|
||||
trace_file: .specify/logs/trace/agentops-70219708-1c20-45a5-964e-107a3bcbb4ea.json
|
||||
trace_id: e2f397b8-e1ce-4256-84c4-948f0486663a
|
||||
trace_file: .specify/logs/trace/agentops-e2f397b8-e1ce-4256-84c4-948f0486663a.json
|
||||
status: success
|
||||
- id: 11-tasks
|
||||
agent: speckit.tasks
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/11-tasks-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: fc199d1f-efc5-407f-917d-b96f0f042975
|
||||
trace_file: .specify/logs/trace/agentops-fc199d1f-efc5-407f-917d-b96f0f042975.json
|
||||
trace_id: 134de506-7f2a-4435-86db-50eb32a4febd
|
||||
trace_file: .specify/logs/trace/agentops-134de506-7f2a-4435-86db-50eb32a4febd.json
|
||||
status: success
|
||||
- id: 12-reviewcode
|
||||
agent: okr.reviewcode
|
||||
artifact: docs/output/output_logs/001-okr-web-app/casan/12-reviewcode-output.md
|
||||
verdict: APPROVED
|
||||
trace_id: 790ad863-fef7-418e-9716-ea0ab1c2e5c1
|
||||
trace_file: .specify/logs/trace/agentops-790ad863-fef7-418e-9716-ea0ab1c2e5c1.json
|
||||
verdict: REJECTED
|
||||
trace_id: 1b1be6e9-0894-487a-ba0e-c369d243da58
|
||||
trace_file: .specify/logs/trace/agentops-1b1be6e9-0894-487a-ba0e-c369d243da58.json
|
||||
status: success
|
||||
|
||||
Reference in New Issue
Block a user