thanhnvandClaude Opus 4.8
f24ed21324
Closes the last fully-[planned] Track-C dimension (was scored 1). - incident.sh raise <event>: classify severity via incident-severity.map (LOW/MED/HIGH/CRIT), record a structured entry (owner routing), and for HIGH/CRIT auto-engage the scoped kill-switch + dispatch an alert (reuses H6 alert-dispatch.sh). Exit 2 on HIGH/CRIT so a pipeline gate goes red. - kill-switch.sh engage/clear/check/status, scoped by project/model/provider (+ global). `check` exits 2 when engaged so gates honor it. - casan-harness.sh honors an engaged kill-switch before running (opt-in CASAN_KILLSWITCH_ENFORCE=1, default OFF → baseline unchanged). - incident-runbook.md: severity→owner→response + postmortem template + prod TODO. - phase-c7-incident-tests.sh: 15 checks — severity grading, auto kill-switch on HIGH/CRIT, MED-only records, lifecycle, global scope, structured record, and the production wrapper refusing to run under an engaged switch. Baselines: run-casan4 35/35, adversarial 44/44. New suite total: 175 → 190. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
40 lines
1.8 KiB
Markdown
40 lines
1.8 KiB
Markdown
# CASAN Incident Runbook (C7 / V23)
|
|
|
|
When a gate raises an incident (`incident.sh raise <event>`), it is classified,
|
|
recorded to `logs/level5/incidents.jsonl`, and for HIGH/CRIT the scoped
|
|
kill-switch is engaged automatically + an alert is dispatched.
|
|
|
|
## Severity → owner → response
|
|
|
|
| Severity | Owner (on-call) | Auto-action | Human step |
|
|
|---|---|---|---|
|
|
| **CRIT** | security-oncall | kill-switch engaged + alert | Contain now; verify blast radius; do NOT clear until root cause known |
|
|
| **HIGH** | ops-oncall | kill-switch engaged + alert | Assess; clear switch only after fix + reviewer sign-off |
|
|
| **MED** | tech-lead | recorded + alert | Triage within SLA; batch-fix |
|
|
| **LOW** | triage | recorded | Review in retro |
|
|
|
|
## Kill-switch operations
|
|
```bash
|
|
kill-switch.sh status # what is engaged
|
|
kill-switch.sh check <scope> <id> # gates honor this (exit 2 = stop)
|
|
kill-switch.sh clear <scope> <id> <reason># turn off (production: reviewer-approved)
|
|
```
|
|
Scopes: `project` · `model` · `provider` · `global` (global stops everything).
|
|
|
|
## Event → severity
|
|
See `incident-severity.map`. Examples: `secret-to-cloud`=CRIT, `tool-write-sensitive`=CRIT,
|
|
`dependency-postinstall`=HIGH, `audit-chain-broken`=HIGH, `cost-budget-exceeded`=MED.
|
|
|
|
## Postmortem template (fill after resolution)
|
|
- **Incident**: <id / timestamp / event / severity>
|
|
- **Detection**: which gate fired, what signal
|
|
- **Blast radius**: scope, what was stopped by the kill-switch
|
|
- **Root cause**:
|
|
- **Fix**:
|
|
- **Prevent recurrence**: new test/gate added (link the fail-able check)
|
|
- **Kill-switch cleared by**: <reviewer> at <time>, reason
|
|
|
|
## Production TODO
|
|
Managed alert channel (Slack/PagerDuty) + on-call rota + auto issue creation;
|
|
kill-switch clear gated by reviewer approval (tie to approval-identity C4).
|