56 lines
3.0 KiB
Markdown
56 lines
3.0 KiB
Markdown
# CASAN PLAN 03 — Cloud Provider Patch (OpenAI/Anthropic)
|
|
|
|
> Status 2026-07-06: **MVP implemented + deterministic tests added**. The router
|
|
> can call OpenAI/Anthropic when real API keys are present, while local tests
|
|
> verify endpoint pinning, provider usage parsing, and fail-closed malformed
|
|
> payload handling without requiring paid keys.
|
|
|
|
## Delivered
|
|
|
|
| Capability | Where | Verification |
|
|
|---|---|---|
|
|
| Model spec prefixes `openai:<model>` and `anthropic:<model>` | `.specify/scripts/bash/model-call.py` | `phase3-model-router-tests.sh` |
|
|
| Endpoint allowlist/SSRF posture | Cloud hosts are hard-pinned to `api.openai.com` and `api.anthropic.com`; Ollama remains `127.0.0.1:11434` only | router SSRF test |
|
|
| Honest unavailable state | Missing `OPENAI_API_KEY` / `ANTHROPIC_API_KEY` exits non-zero; no fake PASS or fake token usage | router cloud-unavailable test |
|
|
| Real provider token usage path | OpenAI `usage.prompt_tokens/completion_tokens`; Anthropic `usage.input_tokens/output_tokens` | deterministic monkeypatched cloud parser test |
|
|
| Malformed provider payload fail-closed | Missing usage/schema exits non-zero via `provider_usage_invalid` | deterministic parser test |
|
|
| Provider telemetry source tags | Usage rows write `openai_api_real_tokens` or `anthropic_api_real_tokens` | `model-call.py` provider log path |
|
|
|
|
## How to run live when keys exist
|
|
|
|
```bash
|
|
cd AINative_OKR_CASAN5
|
|
export CASAN_MODEL_PRIMARY=openai:gpt-4o-mini
|
|
export OPENAI_API_KEY=...
|
|
bash .specify/scripts/bash/model-router.sh /tmp/prompt.txt /tmp/out.json --role classify
|
|
|
|
export CASAN_MODEL_PRIMARY=anthropic:claude-sonnet-4-5
|
|
export ANTHROPIC_API_KEY=...
|
|
bash .specify/scripts/bash/model-router.sh /tmp/prompt.txt /tmp/out.json --role judge
|
|
```
|
|
|
|
Expected evidence:
|
|
|
|
- command exits `0` for well-formed provider output;
|
|
- `/tmp/out.json` has non-zero `input_tokens` and `output_tokens`;
|
|
- `.specify/logs/level5/provider-usage.jsonl` gets a row with provider
|
|
`openai` or `anthropic` and cost source `*_api_real_tokens`;
|
|
- malformed or unreachable provider exits non-zero and does not fabricate a
|
|
successful verdict.
|
|
|
|
## Remaining production work
|
|
|
|
| Priority | Work | Done when |
|
|
|---|---|---|
|
|
| P1 | Run live smoke with real org keys on the target Mac/CI | Evidence file records real provider response, token counts, and no secret leakage |
|
|
| P2 | Add provider cost lookup table for selected models | `cost_usd` is computed from current provider pricing instead of `0.0` |
|
|
| P3 | Wire real provider billing usage API, not only per-call response usage | H6 reconcile uses OpenAI/Anthropic ground truth APIs with schema-versioned fetchers |
|
|
| P4 | Add model allowlist policy for approved cloud models | Unknown cloud model names require approval or block |
|
|
| P5 | Decide default failover order for local -> cloud or cloud -> local | `model-fallback.sh` policy is explicit per role |
|
|
|
|
## Notes for Plan-02
|
|
|
|
Plan-02 source generation should call `model-router.sh --role generate`, not
|
|
provider SDKs directly. This keeps H4 prompt handling, H6 telemetry, endpoint
|
|
allowlisting, and provider usage logging in one path.
|