Build production clients around explicit model selection, bounded work, and a request outcome that can be complete, failed, or uncertain. Generation and billing do not always finish at the same moment.
Configure a production client
Keep the API key in a server-side secret store. Use a separate key for each environment with a suitable model allowlist, spending limit, expiry, and retention setting. Use the exact catalog ID from Models; a client's default model name is not automatically mapped to RedRouter.
Send a realistic output limit on every request. It bounds generation and affects the credit hold. Leave room for output when sizing context. The token-counting endpoint estimates input; its capped estimate does not prove that a prompt fits a model's context.
Reuse connections. Set a connection timeout and an application deadline appropriate to the task. Non-streaming requests have a 110-second total server deadline. Use streaming for long work, and configure proxies to pass SSE without buffering or terminating healthy long-lived streams. Read terminal events and usage even after a finish reason arrives.
Bound concurrency and queues
Coordinate dispatch across all workers sharing an account. Separate keys share account limits; adding a key does not add capacity. Start below the effective concurrency shown in Settings, then adjust using observed latency, refusals, and the rate-limit headers.
Use a bounded queue with a deadline for queued work. Acquire a client-side permit before dispatch and release it after the response body is consumed or the attempt ends. Account for uncertain disconnected attempts before increasing parallelism: the server may still be generating and holding credit.
For steady traffic, a useful starting estimate is concurrency ≈ requests per second × average request duration in seconds. For example, 0.5 requests/second with an eight-second average occupies about four slots. This is sizing arithmetic, not a throughput promise; bursts and long generations need headroom.
When refusals rise, slow dispatch and let in-flight work finish. An immediate retry can compete with the original generation and worsen both concurrency and credit pressure.
Decide when to retry
| Observed outcome | Retry decision | Billing consideration |
|---|---|---|
| Invalid fields, unsupported capability, unknown model | Fix the request first. | Admission refusal; no generation charge. |
| Authentication, key permission, trial lock, credit limit | Repair access or funding first. | Retrying the same request cannot repair the condition. |
| 429 request-rate or concurrency refusal | Wait for Retry-After; reduce dispatch. | Refused generation is uncharged; existing requests may still hold credit. |
Daily spend ceiling or supplier_budget_exhausted | Wait for the indicated reset; do not run a short retry loop. | Existing holds and settled spend affect admission. |
| 503 capacity, model busy, unavailable, or usage brake | Honor Retry-After and use a bounded retry. | These refusals are uncharged. |
Server request_timeout on a non-streaming generation | Switch to streaming or reduce the task, then retry if still needed. | The failed buffered request delivers no output and is uncharged. |
| Connection reset, local timeout, unknown 5xx, early EOF | Treat the outcome as uncertain; inspect Usage before retrying. | Generation may have started or continued after disconnection. |
| Streamed partial answer followed by an error | Discard it as a complete result; inspect Usage and retry only if the task still needs it. | Partial delivery with reported usage can be charged. |
| Complete answer with missing usage | Keep the answer; do not regenerate solely to obtain token counts. | Missing counts are not estimated as actual usage. |
Messages uses a different error envelope and maps some statuses; see Errors. A model_error after output starts is different from a model refusal before generation. A generic network or SDK exception does not establish that a request was uncharged.
Assign one layer responsibility for retries. If your SDK retries automatically and your job runner also retries, the total attempts can multiply. Choose a small attempt cap and a total elapsed-time budget. For retry number n, a practical delay is a random value between zero and min(cap, base × 2^n); never retry earlier than a supplied Retry-After. Jitter spreads workers across time.
Every retry is a new generation. There is no documented generation idempotency contract: an Idempotency-Key or a repeated client identifier must not be treated as duplicate suppression. Give application jobs stable IDs, keep an attempt ledger, and make downstream side effects idempotent in your own system.
Handle disconnects and cancellation
Closing a socket stops delivery to your client. RedRouter may continue reading the model response, within its deadlines and output cap, to collect actual usage. Do not use a disconnected socket as proof that generation stopped or that its charge will be zero.
Keep three pieces of state separately: whether your user still needs the answer, whether you received a valid terminal response, and whether the request's financial outcome is final. A user navigating away can change the first without resolving the other two.
For tools, buffer each complete tool call, validate its arguments, and apply your own permissions before execution. Never execute argument fragments. Use an application-level execution ID to prevent duplicate external effects if the generation or job is retried. RedRouter returns tool calls; your client owns execution.
Reconcile client and account records
Record the response's x-request-id, your job and attempt IDs, endpoint, model, status, error code when supplied, timestamps, time to first output, total duration, terminal event, and reported usage. Keep keys and prompt content out of operational logs unless your own policy explicitly calls for them.
Use Usage to review the model, source key, token counts, cost, and request state. Unknown token counts are unknown, not zero. A zero final charge is different from an unfinished hold. Use Billing to interpret pending or failed requests, and retain request IDs when escalating discrepancies.
Alert on sustained rate-limit errors, growing queues, incomplete streams, credit exhaustion, and unusually high cost per completed job. Measure completed application tasks as well as HTTP success: a 200 stream can fail later, and valid text can still fail your application's schema or quality checks.
Check security controls
- Keep API keys in a server-side secret store, out of browser code, repositories, and diagnostic logs.
- Use separate environment keys with narrow model allowlists and credit ceilings. Revoke exposed keys and rotate replacements through API Keys.
- Configure content storage or account Zero Data Retention before sensitive traffic, and review retention in your own application and tools.
- Treat model output and tool arguments as untrusted input. Validate complete calls, scope filesystem and network access, and require approval for effects your application cannot safely automate.
- Prevent duplicate tool effects with application execution IDs, especially across generation and job retries.
- Log request identifiers and outcomes without credentials or unnecessary content; restrict access to your own transcripts and payment references.
Verify before scaling
Exercise a successful request, a stream, a tool round trip if used, a validation error, a simulated midstream disconnect, and your retry limits. Use recorded or local fixtures for failure paths so the test does not create uncontrolled paid traffic. Confirm that your application handles null usage, unexpected EOF, and a non-JSON error from a network intermediary.
Review your key restrictions and retention controls, confirm credits and effective limits, and increase traffic gradually. Use the troubleshooting guide for symptoms and the migration guide when adapting an existing client.