Prompt caching

Reduce repeated-input cost when a model reports cache reads. RedRouter uses the selected model's cached-input price only for reported cache hits; cache availability and hit rate are not guaranteed.

Keep reusable context stable

Place reusable instructions, reference material, and stable tool definitions before changing user input. Preserve their order and serialization between requests. This gives prefix-based caching a useful opportunity where the model supports it, without changing your application's correctness if every request is a miss.

For example, a repeated analysis job can keep its instruction and rubric fixed while changing only the final observation:

import json
import os
from urllib.request import Request, urlopen

instruction = "Classify the observation as access_control, configuration, or other. Return one label."
observations = [
    "Every employee can edit billing settings.",
    "A test service exposes its debug interface to the office network.",
]
for observation in observations:
    body = {
        "model": "red-cyber-1",
        "max_tokens": 64,
        "messages": [
            {"role": "system", "content": instruction},
            {"role": "user", "content": observation},
        ],
    }
    request = Request("https://redrouter.org/v1/chat/completions",
                      data=json.dumps(body).encode(), headers={
        "Authorization": "Bearer " + os.environ["REDROUTER_API_KEY"],
        "Content-Type": "application/json",
    })
    with urlopen(request, timeout=120) as response:
        result = json.load(response)
    print(json.dumps({"usage": result.get("usage")}))

Set REDROUTER_API_KEY before running. This performs two ordinary billable generations. The short example demonstrates stable input layout; it does not promise a cache hit, and it makes no assumption about minimum prefix length or cache lifetime.

Read cache usage correctly

For Chat and Responses, cached tokens are already included in total input. For Messages, input_tokens contains the noncached portion when a cache-read count is reported. Do not add a cached subset twice.

Unknown usage is null in collected Chat and Responses JSON, while Messages uses an empty object. A Chat stream can omit its usage chunk entirely when actual counts are unavailable. A missing cached-token field means no reported cache measurement, rather than proof about the underlying cache. In Chat streaming, request stream_options: {"include_usage":true} and read through [DONE]; usage can arrive after the finish reason.

Understand pricing and credit holds

For reported counts, input cost is the noncached input at the normal input rate plus cache reads at the cached-input rate. Output uses the output rate. Rates are shown in Models & pricing, in USD per million tokens. Unreported cache reads use the ordinary input rate.

The initial credit reservation is conservative and does not assume a future cache hit. A request can need more available credit than its final cache-discounted charge. Keep sufficient headroom for concurrent credit holds; see Billing and Production.

Compatibility hints and limits

Text-part and Messages cache_control hints are accepted and ignored. Responses prompt_cache_key is accepted as a string and ignored. There is no explicit cache creation, invalidation, inspection, or lifetime API, and these hints do not pin a cache entry or change routing.

Automatic model caching is separate from RedRouter request-log retention, saved chats, and Zero Data Retention. A cache hit is a usage/pricing observation, not a retention-policy claim. Measure cache savings from reported usage across your own representative workload; do not assume a fixed hit rate in cost forecasts.