Choose a model by measuring complete tasks under the constraints your application will use. A lower refusal rate, a long context window, or a lower token price alone does not establish better task performance.
Define a successful task
Write the acceptance criterion before running the comparison. Examples include producing schema-valid records with correct labels, answering from supplied sources with supported citations, or completing a tool loop with valid arguments and the expected result.
Separate several outcomes:
| Outcome | What to record |
|---|---|
| Correct completion | Meets the task's predefined acceptance criterion. |
| Incorrect completion | Produces an answer or action that fails the criterion. |
| Unnecessary refusal | Declines an in-scope task that your evaluation expected it to answer. |
| Appropriate abstention | Correctly reports insufficient evidence or an out-of-scope request. |
| Protocol or tool failure | Malformed arguments, missing results, incompatible parameters, or interrupted streams. |
| Budget exhaustion | Reaches an output, time, tool-turn, or spending bound. |
Refusal labels require judgment. Do not count every uncertainty statement as a refusal, or every non-refusal as a success.
Build a representative test set
Start with a small set of real task shapes that you are permitted to use. Include ordinary cases, difficult cases, and negative controls where the right behavior is to decline an action or acknowledge missing evidence. Replace sensitive content with approved fixtures where possible.
For dataset workflows, keep evaluation examples separate from prompts or examples used to tune generation. Check for duplicates and near duplicates before splitting records. See dataset generation.
Use a stable task ID and record expected outcomes before collecting model outputs. Record the date and exact model ID; for models with undisclosed checkpoint details, mark the identity limitation rather than implying a reproducible weight revision.
Keep the comparison fair
Use the same task inputs, tool implementation, and comparable sampling settings. Record every setting rather than assuming parameter names behave identically across API formats. The compatibility matrix distinguishes effective controls from ignored fields.
Keep output budgets explicit and within each model's published limits. Reasoning can consume output budget. Compare an equal-budget result separately from any experiment that gives one model additional retries or tool turns.
For agent runs, pin the client version and use the same environment and target fixture. Existing security-agent checks are short compatibility observations; they do not establish general security effectiveness or production reliability.
Record attempts, usage, and latency
One task can require multiple billable requests. Retain a record for every attempt, including retries and failures:
{
"task_id": "schema-extraction-001",
"model": "red-cyber-1",
"api": "chat_completions",
"client_version": "record-installed-version",
"settings": {"max_tokens": 512, "temperature": 0},
"attempt": 1,
"http_status": null,
"request_id": null,
"finish_reason": null,
"elapsed_ms": null,
"time_to_first_token_ms": null,
"usage": null,
"actual_cost_usd": null,
"outcome": "not_run"
}
This is a record template, not a measured result. Replace unknown values only with observations. A missing usage count is unknown; do not turn it into zero tokens. Obtain final financial information from account Usage and billing, including its settlement behavior.
Measure time to first token separately from time to complete the answer. A quickly opened stream may still take a long time to produce useful output. For streamed requests, consume the terminal events before marking an attempt complete.
Report useful metrics
- Task success rate: successful tasks divided by all attempted tasks, with failures included.
- Cost per successful task: total actual spend across all attempts divided by successful tasks. Report undefined when no tasks succeed.
- Tool validity: valid, executable calls divided by returned tool calls. Assess argument correctness separately from JSON syntax.
- Unnecessary-refusal rate: unnecessary refusals divided by tasks expected to receive an answer.
- Latency: median and tail completion time, plus time to first token for streaming workloads.
- Recovery: success and additional spend after a bounded retry or repair attempt.
Publish the sample size and the count behind each percentage. Report results per workload; an average across unrelated tasks can hide important failures. Small samples are exploratory evidence, and repeated sampling can change conclusions.
Review and publish responsibly
Inspect incorrect outputs and disagreements manually. If using an automated judge, disclose the judge, rubric, and known limitations, and check it against human-reviewed cases. Avoid evaluating a model solely with a judge derived from that same model.
Share prompts, settings, environment, scoring rules, and representative failures when permission allows. Exclude credentials, private prompts, confidential outputs, and customer identifiers. State what was tested locally, what used the production API, and what remains unverified.
Use the results to choose a model and budget for that workload. Re-run a focused subset after changing models, prompts, tool schemas, clients, or limits; see production for rollout safeguards.