Measured: time-to-first-token and throughput for GPT-OSS 120B, DeepSeek V4 Flash, GLM 5.3 Flash and Llama 3.3 70B
We timed every chat model on the public endpoint with a fixed prompt: median time to first token, output tokens per second and total time, plus speech and transcription speed. Numbers, method and caveats.
Taylor Hawkes · September 16, 2026
Provider pricing pages tell you the cost per token; almost none tell you how long you will wait. We ran the same prompt against every chat model we serve, three times each, and recorded the median. The method is at the bottom; here are the numbers as measured on 2026-09-16.
| Model | Time to first token | Output speed | Total (≈120-word answer) |
|---|---|---|---|
openai/gpt-oss-120b | 0.93 s | 132.9 tok/s | 2.53 s |
deepseek-ai/DeepSeek-V4-Flash | 1.42 s | 55 tok/s | 8.69 s |
deepseek-ai/DeepSeek-V4.1-Flash | 0.41 s | 388.7 tok/s | 1.49 s |
zai-org/GLM-5.3-Flash | 0.38 s | 265.6 tok/s | 1.85 s |
meta-llama/Llama-3.3-70B-Instruct-Turbo | 1.29 s | 67.5 tok/s | 2.92 s |
What stands out
- GLM 5.3 Flash and DeepSeek V4.1 Flash are the fast ones — under half a second to first token and 265–390 tokens per second. For interactive UIs where perceived latency matters, they are the default picks.
- GPT-OSS 120B sits in the middle (≈0.9 s, ≈130 tok/s) and is the strongest of the set for tool use and reasoning; a fair trade for agent workloads.
- DeepSeek V4 Flash is the cheapest per token here and the slowest to start (≈1.4 s TTFT, 55 tok/s). For batch summarisation over its 1M-token context that is irrelevant; for a chat box it is noticeable.
- Llama 3.3 70B is served for compatibility; it is neither the fastest nor the cheapest, and the model page says so.
Speech and transcription
| Model | Task | Measured |
|---|---|---|
hexgrad/Kokoro-82M | Text to speech | 300 characters → 0.84 s |
canopylabs/orpheus-3b-0.1-ft | Text to speech | 300 characters → 58.5 s |
openai/whisper-large-v3 | Speech to text | 19.5 s of audio → 0.92 s (0.047× real time) |
nvidia/parakeet-tdt-0.6b-v3 | Speech to text | 19.5 s of audio → 0.7 s (0.036× real time) |
Two things worth knowing before you pick a speech model: Kokoro returns a 300-character paragraph in under a second, and Orpheus takes about a minute for the same text. Orpheus sounds more expressive; it is a pre-rendering tool, not an interactive one. Both transcription models run at roughly 4–5% of real time, so a 20-minute recording comes back in under a minute.
Method and caveats
- Client in Europe, public endpoint
api.inferenceapis.com, HTTPS, no warm-up. - Chat: prompt "Write a short paragraph (about 120 words) explaining what an API rate limit is.",
max_tokens400, streaming on; time to first token is the first chunk with non-emptycontent; throughput is completion tokens divided by generation time after the first token. - Speech: a 300-character paragraph to mp3. Transcription: the Kokoro output (about 20 seconds of audio) uploaded as mp3.
- Three runs each, median reported. Upstream capacity varies by hour; expect ±30%. We will re-run and update the model pages when we change providers or models.
The raw JSON is what the model and comparison pages read: compare any two models.