Inference latency, measured and published
Most inference providers publish no latency figures at all. Below are ours, measured against the public Enclave42 API on 11 August 2026, with the protocol, the raw data, and the script that produced them.
What was measured
Three models, one per access tier, at three prompt lengths, with 30 requests per combination — 270 requests in total. 269 succeeded. Requests were sent over HTTPS from a client outside our infrastructure, in streaming mode, with temperature: 0 and deterministic prompts.
Measured input lengths were 191, 1,861 and 18,605 tokens, read from the API's own usage block rather than estimated.
Results
| Model | Tier | Prompt tokens | Output tokens (median) | TTFT p50 | TTFT p95 | Total latency p50 | Total latency p95 | Throughput p50 |
|---|---|---|---|---|---|---|---|---|
| mistral-nemo-tee | entry | 191 | 152 | 2,826 ms | 4,203 ms | 4,283 ms | 5,794 ms | 104.3 tok/s |
| mistral-nemo-tee | entry | 1,861 | 161 | 2,777 ms | 3,835 ms | 4,598 ms | 6,414 ms | 95.8 tok/s |
| mistral-nemo-tee | entry | 18,605 | 256 | 3,143 ms | 4,504 ms | 7,392 ms | 8,634 ms | 63.2 tok/s |
| qwen3-32b-tee | standard | 191 | 612 | 7,422 ms | 14,361 ms | 36,376 ms | 65,303 ms | 23.6 tok/s |
| qwen3-32b-tee | standard | 1,861 | 618 | 9,008 ms | 14,420 ms | 32,613 ms | 49,294 ms | 27.1 tok/s |
| qwen3-32b-tee | standard | 18,605 | 634 | 15,083 ms | 26,380 ms | 74,765 ms | 132,002 ms | 9.8 tok/s |
| qwen3-6-27b-tee | advanced | 191 | 1,024 | 7,925 ms | 91,451 ms | 25,574 ms | 155,440 ms | 66.2 tok/s |
| qwen3-6-27b-tee | advanced | 1,861 | 1,024 | 6,263 ms | 10,121 ms | 17,858 ms | 28,983 ms | 88.9 tok/s |
| qwen3-6-27b-tee | advanced | 18,605 | 1,024 | 6,290 ms | 8,962 ms | 27,540 ms | 32,963 ms | 48.3 tok/s |
Why total latency differs so much between models
These models do not perform the same amount of work. mistral-nemo-tee produced a median of 152 to 256 output tokens. qwen3-32b-tee produced 612 to 634. qwen3-6-27b-tee produced 1,024 in every single request.
Both qwen models are reasoning models: they emit a chain of thought before any answer, and that chain counts as generated output. A total latency of 74 seconds on qwen3-32b-tee with a long prompt reflects roughly four times the generation work of the mistral row above it, not four times the slowness per token.
max_tokens was set to 256 for mistral-nemo-tee and 1,024 for both qwen models — at 256, the reasoning models exhausted their budget before producing any answer at all. Throughput figures are therefore not directly comparable across models with different budgets.
One consequence we have not investigated: qwen3-6-27b-tee reached the 1,024-token ceiling in 100% of requests, including on the 191-token prompt. Its measurements describe 1,024 tokens of generation, not a completed answer.
Outliers
Two of the thirty short-prompt requests to qwen3-6-27b-tee exceeded 90 seconds to first token, against a median of 7.9 seconds. This is what pushes that row's p95 figures far above the rest of the table. We have not investigated the cause. The raw measurements are in the CSV below.
One request out of 270 failed, with the stream terminated mid-response on qwen3-6-27b-tee. It is recorded in the CSV as a failure and excluded from the percentiles.
Method
TTFT is the time from sending the request to the first streamed event carrying non-empty content — including reasoning_content, which is what a user actually sees appear on screen first with reasoning models. Total latency is the time to the final [DONE] event. Throughput is output tokens divided by the generation window alone, from first to last content event, not from the start of the request.
All token counts are read from the API's usage block. Percentiles use nearest-rank over 30 samples. We report medians and p95, never averages alone.
Reproduce this yourself
The script that produced these numbers is in our repository, has no dependencies, and runs on any machine with Node.js. Prompts are deterministic: a third party running it obtains exactly the same inputs. On Enclave42, an API key is bound to a single model, so reproducing the full run requires one key per model.
What these numbers are and are not
They are the latencies a client of the Enclave42 API observed on 11 August 2026, under the load conditions of that moment. They are measurements, not a service level commitment. Latency varies with model, prompt length, generation budget and infrastructure load.