Time to first token (TTFT) explains a familiar LLM behaviour: a noticeable pause before the first word, followed by a stream of much faster tokens. If a model needs 1.5 seconds to begin but delivers later tokens roughly every 30 ms, the gap is usually the result of how transformer inference works, not simply a slow model.

TTFT matters because it determines whether chat, copilots and agent interfaces feel responsive. It also exposes where latency is really being introduced: input processing, retrieval, queueing, network delivery or the model itself.

This guide separates TTFT from generation speed, explains the prefill/decode split, and shows how to improve first-token latency without damaging the rest of the user experience.

Time to first token is not generation speed

Streaming LLM applications need at least three measurements:

• Time to first token (TTFT): from request submission to the first meaningful token received by the client.
• Inter-token latency (ITL), also called time per output token: the gap between subsequent streamed tokens.
• Total response time: the time until the answer is complete.

According to NVIDIA’s LLM benchmarking documentation, TTFT generally includes queueing, prefill and network latency. It is therefore a user-facing end-to-end measure, not merely a model benchmark.

TTFT
≈ network and gateway work
+ queueing delay
+ tokenisation and prompt assembly
+ retrieval or other orchestration
+ prefill computation
+ first decode step
+ stream delivery

After the first output token, a rough completion-time model is:

total response time
≈ TTFT + (output tokens - 1) × ITL

With a 1.5-second TTFT, 30 ms ITL and a 100-token answer, total response time is about 4.47 seconds. Reducing TTFT by 800 ms changes the experience immediately because the user sees the response start much earlier.

Why time to first token has a prefill cost

Most LLMs generate text autoregressively: each output token depends on everything that came before it. The input prompt must be processed before the model can generate the first answer token.

Prefill: reading the entire effective prompt

The model receives far more than the latest user message. An application may supply system instructions, tool schemas, conversation history, retrieved documents and user-specific context. During prefill, the model processes that input and creates attention state called the key-value cache, or KV cache.

input prompt
  → tokenise
  → process tokens through transformer layers
  → create KV-cache state
  → predict the first output token

The cache is essential because it prevents the model from recalculating all earlier attention work on every generation step. However, it has to be built first. Longer input contexts normally create more prefill work and increase TTFT. NVIDIA makes this relationship explicit in its TTFT metric guidance.

Decode: generating later tokens one at a time

Once prefill is complete, the model uses the existing KV cache with the previous output token to generate the next token, then appends new KV values and repeats.

existing KV cache + previous token
  → next token
  → append state
  → repeat

Decode is still computationally expensive, but it generally handles one newly generated token at a time. That difference creates the visible latency gap: a slower start, then a comparatively steady stream.

Prefill and decode also exercise hardware differently. Prefill is often compute-heavy across many prompt tokens. Decode is frequently constrained by repeated movement of model and cache data through memory. One batching or GPU configuration is rarely perfect for both phases.

What makes time to first token slow in production?

Time to first token rises with effective prompt size

A user can ask a short question while the model receives tens of thousands of input tokens:

system instructions:            1,500 tokens
tool schemas:                   4,000 tokens
conversation history:           8,000 tokens
retrieved documentation:       12,000 tokens
user message:                      80 tokens
-------------------------------------------
effective input:               25,580 tokens

Common causes are endlessly replaying the full conversation, attaching every tool definition, retrieving large overlapping chunks and putting raw logs or JSON into the prompt. The relevant measure is not the length of the user’s sentence; it is the complete token count sent to the model.

This is a context-engineering problem as well as a latency problem. The site’s guide to context engineering for AI agents explores the related question of balancing useful evidence against unnecessary context. Context should earn its place in the prompt.

Time to first token includes queueing under load

A system can have impressive model throughput and still produce poor TTFT when its inference workers are busy. A new request may wait behind active decode streams or long prompt-prefill jobs.

P50 TTFT:  450 ms
P95 TTFT: 2.8 s
P99 TTFT: 8.1 s

These figures describe a service with an acceptable median but a poor tail experience. One in twenty users waits almost three seconds before seeing any output. Record queue time separately from prefill time; otherwise capacity, routing and model-compute problems become difficult to distinguish.

Time to first token can be dominated by RAG work

For retrieval-augmented generation, the model may not be the first dependency:

user request
  → authentication and policy checks
  → query rewrite or classification
  → embeddings and vector search
  → reranking and document fetch
  → prompt assembly
  → model prefill
  → first token

A slow vector index, remote data source, reranker or permission check can increase TTFT while the LLM has not started. NVIDIA’s Enterprise RAG guidance similarly separates retrieval’s impact on first-token latency from its smaller impact on later-token cadence.

Networks, buffering and cold starts also matter

For short prompts, connection setup, cross-region calls, TLS, proxies and frontend buffering can be significant. A server may emit a token promptly while a proxy or browser waits before showing it. Cold starts are a different category again: model loading, GPU allocation, compilation and cache warming can make the first request much slower than normal traffic. Measure cold and warm requests separately.

Measure time to first token before optimising it

Use timestamps that map to real ownership boundaries:

request_received_at
retrieval_started_at
retrieval_completed_at
prompt_ready_at
inference_queued_at
inference_started_at
first_token_received_at
response_completed_at

These let a team derive retrieval duration, prompt-assembly time, queueing, server-side prefill, client-observed TTFT and post-first-token generation time. Add the dimensions that explain differences: model, region, input tokens, output tokens, retrieved tokens, cache-hit status, queue depth, request type and selected tools.

Use P50, P95 and P99 results for interactive workloads. An average can conceal exactly the tail latency that frustrates users.

How to reduce time to first token without reducing answer quality

Reduce avoidable prompt work first

For many products, the most valuable TTFT improvement is a deliberate prompt budget:

stable instructions:          1,000 tokens
conversation summary:         1,000 tokens
recent conversation turns:    2,000 tokens
retrieved evidence:           4,000 tokens
user request:                   500 tokens
---------------------------------------
target input budget:          8,500 tokens

Summarise older turns, retain recent turns verbatim when necessary, deduplicate retrieved passages and define strict limits for tool schemas and evidence. Do not strip context indiscriminately. The objective is to remove redundant material while keeping the information required for a correct answer.

Use cacheable prefixes to reduce time to first token

When requests begin with the same tokens, an inference system can sometimes reuse previously computed KV-cache blocks. vLLM’s prefix-caching design describes this as avoiding redundant prompt computation for requests with a shared prefix.

stable system instructions
stable tool definitions
stable tenant or product material
conversation-specific context
retrieved documents
current user message

Keep stable material stable and ordered consistently. Avoid placing timestamps, random IDs or user-specific values near the start unless they are required there. One dynamic early token can prevent reuse of an otherwise useful prefix.

Caching has provider-specific rules around prefix length, retention, privacy and billing. OpenAI, for example, documents prompt_cache_key and optional extended retention in its API reference. Treat caching as a measured optimisation rather than an assumption.

Bound retrieval work before time to first token suffers

Run independent retrieval steps in parallel. Precompute embeddings and useful metadata. Cache common results where the access model permits it. Set deadlines for slow sources and apply reranking where it produces enough quality gain to justify its delay.

Beginning a factual answer before the needed evidence arrives is usually a bad trade. A progress state such as “Searching your workspace” can be honest and useful; implying that the answer has already been grounded is not.

Tune batching around both TTFT and inter-token latency

Batching improves utilisation but can hurt first-token latency if new work waits too long to join a batch. Conversely, aggressively admitting a long prompt can disrupt responses that are already streaming.

Chunked prefill processes large prompts in smaller segments so the server can interleave prompt work with active decoding. vLLM explains that chunked prefill can balance compute-bound prefill with memory-bound decode.

• Prioritising full prefills can improve new-request TTFT.
• Prioritising decode can protect the smoothness of existing streams.
• Batch-token budgets and chunk sizes should be tuned against P95 TTFT and P95 ITL together.

There is no universal batch size. A configuration should reflect the product’s service-level objectives and observed traffic shape.

Disaggregate only when the time to first token evidence supports it

At high sustained concurrency, dedicated prefill and decode worker pools can be worthwhile:

incoming request
  → prefill workers
  → KV-cache/state transfer
  → decode workers
  → streamed response

vLLM’s disaggregated-prefill documentation explains why the approach can tune TTFT and inter-token latency separately, while also marking the capability experimental. It adds scheduling, cache-transfer, failure-handling and observability complexity, so it should follow measured contention rather than architectural fashion.

A time to first token diagnosis example

Suppose an assistant reports:

P50 TTFT: 1.6 s
P95 TTFT: 5.4 s
ITL:      32 ms

Instrumenting the request path shows:

P50 retrieval:   180 ms
P50 queueing:    920 ms
P50 prefill:     380 ms
P50 transport:   120 ms

Decode is not the first bottleneck. The priority is capacity and routing for interactive requests, followed by prompt budgeting and cache reuse. Only then should the team consider whether token cadence needs further work.

The practical takeaway

The pause before an LLM starts writing is a normal result of preparing the model to answer. The system may be retrieving evidence, constructing a prompt, waiting for capacity, processing the full input and creating KV-cache state before it can deliver the first stream event.

Optimise time to first token by removing avoidable pre-generation work, measuring queueing independently, preserving cacheable prompt prefixes and tuning serving policy for the experience the product promises. When TTFT, ITL, token counts, retrieval, cache use and queueing are measured separately, “the model is slow” becomes a specific engineering problem with actionable trade-offs.

The post Why LLMs Pause Before They Start: Time to First Token Explained appeared first on Alpesh Kumar.