Why is the first AI reply so slow over an API?

Where the time goes before the first word

The wait before the first word has three parts: waiting for your turn, reading the prompt, and starting to write. Reading grows with the prompt. A fresh chat with a short card is read in a moment; reopening a story with 40,000 tokens of history takes noticeably longer, because every one of those tokens is processed before the reply can begin.

Writing speed after the first word is a separate thing. A reply that starts slowly and then flows normally points to reading or queueing, not a slow model.

Causes, most common first

• Reading from scratch: the first message of a session, an old chat reopened, or anything that changed near the top of the prompt. Reuse only works when the beginning is identical to last time.
• A top that changes every turn: macros such as {{time}}, {{date}}, {{random}} or {{idle_duration}} in the card or system prompt, or lorebook entries switching on and off above the character definition.
• A chat past its context size: once the app trims the oldest messages every turn, the start keeps shifting, and every reply is read in full.
• A queue at busy times.
• Streaming off, so nothing appears until the whole reply is finished.
• A model that was idle or restarting and needs a moment.

What actually shortens it

• Turn streaming on, so words appear as they are written. In SillyTavern it is Streaming under AI Response Configuration; labels can change between versions.
• Move changing macros out of the card and system prompt, or delete them.
• Place lorebook entries that come and go lower in the prompt, for example at a chat depth, rather than before the character definition.
• In very long stories, write a short summary of older events and start a new chat from it, so the app is not trimming every turn.
• Let one reply finish before swiping or sending from another tab, so your own requests do not queue behind each other.

Which cause do you have?

• Only the first reply after a break is slow, the rest are quick: normal reading from scratch. Nothing to fix.
• Every reply in one long chat is slow: something changes at the top each turn, or the chat is past its context size.
• Slow at busy evening hours only: the queue.
• The whole reply appears at once after a long pause: streaming is off.
• The first reply after a quiet spell takes a long time or fails, then all is well: the model was starting.

First-reply speed on UnsavedAI

Our models keep the start of your chat in short-term memory between messages, so with a stable prompt only your new lines are read. Free writes one reply at a time on spare capacity; Priority goes ahead of waiting Free requests and writes two at once. gemma-4-31b, being larger, can take a little longer to start than gemma-4-12b.

If waiting runs long you get a message rather than silence: "Your earlier requests were still running and this one waited too long. Try again shortly." or "The queue wait limit was reached. Try again shortly." A model that is restarting answers "No healthy model worker is available."; Models shows when it is back. The long-roleplay context guide covers prompt ordering in more depth.

Questions people ask