"Context length exceeded" error: why chats fail as they grow

What the error is telling you

Every model has a fixed window of tokens it can handle in one request, and that window holds both the prompt and the answer. Providers word the refusal differently: "maximum context length", "context_length_exceeded", "prompt is too long" or "input too long". They all mean the bundle your app built does not fit.

Your chat app builds that bundle, so the app is where the fix lives. Nothing is broken on the model's side, and swiping or retrying sends the same oversized prompt again, so it fails again.

Causes, most common first

• The app's context setting is above the model's limit, often left over from a preset or a previous model with a bigger window. It hides for a while: short chats fit, and the failure only starts once history fills the setting.
• The setting equals the limit exactly, so the prompt leaves no room for any reply.
• The app does not trim history at all. Some simple front ends and home-made scripts send every message ever written.
• One huge addition: a pasted document, a lorebook whose broad keywords pull in dozens of entries, or a group chat sending several full cards.
• Counting mismatch: the app estimates tokens with a different tokenizer, so what it thinks fits is slightly too big for the model.

Fixing it step by step

First find the model's real limit on the provider's model page. Subtract your reply length (Max Response Length in SillyTavern, Max new tokens in JanitorAI) and keep about 4,000 tokens spare. Type the result into the app's context setting: Context Size under AI Response Configuration in SillyTavern, Context size in JanitorAI's generation settings. Labels can change between versions.

If the setting is already well below the limit, find what is big. SillyTavern can show how the last prompt was split between card, lore, examples and history (where it lives can change between versions). Narrow lorebook keywords, trim a bloated card, or move a long pasted text out of the chat. In an app with no trimming, start a fresh chat and open it with a short summary of the story so far.

Which cause do you have?

• Fails on the very first message of a new chat: the card, lorebook or instructions alone are too big, or the reply length is set so high that prompt plus reply cannot fit. Some providers refuse in that case rather than shorten the reply.
• Worked for a hundred messages, then every message fails: the context setting is above the limit.
• Started right after switching model or provider: the new window is smaller than the old one.
• Fails on one message, works on the next, in a chat near the edge: the counting mismatch. Widen the margin by another 2,000 to 4,000.

The same error on UnsavedAI

Our gateway says "Prompt must leave room for output within the …-token context." where the gap is the model's limit, 61,440 for both gemma-4-12b and gemma-4-31b. A rarer form is "The prompt exceeds the available context."

We shorten a reply that would not fit instead of refusing, so this error only appears when the prompt itself leaves no room. That means lowering Max Response Length does not help; lowering the context setting does. Use 56,000 on either plan, and context size for roleplay explains where that number comes from. If replies arrive but stop early in long chats, that is the shortened-reply case covered in why replies cut off.

Questions people ask