Handle streaming, queues, and API retries
Build a chat experience that handles waiting, cancellation, and temporary capacity errors.
Separate waiting from generation
Time to first token includes queueing and prompt processing. Once output begins, display it incrementally. A connection opening is not evidence that the model has started producing text.
Retry with restraint
On a temporary rate or capacity failure, honor Retry-After when present and use exponential backoff with jitter. Bound the retry count. Immediately retrying in parallel adds pressure and can make waits worse.
Do not duplicate a partial answer
If a stream fails after text arrives, show the interruption and let the user choose whether to retry. A new generation may differ. Cancel unwanted requests promptly so capacity can be released.
Your next step
Check available models, open your console, or read the API documentation. Public signup and paid plans are still being prepared.