Building AI-Powered Apps · Unit 2 of 8
Unit 2: Tokens, cost, limits and streaming
Counting input and output tokens, working out what every request costs, handling rate limits and errors with backoff, and streaming answers so they feel fast.
Opens the first lesson of this unit in Building AI-Powered Apps. Every lesson is open from the start.
Lesson 1: Counting tokens
Goal: Estimate input and output tokens and set max_tokens sensibly.
Providers measure everything in tokens: limits, speed and price. In English a token averages about four characters, or three-quarters of a word; code, numbers and many other languages use more tokens per word. Each provider has its own tokenizer, so exact counts differ a little between models.
| Text | Characters | Rough tokens (characters ÷ 4) |
|---|---|---|
| "Where is order 5512?" | 20 | about 5 |
| A 150-word support email | about 800 | about 200 |
| A 10-page policy document | about 30,000 | about 7,500 |
Input tokens are everything you send: the system prompt, history, context and the new message. Output tokens are what the model writes. max_tokens caps the output; if a reply reaches the cap, it is cut off and the stop reason says so. Set max_tokens high enough for a complete answer and low enough to stop a runaway reply.
Worked example: estimating one request
- System prompt 1,200 characters, history 2,000, new message 400: 3,600 characters, about 900 input tokens.
- A typical reply is about 120 words, roughly 160 output tokens.
- max_tokens of 400 leaves room for a longer answer without allowing an essay.
Rough estimates are enough for planning; use the provider's reported usage for billing and limits.
- Input tokens
- Tokens in everything sent to the model: system prompt, history, context and message.
- Output tokens
- Tokens the model generates in its reply.
- max_tokens
- The request setting that caps how many output tokens the reply may use.
- Tokenizer
- The tool that splits text into a model's tokens.
Remember: Characters ÷ 4 ≈ tokens. Input is everything you send; output is the reply, capped by max_tokens.
Lesson 2: The cost of every request
Goal: Calculate request and monthly costs and name the main ways to reduce them.
Providers charge per million tokens, usually with a higher price for output than for input, and higher prices for larger models. The examples in this course use invented prices to keep the arithmetic easy; real providers publish their own, and they change often.
| Model (invented prices) | Input, per million tokens | Output, per million tokens |
|---|---|---|
| Small | 1 dollar | 5 dollars |
| Large | 3 dollars | 15 dollars |
Worked example: the cost of a support assistant
Each turn sends about 2,000 input tokens and gets back about 300 output tokens. The app handles 10,000 turns a day on the large model.
- Input: 2,000 × 10,000 = 20 million tokens a day, × 3 dollars per million = 60 dollars.
- Output: 300 × 10,000 = 3 million tokens a day, × 15 dollars per million = 45 dollars.
- Total: about 105 dollars a day, roughly 3,150 dollars over a 30-day month.
- On the small model the same traffic costs 20 + 15 = 35 dollars a day.
Cost = tokens × requests × price. Multiply it out before launch, not after the first bill.
- Trim the system prompt and context to what the task needs.
- Limit how much conversation history is re-sent (Unit 1).
- Route easy requests to a smaller model and only hard ones to a larger one.
- Use the provider's prompt caching for repeated prompt beginnings, and cache whole answers to identical questions in your own app.
- Cap max_tokens and turns per conversation, and set per-user daily limits so one user cannot run up the bill.
- Price per million tokens
- How providers quote prices, usually separately for input and output.
- Model routing
- Sending each request to the cheapest model that can handle it well.
- Response caching
- Storing answers to repeated questions so the model is not called again.
Remember: Cost = (input tokens × input price + output tokens × output price) × requests. Trim, route, cache, cap.
Lesson 3: Rate limits, errors and retries
Goal: Handle HTTP errors correctly, including 429 with exponential backoff and jitter.
Providers limit how much you can send: requests per minute and tokens per minute, per account or per key. Go over them and you get an error instead of an answer. Your app must expect errors and handle them calmly.
| HTTP status | Meaning | What your app should do |
|---|---|---|
| 400 | Your request is malformed: bad JSON, too long, a wrong field | Fix the request. Do not retry it unchanged |
| 401 or 403 | Missing or invalid key, or not allowed | Check the configuration and alert a developer |
| 429 | Too many requests: you are rate limited | Wait and retry with backoff; respect any Retry-After header |
| 500, 502 or 503 | An error or overload on the provider's side | Retry a few times with backoff, then show a friendly message |
| Timeout | No answer in time | Retry once or twice; consider streaming or a smaller request |
Exponential backoff means waiting longer after each failed attempt, for example 1, 2, 4 and then 8 seconds, plus a little random jitter so thousands of clients do not all retry at the same instant. Cap the number of retries, and never retry errors that will not fix themselves, such as a 400.
Hands-on lab in the room: Simulated model API and budget planner. Send requests to a simulated model API, watch token counts and costs, hit the rate limit and handle it with backoff. The model and its answers are simulated; the arithmetic is real.
async function callWithRetry(send, tries = 4) {
for (let attempt = 0; attempt < tries; attempt++) {
const res = await send();
if (res.status !== 429 && res.status < 500) return res; // success, or an error retrying will not fix
const wait = 1000 * 2 ** attempt + Math.random() * 250; // 1 s, 2 s, 4 s... plus jitter
await new Promise(r => setTimeout(r, wait));
}
throw new Error('The service is busy. Please try again shortly.');
}
- Rate limit
- A cap on how many requests or tokens you may send per minute.
- HTTP 429
- The "too many requests" error returned when you go over a rate limit.
- Exponential backoff
- Waiting longer after each failed attempt, often doubling the wait.
- Jitter
- A small random addition to retry waits so clients do not retry in lockstep.
Remember: 429 and 5xx: back off and retry a few times. 400: fix the request. Always cap retries.
Lesson 4: Streaming and latency
Goal: Explain streaming and time to first token, and know when not to stream.
A long answer can take many seconds to generate. With streaming, the provider sends the reply in small pieces as they are generated, usually as server-sent events, and your app shows the text as it arrives. The total time is about the same, but the wait before anything appears, the time to first token, drops from seconds to a fraction of a second.
Worked example: two ways to wait
- Without streaming: the user watches a spinner for 6 seconds, then sees a 250-word answer all at once.
- With streaming: the first words appear after about half a second, and the rest flows in over the next 6 seconds.
- Same total time, very different feel, and the user can stop a reply that is heading the wrong way.
Streaming improves how fast an app feels, not how fast the model writes.
In a typical setup your server opens a streaming request to the provider and passes each piece on to the browser as it arrives. Handle the edges: if the user closes the page, cancel the provider request so you stop paying for tokens nobody will read; if the stream breaks halfway, show what arrived plus a retry button; and record usage, which usually arrives at the end of the stream.
Watch out: Do not stream output that must be checked before anyone sees it. If a reply must be valid JSON, pass a safety check or be approved, collect the whole reply on the server, check it, then send it. Streaming is for text a person reads as it is written.
- Streaming
- Receiving and showing a reply in pieces as it is generated.
- Server-sent events
- A web standard for a server to push a stream of messages to a browser.
- Time to first token
- How long until the first piece of a reply arrives.
- Latency
- The delay between sending a request and getting a response.
Remember: Stream text people read; collect and check anything code must validate. Cancel when the user leaves.
Practise it in Building AI-Powered Apps
Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Simulated model API and budget planner.
Every lesson is free: no ads, no lives and no paywall.