Skip to content
FORGENO ADS GAMESUnlock all $8

Building AI-Powered Apps · Unit 2 of 8

Unit 2: Tokens, cost, limits and streaming

Counting input and output tokens, working out what every request costs, handling rate limits and errors with backoff, and streaming answers so they feel fast.

Opens the first lesson of this unit in Building AI-Powered Apps. Every lesson is open from the start.

Lesson 1: Counting tokens

Goal: Estimate input and output tokens and set max_tokens sensibly.

Providers measure everything in tokens: limits, speed and price. In English a token averages about four characters, or three-quarters of a word; code, numbers and many other languages use more tokens per word. Each provider has its own tokenizer, so exact counts differ a little between models.

TextCharactersRough tokens (characters ÷ 4)
"Where is order 5512?"20about 5
A 150-word support emailabout 800about 200
A 10-page policy documentabout 30,000about 7,500

Input tokens are everything you send: the system prompt, history, context and the new message. Output tokens are what the model writes. max_tokens caps the output; if a reply reaches the cap, it is cut off and the stop reason says so. Set max_tokens high enough for a complete answer and low enough to stop a runaway reply.

Worked example: estimating one request

  1. System prompt 1,200 characters, history 2,000, new message 400: 3,600 characters, about 900 input tokens.
  2. A typical reply is about 120 words, roughly 160 output tokens.
  3. max_tokens of 400 leaves room for a longer answer without allowing an essay.

Rough estimates are enough for planning; use the provider's reported usage for billing and limits.

Input tokens
Tokens in everything sent to the model: system prompt, history, context and message.
Output tokens
Tokens the model generates in its reply.
max_tokens
The request setting that caps how many output tokens the reply may use.
Tokenizer
The tool that splits text into a model's tokens.

Remember: Characters ÷ 4 ≈ tokens. Input is everything you send; output is the reply, capped by max_tokens.

Lesson 2: The cost of every request

Goal: Calculate request and monthly costs and name the main ways to reduce them.

Providers charge per million tokens, usually with a higher price for output than for input, and higher prices for larger models. The examples in this course use invented prices to keep the arithmetic easy; real providers publish their own, and they change often.

Model (invented prices)Input, per million tokensOutput, per million tokens
Small1 dollar5 dollars
Large3 dollars15 dollars

Worked example: the cost of a support assistant

Each turn sends about 2,000 input tokens and gets back about 300 output tokens. The app handles 10,000 turns a day on the large model.

  1. Input: 2,000 × 10,000 = 20 million tokens a day, × 3 dollars per million = 60 dollars.
  2. Output: 300 × 10,000 = 3 million tokens a day, × 15 dollars per million = 45 dollars.
  3. Total: about 105 dollars a day, roughly 3,150 dollars over a 30-day month.
  4. On the small model the same traffic costs 20 + 15 = 35 dollars a day.

Cost = tokens × requests × price. Multiply it out before launch, not after the first bill.

  • Trim the system prompt and context to what the task needs.
  • Limit how much conversation history is re-sent (Unit 1).
  • Route easy requests to a smaller model and only hard ones to a larger one.
  • Use the provider's prompt caching for repeated prompt beginnings, and cache whole answers to identical questions in your own app.
  • Cap max_tokens and turns per conversation, and set per-user daily limits so one user cannot run up the bill.
Price per million tokens
How providers quote prices, usually separately for input and output.
Model routing
Sending each request to the cheapest model that can handle it well.
Response caching
Storing answers to repeated questions so the model is not called again.

Remember: Cost = (input tokens × input price + output tokens × output price) × requests. Trim, route, cache, cap.

Lesson 3: Rate limits, errors and retries

Goal: Handle HTTP errors correctly, including 429 with exponential backoff and jitter.

Providers limit how much you can send: requests per minute and tokens per minute, per account or per key. Go over them and you get an error instead of an answer. Your app must expect errors and handle them calmly.

HTTP statusMeaningWhat your app should do
400Your request is malformed: bad JSON, too long, a wrong fieldFix the request. Do not retry it unchanged
401 or 403Missing or invalid key, or not allowedCheck the configuration and alert a developer
429Too many requests: you are rate limitedWait and retry with backoff; respect any Retry-After header
500, 502 or 503An error or overload on the provider's sideRetry a few times with backoff, then show a friendly message
TimeoutNo answer in timeRetry once or twice; consider streaming or a smaller request

Exponential backoff means waiting longer after each failed attempt, for example 1, 2, 4 and then 8 seconds, plus a little random jitter so thousands of clients do not all retry at the same instant. Cap the number of retries, and never retry errors that will not fix themselves, such as a 400.

Hands-on lab in the room: Simulated model API and budget planner. Send requests to a simulated model API, watch token counts and costs, hit the rate limit and handle it with backoff. The model and its answers are simulated; the arithmetic is real.

async function callWithRetry(send, tries = 4) {
  for (let attempt = 0; attempt < tries; attempt++) {
    const res = await send();
    if (res.status !== 429 && res.status < 500) return res;  // success, or an error retrying will not fix
    const wait = 1000 * 2 ** attempt + Math.random() * 250; // 1 s, 2 s, 4 s... plus jitter
    await new Promise(r => setTimeout(r, wait));
  }
  throw new Error('The service is busy. Please try again shortly.');
}
Rate limit
A cap on how many requests or tokens you may send per minute.
HTTP 429
The "too many requests" error returned when you go over a rate limit.
Exponential backoff
Waiting longer after each failed attempt, often doubling the wait.
Jitter
A small random addition to retry waits so clients do not retry in lockstep.

Remember: 429 and 5xx: back off and retry a few times. 400: fix the request. Always cap retries.

Lesson 4: Streaming and latency

Goal: Explain streaming and time to first token, and know when not to stream.

A long answer can take many seconds to generate. With streaming, the provider sends the reply in small pieces as they are generated, usually as server-sent events, and your app shows the text as it arrives. The total time is about the same, but the wait before anything appears, the time to first token, drops from seconds to a fraction of a second.

Worked example: two ways to wait

  1. Without streaming: the user watches a spinner for 6 seconds, then sees a 250-word answer all at once.
  2. With streaming: the first words appear after about half a second, and the rest flows in over the next 6 seconds.
  3. Same total time, very different feel, and the user can stop a reply that is heading the wrong way.

Streaming improves how fast an app feels, not how fast the model writes.

In a typical setup your server opens a streaming request to the provider and passes each piece on to the browser as it arrives. Handle the edges: if the user closes the page, cancel the provider request so you stop paying for tokens nobody will read; if the stream breaks halfway, show what arrived plus a retry button; and record usage, which usually arrives at the end of the stream.

Watch out: Do not stream output that must be checked before anyone sees it. If a reply must be valid JSON, pass a safety check or be approved, collect the whole reply on the server, check it, then send it. Streaming is for text a person reads as it is written.

Streaming
Receiving and showing a reply in pieces as it is generated.
Server-sent events
A web standard for a server to push a stream of messages to a browser.
Time to first token
How long until the first piece of a reply arrives.
Latency
The delay between sending a request and getting a response.

Remember: Stream text people read; collect and check anything code must validate. Cancel when the user leaves.

Practise it in Building AI-Powered Apps

Each lesson ends with three questions that check you can use the idea, not just recognise it. Wrong answers explain why and come back in Review. The unit ends with a 6-question quiz; pass it with 80% or more to master the unit. Hands-on lab: Simulated model API and budget planner.

Every lesson is free: no ads, no lives and no paywall.