Rate Limiting a Free Tier AI App: Token Buckets, Budgets and Abuse Prevention

Your free tier has been live for three days and the provider bill is ten times what your users should have cost. Someone found the endpoint, put a script on it, and your "10 requests a minute" limit was a speed bump, not a wall. To rate limit a free tier AI app properly you need three layers: a token bucket for speed, a per-user budget for cost, and a global breaker for the day everything else fails.

The short version: limit requests per minute to stop bursts, limit tokens (or rupees) per user per day to stop expensive requests, and put a hard daily spend ceiling on the whole app. Below is a small Python implementation I ran against simulated traffic, with the real output, including the part where the global breaker also locks out honest users. That trade-off is the most useful thing in this post.

What are you actually protecting?

Abuse of an AI app is mostly one of four things:

• A script on one account. A loop hitting your endpoint as fast as it can. Easy to stop.
• Many throwaway accounts. Each stays under its limit. Hard to stop, only containable.
• Expensive requests from honest users. A pasted 200-page PDF, or a prompt that triggers a long agent loop. Not malicious, still costly.
• Resale. Someone fronts your free tier as their own product. You see it as steady, evenly spaced load.

A request counter handles the first. Only a cost-based budget handles the third. Only a global ceiling bounds the second and fourth. If you want the arithmetic of what a single request costs, see LLM API cost per 1,000 requests; you need that number to set any of the limits below.

Layer 1: a token bucket per user

A token bucket holds up to capacity tokens and refills at a steady rate. Each request takes one. A user can burst up to the capacity, then is held to the refill rate. Do not confuse these "tokens" with LLM tokens; I'll call them bucket tokens.

from dataclasses import dataclass

@dataclass
class Bucket:
    capacity: float
    refill_per_sec: float
    tokens: float = None
    last: float = 0.0

    def take(self, n, now):
        if self.tokens is None:
            self.tokens = self.capacity
            self.last = now
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill_per_sec)
        self.last = now
        if self.tokens >= n:
            self.tokens -= n
            return True
        return False

Passing now in, instead of calling time.time() inside, makes it testable. I used a fake clock so the simulation below runs instantly and gives the same result every time.

My settings: capacity 5, refill 10 per minute. A real person chatting never notices. A script gets 5 requests, then roughly one every six seconds.

Layer 2: a per-user daily budget in tokens

Requests are not equal, so budget what costs money. Count LLM tokens (input plus output) per user per day, and reject a request before the model call if the estimate would push the user over.

Two details matter:

1. Estimate before, settle after. Before the call, estimate as input length plus your max_tokens cap. After the call, record the real usage numbers from the API response. If you only count afterwards, a single huge request goes through before the limit can say no.
2. Cap the request, not just the user. Set a maximum input size and a max_tokens ceiling per request. A budget of 60,000 tokens a day is meaningless if one request can use 200,000. The rate limits guide for the Claude API covers the vendor-side limits you'll also hit.

Pick the daily number from your own unit economics, not from a guess. The companion post, AI app gross margin per user, shows how to turn "I can afford Rs 20 of model cost per free user per month" into a request budget.

Layer 3: a global spend breaker

Per-user limits assume users are separate people. A person with 40 accounts is 40 users. So the last layer ignores identity: track total estimated spend for the day, and when it crosses a ceiling, stop serving the free tier until reset.

Here is the guard that combines all three layers. This is the code I ran (Python 3, standard library only):

from dataclasses import dataclass, field

@dataclass
class Guard:
    clock: object
    req_capacity: int = 5
    req_per_min: int = 10
    daily_token_budget: int = 60_000
    global_daily_usd: float = 3.0
    usd_per_token: float = 4e-6      # blended input+output, illustrative
    buckets: dict = field(default_factory=dict)
    used_tokens: dict = field(default_factory=dict)
    global_usd: float = 0.0

    def check(self, user, est_tokens):
        now = self.clock.now()
        if self.global_usd >= self.global_daily_usd:
            return False, "global_budget"
        if self.used_tokens.get(user, 0) + est_tokens > self.daily_token_budget:
            return False, "user_daily_budget"
        b = self.buckets.setdefault(user, Bucket(self.req_capacity, self.req_per_min / 60))
        if not b.take(1, now):
            return False, "rate_limited"
        return True, "ok"

    def record(self, user, actual_tokens):
        self.used_tokens[user] = self.used_tokens.get(user, 0) + actual_tokens
        self.global_usd += actual_tokens * self.usd_per_token

The order of checks is deliberate. The global breaker goes first because it's the cheapest rejection. The budget check goes before the bucket so a rejected request doesn't consume a bucket token. Note the usd_per_token figure is a made-up blended rate for the simulation, not a vendor price; use your own from the cost post linked above.

Does it hold up against abuse? A simulation

I generated three kinds of traffic over a two-hour window (7,200 seconds), with a seeded random generator so you can reproduce it:

• 20 normal users. One request every 60 to 300 seconds, 1,500 tokens each.
• 1 script user. Two requests a second for 10 minutes, 3,000 tokens each, one account.
• 40 sockpuppet accounts. Each sends 8 requests in a burst, 3,000 tokens each, spread across the window.

Output of python3 sim.py:

global breaker tripped at t=3050 s (sock10)
normal {'ok': 354, 'global_budget': 469}
script {'ok': 20, 'rate_limited': 161, 'user_daily_budget': 1019}
socks {'ok': 54, 'rate_limited': 30, 'global_budget': 236}
global spend USD: 3.01
script tokens used: 60000 of 60000 budget
sockpuppet tokens served: 162000 across 11 accounts
without global breaker, spend USD: 7.39

What this says:

• The script was contained by its own account's limits. It got 20 requests through (exactly its 60,000-token budget at 3,000 each) out of 1,200 attempts. The bucket and the budget did their jobs.
• The sockpuppets were not contained by per-user limits. Each account stayed inside its own limits, and the breaker only stopped them at about $3 of spend after 11 accounts. Without the breaker the same traffic cost $7.39. That figure uses my illustrative price, so read it as a ratio: the breaker cut simulated spend by about 60 percent.
• The breaker hurt honest users. 469 normal requests were rejected with global_budget after t=3050 seconds, because the abuser spent the shared pool. That is the cost of a global ceiling: you are choosing a bounded bill over availability. Honest users get an error until reset.

In my first run, with a 20,000-token daily budget, the per-user limit also rejected 563 normal requests. A budget set without looking at normal usage blocks the people you want to keep. That run is the reason the final number is 60,000: run your own logs through the simulator before choosing.

How do you soften the global breaker?

A hard stop that punishes everyone is a bad failure mode. Options, roughly from cheap to involved:

1. Degrade, don't stop. When the breaker trips, route the free tier to a smaller model, shorter outputs or cached answers rather than refusing. See AI model routing and semantic caching.
2. Reserve a pool for paying users. Keep a separate budget so paid traffic never hits the free tier's ceiling.
3. Set the ceiling per hour as well as per day. A burst at 2am shouldn't spend the whole day's allowance.
4. Make signup cost something. Verified email, then a phone number or a small card check before the free tier switches on. Each step cuts sockpuppets, and each step also cuts honest signups. Add them when your logs show the abuse, not before.
5. Add per-IP and per-device limits as an extra key, not a replacement. Shared networks (offices, colleges, mobile carriers) put many real users behind one IP, so keep the IP limit generous.

Moving this to production

The code above is a sketch. Before you rely on it:

• Share the state. Run the counters in Redis or similar so all app instances agree, and make check-and-decrement atomic. In-process dictionaries reset on deploy and split across workers.
• Use real usage, not estimates, for the ledger. Read the usage fields from each response and settle the difference.
• Return useful errors. Send HTTP 429 with a Retry-After header for the bucket, and a different, clearly worded error for "daily limit reached" so the client can show a message instead of retrying forever.
• Log every rejection with the user, reason and estimated tokens. That log is how you tune the numbers and spot a pattern like evenly spaced requests all day.
• Watch the prompt side. Cost abuse often arrives as a prompt that makes your app do more work, such as asking for long outputs or looping an agent. Cap tool-call counts per request. The prompt injection defence guide covers the related security side, and FastAPI production patterns has the surrounding service structure.
• Test it. Replay a day of real traffic through the simulator, then replay it with a scripted abuser added. If the abuser still costs more than your threshold, tighten the budget or the breaker.

A quick settings table

Starting points to adapt, not recommendations backed by measurement. They come from the simulation above and need replacing with your own numbers.

If you only do one thing from this post, do the last row. A spend ceiling is the only control that still works when everything else has been bypassed.