HTTP 429 rate_limit_exceeded (Groq)

What the error looks like

Groq returns HTTP 429 Too Many Requests. The body names the model, the tier, and the specific limit — tokens per minute in this example:

{
  "error": {
    "message": "Rate limit reached for model `openai/gpt-oss-120b`
                in organization `org_...` service tier `on_demand`
                on tokens per minute (TPM): Limit 8000, Used 7600, Requested 950.
                Please try again in 4.1s. ...",
    "type": "tokens",
    "code": "rate_limit_exceeded"
  }
}

The type field is tokens or requests depending on which counter tripped, and the message says whether the window is per minute or per day. That distinction decides your fix.

The five Groq limits

Every request is checked against several counters at once. You are throttled by whichever runs out first.

• RPM — requests per minute
• RPD — requests per day
• TPM — tokens per minute (prompt + completion)
• TPD — tokens per day
• ASH / ASD — audio seconds per hour / per day (Whisper transcription)

Current free-tier limits

Groq's published Free Plan limits for the models most people hit (values as of September 2026; always confirm on Groq's rate-limits page):

What 200,000 tokens a day actually buys you. A typical chat turn — a short system prompt, the user message, and a paragraph back — is roughly 400 tokens, so ~500 turns/day before TPD is exhausted. If you send conversation history on every call, that drops quickly. The 8,000 TPM cap separately means you cannot send more than about 8K tokens in any one minute, so a single long-context request can trip TPM on its own.

Read the rate-limit headers

Groq includes these headers on every response, not just errors, so you can watch your remaining budget before you hit the wall. Note the mix: the request headers are daily, the token headers are per minute.

Fix 1 — Retry with backoff (per-minute limits)

For RPM/TPM errors, honor retry-after and retry. Add jitter so parallel workers do not all retry in the same second.

import os, time, random
from openai import OpenAI, RateLimitError

client = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)

def chat(messages, retries=4):
    for attempt in range(retries):
        try:
            return client.chat.completions.create(
                model="openai/gpt-oss-120b", messages=messages
            )
        except RateLimitError as e:
            # Groq sets retry-after only on 429; fall back to exponential backoff.
            wait = float(e.response.headers.get("retry-after", 2 ** attempt))
            time.sleep(wait + random.random())
    raise RuntimeError("Groq rate limit: retries exhausted")

If you use the official SDKs, max_retries handles simple cases, but it will not help with daily limits — it just retries into the same closed window.

Fix 2 — Reduce what you send (daily limits)

• Trim history. Send the last few turns, not the whole conversation.
• Cap max_tokens. Completions count against TPD too.
• Use a smaller model for cheap steps (classification, routing) and reserve the large model for the final answer.
• Cache identical prompts.

This buys headroom; it does not remove the ceiling. Once you have real users, a daily cap becomes an outage schedule.

Fix 3 — Upgrade… if you can

The intended fix is Groq's paid Developer tier, which raises every limit. At the time of writing, Groq's billing page shows "Developer tier upgrades are temporarily unavailable due to high demand." If that is what you see, there is no self-serve way to raise your limits, and Enterprise requires a sales conversation.

We cover the practical options in Groq Developer tier unavailable. The short version is Fix 4.

Fix 4 — Fail over to a second provider (daily limits, no upgrade)

Groq speaks the OpenAI wire format, so any OpenAI-compatible endpoint is a drop-in second provider. Keep Groq as primary for speed and cost, and route to the fallback only when Groq returns 429. Your request shape does not change — only the client does.

import os
from openai import OpenAI, RateLimitError

groq = OpenAI(
    base_url="https://api.groq.com/openai/v1",
    api_key=os.environ["GROQ_API_KEY"],
)
fallback = OpenAI(
    base_url="https://api.inferenceapis.com/v1",
    api_key=os.environ["INFERENCEAPIS_KEY"],
)

def chat(messages):
    try:
        return groq.chat.completions.create(
            model="openai/gpt-oss-120b", messages=messages
        )
    except RateLimitError:
        # Same OpenAI-compatible call; only the client (base URL + key) changes.
        return fallback.chat.completions.create(
            model="llama-3.3-70b-versatile", messages=messages
        )

import OpenAI from "openai";

const groq = new OpenAI({
  baseURL: "https://api.groq.com/openai/v1",
  apiKey: process.env.GROQ_API_KEY,
});
const fallback = new OpenAI({
  baseURL: "https://api.inferenceapis.com/v1",
  apiKey: process.env.INFERENCEAPIS_KEY,
});

export async function chat(messages) {
  try {
    return await groq.chat.completions.create({ model: "openai/gpt-oss-120b", messages });
  } catch (err) {
    if (err.status !== 429) throw err;
    return await fallback.chat.completions.create({ model: "llama-3.3-70b-versatile", messages });
  }
}

If you already run a gateway, the same idea is one config block:

model_list:
  - model_name: chat
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY

  - model_name: chat-fallback
    litellm_params:
      model: openai/llama-3.3-70b-versatile     # any OpenAI-compatible endpoint
      api_base: https://api.inferenceapis.com/v1
      api_key: os.environ/INFERENCEAPIS_KEY

router_settings:
  fallbacks: [{ "chat": ["chat-fallback"] }]   # used when "chat" returns 429

Frequently asked questions

Why am I still getting 429 after waiting?

You hit a daily limit (RPD or TPD), not a per-minute one. Check the error message — it names the window. Daily limits reset only when the 24-hour window rolls over; x-ratelimit-reset-requests shows how long that is.

Does retry-after appear on every response?

No. Groq only sets retry-after on a 429. The x-ratelimit-* headers are present on every response, which is what makes proactive monitoring possible.

How do I increase my Groq rate limit?

Upgrade to the Developer tier from Groq's billing page. If the upgrade is showing as unavailable, the only self-serve alternative is to route overflow to another OpenAI-compatible provider, as shown in Fix 4.

Do free-tier limits apply per model or per account?

Per model. Each model has its own RPM/RPD/TPM/TPD counters, so exhausting openai/gpt-oss-120b does not block whisper-large-v3-turbo.

Where Inference APIs fits

Fix 4 above routes overflow to a second OpenAI-compatible endpoint. Inference APIs is one: it accepts Groq-style model ids and both /v1 and /openai/v1 paths, and bills per request with no daily cap.

Related entries

• Groq: "Developer tier upgrades are temporarily unavailable"
• 404 model_not_found: llama-3.3-70b-versatile (Groq)