Inference APIs

404 model_not_found: gemma2-9b-it (Groq)

Gemma 2 9B was the small, permissively licensed model on Groq's free tier, and 404 The model `gemma2-9b-it` does not exist or you do not have access to it is what its users have seen since 8 October 2025. The retirement is a year old, but the id is still hard-coded in enough tutorials, Discord bots and classroom projects that it keeps surfacing. The replacement Groq named at the time, llama-3.1-8b-instant, has since been retired as well, so following the original notice no longer works.

Last verified November 18, 2026 against Groq deprecations, Groq model list · 5 min read

ProviderGroq
HTTP status404 Not Found
Error codemodel_not_found
MessageThe model `gemma2-9b-it` does not exist or you do not have access to it.
Announced / shut down8 August 2025 / 8 October 2025
When it happensSince 2025-10-08, on any self-serve request that names this model
Can you wait it out?No — the model was retired; the request fails until you change the model or the provider
Short answer
  • Fastest: change the model string to llama-3.1-8b-instant on Groq and raise max_tokens, because it reasons before answering. Different model, so re-check your prompts.
  • Least code change: keep gemma2-9b-it and change only the base URL; Inference APIs accepts the id and answers with Llama 3.3 70B.
  • So it does not happen again: read the model id from configuration and add a fallback for 404 and 429.

What happened

Groq's deprecations page lists the notice: announced 8 August 2025, shut down 8 October 2025, all self-serve tiers, replacement llama-3.1-8b-instant. That replacement was retired on 16 August 2026 for free and developer tiers in favour of openai/gpt-oss-20b. Gemma 7B had gone the same way in December 2024, replaced by Gemma 2 9B. Two hops in under two years is typical of Groq's small-model slot; the deprecations tracker has the chain.

First, find where the id is pinned

The id is often in more places than the one that errored: a default in a settings file, an environment variable on the server, a second call for summarisation or titles, a framework default.

Find every copy of the id
grep -rn -E "gemma2-9b-it" . --include='*.py' --include='*.ts' --include='*.js' --include='*.json' --include='*.yaml' --include='*.yml' --include='*.toml' --include='*.env*'
# and the places grep cannot see: environment variables on the server, a hosted secrets store, the provider dashboard's default model

Option 1 — switch to Groq's recommended replacement

The current end of the chain on Groq is openai/gpt-oss-20b. What is different from Gemma 2 9B:

  • It reasons before answering. Give it max_tokens of 600 or more, or send reasoning_effort: "low"; a small budget returns empty content.
  • Licensing. Gemma's terms were a reason some teams chose it; GPT-OSS is Apache 2.0, which is more permissive, so that is rarely a blocker.
  • Tool calling and JSON mode work on GPT-OSS; Gemma 2 on Groq had no tool calling, so this is a gain.
One string changes
resp = client.chat.completions.create(
    model="llama-3.1-8b-instant",   # was "gemma2-9b-it"
    max_tokens=800,                 # reasoning counts against it
    messages=[{"role": "user", "content": "Hello"}],
)

Option 2 — keep the old id, change the base URL

No pay-as-you-go host we know of serves Gemma 2 9B under this id any more. Inference APIs accepts gemma2-9b-it (and gemma-7b-it) so that a broken app answers again, routing it to Llama 3.3 70B at $0.20 / $0.50 per million tokens: a larger instruct model with no thinking phase, so prompts and max_tokens written for Gemma keep working unchanged.

Base URL and key change, model string stays
from openai import OpenAI

client = OpenAI(
    base_url="https://api.inferenceapis.com/v1",   # was https://api.groq.com/openai/v1
    api_key=os.environ["INFERENCE_API_KEY"],
)
resp = client.chat.completions.create(
    model="gemma2-9b-it",   # the retired Groq id, accepted as an alias
    messages=[{"role": "user", "content": "Hello"}],
)

Gemma was chosen for being small and cheap. If that is the point, send openai/gpt-oss-20b here instead at $0.04 / $0.19 with the larger token budget described above; the alias is for getting unblocked, not for staying. Prices and capabilities: Llama 3.3 70B here.

Option 3 — stop pinning one provider and one id

Put the model id in configuration and let a 404 or 429 fall through to a second provider. A retirement then degrades to a log line instead of an outage, and you choose when to migrate. LiteLLM's router does this in a few lines; the same shape exists in most frameworks.

LiteLLM proxy: Groq first, fall back on 404/429
# LiteLLM proxy config: Groq first, Inference APIs when Groq answers 404 or 429
model_list:
  - model_name: chat
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY
  - model_name: chat-fallback
    litellm_params:
      model: inferenceapis/openai/gpt-oss-120b
      api_key: os.environ/INFERENCEAPIS_API_KEY

router_settings:
  fallbacks: [{ "chat": ["chat-fallback"] }]

The deprecations tracker lists every Groq retirement with its date and replacement, so you can see the next one coming.

Frequently asked questions

Will Groq host Gemma 3?

Groq's current list has no Gemma model. Check the list rather than assuming; it changes every month or two.

Why does a one-year-old retirement still matter?

Because the id lives in code that is copied, not written: tutorials, starter kits and bots. Searching your dependencies for the string finds more copies than searching your own files.

Can I keep Gemma's prompts on Llama 3.3 70B?

Mostly yes. Both are instruct models without a thinking phase, so prompts, max_tokens and output parsers carry over. Llama is larger and more capable, so answers tend to be longer; set max_tokens deliberately.

Where Inference APIs fits

Option 2 above keeps the old id working with a base-URL change: Inference APIs accepts gemma2-9b-it and serves it with Llama 3.3 70B at $0.20 / $0.50 per million tokens, billed per request with no daily cap. Groq's own replacement, openai/gpt-oss-20b, is served here too at $0.04 / $0.19.

Something changed or wrong? Tell us and we will re-verify the entry.