Use Inference APIs with LiteLLM

Use the openai/ prefix with api_base to route any LiteLLM call — SDK or proxy — to Inference APIs, including as a fallback provider.

Settings

Setting	Value
Base URL	`https://api.inferenceapis.com/v1` (also `/openai/v1`)
API key	From API Keys; send as `Authorization: Bearer …`
Chat model ids	`openai/gpt-oss-120b`, `deepseek-ai/DeepSeek-V4-Flash`, `zai-org/GLM-5.3-Flash`, `meta-llama/Llama-3.3-70B-Instruct-Turbo` — all models
Audio model ids	`openai/whisper-large-v3` (transcription), `hexgrad/Kokoro-82M` (speech)

Which model

As a fallback for Groq, use the same id on both sides: openai/gpt-oss-120b. Prices and measured speed: chat models.

Configuration

Python SDK

import os, litellm

resp = litellm.completion(
    model="openai/openai/gpt-oss-120b",        # openai/ prefix = OpenAI-compatible endpoint
    api_base="https://api.inferenceapis.com/v1",
    api_key=os.environ["INFERENCE_API_KEY"],
    messages=[{"role": "user", "content": "Hello"}],
)
print(resp.choices[0].message.content)

Proxy config.yaml with fallback

model_list:
  - model_name: chat
    litellm_params:
      model: groq/openai/gpt-oss-120b
      api_key: os.environ/GROQ_API_KEY
  - model_name: chat-fallback
    litellm_params:
      model: openai/openai/gpt-oss-120b
      api_base: https://api.inferenceapis.com/v1
      api_key: os.environ/INFERENCE_API_KEY

router_settings:
  fallbacks: [{ "chat": ["chat-fallback"] }]

Verify

Smoke test

litellm --model openai/openai/gpt-oss-120b --api_base https://api.inferenceapis.com/v1 --api_key $INFERENCE_API_KEY --test

How this guide was checked

Run end to end litellm 1.101.0: the SDK call below and a call using the Groq-style alias openai/llama-3.3-70b-versatile ran against the live API on 2026-09-17. The proxy config is the documented LiteLLM format; we did not run a proxy.

Gotchas

The double prefix is intentional: the first openai/ tells LiteLLM the protocol, the rest is our model id.
Set drop_params: true if a caller sends parameters a model does not support.

If something fails

401 — key missing or wrong · 402 insufficient_balance — add credits on Billing · 404 model_not_found — check the id against the model list (aliases such as gpt-oss-120b work too) · 503 model_unavailable — the model is being enabled. Full details in the API docs.