Switching OpenAI-compatible providers

Most hosted LLM providers expose the OpenAI chat-completions wire format. That makes moving between them a configuration change rather than a code change — as long as three things line up. This page shows the change in the common SDKs and what to check.

The change

Two values move: the base URL and the API key. The request body, the SDK, and usually the model id stay the same.

Python · openai SDK

from openai import OpenAI

client = OpenAI(
    base_url="https://api.inferenceapis.com/v1",   # was e.g. https://api.groq.com/openai/v1
    api_key=os.environ["INFERENCEAPIS_KEY"],
)
resp = client.chat.completions.create(
    model="llama-3.3-70b-versatile",
    messages=[{"role": "user", "content": "hello"}],
    stream=True,
)

Node.js · openai SDK

import OpenAI from "openai";

const client = new OpenAI({
  baseURL: "https://api.inferenceapis.com/v1",
  apiKey: process.env.INFERENCEAPIS_KEY,
});
const resp = await client.chat.completions.create({
  model: "llama-3.3-70b-versatile",
  messages: [{ role: "user", content: "hello" }],
});

LangChain · ChatOpenAI

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="llama-3.3-70b-versatile",
    base_url="https://api.inferenceapis.com/v1",   # older versions: openai_api_base=
    api_key=os.environ["INFERENCEAPIS_KEY"],
)

LiteLLM · any OpenAI-compatible endpoint

import litellm

resp = litellm.completion(
    model="openai/llama-3.3-70b-versatile",       # "openai/" prefix = OpenAI-compatible
    api_base="https://api.inferenceapis.com/v1",
    api_key=os.environ["INFERENCEAPIS_KEY"],
    messages=[{"role": "user", "content": "hello"}],
)

Many tools (agent CLIs, gateways, evaluation harnesses) read the standard environment variables, so you may not need to touch code at all:

Environment variables · most tools honor these

export OPENAI_BASE_URL="https://api.inferenceapis.com/v1"
export OPENAI_API_KEY="$INFERENCEAPIS_KEY"

Three things to check first

Check	Why it matters	How to confirm
Model id	Providers name the same weights differently (`llama-3.3-70b-versatile` vs `meta-llama/Llama-3.3-70B-Instruct-Turbo`). A mismatch returns `404 model_not_found`.	Call `GET /v1/models` on the new endpoint and look for the id you send. Inference APIs accepts Groq-style ids directly.
Base path	Some providers mount the API at `/v1`, others at `/openai/v1`. Getting it wrong is a 404 on every request.	Read the provider's docs; Inference APIs accepts both paths.
Feature parity	Streaming, tool calling, JSON mode and vision inputs are not universal. Code that relies on one will fail in a new way, not a 404.	Send one request per feature you use before switching traffic.

Keeping both: primary + fallback

You do not have to choose. A common pattern is to keep the current provider as primary and route to a second endpoint only on a 429 or a model-not-found error. The failover example in the 429 entry shows this in Python, Node and LiteLLM.

Where Inference APIs fits

Inference APIs is one such OpenAI-compatible endpoint: https://api.inferenceapis.com/v1, pay-per-request, with common Groq and OpenAI model ids accepted as aliases. See the API docs and the model list.