Embeddings and rerank API

Quickstart

import os, requests
from openai import OpenAI

BASE = "https://api.inferenceapis.com/v1"
client = OpenAI(base_url=BASE, api_key=os.environ["INFERENCE_API_KEY"])

docs = ["Paris is the capital of France.", "Bananas are yellow.", "Berlin is in Germany."]
vectors = [d.embedding for d in client.embeddings.create(model="BAAI/bge-m3", input=docs).data]
query = client.embeddings.create(model="BAAI/bge-m3", input="What is the capital of France?").data[0].embedding

# ...nearest-neighbour search in your vector store returns candidates; then rerank them:
ranked = requests.post(f"{BASE}/rerank",
    headers={"Authorization": f"Bearer {os.environ['INFERENCE_API_KEY']}"},
    json={"model": "Qwen/Qwen3-Reranker-8B", "query": "What is the capital of France?", "documents": docs, "top_n": 2},
).json()["results"]
print(ranked)   # [{'index': 0, 'relevance_score': 0.97...}, ...]

Parameters and response shapes: embeddings and rerank in the API documentation, and the model pages above.

What it costs

Embedding a corpus is cheap everywhere. What stops indexing jobs is not the price but the quota: free tiers cap embedding requests per day, and a re-index of a real corpus runs into that ceiling within minutes. We do not meter requests or tokens per minute or per day.

Measured throughput

We ran a bulk-indexing load through this endpoint on September 17, 2026: texts of about 120 tokens, 256 per request, sent by parallel workers.

At the BGE-M3 rate a corpus of one million 500-token documents indexes in a little over two hours for about $10. Two honest caveats. In one earlier run the backend was briefly saturated and refused about half of a burst with 429; the gateway now retries those itself, up to three times with backoff, before you ever see one, and if one still reaches you it is safe to retry. And Qwen3 Embedding 8B takes about ten seconds per request whatever the batch size, so send large batches and run them in parallel. The largest batch the backend accepts is 1,024 texts.

Pricing versus alternatives

Per one million input tokens; list prices as of September 2026.

The price is not the argument; OpenAI's small model costs the same. The argument is ownership. A closed embedding model can be deprecated, and when it is, every vector you stored has to be rebuilt on the vendor's schedule. With open weights the model that made your vectors can always be run somewhere.

Frequently asked questions

Is it compatible with the OpenAI embeddings API?

Yes. POST /v1/embeddings takes model, input as a string or an array, and optional dimensions and encoding_format, and returns the same response shape. We ran it with the OpenAI Python SDK and with LangChain's OpenAIEmbeddings. In LangChain set check_embedding_ctx_length=False, or it sends OpenAI token ids that open models do not accept.

Can I keep my existing OpenAI vectors?

No, and no provider can offer that. Vectors from different models live in different spaces, so text-embedding-3-small is deliberately not aliased here. Moving to an open-weight model means re-embedding once; after that the model can never be withdrawn from you, because the weights are public.

Will a bulk indexing job hit a rate limit?

We do not meter requests or tokens per minute or per day. Send up to 1,024 texts per request and run 4 to 8 requests in parallel; in our load test that sustained 2 to 4 million tokens per minute on BGE-M3. When the backend is momentarily saturated the gateway retries for you, and a 429 that still reaches you means back off for a few seconds, not that a quota ran out.

Which model should I choose?

BGE-M3 is the safe default: multilingual, 1,024 dimensions, 8K-token inputs, widely used. Qwen3 Embedding 8B is larger, accepts 32K-token inputs and lets you choose the vector length with dimensions, which helps when storage is the constraint. Both cost the same.

What is reranking for?

An embedding search finds passages that are near the query; a reranker reads the query and each passage together and scores whether the passage answers it. Fetch 20 to 100 candidates with embeddings, rerank, and put the best few in the prompt. It is the cheapest accuracy improvement most RAG pipelines can make.

Where do the models run, and is my text stored?

On DeepInfra's GPUs in the United States. DeepInfra states that inputs and outputs are not stored and not used for training, and this gateway keeps only token counts. Details on the trust page.