Model providers
How to point the stack at each model provider, what has actually been exercised, and what has not. The variable reference lives in configuration.md; this page is the practical side: copy a block, know what breaks.
Everything below is selected by LLM_PROVIDER in .env. Embeddings are a separate setting group and are covered on their own.
Verified versus untested
Read this table before choosing a provider. “Verified” means a test in this repo exercises it, and the second column says how. Nothing in CI calls a live model; the only live calls ever made were recording sessions against Azure OpenAI.
| Provider | How to select | Verified how | Not verified |
|---|---|---|---|
replay | LLM_PROVIDER=replay | Every CI eval and tutorial run; 102 fixtures recorded on gpt-4.1-2025-04-14 | Anything about a model other than the one recorded |
azure | LLM_PROVIDER=azure | Fixtures were recorded through it | Current model availability for new subscriptions (see caveats) |
openai | LLM_PROVIDER=openai | Stubbed-transport tests of the client wiring | No recorded run in this repo used api.openai.com |
anthropic | LLM_PROVIDER=anthropic | Stubbed httpx2 transport (Python) and stub handler (.NET), tests/test_anthropic_provider.py | Never run against the live API. Tool-choice quality, prompt caching, real error shapes, streaming edge cases |
| OpenAI-compatible endpoint (Ollama, vLLM, LM Studio, OpenRouter, LiteLLM) | LLM_PROVIDER=openai + LLM_BASE_URL | Ollama and the 2-tool loop on a few local models were tried by hand (see the note in .env.example); the wiring has stubbed tests | vLLM, OpenRouter and LiteLLM have not been run by this repo at all |
| Gemini compatible API | openai + Gemini’s LLM_BASE_URL + LLM_API=chat_completions | tests/test_gemini_compat_endpoint.py, stubbed | No live call to Google |
replay (no key, no network)
LLM_PROVIDER=replay
Serves recorded fixtures keyed by the exact request (messages, tool schemas, instructions). It is what CI, the keyless demo and the tutorials use. A request with no fixture fails with a replay miss, not a model answer, so it is the wrong choice for anything you have not recorded. The key ignores the model name, so fixtures replay under any provider setting. See agents/python/evals/fixtures/README.md for recording.
Azure OpenAI
LLM_PROVIDER=azure
AZURE_OPENAI_ENDPOINT=https://<resource>.openai.azure.com/
AZURE_OPENAI_KEY=...
AZURE_OPENAI_DEPLOYMENT=<your deployment name>
AZURE_OPENAI_API_VERSION=2025-03-01-preview
Azure is always Chat Completions on the Python stack. The deployment name is what you called the deployment, not necessarily the model id. For production, prefer managed identity over a key; the Azure deployment path in azure-deployment.md does that.
OpenAI
LLM_PROVIDER=openai
OPENAI_API_KEY=sk-...
LLM_MODEL=gpt-4.1
The Python stack uses the Responses API here (LLM_API=responses, the default). The .NET stack always uses Chat Completions.
Anthropic (Claude)
LLM_PROVIDER=anthropic
ANTHROPIC_API_KEY=sk-ant-...
ANTHROPIC_MODEL=claude-sonnet-5-5
EMBEDDING_PROVIDER=openai # Claude has no embeddings API
LLM_MODEL is ignored for this provider. Claude is wired on both stacks and covered by stubbed-transport tests only. Cost figures are meaningless until prices are added to shared/cost.py: Claude models fall back to the default entry. Full table in configuration.md.
Any OpenAI-compatible endpoint
One mechanism covers all of these: LLM_PROVIDER=openai, LLM_BASE_URL pointing at the server, a non-empty OPENAI_API_KEY (many servers ignore its value), and LLM_API=chat_completions when the server has no /responses route. A 404 on /responses is the symptom of a missing LLM_API=chat_completions.
| Server | LLM_BASE_URL | Notes |
|---|---|---|
| Ollama | http://localhost:11434/v1 | Start with OLLAMA_CONTEXT_LENGTH=64000; the 4K default silently truncates agent loops. Inside a container use http://host.docker.internal:11434/v1 |
| LM Studio | http://localhost:1234/v1 | Load a model in the server tab first |
| vLLM | http://<host>:8000/v1 | Start it with tool calling enabled for your model (its tool-call parser flag is model specific; check vLLM’s docs) |
| OpenRouter | https://openrouter.ai/api/v1 | Key in OPENAI_API_KEY; the model id is OpenRouter’s vendor/model form |
| LiteLLM proxy | http://localhost:4000/v1 | See below |
| Gemini | https://generativelanguage.googleapis.com/v1beta/openai/ | Gemini key in OPENAI_API_KEY; set EMBEDDING_* explicitly |
Choose a model that does tool calling reliably. Every specialist reads the database through tools; a model with weak function calling does not error, it stops calling tools and answers from memory. If answers stop mentioning real products or orders, check that first.
LiteLLM
LiteLLM is a proxy that speaks the OpenAI API in front of many providers, which makes it the way to reach a provider this repo has no code for. This repo ships no LiteLLM compose profile; run it however you run other services and point the stack at it:
LLM_PROVIDER=openai
LLM_BASE_URL=http://localhost:4000/v1 # http://host.docker.internal:4000/v1 from a container
LLM_API=chat_completions
OPENAI_API_KEY=<the proxy's key, or any non-empty string if it has none>
LLM_MODEL=<a model_name from the proxy's config>
Untested here. Two things to check yourself: that the routed model supports tool calls, and the cache-token caveat below.
Embeddings
Embeddings are configured separately from chat, and defaulting to the chat provider is often wrong:
EMBEDDING_PROVIDER=openai_compatible # openai | azure | openai_compatible | replay
EMBEDDING_BASE_URL=http://localhost:11434/v1
EMBEDDING_MODEL=nomic-embed-text
EMBEDDING_DIM=768
- Anthropic has no embeddings API. With no embedding provider configured, the stack starts and semantic search falls back to its full-text arm.
- A Gemini or other compatible chat base URL is not an embeddings endpoint. Left empty,
EMBEDDING_PROVIDERfollows the chat provider and sendstext-embedding-3-smallrequests there. - The dimension is a database property (
vector(1536)in the baseline). Changing it is an explicit step:uv run --project agents/python python scripts/reset_embeddings.py --dim N, which discards stored vectors. Details in configuration.md.
Known caveats
- Cache-token fields can be remapped by a proxy. Usage and cost accounting reads the cached input count from the provider’s response (
shared/run_usage.pyaccepts MAF’scache_read_input_token_countand the wire’scached_input_token_count). A proxy such as LiteLLM or OpenRouter may rename or drop those fields when it translates between providers, so cached tokens can read as zero and cost can look higher than the bill. The mechanism for a cached price exists; no cached-input rate has been entered (E-2d). - Anthropic has never run live. Treat it as a configured, stub-tested option, not a supported one, until a keyed smoke run is recorded.
- Model availability moves. The default
gpt-4.1and its Azure deployment name are what the fixtures were recorded on. Per the maintainer, gpt-4.1 is closed to new Azure customers, and the gpt-5.x family rejects atemperatureparameter. Neither claim was re-checked against Azure documentation for this page. The stack sendsLLM_TEMPERATURE(default 0.2) on every run, so a model that rejects it fails until a change that letsLLM_TEMPERATUREbe left empty (not sending the parameter) lands; that change is in flight in another pull request. Until then, do not point the stack at a model that rejects temperature. - Per-agent models (
model:block in the prompt YAML) need that provider’s own credentials, and the globalLLM_BASE_URLis not inherited by an agent that switches provider. See configuration.md. - Replay scores do not transfer. The eval scorecard (
docs/eval-scorecard.md) describes a recorded gpt-4.1 session. Switching provider changes behaviour; it does not change those numbers. Measuring another model needs a fresh, keyed recording, which is human-dispatch only.
Gotchas checklist
- 404 on
/responses: setLLM_API=chat_completions. LLM_MODELset but ignored: you are onanthropic, which readsANTHROPIC_MODEL.- Semantic search empty after switching provider: set
EMBEDDING_*explicitly. - Answers with no real data: the model is not calling tools.
- Cost looks wrong on a proxy or on Claude: unpriced model or remapped cache fields.
Source: docs/model-providers.md — this page is generated from the repository.