Streamlining AI Development with LiteLLM Proxy: A Comprehensive Guide

At some point I had three side projects each talking to a different LLM provider, with API keys pasted into three .env files and three slightly different client wrappers. Swapping GPT-4o for Claude in any of them meant editing code. That is a silly amount of friction for what is, underneath, the same chat-completion call, so I put a LiteLLM proxy in front of everything and never went back.

This post documents that setup: LiteLLM proxy plus Open WebUI, PostgreSQL, and Redis, all in Docker Compose. The full configuration lives in my litellm-openwebui repo.

What LiteLLM proxy actually is

LiteLLM is an open-source proxy for large language models. It presents a standardized API and translates behind the scenes to whichever provider you configured: OpenAI, Anthropic, or local models running through Ollama, among many others.

The practical consequence is that application code is written once, against the OpenAI SDK, and the model becomes a string in a request:

# Your code remains the same regardless of which model you're using
response = openai.ChatCompletion.create(
    model="claude-3-5-sonnet",  # Can be switched to any other model
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing."}
    ]
)

The features I actually use

LiteLLM’s docs list a lot of capability. Four things carry the weight in my setup.

One API for every model. No separate code paths per provider, and A/B testing a new model is a one-line change in a request, not a refactor. When a new model ships, adopting it means adding an entry to the proxy config.

Central key management. Provider keys are configured once on the proxy. Rotating a key touches one file instead of every application, and usage across all apps shows up in one dashboard. Before this, my keys were scattered across projects and I genuinely could not have told you which app was spending what.

Routing and fallbacks. Fallback chains are declarative: if a provider is down or throttling, requests route to a backup without the calling app noticing. You can also alias groups of models or route by usage.

# Example of fallback configuration in LiteLLM
router_settings:
  routing_strategy:
    - model_name: gpt-4o
      fallbacks: [claude-3-5-sonnet, llama3.3]

Logging and caching. The proxy tracks token usage and cost per model and per request, and the Redis-backed response cache means repeated queries don’t hit the provider again. Caching helps most on demo and internal-tool traffic, where the same prompts recur constantly; how much it saves depends entirely on how repetitive your traffic is, so measure your own before assuming anything.

Open WebUI as the demo client

LiteLLM is backend infrastructure; Open WebUI is the fastest way to see it working. It is a chat interface that speaks the OpenAI API, so pointing it at the proxy takes two environment variables. From the UI you can pick any model configured in LiteLLM, switch models mid-conversation, keep chat history across models, and upload files. It doubles as a useful smoke test: if a model works in Open WebUI, the proxy config for it is right.

Setting up the environment

Architecture overview

The setup consists of four main components:

1. LiteLLM Proxy: the core service that handles all LLM API interactions
2. PostgreSQL: configuration storage and usage tracking
3. Redis: cache for responses
4. Open WebUI: chat interface for interacting with models

Step-by-step setup

1. Environment configuration

Everything is driven by environment variables. A simplified .env:

# Database configuration
POSTGRES_USER=llmlite
POSTGRES_PASSWORD=secure_password
POSTGRES_DB=llmlite

# LiteLLM configuration
LITELLM_MASTER_KEY=sk-master-secure_key
LITELLM_ADMIN_KEY=sk-admin-secure_key
UI_USERNAME=admin
UI_PASSWORD=secure_admin_password

# API keys for providers
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
OLLAMA_BASE_URL=http://host.docker.internal:11434

2. Model configuration

The entrypoint.sh script generates LiteLLM’s configuration from those variables at container start:

cat > /app/generated_config.yaml <\< EOF
general_settings:
  disable_auth: true
  port: 4000
  host: "0.0.0.0"
  store_model_in_db: true
  store_prompts_in_spend_logs: true

model_list:
  # OpenAI Models
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: "${OPENAI_API_KEY}"

  # Anthropic Models
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: "${ANTHROPIC_API_KEY}"

  # Ollama Models
  - model_name: llama3.3
    litellm_params:
      model: ollama/llama3.3
      api_base: "${OLLAMA_BASE_URL}"
EOF

This defines the available models, maps friendly names to provider-specific identifiers, and attaches the right API key to each.

3. Container orchestration

The Docker Compose configuration ties the services together:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    depends_on:
      - postgres
      - redis
    environment:
      # Configuration variables
    volumes:
      - ./entrypoint.sh:/app/entrypoint.sh
    entrypoint: ["/bin/bash", "/app/entrypoint.sh"]

  postgres:
    image: postgres:16-alpine
    environment:
      # Database configuration
    volumes:
      - postgres_data:/var/lib/postgresql/data

  redis:
    image: redis:alpine
    volumes:
      - redis_data:/data

  open_webui:
    image: ghcr.io/open-webui/open-webui:latest
    environment:
      - OPENAI_API_BASE_URL=${LITELLM_API_BASE:-http://litellm:4000/v1}
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY:-sk-master-123456789}
    volumes:
      - open_webui_data:/app/backend/data

4. Initial admin setup

A PowerShell script (fix.ps1) creates the initial admin user for LiteLLM:

$body = @{
    user_id         = "default_user_id"
    team_id         = "default_team_id"
    user_role       = "proxy_admin"
    auto_create_key = $true
} | ConvertTo-Json

$headers = @{
    Authorization = "Bearer sk-master-123456789"
}

Invoke-RestMethod -Method POST -Uri "http://localhost:4000/user/new" -Headers $headers -Body $body -ContentType "application/json"

Without this step you can’t get into the LiteLLM dashboard, which is easy to forget on a fresh install.

Configuration beyond the basics

For heavier requirements, three options are worth knowing about.

Custom routing rules

router_settings:
  routing_strategy: "usage_based"
  model_group_alias:
    - group_name: "gpt-4-class"
      models: ["gpt-4o", "claude-3-5-sonnet"]

Rate limiting

litellm_settings:
  rate_limit_type: "token" # or "request"
  rate_limits:
    - model: ["gpt-4o"]
      tpm: 1000000  # tokens per minute
      rpm: 6000     # requests per minute

Custom deployments

model_list:
  - model_name: "company-fine-tuned-llama"
    litellm_params:
      model: "ollama/company-llama"
      api_base: "http://internal-ollama-server:11434"

Rolling it out without breaking things

Don’t wire every application through the proxy on day one. The order that has worked for me:

1. Phase 1: deploy LiteLLM with basic configuration, routing to your existing providers
2. Phase 2: point application code at the LiteLLM endpoint
3. Phase 3: add caching and fallbacks
4. Phase 4: tune routing and model selection based on observed usage

Integration patterns

Two patterns have held up well in application code.

Service layer. Keep all LiteLLM interaction behind one class, so retries, model selection, and error handling live in a single place:

// AI service abstraction
class AIService {
  constructor(endpoint, apiKey) {
    this.client = new OpenAI({
      baseURL: endpoint,
      apiKey: apiKey
    });
  }

  async generateResponse(prompt, modelPreference = null) {
    // Handle model selection, error handling, retries, etc.
  }
}

Feature flags for model access. Useful when premium models should only serve certain tiers:

def get_model_for_feature(feature_name, user_tier):
    if feature_flags.is_enabled("use_premium_models", user_tier):
        return "gpt-4o"
    return "llama3.3"  # fallback to local model

What to measure

If you can’t say whether the proxy helped, you skipped this part. Track:

1. Cost: AI spend before and after, per model
2. API reliability: uptime and error rates through the proxy
3. Development velocity: time spent on AI integration tasks
4. Response latency: end-to-end, since the proxy adds a hop
5. Model experimentation: how often the team actually tries new models now that it’s cheap to do

Monitoring and troubleshooting

Health checks

LiteLLM provides built-in health check endpoints:

# Check LiteLLM proxy health
curl http://localhost:4000/health

# Check specific model availability
curl http://localhost:4000/health/readiness

Common issues

1. Model not available

# Error: Model not found
# Solution: Check model configuration in config.yaml
docker logs litellm-proxy-container

2. API key issues

# Error: Authentication failed
# Solution: Verify API keys in environment variables
echo $OPENAI_API_KEY | cut -c1-10  # Check first 10 chars

3. Rate limiting

# Error: Rate limit exceeded
# Solution: Configure rate limiting in LiteLLM config
# Or implement retry logic in your application

Debugging configuration

Turn on verbose logging when something misbehaves:

general_settings:
  set_verbose: true
  log_level: "DEBUG"

Where I’d start

Run Phase 1 this weekend: clone the litellm-openwebui setup, put your existing provider keys in .env, and docker compose up. Point Open WebUI at it, confirm your usual models answer, then migrate one application to the proxy endpoint before touching fallbacks or caching. The rest of the rollout phases can wait until that one app has run quietly for a week.