Streamlining AI Development with LiteLLM Proxy: A Comprehensive Guide

At some point I had three side projects each talking to a different LLM provider, with API keys pasted into three .env files and three slightly different client wrappers. Swapping GPT-4o for Claude in any of them meant editing code. That is a silly amount of friction for what is, underneath, the same chat-completion call, so I put a LiteLLM proxy in front of everything and never went back.

This post documents that setup: LiteLLM proxy plus Open WebUI, PostgreSQL, and Redis, all in Docker Compose. The full configuration lives in my litellm-openwebui repo.

The short version

#

• LiteLLM exposes one OpenAI-compatible endpoint in front of OpenAI, Anthropic, and local Ollama models; your application code stops caring which provider answers.
• API keys live in one place (the proxy), not scattered across every app that calls a model.
• Fallback routing and Redis-backed caching are configuration, not application code.
• Open WebUI sits on top as a ready-made chat client, which makes the proxy easy to exercise and demo.
• The whole stack is a docker compose up away; config below.

What LiteLLM proxy actually is

#

LiteLLM is an open-source proxy for large language models. It presents a standardized API and translates behind the scenes to whichever provider you configured: OpenAI, Anthropic, or local models running through Ollama, among many others.

The practical consequence is that application code is written once, against the OpenAI SDK, and the model becomes a string in a request:

# Your code remains the same regardless of which model you're using
response = openai.ChatCompletion.create(
    model="claude-3-5-sonnet",  # Can be switched to any other model
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Explain quantum computing."}
    ]
)

The features I actually use

#

LiteLLM’s docs list a lot of capability. Four things carry the weight in my setup.

One API for every model. No separate code paths per provider, and A/B testing a new model is a one-line change in a request, not a refactor. When a new model ships, adopting it means adding an entry to the proxy config.

Central key management. Provider keys are configured once on the proxy. Rotating a key touches one file instead of every application, and usage across all apps shows up in one dashboard. Before this, my keys were scattered across projects and I genuinely could not have told you which app was spending what.

Routing and fallbacks. Fallback chains are declarative: if a provider is down or throttling, requests route to a backup without the calling app noticing. You can also alias groups of models or route by usage.

# Example of fallback configuration in LiteLLM
router_settings:
  routing_strategy: 
    - model_name: gpt-4o
      fallbacks: [claude-3-5-sonnet, llama3.3]

Logging and caching. The proxy tracks token usage and cost per model and per request, and the Redis-backed response cache means repeated queries don’t hit the provider again. Caching helps most on demo and internal-tool traffic, where the same prompts recur constantly; how much it saves depends entirely on how repetitive your traffic is, so measure your own before assuming anything.

Open WebUI as the demo client

#

LiteLLM is backend infrastructure; Open WebUI is the fastest way to see it working. It is a chat interface that speaks the OpenAI API, so pointing it at the proxy takes two environment variables. From the UI you can pick any model configured in LiteLLM, switch models mid-conversation, keep chat history across models, and upload files. It doubles as a useful smoke test: if a model works in Open WebUI, the proxy config for it is right.

Setting up the environment

#

Architecture overview

#

The setup consists of four main components:

1. LiteLLM Proxy: the core service that handles all LLM API interactions
2. PostgreSQL: configuration storage and usage tracking
3. Redis: cache for responses
4. Open WebUI: chat interface for interacting with models

Step-by-step setup

#

1. Environment configuration

#

Everything is driven by environment variables. A simplified .env:

# Database configuration
POSTGRES_USER=llmlite
POSTGRES_PASSWORD=secure_password
POSTGRES_DB=llmlite

# LiteLLM configuration
LITELLM_MASTER_KEY=sk-master-secure_key
LITELLM_ADMIN_KEY=sk-admin-secure_key
UI_USERNAME=admin
UI_PASSWORD=secure_admin_password

# API keys for providers
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key
OLLAMA_BASE_URL=http://host.docker.internal:11434

2. Model configuration

#

The entrypoint.sh script generates LiteLLM’s configuration from those variables at container start:

cat > /app/generated_config.yaml << EOF
general_settings:
  disable_auth: true
  port: 4000
  host: "0.0.0.0"
  store_model_in_db: true
  store_prompts_in_spend_logs: true

model_list:
  # OpenAI Models
  - model_name: gpt-4o
    litellm_params:
      model: openai/gpt-4o
      api_key: "${OPENAI_API_KEY}"
      
  # Anthropic Models
  - model_name: claude-3-5-sonnet
    litellm_params:
      model: anthropic/claude-3-5-sonnet-20241022
      api_key: "${ANTHROPIC_API_KEY}"
      
  # Ollama Models
  - model_name: llama3.3
    litellm_params:
      model: ollama/llama3.3
      api_base: "${OLLAMA_BASE_URL}"
EOF

This defines the available models, maps friendly names to provider-specific identifiers, and attaches the right API key to each.

3. Container orchestration

#

The Docker Compose configuration ties the services together:

services:
  litellm:
    image: ghcr.io/berriai/litellm:main-stable
    depends_on:
      - postgres
      - redis
    environment:
      # Configuration variables
    volumes:
      - ./entrypoint.sh:/app/entrypoint.sh
    entrypoint: ["/bin/bash", "/app/entrypoint.sh"]
    
  postgres:
    image: postgres:16-alpine
    environment:
      # Database configuration
    volumes:
      - postgres_data:/var/lib/postgresql/data
      
  redis:
    image: redis:alpine
    volumes:
      - redis_data:/data
      
  open_webui:
    image: ghcr.io/open-webui/open-webui:latest
    environment:
      - OPENAI_API_BASE_URL=${LITELLM_API_BASE:-http://litellm:4000/v1}
      - OPENAI_API_KEY=${LITELLM_MASTER_KEY:-sk-master-123456789}
    volumes:
      - open_webui_data:/app/backend/data

4. Initial admin setup

#

A PowerShell script (fix.ps1) creates the initial admin user for LiteLLM:

$body = @{
    user_id         = "default_user_id"
    team_id         = "default_team_id"
    user_role       = "proxy_admin"
    auto_create_key = $true
} | ConvertTo-Json

$headers = @{
    Authorization = "Bearer sk-master-123456789"
}

Invoke-RestMethod -Method POST -Uri "http://localhost:4000/user/new" -Headers $headers -Body $body -ContentType "application/json"

Without this step you can’t get into the LiteLLM dashboard, which is easy to forget on a fresh install.

Configuration beyond the basics

#

For heavier requirements, three options are worth knowing about.

Custom routing rules

#

router_settings:
  routing_strategy: "usage_based"
  model_group_alias:
    - group_name: "gpt-4-class"
      models: ["gpt-4o", "claude-3-5-sonnet"]

Rate limiting

#

litellm_settings:
  rate_limit_type: "token" # or "request"
  rate_limits:
    - model: ["gpt-4o"]
      tpm: 1000000  # tokens per minute
      rpm: 6000     # requests per minute

Custom deployments

#

model_list:
  - model_name: "company-fine-tuned-llama"
    litellm_params:
      model: "ollama/company-llama"
      api_base: "http://internal-ollama-server:11434"

Rolling it out without breaking things

#

Don’t wire every application through the proxy on day one. The order that has worked for me:

1. Phase 1: deploy LiteLLM with basic configuration, routing to your existing providers
2. Phase 2: point application code at the LiteLLM endpoint
3. Phase 3: add caching and fallbacks
4. Phase 4: tune routing and model selection based on observed usage

Integration patterns

#

Two patterns have held up well in application code.

Service layer. Keep all LiteLLM interaction behind one class, so retries, model selection, and error handling live in a single place:

// AI service abstraction
class AIService {
  constructor(endpoint, apiKey) {
    this.client = new OpenAI({
      baseURL: endpoint,
      apiKey: apiKey
    });
  }
  
  async generateResponse(prompt, modelPreference = null) {
    // Handle model selection, error handling, retries, etc.
  }
}

Feature flags for model access. Useful when premium models should only serve certain tiers:

def get_model_for_feature(feature_name, user_tier):
    if feature_flags.is_enabled("use_premium_models", user_tier):
        return "gpt-4o"
    return "llama3.3"  # fallback to local model

What to measure

#

If you can’t say whether the proxy helped, you skipped this part. Track:

1. Cost: AI spend before and after, per model
2. API reliability: uptime and error rates through the proxy
3. Development velocity: time spent on AI integration tasks
4. Response latency: end-to-end, since the proxy adds a hop
5. Model experimentation: how often the team actually tries new models now that it’s cheap to do

Monitoring and troubleshooting

#

Health checks

#

LiteLLM provides built-in health check endpoints:

# Check LiteLLM proxy health
curl http://localhost:4000/health

# Check specific model availability
curl http://localhost:4000/health/readiness

Common issues

#

1. Model not available

#

# Error: Model not found
# Solution: Check model configuration in config.yaml
docker logs litellm-proxy-container

2. API key issues

#

# Error: Authentication failed
# Solution: Verify API keys in environment variables
echo $OPENAI_API_KEY | cut -c1-10  # Check first 10 chars

3. Rate limiting

#

# Error: Rate limit exceeded
# Solution: Configure rate limiting in LiteLLM config
# Or implement retry logic in your application

Debugging configuration

#

Turn on verbose logging when something misbehaves:

general_settings:
  set_verbose: true
  log_level: "DEBUG"

The bottom line

#

• A proxy layer between apps and LLM providers is worth having even for side projects; the payoff grows with every additional app and model.
• Write application code against one OpenAI-compatible endpoint and treat model choice as configuration. Vendor lock-in stops being a code problem.
• Keys, routing, fallbacks, caching, and spend tracking belong in the proxy, not reimplemented per application.
• Latency and cache hit rates are yours to measure; don’t trust anyone’s generic savings claim, including one from a blog post.

Where I’d start

#

Run Phase 1 this weekend: clone the litellm-openwebui setup, put your existing provider keys in .env, and docker compose up. Point Open WebUI at it, confirm your usual models answer, then migrate one application to the proxy endpoint before touching fallbacks or caching. The rest of the rollout phases can wait until that one app has run quietly for a week.