LLM Gateway

Model	Requests	Tokens	Cost
Loading...

Model ID	Provider	Type
Loading...

Model	Requests	Tokens	Cost	Period
Loading...

Key Prefix	Name	Team	Tier	RPM Limit	Status	Actions
Loading...

ID	Status	Total	Progress	Created	Actions
Loading...

ID	Name	Org ID	Max Budget	Created	Actions
Loading...

ID	User ID	Email	Role	Team	Org	Max Budget	Actions
Loading...

ID	Name	Admin	Max Budget	Created	Actions
Loading...

Scope	PII Action	Max Input Tokens	Max Output Tokens	Blocked Keywords	Enabled	Actions
Loading...

ID	Action	Entity Type	Entity ID	Actor	Details	Timestamp
Loading...

What is LLM Gateway?

LLM Gateway is a unified API proxy for large language models. It sits between your application and LLM providers (OpenAI, Anthropic, Groq, DeepSeek, Gemini, Ollama), giving you a single OpenAI-compatible endpoint with built-in cost tracking, caching, rate limiting, guardrails, and multi-provider routing.

Think of it as a smart reverse proxy for AI. Your application sends requests to the gateway using the standard OpenAI API format. The gateway handles provider translation, retries, failover, caching, and observability — so you don't have to.

Key insight: You change one line — the base_url — in any OpenAI-compatible client library and instantly gain access to all providers, caching, budgets, and routing. No code changes beyond that.

Architecture at a glance

  Your App                    LLM Gateway                        Providers
 +---------+    OpenAI API    +-------------------+    Native    +-----------+
 |  Python |  ------------->  | Auth & Rate Limit |  -------->  |  OpenAI   |
 |  Node   |                  | Cache Check       |             |  Claude   |
 |  Go     |                  | Guardrails        |             |  Groq     |
 |  curl   |                  | Router + Retry    |             |  DeepSeek |
 +---------+  <-------------  | Cost Tracking     |  <--------  |  Gemini   |
               cached/fresh   | Budget Check      |             |  Ollama   |
                              +-------------------+             +-----------+
                                     |
                             +-------+-------+
                             |  PostgreSQL   |  Redis  |
                             |  (budgets,    | (cache, |
                             |   keys, audit)|  RPM)   |
                             +---------------+---------+

Quickstart

Get the gateway running and make your first request in under 5 minutes.

Prerequisites: Go 1.25+, PostgreSQL, Redis. Provider API keys (at least one of OpenAI, Anthropic, etc.).

Step 1: Configure

Create a configs/.env file with your provider keys and a master key for admin access:

configs/.env

APP_NAME=llm-gateway
HTTP_PORT=9000

# Provider API keys (set at least one)
OPENAI_API_KEY=sk-your-openai-key
ANTHROPIC_API_KEY=sk-ant-your-anthropic-key

# Gateway auth
GATEWAY_API_KEYS=sk-gateway-key-1,sk-gateway-key-2
GATEWAY_MASTER_KEY=sk-master-admin-2026

# Infrastructure
DB_HOST=localhost
DB_PORT=5432
DB_USER=postgres
DB_PASSWORD=postgres
DB_NAME=llmgw
DB_DIALECT=postgres
REDIS_HOST=localhost
REDIS_PORT=6379

# Cache responses for 5 minutes
CACHE_TTL_SECONDS=300

Step 2: Start

Terminal

# Start dependencies
docker compose up -d postgres redis

# Run the gateway (migrations auto-apply on first start)
go run .

Step 3: Make your first request

curl

curl -X POST http://localhost:9000/v1/chat/completions \
  -H "Authorization: Bearer sk-gateway-key-1" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Step 4: Try another provider

Same endpoint, same format — just change the model prefix:

Switch to Claude

curl -X POST http://localhost:9000/v1/chat/completions \
  -H "Authorization: Bearer sk-gateway-key-1" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "anthropic/claude-sonnet-4-20250514",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

Model naming: All models use provider/model-name format. The gateway translates the request to each provider's native API automatically. Example: openai/gpt-4o, anthropic/claude-sonnet-4-20250514, groq/llama-3.3-70b-versatile, gemini/gemini-2.0-flash, ollama/llama3.

Use Cases

1. Multi-Provider Failover for Production Apps

If your production app calls Claude and Anthropic has an outage, requests automatically fail over to OpenAI (or any fallback chain you define). No code changes, no downtime.

configs/.env

# Primary: Anthropic Claude. Fallback: OpenAI, then local Ollama.
FALLBACK_CHAIN=anthropic,openai,ollama

2. Cost Control for Team API Access

Give each developer or team their own API key with a monthly spend budget. When the budget runs out, requests are rejected — not your credit card.

curl — create a $50 budget key

curl -X POST http://localhost:9000/key/generate \
  -H "Authorization: Bearer sk-master-admin-2026" \
  -d '{
    "name": "intern-project",
    "team_id": "1",
    "max_budget": 50.00,
    "rate_limit_rpm": 30,
    "allowed_models": ["openai/gpt-4o-mini", "groq/*"]
  }'

The intern can only use cheap models, at max 30 RPM, capped at $50 total spend. You can monitor their usage on the Spend dashboard.

3. Saving Money on Anthropic/Claude API Calls

The gateway's Redis response cache saves you real money. If the same question is asked twice within the TTL window (default 5 minutes), the second call is served from cache — zero API tokens consumed.

Example: A customer support chatbot gets the same FAQ questions repeatedly. With caching enabled, duplicate questions cost $0.00 after the first call. At $3/$15 per million tokens for Claude Sonnet 4, caching 30% of requests saves ~30% on your Claude bill.

4. PII Protection Before Data Reaches the LLM

Guardrails automatically detect and block (or redact) personal data before it ever leaves your network. HIPAA, GDPR, SOC2 compliance teams love this.

configs/.env

GUARDRAIL_ENABLED=true
GUARDRAIL_PII_ACTION=redact    # emails, SSNs, phone numbers, credit cards replaced
GUARDRAIL_BLOCKED_KEYWORDS=internal,confidential,secret-project-name

5. A/B Testing Models

Use different routing strategies or virtual keys to send traffic to different models. Compare quality, speed, and cost from the Spend dashboard.

Two keys, two models, one dashboard

# Key for variant A (Claude)
curl -X POST http://localhost:9000/key/generate \
  -d '{"name": "ab-variant-a", "allowed_models": ["anthropic/claude-sonnet-4-20250514"]}'

# Key for variant B (GPT-4o)
curl -X POST http://localhost:9000/key/generate \
  -d '{"name": "ab-variant-b", "allowed_models": ["openai/gpt-4o"]}'

# Compare results: GET /spend/report?group_by=key

6. Centralized AI for Enterprise (Multi-Tenancy)

Run one gateway for your entire company. Each business unit gets an Organization, each project gets a Team, each developer gets a User — all with separate budgets, rate limits, and audit trails.

Hierarchy

Organization: "Acme Corp" ($10,000/month)
  +-- Team: "Engineering" ($5,000/month)
  |     +-- User: "alice" ($500/month) --> Key: sk-alice-key
  |     +-- User: "bob"   ($500/month) --> Key: sk-bob-key
  +-- Team: "Marketing" ($2,000/month)
        +-- User: "carol" ($200/month) --> Key: sk-carol-key

7. Batch Processing for Data Pipelines

Need to classify 10,000 documents? Submit them as a batch. The gateway processes them through a worker pool (configurable concurrency) and stores results in PostgreSQL. Poll for completion, then fetch results.

8. Drop-in OpenAI Replacement with Local Models

Point the gateway at your local Ollama instance. Your existing OpenAI-client code works unchanged — just change the base_url and model prefix to ollama/.

Cost Optimization & Token Savings

The gateway provides multiple layers to reduce your LLM spend — especially valuable for expensive models like Claude Opus or GPT-4o.

1. Response Caching (biggest win)

Identical requests (same model + messages + temperature + max_tokens) are served from Redis instead of calling the provider API. This is fully transparent — cached responses include "cached": true in the response.

Enable caching (configs/.env)

# Cache identical responses for 5 minutes
CACHE_TTL_SECONDS=300

# For stable prompts (e.g. classification), increase TTL
CACHE_TTL_SECONDS=3600   # 1 hour

When caching helps most: FAQ bots, document classification, data extraction pipelines, any workload where the same inputs repeat. At Claude Sonnet pricing ($3/$15 per 1M tokens), caching 50% of a 1M-request-per-month workload saves ~$4,500/month.

2. Smart Model Routing

Use cheap models for simple tasks, expensive models for complex ones. Per-key allowed_models restrictions enforce this at the gateway level.

Route by task complexity

# Cheap key for classification/extraction (90% of requests)
curl -X POST http://localhost:9000/key/generate \
  -d '{"name": "classifier", "allowed_models": ["groq/llama-3.3-70b-versatile", "deepseek/deepseek-chat"]}'

# Premium key for creative/complex tasks (10% of requests)
curl -X POST http://localhost:9000/key/generate \
  -d '{"name": "premium-tasks", "allowed_models": ["anthropic/claude-sonnet-4-20250514", "openai/gpt-4o"]}'

Cost comparison per 1M tokens (input/output)

Model                          Input     Output    Savings vs Claude Sonnet
---                            ---       ---       ---
groq/llama-3.3-70b             $0.59     $0.79     ~80% cheaper
deepseek/deepseek-chat         $0.14     $0.28     ~95% cheaper
gemini/gemini-2.0-flash        $0.10     $0.40     ~96% cheaper
openai/gpt-4o-mini             $0.15     $0.60     ~93% cheaper
anthropic/claude-sonnet-4      $3.00     $15.00    baseline
openai/gpt-4o                  $2.50     $10.00    ~17% cheaper

3. Fallback Chains (reliability + cost)

Try a cheap provider first, fall back to premium only if it fails. Your app gets reliable responses without always paying premium prices.

configs/.env — cheapest first

# Try DeepSeek first ($0.14/1M), then Groq ($0.59/1M), then Claude ($3/1M)
FALLBACK_CHAIN=deepseek,groq,anthropic

4. Budget Caps

Set hard spend limits per key, user, team, or org. When the budget is exhausted, requests return an error instead of running up your bill.

5. Rate Limiting

Redis-backed sliding window rate limiting (RPM and TPM) prevents runaway scripts from burning through your quota. Set per virtual key.

6. Output Token Limits via Guardrails

Set GUARDRAIL_MAX_OUTPUT_TOKENS to truncate verbose model responses. Fewer output tokens = lower cost.

Client Integration

The gateway is OpenAI API-compatible. Any library that works with OpenAI works with the gateway — just change the base_url. Here are examples in popular languages.

Python (OpenAI SDK)

python

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:9000/v1",
    api_key="sk-gateway-key-1"  # your gateway key
)

# Works with any provider through the gateway
response = client.chat.completions.create(
    model="anthropic/claude-sonnet-4-20250514",
    messages=[{"role": "user", "content": "Explain quantum computing in 3 sentences."}],
    max_tokens=200
)

print(response.choices[0].message.content)
print(f"Tokens: {response.usage.total_tokens}")

# Embeddings work too
emb = client.embeddings.create(
    model="openai/text-embedding-3-small",
    input=["Hello world"]
)
print(f"Dimensions: {len(emb.data[0].embedding)}")

Python (Anthropic SDK → via Gateway)

python — use Anthropic SDK through the gateway

# pip install anthropic
import anthropic

# Point Anthropic SDK at the gateway
client = anthropic.Anthropic(
    base_url="http://localhost:9000",
    api_key="sk-gateway-key-1"
)

# Requests automatically route through the gateway
# with caching, cost tracking, guardrails, etc.
message = client.messages.create(
    model="anthropic/claude-sonnet-4-20250514",
    max_tokens=1024,
    messages=[{"role": "user", "content": "Hello, Claude!"}]
)

Node.js / TypeScript

typescript

import OpenAI from 'openai';

const client = new OpenAI({
  baseURL: 'http://localhost:9000/v1',
  apiKey: 'sk-gateway-key-1',
});

const response = await client.chat.completions.create({
  model: 'groq/llama-3.3-70b-versatile',  // fast + cheap via Groq
  messages: [{ role: 'user', content: 'Write a haiku about Go.' }],
});

console.log(response.choices[0].message.content);

Go

go

import "github.com/sashabaranov/go-openai"

cfg := openai.DefaultConfig("sk-gateway-key-1")
cfg.BaseURL = "http://localhost:9000/v1"
client := openai.NewClientWithConfig(cfg)

resp, _ := client.CreateChatCompletion(ctx, openai.ChatCompletionRequest{
    Model: "deepseek/deepseek-chat",
    Messages: []openai.ChatCompletionMessage{
        {Role: "user", Content: "Hello from Go!"},
    },
})

curl

bash

export GATEWAY_URL="http://localhost:9000"
export GATEWAY_KEY="sk-gateway-key-1"

# Chat completion
curl -X POST $GATEWAY_URL/v1/chat/completions \
  -H "Authorization: Bearer $GATEWAY_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemini/gemini-2.0-flash",
    "messages": [{"role": "user", "content": "Hello!"}]
  }'

WebSocket Streaming (JavaScript)

javascript

const ws = new WebSocket('ws://localhost:9000/v1/chat/completions/stream');

ws.onopen = () => {
  ws.send(JSON.stringify({
    model: 'openai/gpt-4o',
    messages: [{ role: 'user', content: 'Tell me a story' }]
  }));
};

ws.onmessage = (event) => {
  if (event.data === '[DONE]') { console.log('\n--- done ---'); return; }
  const chunk = JSON.parse(event.data);
  const text = chunk.choices?.[0]?.delta?.content;
  if (text) process.stdout.write(text);
};

Chat Completions

POST /v1/chat/completions

The primary endpoint. Supports function calling (tools), temperature, top_p, max_tokens, stop sequences, and response format. The gateway translates this to each provider's native format.

Request

curl -X POST http://localhost:9000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [
      {"role": "system", "content": "You are a helpful assistant."},
      {"role": "user", "content": "What is 2+2?"}
    ],
    "temperature": 0.7,
    "max_tokens": 1000
  }'

Response

{
  "id": "chatcmpl-abc123",
  "object": "chat.completion",
  "model": "gpt-4o",
  "choices": [{
    "index": 0,
    "message": {"role": "assistant", "content": "2 + 2 = 4."},
    "finish_reason": "stop"
  }],
  "usage": {"prompt_tokens": 25, "completion_tokens": 8, "total_tokens": 33},
  "provider": "openai",
  "cached": false,
  "cost": 0.000143
}

With function calling (tools)

curl -X POST http://localhost:9000/v1/chat/completions \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/gpt-4o",
    "messages": [{"role": "user", "content": "What is the weather in Paris?"}],
    "tools": [{
      "type": "function",
      "function": {
        "name": "get_weather",
        "description": "Get current weather for a location",
        "parameters": {
          "type": "object",
          "properties": {"location": {"type": "string"}},
          "required": ["location"]
        }
      }
    }]
  }'

Tool compatibility: If a model doesn't support native function calling, the gateway automatically injects tool definitions as a system prompt and extracts structured tool calls from the response. This means function calling works with every model.

Parameters

Name	Type	Required	Description
`model`	string	Yes	Model ID in `provider/model` format
`messages`	array	Yes	Array of message objects with `role` and `content`
`temperature`	float	No	Sampling temperature (0-2). Default: 1.0
`max_tokens`	int	No	Maximum tokens in the response
`top_p`	float	No	Nucleus sampling threshold
`stop`	array	No	Stop sequences
`tools`	array	No	Function definitions for tool calling
`tool_choice`	string/object	No	`"auto"`, `"none"`, or `{"type":"function","function":{"name":"..."}}`

Streaming (WebSocket)

WS /v1/chat/completions/stream

For real-time streaming, connect via WebSocket. Send the same JSON request format. The server streams chat.completion.chunk objects and ends with [DONE].

Each chunk looks like:

{
  "id": "chatcmpl-...",
  "object": "chat.completion.chunk",
  "choices": [{
    "index": 0,
    "delta": {"role": "assistant", "content": "token "},
    "finish_reason": null
  }]
}

Note: The HTTP /v1/chat/completions endpoint does not support "stream": true. Use the WebSocket endpoint for streaming.

Embeddings

POST /v1/embeddings

Generate vector embeddings. Supports single or batch inputs. Compatible with OpenAI embedding models.

curl

curl -X POST http://localhost:9000/v1/embeddings \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "openai/text-embedding-3-small",
    "input": ["Hello world", "Goodbye world"],
    "dimensions": 1536
  }'

List Models

GET /v1/models

Returns all available models across all configured providers. Use this to discover which models are available.

curl

curl http://localhost:9000/v1/models -H "Authorization: Bearer YOUR_API_KEY"

Available Models by Provider

Provider	Models	Capabilities
OpenAI	`gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `gpt-3.5-turbo`	Chat, Tools, Vision, Streaming, Embeddings
Anthropic	`claude-sonnet-4-20250514`, `claude-haiku-4-20250414`	Chat, Tools, Vision, Streaming
Groq	`llama-3.3-70b-versatile`, `llama-3.1-8b-instant`, `mixtral-8x7b-32768`	Chat, Some Tools, Streaming
DeepSeek	`deepseek-chat`, `deepseek-reasoner`	Chat, Tools, Streaming
Gemini	`gemini-2.0-flash`, `gemini-2.0-flash-lite`, `gemini-1.5-pro`	Chat, Tools, Streaming
Ollama	`llama3`, `llama3.1`, `mistral`, `codellama`, `gemma2`	Chat, Streaming (local)

Images, Audio & More

POST /v1/images/generations

Image generation

curl -X POST http://localhost:9000/v1/images/generations \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"model": "openai/dall-e-3", "prompt": "A futuristic city at sunset", "size": "1024x1024"}'

Additional Endpoints

Endpoint	Method	Description
`/v1/images/edits`	POST	Edit images with prompts
`/v1/images/variations`	POST	Generate image variations
`/v1/audio/speech`	POST	Text-to-speech (TTS)
`/v1/audio/transcriptions`	POST	Speech-to-text (STT)
`/v1/moderations`	POST	Content moderation
`/v1/rerank`	POST	Document reranking
`/v1/completions`	POST	Legacy completions (text-in/text-out)
`/health`	GET	Health check: `{"status":"ok","models":N}`

Routing & Reliability

The gateway includes enterprise-grade routing with retries, circuit breakers, cooldown, and multiple load-balancing strategies.

Routing Strategies

Strategy	Config Value	Best For
Simple	`simple` (default)	Single-provider setups. No overhead.
Round Robin	`round-robin`	Even distribution across providers.
Weighted	`weighted`	Send more traffic to preferred providers.
Least Busy	`least-busy`	Route to provider with fewest in-flight requests.
Latency	`latency`	Route to fastest provider (EMA tracking).
Usage	`usage`	Balance token usage across providers.

Retries & Backoff

Configurable exponential backoff with jitter. Network errors and 5xx responses are retried; auth errors (401/403) are not.

configs/.env

RETRY_MAX=3                    # Max retry attempts
RETRY_BACKOFF_BASE_MS=500      # Base: 500ms, then 1s, then 2s

Circuit Breaker & Cooldown

After consecutive failures, a provider enters cooldown (requests rejected immediately) until the period expires. This prevents cascading failures.

configs/.env

COOLDOWN_THRESHOLD=5           # Failures before cooldown
COOLDOWN_PERIOD_SECONDS=60     # Cooldown duration
CB_THRESHOLD=5                 # Circuit breaker (GoFr HTTP service level)
CB_INTERVAL_SECONDS=30

Per-Key Fallback Override

Different keys can have different fallback chains:

configs/.env

# Key sk-key-1 fails over openai->anthropic; sk-key-2 uses anthropic->ollama
GATEWAY_KEY_CONFIG=sk-key-1:openai,anthropic;sk-key-2:anthropic,ollama

Virtual Keys

Virtual keys are scoped API keys with model restrictions, rate limits, budgets, and team membership. The master key creates and manages virtual keys.

Generate a key with restrictions

curl -X POST http://localhost:9000/key/generate \
  -H "Authorization: Bearer MASTER_KEY" \
  -d '{
    "name": "production-app",
    "team_id": "1",
    "org_id": "1",
    "tier": "premium",
    "rate_limit_rpm": 100,
    "rate_limit_tpm": 90000,
    "max_budget": 50.00,
    "allowed_models": ["openai/*", "anthropic/*"],
    "expires_in_days": 90
  }'

Key Management Endpoints

Endpoint	Method	Auth	Description
`/key/generate`	POST	Master	Create a new virtual key
`/keys`	GET	Master	List all keys
`/key/info?id=N`	GET	Master	Get key details
`/key/{id}/rotate`	POST	Master	Rotate (deactivate old, generate new)
`/key/{id}`	DELETE	Master	Delete a key
`/key/self`	GET	Any key	View your own key info

Rate limiting: Per-key RPM (requests per minute) uses a Redis sliding window. TPM (tokens per minute) tracks total token consumption. Both are enforced before the request reaches the provider.

Multi-Tenancy

The gateway supports a 4-level hierarchy for enterprise deployments. Each level can have an independent budget cap.

Hierarchy

Organization ($10,000)  -->  Team ($5,000)  -->  User ($500)  -->  Virtual Key ($50)
      |                          |                    |                    |
   budget cap              budget cap            budget cap          rate limits
   admin_email             org_id ref            team_id ref         model restrictions
                                                 org_id ref          budget cap

Management Endpoints (all require Master Key)

Resource	Create	List	Delete
Organizations	`POST /organizations`	`GET /organizations`	`DELETE /organizations/{id}`
Teams	`POST /teams`	`GET /teams`	`DELETE /teams/{id}`
Users	`POST /users`	`GET /users`	`DELETE /users/{id}`

Validation: When creating a team, the org_id must reference an existing organization. When creating a user, both org_id and team_id are validated. The same applies when generating virtual keys.

All management actions are logged to the audit log (GET /audit/log), filterable by entity type and action.

Spend & Usage Tracking

Every chat completion request is logged with token counts and computed cost (based on built-in pricing tables). Query spend data with flexible grouping.

Spend reports

# By model (which models cost the most?)
curl "http://localhost:9000/spend/report?group_by=model" \
  -H "Authorization: Bearer MASTER_KEY"

# By team (which team spends the most?)
curl "http://localhost:9000/spend/report?group_by=team" \
  -H "Authorization: Bearer MASTER_KEY"

# Group by: model | provider | key | team | user | org
# Date filters: ?start_date=2026-01-01&end_date=2026-01-31

# Self-service (any key holder can see their own spend)
curl "http://localhost:9000/spend/self?group_by=model" \
  -H "Authorization: Bearer YOUR_VIRTUAL_KEY"

Custom pricing: Override default token prices for custom or fine-tuned models:
CUSTOM_PRICING=my-fine-tuned-model:0.005:0.015,another-model:0.001:0.002
Format: model:input_per_1k:output_per_1k

Guardrails & Content Filtering

Guardrails run pre-call (validate input) and post-call (filter output) on every chat completion. Configure globally via env vars, or per-key via the database.

What Gets Checked

Check	Phase	Description
Keyword Blocking	Pre-call	Rejects requests containing blocked words (case-insensitive)
PII Detection	Pre & Post	Detects emails, phone numbers, SSNs, credit cards, IPs
Input Token Limit	Pre-call	Rejects if input exceeds max tokens
Output Truncation	Post-call	Truncates output if it exceeds max tokens
PII Redaction	Post-call	Replaces PII in model output with `[REDACTED_TYPE]`

Global defaults (configs/.env)

GUARDRAIL_ENABLED=true
GUARDRAIL_PII_ACTION=redact          # block | redact | log | none
GUARDRAIL_BLOCKED_KEYWORDS=hack,exploit,jailbreak,password
GUARDRAIL_MAX_INPUT_TOKENS=4096      # 0 = unlimited
GUARDRAIL_MAX_OUTPUT_TOKENS=4096

Per-Key Overrides (Database)

Manage via the admin UI (Guardrails page) or the API:

API

# Create/update a guardrail config
curl -X POST http://localhost:9000/guardrails \
  -H "Authorization: Bearer MASTER_KEY" \
  -d '{
    "key_hash": "sha256_of_virtual_key",
    "pii_action": "block",
    "max_input_tokens": 8192,
    "blocked_keywords": "confidential,internal",
    "enabled": true
  }'

# List all configs
curl http://localhost:9000/guardrails -H "Authorization: Bearer MASTER_KEY"

# Delete a config
curl -X DELETE http://localhost:9000/guardrails/1 -H "Authorization: Bearer MASTER_KEY"

Priority: Per-key DB config > Global DB config (key_hash=NULL) > Environment variable defaults.

Batch Processing

Submit multiple requests as a batch for asynchronous processing. Useful for data pipelines, bulk classification, or any workload where you don't need instant responses.

Submit a batch

curl -X POST http://localhost:9000/v1/batches \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{
    "requests": [
      {
        "custom_id": "classify-1",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "groq/llama-3.3-70b-versatile",
          "messages": [{"role":"user","content":"Classify: positive or negative? I love this product"}]
        }
      },
      {
        "custom_id": "classify-2",
        "method": "POST",
        "url": "/v1/chat/completions",
        "body": {
          "model": "groq/llama-3.3-70b-versatile",
          "messages": [{"role":"user","content":"Classify: positive or negative? Terrible experience"}]
        }
      }
    ]
  }'

Check status & get results

# Poll status
curl http://localhost:9000/v1/batches/BATCH_ID -H "Authorization: Bearer KEY"
# {"id":"...","status":"completed","total_requests":2,"completed_requests":2}

# Get results
curl http://localhost:9000/v1/batches/BATCH_ID/results -H "Authorization: Bearer KEY"

# List all batches
curl http://localhost:9000/v1/batches?limit=50 -H "Authorization: Bearer KEY"

# Cancel a running batch
curl -X POST http://localhost:9000/v1/batches/BATCH_ID/cancel -H "Authorization: Bearer KEY"

Worker pool: Batch requests are processed by a configurable worker pool (default: 5 workers, 120s timeout per task). Adjust via BATCH_WORKERS and BATCH_TASK_TIMEOUT_SECONDS.

Configuration Reference

All configuration is via environment variables in configs/.env. The gateway reads these at startup.

Provider Keys & URLs

Variable	Default	Description
`OPENAI_API_KEY`	empty	OpenAI API key
`ANTHROPIC_API_KEY`	empty	Anthropic API key
`GROQ_API_KEY`	empty	Groq API key
`DEEPSEEK_API_KEY`	empty	DeepSeek API key
`GEMINI_API_KEY`	empty	Google Gemini API key
`TOGETHER_API_KEY`	empty	Together AI API key
`FIREWORKS_API_KEY`	empty	Fireworks AI API key
`PERPLEXITY_API_KEY`	empty	Perplexity API key
`XAI_API_KEY`	empty	xAI (Grok) API key
`MISTRAL_API_KEY`	empty	Mistral API key
`COHERE_API_KEY`	empty	Cohere API key
`AZURE_OPENAI_API_KEY`	empty	Azure OpenAI API key
`AZURE_OPENAI_ENDPOINT`	empty	Azure OpenAI endpoint URL
`CEREBRAS_API_KEY`	empty	Cerebras API key
`SAMBANOVA_API_KEY`	empty	SambaNova API key
`AI21_API_KEY`	empty	AI21 API key
`OPENROUTER_API_KEY`	empty	OpenRouter API key
`NOVITA_API_KEY`	empty	Novita AI API key
`NVIDIA_API_KEY`	empty	NVIDIA NIM API key
`CLOUDFLARE_API_TOKEN`	empty	Cloudflare Workers AI token
`CLOUDFLARE_ACCOUNT_ID`	empty	Cloudflare account ID
`HUGGINGFACE_API_KEY`	empty	Hugging Face API key
`OPENAI_BASE_URL`	`https://api.openai.com`	Custom OpenAI-compatible endpoint
`OLLAMA_BASE_URL`	`http://localhost:11434`	Local Ollama server

Per-Provider Timeouts (milliseconds, 0 = no timeout)

Variable	Default
`OPENAI_TIMEOUT_MS`	0
`ANTHROPIC_TIMEOUT_MS`	0
`GROQ_TIMEOUT_MS`	0
`DEEPSEEK_TIMEOUT_MS`	0
`GEMINI_TIMEOUT_MS`	0
`OLLAMA_TIMEOUT_MS`	0
`TOGETHER_TIMEOUT_MS`	0
`FIREWORKS_TIMEOUT_MS`	0
`PERPLEXITY_TIMEOUT_MS`	0
`XAI_TIMEOUT_MS`	0
`MISTRAL_TIMEOUT_MS`	0
`COHERE_TIMEOUT_MS`	0
`AZURE_TIMEOUT_MS`	0
`BEDROCK_TIMEOUT_MS`	0

Gateway & Auth

Variable	Default	Description
`DEFAULT_PROVIDER`	`openai`	Provider used when no prefix given
`GATEWAY_API_KEYS`	empty	Comma-separated static API keys
`GATEWAY_MASTER_KEY`	empty	Admin master key for management endpoints
`GATEWAY_KEY_CONFIG`	empty	Per-key fallback overrides (`key:p1,p2;key2:p3,p4`)

Routing & Reliability

Variable	Default	Description
`ROUTING_STRATEGY`	`simple`	simple \| round-robin \| weighted \| least-busy \| latency \| usage
`RETRY_MAX`	`3`	Maximum retry attempts
`RETRY_BACKOFF_BASE_MS`	`500`	Base backoff in milliseconds (exponential)
`COOLDOWN_THRESHOLD`	`5`	Consecutive failures before cooldown
`COOLDOWN_PERIOD_SECONDS`	`60`	Cooldown duration
`FALLBACK_CHAIN`	empty	Comma-separated fallback providers
`REQUEST_QUEUE_ENABLED`	`false`	Enable request queuing
`REQUEST_QUEUE_SIZE`	`100`	Queue buffer size

Cache, Batch & Metrics

Variable	Default	Description
`CACHE_TTL_SECONDS`	`300`	Redis response cache TTL
`BATCH_WORKERS`	`5`	Concurrent batch workers
`BATCH_TASK_TIMEOUT_SECONDS`	`120`	Per-task timeout for batch processing
`CUSTOM_PRICING`	empty	Custom pricing: `model:in:out,model2:in:out`

Dashboard

Generate Key

View Spend

Access Control

Recent Activity

No recent activity

System Info

My Dashboard

My Usage

Key Details

My Key

My Usage

Usage Breakdown

Models

Spend & Usage

Usage by Model

Virtual Keys

Batch Jobs

Access Control

Guardrails

Guardrail Configurations

Settings

Provider Configuration

Audit Log

API Playground

Request

Response

Ready to test

Documentation

What is LLM Gateway?

Quickstart

Step 1: Configure

Step 2: Start

Step 3: Make your first request

Step 4: Try another provider

Use Cases

1. Multi-Provider Failover for Production Apps

2. Cost Control for Team API Access

3. Saving Money on Anthropic/Claude API Calls

4. PII Protection Before Data Reaches the LLM

5. A/B Testing Models

6. Centralized AI for Enterprise (Multi-Tenancy)

7. Batch Processing for Data Pipelines

8. Drop-in OpenAI Replacement with Local Models

Cost Optimization & Token Savings

1. Response Caching (biggest win)

2. Smart Model Routing

3. Fallback Chains (reliability + cost)

4. Budget Caps

5. Rate Limiting

6. Output Token Limits via Guardrails

Client Integration

Python (OpenAI SDK)

Python (Anthropic SDK → via Gateway)

Node.js / TypeScript

Go

curl

WebSocket Streaming (JavaScript)

Chat Completions

Parameters

Streaming (WebSocket)

Embeddings

List Models

Available Models by Provider

Images, Audio & More

Additional Endpoints

Routing & Reliability

Routing Strategies

Retries & Backoff

Circuit Breaker & Cooldown

Per-Key Fallback Override

Virtual Keys

Key Management Endpoints

Multi-Tenancy

Management Endpoints (all require Master Key)

Spend & Usage Tracking

Guardrails & Content Filtering

What Gets Checked

Per-Key Overrides (Database)