One implementation, many endpoints
internal/llm/openai/ speaks the OpenAI Chat Completions API. Because base_url is configurable, the same code serves every OpenAI-compatible endpoint: OpenAI itself, OpenRouter, together.ai, DeepInfra, self-hosted vLLM, Ollama's OpenAI shim, LM Studio, and LiteLLM.
The provider name is one of openai, openrouter, or ollama — each corresponds to its own config block with a preset base_url (see setDefaults in internal/config/config.go). Use openai as the generic slot and override base_url when pointing at a self-hosted backend.
Endpoint recipes
Direct OpenAI. api.openai.com/v1 is the SDK default — no base_url override needed.
provider: openai
openai:
api_key: sk-...
model: gpt-5
max_tokens: 4096
Tool use: yes (native tools array). Streaming: yes (SSE).
OpenRouter aggregates dozens of providers behind one API. Model IDs use the provider/model convention:
provider: openrouter
openrouter:
api_key: sk-or-...
model: anthropic/claude-sonnet-4-6
base_url defaults to https://openrouter.ai/api/v1. Tool use depends on the underlying provider — Anthropic and OpenAI models work, most open-weights models do not.
Local Ollama exposes a Chat Completions-compatible shim at http://localhost:11434/v1:
provider: ollama
ollama:
model: llama3.1:8b
ollama.api_key defaults to not-required (the shim ignores it, but the SDK rejects empty strings — see New in internal/llm/openai/client.go). ollama.base_url defaults to http://localhost:11434/v1.
Tool use: yes as of Ollama 0.4+ (via the tools array in the Chat Completions request). Older builds return plain text.
vLLM is the production-grade self-hosted engine. Start it with --api-key if you want auth:
python -m vllm.entrypoints.openai.api_server \
--model mistralai/Mixtral-8x22B-Instruct-v0.1 \
--host 0.0.0.0 --port 8000 \
--api-key sk-vllm-secret
provider: openai
openai:
api_key: sk-vllm-secret
base_url: http://vllm.internal:8000/v1
model: mistralai/Mixtral-8x22B-Instruct-v0.1
max_tokens: 4096
Tool use: yes for models with a tool-use chat template (Hermes-2-Pro, Mistral-Nemo, Llama-3.1-8B-Instruct and above). Streaming: yes. See Guides: Self-hosted vLLM for the full deployment.
LM Studio ships an OpenAI-compatible server on http://localhost:1234/v1:
provider: openai
openai:
api_key: not-required
base_url: http://localhost:1234/v1
model: lmstudio-community/Meta-Llama-3.1-8B-Instruct-GGUF
Tool use: not supported in current builds (as of mid-2026). The endpoint accepts a tools array but ignores it and returns plain text. Use for chat-only workloads or wait for the feature to land.
LiteLLM is a proxy that fronts many providers behind one API. Point rousseau at it:
provider: openai
openai:
api_key: sk-litellm-...
base_url: http://litellm.internal:4000
model: bedrock/anthropic.claude-sonnet-4-6-20260101-v1:0
Note: LiteLLM's default port is 4000, and its /v1 prefix is optional depending on how it is deployed. Follow the LiteLLM docs for your version.
Tool use: passes through to the underlying provider. Streaming: yes. Useful for teams that want a single choke-point for LLM traffic (rate limiting, budget tracking, audit).
Configuration reference
| Field | Default | Effect |
|---|---|---|
api_key |
required | Bearer token. Use not-required for local endpoints that ignore auth. |
model |
required | Model identifier. No universal default across endpoints. |
base_url |
depends on provider name | Overrides the endpoint. See presets in setDefaults. |
max_tokens |
SDK default | Caps output tokens per completion. |
The openai, openrouter, and ollama provider names each map to their own config block (OpenAIConfig, OpenAIConfig, OpenAIConfig); they share the same shape but let you configure multiple endpoints in one config.yaml and switch between them by changing provider:.
Streaming
The provider implements agent.StreamingProvider via SSE. Every endpoint above supports streaming; Ollama's shim requires a recent build (0.5+).
Tool use
Tool definitions from the Registry are converted to OpenAI's tools array in internal/llm/openai/client.go. Not every OpenAI-compatible endpoint supports tool use — check your backend before enabling. Ollama supports it as of 0.4; older LM Studio builds do not.
Approval policies apply for endpoints that do return tool_calls. Endpoints without tool-use support will return plain text and the Registry will not be consulted.
Gotchas
- Model naming. Every endpoint has its own convention: OpenAI (
gpt-5), OpenRouter (anthropic/claude-sonnet-4-6), Ollama (llama3.1:8b), vLLM (the HuggingFace name). There is no cross-endpoint portability. - Empty API key. The SDK rejects empty strings; pass
not-required(or any placeholder) for local endpoints that do not need auth. - BaseURL trailing slash. Include the
/v1path segment. Do not include a trailing slash. - Timeouts. Local Ollama on a CPU can take tens of seconds per turn — increase your HTTP client timeout if you wrap the provider yourself.
rousseauuses the SDK default. - Tool use variance. OpenAI and Anthropic-behind-OpenRouter reliably support tools. Ollama needs a recent build and a model with a tool-use chat template. LM Studio does not support tools. If tool_calls arrive as plain text, the
Registryis not consulted. - Reasoning models. OpenAI o1/o3 series behave differently:
max_tokensis replaced bymax_completion_tokensand system prompts are limited. The SDK handles this, but expect longer per-turn latency.
Troubleshooting
openai: complete: 401 Unauthorized
Wrong or missing API key. For OpenRouter, use the sk-or-… token. For local endpoints, ensure api_key is non-empty even if the endpoint ignores it.
openai: complete: 404 model not found
The model string does not match anything the endpoint recognises. For OpenRouter, include the provider prefix (anthropic/claude-sonnet-4-6, not claude-sonnet-4-6). For Ollama, ensure the model is pulled (ollama pull llama3.1:8b).
The model ignores my tools
The endpoint does not support tool use for this model. Verify by pointing at the same model via a known-good endpoint (OpenAI, Anthropic direct, OpenRouter with an Anthropic model). See the tool-use column in the recipes above.
context deadline exceeded on local Ollama
CPU inference is slow. Options: (1) increase your caller's timeout, (2) run Ollama on a GPU host, (3) switch to a smaller model (llama3.1:8b vs 70b).
Streaming stalls halfway through a response
Some proxies (LiteLLM, corporate egress proxies) buffer SSE. Set the proxy to disable buffering for text/event-stream or run rousseau on the same network segment as the endpoint.
Related pages
- Guides: Self-hosted vLLM — production deployment.
- Providers: Anthropic — the direct API alternative for Claude.
- Guides: Multi-provider — running different providers per transport.
- Guides: Rate limits — provider-by-provider retry playbook.
- Configuration — the
openai/openrouter/ollamastanzas in context.
Further reading
internal/llm/openai/client.go—Complete, message conversion, tool schema.internal/llm/openai/client.go— streaming implementation.internal/config/config.go—OpenAIConfigstruct,setDefaultsforbase_urlpresets.