Guide: Self-hosted vLLM · rousseau-agent docs

Scenario

You have a vLLM instance serving an open-weights coding model on an internal box (llm.internal:8000). No inference traffic can leave the network. Point rousseau at it and treat the endpoint like any other OpenAI-compatible target.

vLLM implements the OpenAI Chat Completions schema, so rousseau's openai provider works unchanged. LM Studio, Ollama, and Text Generation Inference are the same pattern.

Prerequisites

vLLM already up on http://llm.internal:8000/v1 with /v1/chat/completions responding to a curl smoke test.
The model tag you launched vLLM with (e.g. Qwen/Qwen3-Coder-30B).

Step 1 — Confirm vLLM

curl -fsS http://llm.internal:8000/v1/models
curl -fsS http://llm.internal:8000/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{
    "model": "Qwen/Qwen3-Coder-30B",
    "messages": [{"role": "user", "content": "say hi"}]
  }' | jq .

Both should return without error. If the second call 4xx's, fix vLLM first — rousseau's client is a thin JSON shim and inherits its error surface.

Step 2 — Wire rousseau to vLLM

Edit ~/.config/rousseau/config.yaml:

provider: openai

openai:
  base_url: http://llm.internal:8000/v1
  api_key: not-required        # vLLM ignores the key but the client sends one
  model: Qwen/Qwen3-Coder-30B
  max_tokens: 4096

log:
  level: info
  format: json

The openai provider shares its schema with openrouter and ollama; the only difference is the preset base_url. Setting base_url explicitly overrides the default.

Step 3 — Smoke-test in the TUI

rousseau chat

Type explain the difference between goroutines and threads in two paragraphs. and send. If the reply streams in, the wiring is correct.

If it doesn't:

rousseau doctor

The provider.selected row will show openai; a fail on provider.openai.base_url reachability means either DNS or the internal network path is broken, not rousseau.

Step 4 — Turn on tool use

Coding models vary in tool-use fidelity. The rousseau agent loop expects the model to emit tool_use blocks whose JSON validates against the tool's InputSchema. If your vLLM model does not natively support the OpenAI tool-use schema:

Start with provider: openai + a model that supports it (recent Qwen, Mistral, Llama 3.1 8B+ variants advertise this).
Or wrap vLLM in a shim like vLLM's OpenAI-compatible tool_choice adapter and re-verify.

Once tool use works, the coding tools (read, write, edit, grep, bash) become available exactly as they do with any other provider.

Step 5 — Consider approval policies

Self-hosted models tend to be less risk-aware than frontier models. Locking the bash tool with a pattern-mode approver is prudent:

agent:
  approver:
    mode: pattern
    default: deny
    allow:
      - {tool: read,  match: ".*"}
      - {tool: grep,  match: ".*"}
      - {tool: edit,  match: "^./workspace/.*"}
      - {tool: bash,  match: "^(ls|cat|grep|rg|find|git status|git diff) "}
    deny:
      - {tool: bash,  match: "rm -rf|sudo|curl|wget|chmod|chown"}

See Guides: Audit + Approval Policies for a deeper walkthrough.

Step 6 — Watch performance

Self-hosted endpoints often benefit from higher max_iterations (the agent loop may need more round-trips to hit the same conclusion), and always from enabling session compression:

agent:
  max_iterations: 48
  compression:
    enabled: true
    trigger_messages: 60
    keep_recent: 8

Compression is off by default because it uses an LLM turn to summarise; on a pay-per-token public API this can be wasteful. On a self-hosted endpoint the token cost is zero, so leave it on.

Alternatives to vLLM

Same recipe applies to:

Ollama — use provider: ollama (defaults base_url to http://localhost:11434/v1 and api_key to not-required).
LM Studio — use provider: openai and point base_url at the LM Studio server (http://host:1234/v1).
TGI (Text Generation Inference) — use provider: openai and point base_url at the TGI OpenAI compatibility endpoint.
OpenRouter — use provider: openrouter (defaults base_url to https://openrouter.ai/api/v1).

Caveats

rousseau does not stream when the provider does not stream. Some vLLM builds ship streaming disabled — turn it on for a better TUI experience.
Prompt caching (internal/llm/anthropic uses cache_control markers) is Anthropic-specific and does nothing against vLLM. This mostly matters for long-lived sessions on pay-per-token providers.
The openai-compatible provider page is the definitive reference for every knob.

OpenAI-compatible provider — every config field.
Audit + approval policies — safety posture for less-aligned models.
Offline — running rousseau with no outbound internet.