Self-hosted inference
Run open models on your own infrastructure with vLLM or Ollama, and use them from the copilot.
Availability — self-hosted inference is part of emma's multi-provider support and is available when it's enabled for your company. If you don't see these settings, ask your emma administrator.
emma can stand up an inference server on your own infrastructure so open models run under your control. It supports two engines — vLLM (the default) and Ollama — and once a server is running, its model is available like any other in the model picker.
In a nutshell:
- vLLM — the default, higher-throughput engine; pulls models from Hugging Face (needs an HF token)
- Ollama — the quick-start path; pulls from the Ollama library (no HF token), serves locally
- Or bring an existing endpoint — point emma at any OpenAI-compatible URL you already run
1. Choose an engine
Ask emma to set up an inference server and pick the engine that fits.
The default. Higher throughput, pulls models from Hugging Face — provide a Hugging Face token for gated or private models.
Try this — Set up a vLLM inference server for <model>
The no-Hugging-Face, quick-start path. Installs Ollama, pulls from the Ollama library, and serves the model locally (not exposed publicly).
Try this — Set up an Ollama server for <model>
2. Let emma provision it
emma provisions the server on your infrastructure and reports when it's ready. Setup runs synchronously with a time cap; if a matching server for the same engine and model already exists, emma reuses it.
A self-hosted model runs on infrastructure you pay for. Size the host to the model, and remember the server keeps running (and costing) until you tear it down.
3. Use the model
Once the server is up, select its model in the model picker, or assign it to a pipeline stage. To use an endpoint you manage yourself instead of having emma provision one, add it as an OpenAI-compatible URL — see Bring your own model keys.
What's next
- Choosing models — select your self-hosted model
- Bring your own model keys — or point at an endpoint you run
- The copilot — what the model powers
Last updated on 17 Jul 2026