Open WebUI and Local Models: How the Pieces Fit Together

Open WebUI is a front end. It stores conversations, users, prompts, documents and settings, and it forwards chat requests to a model backend that runs somewhere else, usually on the same machine. Understanding that split explains most of what goes right and wrong with a self-hosted setup.

The backend does the thinking

The interface talks to a model server over HTTP, either a native local-model API or an OpenAI-compatible endpoint. Response quality, speed and memory use are properties of that backend and the model you loaded, not of the interface. Switching front ends will not make a small quantized model reason better.

This also means two independent things can break. If the UI loads but no models appear, the connection to the backend is wrong, and the Ollama connection fixes work through the four causes in the order worth checking them. If models appear but responses are slow or truncated, the problem is model choice, quantization or context configuration.

The same split is what makes front ends broadly interchangeable at the level of answer quality. Open WebUI compared with LibreChat sets out where the two genuinely differ, which is operations and licensing rather than output.

Model size, quantization and memory

A model must fit in memory alongside its context. Quantization reduces the precision of the weights so a larger model fits in less memory, trading some accuracy for the ability to run at all. When a model does not fit in GPU memory, part of it spills to system RAM and the CPU, and generation slows sharply. That single fact explains most reports of unexpected slowness.

Plan for everything being resident at once: the chat model, any embedding model used for document search, and the context of the conversations in flight. The VRAM and context sizer estimates that total for a given parameter count, quantization and context length, which is the number to check before pulling a model rather than after.

Context windows are a hard limit

Every model has a maximum context length covering the system prompt, the conversation history, any retrieved document text and the response being generated. When a request exceeds it the backend rejects it rather than the interface trimming for you: Open WebUI does no automatic trimming apart from the optional Context Compaction feature, which is off by default. A backend such as Ollama will instead shift older tokens out of its own window, so what you lose depends on which layer hits the limit first, and pasting a large document can push out instructions you gave earlier.

If a model appears to forget rules you set, check the context length configured on the backend before blaming the model.

How document search actually works

Uploading documents does not teach the model anything. The text is split into chunks, each chunk is converted into a vector by an embedding model, and the vectors are stored. At query time your question is embedded the same way, the nearest chunks are retrieved, and those chunks are inserted into the prompt.

Several consequences follow. The embedding model matters as much as the chat model, and changing it invalidates everything already indexed. Chunk size is a tradeoff: small chunks retrieve precisely but lose surrounding context, while large chunks carry context but crowd the prompt and dilute the match. Retrieval only finds text that is semantically close to the wording of the question, so a question phrased differently from the source may retrieve nothing useful, and the model will then answer from its own training rather than saying it found nothing. Scanned PDFs with no text layer produce no usable chunks at all.

Persistence and access

The application keeps its state in a data directory. If that directory is not on a volume that survives container recreation, every account, conversation and indexed document disappears at the next update. Check this before you accumulate anything worth keeping; the Docker install guide covers the volume mount, the image tag to pin, and the reverse-proxy settings that a public instance needs.

The first account created becomes the administrator, so create it immediately after first start rather than leaving the instance open. If you expose the interface beyond your own network, put it behind TLS and treat it as an authenticated application, because conversation history and uploaded documents are sitting inside it.

Common mistakes

Running the embedding and chat models on hardware sized for only one of them. Indexing a large document set with an embedding model you plan to replace. Blaming the interface for backend behaviour. Leaving the data directory ephemeral. Assuming that retrieval guarantees an answer came from your own documents.