Talking to Your Own Documents with RAG

What RAG actually does

Your model only knows what it was trained on - it has never seen your tax documents, your notes, or the manual for your specific dishwasher.

Retrieval-augmented generation, RAG, fixes that by pulling relevant chunks of your own documents into the conversation before the model answers, so it can reference real content it wasn't trained on.

Open WebUI has this built in. You don't need a separate vector database you manage by hand, or to understand embeddings to use this - it's a normal part of the interface you already set up.

Creating a knowledge collection

In Open WebUI, go to Workspace → Knowledge and create a new collection. Give it a name that describes what's in it - "Home Manuals" or "Project Notes," whatever fits.

Upload the files you want included. Open WebUI processes them in the background, breaking them into chunks it can search through later. Larger documents take a bit longer to process - that's normal.

The first time you use Knowledge, Open WebUI downloads a small embedding model in the background - a separate model from your chat model, used specifically to turn document text into a searchable form. That one-time download is why the very first upload can feel slower than the ones after it.

Using it in chat

Start a new chat and type # - Open WebUI will show your available knowledge collections. Select one, then ask your question normally.

The model now has relevant chunks from your documents pulled into its context, and will answer using that material specifically instead of just its general training.

Open WebUI shows which document chunks it actually used to build the answer, usually as a small citations or sources indicator you can expand under the response. It's worth checking, especially for anything you're relying on - it tells you whether the model actually grounded its answer in your document or drifted back toward general knowledge.

Updating a collection later

If a source document changes - a manual gets a new version, your notes get revised - remove the outdated file from the collection and upload the new one. Open WebUI doesn't automatically detect that a file changed on your computer; it only knows about what you've explicitly uploaded, so an edited file needs a manual re-upload to actually update what the model can see.

Adding new documents to an existing collection is simpler - just upload additional files the same way you did the first ones. They join what's already there.

What actually goes wrong

The model answers from general knowledge instead of your documents

This almost always means the collection wasn't actually referenced in that message. Confirm you typed # and selected the collection for that specific chat - it doesn't carry over automatically between conversations.

A PDF was uploaded but nothing useful comes back

Check whether the PDF is actually text, not a scanned image - Open WebUI's default extraction reads a text layer, not the image itself. If you specifically need scanned documents to work, Admin Settings → Documents lets you switch the content extraction engine to one that includes OCR (Mistral OCR is a built-in option) - that's a deliberate, separate step, not something to expect out of the box.

The wrong content gets retrieved, not just an incomplete answer

This is different from the context-window problem below - here, the retrieved chunks themselves are the wrong ones, not too many of the right ones. Common causes: very large or poorly structured documents that are hard to chunk meaningfully, duplicate copies of the same document sitting in a collection, or a knowledge collection that's grown too broad and unfocused for the question being asked. Splitting one huge document into a few smaller, well-organized files - and keeping collections scoped to one real topic rather than "everything" - usually fixes this faster than any setting.

It finds the right document, but the answer is incomplete or ignores part of it

This one is different from the retrieval-quality problem above - it happens even when the correct chunks were retrieved. The document content, the retrieved chunks, and the conversation so far all have to fit inside the model's context window at once. If a question pulls in several chunks from a large document on top of an already-long conversation, the total can exceed what the model can actually see, and it silently drops or truncates part of it rather than throwing an error.

What to check: reduce how much conversation history has built up before asking a document-heavy question (a fresh chat helps), or increase the model's context window if your hardware has the RAM to support it - Persistent Instructions With Ollama Modelfiles covers adjusting num_ctx directly via a Modelfile. A model that seems to "forget" part of a long document is almost always this, not a RAG configuration mistake.

Common questions