Run It

Talking to Your Own Documents with RAG

Set up retrieval-augmented generation in Open WebUI so your local model can answer questions using your own files, not just what it was trained on.

15 minute read

What RAG actually does

Your model only knows what it was trained on - it has never seen your tax documents, your notes, or the manual for your specific dishwasher. Retrieval-augmented generation, RAG, fixes that by pulling relevant chunks of your own documents into the conversation before the model answers, so it can reference real content it wasn't trained on.

Open WebUI has this built in. You don't need a separate tool or a vector database you manage by hand - it's a normal part of the interface you already set up.

Creating a knowledge collection

In Open WebUI, go to Workspace → Knowledge and create a new collection. Give it a name that describes what's in it - "Home Manuals" or "Project Notes," whatever fits.

Upload the files you want included. Open WebUI processes them in the background, breaking them into chunks it can search through later. Larger documents take a bit longer to process - that's normal.

Keep collections focused. A collection of 200 unrelated PDFs will retrieve worse results than five well-organized collections of related documents. Retrieval works better when there's less irrelevant material to sift through.

Using it in chat

Start a new chat and type # - Open WebUI will show your available knowledge collections. Select one, then ask your question normally.

The model now has relevant chunks from your documents pulled into its context, and will answer using that material specifically instead of just its general training.

Try it on something real: upload a manual or a set of notes you actually reference often, and ask it a specific question you'd normally have to search for yourself. That's the moment this feature actually earns its place in your workflow.

What actually goes wrong

The model answers from general knowledge instead of your documents

This almost always means the collection wasn't actually referenced in that message. Confirm you typed # and selected the collection for that specific chat - it doesn't carry over automatically between conversations.

A PDF was uploaded but nothing useful comes back

Check whether the PDF is actually text, not a scanned image. A scanned document with no text layer has nothing for the retrieval system to index, even though it looks like normal text when you view it.

Answers reference the wrong part of a large document

This is a retrieval quality issue, not a bug - very large or poorly structured documents are harder to chunk meaningfully. Splitting one huge document into a few smaller, well-organized files in the same collection usually improves results.

Common questions

Do my documents leave my machine when I upload them?

No. Open WebUI processes and stores them locally, the same as everything else in this setup. Nothing is sent to a third party.

What file types can I use?

PDFs, plain text, Word documents, and markdown files all work well. Scanned image-only PDFs will not, since there is no text layer for the model to read.

Go deeper

This guide covers one solid path. Here's where to go if you want something different.

r/LocalLLaMA Search for RAG-specific threads if you want to go deeper into chunking strategy and retrieval tuning. Local AI Master Their curriculum covers RAG in more structured depth than one guide can.

Written from hands-on security operations experience. More about this site →