Run It

Choosing Your First Model

Parameter count, quantization, and picking a model that actually matches your hardware instead of just grabbing the biggest name you've heard of.

14 minute read

What parameter count means

Every model name comes with a number - 3B, 7B, 13B, 70B. That's the parameter count, roughly the number of internal values the model tunes during training, in billions. More parameters generally means more capability: better reasoning, more nuance, fewer obvious mistakes.

But parameter count is also the single biggest factor in how much RAM a model needs and how fast it responds. A bigger model isn't just "smarter," it's a direct tradeoff against speed and hardware requirements. There's no universally correct answer - only the right size for what you're actually running it on.

Quantization, briefly

You'll see model names with tags like Q4 or Q8 attached. That's quantization - a way of compressing the model's internal numbers to take up less memory, at a small cost to precision. Ollama defaults to a reasonable quantization level automatically, so you don't need to think hard about this to get started.

The practical version: lower quantization numbers (like Q4) use less RAM and run faster, at a slight quality cost. Higher numbers (like Q8) are closer to the full-precision model but need more RAM. Ollama's defaults are a sensible middle ground for almost everyone starting out.

Actually picking one

With 16GB of RAM, stay in the 7-8B range - models like llama3.2 or mistral run comfortably. With 32GB, you have real room to try 13B models too. Both Llama and Qwen are widely used, actively maintained open model families worth having installed to compare against each other.

ollama pull llama3.2
ollama pull mistral

Run the same real question through both and see which one you actually prefer the answers from. Benchmarks are a starting point, not a verdict - your own tasks are the only test that actually matters.

A reasonable default: if you only want one model installed to start, llama3.2 is a solid, well-rounded choice for general use at the 7-8B size.

What actually goes wrong

You pull a model too big for your RAM and everything grinds to a halt

If responses are extremely slow or the system feels frozen after pulling a new model, it's very likely too large for the RAM available. Check what's installed and remove it if needed:

ollama list
ollama rm <model-name>

Two models with similar names give noticeably different answers

Different quantization levels of the same base model can genuinely behave a little differently, not just run at different speeds. If a specific answer matters a lot, it's worth trying the same prompt on a higher-quantization version before trusting a borderline response.

Common questions

Is a bigger model always a better model?

Not for your purposes. A bigger model is more capable in general, but if it doesn't fit comfortably in your RAM, a smaller model that actually runs at a good speed will serve you better day to day.

Can I have more than one model installed at once?

Yes, and most people end up with two or three - a fast small model for quick questions and a larger one for anything that needs more reasoning.

Go deeper

This guide covers one solid path. Here's where to go if you want something different.

r/LocalLLaMA Real, current opinions on which models are worth running - this shifts often as new releases land. Local AI Master Their model directory covers far more options with benchmarks than one guide can.

Written from hands-on security operations experience. More about this site →