Run LLMs Locally with Ollama: A Beginner's Guide (2026)

You can run LLMs locally with Ollama in about ten minutes: install one program, pull a model file, and talk to it in a terminal or over a local HTTP API. The harder part is choosing a model that fits your RAM and knowing what it will get wrong. This guide covers both, and every command and output below was run on 2026-10-08.

The setup I used: a 4-core Linux container with 16 GB of RAM and no GPU, Ollama 0.40.1, shared with other jobs. That is a weak machine on purpose. If something works here, it works on your laptop. Speeds are a different story, which I'll come back to.

What does "running locally" actually mean?

Ollama is a local server. It downloads a model as a single file (a GGUF, a format for quantized weights), loads it into memory, and answers requests on localhost:11434. Your prompts never leave the machine unless you point something else at it.

That buys you three things: privacy for sensitive text, no per-token bill, and it works offline once the model is downloaded. It costs you quality and speed compared with a hosted frontier model. If you want the full picture of what's available, see open-weight models in 2026 and small language models on device.

How do I install Ollama?

On Linux the documented route is the install script from ollama.com. I downloaded and read it first, which is a good habit with any curl | sh:

curl -fsSL https://ollama.com/install.sh -o install.sh
less install.sh
sh install.sh

In my container I did not use the script, because it sets up a system service. I downloaded the release tarball (ollama-linux-amd64.tar.zst from the official download URL, about 1.4 GB because it bundles GPU libraries), unpacked it into a scratch folder and ran the server by hand:

./bin/ollama serve
./bin/ollama --version

The version printed 0.40.1. For macOS and Windows, Ollama ships installers on its download page. I did not run those, so I won't describe their steps; follow the page.

How do I pick a model size for my RAM?

Weights take roughly parameters times bits per weight, divided by 8. Add the KV cache, which grows with context length, and some overhead. Here is the weights-only table, computed with a script (parameters x bits / 8, decimal GB):

Two corrections from real files, which I cover properly in the quantization post. First, "4-bit" files are bigger than 4 bits per weight: the Llama 3.2 1B Q4_K_M file I pulled is 808 MB, which works out to 5.23 bits per weight, not 4. Second, the default tag is not always what you think. ollama show llama3.2:1b reported quantization Q8_0, and the file is 1.3 GB.

A workable rule for beginners, which is my judgement and not a vendor guideline:

• 8 GB RAM: stay at or below roughly 3B parameters at 4-bit, and keep the context short.
• 16 GB RAM: 7B to 8B at 4-bit is the sweet spot; 14B at 4-bit is possible but leaves little headroom.
• 32 GB or more: 14B to 32B at 4-bit, depending on what else you run.

If the model plus cache does not fit, the OS starts swapping and generation slows to a crawl. Pick the next size down rather than hoping.

How do I pull a model and run my first prompt?

Pull once, then run:

ollama pull llama3.2:1b
ollama list
ollama run llama3.2:1b

ollama list printed this on my machine:

NAME                           ID              SIZE      MODIFIED
llama3.2:1b                    baf6a787fdff    1.3 GB    48 minutes ago

I chose a 1B model deliberately. It downloads fast and shows the limits of small models clearly, which is the honest thing to teach first. Browse the model library on ollama.com for newer and larger options; I haven't verified which are best this month, and the list changes quickly.

How do I call a local model from Python?

Ollama's HTTP API needs no SDK. This script uses only the standard library and prints the timing fields the server returns. I ran a version of it that also accepts a model name as a second argument:

import json
import sys
import urllib.request

HOST = "http://localhost:11434"

def chat(prompt, model="llama3.2:1b", system=None, temperature=0):
    messages = []
    if system:
        messages.append({"role": "system", "content": system})
    messages.append({"role": "user", "content": prompt})
    body = json.dumps({
        "model": model,
        "messages": messages,
        "stream": False,
        "options": {"temperature": temperature, "seed": 7},
    }).encode()
    req = urllib.request.Request(f"{HOST}/api/chat", data=body,
                                 headers={"Content-Type": "application/json"})
    with urllib.request.urlopen(req) as r:
        data = json.load(r)
    secs = data["total_duration"] / 1e9
    toks = data["eval_count"]
    tps = toks / (data["eval_duration"] / 1e9)
    return data["message"]["content"], secs, toks, tps

if __name__ == "__main__":
    text, secs, toks, tps = chat(sys.argv[1])
    print(text)
    print(f"--- {toks} tokens in {secs:.1f}s ({tps:.1f} tokens/s)")

I ran it with the prompt "Explain in two sentences what a context window is." Output from the Q8_0 model:

A context window refers to the amount of time a model or AI system has to consider the context of a conversation or input, allowing it to understand the nuances and subtleties of language. By providing a context window, a model can better comprehend the intent and meaning behind a user's input, enabling more accurate and effective responses.
--- 67 tokens in 200.8s (0.4 tokens/s)

Read that answer twice. It is fluent and wrong: a context window is a limit on how many tokens the model can read at once, not an amount of time. The Q4_K_M build gave an equally confident, equally wrong answer ("a small, fixed-size area of memory..."). That is what a 1B model does, and it is why you should never trust a small model's explanation of anything you can't check.

About the speed: 0.4 tokens per second is not what you'll see. That run happened while several other jobs shared the container's four cores (load average between roughly 6 and 12 on 4 cores while that ran), and total_duration includes loading the model. I'm reporting it because it's what I measured, not because it's typical. Measure on your own machine.

Ollama also exposes an OpenAI-compatible endpoint, so code written for hosted APIs often works by changing the base URL. I tested it with curl:

curl localhost:11434/v1/chat/completions -d '{"model":"llama3.2:1b-instruct-q4_K_M","messages":[{"role":"user","content":"Say hi in five words."}],"max_tokens":20,"temperature":0}'

The response came back in the standard chat-completions shape with "content":"\"Hello, how are you?\"". That's four words, not five, a second small reminder about counting and small models.

How good is a small local model, really?

I wrote 15 tasks with automatic pass/fail checks (arithmetic, JSON output, extraction, sorting, a primality question, format constraints) and ran them at temperature 0 against four builds of the same 1B model. The checkers are strict: "The email address is: x" fails an "only the address" instruction. It's a smoke test, not a benchmark, and n=15 means a single flipped answer moves the score by 7 points.

Tasks every build above Q2 failed: 144 / 12 + 7 (answered 12), reversing "prompt" (answered "trompr"), "is 91 prime?" (answered yes; 91 is 7 times 13), and converting 2.5 hours to minutes (answered 1575). It passed 391 for 17 times 23, the French translation, sorting, the first five primes and the capital of Australia. Counting the r's in "strawberry" was wrong at fp16 and Q8_0 and right at Q4_K_M, which is noise, not evidence that 4-bit is better.

The practical read: a 1B model is fine for formatting, short rewrites and classification you can verify. It is not a calculator or a fact source. Moving up to 7B or 8B helps a lot, but I did not run those here, so I won't give you numbers for them.

What should I use local models for?

Good fits, in my view: summarising or classifying text you can't send to a cloud API, drafting where you'll edit anyway, offline use, and cheap bulk tasks with a checker (for example, the JSON schema validates or it doesn't). Bad fits: anything needing exact arithmetic, current facts, or long reasoning chains without tools.

If you want to connect a local model to an agent framework, the OpenClaw with Ollama guide and the LM Studio guide show two routes. For fitting long inputs into a small context, see token counting and context management.

Should I run it on a VPS instead of my laptop?

Only if you need it always on, or your laptop can't hold the model. A VPS with enough RAM can serve Ollama to your other tools, but CPU-only inference is slow, and you must not expose port 11434 to the internet without authentication in front of it. Ollama's API has no login of its own.

Some links on this page are affiliate links; they cost you nothing and I only link tools I'd mention anyway. If you do want a box for this, Hostinger's KVM plans are one option, covered in my Hostinger VPS review. Check the RAM first against the table above. If a managed API is cheaper for your volume, use that instead.

What to do next

1. Pull a 1B model and run the Python script above to confirm your setup.
2. Pick the largest model whose Q4 file leaves a third of your RAM free.
3. Write five prompts from your real work with an automatic check each, and run them on two model sizes before committing.
4. Read what you lose to quantization before choosing between Q4 and Q8.

What I could not verify

I did not run macOS or Windows installers, GPU acceleration, or any model above 1.2B parameters. Speeds were measured on a heavily shared CPU and are not representative. Model availability on ollama.com changes often, so check the library page for current names.