How to Run an Ollama Model Locally and Call It From Code

To run a model locally with Ollama, install Ollama, then run ollama run gemma4:e2b in a terminal. Ollama downloads the model the first time and opens a chat on your own computer. Type /bye to leave.

That's the whole core loop. The rest of this post covers installing on each OS, picking a model that fits your memory, calling the local server from code, and the mistakes that trip people up. Everything here comes from the official Ollama docs, and model names change often, so check the Ollama quickstart and the model library before you copy a tag.

Does Ollama run models locally?#

Yes, with one exception worth knowing. Ollama has two modes: local models that run on your machine, and cloud models that run on Ollama's servers. The quickstart says you can "sign in to use cloud models, or choose a local model."

Local models need no API key and no account. Your prompts go to a server on localhost, port 11434. Cloud models are different: requests go to https://ollama.com and need an API key. So if you're searching for how to "run a cloud model locally", the honest answer is that you can't. A cloud model is hosted remotely. If you want everything on your own hardware, pull a local model instead.

Install Ollama on Mac, Windows, and Linux#

You can download the app from the Ollama site for macOS, Windows, or Linux, or use the install commands from the Ollama GitHub README.

A manual download works on all three. There's also an official Docker image, ollama/ollama, on Docker Hub.

After installing, open the app or just type this in a terminal:

ollama

Follow the setup prompts. You'll be asked to run a model or connect Ollama to other tools. On Linux, if the server isn't already running, start it yourself:

ollama serve

The install steps are the same idea on every platform, so "how to run Ollama locally on Windows" and "on Linux" differ only in the install command above.

Pull and run a model#

There are two commands to remember. ollama pull downloads a model without starting a chat. ollama run downloads it if needed and opens a chat.

ollama pull gemma4:e2b
ollama run gemma4:e2b

The quickstart uses Gemma 4 E2B as its local example. The download is about 7.2 GB. The README shows ollama run gemma4 with no size tag, and the full list of models lives at ollama.com/library.

The part after the colon is the tag, which usually picks a variant or size. Tags differ per model, so read the model's page rather than guessing one.

Pick a size that fits your RAM#

The pack gives one concrete number to anchor on. For Gemma 4 E2B, the download is about 7.2 GB, and the docs recommend 8 GB of available VRAM, or unified memory on a Mac. Two more points from the docs:

• Larger context windows need more memory.
• With less VRAM, Ollama can use system RAM, but responses may be slower.

So if you don't have a big GPU, it still runs. It's just slower. For "run a local model on GPU", there's no flag in the quickstart to set. The docs frame it as memory: enough VRAM (or unified memory on a Mac) is what you want.

Here's my rule of thumb, and it's advice, not an official number: the model's download size should be comfortably below your free memory, because the context window needs room too. This small script encodes that. Plug in your own numbers from the model's page.

def verdict(model_gb, free_gb, headroom=1.25):
    need = model_gb * headroom
    if free_gb >= need:
        return "should fit"
    if free_gb >= model_gb:
        return "tight: expect a small context window or slow replies"
    return "too big: pick a smaller model"

# 7.2 GB is the Gemma 4 E2B download size from the Ollama quickstart
for free in (4, 8, 12, 16):
    print(f"{free:>2} GB free -> {verdict(7.2, free)}")

I'd start with the smallest model that does your job and move up only if the answers aren't good enough. Check the model library for current sizes before you pull anything large.

Call the local model from code#

Once a model is pulled and the server is running, you can hit it over HTTP. The simplest request, straight from the quickstart:

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "gemma4:e2b",
    "messages": [
      { "role": "user", "content": "Say hello in one sentence." }
    ],
    "stream": false
  }'

Read the answer from message.content. Setting "stream": false returns one complete response instead of a stream.

Ollama also exposes compatible endpoints on the same server, so existing client code can often point at it:

• http://localhost:11434/v1/chat/completions (OpenAI Chat Completions). Read choices[0].message.content.
• http://localhost:11434/v1/responses (OpenAI Responses). These requests are stateless, so send the conversation history each time. previous_response_id and conversation aren't supported.
• http://localhost:11434/v1/messages (Anthropic Messages). Send the anthropic-version: 2023-06-01 header and a max_tokens value.

These compatibility layers cover a subset of the original APIs, so check the docs if you depend on a specific feature.

Python and JavaScript libraries#

The official Python library installs with pip install ollama:

from ollama import chat

response = chat(model='gemma4:e2b', messages=[
    {'role': 'user', 'content': 'Why is the sky blue?'},
])
print(response.message.content)

For JavaScript, run npm i ollama:

import ollama from "ollama";

const response = await ollama.chat({
  model: "gemma4:e2b",
  messages: [{ role: "user", content: "Why is the sky blue?" }],
});
console.log(response.message.content);

The README examples use the tag gemma4. I swapped in gemma4:e2b so it matches the model you pulled earlier. Use whatever tag you actually have.

No library at all#

If you'd rather avoid a dependency, plain Python works against the same endpoint. This one needs the server running, so it isn't a runnable block here:

import json, urllib.request

body = json.dumps({
    "model": "gemma4:e2b",
    "messages": [{"role": "user", "content": "Say hello in one sentence."}],
    "stream": False,
}).encode()

req = urllib.request.Request(
    "http://localhost:11434/api/chat",
    data=body,
    headers={"Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as r:
    print(json.load(r)["message"]["content"])

The parsing side is easy to test offline. This script shows how the response shape maps to the text you want, using a hand-written sample (not real model output):

import json

# Hand-written sample shaped like a /api/chat reply
sample = '{"message": {"role": "assistant", "content": "Hello there."}}'
reply = json.loads(sample)
print(reply["message"]["content"])

# The OpenAI-style endpoint nests it differently
openai_sample = '{"choices": [{"message": {"content": "Hello there."}}]}'
print(json.loads(openai_sample)["choices"][0]["message"]["content"])

Run a GGUF or local model file#

People search for "ollama run local model file" and "ollama run gguf model locally". The sources I'm working from don't include the steps, so I won't guess at a command. What they do show is that the Ollama repository has docs called "Importing models" and a "Modelfile reference". Start there for bringing your own model file in. Ollama's main path is pulling models from its library by name.

Run models without Ollama#

Ollama isn't the only way. The README names llama.cpp as its supported backend, so you can also look at that project directly if you want a lower-level tool. I'm not covering llama.cpp setup here, because it's a different tool with its own docs. For most developers who just want a local API, Ollama is the shorter path.

Common mistakes and gotchas#

• Server not running on Linux. If requests to localhost:11434 fail, run ollama serve. The quickstart says to do this if it isn't already running.
• Calling a model you haven't pulled. A request for gemma4:e2b needs that model on disk first. Run ollama pull gemma4:e2b.
• Mixing up cloud and local URLs. Cloud uses https://ollama.com with a bearer API key. Local uses http://localhost:11434 with none. Cloud model names like gemma4:31b in the docs are for the cloud examples.
• Putting an API key in browser code. The docs say to keep your key on your application's server, outside browser code and source control. Local use doesn't need a key at all.
• Expecting the Responses endpoint to remember the chat. It's stateless. Send the history every time.
• Picking a model too big for your memory. It may still run using system RAM, but slowly. Larger context windows also need more memory.
• Forgetting "stream": false. Without it, the native API streams, and a simple JSON parse of the whole body can break. Set it when you want one object back.

Quick reference checklist#

1. Install: download the app, or use the install command for your OS.
2. Start: run ollama, or ollama serve on Linux if the server isn't up.
3. Pull: ollama pull gemma4:e2b.
4. Chat: ollama run gemma4:e2b, then /bye to exit.
5. Check memory: the Gemma 4 E2B example is about 7.2 GB, with 8 GB of VRAM or unified memory recommended.
6. Call it: POST http://localhost:11434/api/chat with "stream": false, then read message.content.
7. Use a library if you prefer: pip install ollama or npm i ollama.
8. For coding agents, try ollama launch claude. Ollama offers to install Claude Code if needed.
9. Before pulling anything, check the Ollama quickstart and the GitHub repo for current commands and model names.

Quick answers

Saved you an hour? Send a tip.

USDC or ETH on Base. Keeps the notes free and the examples running.

Base

USDCETH

$2$5$10 Other amount in USDC, from 1 to 1000$

0x9dFD2Ee86182f92464b50809C6702B999a0Cd54f Copy

Open in wallet · 5 USDC

On a computer? Scan the code with your phone wallet, or copy the address.

USDC or ETH on Base works best. Other tokens on Base, Ethereum or Arbitrum also reach this address. Tips are optional and unlock nothing; every guide stays free.