Ollama basics: install, pull, serve and API

Introduction

Ollama packs model weights, runtime and an OpenAI-compatible API into one command. This lesson takes you from zero to a serving local model: install, pull, pick a size, customize with a Modelfile, call the API from code, and point OpenCode at it. Hub page: Ollama: local models the easy way.

Everything stays on your machine: no API bills, no data leaving, and it works offline.

Install on any OS

Linux and WSL2, one command:

curl -fsSL https://ollama.com/install.sh | sh

macOS and Windows: download the app from the official site:

https://ollama.com

Check it works:

ollama --version

Pull, run, manage models

Download once, reuse offline later:

ollama pull qwen3:8b

Chat in the terminal:

ollama run qwen3:8b

List downloaded models:

ollama list

Remove one you no longer need:

ollama rm qwen3:8b

Update weights later by pulling the same tag again. Inside ollama run, /bye exits the chat, /show prints the model configuration.

Which model and size

Rule of thumb: 7 to 8 billion parameters fits 8 GB of RAM and handles chat and drafting; 70 billion wants a serious GPU and pays off for code and reasoning. Start small — a fast small model you actually use beats a huge one you never start.

• General chat and drafting: the smallest Qwen or Llama that answers coherently on your hardware.
• Code: a code-tuned variant one size up, quantized to fit — Q4 buys most of the quality back for a third of the memory.
• Embeddings for RAG: a small dedicated embedding model, not the chat model doing double duty. Pairs with RAG over local documents.

Browse the library for tags and sizes:

https://ollama.com/library

Judge with your own tasks, not leaderboards: three representative prompts, scored by you, separate good enough locally from needs the cloud in an afternoon.

Custom models with Modelfile

A Modelfile bakes a system prompt and parameters into a named model. File Modelfile beside your project:

FROM qwen3:8b PARAMETER temperature 0.2 SYSTEM You are a concise senior reviewer. Answer with code first.

Build and run it:

ollama create my-reviewer -f Modelfile

ollama run my-reviewer

Serve and HTTP API

ollama serve exposes two APIs on localhost port 11434: the native Ollama REST API and an OpenAI-compatible one, so coding assistants and scripts point at it by changing one base URL.

List models over HTTP:

curl http://localhost:11434/api/tags

One-shot generation, native API:

curl http://localhost:11434/api/generate -d "{\"model\": \"qwen3:8b\", \"prompt\": \"Why is the sky blue?\", \"stream\": false}"

Chat, OpenAI-compatible endpoint:

curl http://localhost:11434/v1/chat/completions -H "Content-Type: application/json" -d "{\"model\": \"qwen3:8b\", \"messages\": [{\"role\": \"user\", \"content\": \"Hi\"}]}"

Call it from code

Same code works against real OpenAI by changing base_url only:

from openai import OpenAI client = OpenAI(base_url="http://localhost:11434/v1", api_key="ollama") resp = client.chat.completions.create( model="qwen3:8b", messages=[{"role": "user", "content": "Explain LoRA in one paragraph."}], ) print(resp.choices[0].message.content)

Wire into OpenCode

With ollama serve running, OpenCode discovers local models automatically — open /models and pick ollama/qwen3:8b or your ollama/my-reviewer. No key needed. Pin it per project as in Connect a model, and use it as the private fallback when free cloud tiers disappear, as in Three models, one project. Keep one model loaded: swapping evicts RAM and stalls.

Env vars and keep-alive

• OLLAMA_HOST: bind address, default localhost:11434. Set to 0.0.0.0:11434 for LAN access.
• OLLAMA_MODELS: where weights live. Move it to a big disk when the default fills up.
• OLLAMA_KEEP_ALIVE: how long a loaded model stays in RAM, for example 30m, so the evening session does not pay the load cost twice.
• OLLAMA_NUM_PARALLEL: parallel requests served by one loaded model.

Troubleshooting

• Connection refused on 11434: the server is not running — start ollama serve (the desktop app starts it automatically).
• Out of memory: pull a smaller tag or a lower quantization, shorten context, keep one model loaded.
• Slow answers on CPU: expected — use a Q4 quantized small model and fewer context tokens.
• Model answers off-topic: you skipped the system prompt — bake it into a Modelfile instead of repeating it per request.
• OpenCode sees no local models: confirm ollama list shows the model and the server answers on http://localhost:11434.

Article author: Arthur Isaev