How to run an AI model on your own computer
A beginner's walkthrough for running open language models locally with Ollama or LM Studio: check your memory, pick a model, run it and connect other apps to it.
Short answer. Install Ollama or LM Studio, choose a model that fits your computer’s memory (a model of 4 to 8 billion parameters for an 8 to 16 GB machine), and run it. Everything then happens on your own computer, and you can even switch off the internet.
What you need
- A reasonably recent computer. An Apple silicon Mac, or a Windows or Linux PC, ideally with a graphics card. LM Studio recommends 16 GB or more of RAM.
- Enough memory for the model. This is the main limit. How much memory do I need shows how to work it out in a minute.
- Disk space. A model is a file of about 1 GB to tens of GB.
- An internet connection for the first download. After that, running a model works offline.
Step 1: choose a tool
| Tool | Best for | Interface |
|---|---|---|
| Ollama | The simplest first try, and giving other apps a local model | Command line, plus a local API |
| LM Studio | People who prefer a graphical app, with a built-in model browser | Desktop app |
| llama.cpp | Maximum control, unusual hardware, squeezing a big model into limited memory | Command line and server |
| MLX LM | Mac owners who want a Python package | Command line and Python |
| Open WebUI | A shared, ChatGPT-style web page for a team or household, on top of Ollama | Web app |
For most beginners the first two are the right start. Engines such as vLLM and SGLang are built to serve many users on GPU servers, not for a first experiment. The tools page lists them all, and choosing an inference engine compares them in more detail.
Step 2: choose a model that fits
Start from your memory and from the smallest model that does the job. The numbers below are our 4-bit estimates for the smallest variant of each model.
| Your memory | Good starting points |
|---|---|
| About 8 GB | Gemma 4 E2B (3.5 GB), MiniCPM5-2B (0.7 GB), LFM3 (1.2 GB) |
| About 16 GB | Olmo 3.1 7B (5 GB), Granite 4.2 8B, gpt-oss 20B (OpenAI says it fits in 16 GB) |
| 24 to 32 GB | Qwen3.8-27B (19 GB), Muse Glimmer (20 GB), Gemma 4 31B |
The small models for laptops list explains each of these, and the VRAM calculator estimates other cases. When in doubt, pick the smaller one: a model that fits comfortably is faster and more pleasant than one that barely fits.
Step 3: run it
With Ollama:
- Install Ollama for your system from its website.
- Open a terminal and run
ollama run gemma4. The first time, it downloads the model. Then you can chat right in the terminal. - Use
ollama listto see what you have downloaded andollama pull <name>to fetch another model.
Ollama’s library at ollama.com/library lists ready-to-run versions of many open models. These are repackaged, and often quantized to fit consumer hardware, so check the tag before you compare quality with the original release.
With LM Studio:
- Install the app. It runs on Apple silicon Macs (macOS 14 or newer), on Windows and on Linux. Intel Macs are not supported.
- Use the built-in model browser to search for a model and download it.
- Open the chat tab, load the model and start typing. You can also start a local server from the app.
Step 4: use it from other apps
Both tools can run a local server. Ollama’s API listens on your own machine, by default on port 11434, and many apps can talk to it, either directly or through the OpenAI-compatible API format. Ollama also offers a launcher that connects coding assistants such as Claude Code or Codex to a model it serves.
If you want a ChatGPT-style web page for your household or team, put Open WebUI on top. Read its license first: it is free to use, but it keeps a branding condition and is not open source in the strict sense.
If it feels slow
- Check that the model fits in fast memory. In Ollama,
ollama psshows whether the model is running on the GPU or partly on the processor. A model that spills into system memory is much slower. - Choose a smaller model or a lower precision. Quantization explained shows the trade-off.
- Shorten the context. A long context uses memory for the KV cache.
- Close other heavy apps so the model has more room.
Privacy and safety
- Check where a model really runs. Local means the computation stays on your computer. Ollama can also run models in its own cloud, and LM Studio offers optional hosted models. Those send your prompts away, so confirm that a model runs locally before you give it sensitive data.
- Download from official sources. Use the developer’s own page or a well-known publisher. Formats such as safetensors and GGUF are designed to hold only data, and you should be careful with any model that asks you to run extra code.
- Do not expose your local server to the internet unless you add authentication and know what you are doing.
- Check the license if you use the model for work. The license of the original model still applies to a GGUF copy made from it.
When you outgrow your laptop
If you want to serve many users, or run a very large model, look at vLLM or SGLang on a GPU server, and use the models by VRAM page to see which hardware class a model needs. Self-hosting vs API helps you decide whether buying hardware beats paying per token.
Frequently asked questions
- Do I need a GPU to run a local AI model?
- No, but it helps. Small models run on a modern processor or on an Apple silicon Mac, and a graphics card or unified memory makes them much faster.
- Which is easier, Ollama or LM Studio?
- LM Studio is a graphical app, so it is the easiest if you avoid the terminal. Ollama starts from the command line and runs a local API that many apps support. Both are free to start with.
- Can I use a local model with coding tools and chat apps?
- Yes. Ollama and LM Studio both run a local server that other programs can talk to, and Ollama can connect coding assistants such as Claude Code or Codex to a model on your machine.
- Is a local model really private?
- The computation happens on your computer and works offline. Check any cloud features, though: Ollama can also run models in its own cloud, and LM Studio offers optional hosted models, and those send your prompts off your machine.
- How much disk space do models take?
- Roughly the size of the file you download: about 1 GB for a very small model, around 20 GB for a 30B model at 4-bit and hundreds of GB for the biggest open models. The model page lists download sizes.
Key terms in this guide
- GGUF GGUF is a single-file model format used by llama.cpp and the tools built on it, which bundles quantized weights with the information needed to run them.
- Inference Inference is the act of running a trained model to get an output, such as answering a prompt, as opposed to training it.
- OpenAI-compatible API An OpenAI-compatible API is a server that accepts the same requests as the OpenAI API, so apps written for it can use an open model instead.
- Quantization Quantization stores a model's weights with fewer bits, for example 4 instead of 16, to cut memory use and speed up inference at a small cost in quality.
- Unified memory Unified memory is a design, used by Apple silicon and some other chips, in which the CPU and GPU share one pool of memory, so a large model can use most of it.
- VRAM VRAM is the memory on a graphics card; it decides which models a GPU can hold, because the weights must fit in memory for the model to run quickly.
Models mentioned in this guide
| Model | Released | Parameters | Context | License | Smallest variant needs |
|---|---|---|---|---|---|
| Gemma 4 Google DeepMind | 5.1B – 30.7B | 256K | Apache 2.0 (Permissive) | 3.52 GB Laptop or phone | |
| gpt-oss OpenAI | 21B – 117B 5.1B active | 128K | Apache 2.0 (Permissive) | 15 GB Laptop or phone | |
| MiniCPM5-2B OpenBMB | 1.08B – 2.52B | 128K | Apache 2.0 (Permissive) | 0.745 GB Laptop or phone | |
| Olmo 3.1 Allen Institute for AI (Ai2) | 7.3B – 32.2B | 64K | Apache 2.0 (Permissive) | 5.04 GB Laptop or phone | |
| Granite 4.2 IBM | 3.7B – 29.3B | 128K | Apache 2.0 (Permissive) | 2.55 GB Laptop or phone | |
| Qwen3.8-27B Alibaba Qwen | 27.8B | 256K | Apache 2.0 (Permissive) | 19 GB Gaming GPU |