Can You Run a Local LLM Without a GPU?
Yes — you can run a local LLM without a GPU. CPU-only inference is entirely possible and practical for many developer use cases. The trade-off is speed: CPU inference is slower than GPU inference, sometimes significantly so. But for batch tasks, one-shot prompts, and automation pipelines where you’re not waiting interactively for every token, CPU-only local LLMs work well.
[IMAGE: llama.cpp running local LLM inference on CPU only without a dedicated GPU on a developer workstation]
Can you run a local LLM without a GPU?
Yes. Any modern multi-core CPU with sufficient RAM can run a quantised local LLM. The standard approach is to use a 4-bit quantised (GGUF Q4) model via llama.cpp or Ollama in CPU mode. On a typical developer workstation with a modern 8-core CPU and 16 GB RAM, a 7B Q4 model runs at approximately 2–8 tokens per second — functional for scripting, documentation, and batch tasks. On high-core-count CPUs (16+ cores), speeds improve. Apple Silicon Macs are the standout exception: the unified memory architecture delivers near-GPU speeds on CPU-only hardware.
How CPU-only LLM inference works
GPU inference is fast because GPUs contain thousands of parallel processing cores optimised for the matrix multiplication operations that LLM inference requires. A GPU with 6 GB VRAM can run a 7B model with all weights in high-bandwidth GPU memory, processing many operations in parallel.
CPU inference uses the same underlying operations but with far fewer parallel cores and much lower memory bandwidth. A modern CPU might have 8–32 cores and system RAM with a fraction of the bandwidth of GPU VRAM. The result: correct output, but at a fraction of the speed.
Why CPU inference is still practical:
– Quantisation (4-bit, GGUF format) dramatically reduces memory bandwidth requirements, making CPU inference more viable
– For non-interactive tasks (batch processing, scripted automation), low tokens-per-second isn’t a problem — you just wait
– Modern CPUs with AVX2/AVX-512 instruction support get optimised execution paths in llama.cpp, improving speeds significantly
– Apple Silicon’s unified memory means the CPU and GPU share the same high-bandwidth memory pool — Metal-accelerated inference on M-series Macs is not the same as x86 CPU-only inference
How to run a local LLM on CPU only
Use llama.cpp for CPU inference
llama.cpp is the underlying inference engine for most local LLM tools. For maximum CPU performance with manual control:
1. Build llama.cpp:
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make -j$(nproc)
2. Download a quantised GGUF model from HuggingFace (search for Q4_K_M variants of your chosen model):
# Example: download Llama 3 8B Q4_K_M
wget https://huggingface.co/[repo]/llama-3-8b-q4_k_m.gguf
3. Run inference:
“`./llama-cli -m llama-3-8b-q4_k_m.gguf \
-p “Write a Python function that parses a JSON file” \
-n 200 \
–threads $(nproc)
**Tune for CPU performance:**
- `--threads N` — set to the number of physical CPU cores (not hyperthreads)
- `--ctx-size 2048` — reduce context size to lower memory usage
- `--batch-size 512` — adjust batch processing size; experiment for your hardware
---
### Use Ollama in CPU mode
Ollama automatically falls back to CPU mode when no compatible GPU is detected. There's no special configuration required:
Install Ollama normally
curl -fsSL https://ollama.com/install.sh | sh
Pull a model
ollama pull phi3
Run — will use CPU automatically if no GPU is available
ollama run phi3
Verify which compute device Ollama is using:
ollama ps
If the output shows `100% CPU`, Ollama is running in CPU mode. If it shows a GPU name with a percentage, the GPU is being used.
**Force CPU mode on a machine that has a GPU** (useful for testing):
CUDA_VISIBLE_DEVICES=”” ollama run phi3
---
### Use quantised models (GGUF / Q4)
Quantisation is the essential enabler for CPU inference. Without it, a 7B model at FP16 precision needs ~14 GB of memory bandwidth per inference step — too slow on CPU. At Q4 quantisation, the same model needs ~4–5 GB, and CPU inference becomes practical.
**Quantisation levels for CPU inference, best to worst:**
- **Q4_K_M** — best balance of quality and speed for CPU; the standard recommendation
- **Q5_K_M** — slightly better quality, slightly slower; use if your CPU has bandwidth to spare
- **Q3_K_M** — faster, lower quality; use on very constrained hardware
- **Q2_K** — very fast, noticeably degraded output; avoid unless hardware is severely limited
When pulling models in Ollama:
Default pull (typically Q4_K_M)
ollama pull codellama
Explicitly specify quantisation
ollama pull codellama:7b-code-q4_0
---
## Best local LLMs to run without a GPU
These models offer the best quality-to-speed ratio on CPU-only hardware:
| Model | Quantisation | RAM needed | Relative CPU speed | Best for |
|---|---|---|---|---|
| Phi-3 Mini 3.8B | Q4_K_M | ~4 GB | ★★★★★ | Fast CPU inference; coding + Q&A |
| Gemma 2B | Q4_K_M | ~3 GB | ★★★★★ | Lightweight; simple tasks |
| Mistral 7B | Q4_K_M | ~6 GB | ★★★☆☆ | General tasks; Apache 2.0 |
| Llama 3 8B | Q4_K_M | ~6 GB | ★★★☆☆ | Best small-model quality |
| CodeLlama 7B | Q4_K_M | ~6 GB | ★★★☆☆ | Coding tasks on CPU |
| Gemma 7B | Q4_K_M | ~6 GB | ★★★☆☆ | Reasoning + coding |
> RAM needed estimates are for model weights only; plan for 16 GB total system RAM for comfortable operation. CPU speed ratings are relative and qualitative.
**Recommendation for CPU-only machines:**
Start with **Phi-3 Mini** (3.8B Q4). It runs the fastest on CPU, requires the least RAM, and consistently surprises with its quality for its size — particularly for Python and general developer Q&A. If you need more capability, step up to **Llama 3 8B Q4** — better output at the cost of lower inference speed.
---
## How fast is CPU inference for local LLMs?
CPU inference speed varies widely depending on your processor, core count, and whether the binary is optimised for your instruction set. Indicative ranges:
- **Modern 8-core consumer CPU (e.g., Intel i7 / AMD Ryzen 7):** 2–6 tokens/second for a 7B Q4 model
- **High-core-count workstation CPU (e.g., AMD Threadripper, Intel Xeon):** 5–15 tokens/second for a 7B Q4 model
- **Apple Silicon M2/M3 (unified memory, Metal acceleration):** 25–50+ tokens/second for a 7B model — significantly faster due to the unified memory architecture
> All token/second figures above are — these are indicative ranges based on community reports and should not be published as authoritative benchmarks without verified testing data.
**For interactive chat:** 2–6 tokens/second on CPU produces readable output in real time, but the experience feels noticeably slower than GPU inference. Many developers find it acceptable for a chat window; most find it too slow for IDE inline completions.
**For batch processing and scripted automation:** Speed matters much less. If your use case is running a script that calls the model 100 times to process a batch of files overnight, CPU inference is entirely practical.
---
## Running a local LLM on a workstation without a GPU
Most developer workstations — even those without a discrete GPU — can run a useful local LLM in 2026. The key checks:
**Does your CPU support AVX2?**
Linux
grep avx2 /proc/cpuinfo | head -1
macOS
sysctl -a | grep avx2
“`
AVX2 support significantly improves llama.cpp CPU inference speed. Most CPUs from 2013 onwards support it; Intel Haswell (4th gen) and AMD Ryzen (1st gen) and newer are fine.
Do you have enough RAM?
– 16 GB total system RAM is the practical minimum for a comfortable 7B Q4 model experience
– 32 GB gives you headroom to run the model while also running your normal dev tools without swapping
Is your OS supported?
Ollama and llama.cpp support Linux, macOS, and Windows 10/11. All three platforms support CPU-only inference.
For the full picture on hardware requirements including GPU tiers, see the full local LLM hardware requirements guide. Once you’ve confirmed your setup can handle CPU inference, the step-by-step local LLM setup guide walks you through the full installation. For coding-specific model recommendations that run well on CPU, see the guide to best local coding models to run on limited hardware.
[IMAGE: Graph comparing CPU-only local LLM inference speeds for GGUF Q4 quantised models at different parameter sizes]
Frequently asked questions
Can you run an LLM on CPU only?
Yes. Using a Q4-quantised GGUF model via llama.cpp or Ollama, you can run a 7B parameter LLM on any modern CPU with 16 GB of RAM. Inference is slower than GPU inference — typically 2–8 tokens per second on a modern 8-core CPU — but the output quality is the same. CPU-only inference is practical for batch tasks, scripted automation, and one-shot prompts. Apple Silicon Macs are the best CPU-only hardware due to their unified memory architecture.
What is the best local LLM without a GPU?
Phi-3 Mini (3.8B Q4) is the best starting point for CPU-only inference — it runs faster than 7B models due to its smaller size, requires less RAM, and delivers surprisingly strong quality for coding and developer Q&A tasks. For higher output quality at the cost of slower inference, Llama 3 8B Q4 or Mistral 7B Q4 are good choices. All are available through Ollama.
How fast is CPU inference for local LLMs?
On a modern 8-core consumer CPU, a 7B Q4 model typically runs at 2–6 tokens per second. On a high-core-count workstation CPU, speeds can reach 5–15 tokens per second. Apple Silicon Macs are substantially faster due to unified memory and Metal acceleration. GPU inference on a mid-range NVIDIA GPU runs at 30–80+ tokens per second for comparison. Verify these figures against current community benchmarks for your specific hardware before relying on them.
Last updated: 2026. Verify tokens-per-second benchmarks with standardised testing on your specific hardware before relying on them.