Why Claude Code Is Slow on Local LLMs
Even a 0.5B model took about 47 seconds just to read 16k tokens on CPU. The slowness is not generation; it is how much you send every turn.
Point Claude Code at a local model and the wait before each reply feels long. We once saw three minutes go to reading the system prompt (prefill) alone. Here is where the time goes, measured on our own CPU.
For the agent types themselves, before local versus cloud, see Choosing a Coding Agent.
Claude Code can talk to local models
Ollama documents running Claude Code against local models by setting variables such as ANTHROPIC_BASE_URL=http://localhost:11434 and launching claude --model <name>. The same page recommends a 64k+ context window for local models (https://docs.ollama.com/integrations/claude-code, checked 2026-10-16). Claude Code sends a lot per request.
Measuring prefill
On our test box (no GPU, 8 cores, 15 GB RAM), using llama.cpp build 11671’s llama-bench with Qwen2.5-0.5B-Instruct Q4_K_M (about 463 MiB), on 2026-10-16:
./llama-bench -m qwen2.5-0.5b-instruct-q4_k_m.gguf -p 512,4096,16384 -n 64 -r 1 -t 8| Test | Tokens/s | Time (computed) |
|---|---|---|
| prefill 512 | 607.90 | ~0.8 s |
| prefill 4,096 | 652.20 | ~6.3 s |
| prefill 16,384 | 345.42 | ~47.4 s |
| generate 64 | 142.95 | ~0.4 s |
One run each, so no variance. A second run gave 648.70 (4,096), 343.78 (16,384) and 138.03 (generation), nearly identical, but 888.01 at 512 tokens, so don’t trust the short-input figure. Still, the trend is clear: long inputs slow down per token (16k ran at about half the 4k speed), while generation is fast. The wait is reading the context you send every turn, not writing the answer. Real-world models are heavier per token than this 0.5B one.
Choose by tokens sent per turn
The system prompt, built-in tool definitions and history go out every turn; MCP tool definitions are deferred by default, so only tool names and server instructions load until a tool is used. In Claude Code, /context shows what fills the context and /mcp disables unused servers (https://code.claude.com/docs/en/costs, checked 2026-10-16). In the cloud that costs money; locally it costs time.
If you pay for the cloud instead: what Claude Code costs.
Question the speculative decoding default
llama.cpp sets the draft length cap with --spec-draft-n-max; the default is 3 (llama-server --help, build 11671). On our setup, 2 was fastest, 65% faster than default; 4 and above were slower. The best value depends on your models and hardware. Measure it.
Don’t starve thinking models
With a thinking model, a small max_tokens returned an empty content: thinking used up the budget. Leave room for it.
Summary
- Local slowness comes from per-turn prefill, not generation.
- Longer context is slower per token (16k at about half the 4k rate here).
- Pick harnesses by tokens per turn; disable unused MCP servers.
- Measure the draft length instead of keeping the default of 3.
Product differences: Claude Code vs Cursor vs Codex.