You probably know it is dog slow if you either run a large model, or a large context window !
Here are the technologies you should look into when running on CPU
The obvious are quantization and MOE, but since they are the obvious, i will skip those
1. Multi-Token Prediction (MTP): It effectively doubles or triples your token generation speed on a CPU without changing the model’s size
DeepSeek-V3 onwards
2. Llamafile: Llamafile introduced custom, dependency-free CPU kernels utilizing vectorized outer products tailored precisely to modern CPU instruction sets (like AVX-512 and Zen4/Skylake architectures)
This unlocked raw CPU speed gains ranging from 30% to over 500% purely for prompt processing and matrix execution on standard system RAM
It is best with AVX-512, but also gives a significant boost on older AVX2.
Ollama Automatically makes use of this
3.MLA: Multi-head Latent Attention (MLA) in V3/R1, which they have scaled further into Compressed Sparse Attention (CSA) in their latest architectures
MLA dramatically compresses the KV cache into a tiny latent vector.
DeepSeek V3 onwards