{"id":6677,"date":"2026-10-14T15:31:33","date_gmt":"2026-10-14T15:31:33","guid":{"rendered":"https:\/\/www.voodoo.business\/blog\/?p=6677"},"modified":"2026-10-14T15:31:34","modified_gmt":"2026-10-14T15:31:34","slug":"running-llms-on-a-cpu","status":"publish","type":"post","link":"https:\/\/www.voodoo.business\/blog\/2026\/10\/14\/running-llms-on-a-cpu\/","title":{"rendered":"Running LLMs on a CPU"},"content":{"rendered":"\n<p class=\"wp-block-paragraph\">You probably know it is dog slow if you either run a large model, or a large context window !<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Here are the technologies you should look into when running on CPU<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">The obvious are quantization and MOE, but since they are the obvious, i will skip those<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">1. Multi-Token Prediction (MTP): It effectively doubles or triples your token generation speed on a CPU without changing the model&#8217;s size<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">DeepSeek-V3 onwards<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">2. Llamafile: Llamafile introduced custom, dependency-free CPU kernels utilizing vectorized outer products tailored precisely to modern CPU instruction sets (like AVX-512 and Zen4\/Skylake architectures)<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">This unlocked raw CPU speed gains ranging from <strong>30% to over 500%<\/strong> purely for prompt processing and matrix execution on standard system RAM<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">It is best with AVX-512, but also gives a significant boost on older AVX2.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">Ollama Automatically makes use of this<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">3.MLA: <strong>Multi-head Latent Attention (MLA)<\/strong> in V3\/R1, which they have scaled further into <strong>Compressed Sparse Attention (CSA)<\/strong> in their latest architectures<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">MLA dramatically compresses the KV cache into a tiny latent vector.<\/p>\n\n\n\n<p class=\"wp-block-paragraph\">DeepSeek V3 onwards<\/p>\n","protected":false},"excerpt":{"rendered":"<p>You probably know it is dog slow if you either run a large model, or a large context window ! Here are the technologies you should look into when running on CPU The obvious are quantization and MOE, but since they are the obvious, i will skip those 1. Multi-Token Prediction (MTP): It effectively doubles [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[176],"tags":[],"class_list":["post-6677","post","type-post","status-publish","format-standard","hentry","category-ai"],"_links":{"self":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/6677","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/comments?post=6677"}],"version-history":[{"count":2,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/6677\/revisions"}],"predecessor-version":[{"id":6679,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/posts\/6677\/revisions\/6679"}],"wp:attachment":[{"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/media?parent=6677"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/categories?post=6677"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.voodoo.business\/blog\/wp-json\/wp\/v2\/tags?post=6677"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}