tl;dr
- Should you run LLM’s at home in the first place?
- Hardware recommendations
- Models to run
- How to interact and build with local LLMs
Why run LLMs at home?
The vast majority of your day to day usage can be handled by a local model that has web search enabled. As the field progresses, about 80% of your day to day usage will be able to be accomplished by a model that you can run on hardware you own.
Many of you probably already own hardware that is capable of running strong local LLMs; mid-to-high-end Macs and also gaming PCs made in the last 5 years will be totally adequate for running a model that is functionally the same as GPT 5.3 instant.
The only thing that you are missing to run your own LLM is the knowledge of how to do so, which is what this article is here to teach you.
Why memory matters most
The main bottlenecks for running models at home have to do with memory. The two things you want to look out for are total memory availability, and memory bandwidth. These two factors will determine how large of a model you can run (bigger is better) and how fast the model wil run.
Total Memory
The total amount of memory you have will determine the size of the model that you can run. The rough back of the envelope math says that 1 billion parameters will require 1 GB of memory if you are running the model in 8 bit (more on the number of bits and quantization later).
In reality you will want more than the bare minimum however. There is an additional memory used to help with more efficient inference (can be anywhere from 0 to 3 GB depending on the inference engine).
You will also need memory to store the KV cache. The KV cache is for storing the calculations of the previous tokens that have been generated. Without the KV cache we would have to recalulate the entire prompt and also any other tokens we have generated just to get the next token.
The memory use of the KV cache scales linearly with sequence length (the number of tokens) thanks to Flash Attention. You usually want to use have at least 32K max sequence length for local models for day to day chatting, and 200k+ for larger agentic applications like coding.
Note that that long context understanding for open source models is fairly lacking, so expect the model to feel measurably dumber) when using over 32k context.
Memory Bandwidth
The second key thing to pay attention to is the memory bandwidth of your system.
For computations to happen on your CPU (or GPU) we need to get the model weights from the system memory to the compute registers. The system memory is much larger than the internal registers, which are usually only a couple of MB.
Because of this, coupled with the sequential nature of LLM architectures, the speed that we can get the small pieces of the model weights from system memory to the registers is the main determiner for how fast your LLM will run.
This is only the case for low batch sizes (you are the only one using the model), which is what you tend to have when using LLMs at home. This is does not hold when you are serving the model at scale to dozens or hundreds of users at once.
This is why Mac’s will often be just as good as GPUs for LLM inference, even though they have up to 100x less computational ability.
Raw compute is still a nontrivial factor, but you will need it alongside strong memory bandwidth speeds to see performance gains.
Where memory bandwidth does not matter as much is prompt prefilling (PP). Prompt prefilling is the initial processing of all of the tokens in your prompt. Because the model receives all of the prompt tokens at once when you submit a request, it is able to calculate all of the values in parallel, making this a computationally bound task for medium to large prompts (100s of tokens or more).
So for CPU systems (including Macs) you will see much slower prompt processing speeds than with a GPU. So if you are regularly doing tasks such as summarization where your prompts frequently contain a large number of tokens, it may be good to get a GPU instead of a Mac or CPU system.
Models
Model types
Dense
Dense LLM’s mean that all of the parameters are used per inference pass (whereas MoE models, which we will discuss next, only use a subset of their parameters). That means if your dense model has 8 billion paramaters, all 8 billion will be used for each token during inference.
Dense models will often be slower than their spare MoE counterparts, but they do have higher intelligence per parameter. This means that if you are heavily constrained in terms of total memory, or you want to maximize the intelligence of the model you can fit on your hardware, then dense models are the way to go.
MoE
One way to reduce the amount of burden on the memory bandwidth and compute requirements when running LLMs is to use Mixture of Experts (MoE) models.
MoE’s split the model into multiple smaller pieces (experts) and only runs one or two of them for each token that is generated.
Common misconception: many people thing only one expert is used from the model for their request. In reality, MoE models have different experts in every layer, and are chosen differently for pretty much every token.
These experts are usually around an order of magnitude smaller than the model as a whole, allowing for less memory bandwidth being needed and also less compute, since we are only using a small fraction of the model at a time.
Model makers will specify the number of parameters being used at inference as “active” parameters, alongside the total model parameters. For instance Qwen3 30B A3B has 3 billion active parameters and 30 billion total parameters. Note that when talking about model sizes, if there is no quantifier for what base is actually being used, you can assume that it is billions.
Modern MoE models tend to have a bit of performance degradation when compared to their fully dense counterparts of the same size, but they allow you to run much larger models at acceptable speeds even if you do not have the most powerful hardware.
Reasoning vs instruct
When choosing a model, one of the key decisions is whether to use a reasoning or instruct model.
Reasoning models generate thinking tokens internally before producing their final answer. These thinking tokens are usually hidden from the user.
This approach gives reasoning models a significant boost on tasks like math, STEM problems, and complex planning sequences. Some reasoning models even allow you to control the amount of reasoning being used.
The tradeoff is latency. The more reasoning the model does, the longer you will wait before seeing a response.
Whether a reasoning model is worth it depends on your use case. For complex analytical tasks, the performance boost is substantial. For simple queries or creative writing, you may not see much benefit over a standard instruct model.
Instruct models (non-reasoning models) show their response immediately from the first token, with no hidden thinking phase
Hardware
Mac
M series Macs are the best all around platform for running LLMs. The offer strong memory bandwidth, in a lower power package, while also being able to support up to 512GB of total memory.
The largest consumer GPU only has 32GB of memory (a RTX 5090).
| chip | max unified memory | memory bandwidth | q8_0 tg (tokens/s) | approx system price (usd) | $/gb | $ per (q8 token/s) |
|---|---|---|---|---|---|---|
| m1 | 16 gb | 68.25 gb/s | 7.91 | $470 | 29.4 | 59.4 |
| m1 pro | 32 gb | 200 gb/s | 22.34 | $790 | 24.7 | 35.4 |
| m1 max | 64 gb | 400 gb/s | 40.20 | $1,300 | 20.3 | 32.3 |
| m1 ultra | 128 gb | 800 gb/s | 59.87 | $2,700 | 21.1 | 45.1 |
| m2 | 24 gb | 102.4 gb/s | 21.91 | $600 | 25.0 | 27.4 |
| m2 pro | 32 gb | 200 gb/s | 23.01 | $1,000 | 31.2 | 43.5 |
| m2 max | 96 gb | 409.6 gb/s | 41.83 | $1,950 | 20.3 | 46.6 |
| m2 ultra | 192 gb | 819.2 gb/s | 66.64 | $4,800 | 25.0 | 72.0 |
| m3 | 24 gb | 100 gb/s | 21.34 | $960 | 40.0 | 45.0 |
| m3 pro | 36 gb | 150 gb/s | 17.53 | $1,550 | 43.1 | 88.4 |
| m3 max | 128 gb | 300 or 400 gb/s | 42.75 | $3,900 | 30.5 | 91.2 |
| m3 ultra | 512 gb | 819 gb/s | 63.93 | $10,450 | 20.4 | 163.5 |
| m4 | 32 gb | 120 gb/s | 13.54 | $1,000 | 31.2 | 73.9 |
| m4 pro | 64 gb | 273 gb/s | 30.69 | $3,350 | 52.3 | 109.2 |
| m4 max | 128 gb | 546 gb/s | 54.05 | $3,300 | 25.8 | 61.1 |
Token per second speeds for the above table is gotten from the Llama.cpp benchmarking discussion. Pricing is a rough estimate based on eBay prices for M1-M3 series, and M4 series is based on current Apple prices.
One thing to be aware of is that Mac will struggle a lot with image generation vs a GPU (since image generation is computationally bound, not memory bandwidth bound).
GPU
30 Series| gpu | vram (gb) | memory bandwidth | token t/s | used price | $/gb vram | $/token |
|---|---|---|---|---|---|---|
| rtx 3050 | 8 | 224 gb/s | $175 | 21.88 | ||
| rtx 3060 | 12 | 360 gb/s | 75.57 | $235 | 19.58 | 3.11 |
| rtx 3060 ti | 8 | 448 gb/s | $210 | 26.25 | ||
| rtx 3070 | 8 | 448 gb/s | $210 | 26.25 | ||
| rtx 3070 ti | 8 | 608 gb/s | $245 | 30.63 | ||
| rtx 3080 | 12 | 912 gb/s | 139.65 | $435 | 36.25 | 3.11 |
| rtx 3080 ti | 12 | 912 gb/s | $430 | 35.83 | ||
| rtx 3090 | 24 | 936 gb/s | 158.16 | $780 | 32.50 | 4.93 |
| rtx 3090 ti | 24 | 1008 gb/s | 171.19 | $900 | 37.50 | 5.26 |
| gpu | vram (gb) | memory bandwidth | token t/s | used price | $/gb vram | $/token |
|---|---|---|---|---|---|---|
| rtx 4060 | 8 | 272 gb/s | $285 | 35.63 | ||
| rtx 4060 ti (16g) | 16 | 288 gb/s | 63.86 | $400 | 25.00 | 6.26 |
| rtx 4070 | 12 | 504 gb/s | $420 | 35.00 | ||
| rtx 4070 super | 12 | 504 gb/s | $480 | 40.00 | ||
| rtx 4070 ti | 12 | 504 gb/s | $530 | 44.17 | ||
| rtx 4070 ti super | 16 | 672 gb/s | 132.26 | $640 | 40.00 | 4.84 |
| rtx 4080 | 16 | 716 gb/s | 142.49 | $840 | 52.50 | 5.90 |
| rtx 4080 super | 16 | 736 gb/s | 148.33 | $800 | 50.00 | 5.39 |
| rtx 4090 | 24 | 1008 gb/s | 186.21 | $2350 | 97.92 | 12.62 |
| gpu | vram (gb) | memory bandwidth | token t/s | new price | $/gb vram | $/token |
|---|---|---|---|---|---|---|
| rtx 5050 | 8 | 320 gb/s | $249 | 31.13 | ||
| rtx 5060 | 8 | 448 gb/s | $299 | 37.38 | ||
| rtx 5060 ti | 16 | 672 gb/s | 90.94 | $429 | 26.81 | 4.72 |
| rtx 5070 | 12 | 672 gb/s | 127.54 | $549 | 45.75 | 4.30 |
| rtx 5070 ti | 16 | 896 gb/s | $749 | 46.81 | ||
| rtx 5080 | 16 | 960 gb/s | $999 | 62.44 | ||
| rtx 5090 | 32 | 1792 gb/s | 290.02 | $1999 | 62.47 | 6.89 |
For GPUs, you will always want more VRAM than less. I personally recommend the RTX 3090 with its 24 gigabytes of memory as the best price to performance GPU out there right now. You can go with one of the more cost-effective 16 gigabyte GPUs, but you will eventually run into a case where you wish you had more VRAM available.
Another thing to note with the GPU pricing is that this does not include the rest of the computer around it. So if you do not have an existing machine to put these GPUs in, then you will likely need to spend another $200-$600 for the rest of the computer.
CPU
For CPU based systems, there is one special CPU that you will want to look at. That is the AMD AI Max+ 395 CPU, which is an x86 version of the Apple M series processors. It is the best value if you want to get a new system with 128GB of decently fast memory, as you are able to get 128GB of memory under 2.5K.
For more traditional CPU servers, like Ryzen and Xeon systems, you will be bottlenecked by the speed of your system memory, and with the current RAM apocalypse, I would advise against building any traditional CPU based systems.
Misc
-
- Worse memory bandwidth than a similaly priced M series Mac, so it will be slower. Could be a good deal around the $2k range of you can get it, but does not live up to the original $3K price tag, let alone the $4k it is now.
-
AMD
- AMD cards on paper look as good as Nvidia GPUs in terms of memory bandwidth and flops, but they lack the software support ecosystem that Nvidia has. They have made good progress in the past year or so, but we are still another year or two away from me feeling comfortable recommending them to someone who wants something that works well out of the box.
-
Intel
- The upcoming B50 48GB has a chance to be a huge disruptor. Intel does not have the historical GPU software bloat that AMD has, allowing them to quickly iterate and build out support for their GPUs. They have been moving fast and are catching up to Nvidia quickly, so be sure to look out for them in the current months.
Power Usage
The final thing to look out for is power usage. When buying equipment you don’t have just the up front cost of the hardware, but also the cost of the power that you use when running the models.
This amount can very quickly become nontrivial, especially with a GPU, so you should always have this in mind.
You can get a rough price for the elctric usage based on the following formula:
Total power cost per month = (watts of your system at full load)/1000 * hours of use per day * (cost per kilowatt hour) * 30 days
So for instance for me with a 3090 GPU that pulls 350 watts (plus 50 watts for the rest of the system) running for 3 hours a day at $0.20 per kilowatt hour we get
400/1000 * 3 * $0.20 * 30 = $7.20, ie 1/3 the price of a ChatGPT subscription.
My M4 Mac Mini only uses 65 watts at full power, so we end up with only $1.17 when plugging it into the above formula.
For the amount of time used, you only need to count when the model is actually running (generating/processing tokens). Having the model loaded into memory idlely uses a negligible amount of power.
Also if you are planning on having a multi-GPU system, you need to be aware of the limitations of a traditional US outlet. They provide 120 volts at 15 amps, meaning you can use at most 1800 watts on a single circuit. This limits you to around four 300 watt gpus, since the system will pull ~1400 watts at max load, and you will usually have other things running on the curcuit and you also want to leeave an extra 10-15% of headroom as well.
Running models
Quantization
LLMs tend to be trained in 16 bit, but the community has found that we dont need that many bits when we are doing inference with these models.
The act of reducing the number of bits that the model uses is called quantization. Quantization has major two benefits for us; it reduces the amount of total memory needed to load the model, and then because the model is smaller (in GB), the amount of data we need to send back and forth to the compute registers is also decreased, meaning we are able to better use our limited memory bandwidth since we are sending less data.
| Number of parameters | 16 bit | 8 bit | 4 bit |
|---|---|---|---|
| 4 billion | 8 GB | 4 GB | 2 GB |
| 8 billion | 16 GB | 8 GB | 4 GB |
| 30 billion | 60 GB | 30 GB | 15 GB |
Take for instance a 4 billion parameter model. With its native 16 bit parameters, we would need 8 GB of memory to load the model. If we were to quantize the model down to 4 bits instead, it would only take 2GB of memory. This is super helpful for loading larger models into cheaper consumer hardware. Also because we decreased the model size by 4x, there is 4x less data that needs to be moved back and forth, so inference should be ~4x faster.
When you quantize, the actual computations are still done in 16 bit, the weights are upcast from 4 bit to 16 bit for the computation and then downcast back to 4 bit to be used by the next layer. This is usually denoted as W4A16, meaning the weights are in 4 bit and the activation (computations) are done in 16 bit.
There is work on use 8 bit and even 4 bit as the activation datatype, but they are still a bit unstable and do not have wide spread support. So when you see that GPUs have a bunch of FLOPs at 4 or 8 bit, realize that you will most likely not be using 4 or 8 bit for computations, so you wont see the speed ups the extract compute gives you. The FP16 FLOPs is what you want to compare.
Traditionally these quantized models (quants) are not made by the labs that release the models but rather the community, so there can be a large amount of variance between different quants.
Unsloth, bartowski, Kiwi, cyan, and MLX community are all good quant makers on Huggingface that I trust.
Different Hardware
Different hardware (and inference engines) support different levels of quantization.
GPU only inference engines like vLLM and SGLang only support 4bit (AWQ and GPTQ), 8 bit (FP8), and 16 bit.
For CPU only (including Macs) or CPU and GPU hybrid systems, you can choose basically any bit level that you want. The format used for them is called GGUF, and most quantization repos on Huggingface will have everything from 2 bit all the way up to 8 bit quants available. I personally recommend using the Q4_K_M quants since they strike a good balance between size and accuracy.
Performance degradation
Do we really get all of this for free with no performance loss? We effectively do.
At 8 bit is has been shown there is effectively no loss, and as we have gotten better at quantizing models, a good 4 bit quant is now essentially lossless. 3 bit and low is showing signs of progress in terms of quality preservation, but it is still highly model and quantization dependent, so you will have to measure it on a case by case basis.
It has been shown that 4 bit is the optimal quant to run at when comparing memory usage and performance. For instance it is better to run a 4 bit 20 billion parameter model than a 8 bit 10 billion parameter model.
This does not tend to be the case for any multimodal LLMs or LLMs with unique architectures. These models tend to be more brittle and hard to quantize, so I do not recommend going below 8 bit for them if you can.
Llama cpp
Llama cpp is king of at home inference right now. It can be used on Macs, regular x86 based CPU systems, CPU and GPU hybrid systems (where the model is split between the CPU and GPU, allowing you to run larger models, while still getting a speed boost from the GPU) and GPU only systems (although they leave some performance on the table compared to GPU native solutions).
It’s build in C++ and is made to be super portable, so you can use it on pretty much any hardware + OS combo you can think of.
Its CPU performance is the bast out there, and also allows you offload some of the parameters to the GPU if you can fit the entire model on there.
It does have a little bit of a learning curve, but there is a large community with a multitude of resources available to you.
MLX
MLX is a machine learning library made specifically for Macs. The team is super quick to support all of the new models that come out, and it will be 10-20% faster than Llama cpp on a Mac.
vLLM/SGLang
If you have a GPU and the model you want to run can fit in its memory, then vLLM or SGLang is the way to go. vLLM is a production grade open source inference that is made to run models as fast as humanly possible at scale.
They have a large community that keep them up to date with all of the latest features and models, and my go to for deploying models myself in production.
vLLM tends to be a bit nicer for running on local non-enterprise GPUs, so I would start with that. SGLang tends to be about 10% faster than vLLM when you can get it to work.
Others
-
Ollama
- Originally built as a Llama cpp wrapper, it offers convenience at the price of reliability, quality, and correctness. They now have diverged from Llama cpp and are unable to match the speed and quality that Llama cpp has. It offers convenience of running the models at the expense of everything else. Spend the extra 15 minutes to learn Llama cpp instead.
-
LMStudio, Jan
- Llama cpp wrappers, mostly frontend niceties
-
Aphrodite engine
- If batch size 1 inference on your GPU is what you care about, then this is you best option. Lacks the wide community support that other inference engines have. If you are using very mainstream architectures like Qwen, then this could be a good choice