For those looking to run LLMs at home, one of the biggest questions you need to answer is what model would you like to run? To determine what model you want to run, you need to first know what type of questions you are going to be asking the model.
For instance, will you be attaching it to a search engine or local knowledge base and asking it rag style queries? Will you be relying on its internal knowledge to answer basic questions about different programs or apps you may use? Or will you use it for things like data processing and coding? Your answers to these questions will determine the model that you’re going to want to use.
For many of these tasks, there will not be a benchmark that is directly analogous to your use case, making it very difficult to use existing information to determine what model you should be running. For the open source models that you can run at home, because of their small size, there can be wide differences between what they are good at, depending on what had been in their pretraining data.
This is especially apparent when asking the model’s to recall information directly from their weight (no RAG). It has been shown in benchmarks like AA-omniscience that the amount of information that a model is able to store is directly related to its size. That means that these small models will only have remembered a small fraction of what they were trained on, and can vary widely from model to model.
To know what model to use, and if it even makes sense to run a local model, you should be making your own benchmarks to be able to measure their capabilities for your tasks. You don’t necessarily need many questions. As few as ten can suffice depending on your problem and how representative the questions are to what you will be asking.
The questions very often can be generated by a large LLM like GPT 5.2, Gemini Pro 3, or Claude Opus 4.5. You want to make sure these questions are written in a similar way to how you’d be asking them in the real world. Often, when LLMs generate the questions themselves, they are very well worded, don’t use any abbreviations, and have perfect grammar. But very often, the questions we ask to LLMs are not nearly as nicely formatted as this. So you’re going to want to make sure that the questions have the same look and feel so that you’re getting, so that you’re getting an accurate measurement from the benchmark.
You want to make the data set difficult but not too difficult. You usually want at most only one or two models that are scoring 0% on it, if at all. And you don’t want any models going above 80% accuracy either. That way, new models have room to grow into the benchmark. With this benchmark, you can also use it as a signal while you are optimizing your local LLM environment as you go and tweak prompts and retrieval setups and other infrastructure surrounding the model.
To give an example, I recently have been using and learning NeoVim, which involves a whole bunch of new keybinds that I have to go and learn. I find myself asking ChatGPT what many of these keybinds are. But it’d be more convenient if I could use a local model which is able to run much faster to find this information out.
To find out which local LM I should use, I had uh GPT 5.2 in Codex generate 50 questions of varying difficulty about NeoVim that I then went and had my candidate Candidate local LLMs go and answer.
