Quick answer
To run an open-weight AI model, you need roughly enough fast memory to hold all of its parameters, plus about 10 to 20% extra. Multiply the number of parameters by the bytes each one takes: 2 bytes at full 16-bit precision, 1 byte at 8-bit and half a byte at 4-bit. So a 7-billion-parameter model needs about 5 to 6 GB at 4-bit, while a giant like Kimi K3.1 Pro (2.8 trillion parameters) needs well over 1.4 TB. On a PC that memory is usually your graphics card’s; on an Apple silicon Mac it is the shared unified memory.
📘 Main article: This guide follows our round-up Anthropic IPO: Why It’s an AI Bubble Test, Plus Kimi K3.1 Pro, Genie 4 and OpenAI’s Maths Claim, where Kimi K3.1 Pro turned out to need around 44 high-end graphics cards to run.
Key takeaways
- Memory, not processor speed, decides whether a model will load at all.
- Use parameters × bytes per parameter + 10–20% as a quick estimate.
- Quantised (compressed) versions at 4-bit need about a quarter of the memory of the full model, with some loss of quality.
- Mixture-of-experts models still need memory for all their parameters, not just the active ones.
- If it will not fit, use a smaller model, rent a cloud GPU or use an API.
In this guide
- How much memory do you need to run an open-weight AI model?
- What is the simple rule for working it out?
- How much memory do common model sizes need?
- Do “active parameters” reduce the memory you need?
- How do you check what your own computer has?
- What can you do if the model is too big?
- Frequently asked questions
- The bottom line
- Related posts
How much memory do you need to run an open-weight AI model?
An open-weight model is one you can download and run yourself, such as Llama, Qwen, Gemma, DeepSeek or Kimi. Running it locally keeps your data on your own machine, but only if the whole model fits in fast memory.
On a Windows or Linux PC, the memory that matters most is your graphics card’s VRAM (video memory). On an Apple silicon Mac, the processor and graphics share one pool called unified memory, which is why high-memory Macs are popular for this. Ordinary system RAM can be used as overflow, but the model then runs much more slowly.
What is the simple rule for working it out?
Every model is a big list of numbers called parameters (or weights). How much space each number takes depends on its precision:
| Precision | Bytes per parameter | Typical use |
|---|---|---|
| 16-bit (FP16 / BF16) | 2 | The full model as released |
| 8-bit | 1 | Near full quality, half the size |
| 4-bit | 0.5 | The most common home setting; small quality loss |
Then add 10 to 20% for the working memory a model uses while it reads your prompt and writes its answer. Longer conversations and bigger documents use more.
💡 Tip: Worked example: a 27-billion-parameter model at 4-bit is 27 × 0.5 = 13.5 GB, plus about 20% gives roughly 16 GB. A 24 GB graphics card would handle it comfortably.
How much memory do common model sizes need?
| Model size | Approx. memory at 4-bit | Approx. memory at 16-bit | What can usually run it |
|---|---|---|---|
| 3–4 billion | 2–3 GB | 7–9 GB | Most modern laptops |
| 7–8 billion | 5–6 GB | 15–18 GB | An 8 GB graphics card or a 16 GB Mac |
| 27–32 billion | 16–22 GB | 55–75 GB | A 24 GB graphics card or a 32 GB Mac |
| 70 billion | 40–45 GB | 140+ GB | Two big graphics cards or a 64 GB+ Mac |
| 2.8 trillion (Kimi K3.1 Pro) | 1.4+ TB | 5.6+ TB | Dozens of data-centre or high-end cards |
These are rough estimates using the rule above, not exact figures. The download page for a model on Hugging Face often lists the file size of each quantised version, which is the best guide to how much memory it needs.
The Kimi example matches what Matt Wolfe worked out in his video: about 44 RTX 5090 cards at 32 GB each, or at least three 512 GB Mac Studios.
Do “active parameters” reduce the memory you need?
No. Some large models, including Kimi K3.1 Pro, use a mixture-of-experts design. Only part of the model (Kimi uses about 104 billion of its 2.8 trillion parameters) does the work for each word it writes. That makes answers faster and cheaper to compute, but the whole model still has to be loaded, because a different part may be needed for the next word.
⚠️ Warning: Do not size your hardware on “active” parameters. Always use the total parameter count.
How do you check what your own computer has?
- Windows: open Task Manager, go to Performance and click your GPU. “Dedicated GPU memory” is your VRAM.
- Mac: open the Apple menu, choose About This Mac and look at Memory. On Apple silicon, you can use roughly two-thirds to three-quarters of it for a model, leaving the rest for the system.
- Linux with an Nvidia card: run
nvidia-smiin a terminal to see total and used memory. - Compare your figure with the model’s file size plus about 20%. If it fits, try it in a free tool such as Ollama or LM Studio.
💡 Tip: Before running any downloaded model, check where it came from. Our guide on checking a downloaded AI model before you run it covers the basics.
What can you do if the model is too big?
- Pick a smaller model. Many 7 to 30 billion parameter models are good enough for drafting, summarising and sorting.
- Use a more compressed version. A 4-bit file is about a quarter of the full size. Going lower than 4-bit often hurts quality noticeably.
- Rent a cloud GPU by the hour for big jobs, and set a budget first. See how to avoid surprise cloud GPU bills.
- Use the model through an API. For huge models like Kimi K3.1 Pro, this is the only realistic option for most people. Our piece on running a model locally or through an API weighs up the choice.
Not sure local AI is worth it? Start small: install a free tool, run a 7-billion-parameter model on one routine task this week and see if the quality is good enough.
Frequently asked questions
How much RAM do I need to run an AI model locally?
For a 7 to 8 billion parameter model at 4-bit, about 8 GB of graphics memory or a 16 GB Mac is a comfortable starting point. Larger models need proportionally more.
Can I run an AI model without a graphics card?
Yes, small models can run on a normal processor and RAM, but they are much slower. A graphics card or Apple silicon Mac makes a big difference.
What does 4-bit quantised mean?
It means each parameter is stored in half a byte instead of two, cutting memory needs by about three-quarters with a small drop in quality.
Why does a longer conversation use more memory?
The model keeps a working record of everything in the conversation so far. Longer chats and big documents make that record larger.
Can I run Kimi K3.1 Pro at home?
Not realistically. It needs over 1.4 TB of fast memory even when compressed, so most people will use it through an API.
The bottom line
Working out whether a model will run is simple maths: parameters times bytes per parameter, plus a safety margin. Check your own memory, start with a model that fits comfortably, and use the cloud or an API for anything bigger. Your next step: look up your graphics or unified memory today and write down the largest model size it can hold.
Related posts
- Anthropic IPO: Why It’s an AI Bubble Test, Plus Kimi K3.1 Pro, Genie 4 and OpenAI’s Maths Claim
- Should You Run Muse Spark Open Locally or Use It Through an API?
- How to Avoid Surprise Bills When You Rent a Cloud GPU for AI
