Start with the work
"Which GPU should we buy?" can't be answered until three other questions have been:
- What will people use it for? Drafting and summaries, questions about the firm's documents, translation, or agents that read incoming mail all put a different load on the machine.
- How good do the answers need to be? Bigger models follow instructions better and reason better. Smaller ones are quicker and cheaper. You find out which you need by testing, and benchmarks published by someone else won't tell you.
- How many people will use it at the same moment? That's a different number from how many have an account.
A good first step is a test set: 50 to 100 real questions or tasks from the firm, each with a note on what a good answer looks like. Every candidate model gets run against it. Later, the same set is how you check that a model update hasn't made things worse.
Memory for the model itself
The model's weights have to fit in GPU memory. Roughly, that's the number of parameters times the bytes per parameter:
| Precision | Bytes per parameter | 8B model | 32B model | 70B model |
|---|---|---|---|---|
| 16-bit (FP16/BF16) | 2 | 16 GB | 64 GB | 140 GB |
| 8-bit | 1 | 8 GB | 32 GB | 70 GB |
| 4-bit (quantised) | ~0.55 | ~5 GB | ~18 GB | ~40 GB |
Quantising to 4 or 8 bits saves a lot of memory and costs a little quality. That's why most private setups use it. Measure the loss on your own test set before deciding it doesn't matter.
Memory for conversations
The second thing that eats GPU memory is the key-value cache, which is the model's working memory for each conversation in progress. It grows with the length of the context and with the number of requests being handled at once.
For a model shaped like Llama 3 8B (32 layers, 8 key-value heads of 128 dimensions), every token kept in context costs:
2 (key and value) × 32 layers × 8 heads × 128 dims × 2 bytes = 128 KiB per token
8,000-token context ≈ 1 GiB per active request
Do the same sum for a 70B model from the same family (80 layers) and you get 320 KiB per token, so about 2.5 GiB per request at 8,000 tokens. What fills a GPU is long documents pasted into prompts. The number of accounts barely matters.
People versus requests
Fifty people with access almost never means fifty requests at once. People read, think and type between prompts. Serving software such as vLLM batches whatever requests are running, so one GPU can serve a lot of people as long as there's room for the key-value cache.
Agents change that. An agent that reads every incoming email runs all day and may call the model several times per email. Size for agents separately, as a constant background load.
Rough sizes
| Team | Model size | GPU memory |
|---|---|---|
| Up to 10 people | 7–14B, quantised | 16–24 GB |
| 10 to 50 people | 24–32B, quantised | 48 GB in total |
| 50 or more, or lots of agents | 70B | 80 GB or more, over two to four GPUs |
In those ranges you're looking at workstation cards (24 to 32 GB), professional cards (48 to 96 GB) and data-centre cards with 80 GB or more. Workstation cards are good value for a pilot or a small team. The professional and data-centre ones bring more memory, ECC and cooling that suits a rack.
The document index
A document assistant cuts files into passages, turns each passage into a vector with an embedding model, and stores the vectors in an index. Next to the model, the index is small:
1,000,000 passages × 1,024 dimensions × 4 bytes ≈ 4 GB
Embedding models are small enough to share the GPU or run on the CPU. The real work is in getting documents in: reading text out of scanned PDFs, keeping the index current when files change, and copying the file-share permissions into the index so people only find what they're allowed to open.
The rest of the machine
- RAM and disks. Enough memory to load models quickly, and fast NVMe for the index and the model files.
- Power and heat. A server with several GPUs can draw more than a kilowatt under load. Check the room before ordering anything.
- Downtime. Decide whether it's acceptable for the assistant to be offline during maintenance. If it isn't, you need a second server.
- Backups. The configuration, the index and the conversation logs need backing up and test restores like any other system.
What we'd suggest
Run a pilot first: one use case, one GPU, your own test set. Measure quality and load, and size the production machine from what you measured. Most of the money saved comes from not buying for a workload you only imagined.
