Run big LLMs on cheap, mismatched GPUs
Updated June 2026
In a GPU shortage, the smart way to a 70B is not one $12,000 card — it is several cheap cards pooled together. Here is how to combine mismatched GPUs into a single endpoint, and which cards give you the most VRAM per dollar to do it.
Pool cheap VRAM instead of buying one big card
A 70B model needs roughly 48GB of VRAM at Q4 — normally that means a single workstation card costing five figures, or two matched 24GB cards. But there is a cheaper way: pool the VRAM of several mismatched cards you already have (or can buy cheaply), across one or more machines, so they act as one big pool.
The open-source tool for this is Tightwad (MIT-licensed, free). Its pitch: "pool your mismatched CUDA + ROCm + Metal cards into one OpenAI-compatible endpoint, so a model that fits on no single machine runs across all of them." It uses speculative decoding for speed (~1.86× in its own benchmarks), supports Mixture-of-Experts models, and runs entirely on your hardware — no cloud bill.
Tightwad — pool mismatched GPUs into one local LLM endpoint. Open-source, runs in your homelab.
Why this changes how you buy GPUs
Pooling flips the goal from "one card big enough" to "the most cheap VRAM you can combine." That favors high-value cards on $/GB VRAM — even across brands, since Tightwad mixes NVIDIA, AMD and Apple. Two cheaper 16–24GB cards can out-VRAM a single flagship for less money. Rank candidates by $/GB on the GPU tracker and pair with the NVIDIA vs AMD vs Intel guide for the software trade-offs.
The build, end to end
- Buy cheap VRAM: pick high-$/GB cards (mixed brands OK) — see the live table below.
- Feed them: 64GB+ DDR5 and a fast NVMe SSD for weights and offload.
- Pool them: set up Tightwad to combine the cards into one endpoint (its homelab guide is a ~30-minute walkthrough).
- Or go appliance: if you would rather not pool, compare a unified-memory mini PC.
Live GPU prices ($/GB VRAM)
Pooling GPUs for local AI — FAQ
Can you run a 70B model on cheap GPUs instead of one expensive card?
Yes — by pooling several cards across your machines so their VRAM combines. A 70B that needs ~48GB at Q4 can run across, say, a couple of 24GB cards (even mismatched brands) instead of buying a single 96GB workstation card. Free open-source tools like Tightwad pool mismatched GPUs into one endpoint to do exactly this.
What is Tightwad?
Tightwad is open-source (MIT) software that pools mismatched CUDA, ROCm and Apple Metal GPUs across machines into a single OpenAI-compatible endpoint, so a model that fits on no single machine runs across all of them. It uses speculative decoding for speed (~1.86x measured in its benchmarks) and runs entirely on your own hardware with no cloud cost.
Do the GPUs have to match?
No — that is the point. Tightwad is built for mismatched hardware: you can combine NVIDIA, AMD and Apple cards of different sizes over your network. That makes the cheapest-$/GB cards from any brand useful together, rather than forcing one matched set.
How does this change what GPU I should buy?
It shifts the goal from "one card big enough for the model" to "the most cheap VRAM you can pool." Instead of paying a flagship or workstation premium, you can buy several high-$/GB-value cards (see the live tracker) and pool them. Watch $/GB VRAM and buy on dips.