The honest answer is that almost nobody should pick one. The useful question is which parts of your work belong on your own hardware and which belong behind someone else's API key. Cloud models are still better at the hardest tasks and cheaper for most people, because you pay nothing when the machine sits idle. Local models are now good enough that keeping sensitive work offline no longer means accepting a toy. The deciding factors are memory capacity, token volume, and whether your data is allowed to leave the building.
Everything below is a decision test, not a cheerleading exercise for either side.
What changed: open weights stopped being a hobby
The practical shift is that serious labs now ship weights you can download and run under permissive licences. OpenAI's gpt-oss release is the clearest example: two models published under Apache 2.0, with the company stating that the 120B version runs on a single 80 GB GPU and the smaller one targets consumer hardware. The project's GitHub repository describes gpt-oss-120b as 117B parameters with 5.1B active and gpt-oss-20b as 21B parameters with 3.6B active, both post-trained with MXFP4 quantization so the 20B model fits within 16 GB of memory.
That matters more than any benchmark chart. A mixture-of-experts model with a few billion active parameters reads like a large model on disk and behaves like a small one at generation time. It is the reason a laptop can now answer at reading speed instead of one word per second.
What has not changed: the model you can fit at home is still not the model a provider runs in a data centre. Treat local inference as sufficient for a defined set of tasks, not as a drop-in replacement for the best hosted model on your hardest work.
Memory, not money, sets your local ceiling
Your GPU's compute rating decides how fast tokens come out. Its memory decides whether the model runs at all. Weights have to fit, and the key-value cache for your context sits on top of them.
Rough arithmetic: at 16-bit precision a model needs about two bytes per parameter, so a 70B model wants roughly 140 GB before you have loaded a single token of context. Four-bit quantization cuts that by about four times, which is why nearly every local setup ships quantized by default. Community hardware guides such as this Ollama VRAM breakdown and this quantization-focused guide converge on similar figures.
Translated into hardware classes, the tiers look like this:
Memory available to the model | What runs comfortably at 4-bit | Typical hardware class |
8–12 GB | 7B–8B general models | Mainstream desktop or laptop GPU |
16 GB | 12B–14B models, plus gpt-oss-20b at MXFP4 | Mid-range GPU or 16 GB unified-memory machine |
24–32 GB | 27B–32B models with usable context | High-end consumer GPU |
48–64 GB | 70B at 4-bit, with context trade-offs | Dual GPUs or large unified-memory systems |
80 GB and above | gpt-oss-120b and large MoE models | Data-centre GPU or high-memory unified systems |
The volume where local starts paying for itself
Hosted inference is billed per million tokens, split between input and output, and output usually costs several times more than input. Published rates move constantly, so treat the following as a tier map rather than a price list.
Tier | Input per 1M tokens | Output per 1M tokens | Rates reported in October 2026 |
|---|---|---|---|
Small / fast | $0.10 – $1.00 | $0.40 – $5.00 | Haiku-class listed at $1 / $5 |
Mid | $2.00 – $3.00 | $10 – $15 | Sonnet-class listed at $2 / $10; Gemini 3.1 Pro listed at $2 / $12 for prompts up to 200K tokens |
Frontier | $5.00 and up | $25 – $30 | Opus-class listed at $5 / $25 |
Now the local side. A capable 24–32 GB workstation at $2,500, amortized over three years, costs about $69 a month. Assume three hours of real generation a day at 400 W, idle the rest at 60 W, and electricity at $0.18/kWh: roughly 75 kWh and about $13 a month. Call it $83 a month before your time.
Break-even depends on which cloud tier you are replacing. Using a blended rate of 75% input and 25% output tokens:
The pattern is the point. Replacing a cheap small model with local hardware almost never pays back; replacing heavy frontier usage can pay back in months. And if your entire AI usage is one $20-a-month chat subscription, a GPU purchase is a hobby expense, not a saving.
here your prompt actually goes
Privacy arguments get vague fast, so look at the path a request takes.
With local inference, the text never leaves the device, there is no retention policy to read, no sub-processor list to audit, and no outage that can take your tooling down. That property is binary and it is the strongest argument local has.
With the cloud, the text leaves your network. Major providers offer business terms that exclude API data from training by default, and enterprise agreements can add zero-retention processing and data-residency options. Those controls are real, but they are contractual rather than physical, and the default consumer tiers are not the same as the business ones.
Regulatory exposure adds a second layer: obligations under the EU AI Act and similar frameworks continue to phase in, and the detail of later phases has been subject to amendment. If compliance drives your architecture, verify current requirements with counsel rather than with a vendor blog post, including this one.
What local still loses
Capability is the easy part to admit. Hosted frontier models remain ahead on multi-step reasoning, long-context work, reliable tool calling, and native handling of images, audio and documents. Local open-weight models have closed enough ground for drafting, summarizing, classification, code completion and extraction. They have not closed it on the hard end.
Three constraints people discover after the purchase:
Long context is punishing. The KV cache grows with the prompt. A model that fits with 8K context can run out of memory at 64K, and prompt processing on a long document is slower than generation.
Concurrency is not free. One user with a GPU is fine. Five teammates hitting the same box need a real serving stack such as vLLM, plus batching, queueing and monitoring.
You become the ops team. Driver updates, runtime updates, model upgrades, evaluation after each swap, and a backup plan for the day the machine fails.
Cloud has its own failure modes: deprecations that retire the model your prompts were tuned against, silent behaviour changes between versions, rate limits during traffic spikes, and spend that scales with success.
A decision test for individuals
Start from constraints, not from enthusiasm.
Run local if your work involves material you are contractually barred from sharing; you need inference offline or on an air-gapped machine; you want a fixed monthly cost you cannot overshoot; you already own a 16 GB-plus GPU or a high-memory unified-memory machine; or you are building something where per-request cost would kill the idea.
Stay on the cloud if your usage is spiky; you need the strongest available reasoning for a small number of hard problems; you work with images, audio and long documents; your budget cannot absorb hardware up front; or your time is worth more than the ops work.
The common pattern among heavy users is a split: a small local model handling the high-volume, low-stakes, sensitive traffic, and a hosted model called deliberately for the hard 10%. Tools like Ollama, LM Studio and llama.cpp expose an OpenAI-compatible endpoint, so routing between the two is a base-URL change rather than a rewrite.
A decision test for small businesses
The calculation shifts because three costs appear that individuals ignore: engineering time, availability, and audit evidence.
Question | Points to local | Points to cloud |
|---|---|---|
Does customer or patient data enter the prompt? | Yes, with no approved processor | Yes, with a signed DPA and retention controls |
What happens if the model is unavailable for a day? | Work stops; you need on-site control | Degraded but tolerable |
Who maintains the stack? | Someone already runs servers | Nobody wants to |
How predictable is volume? | Steady, high, continuous | Spiky or seasonal |
How hard are the tasks? | Narrow, repetitive, well-defined | Open-ended reasoning |
What does procurement need? | Physical data control | Vendor certifications and contracts |
Two numbers decide most small-business cases. The first is monthly token volume against the break-even table above. The second is the hourly cost of the person who would maintain the machine; at typical salaries, four hours a month of maintenance costs more than most teams spend on API calls.
A workable middle path: keep production on a hosted API with business terms, and run one local box for the narrow category of work that cannot legally or contractually go out. That box earns its keep through compliance, not through saved tokens.
Questions readers ask before buying a GPU
Can a local model replace a frontier cloud model for coding? Partly. Open-weight coding models handle completion, refactoring, test generation and explanation of existing code well on a 24–32 GB machine. They are weaker on long multi-file reasoning and agentic workflows that chain dozens of tool calls. Most developers who try the split keep local inference for inline completion and route architectural or debugging sessions to a hosted model.
How much VRAM do I actually need to start? Sixteen gigabytes is the first genuinely comfortable tier. It covers 12B–14B models at four-bit quantization plus workable context, and OpenAI states gpt-oss-20b runs within 16 GB of memory thanks to MXFP4 quantization. Eight to twelve gigabytes still runs 7B–8B models well enough for summarizing, classification and drafting. Below that, expect compromises you will notice.
Is running models locally cheaper than paying per token? Only at volume. Against a cheap hosted small model, a roughly $83-a-month local setup needs about 98 million tokens a month to break even. Against frontier-tier rates, the figure drops to about 8 million. Both assume $2,500 of hardware over three years and $0.18/kWh electricity, and neither counts your maintenance time.
Does using a cloud API mean my data trains the model? Not by default on business API tiers from the major providers, which exclude API inputs and outputs from training and offer retention controls. Consumer chat subscriptions follow different defaults and different settings. Check the specific product terms attached to your account, because the policy that applies is the one in your contract, not the one in the marketing page.
What about fine-tuning on private data? Fine-tuning an open-weight model locally keeps both the data and the resulting weights under your control, which is the main reason regulated teams choose it. The trade-off is cost and skill: you need a training-capable GPU, a clean labelled dataset, and an evaluation process. For most tasks, retrieval over your own documents beats fine-tuning at a fraction of the effort.
Spend two weeks measuring before you spend anything
Pull your token counts from the provider dashboard. Multiply by the blended rate for the tier you actually use. Compare that number against roughly $83 a month plus the hours you would spend maintaining a machine. If the API bill is lower, the hardware is a preference, not an investment, and that is a perfectly honest reason to buy it.
If privacy is the driver rather than cost, the arithmetic is irrelevant and the decision is already made: install a runtime, start with a model that fits your memory with room for context, and build your test set before you build the workflow. If the local model fails that test set, you have learned something cheaply. If it passes, you have a system that keeps working when the network does not.
Discussion
Loading discussion…