Local LLM Inference in 2026: Models, Hardware, Performance, and Cost

In this article, local inference means running accessible weights inside a boundary controlled by the owner: a phone, laptop, workstation, or a small office server. A device may expose a local HTTP API to several people. The decisive property is the data and operations boundary, not whether the computer literally sits on a desk.

Local does not have to replace a cloud API. It is compelling when data cannot leave, connectivity is unreliable, network-independent latency matters, a specific model or adapter must be pinned, or utilization is stable enough to amortize the device. Cloud handles bursts, exposes frontier models earlier, and externalizes drivers, cooling, and updates.

Four meanings of efficient

• —The model fits together with context and working buffers without constant CPU offload.
• —First-token and streaming pace match the use case: a background job tolerates seconds; a code editor does not.
• —Quality passes the task set, including format, target language, tools, and refusal behavior.
• —Fully loaded cost, privacy, and operations beat the available cloud or smaller-model alternative.

Local product = exact weights × exact format × runtime × context × concurrency × license. A family name without the other factors is not a deployable configuration.

Three distinct local boundaries

The first is a model running directly on the user's device. The request, cache, and result never leave the phone or laptop; the application works offline and can use local files without a separate server. Power draw, install size, cold-start time, and compatibility with the NPU, Metal, or integrated graphics matter here. Peak speed is rarely the only objective: the model must not drain the battery or evict the primary application from memory.

The second is a personal workstation. The model listens only on localhost or the home network, while a code editor, RAG pipeline, or agent reaches it through a compatible API. This setup can accommodate a discrete GPU, a large SSD, and a persistent server process, but it is still optimized for one active person. Demand usually arrives in bursts: a few minutes of agent activity followed by long idle periods. Fast first-token latency, the ability to release memory, and quiet cooling can therefore matter more than maximum combined throughput across concurrent requests.

The third is a team server inside an office, branch, or private cloud. Data remains on a controlled network, but the system is no longer personal: it needs a queue, authentication, quotas, access logs, redundancy, and a model-update policy. The same computer may physically sit under a desk, yet operationally it is a production service. Comparing it only with a desktop-app subscription is misleading; the real alternative is a cloud API with comparable SLOs, observability, and user count.

Finally, local does not automatically mean private. A model loader may contact an external registry, the shell may send telemetry, an agent tool may call a search engine, and conversation backups may land in the cloud. For sensitive workloads, audit the entire data path: name resolution and weight downloads, request logs, plugins, updates, backups, and outbound connections. The trust boundary is set by the outermost component that sees the original request, not by where matrix multiplication happens.010101I covered the same frame on my channel through Steve Wilson's book — he led the OWASP Top 10 for LLM Applications project: a model is not a standalone function but a system component with users, tools, logs, and permissions. Its supply chain is wider than the usual one: weights, training data, prompt templates, and external providers all need versioning and review too.Knizhny kub · a book on LLM application security (in Russian)01I covered the same frame on my channel through Steve Wilson's book — he led the OWASP Top 10 for LLM Applications project: a model is not a standalone function but a system component with users, tools, logs, and permissions. Its supply chain is wider than the usual one: weights, training data, prompt templates, and external providers all need versioning and review too.Knizhny kub · a book on LLM application security (in Russian)

The weight floor can be estimated before downloading. For a model with P billion parameters and b bits per parameter, the arithmetic minimum is P × b / 8 gigabytes. A GGUF, GPTQ, AWQ, or FP4 artifact also stores scales, group coefficients, and metadata. Runtime tensors, graphs, the tokenizer, a vision/audio encoder, and the operating system consume memory as well.

Q4/Q5 weight quantization reduces artifact size and memory traffic, but does not promise equal speed or quality. llama.cpp Q4_K_M, AWQ W4A16, GPTQ Int4, and Blackwell MXFP4 are different layouts with different kernels. “Four bit” without the artifact and backend is an incomplete configuration.

The second large term is the KV cache. Its size depends on layer count, KV heads, context length, head dimension, cache precision, and the number of requests processed in parallel: each active chat or generation needs its own KV state. A documented 128K, 256K, or one-million-token architecture limit is therefore not a free local API setting.

A memory worksheet, not a “B → GB” lookup table

Start with the actual file size, not the parameter count. If a Q4 artifact occupies 19 GB on disk, use 19 GB in the memory budget rather than the 17.5 GB arithmetic floor. Next, launch the intended runtime once and record resident memory before and immediately after loading: some engines unpack part of the tensors, reserve kernel workspace, or keep copies of service metadata. The post-load reading already includes resident weights, so the file size must not be added again. The file remains a floor and sanity check; the delta from the pre-load process shows the cost of that format and runtime.

Then add KV cache for the working context rather than the advertised maximum. With Multi-Head Attention (MHA), the number of KV heads matches the number of attention heads; with Grouped-Query Attention (GQA), several query heads share a key-value pair; with Multi-Query Attention (MQA), there is one such pair per layer. Two models with the same size and context window can therefore need radically different cache capacity. KV precision is also separate from weight precision: Q4 weights do not imply a four-bit cache. FP16, FP8, and Q8 KV change memory, speed, and sometimes quality in different ways, and support for each mode depends on the engine.

When estimating chat memory, use the distribution of context already occupied when generation starts rather than the model's maximum context window—for example, 256K tokens. A typical request might start with 4K tokens of instructions and documents, the P95 might be 24K, and an occasional repository analysis might reach 80K. Designing the whole system around the rare maximum is expensive; truncating everything to the average is risky. A practical setup guarantees P95, uses a separate profile or lower concurrency for the long tail, and makes the cost of a large context visible to the user.

Concurrency multiplies per-sequence state almost linearly, but not the weights. One process serving four chats does not hold four model copies, yet it does hold four KV sets and their working buffers. Images add tokens and vision-encoder memory; speculative decoding adds a draft model or its cache; LoRA adapters add their own tensors. With unified memory, a nominal 128 GB cannot all be promised to the model: the OS, applications, and display buffers share the same pool. On a discrete card, system RAM does not become VRAM without offload over a slower bus.

Fit test = peak resident memory of the target process set at P95 context and working concurrency + multimodal/draft components not covered by that measurement + OS and application reserve. That peak already includes weights, runtime overhead, and KV cache.

On a 24 GB card, a 27B dense Q4 with a 13.5 GB arithmetic floor looks spacious. The real artifact, long KV, images, and CUDA graphs can consume that reserve. For Qwen3.6-35B-A3B, the Q4 floor is 17.5 GB, but the official FP8 checkpoint is about 37.5 GB. It belongs on 48 GB, while a community Q4 on 24 GB needs short context and quality validation.

Dense and MoE answer different questions

A dense model touches almost all weights for every token. A mixture-of-experts model normally keeps all experts resident while its router activates a subset. A 35B-A3B model therefore stores roughly 35B parameters while activating about 3B per token. Those 3B already include the shared expert, so it should not be added again. This is the source of modern local MoE speed—and the reason active parameters must never be treated as a memory requirement.020202I covered a talk on the accelerator market on my channel, and the same point arrives from the hardware side: a consumer and a datacenter card can have comparable peak compute while their memory bandwidth differs threefold — 1 against 3 TB/s. Capacity and rate really are sold separately.Knizhny kub · where accelerator hardware is heading (in Russian)02I covered a talk on the accelerator market on my channel, and the same point arrives from the hardware side: a consumer and a datacenter card can have comparable peak compute while their memory bandwidth differs threefold — 1 against 3 TB/s. Capacity and rate really are sold separately.Knizhny kub · where accelerator hardware is heading (in Russian)

Open weights no longer form one ladder where more parameters automatically mean a better local product. Multimodality, tool support, attention mechanism, active expert fraction, available quantization formats, and license terms can matter more than a small difference in parameter count. The table is a tiered shortlist, not a quality ranking.

The most practical candidates

On an existing laptop, begin with 3–12B: Gemma 4 E4B/12B, Ministral 3, or Qwen3.5 4B/9B. For a new 24–32 GB GPU system, the center of gravity is gpt-oss-20b, Qwen3.6-27B/35B-A3B, Gemma 4 26B-A4B/31B, and Nemotron 3 Nano. These models can do useful coding, document, and tool work without distributed placement.

Parameter count does not determine quality

Compare models on the form of output that will enter the product. A coding agent must do more than solve an isolated algorithm problem: it must find the right file, preserve repository constraints, produce an applicable patch, call tools correctly, and stop after completing the task. For RAG, check citations to supplied passages and resistance to conflicting instructions inside a document. A Russian-language assistant needs separate tests for morphology, mixed Russian-English code, and local names rather than only a translated general benchmark.

Evaluate the exact weight build, not just the model architecture. Results for BF16 or an official FP8 release do not automatically carry over to a third-party Q4 build: the quantization method and conversion quality can change model behavior. Failures may concentrate in rare capabilities rather than average answers: strict JSON, long reasoning, tool selection, or reading small text in images. Every eval row should therefore record the full filename, hash, chat template, sampling parameters, and tokenizer version. If the exact artifact cannot be reproduced, its score cannot justify a hardware purchase.

Reasoning mode changes the economics as well. A model may answer more accurately while producing several times more hidden or visible tokens, so a high tokens-per-second figure does not guarantee less time to an accepted result. A defined metric is accepted tasks per wall-clock time. For a single-attempt view, expected time to an accepted result is roughly end-to-end cycle latency divided by acceptance rate, including retries. In background batch analysis, waiting several minutes for a single task and automatically retrying failed attempts may be acceptable. In an IDE extension, even a capable model loses if the first useful fragment arrives after the developer has switched to another task.

A license is as hard a filter as memory

“Open weights” is not a single legal regime. Each release has its own terms: gpt-oss and many Qwen and Mistral models use Apache 2.0, while Gemma has separate Google terms. Apache 2.0 generally permits commercial use. If you redistribute the model or a derivative, however, you must include the license text, retain required attribution notices, and review the separate licenses for the tokenizer, code, and other bundled components. Llama, Nemotron, Kimi, and MiniMax each have their own terms, use restrictions, or thresholds that introduce additional obligations. Check the license named in a model registry against the text for that exact release: a family can change terms between versions.

Data and outputs need separate review. Permission to download weights does not grant rights to voices, images, documents, or code sent to the model; speech synthesis requires the voice owner's consent; local RAG does not waive trade-secret rules or retention limits. For a product, store the license version, source page, download date, and restrictions next to the weight hash. That turns a future model swap from an investigation into a routine dependency update.030303I covered a concrete release that shows how this looks in practice on my channel: GigaChat 3 Ultra shipped under MIT — a mixture-of-experts model of roughly 702B parameters with about 36B active per generation step, with standard-tooling compatibility called out as its own line. Licence, regime, and compatibility are declared together; none of them follows from size.Knizhny kub · the GigaChat 3 Ultra tech report (in Russian)03I covered a concrete release that shows how this looks in practice on my channel: GigaChat 3 Ultra shipped under MIT — a mixture-of-experts model of roughly 702B parameters with about 36B active per generation step, with standard-tooling compatibility called out as its own line. Licence, regime, and compatibility are declared together; none of them follows from size.Knizhny kub · the GigaChat 3 Ultra tech report (in Russian)

The 100–120B tier now runs on one 80–128 GB device through MoE and trained FP4 formats. OpenAI explicitly gives a 16 GB fit for gpt-oss-20b and 80 GB for gpt-oss-120b. That is capacity guidance, not a tokens/s promise: DGX Spark, M3 Ultra, and RTX PRO 6000 read memory at very different rates.

The 284B–1T tier is local only in the sense of an owned cluster. DeepSeek V4 Flash publishes an approximately 160 GB mixed-format checkpoint; Kimi K2.7 Code is around 595 GB. Two small nodes may technically start a large quant, but networking and partial offload can make it slower than a strong 20–35B model on one card.

A long window is a capability, not a recommendation

A 262K- or million-token context can help with rare tasks, but it does not replace retrieval and data structure. As the window grows, prompt-processing time, KV cache, and the chance that the model loses an important detail among irrelevant material all increase. Needle-in-a-haystack performance is not the same as analyzing a long contract or repository. Compare at least three modes: a short request without RAG, selected passages with citations, and the complete long document. The middle option often produces a better answer with lower latency.

Multimodal models consume more than text tokens. Resolution, frame count, image downscaling, and page tiling all change the visual sequence length. A model that fits a text test can run out of memory on a batch of scanned pages. Before choosing a 24 or 32 GB configuration, run the model on real PDFs, screenshots, and photos. Include the maximum number of files or pages the system must process in a single request. If the main task is document OCR, a compact specialist model may be cheaper and more reliable than a general model with 30B parameters.

Not every local AI task needs a general generator

The generator is only one part of RAG. Qwen3 Embedding or Granite Embedding can run on CPU or a small GPU, Qwen3-Reranker sees a short top-K, and the large LLM receives only selected passages. This pipeline is often faster and more accurate than placing an entire corpus into a model's advertised 256K context.

Start with memory capacity: it must hold the model weights, KV cache, and runtime overhead at the same time. When the system generates one response, a large dense model is often limited by how quickly its weights can be read from memory. For local chat, memory bandwidth is therefore more useful than advertised TOPS—the theoretical number of trillions of operations per second. A rough upper bound on generation speed is simple: divide measured memory bandwidth by the weight bytes read to produce one token. Peak specification bandwidth gives only an ideal ceiling. Actual speed is lower because attention, context processing, synchronization, and incomplete device utilization add overhead.

Why the current Apple lineup looks unusual

The Mac Studio generation documents M4 Max up to 128 GB and M3 Ultra up to 512 GB, yet the current US configurator offers only 36/64 GB and 96 GB respectively. The MacBook Pro M5 Max is available with 128 GB and up to 614 GB/s, making it the highest-capacity currently orderable Apple option in August 2026. The article separates “the platform supports” from “you can order today”; otherwise a historical specification becomes false purchase advice.040404Personal experience here: I bought a mini-PC on a Ryzen AI Max+ 395 with 128 GB of unified memory, of which 96 GB can be assigned to the GPU, and wrote about it on my channel. Quantized 70–80B models fit comfortably — but ROCm would not come up for me on a non-LTS Ubuntu. That is exactly the budget line a specification sheet never shows.Knizhny kub · a mini-PC on Ryzen AI Max+ 395 (in Russian)04Personal experience here: I bought a mini-PC on a Ryzen AI Max+ 395 with 128 GB of unified memory, of which 96 GB can be assigned to the GPU, and wrote about it on my channel. Quantized 70–80B models fit comfortably — but ROCm would not come up for me on a non-LTS Ubuntu. That is exactly the budget line a specification sheet never shows.Knizhny kub · a mini-PC on Ryzen AI Max+ 395 (in Russian)

Discrete GPU or unified memory

• —RTX 5090 provides 1.79 TB/s and the broadest CUDA path, but is capped at 32 GB and has no NVLink.
• —R9700 and Arc Pro B70 provide 32 GB for less, with more work to validate the exact OS–driver–runtime combination.
• —Mac, Framework, and DGX Spark address 64–128 GB without PCIe offload, but their 256–819 GB/s limits large dense models sooner.
• —RTX PRO 6000 combines 96 GB with 1.79 TB/s, but price and 600 W move it from personal to professional infrastructure.

CPU, NPU, and multiple GPUs each have a niche

A CPU is useful for more than emergency fallback. Small models, embeddings, reranking, and background batch work can use a machine that is already paid for, without a separate card. Performance depends heavily on memory channels and vector instructions, so two computers with the same RAM capacity are not equivalent. For an occasional request, 5–10 tok/s may be more economical than keeping a large GPU idle. llama.cpp can also leave some layers on CPU to test a model larger than VRAM, but treat that as a capacity experiment rather than a promise of interactive speed.

Laptop and phone NPUs are well suited to a stable small model when battery life and silence are critical. Their advertised TOPS cannot be compared directly with a discrete GPU's memory bandwidth: supported operators, weight formats, maximum tensor size, and compiler tooling constrain the model that can actually run. A packaged vendor demo is not enough. Confirm that its tools can convert and run the model architecture you need, then update the model without releasing a new version of the app.

Multiple consumer GPUs add capacity only after the model is partitioned; they do not become one transparent memory pool. RTX 5090 has no NVLink, so layers or tensors move over PCIe, synchronization reduces speed, and the chassis must power and cool two 575 W cards. Two less expensive 32 GB cards may provide 64 GB of capacity, but they do not guarantee the pace of one professional 96 GB card. Such a purchase needs a measured run of the exact engine using the intended partitioning scheme.

Unified memory wins on simplicity: one large file is available to CPU and GPU without manual transfer, and the system is quieter than a typical multi-card workstation. The tradeoff is shared bandwidth and competition with applications. Discrete VRAM offers higher bandwidth and mature specialized kernels, but creates a hard capacity wall. A practical rule follows: for 7–35B and interactive work, start with a fast discrete card; for 70–120B and one user, look at a large unified pool; for a shared server that needs both capacity and concurrency, model a server GPU as a separate system.

Tokens per second are meaningful only next to model, quant, backend, version, occupied-context length, and test mode. llama-bench separates prompt processing (pp) and text generation (tg); tokenization and sampling are excluded. Chat speed also includes queueing, context construction, and streaming delivery.

What can still be inferred

First, a 20–35B MoE on one modern card can stream far faster than reading pace. Second, 128 GB at 256–273 GB/s is primarily a capacity play: gpt-oss-120b remains fast because only 5.1B parameters are active, while dense 70B on the same device will be much slower. Third, context has a cost: the published DGX Spark gpt-oss-120b run falls from about 59 to 43 tok/s as occupied context moves from empty to 32K.

Software can also change the result without a new GPU. In a separate llama.cpp test, operation fusion and concurrent CUDA streams together moved Qwen3 30B-A3B tg128 on RTX 5090 from about 247 to 352 tok/s. The environment variable was named GGML_CUDA_GRAPH_OPT, but the author explicitly noted that the mechanism had nothing to do with CUDA Graphs. This is not a universal speedup; it is a reminder to record the code revision, flags, and warm-up together with the card name.050505I covered the same measurement discipline on my channel through Maxime Labonne's talk at Liquid AI: counting nominal operations is not enough, and picking a model off a leaderboard is worse. What has to be measured is prefill, decode, memory footprint, quantization impact, and cold start — on your own hardware and your own runtime.Knizhny kub · Liquid AI on training small models (in Russian)05I covered the same measurement discipline on my channel through Maxime Labonne's talk at Liquid AI: counting nominal operations is not enough, and picking a model off a leaderboard is worse. What has to be measured is prefill, decode, memory footprint, quantization impact, and cold start — on your own hardware and your own runtime.Knizhny kub · Liquid AI on training small models (in Russian)

What to measure beyond average throughput

Prompt processing and generation stress the system differently. A large PP figure matters for the first pass over a document, but the user experiences time to first token (TTFT), which also includes tokenization, queueing, cache preparation, and the first decode iteration. TG describes the steady stream of one answer but hides pauses between individual tokens. For a code editor, record P50 and P95 TTFT, average inter-token latency, and the longest stall. Two runs with the same tok/s can feel entirely different.

Occupy the context before measuring generation. An empty-cache test describes a hardware ceiling, not a multi-step agent after twenty messages and tool results. Take P50 and P95 input lengths from a real trace and test several output lengths. A short TG32 run may end before heat and clock reduction appear; a long TG512 run shows steady state but does not represent answers that usually stop after 80 tokens. You need both views.

On a shared server, performance under concurrent generation matters more than the speed of one response. Run 1, 2, 4, and 8 requests at the same time; at each concurrency level, measure aggregate generation throughput, P95 latency, and the share of requests meeting the SLO. Continuous batching raises the combined token rate but may worsen one user's inter-token latency. Also test request cancellation. When an agent stops generating an answer it no longer needs, the system should promptly release the resources held by that request: its KV cache and scheduler slot. Otherwise, a benchmark in which every response runs to completion will overstate how many requests the server can handle concurrently.

Finally, measure accepted work rather than computation alone. For code, that means an applicable patch with passing tests; for extraction, a valid schema with correct fields; for RAG, an answer with a verifiable citation. Accepted tasks divided by energy and time produce a metric that can be compared with cloud alternatives. Raw model speed almost always creates false savings when the model needs twice as many retries.060606The same principle is developed in “The Economics of AI Development: From Tokens to Accepted Work”. Cost per accepted task includes not only the model call but also tools, retries, human review, rework, and expected failure loss.06The same principle is developed in “The Economics of AI Development: From Tokens to Accepted Work”. Cost per accepted task includes not only the model call but also tools, retries, human review, rework, and expected failure loss.

Different runtimes add support for a new model at different times, and that support is often partial. A runtime may recognize the tokenizer but process images on CPU instead of GPU; load FP8 weights without fast kernels for the selected GPU; or run the model in llama.cpp while the current vLLM release still does not support it. Before buying, verify the exact model, weight format, and runtime version. Support for a similar model guarantees nothing.

For one user, a sensible baseline is llama.cpp or MLX through LM Studio, Ollama, or a direct CLI. It quickly proves whether the exact artifact fits and what batch-1 tg is. Move a shared Linux API to vLLM, SGLang, or TensorRT-LLM only after architecture support is verified. Continuous batching, authentication, quotas, prefix caching, and P95 now matter more than one local chat.

• •Pin model revision, tokenizer, chat template, and weight-file hash.
• •Pin OS, firmware, driver, runtime commit/container, and every flag.
• •Run a cold start and sustained thermals; record peak RAM/VRAM and wall power.
• •Validate text, images, JSON/tool calls, long context, and cancellation separately.
• •Update one layer at a time and keep rollback; newer does not guarantee support for an older quant.

A reproducible build matters more than the most convenient interface

A graphical shell is convenient for the first experiment, but a production configuration must survive restarts and upgrades. Preserve the launch command or configuration file, the model source and hash, and the engine, driver, and chat-template versions. Verify recovery after a reboot without a manual click, as well as behavior after a corrupted download. If the product depends on a single file from a community registry, keep a verified copy inside your boundary while honoring its license.

An OpenAI-compatible API simplifies server replacement but does not make semantics identical. Model names, token counting, tool-call format, image support, streaming events, and stop reasons differ. Before switching a client, write contract tests for a normal response, JSON, cancellation, timeout, and context overflow. Moving among Ollama, LM Studio, llama.cpp server, and vLLM then does not require a manual retest of the whole application.

For an offline boundary, decide in advance how security updates and new weights will enter. Complete isolation without a delivery process quickly leaves a vulnerable driver and an outdated model. A signed package, hash verification, a staging system, and a controlled maintenance window are more practical. Locality answers who controls a change; it does not require never updating anything.070707I covered the practical side of this choice on my channel through a long piece on local serving: it shows plainly why Ollama is more convenient but slower than llama.cpp, how tensor placement and modes such as cmoe change the result on MoE models, and why the default quant offered by a GUI is rarely the best one.Knizhny kub · getting more out of local LLMs (in Russian)07I covered the practical side of this choice on my channel through a long piece on local serving: it shows plainly why Ollama is more convenient but slower than llama.cpp, how tensor placement and modes such as cmoe change the result on MoE models, and why the default quant offered by a GUI is rarely the best one.Knizhny kub · getting more out of local LLMs (in Russian)

The use case sets minimum useful quality and workload shape. A voice router benefits from an on-device 3B even when 27B is smarter. A coding agent needs stable diff/JSON and a low token interval. A shared RAG server needs concurrency and access control, even if its batch-1 demo looks identical.

1. Existing computer: begin at zero incremental CapEx

An 8–12B Q4 on 16–32 GB can cover private retrieval and drafting. Measure it on the device you already own. Buying a 32 GB GPU for a model that fails your eval merely produces the wrong answer faster.

2. Developer workstation: 24–48 GB

For coding and agents, choose between discrete-GPU pace and unified-memory convenience. RTX 5090 offers very high interactive speed and a mature CUDA path; R9700/B70 provide less expensive 32 GB with more stack work; M4 Pro/M5 Max are quiet systems where the IDE, code, and model share memory. Evaluate repositories, patch format, and tool loops—not only a public coding score.

3. Large personal model: 96–128 GB

Framework Desktop with AMD Ryzen AI Max+ 395 and 128 GB of memory at $3,449 is the least expensive new route in this table to gpt-oss-120b, but 256 GB/s favors MoE over dense 70B. DGX Spark costs more but adds CUDA, 4 TB SSD, and ConnectX-7. MacBook Pro M5 Max and Mac Studio M3 Ultra provide more bandwidth and Metal/MLX, with different price, thermals, and currently orderable SKUs.

4. A small team is already a service

One person tolerates cold loading and can occupy all memory. Five people create a queue, multiple KV caches, and authentication requirements. At that point, return to the separate longread on LLM inference as a distributed system—continuous batching can contribute more than moving to the next GPU generation.

5. Voice, documents, and edge devices

Dictation, commands, and first-pass document processing benefit from a composed pipeline: a small ASR or OCR model, a 2–4B router, and a larger generator only for difficult requests. It lowers average latency and can keep sensitive raw audio or images on the device. Acceptance tests include end-to-end time from signal to action, quality under noise and accents, battery use, false activations, and offline behavior. No single generator metric describes this system.

6. A research lab buys flexibility

Several GPUs or nodes are justified when the objective is to compare formats, develop kernels, study large models, or reproduce an owned cluster. ECC, fast networking, enough SSD capacity, remote management, and the ability to pin versions create the value. If the end goal is one personal chat, the same configuration will almost certainly lose to one fast card or the cloud on price, complexity, and engineering time. Judge a lab by reproducible experiments, not by the largest model it managed to load once.

Local inference avoids per-token API fees but incurs other costs: hardware, electricity, model storage, engineering support, idle capacity, and potential losses from lower model quality. Fully loaded cost includes the host: an RTX 5090 at $1,999 still needs a motherboard, CPU, RAM, fast SSD, PSU, and cooling. Mac and DGX prices are system prices; Framework excludes storage and OS.

Energy should use measured average wall power on the production profile, not TDP. A device averaging 250 W for eight hours per day consumes roughly 730 kWh per year; the bill depends on local tariffs. An always-on shared server approaches 2,190 kWh before cooling. At low utilization, depreciation still dominates electricity.080808I covered the other half of this comparison — the price of the alternative — on my channel through SemiAnalysis: a one-year H100 contract rose roughly 40%, from $1.70 to $2.35 per GPU-hour, in a few months. The cloud term in your payback formula is not a constant either.Knizhny kub · the H100 rental price index (in Russian)08I covered the other half of this comparison — the price of the alternative — on my channel through SemiAnalysis: a one-year H100 contract rose roughly 40%, from $1.70 to $2.35 per GPU-hour, in a few months. The cloud term in your payback formula is not a constant either.Knizhny kub · the H100 rental price index (in Russian)

Three calculations instead of one break-even point

For occasional personal use, account for the full device price: it spends most of its life idle, while API savings remain small. If the computer would be purchased for development anyway, local AI can be assigned only the incremental memory or GPU cost—but that must be labeled as a different scenario. A shared server adds reserve, administration, monitoring, and the cost of downtime. These three bases cannot be collapsed into one “pays back in N months” line.

The depreciation period should match stack risk. A case and PSU may last five years, a GPU three, while the useful model format may change within months. Calculate optimistic, working, and low utilization, then add a case in which more memory is needed after one year. A consumer card's resale value can reduce risk; soldered memory and specialized nodes are harder to upgrade, though they are sold as complete systems.

Cloud cost is not fixed either: prompt caching, batch mode, and a cheaper model all change the denominator. Conversely, a local system may save not tokens but the time needed to approve data handling or the cost of a privacy incident. The final model should separate financial assumptions from non-financial requirements. If privacy is mandatory, break-even selects among local options rather than removing the constraint itself.

A reliable choice starts not with hardware reviews but with the minimum acceptable quality threshold for a specific use case. First define that threshold and evaluate models against it; then choose the exact weight file, estimate the required memory, confirm software-stack compatibility, and only after that select the device. Exclude any model that falls below the threshold. If a model passes but does not fit in memory, choose a different weight format or a device with more memory. If it fits but responds too slowly, first check memory bandwidth and support for the required GPU kernels — and only then consider a different graphics card.090909On how easily an eval gets replaced by a convenient metric, I covered a Sonar talk on my channel: a model passes the tests and still leaves code the team has to clean up afterwards. The lesson transfers here — your own task set and an independent check of the result matter more than any tokens-per-second table.Knizhny kub · why pass rate is no longer enough (in Russian)09On how easily an eval gets replaced by a convenient metric, I covered a Sonar talk on my channel: a model passes the tests and still leaves code the team has to clean up afterwards. The lesson transfers here — your own task set and an independent check of the result matter more than any tokens-per-second table.Knizhny kub · why pass rate is no longer enough (in Russian)

Minimum protocol before ordering

• •Collect 50–200 real requests with P50/P95 lengths, formats, and accepted answers.
• •Choose two model sizes and two exact artifacts each—not ten family names.
• •Rent or borrow nearby hardware for a day; pin versions and replay the same trace.
• •Record quality, TTFT, PP, TG, P95, peak memory, power, and tool failures.
• •Price the complete system under three utilization cases: occasional, workday, and shared server.
• •Buy context reserve, but not capacity for a model you do not plan to use.

Stop conditions protect the budget

Before testing, write down which result cancels the purchase: quality below the threshold, P95 above the SLO, memory without a 15% reserve, an unsuitable license, or no reproducible build. Otherwise a day of tuning makes it easy to mistake the mere fact that the model started for success. If testing shows that no local model reaches the required quality threshold on the target tasks, the rational choice is a hybrid setup or a cloud service. This costs less than optimizing a model that cannot solve the target task at the required quality level.

Five traps that look like shortcuts

The first is buying for the largest possible model. A larger file may answer more slowly, carry an unsuitable license, and follow output formats less reliably, while a specialist 12–24B model completes the job. The second is treating the calculated Q4 weight size as the total memory required for inference. Beyond the weights, memory is consumed by format metadata, the KV cache, working buffers and compute graphs, the encoder, and the OS. A model that “should fit” under simple arithmetic may not fit entirely in accelerator memory and may start using slower system memory instead.

The third is comparing tokens-per-second figures directly across reviews and benchmark tables. If the tests use different models, quantizations, context lengths, batch sizes, or engine versions, their results are not comparable. Treat external measurements as rough guidance; a final comparison requires replaying the same request set on every candidate. The fourth is buying two cards solely for their combined VRAM. Without measuring partitioning, PCIe traffic, and engine support, this is an expensive way to gain capacity while losing responsiveness.

The fifth is building a decision around a promised future. Announced memory, unverified support for a new architecture, or an expected quant is not a working configuration at the time of selection. A shortlist should contain only combinations for which exact weights, available hardware, a compatible runtime, and a reproducible test all exist. Future options belong on a dated watchlist, not in the current SLO or budget.

The strongest antidote to all five traps is a small reproducible testbed. One request archive, one launch script, and one results table let you change one variable at a time: model first, then quant, then runtime, and only then hardware. The purchase becomes a response to an observed bottleneck. Without such a testbed, every new review resets the decision because it compares a different system.

Replay that testbed after a material model or runtime update, but keep the previous result. A regression can look like higher average speed while P95, memory use, or tool quality gets worse. A version is better only if it clears the same thresholds on the same archive. Over time, this log reveals whether the product is limited by the model itself, memory capacity, bandwidth, the software stack, or an obsolete use case. An upgrade can then answer a concrete question instead of a general fear of missing the next hardware release. The decision remains auditable even after the team, vendor, and owner of the local service change over the years.

The practical 2026 default is deliberately plain: an existing machine and 8–14B to validate the product; then 24–32 GB and a 20–35B MoE for a personal agent; 96–128 GB only after a measured 100–120B advantage. Multi-GPU and 284B+ form a research branch, not a mandatory next step.