Local LLM Inference in 2026: Models, Hardware, Performance, and Cost

In this article, local inference means running accessible weights inside a boundary controlled by the owner: a phone, laptop, workstation, or a small office server. A device may expose a local HTTP API to several people. The decisive property is the data and operations boundary, not whether the computer literally sits on a desk.

Local does not have to replace a cloud API. It is compelling when data cannot leave, connectivity is unreliable, network-independent latency matters, a specific model or adapter must be pinned, or utilization is stable enough to amortize the device. Cloud handles bursts, exposes frontier models earlier, and externalizes drivers, cooling, and updates.

Four meanings of efficient

• —The model fits together with context and working buffers without constant CPU offload.
• —First-token and streaming pace match the use case: a background job tolerates seconds; a code editor does not.
• —Quality passes the task set, including format, target language, tools, and refusal behavior.
• —Fully loaded cost, privacy, and operations beat the available cloud or smaller-model alternative.

Local product = exact weights × exact format × runtime × context × concurrency × license. A family name without the other factors is not a deployable configuration.

A trace here is the saved requests and sequence of model and tool calls that can be replayed. A quant is a particular quantized weight build, such as Q4: compare the exact file, not just its bit width. The runtime is the software that loads the weights and executes requests, including its version and settings.

Three distinct local boundaries

The first is a model running directly on the user's device. The request, cache, and result never leave the phone or laptop; the application works offline and can use local files without a separate server. Power draw, install size, cold-start time, and compatibility with the NPU, Metal, or integrated graphics matter here. Peak speed is rarely the only objective: the model must not drain the battery or evict the primary application from memory.

The second is a personal workstation. The model listens only on localhost or the home network, while a code editor, RAG pipeline, or agent reaches it through a compatible API. This setup can accommodate a discrete GPU, a large SSD, and a persistent server process, but it is still optimized for one active person. Demand usually arrives in bursts: a few minutes of agent activity followed by long idle periods. Fast first-token latency, the ability to release memory, and quiet cooling can therefore matter more than maximum combined throughput across concurrent requests.

The third is a team server inside an office, branch, or private cloud. Data remains on a controlled network, but the system is no longer personal: it needs a queue, authentication, quotas, access logs, redundancy, and a model-update policy. The same computer may physically sit under a desk, yet operationally it is a production service. Comparing it only with a desktop-app subscription is misleading; the real alternative is a cloud API with comparable SLOs, observability, and user count.

Finally, local does not automatically mean private. A model loader may contact an external registry, the shell may send telemetry, an agent tool may call a search engine, and conversation backups may land in the cloud. For sensitive workloads, audit the entire data path: name resolution and weight downloads, request logs, plugins, updates, backups, and outbound connections. The trust boundary is set by the outermost component that sees the original request, not by where matrix multiplication happens.010101I covered the same frame on my channel through Steve Wilson's book — he led the OWASP Top 10 for LLM Applications project: a model is not a standalone function but a system component with users, tools, logs, and permissions. Its supply chain is wider than the usual one: weights, training data, prompt templates, and external providers all need versioning and review too.Knizhny kub · a book on LLM application security01I covered the same frame on my channel through Steve Wilson's book — he led the OWASP Top 10 for LLM Applications project: a model is not a standalone function but a system component with users, tools, logs, and permissions. Its supply chain is wider than the usual one: weights, training data, prompt templates, and external providers all need versioning and review too.Knizhny kub · a book on LLM application security

The weight floor can be estimated before downloading. For a model with P billion parameters and b bits per parameter, the arithmetic minimum is P × b / 8 gigabytes. A GGUF, GPTQ, AWQ, or FP4 artifact also stores scales, group coefficients, and metadata. Runtime tensors, graphs, the tokenizer, a vision/audio encoder, and the operating system consume memory as well.

Q4/Q5 weight quantization reduces artifact size and memory traffic, but does not promise equal speed or quality. llama.cpp Q4_K_M, AWQ W4A16, GPTQ Int4, and Blackwell MXFP4 are different layouts with different kernels. “Four bit” without the artifact and backend is an incomplete configuration.

The second large term is the KV cache. Its size depends on layer count, KV heads, context length, head dimension, cache precision, and the number of requests processed in parallel: each active chat or generation needs its own KV state. A documented 128K, 256K, or one-million-token architecture limit is therefore not a free local API setting.

A memory worksheet, not a “B → GB” lookup table

Start with the actual file size, not the parameter count. If a Q4 artifact occupies 19 GB on disk, use 19 GB in the memory budget rather than the 17.5 GB arithmetic floor. Next, launch the intended runtime once and record resident memory before and immediately after loading: some engines unpack part of the tensors, reserve kernel workspace, or keep copies of service metadata. The post-load reading already includes resident weights, so the file size must not be added again. The file remains a floor and sanity check; the delta from the pre-load process shows the cost of that format and runtime.

Then add KV cache for the working context rather than the advertised maximum. With Multi-Head Attention (MHA), the number of KV heads matches the number of attention heads; with Grouped-Query Attention (GQA), several query heads share a key-value pair; with Multi-Query Attention (MQA), there is one such pair per layer. Two models with the same size and context window can therefore need radically different cache capacity. KV precision is also separate from weight precision: Q4 weights do not imply a four-bit cache. FP16, FP8, and Q8 KV change memory, speed, and sometimes quality in different ways, and support for each mode depends on the engine.

When estimating chat memory, use the distribution of context already occupied when generation starts rather than the model's maximum context window—for example, 256K tokens. A typical request might start with 4K tokens of instructions and documents, the P95 might be 24K, and an occasional repository analysis might reach 80K. Designing the whole system around the rare maximum is expensive; truncating everything to the average is risky. A practical setup guarantees P95, uses a separate profile or lower concurrency for the long tail, and makes the cost of a large context visible to the user.

Concurrency multiplies per-sequence state almost linearly, but not the weights. One process serving four chats does not hold four model copies, yet it does hold four KV sets and their working buffers. Images add tokens and vision-encoder memory; speculative decoding adds a draft model or its cache; LoRA adapters add their own tensors. With unified memory, a nominal 128 GB cannot all be promised to the model: the OS, applications, and display buffers share the same pool. On a discrete card, system RAM does not become VRAM without offload over a slower bus.

Fit test: memory capacity ≥ (peak resident memory of the target process set at P95 context and working concurrency + multimodal/draft components not covered by that measurement) × 1.15–1.25 + OS and application reserve. This is the formula above with a measured M_peak: it already includes weights, runtime overhead, and KV cache.

On a 24 GB card, a 27B dense Q4 with a 13.5 GB arithmetic floor looks spacious. The real artifact, long KV, images, and CUDA graphs can consume that reserve. The opposite mistake happens too: the official FP8 checkpoint of a 35B-parameter MoE takes about 35–38 GB and already targets 48 GB, even though its Q4 floor is 17.5 GB. The file format moves the hardware tier more than the “35B” in the name does.

Dense and MoE answer different questions

A dense model touches almost all weights for every token. A mixture-of-experts model normally keeps all experts resident while its router activates a subset. A 35B-A3B model therefore stores roughly 35B parameters while activating about 3B per token. Those 3B already include the shared expert, so it should not be added again. This is the source of modern local MoE speed—and the reason active parameters must never be treated as a memory requirement.020202I covered a talk on the accelerator market on my channel, and the same point arrives from the hardware side: a consumer and a datacenter card can have comparable peak compute while their memory bandwidth differs threefold — 1 against 3 TB/s. Capacity and rate really are sold separately.Knizhny kub · where accelerator hardware is heading02I covered a talk on the accelerator market on my channel, and the same point arrives from the hardware side: a consumer and a datacenter card can have comparable peak compute while their memory bandwidth differs threefold — 1 against 3 TB/s. Capacity and rate really are sold separately.Knizhny kub · where accelerator hardware is heading

Open weights no longer form one ladder where more parameters automatically mean a better local product. Multimodality, tool support, attention mechanism, active expert fraction, available quantization formats, and license terms can matter more than a small difference in parameter count. The selection order is therefore the reverse of the usual one: first the use case and quality threshold, then the memory tier, and only then the exact weight file. Tiers with current examples live in the snapshot, in section 9; this section covers the rules for choosing among them.

Parameter count does not determine quality

Compare models on the form of output that will enter the product. A coding agent must do more than solve an isolated algorithm problem: it must find the right file, preserve repository constraints, produce an applicable patch, call tools correctly, and stop after completing the task. For RAG, check citations to supplied passages and resistance to conflicting instructions inside a document. A Russian-language assistant needs separate tests for morphology, mixed Russian-English code, and local names rather than only a translated general benchmark.

Evaluate the exact weight build, not just the model architecture. Results for BF16 or an official FP8 release do not automatically carry over to a third-party Q4 build: the quantization method and conversion quality can change model behavior. Failures may concentrate in rare capabilities rather than average answers: strict JSON, long reasoning, tool selection, or reading small text in images. Every eval row should therefore record the full filename, hash, chat template, sampling parameters, and tokenizer version. If the exact artifact cannot be reproduced, its score cannot justify a hardware purchase.

Reasoning mode changes the economics as well. A model may answer more accurately while producing several times more hidden or visible tokens, so a high tokens-per-second figure does not guarantee less time to an accepted result. An accepted task is one whose result clears the predefined quality check. The useful metric is the number of those tasks per wall-clock time.

For attempts of comparable duration, a rough estimate of time to an accepted result is the mean duration of one attempt divided by the share of attempts that pass acceptance.

In background batch analysis, waiting several minutes for a single task and automatically retrying failed attempts may be acceptable. In an IDE extension, even a capable model loses if the first useful fragment arrives after the developer has switched to another task.

A license is as hard a filter as memory

“Open weights” is not a single legal regime. Each release has its own terms: gpt-oss, Gemma 4, and many Qwen and Mistral models use Apache 2.0. Apache 2.0 generally permits commercial use. If you redistribute the model or a derivative, however, you must include the license text, retain required attribution notices, and review the separate licenses for the tokenizer, code, and other bundled components. Llama, Nemotron, Kimi, and MiniMax each have their own terms, use restrictions, or thresholds that introduce additional obligations. Check the license named in a model registry against the text for that exact release: a family can change terms between versions.

Data and outputs need separate review. Permission to download weights does not grant rights to voices, images, documents, or code sent to the model; speech synthesis requires the voice owner's consent; local RAG does not waive trade-secret rules or retention limits. For a product, store the license version, source page, download date, and restrictions next to the weight hash. That turns a future model swap from an investigation into a routine dependency update.030303I covered a concrete release that shows how this looks in practice on my channel: GigaChat 3 Ultra shipped under MIT — a mixture-of-experts model of roughly 702B parameters with about 36B active per generation step, with standard-tooling compatibility called out as its own line. Licence, regime, and compatibility are declared together; none of them follows from size.Knizhny kub · the GigaChat 3 Ultra tech report03I covered a concrete release that shows how this looks in practice on my channel: GigaChat 3 Ultra shipped under MIT — a mixture-of-experts model of roughly 702B parameters with about 36B active per generation step, with standard-tooling compatibility called out as its own line. Licence, regime, and compatibility are declared together; none of them follows from size.Knizhny kub · the GigaChat 3 Ultra tech report

A long window is a capability, not a recommendation

A 262K- or million-token context can help with rare tasks, but it does not replace retrieval and data structure. As the window grows, prompt-processing time, KV cache, and the chance that the model loses an important detail among irrelevant material all increase. Needle-in-a-haystack performance is not the same as analyzing a long contract or repository. Compare at least three modes: a short request without RAG, selected passages with citations, and the complete long document. The middle option often produces a better answer with lower latency.

Multimodal models consume more than text tokens. Resolution, frame count, image downscaling, and page tiling all change the visual sequence length. A model that fits a text test can run out of memory on a batch of scanned pages. Before choosing a 24 or 32 GB configuration, run the model on real PDFs, screenshots, and photos. Include the maximum number of files or pages the system must process in a single request. If the main task is document OCR, a compact specialist model may be cheaper and more reliable than a general model with 30B parameters.

Not every local AI task needs a general generator

The generator is only one part of RAG. An embedding model can run on CPU or a small GPU, a reranker sees a short top-K, and the large LLM receives only selected passages. This pipeline is often faster and more accurate than placing an entire corpus into a model's advertised 256K context. The same holds for speech, document recognition, and command routing: a compact specialist model with hundreds of millions or a few billion parameters is often cheaper, faster, and more stable than a general one. Concrete models for these use cases are collected in section 9.

Start with memory capacity: it must hold the model weights, KV cache, and runtime overhead at the same time. When the system generates one response, a large dense model is often limited by how quickly its weights can be read from memory. For local chat, memory bandwidth is therefore more useful than advertised TOPS—the theoretical number of trillions of operations per second. A rough upper bound on generation speed is simple: divide measured memory bandwidth by the weight bytes read to produce one token. Peak specification bandwidth gives only an ideal ceiling. Actual speed is lower because attention, context processing, synchronization, and incomplete device utilization add overhead.

Discrete GPU or unified memory

• —A discrete card offers the highest bandwidth and the most mature kernels, but its VRAM is a hard wall: whatever does not fit crosses PCIe and generation slows sharply.
• —Cards with the same capacity from another vendor are often cheaper, but support for the exact model and quant must be validated on the OS–driver–runtime combination.
• —Unified memory—Apple silicon, AMD Ryzen AI Max, NVIDIA GB10—addresses 64–128 GB or more without PCIe offload, but its bandwidth is several times lower than a fast discrete card's and limits large dense models sooner.
• —A professional card combines large capacity with high bandwidth, but price and power move the decision from personal to infrastructure class.

The second item is not a theoretical caveat. A specification promises capacity and bandwidth, not that the chosen runtime will come up on your OS and driver version on the first attempt. The time it takes to assemble a working stack is a budget line, just like the device price.040404Personal experience here: I bought a mini-PC on a Ryzen AI Max+ 395 with 128 GB of unified memory, of which 96 GB can be assigned to the GPU, and wrote about it on my channel. Quantized 70–80B models fit comfortably — but ROCm would not come up for me on a non-LTS Ubuntu. That is exactly the budget line a specification sheet never shows.Knizhny kub · a mini-PC on Ryzen AI Max+ 39504Personal experience here: I bought a mini-PC on a Ryzen AI Max+ 395 with 128 GB of unified memory, of which 96 GB can be assigned to the GPU, and wrote about it on my channel. Quantized 70–80B models fit comfortably — but ROCm would not come up for me on a non-LTS Ubuntu. That is exactly the budget line a specification sheet never shows.Knizhny kub · a mini-PC on Ryzen AI Max+ 395

CPU, NPU, and multiple GPUs each have a niche

A CPU is useful for more than emergency fallback. Small models, embeddings, reranking, and background batch work can use a machine that is already paid for, without a separate card. Performance depends heavily on memory channels and vector instructions, so two computers with the same RAM capacity are not equivalent. For an occasional request, 5–10 tok/s may be more economical than keeping a large GPU idle. llama.cpp can also leave some layers on CPU to test a model larger than VRAM, but treat that as a capacity experiment rather than a promise of interactive speed.

Laptop and phone NPUs are well suited to a stable small model when battery life and silence are critical. Their advertised TOPS cannot be compared directly with a discrete GPU's memory bandwidth: supported operators, weight formats, maximum tensor size, and compiler tooling constrain the model that can actually run. A packaged vendor demo is not enough. Confirm that its tools can convert and run the model architecture you need, then update the model without releasing a new version of the app.

Multiple consumer GPUs add capacity only after the model is partitioned; they do not become one transparent memory pool. NVIDIA consumer cards have no NVLink, so layers or tensors move over PCIe, synchronization reduces speed, and the chassis must power and cool two cards drawing 300–600 W each. Two less expensive 32 GB cards may provide 64 GB of capacity, but they do not guarantee the pace of one professional 96 GB card. Such a purchase needs a measured run of the exact engine using the intended partitioning scheme.

Unified memory wins on simplicity: one large file is available to CPU and GPU without manual transfer, and the system is quieter than a typical multi-card workstation. The tradeoff is shared bandwidth and competition with applications. Discrete VRAM offers higher bandwidth and mature specialized kernels, but creates a hard capacity wall. A practical rule follows: for 7–35B and interactive work, start with a fast discrete card; for 70–120B and one user, look at a large unified pool; for a shared server that needs both capacity and concurrency, model a server GPU as a separate system.

How to read a hardware price

A price in a review is useful only with five attributes: date, market, taxes, exact configuration, and availability state. A “from” price refers to base memory, not the configuration the model needs. A graphics-card price excludes the host, PSU, and cooling. A manufacturer's suggested price and the store price differ, and a deposit pre-order is not the same as a device in stock. In the snapshot, every price carries these attributes; when moving it to another market, add VAT, import costs, and the cost of warranty.

The used market stays out of the snapshot tables for the same reason. Second-hand RTX 3090 and 4090 cards, with their 24 GB, are the main real alternative to new cards on memory per dollar, but there is no warranty, the mining and overclocking history is unknown, memory and fans arrive with accumulated wear, and listing prices shift week to week and region to region—they cannot be pinned to a date. They should not be excluded from a personal decision: the same capacity and bandwidth arithmetic applies, with repair risk and inspection time added mentally to the price.

Tokens per second are meaningful only next to model, quant, backend, version, occupied-context length, and test mode. llama-bench separates prompt processing (pp) and text generation (tg); tokenization and sampling are excluded. Chat speed also includes queueing, context construction, and streaming delivery.

How to read other people's benchmarks

A published measurement answers a narrow question: what speed did this model, this weight file, and this software version deliver on this device at this context length. Two rows can be compared only when every condition matches except the one under study. Even a paired post by one author with identical flags stops being a controlled A/B if the systems use different CPUs or drivers. Vendor claims are useful as an upper reference, but their testbed and settings are usually not reproducible. Other people's numbers therefore calibrate scale, not a ranking: “dollars per token” cannot be computed across rows from different experiments.

Several conclusions still carry over between hardware generations. An MoE model with a few billion active parameters on one modern card streams faster than a person reads. A large unified-memory pool with moderate bandwidth is primarily a capacity play: a 100–120B MoE stays fast on it, while a dense 70B is noticeably slower. Finally, context has a cost: generation slows as context fills, so an empty-cache test overstates expectations. The numbers behind these conclusions on the snapshot date are in section 11.

Software changes the result without new hardware

Software can change the result without a new GPU. For example, in a separate llama.cpp test, operation fusion and concurrent CUDA streams together moved Qwen3 30B-A3B tg128 on RTX 5090 from about 247 to 352 tok/s. The environment variable was named GGML_CUDA_GRAPH_OPT, but the author explicitly noted that the mechanism had nothing to do with CUDA Graphs. This is not a universal speedup; it is a reminder to record the code revision, flags, and warm-up together with the card name.050505I covered the same measurement discipline on my channel through Maxime Labonne's talk at Liquid AI: counting nominal operations is not enough, and picking a model off a leaderboard is worse. What has to be measured is prefill, decode, memory footprint, quantization impact, and cold start — on your own hardware and your own runtime.Knizhny kub · Liquid AI on training small models05I covered the same measurement discipline on my channel through Maxime Labonne's talk at Liquid AI: counting nominal operations is not enough, and picking a model off a leaderboard is worse. What has to be measured is prefill, decode, memory footprint, quantization impact, and cold start — on your own hardware and your own runtime.Knizhny kub · Liquid AI on training small models

What to measure beyond average throughput

Prompt processing and generation stress the system differently. A large PP figure matters for the first pass over a document, but the user experiences time to first token (TTFT), which also includes tokenization, queueing, cache preparation, and the first decode iteration. TG describes the steady stream of one answer but hides pauses between individual tokens. For a code editor, record P50 and P95 TTFT, average inter-token latency, and the longest stall. Two runs with the same tok/s can feel entirely different. An acceptance threshold needs a numeric form—for example, “TTFT under 0.5 s for IDE completion”; the number is illustrative, and your threshold comes from your scenario, not from this article.

Occupy the context before measuring generation. An empty-cache test describes a hardware ceiling, not a multi-step agent after twenty messages and tool results. Take P50 and P95 input lengths from a real trace and test several output lengths. A short TG32 run may end before heat and clock reduction appear; a long TG512 run shows steady state but does not represent answers that usually stop after 80 tokens. You need both views.

On a shared server, performance under concurrent generation matters more than the speed of one response. Run 1, 2, 4, and 8 requests at the same time; at each concurrency level, measure aggregate generation throughput, P95 latency, and the share of requests meeting the SLO. Continuous batching raises the combined token rate but may worsen one user's inter-token latency. Also test request cancellation. When an agent stops generating an answer it no longer needs, the system should promptly release the resources held by that request: its KV cache and scheduler slot. Otherwise, a benchmark in which every response runs to completion will overstate how many requests the server can handle concurrently.

Finally, measure accepted work rather than computation alone. For code, that means an applicable patch with passing tests; for extraction, a valid schema with correct fields; for RAG, an answer with a verifiable citation. Accepted tasks divided by energy and time produce a metric that can be compared with cloud alternatives. Raw model speed almost always creates false savings when the model needs twice as many retries.060606The same principle is developed in “The Economics of AI Development: From Tokens to Accepted Work”. Cost per accepted task includes not only the model call but also tools, retries, human review, rework, and expected failure loss.06The same principle is developed in “The Economics of AI Development: From Tokens to Accepted Work”. Cost per accepted task includes not only the model call but also tools, retries, human review, rework, and expected failure loss.

Different runtimes add support for a new model at different times, and that support is often partial. A runtime may recognize the tokenizer but process images on CPU instead of GPU; load FP8 weights without fast kernels for the selected GPU; or run the model in llama.cpp while the current vLLM release still does not support it. Before buying, verify the exact model, weight format, and runtime version. Support for a similar model guarantees nothing. Which runtime versions are current on the snapshot date, and what changed in them, is collected in section 11.

For one user, a sensible baseline is llama.cpp or MLX through LM Studio, Ollama, or a direct CLI. It quickly proves whether the exact artifact fits and what batch-1 tg is. Move a shared Linux API to vLLM, SGLang, or TensorRT-LLM only after architecture support is verified. Continuous batching, authentication, quotas, prefix caching, and P95 now matter more than one local chat.

• •Pin model revision, tokenizer, chat template, and weight-file hash.
• •Pin OS, firmware, driver, runtime commit/container, and every flag.
• •Run a cold start and sustained thermals; record peak RAM/VRAM and wall power.
• •Validate text, images, JSON/tool calls, long context, and cancellation separately.
• •Update one layer at a time and keep rollback; newer does not guarantee support for an older quant.

A reproducible build matters more than the most convenient interface

A graphical shell is convenient for the first experiment, but a production configuration must survive restarts and upgrades. Preserve the launch command or configuration file, the model source and hash, and the engine, driver, and chat-template versions. Verify recovery after a reboot without a manual click, as well as behavior after a corrupted download. If the product depends on a single file from a community registry, keep a verified copy inside your boundary while honoring its license.

An OpenAI-compatible API simplifies server replacement but does not make semantics identical. Model names, token counting, tool-call format, image support, streaming events, and stop reasons differ. Before switching a client, write contract tests for a normal response, JSON, cancellation, timeout, and context overflow. Moving among Ollama, LM Studio, llama.cpp server, and vLLM then does not require a manual retest of the whole application.

For an offline boundary, decide in advance how security updates and new weights will enter. Complete isolation without a delivery process quickly leaves a vulnerable driver and an outdated model. A signed package, hash verification, a staging system, and a controlled maintenance window are more practical. Locality answers who controls a change; it does not require never updating anything.070707I covered the practical side of this choice on my channel through a long piece on local serving: it shows plainly why Ollama is more convenient but slower than llama.cpp, how tensor placement and modes such as cmoe change the result on MoE models, and why the default quant offered by a GUI is rarely the best one.Knizhny kub · getting more out of local LLMs07I covered the practical side of this choice on my channel through a long piece on local serving: it shows plainly why Ollama is more convenient but slower than llama.cpp, how tensor placement and modes such as cmoe change the result on MoE models, and why the default quant offered by a GUI is rarely the best one.Knizhny kub · getting more out of local LLMs

Local inference avoids per-token API fees but incurs other costs: hardware, electricity, model storage, engineering support, idle capacity, and potential losses from lower model quality. Fully loaded cost includes the host: a separate graphics card still needs a motherboard, CPU, RAM, fast SSD, PSU, and cooling. Complete systems such as a Mac or DGX Spark carry a system price, while do-it-yourself kits are sometimes sold without storage and an OS.

Energy should use measured average wall power on the production profile, not TDP. A device averaging 250 W for eight hours per day consumes roughly 730 kWh per year; the bill depends on local tariffs. An always-on shared server approaches 2,190 kWh before cooling. At low utilization, depreciation still dominates electricity.080808I covered the other half of this comparison — the price of the alternative — on my channel through SemiAnalysis: a one-year H100 contract rose roughly 40%, from $1.70 to $2.35 per GPU-hour, in a few months. The cloud term in your payback formula is not a constant either.Knizhny kub · the H100 rental price index08I covered the other half of this comparison — the price of the alternative — on my channel through SemiAnalysis: a one-year H100 contract rose roughly 40%, from $1.70 to $2.35 per GPU-hour, in a few months. The cloud term in your payback formula is not a constant either.Knizhny kub · the H100 rental price index

Three calculations instead of one break-even point

For occasional personal use, account for the full device price: it spends most of its life idle, while API savings remain small. If the computer would be purchased for development anyway, local AI can be assigned only the incremental memory or GPU cost—but that must be labeled as a different scenario. A shared server adds reserve, administration, monitoring, and the cost of downtime. These three bases cannot be collapsed into one “pays back in N months” line.

The depreciation period should match stack risk. A case and PSU may last five years, a GPU three, while the useful model format may change within months. Calculate optimistic, working, and low utilization, then add a case in which more memory is needed after one year. A consumer card's resale value can reduce risk; soldered memory and specialized nodes are harder to upgrade, though they are sold as complete systems.

Cloud cost is not fixed either: prompt caching, batch mode, and a cheaper model all change the denominator. Conversely, a local system may save not tokens but the time needed to approve data handling or the cost of a privacy incident. The final model should separate financial assumptions from non-financial requirements. If privacy is mandatory, break-even selects among local options rather than removing the constraint itself.

A reliable choice starts not with hardware reviews but with the minimum acceptable quality threshold for a specific use case. First define that threshold and evaluate models against it; then choose the exact weight file, estimate the required memory, confirm software-stack compatibility, and only after that select the device. Exclude any model that falls below the threshold. If a model passes but does not fit in memory, choose a different weight format or a device with more memory. If it fits but responds too slowly, first check memory bandwidth and support for the required GPU kernels — and only then consider a different graphics card.090909On how easily an eval gets replaced by a convenient metric, I covered a Sonar talk on my channel: a model passes the tests and still leaves code the team has to clean up afterwards. The lesson transfers here — your own task set and an independent check of the result matter more than any tokens-per-second table.Knizhny kub · why pass rate is no longer enough09On how easily an eval gets replaced by a convenient metric, I covered a Sonar talk on my channel: a model passes the tests and still leaves code the team has to clean up afterwards. The lesson transfers here — your own task set and an independent check of the result matter more than any tokens-per-second table.Knizhny kub · why pass rate is no longer enough

Minimum protocol before ordering

Use one row per exact configuration in the final table: model and weight file, software and version, device, acceptance rate, time to first response, full-cycle P95, peak memory, and price. This shows which candidate meets your thresholds and what it costs.

• •Collect 50–200 real requests with P50/P95 lengths, formats, and accepted answers.
• •Choose two model sizes and two exact artifacts each—not ten family names.
• •Rent or borrow nearby hardware for a day; pin versions and replay the same trace.
• •Record quality, TTFT, PP, TG, P95, peak memory, power, and tool failures.
• •Price the complete system under three utilization cases: occasional, workday, and shared server.
• •Buy context reserve, but not capacity for a model you do not plan to use.

Stop conditions protect the budget

Before testing, write down which result cancels the purchase: quality below the threshold, P95 above the SLO, memory without a 15% reserve, an unsuitable license, or no reproducible build. Otherwise a day of tuning makes it easy to mistake the mere fact that the model started for success. If testing shows that no local model reaches the required quality threshold on the target tasks, the rational choice is a hybrid setup or a cloud service, provided external transfer is permitted for those data. If it is prohibited, revisit the task, quality threshold, or local configuration. Optimizing a model that cannot solve the target task at the required quality level will not resolve the problem by itself.

Five traps that look like shortcuts

The first is buying for the largest possible model. A larger file may answer more slowly, carry an unsuitable license, and follow output formats less reliably, while a specialist 12–24B model completes the job. The second is treating the calculated Q4 weight size as the total memory required for inference. Beyond the weights, memory is consumed by format metadata, the KV cache, working buffers and compute graphs, the encoder, and the OS. A model that “should fit” under simple arithmetic may not fit entirely in accelerator memory and may start using slower system memory instead.

The third is comparing tokens-per-second figures directly across reviews and benchmark tables. If the tests use different models, quantizations, context lengths, batch sizes, or engine versions, their results are not comparable. Treat external measurements as rough guidance; a final comparison requires replaying the same request set on every candidate. The fourth is buying two cards solely for their combined VRAM. Without measuring partitioning, PCIe traffic, and engine support, this is an expensive way to gain capacity while losing responsiveness.

The fifth is building a decision around a promised future. Announced memory, unverified support for a new architecture, or an expected quant is not a working configuration at the time of selection. A shortlist should contain only combinations for which exact weights, available hardware, a compatible runtime, and a reproducible test all exist. Future options belong on a dated watchlist, not in the current SLO or budget.

The protocol above checks all five traps: keep a request archive, launch script, and results table, changing one variable at a time. Replay the test after a material model or software update and keep the previous result. A new version must clear the same quality, P95, and memory thresholds on the same archive; higher average speed alone does not justify an update.

The practical default is deliberately plain and barely depends on the year: an existing machine and a small model to validate the product itself; then one card or memory pool of 24–32 GB and a model that fits it with headroom for a personal agent; 96–128 GB only after a measured advantage of a larger tier. Multi-GPU and models with hundreds of billions of parameters form a research branch, not a mandatory next step. The snapshot below shows which models and devices stand behind these steps today.

The tiers below were selected by the rules from section 3: use case, memory tier, exact weight file, license. This is a shortlist, not a quality ranking: models share a row because of capacity and the shape of work, not test scores. Every model was checked against its Hugging Face card on 31 October 2026: parameter count, license, and the size of the official weight files.

What changed since August

• —The 27–31B tier changed the most. Qwen3.8 27B (Apache 2.0; text, image, and video) replaced Qwen3.6 27B; Meta released the dense Muse Glimmer 30B under Apache 2.0 with an official 16.8 GB Q4_K_M GGUF and an explicit 24 GB recommendation; IBM released Granite 4.2 3B/8B/30B with reasoning modes.
• —NVIDIA released the Nemotron 3.5 line under the OpenMDW-1.1 license. Lightning 30B-A3B with a 1M window continues the Nemotron 3 Nano class; Super VL 120B-A12B takes 75 GB in NVFP4 and, according to its card, fits one DGX Spark.
• —Liquid AI released LFM3 with 1.7B parameters (about 1 GB at four bits by Liquid's estimate) and the LFM2.5-24B-A2B MoE. Their license bars commercial use by companies with $10M or more in annual revenue—which also applies to the earlier LFM2.5.
• —No new small Qwen, gpt-oss, Ministral, or Devstral models appeared this quarter: Qwen3.5 2B/4B/9B, Qwen3.6-35B-A3B, gpt-oss-20b/120b, Ministral 3, and Devstral Small 2 remain current. Llama 4 Scout has not been updated and dropped off the list.
• —The top tier moved even further from “local”: DeepSeek V4.1 Flash grew to 552B and a 510 GB file, Mistral Large 4 has about 1T parameters, and Kimi K3 has 2.8T. GigaChat 3.5 Ultra (432B-A28B, MIT) replaced GigaChat 3 Ultra, and Yandex published open weights for the first time—the base model AliceAI-Foundation-80B-A3B, without instruction tuning.

The most practical candidates

On an existing laptop, begin with 3–12B: Gemma 4 E4B/12B, Qwen3.5 4B/9B, Ministral 3, or Granite 4.2 8B. For a new 24–32 GB system, the center of gravity is gpt-oss-20b, Qwen3.6-35B-A3B, Gemma 4 26B-A4B, Nemotron 3.5 Lightning, and the dense Qwen3.8 27B, Gemma 4 31B, or Muse Glimmer 30B at Q4. These models can do useful coding, document, and tool work without distributed placement.

An example of how the file format moves the hardware tier: Qwen3.6-35B-A3B has a 17.5 GB Q4 floor, but its official FP8 checkpoint is about 37.5 GB. It targets 48 GB, while a community Q4 on 24 GB needs short context and quality validation. The opposite example is Nemotron 3.5 Bolt 74B-A9B: NVIDIA published an official 24 GB package with experts compressed to roughly two bits. That build fits one card, but its quality must be validated separately from the full version.

The 100–125B and 284B+ tiers

The 100–125B tier runs on one 80–128 GB device through MoE and trained FP4 formats. OpenAI explicitly gives 16 GB for gpt-oss-20b and 80 GB for gpt-oss-120b, and NVIDIA lists one DGX Spark for Nemotron 3.5 Super VL. That is capacity guidance, not a tokens/s promise: DGX Spark, Mac Studio, and RTX PRO 6000 read memory at very different rates. The tier name can mislead as well: Qwen3.8-Flash-Next activates 6B of 125B, but with its n-gram embeddings it stores 180B, and its official FP8 weighs 185.5 GB.

Models from 284B up are local only in the sense of an owned cluster. DeepSeek V4 Flash ships as a file of about 167 GB, its successor V4.1 Flash at about 510 GB, and Kimi K3 at about 1.5 TB. Two small nodes may technically start a large quant, but networking and partial offload can make it slower than a strong 20–35B model on one card.

Licenses on the snapshot date

Most models in the table use Apache 2.0 or MIT: gpt-oss, Gemma 4, Qwen3.5/3.6/3.8 27B, Granite 4.2, Muse Glimmer 30B, Ministral 3, Devstral Small 2, DeepSeek, GLM, and GigaChat. Nemotron 3.5 (OpenMDW-1.1), Qwen3.8-Flash-Next (Qwen Community License 1.0, with a separate license for model-as-a-service use), LFM ($10M revenue threshold), Mistral Large 4 (a model-as-a-service restriction), and Kimi K3 carry their own terms. The license names in this paragraph are a snapshot-date guide; check the license text of the exact release before use.

Specialist models by use case

The snapshot covers new hardware with a public US price on 31 October 2026, before tax. The used market is excluded for the reasons in section 4. Phones and NPU laptops are excluded because results depend too heavily on the device and software. If you already own such a device, start there—you may not need new hardware. Points in Figure 8 are specifications, not speed on one shared task.

What changed since August

• —On 25 August Apple refreshed its desktop Macs. Mac mini now ships with M6 (up to 32 GB) and M5 Pro (up to 64 GB, 307 GB/s); Mac Studio with M5 Max (up to 128 GB, 614 GB/s) and M5 Ultra (96/256/512 GB, 1.2 TB/s). The August oddity, when large Mac Studio configurations could not be ordered, is gone: the 512 GB option has been orderable again since 28 October at $22,299.
• —Ryzen AI Max+ PRO 495 moved from the watchlist into the table: Framework Desktop with 192 GB is open for pre-order at $6,799, shipping from November. Capacity grows by half, while bandwidth only moves from 256 to 273 GB/s.
• —NVIDIA raised prices: DGX Spark Founders Edition went from $4,699 to $6,950 and is sold out; RTX PRO 6000 from $13,250 to $16,000. Since 23 October partners have sold a 64 GB DGX Spark from $4,999—cheaper, but the 100–125B class no longer fits.
• —RTX 50 Super has still not launched; press reports say it was postponed because of the GDDR7 shortage. At the end of September TrendForce forecast DRAM contract prices rising 10–15% in Q4, and retail shows it: the lowest US price of RTX 5060 Ti is $766 against a $429 MSRP.

The practical consequence: at the end of 2026, unified memory is the only mainstream way to get more than 96 GB on one device without a server budget, but its price rises with the price of memory. If you need a device for a specific 100–125B model, compare a 128 GB Mac Studio M5 Max, a 128–192 GB Ryzen AI Max+, and a 128 GB DGX Spark by full system price and on your own trace, not by a specification line.

The table records the versions checked on 31 October 2026 and the main change in each runtime since August. A version number here is not advice to upgrade; it is an example of what to record next to every measurement: a month later the same model on the same hardware may run noticeably faster or stop loading.

The quarter's main change concerns Macs. Since version 0.40, Ollama runs supported models on Mac through MLX rather than llama.cpp by default, and LM Studio added its own Mac engine. After an update the same command may run on a different runtime, so before comparing “before” and “after,” check which engine actually loaded the model.

On AMD, ROCm moved to version 10 and officially supports Ryzen AI Max+ 395 and R9700, yet in published measurements Vulkan in llama.cpp still keeps pace with it on generation.

The measurements below follow the rule from section 5: the publication names the model, quant, runtime version, and context length. No new paired RTX 5090 and R9700 measurement appeared after August, so that pair is unchanged. Open tables for Mac Studio M5 Ultra and Ryzen AI Max+ 395 at several context lengths were added. Rows that could not be rechecked were removed: Framework's gpt-oss-120b result and Intel's Arc Pro B70 result.

What can still be inferred

• —A 20–35B MoE on one modern card or on Mac Studio M5 Ultra generates several times faster than a person reads: 127–210 tok/s at short context.
• —Long context is expensive everywhere. On M5 Ultra gpt-oss-20b generation drops from 210 to 85 tok/s at 128K; on DGX Spark gpt-oss-120b drops from 59 to 43 tok/s at 32K. Measure speed at the length your trace actually contains.
• —The runtime can give more than the next purchase. On M5 Ultra, MLX processes a 32K input 1.7 times faster than llama.cpp and keeps 107 tok/s versus 85 at 128K. On DGX Spark a forum user got 61 tok/s in vLLM versus 42 in Ollama on the same model.
• —128 GB at 256–273 GB/s is primarily a capacity play: gpt-oss-120b is fast there thanks to 5.1B active parameters, while a dense 70B model on the same device will be several times slower.

The use case sets minimum useful quality and workload shape. A voice router benefits from an on-device 3B even when 27B is smarter. A coding agent needs stable diff/JSON and a low token interval. A shared RAG server needs concurrency and access control, even if its batch-1 demo looks identical. The model short-lists in the rows are assembled by tool support, license, and community adoption, not by measured quality: only your own eval confirms quality.

1. Existing computer: begin at zero incremental CapEx

An 8–12B Q4 on 16–32 GB can cover private retrieval and drafting. Measure it on the device you already own. Buying a 32 GB GPU for a model that fails your eval merely produces the wrong answer faster.

2. Developer workstation: 24–48 GB

For coding and agents, choose between discrete-GPU pace and unified-memory convenience. RTX 5090 offers very high interactive speed and a mature CUDA path; R9700/B70 provide less expensive 32 GB with more stack work; a 48–64 GB Mac mini M5 Pro or a Mac Studio M5 Max is a quiet system where the IDE, code, and model share memory. What decides the choice is not a public coding score but behavior on your repositories: patch format and tool loops.

3. Large personal model: 96–192 GB

Framework Desktop with AMD Ryzen AI Max+ 395 and 128 GB of memory at $3,449 remains the least expensive new route in this table to gpt-oss-120b, but 256 GB/s favors MoE over dense 70B. The 192 GB PRO 495 version costs almost twice as much and adds capacity, not speed. The 128 GB DGX Spark has risen to $6,950 and adds CUDA, 4 TB SSD, and ConnectX-7. A 128 GB Mac Studio M5 Max offers more bandwidth and MLX for about $5,100.

4. A small team is already a service

One person tolerates cold loading and can occupy all memory. Five people create a queue, multiple KV caches, and authentication requirements. At that point, return to the separate longread on LLM inference as a distributed system—continuous batching can contribute more than moving to the next GPU generation.

5. Voice, documents, and edge devices

Dictation, commands, and first-pass document processing benefit from a composed pipeline: a small ASR or OCR model, a 2–4B router, and a larger generator only for difficult requests. It lowers average latency and can keep sensitive raw audio or images on the device. Acceptance tests include end-to-end time from signal to action, quality under noise and accents, battery use, false activations, and offline behavior. No single generator metric describes this system.

6. A research lab buys flexibility

Several GPUs or nodes are justified when the objective is to compare formats, develop kernels, study large models, or reproduce an owned cluster. ECC, fast networking, enough SSD capacity, remote management, and the ability to pin versions create the value. If the end goal is one personal chat, the same configuration will almost certainly lose to one fast card or the cloud on price, complexity, and engineering time. Judge a lab by reproducible experiments, not by the largest model it managed to load once.

The default as of 31 October 2026 is deliberately plain: an existing machine and 8–14B to validate the product; then 24–32 GB and a 20–35B MoE or a dense 27–31B model for a personal agent; 96–192 GB only after a measured advantage of the 100–125B class. Multi-GPU, a 512 GB Mac Studio, and 284B+ models form a research branch, not a mandatory next step.