Extract more from local LLMs (AI column)

I rarely write about articles on Habr, but I could not resist it - I really liked the big and practical. article About launching local LLMs. This topic is interesting to me, but I have not experimented much before, limiting myself to running standard models in my LMStudio on the thick AMD Ryzen AI Max+. 395. And this article allows you to squeeze more speed, quality and context out of your iron, for example, the author explains on his fingers why ollama is convenient but slower than llama.cpp, as well as what quantization is and what it is and what it gives. If you talk a little more about the main topics, they are

1Difference Between Dense and MoE Models In a Dense model, all parameters are always active, so it is often more stable and “smarter”, but slower. The MoE has only a fraction of the experts active, so such a model can be faster but requires a more tidy run. And here's where the fun begins: if you just unload some of the layers on the GPU, as Ollama often does, the graphics card can be full, but it's inefficient. And llama.cpp is able to intelligently decompose tensors and turn on modes like cmoe / ncmoe, which on MoE models gives a very noticeable gain.

2Quantization - what is it and what gives To be honest, this part seemed super helpful to me, as the author explained that the standard Q4KM default choice is not always the best option. Modern dynamic quanta such as UD-Q4\ K\ XL can give more quality at a close size, and sometimes Q4\ K\ M is approximately at the level of the lighter UD-Q3\ K\ XL. And this is an important practical conclusion: when running locally, you should not blindly take a default quantum from Ollama or LM Studio. It's better to see if there's a version from Unsloth, Ubergarm or other more recent options.

3Use of local models for coding The author shows that even the strong quantization of UD-Q2 \ K \ XL can be working enough to generate and refine code: the model collected prototypes of games, corrected bugs and worked through an agent. This doesn’t mean that Q2 is always the best choice, but it does mean that the old “below Q4 don’t look” rule is no longer always true.

Plus in the article a lot of practical tips that are interesting to test in practice. Try llama.cpp instead of Ollama, especially for MoE models. Look not only for Q4\ K\ M, but also for dynamic quanta of UD-Q4\ K\ XL/UD-Q3\ K\ XL/IQ variants. Remember that a smaller quantum is not always faster because i-quantums may require more calculations per CPU.

• Use it. -fit, -cmoe, -ncmoe Do not try to guess the optimal layout with your hands;
• Increase. -ub and -bIf the speed of processing a large context is important; Connect a small draft model for speculative decoding if the task is similar to coding or translation. Free VRAM: Browser, Windows and unnecessary apps are easy to eat 2–3 GBs that could fit a model or context Look at the release date of the model, not the old lists of “best LLM”, where still pop up models that have long lost to new generations.

The effect of these settings can be very tangible. In the article llama.cpp on the MoE model showed a speed of approximately 3 Twice as high as Ollama on the same iron. Configuration of context processing gave a multiple increase in PP. Speculative decoding accelerated the Dense model to approximately 1.5 Once in code and translation. And the right choice of quantum allowed either to get more quality in the same size, or fit the model into the available VRAM.

All in all, this is a good article for engineers who have already tried local LLMs and are faced with the feeling that “the model looks good, but it works slowly.”

P.S. Ehh, I'd like to dig through my hardware and try these tips at the May holidays, but I'm leaving with my wife and kids for London on Sunday, so I'll put it off for the second May holidays. And with the subscribers of the channel will be the following: 2 For weeks, she shares tourist photos and her thoughts on AI:)

#AI #LLM #LocalLLM #Engineering #Performance #Agents #llamacpp