GigaChat 3 Ultra Preview is a heavy open source (AI column)

Just finished reading today. article Gathering about GigaChat release 3 Ultra, which was published in late November on Habre. I found it interesting and worth studying, but if you briefly summarize the theses from the article, the guys rolled out a new generation of open-weight models under MIT:

• GigaChat 3 Ultra Preview A flagship MoE model at ~702B parameters, of which ~36B is active at the generation step. This is the first such large, initially Russian-language open-source model of this scale, compatible with the standard OSS toolkit. (HuggingFace, vLLM, sglang, etc.). - GigaChat 3 Lightning - Compact ~10B-MoE for local start-up and quick learning.

Next articleThe guys told about some moments - Data. Pretrain-hull inflated to ~14 trillion tokens: 10 language (from Chinese to Uzbek and Kazakh), a lot of code/mathematics and5,5 trillionth (Q&A, reverse-prompts, Olympiad programming, etc.). - Data Infrastructure We have launched our own open-source YT cluster: 10 000 nuclei and>5 PB storage so that sampling and tokenization are performed in minutes instead of days. Yesterday I was at Giga Salut, where I had an interesting conversation with the guys about this topic. (How did you move to YT and what were the results?) - Architecture. Ultra is a huge MoE model inspired by DeepSeek V3: 256 experts, MTP (multi‑token prediction), MLA (multi‑head latent attention)A full stack compatible with existing OSS tools for inferencing and learning. Training. It was difficult to learn the MoE model: the wild volume of communication between the GPU, the imbalance of the load on experts, infrastructure hell with checkpoints on the Internet. 10TB and benchmarks that require dozens of GPUs. - Alignment The general assembly line looked like this. -- Stage 1.5 a large dialog pretrain so that the model communicates normally; RL on chains of reasoning (Chain‑of‑Thought RL); SFT on hand-licked datasets. Ultra Preview does not have a CoT-RL stage.

Also in the model added B2C features: Python-code interpreter, redesigned search (essentially a finished RAG layer) long-term memory of the user.

What is the breakthrough of this release - Scale.. GigaChat Ultra is the largest open-source-LLM project in Russia and Europe and one of the top companies in the world.5 open world models by number of parameters Learning from scratch. This is not an additional training of the Western model: weights and dataset - their own, the model natively learns from Russian and current data, without inheriting other people's restrictions. Compatibility with the OSS stack. The architecture is as close as possible to DeepSeek V3, so further training and deployment can be built on existing tools. (vLLM, sglang, Megatron, Torchtitan, etc.). - Quality.. Ultra confidently overtakes GigaChat 2 Max on key benchmarks (MERA, MMLU-Pro, GSM8K, HumanEval+, etc.) He is a leader in Russian language tests. But so far there are no results of a large number of other benches.

Anyway, the guys from Sber are good. It is nice to see open releases and technical reports about the technologies of Russian companies.

#AI #ML #Software #Engineering #Architecture #Infrastructure #Data