Code World Model: Building World Models for Computation (Category Software)

I saw an interesting one. report Jacob Kahn, researcher at FAIR (which is part of the banned company Meta in Russia). In this talk, Jacob presents a paradigm shift in teaching models for code generation: the transition from learning in syntax to learning in semantics through world modeling. An extended version of this concept can be read in whitepaper, and also touch in repo GitHub. If we talk about the key ideas of the speech, they are

1Criticism of the current approach Most neural models for code learn from the code itself – token sequences that reflect syntax rather than computation. This allows models to master the “form” of code, but real reasoning about programs requires an understanding of execution and computational dynamics.

2а World modeling for code CWM (code world model) embodies an approach where the model learns not only from static code, but also from program execution data (observation-action trajectories). This includes Python execution tracks. (where actions are Python expressions and observations are local variables) Agent interactions in Docker environments.

3Architecture of the model and its training The model has 32 a billion parameters with a unique system of alternating attention (Local attention with window 8192 token and global attention with window 131072 ratio 3:1). Training consists of three stages: pre-training, mid-training 5 trillions of data tokens, world modeling, and post-training with reinforcement learning via GRPO at ~172 billions of tokens.

4New opportunities CWM can serve as a "neural debugger" - simulate code execution step by step, predict the values of variables without real running, independently test and correct the code.

I was interested in the story of the ambitious name “world model” and it turned out that this name is due to several reasons. The title refers to the classic work of Ha & Schmidhuber. (2018) "World Models", which offered to train models on compressed spatial and temporal representations of the environment in reinforcement learning. CWM takes this idea into the realm of code - the environment is now a computing system. The report and article position CWM as “the foundation for future research and prototyping in AI-driven software systems.” This is a bid to create a new research paradigm, not just an improved model. Meta actively promotes the concept of world models through different projects of the team. Chameleon (Multimodal early-fusion model) before Transfusion (Language modeling and diffusion). Jacob Kahn, the speaker of the report, co-authored both projects:)

If you add a little bit of tar to all of the Sammari, it’s not clear how much the capabilities of world modeling capabilities are transferred to programming languages that the model hasn’t seen, to complex codebases, and to long-term support for the created code. The need for a three-stage RL with self-bootstrapping to achieve the stated results raises the question of whether this is a fundamental advantage or simply a consequence of scale and careful work with data. But in any case, this is an interesting scientific work that has great potential and will be interesting to follow the development of this idea.

#Engineering #AI #Software #ML #Whitepaper