Yann LeCun on JEPA: predicting not pixels, but the state of the world (AI column)

I watched Jan LeCun's lecture.World Models: Enabling the Next AI Revolutionread 29 May 2026 A year at ETH Zurich. It turned out to be almost an exploratory manifesto: it is not enough for the physical world to generate the next token - you need to build an abstract state of the world, predict the consequences of actions and plan on top of it. By the way, Jan Lekun at the end of last year left the company banned in Russia Meta, where he had long been Chief Scientist. (I kind of did. wrote about).

LeCun continues his longstanding dispute with the mainstream in the form of an LLM. In pre-training, the autoregression model minimizes the cross-entropy of the next token – increasing the likelihood of a proper continuation. Instruction tuning (SFT)Training on preferences (RLHF), reinforcement learning and distillation change behavior, but at the core remains the same prediction of the next token. According to LeCun, this is enough for a strong language interface, but not for a model of the physical world. Let me be clear: this is not consensus. Reasoning models, search and tools make the boundary less absolute, but the contrast itself helps to understand why JEPA is needed.

In Joint Embedding Predictive Architecture The object of prediction changes: Context encoder translates the visible part of X into a vector representation. Target encoder encodes the hidden fragment or future state of Y. Predictor for X and action A predicts representation of Y. Loss is the distance between the predicted and target vectors, not the error in each pixel.

Simply put, learning brings together two condensed descriptions of the hidden fragment: predicted from the visible part of the image or video and built from the fragment itself. Details vary between versions of JEPA, but the essence is the same: the useful features of the scene must match, and the model gets the right to discard the unpredictable. To understand where the ball will roll, you do not need to guess the movement of each blade of grass - enough objects, geometry, movement and consequences of action. Such a space is less detailed, but suitable for prediction on the far horizon.

The approach has a well-known trap: if both encoders produce a constant vector, the error is zero, although the model has learned nothing, it is the collapse of representations. In I-JEPA and V-JEPA, the target branch does not receive a gradient and is updated as an exponential moving average of the main network. In LeJEPA, instead of a teacher-student scheme, there is a SIGReg regularizer that pushes embeddings to an isotropic Gaussian distribution. LeCun admits that this variant is not yet scalable as well as I-JEPA and V-JEPA.

When the predictor takes action into account, an action-conditioned world model is obtained: it predicts not just the future, but the consequences of a particular robot movement. The system scrolls through variants in latent space and selects a sequence with minimal energy. (energy) A measure of incompatibility with purpose and limitations. In V-JEPA 2The CEM scheduler minimizes the L1 distance to the target image embedding, performs the first action and replans. In fact, it is an MPC, only the dynamics model is learned from the data.

It doesn't necessarily compete with LLM: V-JEPA 2 Visual encoder connected to Llama 3.1 8B for video question answering. The system can be composite: LLM - language, knowledge and tools; world model - state and dynamics; planner - actions.

Separately from the performance, I was interested in figuring out what is already living in production, and what is still research. Autoregression LLM, multimodal models, embeddings, vision encoders and classic MPC have long lived in production - but as separate bricks, not a ready-made JEPA system.

• V-JEPA 2/2.1 It can be taken as open video encoders for classification, search, prediction of actions or as a visual part of VLM. But open weights, Transformers integration, and a commercial license are not yet rollouts.
• V-JEPA 2-A.C.'s been trained about 62 After hours of robotic video and without adapting to a specific installation, they were launched on Franka manipulators in two laboratories. The robots performed reach, grasp and pick-and-place for visual purposes. This is a study, not a serial robot. Hierarchical planning, long horizons, combining vision with sound and touch, claimed safety through guardrail objectives - also so far research. I didn’t find official sources confirming that JEPA already serves Meta AI, Instagram, glasses or a commercial robot.

At the end, LeCun gives the researchers some deliberately provocative advice: Academic groups do not try to beat the industry by scaling LLM, but go to the unresolved tasks of world models and physical AI. To understand the world, study joint-embedding and energy-based models and not consider pixel generation as a ready-made model of the world. Develop regularized methods and maximize information content instead of relying only on contrastive learning; Learn mainly by observation, and include RL economically - on top of good ideas, when it is impossible without it.

For me, this is not a ban on working with LLMs: they are already solving real problems, and LeCun’s theses have yet to be proven. This is advice about choosing a frontier: go where you need to predict the consequences and manage the process, not compete in the beauty of generation.

P.S. Yesterday, my friends and I were discussing LLM, World Models, AI in development and retail... and I realized that I was not good at World Models – I had to get this video from the watch list, which was not reached and watch it this morning:)

#AI #JEPA #WorldModels #Research #Robotics #Engineering