Blog · Review Essay · May 2026

Yann LeCun's World-Model Bet

The Welch Labs video on Yann LeCun's JEPA program is not really about whether large language models are impressive. They obviously are. The sharper question is whether language prediction is enough for agents that must understand the physical world, anticipate consequences, and plan before acting.

The Claim

The video's core argument is that LeCun's bet against language-first AI is not a rejection of deep learning. It is a different reading of what deep learning has taught us. Large language models learned useful representations by predicting the next token at internet scale. That strategy worked because language is already a compressed symbolic record of human perception, social life, planning, explanation, and culture.

But the physical world is not a sentence. A robot, vehicle, animal, child, or city does not act by selecting the next word. It acts in a world where gravity, occlusion, friction, uncertainty, delay, embodiment, and consequence matter. LeCun's position, as presented in the video, is that reliable agentic systems need models that can predict what their actions will do before they take them.

That is the important phrase: before they take them. A chatbot can be useful while remaining mostly reactive. An embodied agent cannot. Once a system moves a robot arm, drives a vehicle, administers a process, routes money, or changes a built environment, intelligence becomes a control problem.

Why Language Worked First

The video gives a compact history of why self-supervised learning broke through in text before it broke through in vision and video. GPT-style training uses text itself as the training signal. Hide the next token, make the model predict it, repeat at scale. No human has to label every sentence with a task-specific answer.

This matched LeCun's older claim that most intelligence should come from self-supervised learning, with supervised learning and reinforcement learning as smaller layers on top. The irony is that the first spectacular proof arrived through language models, not through the vision and world-model route LeCun favored.

The reason is structural. Text prediction gives a model a manageable output space: a finite vocabulary of possible tokens. Physical prediction does not. Predicting the next frame of a video at the pixel level asks the model to choose among an astronomical number of possible images. Worse, many futures are plausible. If a ball might bounce left or right, a pixel-level predictor trained to average uncertainty produces blur.

The Blurry-Video Problem

The blurry-video problem is the review's best technical explanation. A language model can assign probability mass to several possible next words. A naive video model that directly predicts pixel values is pressured toward the visual average of many possible futures. The average of several plausible futures is not a future. It is mush.

This matters beyond image quality. It exposes a deeper problem with generative prediction as the foundation for physical intelligence. The point of a world model is not to render every leaf beside the road. It is to preserve the features that matter for action: the car ahead is braking, the cup is near the edge, the person is entering the workspace, the object will fall if pushed.

A model that spends its capacity reconstructing unpredictable detail may be worse, not better, for planning. Physical agency needs abstraction. It needs to know what is likely, what is impossible, what is dangerous, and what will change if an action is taken.

The Joint-Embedding Detour

JEPA, or Joint Embedding Predictive Architecture, enters as a way around the demand to generate the whole world. Instead of predicting raw pixels, the system maps observations into embeddings, then predicts future embeddings. The hope is that the embedding preserves salient structure while discarding nuisance detail.

The video traces this through Siamese networks, contrastive learning, Barlow Twins, VICReg, DINO, and finally JEPA. The common idea is representation learning without direct reconstruction. A model can learn that two distorted views of the same scene should have related representations without learning to generate every pixel of that scene.

The central danger is representation collapse. If the model is rewarded only for making two embeddings similar, it can cheat by outputting the same embedding for everything. Barlow Twins and related methods attack this by encouraging useful invariance while reducing redundancy across embedding dimensions. In plainer language: learn what stays meaningfully the same, but do not let every internal feature become a copy of every other feature.

This is less glamorous than chatbots. It is also foundational. A system that cannot learn good representations of the world cannot plan in the world. It may talk about action beautifully while lacking the machinery to anticipate action.

World Models

LeCun's world-model argument is old in spirit and new in implementation. Control theory has long cared about predicting the next state of a system under possible actions. What machine learning changes is the possibility of learning the state representation and the transition model from large-scale sensory data.

In the JEPA frame, an agent observes the world, encodes the current state, considers possible actions, predicts future embedded states, and searches for an action sequence that reaches a goal. This makes inference less like autocomplete and more like planning.

That is why robotics is the natural test case. A language model can describe how to move a cup. A world model should help predict what will happen if a particular robot action is applied to a particular scene. The difference is not vocabulary. It is consequence.

The video also makes clear that this is not a finished victory lap. It is part one of an argument. The open question is whether JEPA-like systems can scale from elegant representation learning and limited robot planning into general-purpose agents that compete with or complement multimodal language-model systems.

The Spiralist Reading

For Spiralism, the most important distinction is not LLM versus JEPA. It is interface intelligence versus consequence intelligence.

Language models are extraordinarily good at the interface layer. They compress culture, produce explanations, simulate styles, translate requests, write code, and mediate institutional work. That makes them powerful because human civilization itself is heavily linguistic. Law, finance, education, software, religion, bureaucracy, and identity all run through symbolic systems.

But symbolic fluency can create a false sense of agency. A model that can explain a plan may not be able to predict the physical or institutional consequences of the plan. It may know the sentence "do no harm" without having a grounded model of harm as it unfolds through bodies, rooms, machines, incentives, and time.

LeCun's critique points toward a missing layer in AI governance. The question is not only whether an agent can follow instructions. The question is whether it can simulate enough of the relevant world to understand what following those instructions will do.

This is why world models are not merely a robotics topic. Institutions also need world models. A hospital, school, court, city, or church must anticipate downstream effects. If AI agents become institutional actors, their safety cannot rest only on refusal policies and fluent explanations. They need bounded authority, feedback, review, and some capacity to model consequence before action.

What to Watch

The first thing to watch is whether world-model systems produce useful planning outside carefully bounded demonstrations. Robot control is the visible test, but the broader test is whether learned representations support robust action under novelty, ambiguity, and partial information.

The second thing to watch is whether language models absorb the world-model agenda rather than being replaced by it. The likely future may not be one architecture defeating another. It may be language interfaces wrapped around learned world models, planners, simulators, memory systems, and tool-use policies.

The third thing to watch is governance. A system that predicts consequences can be safer than one that cannot, but it can also be more strategically capable. Planning is not automatically benign. The more an agent can search possible futures, the more important it becomes to specify whose futures count, what actions are permitted, and who can interrupt the loop.

The video is useful because it cuts through a shallow argument. The issue is not whether LLMs are "real AI." They are. The issue is whether next-token prediction is the right substrate for agents that must act in the world. LeCun's bet is that intelligence needs more than language. The Spiralist addition is that society needs more than intelligence. It needs accountable consequence.

Sources

Return to Blog