Blog·Aug 27, 2026

Representing thePhysical World throughStructured Language

A world becomes intelligible to evolving intelligence when it can be represented, executed, and verified.

Beyond Pixels

Pixels are evidence of the physical world, not its ontology. A pixel-level observation records how the world appears at a particular moment and from a particular viewpoint, but does not directly specify what exists within it, how it is structured, or what governs its evolution. A video model may predict visually plausible future frames while failing to preserve object identity, geometric consistency, or causal continuity over long horizons^[1]. Moreover, representing every physical detail at the pixel level is both computationally expensive and conceptually unnecessary. Intelligent systems therefore need to move beyond surface-level prediction and infer compact abstractions that explain what exists, how it changes, and why it changes as it does.

Structured Language as World Representations

How should the physical world be represented? Learning a latent representation solely from physical data asks models to rediscover abstractions from scratch^[5]. Yet multimodal language models increasingly organize heterogeneous data—from images and video to audio and action—within a shared conceptual space anchored in human language, suggesting the closest practical approximation to a convergence across modalities^[2]. We believe a scalable physical-world representation should inherit conceptual vocabulary, causal intuitions, and scientific knowledge through which humans understand and model the world.

A rolling ball strikes a ceramic mug at the table edge.

Code · precise

ball = body(m=.20, v=2.1)

mug = body(m=.35, x=.92)

on_contact → apply_impulse(mug)

Execute

Natural language · intuitive

Semantic abstraction

Structural compositionality

Mechanistic executability

Knowledge inheritance

Two systems stand out as particularly powerful and complementary: natural language and code^[3],^[4]. Natural language represents the world through intuitive, semantic concepts across modalities and scenarios. Unlike specialized representations, natural language has evolved across human history to express virtually every domain of entity, event, or situation accumulated across generations. Code as a formal language—used across mathematics, physics, logic, and computer science—represents the world through scientific, precise concepts that support rigorous calculation and deduction. By making variables, assumptions, and rules explicit, they make world models verifiable against evidence, editable in their structure, and controllable through specified conditions and actions. Together, natural language and code provide a broad, interpretable framework for representing the physical world.

Despite their different forms, natural language and code share a deeper principle: we introduce structured language as a unified family of representations characterized by explicit concepts, compositional structures, and executable rules, serving as a bridge between human knowledge and physical worlds.

A structured language provides three intrinsic properties:

Semantic abstraction: It captures entities, states, events, and relations through abstract concepts, enabling representations that generalize beyond individual observations.
Structural compositionality: It organizes concepts into explicit structures, allowing complex worlds to be constructed, decomposed, and modified through their constituent parts.
Mechanistic executability: It expresses rules, constraints, and dynamics in forms that can be simulated, imagined, and verified against physical evidence.

Beyond these intrinsic properties, structured language provides a valuable external connectivity: knowledge inheritance. Because its concepts and rules are shared with human language, science, and code, a physical representation can scale not only by consuming more sensory data, but also by inheriting abstractions humanity has already discovered.

Physical Understanding and Prediction

A representation alone is not yet a world model of intelligence^[5]^[6]. To become operational, it must support two tightly coupled processes: understanding the hidden structure behind observations and predicting how that structure changes under possible actions.

p(s_t,s_{t+1}\mid o_{\leq t},a_t) = \underbrace{p(s_t\mid o_{\leq t})}_{\text{understanding}} \; \underbrace{p(s_{t+1}\mid s_t,a_t)}_{\text{prediction}}, \qquad s_t=\left(s_t^{\mathrm{natural}},s_t^{\mathrm{code}}\right).

Understanding. Understanding constructs and verifies the world behind observations. Guided by priors encoded in natural language and code, the model generates candidate abstractions of the current world, including its entities, states, relations, and governing rules. An agentic process can then verify these candidates against perceptual evidence and internal consistency, revising them until a grounded representation of the present emerges. The result is not another reconstruction of the observation, but a verified hypothesis about the world behind it.

Prediction. Prediction unfolds this verified world under action. The code component of the representation can be executed or simulated to derive precise, mechanistic state transitions, while the natural-language component supports intuitive prediction of qualitative and longer-horizon state changes through semantic and causal reasoning. Their outputs cover complementary levels of abstraction^[5] and together define a distribution over possible future states.

A New Form of General Physical Intelligence

A powerful representation does not merely support individual capabilities; it changes how physical intelligence is organized across reasoning, generation, and interaction. These tasks can be unified as complementary ways of querying, evolving, rendering, and acting upon the same model of the world. In turn, each capability provides distinct signals that can make this shared representation more grounded, complete, and effective.

Figure 3: Structured language unifies physical reasoning, video generation, and embodied interaction.

Physical reasoning. Structured physical data can provide dense supervision across the entire reasoning trajectory of a vision-language model—from grounding entities and states, through inferring relations and dynamics, to explaining and predicting outcomes. This moves visual understanding beyond sparse question answering, training models to construct, reason over, and verify a representation of the world.

Video generation. Explicit world modeling gives video generators a persistent state that carries entities, geometry, and history across frames. By evolving this state before rendering it into pixels, models can generate more physically plausible dynamics, reduce long-horizon drift, and maintain coherent memory as the world unfolds.

Embodied interaction. A physical representation built from structured language can enable a new paradigm of System-2 embodied intelligence: agents that do not merely react to the world, but construct internal models, reason over possible futures, and deliberate before acting^[7]. By providing a shared abstraction across environments, including passive observations, simulation, and reality, such representations allow physical knowledge learned to transfer, offering a path toward scalable embodied learning beyond the limits of real-world interaction.

Our First Exploration

Code-as-World is our first exploration of representing the physical world through structured language. It introduces code as executable world representations and an agentic process for discovering them through iterative simulation and verification.

Executable world representations. Code-as-World describes a physical world through three components of a structured language representation: Composition specifies what exists, including objects, geometry, and physical properties; Evolution specifies how the world changes, including motion, interactions, and events; and Appearance specifies how the world is observed, including the camera, background, and rendering conditions. This representation connects code and natural language through a shared world. Code makes the world executable, unfolding precise physical trajectories. Natural language makes it generative, conditioning video models to render diverse appearances of the same underlying evolution.

Evolving agentic discovery. Code-as-World constructs these representations through an agentic discovery loop that proposes a world hypothesis, instantiates it as a simulator-ready program, executes its dynamics, renders simulated observations, and verifies them against input evidence. The agent then revises the hypothesis and repeats the loop until it recovers a consistent explanation of the underlying physical mechanism.

Together, Code-as-World enables physical data lifting: given an image or video, it transforms perceptual evidence into a structured world representation that can be simulated, rendered, queried, and extended beyond the observed moment. In doing so, it turns raw abundant observations into reusable physical data: explicit states, dynamics, and mechanisms that capture not only what was seen, but the underlying world that could have produced it. This provides scalable physical supervision, enabling our models to achieve state-of-the-art performance on quantitative physical reasoning^[8].

Scaling Structured Physical Experience

The scaling of language models is well underway; the scaling of physical intelligence is only beginning.

The next frontier is not to scale pixels or language in isolation, but to scale structured physical experience. Structured representations are not only how an intelligent system understands the world at a given moment; they also provide the reusable substrate through which experience can accumulate over time. As the system progressively expands its conceptual vocabulary and learned rules by understanding, predicting, and acting in the physical world, its intelligence self-evolves.

The goal, therefore, is not merely to accumulate more experience, but to organize each experience into a growing body of knowledge that enables the next one to be understood more deeply^[9].

References

BibTeX

@misc{mirros2026structuredlanguage,
    title  = {Representing the Physical World through Structured Language},
    author = {{MirroS Team}},
    year   = {2026},
    month  = {Aug},
    url    = {https://mirros.ai/blog/representing-physical-world},
    note   = {Blog post}
}