HarnessEval: The Era of Harness for Benchmarking

Benchmarks set the North Star of technology, guiding the frontier of exploration and development. Yet evaluating world models in Physical RSI remains fragile, particularly when assessing physical causality, geometric consistency, and observation realism. While humans can easily analyze and spot artifacts in generated content, this capability has never been successfully automated or achieved in existing benchmarks. As a consequence, existing evaluation benchmarks offer limited persuasive power: the reported scores can be neither explained nor verified, and provide little insight into where and why a model fails.

In this blog, we introduce a concept from the LLM ecosystem to the benchmark community: the Harness. A harness is more than a code wrapper; it provides the robust scaffolding to power fully agentic pipelines by formalizing complex human workflows, such as evidence gathering, tool use, and reasoning. Human evaluation is also a workflow, and can therefore be harnessed. When humans evaluate a generated world, we do not merely glance at a static image; we intuitively locate objects, track object permanence over time, and verify causal relationships as well as geometric constraints. We conceptualize this workflow as an agentic pipeline through Harness: a benchmark agent that can spawn specialized sub-agents, equipping each with specific contexts and diagnostic tools to investigate different aspects of models' output. We name our agentic workflow HarnessEval, and it works like Sherlock Holmes, who meticulously discovers hidden clues, connects disparate pieces of evidence, and applies rigorous reasoning to formulate a definitive deduction. This system moves beyond traditional Q&A probing with a static rubric in existing benchmarks; it is an intelligent agent that decomposes the problem, assembles the right tools, and reasons for every evaluation case. The benchmark will evolve into a dynamic, executable agentic system.

HarnessEval-W: A Harness Example for World Model Evaluation

To realize this vision of agentic evaluation, we introduce HarnessEval-W, our first agentic benchmark for evaluating interactive world models. The core of HarnessEval-W is an intelligent agent that dynamically plans the evaluation strategy, actively spawns sub-agents to interpret the intent of the actions performed in the world model, hunts for spatiotemporal evidence in the generation, leverages applicable tools, and generates inspectable reasoning sequences for its final scoring. To design the agentic process in HarnessEval-W, we first establish the evaluation axes of world models, and develop the skill libraries that our harness can operate on. Our HarnessEval-W then evaluates world models on these axes, leveraging the skills.

Three Evaluation Axes

Following common practices in the literature^[1][2], we formulate the physical world model as the future prediction conditioned on historical observations and user-specified actions. Any models that can achieve these capabilities can be evaluated by our benchmark, including but not limited to bidirectional video diffusion models and autoregressive video models. We establish three evaluation axes for world models:

• Observation Quality: Observation quality refers to whether the rendered output is visually reliable. It covers perceptual coherence, structural plausibility, and realism for the videos.
• Transition Correctness: Transition correctness focuses on whether the state transition faithfully executes the requested actions at the appropriate time. We evaluate three transition types: Exploratory transition focuses on changing the observed position in the world. Intentional transition refers to changing a specified entity, relation, or event. Physical transition evaluates the plausibility of physical control.
• World Persistence: World Persistence evaluates whether the sequence of predicted states remains coherent in the evolving world. It includes three representative settings. Drift Resistance tests whether overall layout, style, and appearance remain consistent across a long rollout. Revisit Consistency tests whether a location or object remains the same after the observer leaves and returns. Offscreen Evolution examines whether an endogenous process continues while temporarily unseen. Persistence does not require the world to remain unchanged everywhere. Instead, it requires stable properties to remain invariant while dynamic properties continue to evolve consistently with actions and time.

Hierarchical Agentic Evaluation Workflow

The evaluation of world models is highly context-dependent. Every test case is a unique environment with different physical actions, temporal structures, and observation states. The core of HarnessEval-W is an intelligent agent that dynamically plans the evaluation strategy, spawns sub-agents to interpret the world from specialized perspectives, and finally generates verifiable reasoning traces for the final score. Every step is tailored to the specific world being evaluated. Through this agentic workflow, the system rigorously validates whether the gathered evidence answers the evaluation question, ensuring an interpretable, reproducible, and customized evaluation for every single case.

Skill Libraries. Guided by our evaluation axes, we developed a comprehensive skill library where each skill assesses a specific dimension of the world model. Currently, the library comprises 9 distinct skills. We designed this framework to be extensible, allowing us to dynamically integrate new skills as model capabilities evolve and evaluation demands expand.

1. Render Quality InspectorAesthetic quality, imaging quality, temporal flicker, motion smoothness, artifact and readability gates
2. Physical Plausibility InspectorVisual plausibility, causal fidelity, support/contact, judgeability gate
3. Viewpoint Trajectory VerifierCamera pose recovery, adaptive reference trajectory, Accuracy / Consistency / NavScore
4. Intentional Change VerifierStage 1 expected specification, Stage 2 visible-change questions, preservation diagnostics
5. Physical Response VerifierStage 1 expected specification, Stage 2 response questions, causal-order checks
6. Physical Dynamics VerifierSelected engine checks, trajectory constraints, process consistency, task-specific kinematics
7. Drift Degradation AnalyzerChunk-wise render, motion, appearance, and physical quality curves
8. Return Consistency VerifierReturn trigger, entity consistency, environment consistency, render consistency
9. Offscreen Evolution VerifierReappearance, expected evolution, relation continuity, freeze/reset diagnostics
10. ...

Skill Selection. HarnessEval-W operates hierarchically to generate the reasoning trace. At the first level, HarnessEval-W interprets the context of the evaluation case, and then selects the skills that can legitimately evaluate the case. This step ensures we are asking the right high-level questions for each unique world. For example, in a robotics scenario, we might care mostly about physical realism instead of visual realism.

Skill Decomposition. Once a high-level skill is assigned, we further decompose each evaluation question into smaller measurable subquestions, and delegate each subquestion to a suitable subagent (or tool). For instance, when evaluating a collision, the agent decomposes the problem into tracking bounding boxes, verifying temporal intersection, and calculating velocity. The evidence from all the subagents is then gathered and validated by the parent agent to obtain the final result. Such an agentic pipeline can run recursively and spawn sub-agents to generate reasoning traces.

Intentional Change Verifier

Target Visible

Transition Visible

Intended Change

Target Specificity

Final State

Anchor Preservation

No Extra Event

Judgeable

Constructing an Evidence Tree. The ultimate goal of HarnessEval-W is not just to provide a scalar score indicating model performance, but a transparent evidence tree that meticulously records what was tested, which specific tool supplied the visual grounding, and the complete logical chain that justifies the final score. This interpretable paradigm further delivers actionable insights, helping researchers pinpoint exact failure modes and guiding future model development.

Evaluation Case Construction

The benchmark must reflect the complexity and diversity of real-world environments to probe the limits of world models; this demands a wide spectrum of evaluation cases to cover possible world scenarios. We construct our evaluation cases via an agentic pipeline for scalability. Specifically, we first sample the initial world setup from our predefined scene taxonomy, generate the initial image, and plan the action to perform in the world. We then validate the scene with an LLM reasoning agent to filter out unreasonable test cases. Specifically, we first start from a scene taxonomy that defines world characteristics, covering diverse environments, entities, spatial structures, visual styles, and viewpoints. Next, we sample specific evaluation axes for the world, targeting the transition and persistence metrics defined earlier. Based on these results, an image-generation agent synthesizes the initial frame. An image-grounded planning agent then analyzes this frame to generate the target action (such as a text prompt or camera sequence) tailored to the evaluation axis. To ensure rigorous quality control, a validation agent strictly evaluates the paired action and image to verify that the environment, foreground, midground, and proposed action are contextually coherent. Any candidates that fail this validation are automatically routed back to the sampler, ensuring a high-quality generation process for our evaluation cases.

We provide the evaluation results of our benchmark on xx world models here.

What's Next: Evolving Agentic Evaluation

Evolving world, evolving evaluation.

Evaluation is never a static rubric. It evolves in our evolving world. Our HarnessEval-W is only the start of the agentic benchmark paradigm, and we identify three future directions below:

1. Test-Time Scaling for Agentic Benchmarks. As LLMs generate increasingly complex reasoning traces and leverage external tools (e.g., MCPs) to solve long-term tasks, the evaluation process must scale accordingly. By dedicating more compute at test time, the agentic evaluators can execute deeper search, multi-step verification, and robust tool use to accurately evaluate frontier models. This paradigm shift ensures the evaluator is always as capable as the model being tested.
2. Scaling Skill Libraries. Future world models will scale up and serve a vastly broader range of scenarios, generating highly complex environments with fine-grained physical fidelity. To rigorously evaluate these advancements, the agentic evaluators must scale up accordingly with a rich set of capabilities. We propose continuously scaling a comprehensive "Skill Library." As models simulate more intricate physics and diverse scenes, the evaluator can dynamically retrieve and compose specialized skills from this ever-growing library to accurately assess physical understanding across generated worlds.
3. Recursive Self-Improved Agentic Benchmark. When the system encounters out-of-distribution (OOD) scenarios while evaluating frontier models, it will actively evaluate the evaluator to identify internal skill gaps. Through either external skill expansion or self-driven exploration, the evaluation agent acquires new capabilities and updates its Skill Library, creating a recursively self-improving evaluation loop that never becomes obsolete.

The future of benchmarking is no longer a black-box metric with a static rubric, but an intelligent evaluator with Harness that decomposes the problem, assembles the right tools, and reasons over every evaluation case. We open-source HarnessEval-W as a living benchmark: an executable agentic system that grows new skills and evaluation cases as world models evolve. We invite the broader community to contribute to this agentic benchmark workflow together.

Further Reading

1. 01Fei-Fei Li, A Functional Taxonomy of World Models. ↗
2. 02Hafner et al., Dream to Control: Learning Behaviors by Latent Imagination. ↗

@misc{mirros2026harnesseval,
    title  = {HarnessEval: The Era of Harness for Benchmarking},
    author = {{MirroS Team}},
    year   = {2026},
    month  = {Aug},
    url    = {https://mirros.ai/blog/harnesseval},
    note   = {Blog post}
}