PhyJudge-9B: Physical Plausibility Judge for Generated Video

Project page · Paper · Evaluation code · PhyGround dataset

PhyJudge-9B is a PEFT/LoRA adapter for Qwen3.5-9B, trained to evaluate generated videos on three general dimensions and 13 observable physical laws. It returns one criterion-specific score per call on a 1–5 scale, where a higher score indicates a more plausible or better-aligned video under that criterion.

PhyJudge is an evaluation model. It does not generate video and it is not a generic text-only language-model adapter.

Evaluation dimensions

General dimensions

• SA: semantic alignment with the prompt;
• PTV: physical temporal validity;
• persistence: stability of object existence, identity, and attributes.

Physical-law dimensions

Domain	Laws
Solid-body mechanics	gravity, inertia, momentum, impenetrability, collision, material
Fluid dynamics	buoyancy, displacement, flow_dynamics, boundary_interaction, fluid_continuity
Optics	reflection, shadow

Each call evaluates either one general dimension or one applicable law. A full PhyGround evaluation normally makes 5–7 calls per video: three general dimensions plus 2–4 laws associated with the prompt.

Files

File	Purpose
adapter_config.json	PEFT/LoRA configuration and base-model identifier
adapter_model.safetensors	LoRA weights, approximately 174 MB
additional_config.json	Additional ms-swift training settings
training_args.json	Training hyperparameters
subq+human.yaml	Prompt and 1–5 scoring templates
infer.py	Standalone video-scoring script

Setup

pip install -U \
  "transformers>=5.2.0" \
  "peft>=0.19.1" accelerate pyyaml \
  "qwen-vl-utils[decord]" \
  huggingface_hub

hf download NU-World-Model-Embodied-AI/phyjudge-9B \
  --local-dir ./phyjudge-9B

The card's prompt/parser path was checked with Transformers 5.2.0 and PEFT 0.19.1. Loading the 9B base model in bf16 needs roughly 24 GB of GPU memory in the authors' setup. Actual memory use depends on software versions, video length, frame sampling, and resolution. Reduce --fps or --max-pixels if needed; doing so can also change the score.

Quickstart

Score semantic alignment:

python ./phyjudge-9B/infer.py \
  --adapter-dir ./phyjudge-9B \
  --video /path/to/video.mp4 \
  --caption "A ball rolls down a ramp and knocks over a block." \
  --metric SA

Score one physical law:

python ./phyjudge-9B/infer.py \
  --adapter-dir ./phyjudge-9B \
  --video /path/to/video.mp4 \
  --caption "A ball rolls down a ramp and knocks over a block." \
  --law gravity

The script loads the base model recorded in adapter_config.json, attaches the LoRA adapter, samples the video at 2 FPS by default, and performs deterministic decoding. Output is a JSON object:

{
  "key": "gravity",
  "score": 4,
  "raw": "{\"gravity\": 4}"
}

Use --print-prompt to inspect the rendered prompt. Use --criteria only when you intentionally want to override the released per-law criterion; modified criteria are no longer directly comparable with the published protocol.

Programmatic use

from pathlib import Path
import torch

from infer import (
    build_messages,
    build_prompt,
    decode_generated,
    load_model,
    load_yaml,
    parse_score,
    prepare_inputs,
)

processor, model, adapter_dir = load_model(
    "NU-World-Model-Embodied-AI/phyjudge-9B",
    dtype=torch.bfloat16,
    device_map="auto",
)
cfg = load_yaml(adapter_dir / "subq+human.yaml")

system, user, key = build_prompt(
    cfg,
    caption="A ball rolls down a ramp and knocks over a block.",
    law="gravity",
)
messages = build_messages(system, user, Path("video.mp4"))
inputs = prepare_inputs(
    processor,
    messages,
    next(model.parameters()).device,
    fps=2.0,
    max_pixels=360 * 640,
)

with torch.inference_mode():
    generated = model.generate(**inputs, max_new_tokens=64, do_sample=False)

raw = decode_generated(processor, inputs, generated)
print({"key": key, "score": parse_score(raw, key), "raw": raw})

Training details

• Base model: Qwen/Qwen3.5-9B.
• Method: LoRA via PEFT, rank 32, alpha 64, dropout 0.05.
• Vision encoder: frozen.
• Precision: bf16 with gradient checkpointing.
• Optimizer: AdamW, learning rate 1e-4, cosine schedule.
• Duration: 1 epoch / 294 steps on the subq+human split.
• Framework versions recorded by the release: ms-swift 4.1.2, PEFT 0.19.1, DeepSpeed ZeRO-2.

See training_args.json, additional_config.json, and adapter_config.json for the released configuration.

Reported agreement with human rankings

Under the paper's model-ranking protocol, PhyJudge-9B has 3.3% aggregate relative bias against human rankings; Gemini 3.1 Pro has 16.6% under the same protocol. This is a task-specific benchmark result, not a claim that PhyJudge-9B is generally more capable than a frontier multimodal model.

Intended use

PhyJudge-9B is intended for:

• research evaluation of generated videos under the released PhyGround rubric;
• per-law failure diagnosis and model comparison;
• reproducing or extending the PhyGround benchmark;
• screening results before targeted human review.

It should not be used as the sole authority for safety-critical, scientific, legal, or real-world physical judgments.

Limitations

• Scores inherit coverage limits from PhyGround's 13-law taxonomy and prompt distribution.
• The judge may miss brief events because it samples frames rather than continuously perceiving every instant.
• Scores can change with video decoding, sampling FPS, resolution, prompt wording, or criterion wording.
• The model can return malformed JSON or an incorrect score; infer.py reports null when parsing fails.
• A trained evaluator can reproduce annotation preferences and biases. Human audits remain appropriate for new models and out-of-distribution content.

License

Adapter-specific reuse terms have not yet been separately specified in this repository. The metadata therefore uses license: other rather than implying a license grant that is not present. The Qwen3.5-9B base model is released under Apache-2.0, but its license does not by itself determine the license of this adapter. Maintainers should add a dedicated LICENSE file and replace the metadata above once the intended adapter license is approved.

Citation

@misc{lin2026phygroundbenchmarkingphysicalreasoning,
  title         = {PhyGround: Benchmarking Physical Reasoning in Generative World Models},
  author        = {Juyi Lin and Arash Akbari and Yumei He and Lin Zhao and Haichao Zhang and Arman Akbari and Xingchen Xu and Zoe Y. Lu and Enfu Nan and Hokin Deng and Edmund Yeh and Sarah Ostadabbas and Yun Fu and Jennifer Dy and Pu Zhao and Yanzhi Wang},
  year          = {2026},
  eprint        = {2605.10806},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2605.10806}
}