Symbols or Pixels?

Abstract

We compare two ways of answering the same visual reasoning problem: a video model renders the solution, and a language model states it in text. We pair the released VBVR-Pro video models (5B and 27B total parameters) with zero-shot Qwen language models of similar size and evaluate both on 475 VBVR-Pro-Bench samples (95 tasks).

• Inference cost. With the layout prompt, a text answer takes 480× fewer forward FLOPs at 5B and 1,026× fewer at 27B than a generated video.
• Solve rate. The benchmark's v2 scorer rates each generated video from 0 to 1 against the task's reference solution. At the strict threshold (v2 ≥ 0.9, a level that the reference solution videos reach), text solve rates are 5.5 percentage points (pp) higher than video at 5B and 4.8 pp higher at 27B; neither difference is statistically significant. At the lenient threshold (v2 ≥ 0.7), which also counts near misses such as a correct answer drawn imperfectly, video solve rates are 9.7 pp higher at 5B and 13.3 pp higher at 27B; only the 27B difference is significant.
• Per-sample outcomes. At 27B and v2 ≥ 0.9, only 89 of 475 samples are solved by both methods. An oracle that selects the successful output for each sample would solve 68.2%, compared with 45.9% for text alone.
• Answer type (exploratory). At v2 ≥ 0.9, text has higher solve rates on non-pixel answers at both scales, and video on single-agent paths at 27B; on the text side these are decision-level results (see below).

The comparison is between systems as released and does not isolate the effect of representation. The video models were fine-tuned on VBVR-Pro-SFT, and the language models are zero-shot. The two sides are also graded on different things: a text answer only states the decision, while the v2 score also checks how the video renders it (in 92 of the 95 tasks, that the rest of the scene stays unchanged). A text success therefore assumes that rendering the answer would always succeed.

1 Introduction

A video generator can answer a visual reasoning problem by rendering its solution. VBVR-Pro trains video models to do this on procedural tasks and scores the generated trajectory [2, 3]. Many of these tasks also have a compact answer (a choice, a coordinate, an ordering, or a path). A language model can state such an answer in text without generating any frames, although it may locate positions less precisely than a rendered solution.

We ask two questions on the same benchmark samples:

1. How much inference compute does each method need per answer?
2. On which tasks does each method solve more samples?

We pair the released 5B and 27B VBVR-Pro video checkpoints with zero-shot Qwen models of similar size. Both receive the same first frame and task prompt. The video model generates an 81-frame solution, and the language model writes a final answer. The two methods also differ in training data and in grading (§3, §5).

Text answers take two to three orders of magnitude fewer forward FLOPs (480× at 5B and 1,026× at 27B). Their accuracy is harder to compare. A text answer can name the correct object or path without drawing it, whereas the benchmark's scorer also checks the drawing; a video, in turn, can move toward the goal state without reaching it and still receive partial credit. We therefore report results at two thresholds, together with per-sample outcomes, and treat the text results as decision-level (§5).

2 Related work

Video models as visual reasoners. Video models can solve some visual tasks by generating their solutions [1]. VBVR and VBVR-Pro build procedural training and evaluation suites for this setting [2, 3], and other benchmarks study mazes, puzzles, and games [4, 5]. Process-aware evaluation scores the whole generated trajectory instead of only the last frame [6]. A text answer has only a final state, so this scoring choice affects any comparison between the two.

Representation and task type. Visual generation has been reported to help with paths and geometry [7, 8, 9], and symbolic or program-based methods with discrete tasks [4, 10]. These studies do not measure accuracy and inference cost for paired text and video answers on the same samples, which is what we do on VBVR-Pro-Bench.

3 Experimental setup

3.1 Benchmark and grading

VBVR-Pro-Bench has 100 tasks with five samples each: 50 in-domain tasks (seen during VBVR-Pro training) and 50 out-of-domain tasks. We keep 95 tasks (475 samples; 47 in-domain and 48 out-of-domain) on which both output types can be graded. Three tasks have no consistent final answer, and two fail a scorer check on ground-truth or trivial videos.¹ For each retained task, we fixed the text answer format and the comparison rule before grading, and checked both against known answers.²

• Video grading. The v2 scorer assigns each generated video a single score in [0, 1] for the whole trajectory.³ Each task has its own rule-based scorer. It checks the answer itself and, depending on the task, whether the rest of the scene stays unchanged, how the answer is drawn (for example, the size of a circle), and the motion between frames. We use two thresholds:
  • Strict, v2 ≥ 0.9 (our main definition of a solved video). The video matches the reference solution up to small scorer deviations. The ground-truth videos of the retained tasks score between 0.92 and 1.0 (median 1.0), and a task is excluded when its ground-truth videos do not all reach 0.9 (O-33).
  • Lenient, v2 ≥ 0.7. This also counts near misses. Typical cases are a correct answer with part of the scene changed, a correct answer drawn imperfectly (a circle that is too large, or stray marks), and a partly correct answer that the scorer credits in part. For example, the stable-sort scorer (G-3) multiplies the arrangement score by 0.6 + 0.4 × (a preservation score): with a perfect arrangement, a video needs a preservation score of at least 0.75 to reach 0.9, but only 0.25 to reach 0.7.
  • Choice of thresholds. Both thresholds are our choice (VBVR-Pro reports mean v2 scores only): 0.9 is the level that reference solutions reach, and 0.7 shows how much the near misses change the comparison. Of the 475 G27 videos, 195 reach 0.9, 86 fall between 0.7 and 0.9, and 194 score below 0.7 (G5: 106, 72 and 297).
• Text grading. The final answer is graded as correct or incorrect, so text results do not depend on the video threshold. The language model is not asked to render its answer, so a text success measures the decision only (§5).
• Statistics. Solve rates are task-weighted. Differences are paired by sample and reported as text − video in percentage points (pp). 95% confidence intervals (CIs) come from a task-level bootstrap,⁴ and we call a difference significant when its CI excludes zero.

3.2 Models and inference

We evaluate the released VBVR-Pro checkpoints G5 and G27 (Wan2.2 models fine-tuned on VBVR-Pro-SFT) and pair them with zero-shot Qwen3.5-4B and Qwen3.6-27B. Qwen3.5-9B is included as an additional reference (Table 1).

• Video. Each video model generates an 81-frame solution from the first frame and the task prompt.⁵
• Text. Each language model receives the same task prompt and the first frame at 512² (the resolution used for video generation). The primary layout prompt asks for a structured description of the scene before the answer; the direct prompt asks for the answer only. Decoding is greedy with a 2,048-token limit.⁶

The video evaluation follows the published VBVR-Pro protocol (Table 2). G5 obtains a mean v2 score of 0.474 (published: 0.470), and G27 obtains 0.652 (published: 0.670). These means are over all 500 benchmark samples and only check the reproduction; the paired results use solve rates on the 95-task set.

3.3 Forward FLOPs

We measure inference cost as the forward FLOPs needed to produce one answer, estimated from the model shapes and, for text, the measured token counts.⁷

• Video. The DiT processes 5,376 latent tokens (G5) or 21,504 (G27) for 50 denoising steps, with two passes per step for classifier-free guidance. The estimate omits the video model's text encoder and VAE, so it is a lower bound.
• Text. The estimate covers the vision encoder and the language-model passes over the prompt and generated tokens.

FLOPs count arithmetic only. Text decoding is memory-bound, so wall-clock ratios on real hardware are smaller than the FLOPs ratios.

4 Results

4.1 Inference compute

With the layout prompt, a text answer takes 1.2×10¹³ forward FLOPs at 4B and 8.8×10¹³ at 27B. Generating one video takes 5.8×10¹⁵ (G5) and 9.0×10¹⁶ (G27), so text needs 480× and 1,026× fewer FLOPs (Figure 1, Table 3). The video model runs 100 DiT passes over the full latent video (50 steps, two passes each); the language model encodes one frame and decodes one answer.

Normalized by accuracy, text needs 597× (5B) and 1,147× (27B) fewer FLOPs per solved sample at v2 ≥ 0.9, and 356× and 796× fewer at v2 ≥ 0.7. With the direct prompt, the per-answer ratios are 482× and 1,293×.

4.2 Aggregate solve rates

At v2 ≥ 0.9 with the layout prompt, text solves 27.8% of samples at 5B (video: 22.3%) and 45.9% at 27B (video: 41.0%). The paired differences, +5.5 pp [−4.0, +14.9] and +4.8 pp [−6.5, +16.4], are not significant (Table 3). The two rates measure different things: the text grade checks only the decision, while the v2 score also depends on how the video renders it (§5).

At v2 ≥ 0.7, near-miss videos also count as solved. The differences become −9.7 pp [−20.2, +0.6] at 5B and −13.3 pp [−24.6, −1.9] at 27B, and only the 27B difference is significant. At 27B, it comes from in-domain tasks, which the video model saw during fine-tuning: −27.2 pp [−42.1, −12.8] in-domain versus +0.4 pp [−17.1, +17.5] out-of-domain. The lenient threshold credits trajectories that approach the goal state without reaching it; a text answer has no partial credit.

The layout prompt is also not the strongest text condition. With the direct prompt, text solves 33.1% (4B) and 50.3% (27B) at v2 ≥ 0.9, compared with 27.8% and 45.9% with the layout prompt. The only significant comparison at this threshold is Qwen3.5-4B (direct) against G5, at +10.7 pp [+0.6, +20.4]. Switching from the layout to the direct prompt raises the text solve rate by 5.3 pp (4B) and 4.4 pp (27B), about as much as the text − video differences at v2 ≥ 0.9 (+5.5 and +4.8 pp), so the strict-threshold results do not favor either representation.

4.3 Breakdown by task group

Figures 2–5Pair5B: G5 vs Qwen3.5‑4B27B: G27 vs Qwen3.6‑27BPromptLayoutDirectVideo thresholdv2 ≥ 0.9v2 ≥ 0.7

The aggregate rates combine answer types on which the two methods differ (Figure 2). With the layout prompt:

• Non-pixel answers (labels, counts, orderings, and grid cells): at v2 ≥ 0.9, text − video is +20.8 pp [+7.6, +33.2] at 5B and +18.8 pp [+3.6, +33.6] at 27B.
• Single-agent paths: at 27B and v2 ≥ 0.9, the difference is −29.1 pp [−49.1, −10.9].
• Pixel-position answers: at 27B, the difference is −10.7 pp [−27.1, +5.8] at v2 ≥ 0.9 and −26.7 pp [−42.2, −9.8] at v2 ≥ 0.7; only the second is significant.

These splits are exploratory (§5), and three confounds remain:

• Nine of the 11 single-agent path tasks are in-domain for the video model.
• The language models see the first frame at 512² to match video generation. In earlier 97-task runs, Qwen3.6-27B solved 56.0% of pixel-answer tasks with 1024² input, compared with 38.9% at 512² (pooled over prompts).
• The text rates are decision-level. In the 18 non-pixel selection tasks, the video must also draw its mark, and the v2 scorer checks that the rest of the scene is unchanged (many also check the size of the mark and penalize stray marks); a text answer faces none of these checks.

Text better (CI above 0)Video better (CI below 0)Not significant (CI includes 0)

Loading interactive figure…

4.4 Sample-level overlap

At 27B and v2 ≥ 0.9, 89 of the 475 samples are solved by both methods, 129 by text only, 106 by video only, and 151 by neither (Figure 3). An oracle that selects the successful output for each sample would solve 68.2% (324 of 475), compared with 45.9% for text, the better single method. At 5B, the corresponding rates are 43.8% and 27.8%. These oracle rates are upper bounds; we do not evaluate a method that chooses between text and video.

Samples solved by only one method split by answer type. On non-pixel answers, 85 samples are solved by text only and 38 by video only; on pixel-position answers, the counts are 44 and 68. This pattern suggests routing by answer type (text for discrete targets, video for localization and trajectories), but we have not tested whether a router can make this choice in advance.

Both solveText onlyVideo onlyNeither

Loading interactive figure…

4.5 Error analysis

• Text (Qwen3.6-27B, layout prompt). 24.8% of answers reach the 2,048-token limit before giving a final answer, and 29.3% are parsed but wrong (Figure 4). The direct prompt lowers truncation to 19.6%.
• Video (G27). 86 of 475 samples (18.1%) score between 0.7 and 0.9.

Text grades do not depend on the threshold, so the change in the 27B difference from +4.8 pp (v2 ≥ 0.9) to −13.3 pp (v2 ≥ 0.7) consists entirely of these 86 near-miss videos. The v2 scorer gives partial credit along the trajectory, whereas the text grader sees only the final answer.

CorrectIncorrectUnparsableTruncated at 2,048 tokensVideo v2 scores

Loading interactive figure…

4.6 Qualitative examples

Figure 5 shows five paired examples, each with the generated videos next to the language model's layout, reasoning, and final answer for the same sample. It can be used to check whether a reported success reflects the intended solution, a localization error, or a video score close to the threshold.

video bettertietext better

Loading interactive figure…

ExamplePreviousNext

Input

Task prompt (shared by both models)

Reference answer (512-pixel frame coordinates)

Ground-truth solution video

Video reasoning

Text reasoning

Qwen3.6-27BQwen3.5-4B

Parsed answer

Model output

Prompt sent to the language model

5 Limitations

• Text is graded on the decision only. The v2 scorer also grades the rendered solution. In the scorer code (commit 95265d0), 92 of the 95 tasks check that the rest of the scene is unchanged, and in 94 tasks a video with the correct decision that fails the other checks cannot reach v2 ≥ 0.9 (in at least 86, not even v2 ≥ 0.7). For example, the stable-sort task (G-3) credits the order only if every original object is still found unchanged in the final frame. A text success therefore assumes that rendering the answer would always succeed. The cost gap is less sensitive: 2,000 tokens of rendering code per answer would lower the FLOPs ratio to about 200× (5B) and 460× (27B). The per-task reading of the scorers is in the code release (results/scorer_audit.csv).
• Training and partial credit differ. The video models were fine-tuned on VBVR-Pro-SFT, and the language models are zero-shot. The v2 score gives partial credit along the trajectory, while text receives a binary grade. Reporting two thresholds shows the size of this effect but does not remove it.
• In-domain samples overlap with the training data. Of 235 in-domain samples checked, 168 have an identical counterpart in VBVR-Pro-SFT. G5's mean v2 score on these samples is no higher than on other samples of the same tasks (difference −0.007); this check does not rule out other effects of the overlap.
• The G27 reproduction is slightly low. G27 scores 0.652 overall against 0.670 on its model card, and 0.773 against 0.808 in-domain (Table 2). This could shift the in-domain comparison by a few points.
• Subgroup analyses are exploratory. The answer-kind split was defined after the zero-shot runs, the single-agent path group has only 11 tasks, and subgroup CIs are not adjusted for multiple comparisons.
• Input resolution and decoding affect text. Both methods use a 512² first frame. In earlier 97-task text runs, Qwen3.6-27B solved 56.0% of pixel-answer tasks at 1024² versus 38.9% at 512² (pooled over prompts). The results come from a single greedy run with a 2,048-token limit, and unfinished answers count as failures (24.8% of Qwen3.6-27B layout answers).
• Cost is forward FLOPs only. Video FLOPs follow from the model shapes and text FLOPs from the measured token counts; the estimate omits the video text encoder and VAE. Text decoding is memory-bound, so wall-clock ratios are smaller than the FLOPs ratios. Pretraining compute is not matched, and it is not disclosed precisely enough to compare.

6 Conclusion

On 475 paired VBVR-Pro-Bench samples, text answers take 480× (5B) and 1,026× (27B) fewer forward FLOPs than generated videos, and at v2 ≥ 0.9 their solve rates do not differ significantly. The text side is graded on the decision only, while the video must also render it, so this comparison favors text by an amount we have not measured. At v2 ≥ 0.7, video solves more samples, and the 27B difference (−13.3 pp) is significant; it comes from tasks seen during video fine-tuning. By answer type, text solves more non-pixel answers at v2 ≥ 0.9, and video solves more single-agent paths (27B, v2 ≥ 0.9) and pixel-position answers (27B, v2 ≥ 0.7).

Three experiments would test these conclusions more directly:

• Text that renders its answer. The language model writes code that turns the first frame into the solution video, which is scored with the same v2 scorer and thresholds. Both sides are then graded on task completion, and keeping the ANSWER line separates decision errors from rendering errors.
• Matched training. Fine-tune the language model on VBVR-Pro-SFT.
• A combined solver. Use text for discrete targets and video for localization and paths. At 27B, the oracle rate (68.2%, compared with 45.9% for text alone, with text counted at the decision level) indicates the possible gain.

Citations

@misc{li2026symbols,
  author       = {Li, Bo},
  title        = {Symbols or Pixels? Zero-Shot Language Reasoning versus Video Generation on Visual Reasoning Tasks},
  year         = {2026},
  howpublished = {LMMS Lab Notes},
  url          = {https://www.lmms-lab.com/notes/symbols-or-pixels/}
}

References

1. Thaddäus Wiedemer, Yuxuan Li, Paul Vicol, et al. Video models are zero-shot learners and reasoners. arXiv:2509.20328, 2025. Paper
2. Maijunxian Wang, Ruisi Wang, Juyi Lin, et al. A Very Big Video Reasoning Suite. arXiv:2602.20159, 2026. Paper
3. Junxiang Xu, Ruisi Wang, Fanyi Pu, et al. VBVR-Pro: A Scalable and Verifiable Suite for Native Visual Reasoning. arXiv:2608.26105, 2026. Paper
4. Cheng Yang, Haiyuan Wan, Yiran Peng, et al. Reasoning via Video: The First Evaluation of Video Models' Reasoning Abilities through Maze-Solving Tasks. arXiv:2511.15065, 2025. Paper
5. Zefan Cai, Haoyi Qiu, Tianyi Ma, et al. MMGR: Multi-Modal Generative Reasoning Benchmark and Evaluation. arXiv:2512.14691, 2025. Paper
6. Yifan Li, Yukai Gu, Yingqian Min, et al. Beyond the Last Frame: Process-aware Evaluation for Generative Video Reasoning. arXiv:2512.24952, 2025. Paper
7. Yi Xu, Chengzu Li, Han Zhou, et al. Visual Planning: Let's Think Only with Images. arXiv:2505.11409, 2025. Paper
8. Chengzu Li, Zanyi Wang, Jiaang Li, et al. Thinking in Frames: How Visual Context and Test-Time Scaling Empower Video Reasoning. arXiv:2601.21037, 2026. Paper
9. Jialong Wu, Xiaoying Zhang, Hongyi Yuan, et al. Visual Generation Unlocks Human-Like Reasoning through Multimodal World Models. arXiv:2601.19834, 2026. Paper
10. Wen-Ding Li, Keya Hu, Carter Larsen, et al. Combining Induction and Transduction for Abstract Reasoning. arXiv:2411.02272, 2024. Paper