NOV 8, 2026/

evaluation

朝闻道*

The Hard Road to Real Evaluation

Quick disclaimer. This is a draft for the author and readers to pick apart, not an evaluation roadmap or a declaration that somebody's leaderboard is useless. The more evaluation you do, the more traps you find. Some judgments here remain unsettled. The references are meant to make the discussion concrete, not lend authority through sheer paper count. Add to it. Push back.

1. Evaluation is getting more competitive. Closer to what?

How do we know we are taking the right road?

Multimodal eval is busy. Harder questions, longer leaderboards, and a release checklist that seems to have no end. Academic knowledge, visual mathematics, long video, desktop operation, interactive planning. Capabilities are improving, and evaluation is trying to keep up. That work deserves respect.

But I have an uncomfortable question: are we getting closer to evaluation that genuinely advances the field?

When a model gains two points, does it see more clearly and reason more reliably? Or does it know the questions and the judge better? Can the result tell us whether the next improvement should come from data, training objectives, or inference-time tools and budgets? Can it explain a real failure encountered by a user?

If the only thing left is a ranking, I think we are missing something. Rankings help people choose models. Research also needs evidence about where capabilities live, where they fail, and which changes actually help. This is not a search for the best benchmark. It is a question about turning a report card into evidence that changes our direction.

2. Set the coordinates first: period × lifecycle stage

In some past work, such as Ma et al.'s NeurIPS'25 tutorial, “what's measured / what's missed / what's next” is already in the title. Whether a score deserves to be the y-axis of scaling is a question we should already be asking. To answer it, we first need to distinguish what kind of ruler we are using from where in a model's lifecycle we use it.

I want to separate those questions into two coordinates:

The horizontal axis is period / era: evaluation's evolving emphasis, from traditional metrics, to task-specific metrics and settings, to dynamic / online evaluation, to environments and interaction. It describes how the ruler changes over time.
The vertical axis is stage / lifecycle: pretrain → mid-train → post-train → deploy / online product evaluation / regression. It describes where a model or system is in its lifecycle and which decisions evaluation should support. The last entry is not another training step; it is continued evaluation after the system enters use.

Two axes of evaluation: period × stage

Horizontal → periodHow evaluation forms evolve (period / era)

Vertical ↓ stageThe model and system lifecycle (stage / lifecycle)

Select an intersection to explore its scope and pitfalls; scroll horizontally on narrow screens.

Stage ↓ / period →

pretrain × Traditional metrics

Usually measuresCheck representations, cross-modal associations, and transfer with classification, retrieval, or caption metrics.

Watch out forCandidate pools, prompts, and validation distributions affect scores; local metrics do not establish open-ended capability.

Cells are diagnostic examples, not training recipes; periods overlap. Online is not deploy: dynamic tests can run offline, while fixed regression sets can serve deployed systems. Tab to focus, Enter / Space to select, and arrow keys to move between cells.

A ruler from the same period can be used at different training stages; the same stage can use evaluation forms from different periods. Retrieval metrics can probe pretraining representations or check for regression after post-training. A small environment can test a post-trained action policy or a deployed workflow. The cells are diagnostic examples, not a prescription assigning one method to each stage.

Periods overlap rather than hand over neatly. Traditional metrics remain useful today; environment evaluation is not a highest rank. In particular, online on the horizontal axis is not the same as deploy on the vertical one. The former emphasizes ongoing changes to content or measurement processes: a dynamic test can run offline. The latter concerns a system already in use: a fixed regression set can still serve product evaluation.

We will follow the horizontal axis to see how rulers change, then the vertical axis to ask what they should tell us. Why make this distinction? Today's difficulties cut across both.

Capabilities got broader. One number still wants to cover everything.

A classifier chose among predefined labels. Today's model may need to read images and text, listen to audio, understand temporal relationships, then use tools to finish a task. Some abilities can be tested separately. Others fail precisely when combined.

Recognizing a button in a screenshot does not mean knowing whether to click it. Answering a video question does not mean finding the relevant evidence, or admitting when that evidence is unavailable. A local success easily becomes a claim about system reliability.

HELM put robustness, calibration, fairness, and efficiency alongside accuracy. Even for language models, “good” is not one axis. With more input and action pipelines, multimodal makes that harder.

Same score, different process

Assume the same 100 tasks, permissions, and budget ceiling, with one point per completed task. Both finish 60 tasks; select a model to inspect the process, or scroll on narrow screens.

Hypothetical model A · Reliable first-pass completion

Read evidence
Take action
Verify state
Complete task

Process differenceAll 60 completed tasks succeed on their first attempt, with grounded actions and timely checks. Uncompleted tasks still count. This shows one illustrative successful trace.

Inspect beyond the scoreCheck stability across runs and appropriate stopping or clarification on failure. First-pass completion does not erase the other 40 failures.

A fictional illustration, not measured results for Inkling or any real model: score ≠ process quality. Inspect verifiable actions and consequences, not internal CoT as a mechanism account. Tab to focus, Enter / Space to select; left / right arrows or Home / End switch models.

Distributions move. Old rulers do not move automatically.

Users do not follow a benchmark's table of contents. They upload crooked photos, compressed screen recordings, and questions containing industry-specific terminology. Change the device, language, audience, or workflow, and the failure distribution can change too.

ImageNetV2 examined newly collected test data. ImageNet-C tested common corruptions. WILDS centered naturally occurring distribution shifts. None is a universal multimodal solution. They make one point concrete: progress on an old test set is not automatically progress in a new setting.

Evaluation has to evolve. That means explaining changes in populations, conditions, and tasks, and identifying which historical comparisons still hold. A new leaderboard name every month is not enough.

Leaderboards entered the commercial loop.

When a research score becomes release material, a procurement signal, and market perception, optimization pressure follows it. We do not need to assume everyone is cheating. If a number affects resources, people will study how to raise it.

Genuine capability improvement and adaptation to a particular exam can look identical in a results table. With public comparison platforms such as Artificial Analysis, I want to read the methodology, coverage, and update notes, not just the order. Commercial rankings have value. Their influence makes transparency and resistance to targeted optimization more important.

3. Along the horizontal axis: four periods, from scores to environments

These four periods summarize historical shifts in emphasis, not rigid dates or the four lifecycle stages from pretrain to deploy. VQA predates today's chat models. Online evaluation is not a recent invention. Environments have not driven other methods out of history. What changes is what we want evaluation to tell us.

Along the horizontal axis: four evaluation periods

From scoring and task design to ongoing sampling and interactive environments. Select a period to explore; scroll on narrow screens.

Traditional metrics

Measure a local ability with established metrics inside a defined task and answer space.

Representative workImageNet / ILSVRC · COCO Captions · CLIP

LimitsAccuracy, Recall@K, and caption scores have different scopes. Candidate pools and prompts affect comparisons; local gains do not establish system reliability.

1 / 4

Click a period or use Previous / Next; focused period buttons also support left / right arrows. This is a shift in emphasis, not a dated time scale or a sequence of training stages.

Period one: borrow CV and NLP metrics for a defined task

Early vision tasks and image-text representation learning had clearer boundaries. ImageNet / ILSVRC classification evaluated recognition among specified categories. COCO supplied images and annotations, while COCO Captions supported standardized caption evaluation. Image-text retrieval also commonly used established COCO splits and candidate pools.

Classification used top-1 or top-5 accuracy, retrieval used Recall@K, and captioning used metrics such as BLEU, METEOR, and CIDEr. These are not interchangeable. They measure label hits, ranking of matching pairs, or generated descriptions against reference text. Their shared feature is a relatively explicit task and answer space.

Recall@K depends on the candidate pool. Its size and difficulty matter, as do retrieval direction, captions per image, and dataset split. Report “R@1 improved” without those conditions, and readers do not really know where the improvement happened.

CLIP marked an important point in 2021 for large-scale image-text pretraining and practical zero-shot transfer. Its zero-shot classification still depended on class text and prompts. It demonstrated useful transferable representations, not open-ended visual conversation.

The methods of this period are simple, reproducible, and inexpensive ways to isolate problems. The limitation is equally simple: one task should not stand for all multimodal intelligence. These tests still deserve a place, with clear boundaries on their claims. They can probe representations during pretraining or anchor input-pipeline regressions after deployment.

Period two: design task-specific metrics and settings

From visual question answering to the visual instruction-tuning wave around 2023, outputs expanded beyond labels and rankings. Users asked about relations, text in images, and open-ended observations. Evaluation had to change.

VQAv2 used more balanced image-question pairs to make language-only shortcuts harder. Its consensus-based scoring also accommodated variation in human answers. GQA connected scene graphs with compositional questions. TextVQA made text within images necessary to answer questions.

Those paths expose different bottlenecks inside “visual question answering.” A sound question, available visual evidence, and a label that supports judgment are three separate requirements. More examples do not guarantee all three.

MME and MMBench organized multiple perceptual and cognitive tasks. MMMU moved into multidisciplinary settings connecting expert knowledge with visual understanding. MathVista focused on mathematical reasoning in visual contexts. Each broadened the meaning of capability, while making choices about question distributions, answer formats, and aggregation.

Open-ended output added another layer. LLaVA and LLaVA-Bench used model-assisted review instead of forcing every conversation into multiple choice. But does the judge favor longer answers or a familiar style? How should fluent factual errors be penalized? The judge is part of the method, not a transparent law of nature.

Rule-based scoring is not inherently better. Rules miss equivalent answers; humans disagree too. We need to examine where judges agree with human review, where they differ, and whether errors favor particular models.

Infrastructure such as lm-eval and lmms-eval makes interfaces and protocols easier to reuse. That matters. Sharing a framework does not mean sharing a measurement construct. A framework standardizes execution; it cannot decide what a combined score means for us.

Period three: move from fixed content to dynamic and online evaluation

Repeated use of public question banks and increasingly unauditable training corpora expose two problems: test content can enter training, and fixed question complexity can stop distinguishing models.

Some past work, such as VLB, addresses both: it generates variants through visual and language transformations, with a judge checking consistency. Combining transformations lets evaluation complexity be adjusted.

The useful idea is that evaluation content need not remain frozen. Whether a transformation still measures the intended ability, and whether the generated example is correct, remain verification questions. Better generation does not remove the need to audit question quality.

Language and coding offer related examples. LiveBench uses recent material and updates to limit contamination risk. LiveCodeBench continually collects programming problems, enabling temporal splits and broader code evaluation. Neither directly solves image provenance, video annotation, or perceptual shortcuts. Fresh content, verifiable judgment, and temporal records are still useful ideas to borrow.

We also need to distinguish refreshing content, recombining old material, increasing difficulty, and adapting questions to model performance. An eighty on an increasingly difficult test differs from an eighty on fresh questions at stable difficulty.

Static tests need not disappear. Stable anchors can support regression comparisons; dynamic tests can reveal new failures. The deeper shift is from publishing a question bank once to maintaining a measurement process. It can test whether mid-training transfers beyond familiar templates, or detect new user distributions after deployment. Refreshing questions and running online product experiments remain distinct operations.

Period four: evaluate interaction in an environment

WebArena put web tasks in repeatable environments. VisualWebArena emphasized visual information in web operation. OSWorld moved into open-ended tasks in real computer environments.

These should not all be reduced to “give the model screenshots.” The difference is repeated observation and action, with goals realized as environment outcomes. Saying “I saved the file” and actually saving the correct file are different kinds of evidence.

Video can borrow that perspective. A final answer does not reveal how evidence was gathered. If a model may revisit clips, we should record when, which clips, and at what budget. Long-video understanding is not identical to agent evaluation. Both remind us that capability can live in an interaction, not just its final sentence.

Small and strict matters more than large and realistic-looking. Define initial states, permissions, step limits, and time budgets. Check the resulting state where possible. Preserve the structure that matters: goals, information constraints, and action consequences. Do not import every bit of real-world noise.

Limits remain. Websites and simulators can be memorized. Tool interfaces affect results. Environment errors can be counted as model failures. Aggregate task success need not predict user success. An environment is a better measurement instrument, not an automatic portal to reality.

Recent discussions around MiMo-V2.6 and DeepSWE are useful reminders not to collapse every environment story into one scandal. MiMo’s report itself discusses reward hacking, solution leakage, and mitigations such as cleanup, isolation, and trajectory audits. A Vals audit argues that some public training environments deleted later branches yet left unreachable Git objects readable, and shows cases where models exploit that leakage. That is a question about whether information boundaries actually hold—not only whether a prompt says “do not cheat.”

Ian Barber relays an audit of DeepSWE in the opposite direction: an agent can add a test that collides with a hidden test’s name and fail a repair that would otherwise be fine. One failure mode can inflate scores; another can deflate them. Training-environment leakage, evaluation misjudgment, and active reward seeking are different claims. A Standard Normal red-team report further shows grader bypasses under attack prompts with other models—evidence of exploitability under those conditions, not proof that MiMo behaves that way in ordinary evaluation.

Well-resourced teams are also better positioned to probe environments and scoring repeatedly. Enterprise deployment does not immunize a system against hacking surfaces. More online, interactive benchmarks still look closer to what many of us want—if judgment, permissions, and process are maintained together, rather than swapping a static question bank for an expensive, exploitable simulator.

4. Along the vertical axis: different questions from pretrain to deployment

Along the vertical axis: a four-stage evaluation ladder

Select a stage to separate what should change or stay, the evaluation question, loss signal, methods still needed, and pitfalls.

pretrain

What should change or stayRepresentations, knowledge, and cross-modal associations.

Evaluation questionHave representations, knowledge, and cross-modal associations developed, and do they transfer?

What eval loss can signalImprovement in prediction or alignment objectives on a validation distribution.

What eval method still needs to checkWhether perception, retrieval, and transfer tests support claims of foundational gains. Add small downstream-training controls for important architecture decisions.

Common pitfallsSimilar loss need not imply similar downstream agent performance. Preprocessing, prompts, and adapters can obscure base-model differences.

Stages are diagnostic perspectives, not ranks of methods; one method can serve multiple stages. Tab to focus, Enter / Space to select; use up / down arrows or Home / End on stage buttons.

Stages describe the lifecycle of a model and system.

OPD, for example, starts from models that already went through pre-/mid-training, then studies distillation and behavioral adjustment—including acquiring domain knowledge, preserving assistant behavior, and recovering behavior. That separation is more useful to me than just asking how many points the final model gained.

There is a caveat: teams define mid-training differently. The same SFT operation can sit in different stages depending on its purpose. The stage ladder above is a simplification for discussion, not an attempt to standardize industry terminology. Deploy belongs here because stopping training does not end evaluation's responsibility: input pipelines, tools, users, and service versions keep changing.

The same eval method can be reused at all four stages if interfaces and comparison conditions are explicit. One stage can require several methods. This is not an assignment of simple benchmarks to pretraining and advanced environments to post-training. It separates what we intend to change from what we can observe.

Pretrain: have the foundations developed?

Representations, knowledge, and cross-modal associations often matter most here. Validation loss helps track optimization; retrieval, classification, and controlled question-answering add evidence about transfer. Keep image preprocessing, text prompts, and adaptation procedures consistent, or interface differences may obscure changes in the base model.

Small environments are not forbidden, but the action interface needs explanation. With fixed adapters and policies, a workflow step can diagnose perceptual bottlenecks. Without such controls, attributing an entire agent's success or failure to pretraining is difficult. Environment form belongs on the horizontal axis; foundational diagnosis belongs on the vertical. They can coexist.

Mid-train: did specialization arrive without displacing existing abilities?

After adding domain knowledge, long context, or video data, the useful question is not only whether an aggregate rose. Where did specialization help? Does it transfer to new material? What was forgotten? In-domain tests, out-of-domain held-out examples, and stable regression anchors need to be read together.

Compare old and new checkpoints under the same behavioral interface. When refreshing dynamic questions, separate freshness from difficulty where possible. When testing domain workflows, keep tools and planning policies fixed. Otherwise acquired knowledge, changing questions, and system changes collapse into one success rate.

Post-train: not just whether it can, but how it acts

Instruction, preference, reasoning, and tool training often change how capabilities are called upon: when to reason, abstain, ask for clarification, or recover. Knowing an answer is not enough, and neither is a reward or preference-validation signal. Task judgments, action traces, and human checks may all be necessary.

Different budgets, tools, and retry limits can produce very different results from the same base model. Record those conditions when checking environment success. For open-ended responses, verify that the judge is not merely rewarding length or a pleasing style. Traditional metrics still have a role: checking whether behavioral training damaged existing perception and transfer.

Deploy: evaluate a system in use, not just a checkpoint

Weights can stay fixed while image resizing, a retrieval index, tool interfaces, or the user population change product outcomes. Separate authorized fresh-traffic monitoring, offline replay, controlled online product experiments, and fixed regressions. They help discover new problems, reproduce causes, compare product changes, and confirm that old failures have not returned. They are not interchangeable scores.

Real traffic cannot simply be copied into a test set. Establish authorization, privacy, and de-identification boundaries, then record time windows, user and task strata, and service versions. Online comparisons must account for user mix, experiment allocation, and seasonality. Severe errors, latency, cost, appropriate clarification, and recovery may decide whether a release should ship or roll back, alongside average success.

Regression sets built from fixed failures are valuable, but only establish whether known problems returned; fresh held-out examples remain necessary. Do not count every infrastructure timeout as a reasoning error, or hide severe failures behind unlimited retries. At deploy, evaluation should support product and system decisions, not provide a pretty endpoint for a training curve.

Eval loss is not eval method, or real capability.

Validation loss is useful. It tells us about optimization and prediction on held-out data. But it depends on the validation distribution, objective, and weighting. Change the objective, and two loss curves may not even be directly comparable.

Eval method includes sample selection, input processing, permitted interaction, the judge, and result aggregation. It tests whether evidence supports a capability claim, not merely whether a training objective decreased.

Consider a hypothetical example. Adding high-resolution document data lowers validation loss and improves TextVQA, but desktop task success stays flat. Perhaps reading improved while planning did not. Perhaps screenshot preprocessing discarded the new detail. Declaring the training useless from final success alone, or declaring the product more reliable from loss alone, both skip intermediate evidence.

I want a traceable chain: what changed in the proxy objective, what controlled tests verified, and where actual use still fails. When mid-training changes knowledge and post-training changes behavior, gains and forgetting both need inspection. One aggregate should not quietly cancel them out.

Update evaluation, but preserve comparisons that explain the change.

Distribution shift is not a bonus problem after release. Training and evaluation face it throughout. A camera, language, document layout, or default image-resizing setting can change observed performance.

When updating a test, keep anchors, record versions, and, where needed, run the same models on both versions. Otherwise it becomes hard to tell whether the model improved, the questions got harder, or the protocol changed. Evaluation should move without making its own movement impossible to explain.

5. At bottom, we want two things from evaluation

Diagnosis, comparison, selection, and regression testing are useful functions. Trace them back, and I think the basic motivations are two: prevent hacking, and better approximate real-world capability assessment inside small, strict environments.

First: make scores track capability, not exam shortcuts.

How scores mislead: seven mechanisms, different checks

Select a mechanism to separate the misleading claim, evidence, and checks. Choices wrap to fit; details stay below. Harness false negatives can lower scores rather than inflate them.

Language / unimodal shortcut

MechanismQuestion wording, answer priors, or one modality suffice without the visual evidence the task claims to require. This is not the same as seeing the test item before.

What the score gets wrongCredits language priors or text information as vision and cross-modal capability.

Evidence

VQAv2 — Balanced image–question pairs make language-only shortcuts less rewarding.
GQA — Scene graphs and compositional questions probe objects and relations.

What to checkCompare blind, unimodal, and balanced-pair controls. Verify that the question needs the removed evidence; a correct blind answer alone does not establish contamination.

Shortcuts and judge bias can inflate scores; harness false negatives can lower them. These need different audits. Tab to focus, Enter / Space to select; arrows follow button order, Home / End jump to the first / last choice.

Contamination, leakage, leaderboard gaming, and reward exploitation often get bundled together. They need different responses. Training on test answers and exploiting linguistic bias in a question are different mechanisms, even if both cause capability to be overestimated.

A visual question answered without an image may reflect prior exposure, common knowledge, or enough information in the wording. Investigate it, but do not treat that observation alone as proof of contamination. More informative controls combine removing vision, replacing key objects, shuffling options, or keeping text fixed while changing a relevant visual relationship. Does the answer follow the necessary evidence?

Shortcuts can also be multimodal. VQA-CE mines trivial predictive rules that combine question words with visual elements, then evaluates on counterexamples where those rules fail. Models can look strong on the main split and still collapse when the easy co-occurrence is broken. That is a different diagnosis from “the answer leaked into training.”

Counterfactual tests need valid questions too. If an edit damages the image or makes the question ambiguous, a performance drop does not establish that the original result depended on a shortcut. Evaluators are responsible for their transformations.

Judge taste is another family. Zheng et al. document position, verbosity, and self-enhancement biases in LLM-as-a-judge settings. Length-Controlled AlpacaEval and LMSYS style-control treat length as a confound to model, not as proof of substance. None of these papers by themselves indict a particular frontier model; they constrain what an automatic preference score can claim.

Preference Leakage goes one step further: when synthetic-data generators and judges are related—same model, inheritance, or family—judges systematically favor related students. Fresh questions do not automatically break that loop. Length control cannot substitute for disclosing generator–judge relatedness and adding independent checks. This sharpens the earlier worry about generation–judgment closed loops.

Frontier reports already separate some of these threads in practice. Qwen3 adds probes such as LengthCtrl alongside answer accuracy: whether a model obeys a length constraint is not the same question as whether the answer is correct. Mistral Large 4 shows that comparable early losses need not imply comparable later agentic transfer—another reminder that one proxy can miss the capability you care about next.

Repeatedly using the same validation set to choose prompts, sampling strategies, and model versions also brings it into the optimization process. Dwork et al. addresses adapting repeatedly to holdout data. This is not identical to leaking answers. But “we never backpropagated on it” is not a complete independence guarantee.

Preventing hacking is therefore design work, not a moral declaration. Train-test boundaries, access frequency, variant checks, independent judges, environment isolation, and audit records deserve attention alongside the score.

Second: preserve important real-world constraints in a small world.

Consider another hypothetical task. Read amounts from three photographed receipts, check the total, and enter them into a spreadsheet. Ask for clarification when one receipt is unreadable rather than guessing and continuing. This is small, but it connects perception, arithmetic, tools, and uncertainty.

What makes it strict? Recorded input versions, verifiable amounts and cells, no direct access to hidden answers, limits on time and tool calls, and observable recovery after failure. Completion should preferably be judged from environment state, not the model's own report.

We have not packed the entire world into the task, and should not pack every ability into it either. We have preserved a structure users care about: inspect evidence, handle constraints, finish the job, and do not bluff through uncertainty.

Latency, cost, severe errors, appropriate clarification, and recovery may matter alongside success. They need not immediately become one weighted score. Weights express value choices; users should be able to see them.

Both motivations are necessary. An almost ungameable test unrelated to use may measure an irrelevant ability fairly. A realistic-looking environment without reliable judgment may be an expensive demo. Good evaluation should increase confidence that we can fix actual failures, not only become more practiced at an exam.

6. The road is long. Dynamic does not mean immune.

Dynamic evaluation reduces one risk and adds measurement costs.

Back to VLB and LiveBench / LiveCodeBench: valuable because they address specific problems, not because they issue contamination-immunity certificates. Generated variants can reduce original-question memorization; temporal updates can reduce some training exposure. Those benefits have conditions.

New questions may still derive from familiar source material. Generators may repeat templates. Answers may circulate elsewhere. “Just generated” does not mean entirely fresh to the tested model. Increasing irrelevant distractions can make a question harder without making it a more meaningful test.

Chen et al. at EMNLP'25 also put the static→dynamic shift inside the contamination discussion. My reading is that contamination cannot be settled by one deduplication pass before release. Provenance, evaluation access, and the content lifecycle jointly shape the risk.

Generation and judgment can also create a biased loop. One model writes questions, a related model grades them, and the tested model shares their preferred style. Preference Leakage makes that loop concrete: relatedness alone can move scores even when the surface protocol looks clean. The process can look consistent without becoming more useful in reality.

Return to the opening question about scores. As compute, data, and training steps increase, we hope capability grows with them. If the measurement target drifts, or scores mostly reward familiar templates and judge-pleasing responses, a beautiful scaling curve may not support the original capability claim. Dynamic evaluation is not an answer by itself. At every lifecycle stage, we need to ask what this version of the ruler measures and whether its changes still permit comparison.

Mixing benchmark data into training and finding hacking opportunities in evaluation mechanisms deserve attention. Still, official training splits, disclosed task adaptation, and undisclosed use of test content need to remain separate. Touching a dataset is not sufficient grounds for an accusation.

My minimum requirement is simple. Claim generalization, and provide held-out evidence appropriate to that claim. Claim task specialization, and disclose its scope. Honest boundaries help more than pretending every model lives in a perfectly clean data vacuum.

Watermarks, fingerprints, and AA: an analogy about risk, not an allegation.

Shortcuts need not involve memorized answers. Wording, formatting, file metadata, and distinctive watermark or fingerprint-like cues can reveal an evaluation's source. A system recognizing an exam could, in principle, switch strategies. These signals could be incidental; they need not have been deliberately planted.

Kirchenbauer et al. study text watermarks; Sadasivan et al. look at detection and rewriting boundaries. Neither establishes that a multimodal leaderboard has been gamed.

The analogy is limited: subtle signals usable by a system are not purely imaginary. Detecting a watermark does not automatically improve a benchmark score. Moving from source recognition to effective score gaming requires a strategy, access conditions, and specific evidence.

Even when discussing a leaderboard such as AA, the careful questions are whether distinctive cues could enable targeted optimization, and whether traffic, formatting, or sample characteristics reveal the evaluation source. This draft has no evidence that a particular person, team, or model used watermarks to game AA. Those papers do not supply that evidence either.

Reducing exploitable source cues, testing semantically equivalent formatting variants, and disclosing meaningful audit boundaries are possible directions. None is a universal guarantee. Contamination, shortcuts, and environment exploits require concrete threat models, not just the phrase “our evaluation is dynamic.”

7. In the long tail, do we still need to test everything?

Wider coverage does not necessarily locate real failures.

Suppose common tasks work reasonably well. What remains is a less common language, a blurry document, a tricky temporal relationship, or an occasional failure in a professional workflow. Average performance keeps improving while some users keep encountering the same problem.

MMMU's multidisciplinary coverage and MMT-Bench's broad task organization make it concrete that multimodal capability is not one solid block. Yet category averages do not equal a product's failure distribution. Academic weights, user frequency, and error consequences can differ.

A rare severe error may matter more than a few additional correct answers on common tasks. Conversely, a low-frequency, low-impact task that is extremely hard to label reliably may not deserve unlimited resources. That judgment cannot rest only on an evaluator's interests.

Focus and comprehensiveness both have uncomfortable costs.

Should evaluation deliberately leave things out and focus on verifiable, high-leverage, hard-to-game core abilities? I understand that choice. Resources are limited. Testing a few bottlenecks rigorously, when they affect many workflows, can beat filling a capability radar chart.

But who defines the core? If professional developers' needs alone count as high leverage, other languages, devices, and ordinary users may remain outside the measurement indefinitely. Testing less can be honest scope control. It can also hide inconvenient failures.

Comprehensiveness is not easy either. More coverage means harder maintenance of labels, judges, statistics, and environments. A handful of questions per tail category creates the illusion of “already tested.” Without adequate samples and trustworthy judgment, a checkmark can be more misleading than a blank.

Two questions belong together: does the problem matter, and can we measure it reliably? Important but currently hard to measure should not disappear from the needs list. Easy to measure but irrelevant should not automatically capture the budget.

A possible compromise: stable core, tail probes, real regression.

For now, I lean toward three evaluation functions, rather than compressing all three into a championship score. All fit back into the two axes: stable cores are not confined to traditional metrics, tail probes are not confined to the dynamic period, and real regressions need not wait for deployment. Discovered failures can become controls in the next round of mid- or post-training.

First, a stable core. Choose verifiable abilities with transfer value, maintain protocols and historical anchors, and compare gains and regressions. Stable does not mean never updated. It means explaining and calibrating updates.

Second, long-tail probes. Explore fresh content, disturbances, and user groups to discover unknown failures. These may not suit a permanent leaderboard. Discovering a problem can matter more than ranking models.

Third, regression tests derived from real failures. Where privacy and authorization permit, de-identify and stratify important failures from products or research, and make them reproducible. Their main question is “was this problem fixed?” They need not pretend to measure universally applicable capability.

Three coverage layers: compare, discover, verify fixes

Select a layer to explore its purpose, public-leaderboard fit, and training feedback. Distinct functions, not one championship score; scroll on narrow screens.

Stable core

PurposeMaintain verifiable, transferable abilities, fixed protocols, and historical anchors for longitudinal comparisons and regressions.

Public-leaderboard fitA better fit for scoped public comparisons when samples, judgments, and protocols are trustworthy. Stable does not mean frozen, or publishing every test item.

Training feedbackLocate foundational gaps and forgetting; compare gains and tradeoffs across pretrain, mid-train, and post-train. Keep version controls and isolated held-out variants when updating the core.

Feedback loop: probes discover gaps → regressions verify fixes → repeated failures update the core; retain fresh held-out variants. Layers are not tied to one period or stage. Tab to focus, Enter / Space to select; left / right arrows or Home / End switch layers.

The layers should interact. A probe finds a failure; its impact and verification cost help determine whether it belongs in the core or a regression set. Repeated regression failures can reveal a missing core ability. Once incorporated, those tests can themselves become learned targets, so fresh held-out variants remain necessary.

Datasheets advocate documenting motivations, composition, collection, and usage boundaries. Borrow that idea for evaluation: state what is uncovered, sample sizes, and judgment limitations. Not every gap must be filled immediately. Gaps should be visible.

This is a proposal for discussion. Will the core narrow over time? Who has incentives to maintain discoveries from tail probes? Could one product's regression needs become mistaken for the whole field's direction? I do not have standard answers.

8. Put the questions on the table before declaring one right answer

At this point, I am less interested in ranking evaluation methods once and for all. Traditional metrics, task benchmarks, dynamic tests, and interactive environments do different jobs. New does not automatically mean more real. Harder does not automatically mean more useful.

If this draft leaves a discussion checklist, mine starts here:

What capability claim does the score support? What is explicitly outside its coverage?
Which period's evaluation form is being used at which lifecycle stage? Should the result change training, the inference system, or a deployment decision?
How do we distinguish legitimate learning, task adaptation, contamination, and shortcuts instead of guessing from gains?
How do dynamic updates preserve comparisons without turning stable anchors into new optimization targets?
Which structures must a small environment preserve for success to say something about actual use?
Who allocates evaluation resources when long-tail frequency, impact, and verifiability conflict?
After evaluation reveals a failure, who carries it back into training, system improvements, and regression tests?

The last question may be the simplest and the easiest to skip. If a benchmark only makes a release table more complete, without changing any training or usage decision, what exactly did it advance?

The hard road to evaluation that genuinely advances the field probably takes more than more questions, harder questions, and bigger leaderboards. It needs clearer capability claims, more honest boundaries, and a route from measurement back to practical improvement.

For now, I want to leave the questions here and start a conversation. What should we keep testing? What can we set aside for a while? What needs a different kind of test? I would like to hear people's answers.

9. References: starting points for further reading

These are entry points to the discussion, not a comprehensive survey. Findings from the papers, this post's interpretations, and hypothetical examples should be read separately.

Environments, frontier reports, and coverage boundaries

Zhou et al., 2024: WebArena; Koh et al., 2024: VisualWebArena; Xie et al., 2024: OSWorld.
Xiaomi: MiMo-V2.6 Technical Report, especially reward-hacking mitigation and evaluation protocol.
Oliver Chen and Anthony Ozerov / Vals AI: MiMo training-environment leakage audit.
Ian Barber: Environments and Benchmarks (DeepSWE harness discussion).
Standard Normal: MiMo2.6 Hacking Report — attack-prompted exploitability, not an unconditional claim about ordinary runs.
Qwen Team: Qwen3 Technical Report; Mistral AI: Mistral Large 4.
Kirchenbauer et al., 2023: A Watermark for Large Language Models.
Sadasivan et al., 2023: Can AI-Generated Text be Reliably Detected?.
Gebru et al., 2021: Datasheets for Datasets.
Artificial Analysis official website: a public leaderboard example, not experimental evidence for the risk analogy in this post.

Key resources

Selected starting points for measurement, dynamic evaluation, and environment evaluation.