LLaVA-OneVision-1.5-RL: Unlocking Multimodal Reasoning via Lightweight Reinforcement Learning

Unlocking multimodal reasoning via lightweight reinforcement learning!

Overview

LLaVA-OneVision-1.5-RL presents an RL post-training stage utilizing 67K curated examples with discrepancy-based selection to generate explicit chain-of-thought reasoning, achieving significant performance gains on STEM, coding, and reasoning benchmarks while maintaining visual understanding capabilities.

Our contributions are threefold:

(1) Discrepancy-Driven Data Curation. We identify tasks where model performance gap exists between Pass@N and Pass@1 metrics, targeting “latent capability” rather than knowledge injection.

(2) Rule-Based Reward System. We employ domain-specific verification rules rather than learned preference models, enabling precise feedback across STEM, grounding, spatial reasoning, counting, coding, OCR, and diagram tasks.

(3) Two-Stage Curriculum Training. We design a training curriculum that first stabilizes concise task performance with answer-only RL, then unlocks deeper reasoning through chain-of-thought RL.

RL Data Strategy

Discrepancy-Driven Selection

If a model can solve a task given enough attempts (high Pass@N) but rarely gets it right on the first try (low Pass@1), it already has the latent capability — it just needs to learn to use it reliably. We select tasks with this gap for RL training, filtering out tasks that are too easy (high Pass@1, nothing to learn) or too hard (low Pass@N, beyond current capability).

Reward-Based Sampling

Multiple candidate responses are filtered by average reward scores to exclude trivial and unsolvable cases, focusing on medium-difficulty instances that provide optimal learning signals.

Reward System Architecture

Our RL setup employs a rule-based reward paradigm, where rewards are derived directly from task outcomes rather than learned preference models. Since different answer types require distinct verification strategies, we design answer-type-specific scoring rules via the reward/ module.

def math_reward_fn(completions, answer):
    model_answer = extract_answer(completion, format_strict=True)
    gold_parsed = parse(gt_answer, extraction_config=[
        StringExtractionConfig(), LatexExtractionConfig(), ExprExtractionConfig(),
    ])
    correct = verify(answer_parsed, gold_parsed)
    if not correct:
        correct = is_equal(gt_answer, model_answer)  # SymPy fallback
    return 1.0 if correct else 0.0

def bbox_reward_fn(completions, answer):
    xA = max(predicted[0], gt[0]); yA = max(predicted[1], gt[1])
    xB = min(predicted[2], gt[2]); yB = min(predicted[3], gt[3])
    inter = max(0, xB-xA) * max(0, yB-yA)
    iou = inter / float(areaA + areaB - inter)
    return iou  # ∈ [0, 1]

def multiplechoice_reward_fn(completions, answer):
    predicted = extract_boxed_content(completions)[-1]
    predicted = predicted.strip().strip('.()')
    predicted = predicted[0].upper() if predicted else ""
    return 1 if predicted == answer.upper() else 0

def number_reward_fn(completions, answer):
    answer_str = extract_boxed_content(completions)[-1]
    match = re.findall(r"([0-9\\.]+)", answer_str)
    count = match[-1] if match else ""
    return float(count.strip() == answer.strip())

def html_reward_fn(completions, answer):
    token_score = calculate_token_overlap(gen, ref)
    structure_score = calculate_tag_structure_similarity(gen, ref)
    reward = 0.6 * token_score + 0.4 * structure_score
    return max(0.0, min(1.0, reward))

def svg_reward_fn(completions, answer):
    token_score = calculate_token_overlap(gen, ref)
    structure_score = calculate_structure_similarity(gen, ref)
    image_score = calculate_image_similarity(gen_png, ref_png)  # SSIM
    reward = 0.5 * image_score + 0.25 * (token_score + structure_score)
    return reward  # ∈ [0, 1]

def ocr_reward_fn(completions, answer):
    dist = levenshtein_distance(gt, det)
    length = max(len(target), len(det))
    reward = 1 - min(values)
    return reward if reward >= 0.5 else 0  # threshold

Two-Stage Training Procedure

Interactive Training Pipeline: We utilize Group Relative Policy Optimization (GRPO) within the asynchronous AReaL framework. Click each stage to view the full hyperparameters and configuration.

Put ONLY your final answer within <answer></answer>.

Think and solve the following question step by step. Please put your thinking and analysis procedure within <think></think>. Put ONLY your final answer within <answer></answer>.

Extended Capability Analysis

Spatial & Grounding: RL “fast mode” significantly enhances fine-grained perception on SAT and Ref-L4 benchmarks.

Coding: “Thinking” mode achieves highest scores on Design2Code and UniSVG, demonstrating chain-of-thought benefits for structural code generation.

Performance Results

GRPO Algorithm & AReaL Async Framework

GRPO

GRPO eliminates the critic by sampling G = 16 completions per prompt and using group-normalized rewards as the baseline.

# utils/functional.py — ppo_actor_loss_fn
ratio = torch.exp(logprobs - proximal_logprobs)
clipped_ratio = torch.clamp(ratio, 1.0 - eps_clip, 1.0 + eps_clip_higher)
pg_loss = torch.max(-advantages * ratio, -advantages * clipped_ratio)

# Behavior importance weight (off-policy correction)
behav_kl = proximal_logprobs - old_logprobs
behav_imp_weight = torch.clamp(behav_kl.exp(), max=behav_imp_weight_cap)
pg_loss = pg_loss * behav_imp_weight

AReaL Async Training

AReaL decouples rollout from gradient computation — 2.77× throughput by eliminating GPU idle time.

# trains/grpo.py — main training loop
for global_step in range(start_step, max_steps):
    batch = rollout.prepare_batch(train_dataloader, workflow=workflow)
    batch["prox_logp"] = actor.compute_logp(batch)
    actor.compute_advantages(batch)
    actor.ppo_update(batch)

    rollout.pause()
    actor.update_weights(weight_update_meta)
    rollout.set_version(global_step + 1)
    rollout.resume()

Training Configuration

Acknowledgements

We thank the following projects and frameworks:

• AReaL: Lightning-Fast RL for LLM Reasoning and Agents
• sglang: Fast serving framework for LLMs and vision language models
• lmms-eval: Standardized evaluation framework
• LLaVA: Large Language-and-Vision Assistant
• LLaVA-NeXT: Next-generation multi-modal assistant

@inproceedings{LLaVA-OneVision-1.5,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
  booktitle={arXiv},
  year={2025}
}

@inproceedings{xie2025region,
  title={Region-based Cluster Discrimination for Visual Representation Learning},
  author={Xie, Yin and Yang, Kaicheng and An, Xiang and Wu, Kun and Zhao, Yongle and Deng, Weimo and Ran, Zimin and Wang, Yumeng and Feng, Ziyong and Miles, Roy and Elezi, Ismail and Deng, Jiankang},
  booktitle={ICCV},
  year={2025}
}

@article{lillava,
  title={LLaVA-OneVision: Easy Visual Task Transfer},
  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
  journal={Transactions on Machine Learning Research},
  year={2024}
}