LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training

High performance, low cost, and strong reproducibility.

LLaVA established a low-cost open-source path for connecting vision encoders with large language models, and later versions steadily expanded toward OCR, charts, documents, multi-image reasoning, and video. LLaVA-OneVision consolidates that line into a unified interface across images, charts, documents, multi-image inputs, and video.

The remaining gap in open multimodal systems is often not architecture, but recipe transparency. Strong models such as Qwen2.5-VL show excellent results, yet full data composition, cleaning, sampling, and training schedules are rarely disclosed end-to-end. Our focus is to close that reproducibility gap rather than only release weights.

We release a fully open, concept-balanced 85M pretraining set and a curated 22M instruction set on top of the LLaVA–OneVision framework. The recipe keeps a compact three-stage pipeline, combines offline data packing with Megatron-LM and a distributed optimizer, and trains an 8B model from pretraining to SFT in about ~168 wall-clock hours.

LLaVA–OneVision–1.5 extends the series with RICE–ViT for native-resolution perception, stronger chart/document understanding, and a quality-first data recipe. Beyond checkpoints, we also release the data, packing pipeline, configs, logs, and evaluation commands needed for low-cost reproduction (see the technical report).

Pretraining Dataset (85M) and Concept Balancing

The 85M pretraining corpus mixes eight sources, covering roughly 20M Chinese and 65M English image–text pairs. To reduce long-tail sparsity, we use concept balancing with a shared image–concept embedding space and inverse-frequency resampling, then add high-quality bilingual caption augmentation. The result is better data coverage, stronger rare-concept recall, and more stable multimodal transfer.

Instruction Dataset (22M)

The 22M instruction dataset spans captioning, charts and tables, code and math, grounding, OCR, science, and general VQA. Through aggregation, format standardization, rewriting, bilingual conversion, and safety filtering, we keep category balance and reduce template homogeneity. Adding FineVision further improves downstream performance.

Method

1) Visual Encoder Pretraining

To improve OCR, document understanding, and region-level perception, LLaVA–OneVision–1.5 adopts our in-house MVT v1.5 (RICE–ViT) backbone.

Unlike CLIP/SigLIP-style encoders that mainly emphasize global alignment, RICE–ViT is designed to preserve stronger local semantics through Region Cluster Discrimination:

• it is trained on 450M images and 2.4B candidate regions
• it explicitly models local entities, text blocks, and surrounding context through region–cluster discrimination with region-aware attention
• it uses 2D rotary position encoding (2D RoPE) to support native multi-resolution inputs

Rather than stacking several specialized losses, we use a unified clustering–discrimination objective that improves semantic understanding, OCR, and localization together. We then connect the visual backbone to the language model with a lightweight projector and full-parameter joint training, avoiding redundant adapter layers while keeping the pipeline simple.

2) Three‑Stage Learning Pipeline

3) Offline Parallel Data Packing

To reduce padding waste and improve token utilization, we use offline parallel packing:

• hash–bucket clustering by sample length or length ranges to cut global sorting/scanning costs
• multithreaded concatenation of multiple short samples into fixed–length sequences close to the target length during data prep

The pipeline is deterministic, reproducible, and avoids the runtime overhead of online dynamic packing. On the 85M set, it reaches up to ~11× effective padding compression.

4) Hybrid Parallelism and Efficient Long–Context Training

We use hybrid parallelism and long-context optimization—TP + PP + sequence/context parallelism with a distributed optimizer—to improve utilization and memory efficiency at scale. Native-resolution training preserves details in charts, documents, and dense text regions.

from transformers import AutoTokenizer, AutoProcessor, AutoModelForCausalLM
from qwen_vl_utils import process_vision_info

model_path = "lmms-lab/LLaVA-One-Vision-1.5-8B-Instruct"

# default: Load the model on the available device(s)
model = AutoModelForCausalLM.from_pretrained(
    model_path, torch_dtype="auto", device_map="auto", trust_remote_code=True
)

# default processor
processor = AutoProcessor.from_pretrained(model_path, trust_remote_code=True)

messages = [
    {
        "role": "user",
        "content": [
            {
                "type": "image",
                "image": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen-VL/assets/demo.jpeg",
            },
            {"type": "text", "text": "Describe this image."},
        ],
    }
]

# Preparation for inference
text = processor.apply_chat_template(
    messages, tokenize=False, add_generation_prompt=True
)
image_inputs, video_inputs = process_vision_info(messages)
inputs = processor(
    text=[text],
    images=image_inputs,
    videos=video_inputs,
    padding=True,
    return_tensors="pt",
)
inputs = inputs.to("cuda")

# Inference: Generation of the output
generated_ids = model.generate(inputs, max_new_tokens=1024)
generated_ids_trimmed = [
    out_ids[len(in_ids) :] for in_ids, out_ids in zip(inputs.input_ids, generated_ids)
]
output_text = processor.batch_decode(
    generated_ids_trimmed, skip_special_tokens=True, clean_up_tokenization_spaces=False
)
print(output_text)

# pip install git+https://github.com/EvolvingLMMs-Lab/lmms-eval.git
accelerate launch --num_processes=8 --main_process_port 12399 -m lmms_eval \
 --model=llava_onevision1_5 \
 --model_args=pretrained=lmms-lab/LLaVA-OneVision-1.5-8B-Instruct,attn_implementation=flash_attention_2,max_pixels=3240000 \
 --tasks=mmmu_val,mmmu_pro_standard,mmbench_en_test,mmerealworld,mmerealworld_cn,ai2d,ai2d_no_mask,vstar_bench,chartqa,charxiv,docvqa_test,mathvista_testmini,mmstar,scienceqa \
 --batch_size=1

@inproceedings{LLaVA-OneVision-1.5,
  title={LLaVA-OneVision-1.5: Fully Open Framework for Democratized Multimodal Training},
  author={An, Xiang and Xie, Yin and Yang, Kaicheng and Zhang, Wenkang and Zhao, Xiuwei and Cheng, Zheng and Wang, Yirui and Xu, Songcen and Chen, Changrui and Wu, Chunsheng and Tan, Huajie and Li, Chunyuan and Yang, Jing and Yu, Jie and Wang, Xiyao and Qin, Bin and Wang, Yumeng and Yan, Zizhen and Feng, Ziyong and Liu, Ziwei and Li, Bo and Deng, Jiankang},
  booktitle={arxiv},
  year={2025}
}

@article{lillava,
  title={LLaVA-OneVision: Easy Visual Task Transfer},
  author={Li, Bo and Zhang, Yuanhan and Guo, Dong and Zhang, Renrui and Li, Feng and Zhang, Hao and Zhang, Kaichen and Zhang, Peiyuan and Li, Yanwei and Liu, Ziwei and Li, Chunyuan},
  journal={Transactions on Machine Learning Research}
  year={2024}
}