Kaleidoscope: In-language Exams for Massively Multilingual Vision Evaluation The evaluation of vision-language models (VLMs) has mainly relied on English-language benchmarks, leaving significant gaps in both multilingual and multicultural coverage. While multilingual benchmark...

Check out the paper at:
📜Paper: arxiv.org/abs/2504.07072
💿Data: hf.co/datasets/Coh...
🌐Website: cohere.com/research/kal...
Huge thanks to everyone involved! This was a big collaboration 👏

Today we are releasing Kaleidoscope 🎉

A comprehensive multimodal & multilingual benchmark for VLMs! It contains real questions from exams in different languages.

🌍 20,911 questions and 18 languages
📚 14 subjects (STEM → Humanities)
📸 55% multimodal questions

Me :)