The development of large Vision-Language Models (VLMs) has greatly improved multimodal tasks like image captioning, visual question answering, and cross-modal retrieval by better integrating visual and textual information. However, while VLMs show strong performance, researchers still lack a full understanding of their compositionality — the ability to grasp and generate new combinations of known visual and textual elements. Current benchmarks assess compositionality mainly through objects, relations, and attributes, but overlook deeper aspects like object interactions, counting, and complex compositions. To address these limitations, we introduce MMComposition, a novel benchmark specifically designed to comprehensively evaluate the compositionality of vision-language models (VLMs). MMComposition assesses VLMs across three main dimensions: vision-language (VL) compositional perception, reasoning, and probing. Unlike previous benchmarks that primarily focus on text-to-image retrieval or single-choice questions, MMComposition offers a diverse set of 4,342 tasks, including single-image and multi-image scenarios, single-choice and indefinite-choice questions. This ensures a thorough evaluation of models' ability to handle complex compositional tasks across modalities. Our findings reveal that even state-of-the-art models like GPT-4o struggle with fine-grained compositional reasoning, highlighting the need for further advancements in VLMs' compositional capabilities. Our key contributions are:

1. Introducing MMComposition, a novel, high-quality benchmark for evaluating the compositionality of pre-trained VLMs across perception, reasoning, and probing.
2. Providing a comprehensive experimental evaluation of 54 state-of-the-art VLMs, demonstrating the challenging nature of MMComposition and exposing significant gaps between model and human performance.
3. Analyzing the critical factors in VLM architecture that influence compositionality, offering insights for future improvements in model design and training.