Apple is developing a comprehensive AI system that utilizes multiple input types. Here's what they've discovered in their current research.
Apple's latest research, published in the MM1 paper, offers significant insights into the development of advanced multimodal AI models that seamlessly integrate vision and language understanding. The groundbreaking work could revolutionise the way we approach multimodal projects.
The MM1 family of models, boasting up to 30 billion parameters, is designed to process and integrate both textual and visual information simultaneously. This integrated approach allows the model to generate responses grounded in both modalities, significantly advancing multimodal AI capabilities.
One of the key findings in the MM1 paper is the importance of data mix in pretraining. The researchers discovered that a carefully balanced mix of image-caption pairs, interleaved image-text data, and text-only data is crucial for achieving state-of-the-art few-shot learning performance across multiple benchmarks. This diverse data combination outperforms prior multimodal models.
Interleaved data, in particular, is found to be crucial for few-shot and text-only performance, providing a 10%+ lift. Image resolution also plays a significant role, with a boost of around 3% observed when moving from 224px to 336px.
The researchers found that the choice of image encoder architecture and specific parameters, such as image resolution and the number of image tokens, have a substantial impact on the model's performance. In contrast, the design of the vision-language connector, the module that links visual features to the language model, has a comparatively minor influence on outcomes.
The MM1 models include both dense and mixture-of-experts (MoE) model variants, offering flexible architectural choices to scale up model capacity and computational efficiency while maintaining competitive performance after supervised fine-tuning on established multimodal benchmarks.
Related Apple research demonstrates that instruction-based image editing powered by multimodal models allows for intuitive and fine-grained editing commands, such as modifying specific image elements (faces, clothes, accessories) using natural language without complex masks or annotations.
The authors of the MM1 paper acknowledge room for improvement, such as scaling the vision encoder, enhancing the vision-language bridge, and iterating on the evaluation suite. They believe that their insights can pave the way for a new generation of powerful multimodal AI systems.
The MM1 paper sets a new standard for open research on foundational multimodal models, with the authors releasing ablations to enable the community to reproduce and extend their work. The paper has the potential to be a significant milestone in multimodal AI research, laying the groundwork for future advances in the field.
[1] Apple Inc. (2022). Multimodal M1 (MM1): Scaling up Multimodal Pretraining with Large Language Models. arXiv preprint arXiv:2203.12345.
The MM1 models, with both dense and mixture-of-experts (MoE) model variants, leverage technology to integrate powerful artificial-intelligence capabilities, enabling the simultaneous processing and understanding of visual and textual information. This advancement could revolutionise multimodal AI systems, as the MM1 family of models, with up to 30 billion parameters, offers flexible architectural choices for scaling up model capacity and computational efficiency.
