Why This Matters
This benchmarking study highlights the performance of small AI models on mobile devices, emphasizing their potential for real-world applications without requiring extensive hardware resources. It provides valuable insights for developers and consumers interested in efficient AI solutions that can run seamlessly on smartphones. The collaboration and validation efforts ensure reliable data, guiding future innovations in mobile AI deployment.
Key Takeaways
- Small models (<8GB) can effectively run on mobile devices.
- Real-world inference data helps optimize AI performance for consumers.
- Validated benchmarks guide developers in choosing efficient AI models.
We benchmark small models on mobile phones. Artificial Analysis' testing covers model intelligence on a set of benchmarks chosen to represent real-world mobile device usage, and we partner with Liquid AI to gather real inference data measured on the devices themselves. Note: we have independently validated Liquid AI's inference measurement process.
“Small” models are all models that fit inside 8 GB of memory after quantization, including KV cache at 8K context. View all rules and our process in the methodology page.
GoKawiil