Want smarter insights in your inbox? Sign up for our weekly newsletters to get only what matters to enterprise AI, data, and security leaders. Subscribe Now
Researchers have published the most comprehensive survey to date of so-called “OS Agents” — artificial intelligence systems that can autonomously control computers, mobile phones and web browsers by directly interacting with their interfaces. The 30-page academic review, accepted for publication at the prestigious Association for Computational Linguistics conference, maps a rapidly evolving field that has attracted billions in investment from major technology companies.
“The dream to create AI assistants as capable and versatile as the fictional J.A.R.V.I.S from Iron Man has long captivated imaginations,” the researchers write. “With the evolution of (multimodal) large language models ((M)LLMs), this dream is closer to reality.”
The survey, led by researchers from Zhejiang University and OPPO AI Center, comes as major technology companies race to deploy AI agents that can perform complex digital tasks. OpenAI recently launched “Operator,” Anthropic released “Computer Use,” Apple introduced enhanced AI capabilities in “Apple Intelligence,” and Google unveiled “Project Mariner” — all systems designed to automate computer interactions.
OS agents work by observing computer screens and system data, then executing actions like clicks and swipes across mobile, desktop and web platforms. The systems must understand interfaces, plan multi-step tasks and translate those plans into executable code. (Credit: GitHub)
Tech giants rush to deploy AI that controls your desktop
The speed at which academic research has transformed into consumer-ready products is unprecedented, even by Silicon Valley standards. The survey reveals a research explosion: over 60 foundation models and 50 agent frameworks developed specifically for computer control, with publication rates accelerating dramatically since 2023.
AI Scaling Hits Its Limits Power caps, rising token costs, and inference delays are reshaping enterprise AI. Join our exclusive salon to discover how top teams are: Turning energy into a strategic advantage
Architecting efficient inference for real throughput gains
Unlocking competitive ROI with sustainable AI systems Secure your spot to stay ahead: https://bit.ly/4mwGngO
... continue reading