GoKawiilSteelman Labs · Notes
Computer use is far from solved
Right now, agentic computer use is one of the biggest levers for real-world AI impact. LLM-based agents are transforming software development, but most intellectual work is gated behind using software. When coding agents are so good, it is natural to ask: can they file my taxes in a government portal, fix a text document, test a website? We are not there yet. On OSWorld-V2, a leading benchmark of long-horizon computer tasks, the best model achieves only 20.6% completion rate. On Agents' Last Exam the best result is 26.2%. Users accustomed to the impressive performance of LLMs in chat interactions expect similar results from computer use. But they are met with frustration: agents are unreliable, slow and expensive. At the moment, it's easier to just do the work yourself. OSWorld-V2 completion (binary reward) against output tokens per task, across models and reasoning-effort settings. The frontier tops out around 20.6% — and every extra point of reward costs sharply more tokens. In 2024–2025 every major AI lab shipped a computer-use prototype. At the time, these showed excellent performance on benchmarks such as OSWorld, WebArena, AndroidWorld and WebVoyager, boasting task completion rates from 60.76% up to 97.4%. Those benchmarks were a good fit for the agents. With few exceptions, they presented each task as mostly static environments with clean text representations and small pre-defined action spaces — which is nothing like the real world of messy interfaces. Hence the inflated leaderboards and the gap between user expectations and real performance. So far, the answer to improving computer use has been more of the same: larger models and more tokens. In our opinion, this direction is a dead end. The established approaches look good in cushioned environments, but don't generalize to real GUI work.
The core flaw of computer use Computer use has well-known hard parts such as perception, planning and preserving context across a long task. Those issues are being addressed by the community and some of this genuinely requires smarter models. However, a simpler issue is getting less attention, and it might be the bottleneck: interface-using agents mostly avoid the interface. On OSWorld-V2, the best models often aren't operating the UI at all. In task 052, to book a hotel suite, GPT-5.5 injected JavaScript to read the site's source, found the internal API endpoint, and POSTed the reservation. In task 065, to buy a train ticket, both GPT-5.5 and Claude Opus skipped the website and POSTed straight to the API. To compose two images in GIMP in task 003, GPT-5.5 scripted in Python. Overall, most tasks were not solved as intended, but bypassed instead. OSWorld task 068: instead of playing the game through its UI, Claude Opus 4.7 goes after the state directly — “update last_score to 150 via API” — POSTing the score rather than earning it. This can be overlooked or even seen as advantageous. Why touch the GUI if you can make an API call? However, some work can only be done through the UI. Bypassing the UI can have unexpected side-effects, because the software was not designed to be used this way. More importantly, a lot of work is much easier to get done through the UI — which is important to get economically feasible computer use. Clicking buttons, filling forms and navigating websites are not hard tasks. A person does it without thinking. Reverse-engineering a site's API, disassembling its JavaScript or scripting a compositor from scratch is more complex than the click it replaces. Doing it directly would be cheaper, faster and much more reliable. However, for frontier models it's not an option. The WebGames benchmark challenges models to complete simple tasks that require the reaction time and motor control of an average website user. The human success rate is over 95%, but models lag far behind despite their intelligence. That gap does not narrow with more tokens or parameters. This is why, in long-horizon tasks, models reach for hard solutions: they do not have the ability to use interfaces on a human level, so they have to work around that limitation. This leads to a suboptimal compute allocation. Agents spend most of their actions on perception and action. Figure 10 from OSWorld 2.0 breaks down the action budget: visual grounding, low-level manipulation and tool-use overhead dominate the rest. Planning how to solve a task is supposed to be harder than clicking a button, but most computation goes to seeing and clicking. Reasoning, reflection and error recovery get what's left. Figure 10, OSWorld 2.0 — share of the action budget by category across models. Perception and action dominate; reasoning, reflection and correction get what's left. In our opinion, this is the core issue with agentic computer use: we use trillion-scale reasoners to work around clicks. Proper manipulation would allow agents to spend effort on solving the task instead of hacking around interface limitations. This would enable faster execution, fewer tokens burned in vain, improved reliability and better long-horizon performance.