GoKawiilHi, I’m Mark Nottingham. I write about the Web, protocol design, HTTP, Internet governance, and more. This is a personal blog, it does not represent anyone else. Find out more .
What's Missing in the ‘Agentic’ Story
For much of the history of computing, it was reasonably safe to assume that a machine was doing what you told it to do (and what its creators promised it would do), because its operations were local.
You bought a laptop or desktop with an operating system, and it did what it said on the tin: it ran programs and stored files. You bought a spreadsheet and a word processor, and those programs performed those tasks and didn’t do anything else. Software that didn’t do this was in a separate bucket called ‘malware’ and we had ways of dealing with it.
That assumption has a more general precedent in tools – whether they be staplers, screwdrivers, or telescopes. When you buy a screwdriver, it turns screws; it has no agency of its own. It might do other things, but that’s because you’re misusing the tool, not because it decided to do something else. Most things that people use unambiguously follow this pattern: for example, my mechanical wristwatch can’t do anything but tell me the time.
That pattern is perpetuated in most depictions of computers in fiction (especially sci-fi), which work for people diligently and always on their behalf, usually with minimal intrusion. They unambiguously act in the interest of their users, following in the footsteps of technological optimism which informs much of fiction and influenced a generation of nerds who tried to build it.
All of these experiences combine to lead people to trust computers fairly unquestioningly; they don’t give much thought to the other purposes that might be served. When I use my phone, it’s my phone, and so it’s working for me, right? This is perpetuated in the press: recently, I saw an article in a major newspaper about how to talk to “your” AI agent.
If you scratch the surface just a bit, however, none of this is true when applied to modern technologies, and these assumptions are not safe.
The State of Trust on the Internet
Every time you use an Internet-connected computer, you’re trusting someone (and most likely, a multitude) to act on your behalf. From an application’s code all the way down to the silicon, software and hardware and the network services they use reliably embed the interests of those that create them – and they may or may not be aligned with yours.
... continue reading