The next version of Windows will be stuffed to the gills with AI. You may be asking, “Even more than before?” Yes, and Microsoft hopes to train you to quit using your keyboard and mouse to handle your PC. The company hopes you’ll use your voice to command your PC, like you’re some domineering captain on a ship and your crew is a hapless chatbot who’s desperate to understand your vague whims.
Starting Thursday, Microsoft is pushing more “experimental” features and future apps that will put the company’s Copilot AI directly in front of your Windows experience. Microsoft already ensured there would be a Copilot key on all new PCs that access Microsoft’s chatbot. Now, once you enable it in the Copilot app settings, you can start talking to your computer by screaming “Hey, Copilot” at your screen.
Copilot Vision looks at your screen and tells you what to do
If anybody still misses Cortana, now’s your time to shed a single tear. Microsoft already had its Copilot Vision function available on Edge browsers, but now it’s stretching its legs within the wider Windows software suite. Unlike past voice assistants, the new version of Copilot will have AI image recognition, and it should be able to comprehend what’s happening on your screen. This should mean you may have to issue less detailed prompts to the AI to get it to do what you want. And what does Microsoft expect you to use AI for? Well, it could replace all those how-to articles you see online. If you tell Copilot, “Show me how to get better quality audio in Spotify,” Microsoft said it will highlight the setting you need to hit on your screen.
Microsoft sat me down for a demo with the new Copilot Vision feature, though I’ve yet to try it using my own voice. The Voice dialogue was surprisingly fast in responding to queries about a math problem or about buying a dress online. However, when the user tried to get the AI to point out the right controls for changing image resolution on their Shopify account, the AI circled the wrong part of the page. It’s the curse of all live demos that something will likely go awry, but we can expect some idiosyncracies as Microsoft tries to get us talking to our Windows machines.
Microsoft said this AI vision system can look at an image on your screen and offer descriptions of what it sees. Apparently users would use this to type out a resume based on their own portfolio. In another example, Microsoft showed Copilot humming a mindless tune for a musician to riff off (no, the tune did not sound especially appealing). Copilot can now look at all your browser tabs at once and find products based on what you’re looking at. Google has also promoted AI shopping, though with more virtual “try on” features that create an AI image of yourself to imagine your body in that dress.
You’re going to start seeing new Copilot commercials real soon. These are designed to “teach” you about the fun and pleasure of using Copilot with your voice. But that’s not all. There are more full-blown features supposed to get you using AI. Windows Insiders will be able to access beta features that will put a Copilot function crowding out the other functions on your taskbar, replacing the regular Windows search bar (you can still use it to search for files or settings, just as before).
A Copilot Actions app literally takes over your PC
Microsoft said users already talk to their PCs, though usually for the sake of dictation or notetaking. Plus, speech recognition is already a standardized feature for accessibility purposes. Still, there’s a wide gulf between those use cases and literally talking to your PC without annoying your deskmates trying to work just a few feet away. Instead of offering the ability to type to Copilot Vision out of the gate, Microsoft is limiting it to Windows Insiders beta testers to start.
Beta testers will also be first to try out the newfangled Copilot Actions app. Microsoft described the application as an “AI agent” that can take actions for you across different apps and files. In AI circles, an “agent” is essentially multiple AI models working together to complete a more complicated task. On Windows, this means it could essentially take over your PC, run programs for you, and fulfill your demands. Anthropic’s Claude AI showed off similar PC-takeover capabilities last year.
... continue reading