GoKawiilis a reporter who writes about law, technology, speech, and democracy. A journalist trained as a lawyer, she has been writing about tech for 10 years.
xAI is suing Minnesota Attorney General Keith Ellison over a law passed back in May that broadly targets “nudification” apps, claiming that the statute’s punitive provisions leave the company with “no practical choice but to restrict Grok Imagine’s image-editing features in various ways.” The law, the company argues, violates the First Amendment.
Back in January, Grok flooded the internet with millions of sexually explicit deepfakes, including images of minors. In another era, a content moderation disaster of this magnitude would have had the CEO flipping the kill switch on the product within hours. But in Elon Musk’s brave new world, the chatbot was left to pump out pornographic images for days. “Grok now hitting #1 on the App Store in one country after another!” he posted in the midst of the deepfake epidemic.
Something had gone wrong. Was the law simply not up to the task of curbing an explosion of nonconsensual nudes? Governments around the world were asking fairly similar questions. The EU launched an investigation, as did the United Kingdom. There was backlash from French, Indian, and Malaysian government officials. And while the American federal government was noticeably quieter on the issue, blue states like California and Minnesota took a harder line.
In Minnesota, the state legislature passed a law targeting “nudification” technology, prohibiting site owners from allowing people to “access, download, or use” such software. (The law makes a carve-out for tech that “requires the technical skill of a user to nudify an image or video” — so bespoke sexual harassment through Photoshop is in the clear.) The statute creates a civil cause of action for private individuals who may wish to sue, and also authorizes the state attorney general to seek penalties of up to $500,000 per violation.
The law, touted as the first of its kind, was signed by Gov. Tim Walz in May, and is meant to go into effect on August 1st. It’s a bit of a mystery as to why xAI did not file a lawsuit in the last three months, and instead sued just a couple of days before the statute went into effect. It’s the kind of delayed reaction that makes you wonder if someone at xAI suddenly noticed last week that they were about to be in violation of a fairly punitive law, and then spent the next few days trying to find a law firm willing to work overtime on behalf of a nonconsensual deepfake generator. One can only hope that the lawyers involved are making an absolute killing.
The complaint is pretty reasonable, all things considered. The Minnesota law is overbroad, it argues, because it imposes liability even for images that are consensually generated, or images of adults that are merely sort-of kind-of sexy, or images with “humorous” or “artistic” merit. The First Amendment dictates that laws must be narrowly written so as to avoid banning social and political commentary like this deepfake of Chris Christie and JB Pritzker sumo wrestling.
A real excerpt from page 20 of the lawsuit.
“Liability also attaches … even if the company has deployed near-perfect, state-of-the-art technical controls to prevent the generation of nude images,” the lawsuit states with an excruciating degree of earnestness. Elsewhere, it bemoans the strict liability that attaches for the unsanctioned conduct of its users — “images and videos that users might manage to create in violation of Grok’s terms of service, which is possible only by evading sophisticated, state-of-the-art controls that xAI employs.”
“xAI takes violations of its terms of service seriously,” the lawsuit states, pointing out that the company has since sued one of its users for using Grok to generate CSAM. (The lawsuit against the alleged CSAM-maker was filed over two months after the Minnesota statute was passed.)
... continue reading