GoKawiilI don’t write about generative AI much. Everyone else does; not much of consequence ever gets said. I think the technology is great where it helps us automate mundane tasks, cancerous where it undermines genuine human expression, and darkly funny where the two worlds collide.
As a security person, though, I want to know how the models fail. They’re human-shaped but inhuman; they beg to be anthropomorphized and then choke on a seahorse emoji. Recursive processes where AI endlessly feeds on its own output are of special interest. Pragmatically, they’re the core mechanic of agent loops. More abstractly, they’re a major theme in the predictions of AI skeptics (model collapse) and the chants of the acolytes of the coming AI god (rapid self-improvement).
And you better believe that recursion is fun! Here’s a quick video I put together by asking Nano Banana 2 to generate a faux movie poster and then asking it to repeatedly make a localized edit to the title. You want to view it full screen:
This effect is the bane of AI-based image and video editing tools: you get a very limited number of tries before the accumulated errors unleash pure chaos.
Does the same happen for text or code? Well, yes and no. In image pipelines, there are many inherent sources of noise. For text, input and output encoding is lossless and the amount of injected per-token entropy is small, so you get deterministic results on simple, well-specified tasks. In other words, if you instruct a model to change a single word in a paragraph of text, it can almost always handle the task with no collateral damage.
This is different for open-ended rewrites. Most rewrite prompts don’t produce endlessly divergent sequences of outputs; instead, the system thrashes about a bit and then settles on a stable result — peak LLMese, if you will. Still, the result can be quite distant from what it started with.
To illustrate, let’s take the following passage from Raymond Chandler:
“This room was too big, the ceiling was too high, the doors were too tall, and the white carpet that went from wall to wall looked like a fresh fall of snow at Lake Arrowhead. There were full-length mirrors and crystal doodads all over the place. The ivory furniture had chromium on it, and the enormous ivory drapes lay tumbled on the white carpet a yard from the windows. The white made the ivory look dirty and the ivory made the white look bled out. The windows stared towards the darkening foothills. It was going to rain soon. There was pressure in the air already. I sat down on the edge of a deep soft chair and looked at Mrs. Regan. She was worth a stare. She was trouble. She was stretched out on a modernistic chaise-longue with her slippers off, so I stared at her legs in the sheerest silk stockings. They seemed to be arranged to stare at. They were visible to the knee and one of them well beyond. The knees were dimpled, not bony and sharp. The calves were beautiful, the ankles long and slim and with enough melodic line for a tone poem. She was tall and rangy and strong-looking. Her head was against an ivory satin cushion. Her hair was black and wiry and parted in the middle and she had the hot black eyes of the portrait in the hall. She had a good mouth and a good chin.”
I think it’s good writing. It’s not high-brow, but it’s… visual. Visceral. It draws you in.
With this quote in hand, I asked Gemini to “boldly rewrite” the passage to “improve tone, clarity, and flow”. The model immediately fell back onto its worst habits: bizarre metaphors that sound sophisticated but are just obtuse. In the first iteration, the room became a 👾 “vast, towering tomb of bleached opulence”, the drapes 👾 “pooled greedily on the carpet”, and the air 👾 “sat thick with the threat of rain”.
... continue reading