Skip to content
Tech News
← Back to articles

My Favorite Bugs: Invalid Surrogate Pairs

read original get Unicode Character Encoding Guide → more articles
Why This Matters

This bug highlights the subtle challenges developers face with Unicode handling, particularly with surrogate pairs like emojis, which can cause silent failures in real-time collaborative editors. Understanding and addressing such issues is crucial for building reliable, user-friendly software that handles diverse character sets seamlessly. It underscores the importance of robust Unicode support in modern web applications, especially as emojis and other multi-byte characters become ubiquitous.

Key Takeaways

If you're in the business of building things that run on computers long enough, I think you will eventually acquire a favorite bug story. This is a short story about mine. I've also built an interactive tool where you can explore the concepts underpinning the heart of this bug.

The bug: two emoji enter, none leave

I was working on migrating a legacy editor to a more collaborative experience with my team. TipTap on top (itself a wrapper around ProseMirror), Yjs underneath handling the CRDT magic for real-time syncing. It worked well! Mostly.

In our alpha/early release days, when it was still mostly internal and/or early rollout users, sometimes the editor would just stop saving your content. Silently. You'd keep typing and everything looked fine, but your edits stopped syncing to the Yjs document. The next time you opened the page, everything you'd written after the failure point was gone.

It was utterly terrifying, very rare and almost impossible to diagnose because we could never recreate it. We really tried! My early suspicions generally revolved around shaky wifi connections and wonky websocket behaviors, but no amount of throttling or turning my wifi on and off seemed to recreate the issue. The experience was surprisingly resilient in those scenarios, in my memory. It felt like it happened randomly, never when anyone was looking. No obvious errors picked up in the console, no stack trace, no crash. Just... "Hey, I think my changes didn't save."

Then one day our product manager cracked it. This was not a trivial thing to find. He'd been experiencing it more than anyone else (probably because he was the best at dogfooding our product) and had been methodically narrowing it down.

"I feel like I'm going crazy, but I think it's when I type specific characters together, go back and insert a character between them..."

He'd been using 🟢 and 🔴 in his weekly project status emails to communicate general health. Green for on-track, red for at-risk. Every week the template he was using had both characters already present and he would simply remove the one he didn't need (Generally the red one, I am happy to say!).

On this occasion he'd copied the green circle and pasted it in front of the red one at some point, or maybe vice versa. That specific operation— inserting one multi-byte emoji adjacent to another— was triggering a splice in the underlying CRDT library, which split a surrogate pair down the middle.

I remember being on the call when he showed this to me and one of my direct reports who'd been toiling away at the collaborative editing transition. I must've gotten a little too excited—I live for esoteric bugs—"I feel like you got energized by this," he said. He wasn't wrong.

... continue reading