27.5KB language-agnostic WebGPU syntax highlighter

2026-09-09 | original

read original get Keychron Q1 Pro Mechanical Keyboard → more articles

Why This Matters

gpu-lexer is an experimental 27.5KB library that replaces per-language grammars with a tiny WebGPU-run model that infers token types from context, aiming to highlight code in any language—including ones it never saw in training. If the approach holds up, it points to a much lighter alternative to shipping dozens of grammar files in editors, docs sites, and browser-based tools. The caveat is accuracy: it disagrees with Shiki on 12.57% of token labels in held-out files.

Key Takeaways

A tiny GPU-run model labels tokens by context instead of relying on language-specific grammars, so it can attempt unfamiliar syntaxes.
At 27.5KB it could dramatically reduce the payload of web-based highlighting versus bundling many grammars.
It's explicitly an experiment: 12.57% of labels differ from Shiki on unseen files, and real-world or novel languages may fare worse.

Worth a Look

Keychron Q1 Pro Mechanical Keyboard — If you're the kind of developer who gets excited about a 27.5KB WebGPU syntax highlighter, you probably spend a lot of hours staring at colorful code. The Keychron Q1 Pro is a hot-swappable, gasket-mounted board with QMK/VIA customization and RGB backlighting, so your keys can look as vivid as your editor theme.

See Keychron Q1 Pro Mechanical Keyboard on Amazon → Affiliate link — we may earn a commission on purchases, at no extra cost to you. Product picked by AI based on this article; it is not a tested recommendation.

27.5KB language-agnostic WebGPU syntax highlighter

import { highlight } from 'gpu-lexer' const spans = await highlight('source code') // { // type: 'plain' | 'comment' | 'string' | 'number' | 'keyword' | 'type' | 'function' | 'constant' | 'operator' // start: number // end: number // }[]

gpu-lexer splits source code into simple parts—words, whitespace, newlines, and symbols. Then a tiny WebGPU model combines local and whole-file context to label each part. It is designed for any language: instead of choosing a grammar, it guesses each part's type from the surrounding source, even when it never saw that language or syntax during training. Adjacent labels become the highlight spans returned to your code.

This is an experiment, not a grammar-equivalent highlighter. On files kept out of training, 12.57% of the current model's token labels differ from Shiki. This measures agreement with Shiki—not objective correctness—and unseen languages or real-world code may differ more often.

Explore topics: webgpu gpu-lexer syntax highlighter shiki tokenization