GoKawiilHow much GPU RAM do you actually need to run Qwen3.8 27B without sacrificing quality?
The full BF16 model weighs 55 GB, putting it beyond most consumer hardware. Yet the 17 GB Q4_K_M matches the full model on a popular agentic coding benchmark, Terminal-Bench 2.1. It fits on a 24 GB card such as RTX 4090, still leaving room for about 64k tokens of context.
Compression eventually hits a cliff. At 1 bit, the model performs around random chance on GPQA Diamond, and longer reasoning makes it worse.
Background
Qwen3.8 27B GGUF quantizations available from Unsloth on Hugging Face. So much to choose from! I will check 8-bit Q8_0 (29 GB), 4-bit Q4_K_M (17 GB), 2-bit UD-Q2_K_XL (10.7 GB), and the smallest one possible, 1-bit UD-IQ1_S (6.2 GB).
Previously, I investigated the Qwen3.6 27B model, which was good at generating SVG pelicans even at 12GB, and maintained most of its knowledge up to 16GB. At the same time, in Reddit threads, many complain that all quantizations, even the 8-bit ones, give worse results - with people asking why your local LLM feels dumber than it is. Are these complaints grounded?
Measuring token prediction differences (KL-divergence, top-1 predictions) is easy, but it does not tell us whether the model gets worse at solving tasks. Some noise might be irrelevant for solving tasks, as (say) a quantized model generates an answer of precisely the same quality, paraphrased a bit. In other cases, a single different token might be a logical error, or even abruptly end the output.
Download as PNG 70% 80% 90% 100% 6 10 20 30 50 GB model size on disk same top-1 token as BF16 UD-IQ1_S UD-IQ1_M UD-Q2_K_XL Q4_K_M Q8_0 BF16
So, I focus on directly measuring results on popular benchmarks - GPQA Diamond, instruction-following IFBench, programming Terminal-Bench 2.1. First, to replicate official results of the full model BF16 , and then to see how quantization affects results.
I burned around $3,000 on Modal GPUs when I ran models with llama.cpp using a build from 16 August 2026 as earlier builds do not work for this model. I could have run it on my own laptop, in principle, but (unlike pelican-generation), these are time-consuming benchmarks.
... continue reading