Huawei open-sources MACaQ model quantization toolkit for local AI inference
Huawei has released MACaQ, an internal quantizer that compresses text, image, video and audio models for local inference across llama.cpp, vLLM, Diffusers and PyTorch. The company says MACaQ can quantize a 122-billion-parameter model in 64 minutes using 32.5GB of device memory, and claims it achieves lower perplexity than rivals MR-GPTQ and DuQuant++ or matches their Llama-3-8B results in roughly a tenth of the time.