Tech News
← Home  ·  All topics

Macaq

1 GoKawiil brief on this topic

Huawei open-sources MACaQ model quantization toolkit for local AI inference

Huawei has released MACaQ, an internal quantizer that compresses text, image, video and audio models for local inference across llama.cpp, vLLM, Diffusers and PyTorch. The company says MACaQ can quantize a 122-billion-parameter model in 64 minutes using 32.5GB of device memory, and claims it achieves lower perplexity than rivals MR-GPTQ and DuQuant++ or matches their Llama-3-8B results in roughly a tenth of the time.