Tech News
← Home  ·  All topics

Cpu Inference

3 GoKawiil briefs on this topic · RSS for cpu inference

Open-source llama-modchip runs 284B-parameter DeepSeek model on 64GB CPU-only machine

A project called llama-modchip, released on GitHub, claims to compress the 284.33B-parameter DeepSeek-V4-Flash mixture-of-experts model from roughly 77GB down to about 57-59GB of memory, letting it run on a 64GB CPU-only Linux machine without a GPU. The developers report prefill speeds near 44 tokens per second and CPU decode speeds of roughly 9-15 tokens per second, achieved through custom quantization, cache-tiling and pruning techniques, plus a separate mode running a 30B Nemotron model at much higher speeds on CPU hardware.

AMD merges ZenDNN into Intel's oneDNN, joins UXL Foundation as steering member

AMD is folding its ZenDNN library, previously a downstream fork, into the open-source oneDNN project maintained by Intel under the UXL Foundation. AMD also joined UXL as a steering member, paying $20,000 annually for influence over the foundation's direction. The merger was recommended by the x86 Ecosystem Advisory Group, a joint Intel-AMD initiative formed in 2024.

Jevstiller open-sourced: local model distills Jev with a set disagreement budget

A Show HN project called Jevstiller introduces a local CPU model trained on outputs from the Jev classification API, aiming to answer most requests in about 15 milliseconds instead of the roughly 300 milliseconds a network call to Jev takes. Users set a target agreement rate (e.g. 98%), and the system routes only enough traffic to the local model to keep its combined coverage and error rate within that budget, sending the rest to Jev.