Tech News
← Home  ·  All topics

Inference

67 GoKawiil briefs on this topic · RSS for inference

LemonSeed Studio runs unmodified Linux amdgpu driver on iPad via PCIDriverKit

LemonSeed Studio is an iPad app that embeds the unmodified upstream Linux amdgpu and amdkfd driver as a PCIDriverKit extension, allowing it to control an AMD GPU connected through a Thunderbolt enclosure. The app also bundles the LemonSeed Engine for on-device LLM inference running on that GPU via its HSA runtime, plus amdgpu_mtopg for live GPU telemetry. The project's source is available via git with recursive submodules, and is dual-licensed under MIT or GPL-2.0-only.

Prime Labs rewrites Prime Agent coding assistant from TypeScript to Rust

Prime Labs has released a Rust rewrite of its Prime Agent coding tool, which it says has been downloaded over 300,000 times and processed more than 8 trillion tokens since its August launch. The company says the rewrite was carried out over two weeks by a swarm of more than 2,000 AI subagents running across over 10,000 Prime Sandboxes, consuming over 200 billion tokens via its Prime Inference GLM-5.3 endpoint.

Google releases EmbeddingGemma 2, adding multimodal embeddings on-device

Google launched EmbeddingGemma 2, a follow-up to its original EmbeddingGemma model that reportedly surpassed 20 million downloads. The new 740-million-parameter model is built on the Gemma 4 architecture, released under an Apache 2.0 license, and extends beyond text to embed code, images, video, and audio in a single shared space for on-device inference.

Cerebras Leads AI Inference Hardware Race with $5.55B Nasdaq IPO

Cerebras Systems, a key player in inference-focused AI chips, successfully launched its IPO on Nasdaq, raising $5.55 billion and achieving a valuation of approximately $23 billion. The company's early backing by the author, who invested when it was valued at around $2 billion, underscores its prominence in the hardware infrastructure for running AI models efficiently at scale.

Redis creator releases ds4, a local inference engine for frontier open-weight LLMs

Salvatore Sanfilippo, the creator of Redis, has released DwarfStar 4 (ds4), a lightweight C-based inference engine for running large open-weight language models locally on high-memory Mac, CUDA and ROCm hardware. The tool supports models including DeepSeek V4, V4.1 Flash, GLM 5.x and Qwen3.8 Flash Next, covering text and vision, with a local API, CLI and native agent, and is released under the MIT license.

Team's month-long GLM 5.3 Flash coding trial derailed by costs, outages

A development team set out to run an entire month of coding work on the GLM 5.3 Flash model, keeping within a $68 budget for the first two weeks. The second half of the month saw usage spike to 1 billion tokens across other models, driven by an expensive vibe-coded Wagtail MCP server prototype that alone cost $150 and 5kWh, plus infrastructure capacity issues that forced switches to DeepSeek V4.1 Flash and Qwen 3.8 Flash.

Jevstiller open-sourced: local model distills Jev with a set disagreement budget

A Show HN project called Jevstiller introduces a local CPU model trained on outputs from the Jev classification API, aiming to answer most requests in about 15 milliseconds instead of the roughly 300 milliseconds a network call to Jev takes. Users set a target agreement rate (e.g. 98%), and the system routes only enough traffic to the local model to keep its combined coverage and error rate within that budget, sending the rest to Jev.

Modal Labs nears $750M round at $15.75B valuation, led by Accel

AI inference infrastructure provider Modal Labs is close to finalizing a $750 million funding round led by Accel, valuing the company at $15.75 billion including the new capital, according to a source familiar with the deal. That figure more than triples Modal's $4.65 billion valuation from a $355 million raise just four months earlier. Modal declined to comment on the report.

Developer runs 0.5B BitNet LLM across a 7-node ESP32S3 cluster

An open-source project splits a 0.5-billion-parameter 1.58-bit BitNet language model across seven ESP32S3 microcontrollers connected via SPI daisy-chain. One master unit handles tokenization, embeddings and final sampling, while six compute nodes each process four transformer blocks using 1.58-bit attention and MLP layers with KV caching in PSRAM.

New browser tool 'MicroLLM Lab' benchmarks seven small LLMs locally

MicroLLM Lab is a browser-based tool that lets users load and run up to seven small language models, including ones as small as 135M parameters, directly in the browser. It runs objective test suites using regex and exact-token checks rather than judging writing quality, and measures metrics like peak speed, sustained decode speed across 256 tokens, and total benchmark wall time.

OpenAI's Richard Ho explains Jalapeño inference chip in Hot Chips interview

OpenAI's VP of Hardware, Richard Ho, spoke with Tom's Hardware following the August 2026 Hot Chips reveal of the company's Jalapeño inference ASIC. Ho said the chip's development leaned heavily on AI tools to compress the design timeline, and that efficiency, not raw performance, was the primary driver behind building custom silicon.

Epoch AI: Cost of matching AI performance fell 47% per quarter since 2023

A new Epoch AI report by Emberson and Roodman finds that the price of achieving a given level of AI performance has dropped roughly 47% per quarter over the past three years, a 13-fold annual decline. The report cites OpenAI's o3, which cost about $0.30 per question to score 75% on GPQA Diamond in January 2025, compared with a much newer model reaching similar scores for a fraction of a cent by mid-2026.