Tech News
← Home  ·  All topics

Inference

66 GoKawiil briefs on this topic · RSS for inference

JEDEC previews DDR6 specs: shorter modules, wider bus, higher speeds

JEDEC has outlined early DDR6 specifications, showing registered DIMMs that are about 44% shorter than DDR5 modules while starting at 12,800 MT/s versus DDR5's 9,200 MT/s ceiling. The data bus widens from 64 to 96 bits, and JEDEC expects these changes to combine for over 50% more bandwidth at launch, alongside support for 128-gigabit dies and new low-power modes for data centers.

New Energy-Based Reasoners cut inference iterations and samples versus EqR

Researchers introduced Energy-Based Reasoners (EBRs), a test-time scaling approach that refines answers through iterative descent steps and selects the best output by picking the candidate with the lowest computed energy across multiple generated trajectories. Benchmarked against EqR (using its released checkpoint) and compared with GRAM and TRM models from GRAM's paper, EBRs reportedly matched EqR's best accuracy using 7 times fewer iterations and 43 times fewer samples, with results averaged across three seeds.

Gemini 4 Carbon flagged for unusually slow task completion in benchmark tests

An evaluation comparing Gemini 4 Carbon against Opus and Astra on coding-bug-fixing tasks found it took nearly an hour to complete work that the other models finished in minutes. Users on social media independently reported similar frustrations, describing the model as slow and hard to use despite liking its capabilities. The evaluators traced each step of Carbon's runs to determine what was causing the delay.

Cerebras combines GPUs with its Wafer Scale chips to boost inference efficiency fivefold

Cerebras has moved its disaggregated inference architecture into production, pairing GPUs for prompt processing with its own Wafer Scale Engine systems for output generation. The company says this setup delivers five times more tokens per megawatt while maintaining its existing inference speeds, and is now available both on its own Inference Cloud and for infrastructure partners to deploy in their own data centers.

Optical computing startups attract $2.22 billion in valuations amid AI power crunch

The optical-computing sector's combined market valuation has grown to $2.22 billion, up from $0.4 billion in 2022, as startups in the US, China, Germany and Israel pursue light-based alternatives to electronic chips. Researchers say the surge is driven by AI inference workloads pushing data centers up against power limits, prompting interest in optics as a potentially more energy-efficient computing method. One example cited is UK-based Optalysys, which builds optical Fourier transform chips designed to slot into data-center networking paths.

Opinion: API-based AI pricing and accuracy swings expose businesses to hidden costs

An Entrepreneur contributor argues that businesses relying on AI through third-party APIs are effectively renting a product whose behavior, pricing and accuracy can change without warning. The piece cites research from Stanford, UC Berkeley, Carnegie Mellon and HKUST showing that identical GPT-4 and GPT-3.5 prompts produced sharply different accuracy results across model snapshots taken months apart, including one case where prime-number accuracy fell from 84.0% to 51.1%.

Sam Altman defends comments on sharing memory amid DRAM shortage

OpenAI CEO Sam Altman clarified remarks about wanting everyone to have abundant RAM, saying the comment was misinterpreted. He noted DRAM prices have risen roughly 5-7x due to supply shortages, and argued that pooling memory for shared use could be more efficient than leaving it idle on individual edge devices.

Developers release open-source Edge0-145B on-device AI model

A development team has open-sourced Edge0-145B and Edge0-120B, two on-device AI models, alongside an inference framework and deployment tools. Edge0-145B is described as the first open-source on-device model with 145 billion parameters, running locally on a compatible laptop with reported peak memory of 29GB and decoding speeds above 80 tokens per second.

New guide frames inference engineering as the next platform-team skillset

A field guide aimed at platform engineers outlines why organizations are shifting to running open-source AI models themselves rather than relying solely on hosted frontier models. It cites DeepSeek's R1, released in January 2025, matching OpenAI's o1 on math and coding benchmarks within about a month, and notes Hugging Face now hosts over two million open models. The guide also references Upbound's Project Champagne work running models on its Modelplane tool, where most engineering effort went into infrastructure like TLS, identity and observability rather than model serving itself.

Developer publishes open-source C++/CUDA LLM inference engine with 85-page handwritten notes

An independent developer has released a project that builds a transformer inference engine from scratch in C++17 and CUDA, documented through over six hours of video and an 80-plus page PDF of handwritten notes. The system currently loads Llama 3.2 1B Instruct weights directly from safetensors files, runs a custom Unicode-aware BPE tokenizer, and executes full dense causal prefill across all 16 layers including RoPE, grouped-query attention, SwiGLU MLP, and RMSNorm kernels to produce next-token predictions via GPU argmax.

Open-source llama-modchip runs 284B-parameter DeepSeek model on 64GB CPU-only machine

A project called llama-modchip, released on GitHub, claims to compress the 284.33B-parameter DeepSeek-V4-Flash mixture-of-experts model from roughly 77GB down to about 57-59GB of memory, letting it run on a 64GB CPU-only Linux machine without a GPU. The developers report prefill speeds near 44 tokens per second and CPU decode speeds of roughly 9-15 tokens per second, achieved through custom quantization, cache-tiling and pruning techniques, plus a separate mode running a 30B Nemotron model at much higher speeds on CPU hardware.

Anthropic unveils FlashProve, an open-source tool to verify which AI model generated an output

Anthropic published a blog post detailing FlashProve, an open-source system that provides cryptographic proof confirming which specific AI model produced a given output. The system relies on labs publishing a 'fingerprint' for each released model, which can later be matched to outputs and verified without exposing the model's underlying weights. Anthropic says verification can run locally and could also confirm third-party claims, such as that a cybersecurity scan was performed by a specific frontier model.