Tech News
← Home  ·  All topics

Openai-Compatible Api

3 GoKawiil briefs on this topic · RSS for openai-compatible api

TrnRT open-source inference engine brings GLM-5.3-Flash to AWS Trainium2

RunanywhereAI released TrnRT, an open-source LLM inference engine built for AWS Trainium chips, launched via a Show HN post. It serves an OpenAI-compatible API from a single trn2.48xlarge instance, using fused NKI 'megakernels' per layer and a prefix cache spanning device and host memory, with GLM-5.3-Flash as its first supported model.

Open-source spinf-serve lets developers run bulk AI scoring on a single GPU

A developer has released spinf-serve, an Apache 2.0-licensed inference server that reads a piece of content once and answers many fixed-choice questions about it in a single pass, rather than generating text. It runs on a single 24GB NVIDIA GPU, offers an OpenAI-compatible API matching the existing hosted spinf service, and supports both open instruction models and dedicated decision models.

Developer releases Janus, a Go binary for running GGUF models via Vulkan

A developer has published Janus, an open-source single Go binary that runs local .gguf language models on AMD, Intel or Nvidia GPUs via Vulkan, or on CPU, without requiring Python, Docker or Ollama. It exposes an OpenAI-compatible API alongside a built-in web UI with chat, tool-calling, memory and file-handling features, and supports hot-swapping models without restarting.