GoKawiilAgent loops are often oversimplified. They’re presented as a single loop, when really it’s three loops in a trench coat that make up an “agentic” experience for a customer. I’m here to write (yes, I wrote this, insane right?) yet-another-blog about agent loops. The example code blocks are also pseudo-code and for illustrating these ideas. Also I’ve omitted streaming, which complicates the post but the shape of these stays the same.
I even made this image!
The Inference Loop
The most reductive way to explain Large Language Models is that they take text, and predict the next charact... err... tokens. Unlike your ex, they really do complete your sentences. This is achieved with an inference loop.
Your inference loop has three responsibilities:
Make chat completion API calls (infer the next words) Pass a tool usage request to your tool loop (more later) Manage chat history persistence (of tool results, or more user messages)
When you’re building an agent, the first “outer” loop you’re creating is the inference loop. This is the loop that sends a system prompt, user and assistant messages, and available tools to an LLM’s “chat completion” endpoint. Anthropic has one. OpenAI has one. OpenRouter makes all of them easy to use.
Once an LLM returns its messages, it’s your responsibility to append them to a “chat history” so that a conversation can be continued. This can be an array, a database table, Redis keys, whatever. Your prerogative.
chatMessages = [ { role: "system", content: "You are a couples therapist. You help people either make it work, or make them realize it will never work. The user you're speaking with is named Tom." }, { role: "user", content: "I kinda miss Laney, what should I write to her?" } ] continueInferenceLoop = true while continueInferenceLoop inferred = aiClient.completeChat(messages: chatMessages) chatMessages.push({ role: "assistant", content: inferred.message.content, toolCalls: inferred.toolCalls, }) if inferred.toolCalls.length == 0 continueInferenceLoop = false else # Handle tools (see more later) end end finalResult = chatMessages.last.content puts "Assistant responded with: #{finalResult}"
The API design of (most) Large Language Model providers is a stateless design. This means the model provider has no idea what your previous conversation was with it. You need to provide the entire conversation every time. This is why you see the warning of “Large conversations use tokens faster” — yes I typed an em-dash — because you are sending the tome of messages you’ve built up about whether you should reach out to your ex.
... continue reading