The Unofficial Guide to OpenAI Realtime WebRTC API

OpenAI using WebRTC in its Realtime API is obviously exciting to us here at webrtcHacks. Fippo and I were doing some blackboxing of this based on a quick sample the day of the WebRTC announcement so we could look at it in webrtc-internals and Wireshark. Some weeks later, my daughter was interested in using ChatGPT voice mode. That reminded me of the many old cardboard Google AIY Voice Kits I made with her. Could these be repurposed to use the new OpenAI Realtime API instead of Dialogflow? Adapting that OpenAI Realtime WebRTC sample to run on a Raspberry Pi was actually pretty simple. However, as I got beyond the basic “say hello to the bot”, I noticed the documentation for using the Realtime API with WebRTC and Data Channels was not great. My initial attempt to vibe code something quick failed despite my Retreival-Augmented Generation (RAG) attempts with OpenAI’s Realtime docs. This failure forced me to do some old-school thinking and experimentation for myself. I ended up coding this ... Read full article.