As I pointed out recently, while Whisper is top of mind and still a pretty good transcription model, OpenAI has moved away from it. That said, the fact that Apple’s new transcription API is faster than Whisper is great news. But how accurate is it? We tested it out.
Full disclosure: the idea for this post came from developer Prakash Pax, who did his own tests. As he explains it:
I recorded 15 audio samples in English, randomly ranging from 15 seconds to 2 minutes. And tested against these 3 speech-to-text tools. Apple’s New Transcription APIs
openAI Whisper Large v3 Turbo
Eleven Lab’s scribe v1
I won’t include his results here, otherwise you’d have no reason to head to his interesting post and check it out for yourself.
But he did add this caveat about his methodology. “I’m non-native English speaker. So the results might slightly vary for others,” and his tests got me curious regarding how Apple and OpenAI would pit against NVIDIA’s Parakeet, which is by far the fastest transcription model out there.
How I did it
Since I’m not a native English speaker either, I decided to use a recent 9to5Mac Daily episode, which was 7:31 long.
I used MacWhisper to run OpenAI’s Whisper Large V3 Turbo, and NVIDIA’s Parakeet v2. For Apple’s speech API, I used Finn Vorhees’ excellent Yap project. I ran them on my M2 Pro MacBook Pro with 16GB of RAM.
... continue reading