There are a lot of piano “tutorial” videos out there, but most of them aren’t step-by-step walkthroughs. They mostly look like falling raindrops that light up piano keys. These tutorials are often made with a tool called Synthesia.
I’ve struggled with these videos because I can’t keep up with the falling notes or figure out the chords I need to play. It’s up to you to keep track of the light bars.
For example, I wanted to learn a Telugu song called Samayama, but it only came in video tutorial format.
So I built PianoReader. PianoReader is a web tool that watches piano tutorial videos and spits out the notes and chords. As a bonus, it does it all in the browser without extra server compute.
It takes in a tutorial video and outputs piano tablature that looks like this:
|Left Hand | Right Hand | | -------- | ---------- | | A Maj | A D | | A Maj | A | | D min | D |
How it works
I knew this problem was going to involve some level of computer vision. It needed some kind of classifier or bounding box detection to make it work.
Alternatively, I considered training a model. But I didn’t want to sit and label hundreds of videos to predict notes. I mean, if I did the work to label just one video, I wouldn’t even have to build this :)
HTML Canvas
... continue reading