Synthesia to Midi
Synthesia is a midi visualizer. Originally intended to help people learn to play piano, its more popular function is to display a midi file as a bunch of colorful bars falling downwards towards a keyboard.
This is how I came to know of Synthesia. Often times, I'm curious how to play a specific part myself, and many people don't provide sheet music or a midi file to view, so I decided to try and make them myself.
My program simply checks the color values of the keys on the bottom of the screen (they light up when a note is played on them) and compares them to values known to be white (from the beginning of the video when nothing is played). From there, it analyses each key's data for continuity and places note start and stop commands appropriately in the midi file using a library I downloaded.
Processing isn't the best for video analysis, as there aren't actually functions to step through video files, so I had to play and pause the video quickly and then grab the screens pixels. It isn't the best, but it works decently.
Here are some of the tests I ran on the program. On the left is the video I fed in (the first 20-ish seconds of this video chopped up. On the right, is the midi file generated by the program, converted to audio through Musescore.
The notes seem to work pretty well, although the timings could be a lot better. This is because I haven't gotten around to adding a tempo and beat identifier algorithm to it yet, so it ends up quantizing the notes to an unrelated tempo.
INPUT
OUTPUT
(This one is a a little un-satisfying since I cut the video right before the last note, sorry!)
Here's the Processing file, in case you wanted to try it for yourself.
I can't upload a .pde file, so you'll have to copy-paste this into an empty sketch.