Mixed Reality with Keyboard and Kinect

Finally! Tango Chen (aka Jingzhou Chen) is back with a new Kinect project!

MIDI Reality – Mixed Reality with Kinect and MIDI Keyboard

“Tech & music are my favorites. I’d love to do them together.
So I had this idea to try covering a song with a MIDI keyboard, and a Kinect v2 sensor for mixed-reality floating caption effects.
That was when the song (Life Is Worth Living by Justin Bieber) was still new. However, the recording process’s been so procrastinated.
Since it had taken for a very long time, I decided to not making a better cover, but this somewhat bad, weird and incomplete one. And, explaining how that was done more focusedly.
You can consider it as how most mixed reality / augmented reality effects are done behind nowadays. Literally overlaying parts of(or full) virtuality on a reality layer. I can not be 100% sure though, cause I haven’t been looking into any references for this project.
Furtherly, using regular AR marker technology to track position of MIDI keyboard, Kinect can capture around for a movable camera view(Requires a stored 3D background data also). Or just use HoloLens!”