New Zealand design studio Custom Logic has turned the Kinect into full motion-interactive instrument with one hell of a visual display to boot. The project, titled V Motion, was developed as an ad campaign for V, a New Zealand energy drink company. In a recent blog post, the team behind the technology discussed some of the challenges of developing a real-time virtual instrument.
“In the beginning, none of us were really sure what we were building,” wrote Jeff Nusz of the Custom Logic team. “The starting point was the idea of using the Kinect camera to detect the movements of the musician.” Custom Logic began with BiKinect, a tool suite for Kinect developers created by Ben Kuper, as the basis for the project.
The team’s goal was to create a fully functioning virtual interface for a complex piece of music software called Ableton Live. This involved turning many of the dials, switches, and sliders within the program into recognizable gestures for Microsoft’s Kinect camera. Custom Logic came up with an interface where the player can create synth sounds, and even drum hits, by fully extending their arms in separate directions.
Additionally, users can tweak filters by forming and extending a circle shape with their hands, and dampen the sound by adjusting the height of the player’s chest from the ground. Check out the practice video below to see just how some of the gestures work.
One of the biggest challenges for the team was reducing the significant lag that, in early builds, made keeping a rhythm mostly impossible. Nusz said they were forced to introduce another Kinect camera and computer rig to effectively reduce the cumbersome lag, but an additional set of headaches arose around the complications that come with two Kinects sharing the same information points. The developers solved the problem by attaching one of the cameras to a “battery operated wiggling device” that prevented the shaking Kinect from focusing on points of data that were not its own.
Once the rig was finished, Custom Logic took to the streets of Auckland to film the promotional music video of an original song called “Can’t Help Myself.” The song was performed by “motion artist” Josh Cesan in front of the interface projected onto a 100 by 40 foot wall in downtown Auckland.
“This technology is amazing and it feels like the tip of the iceberg,” wrote Nusz. “I’d love to see v2.0… multiple musicians, flexible loop editing, realtime VJing.” Nusz said that with the growing complexity of motion capture technology the team will be able to add new features and improve the overall performance of the V Motion project into the future.