The Kinect Is Actually an Amazing Movie Camera

Remember when you would watch video games on your family’s big wood-panelled tube TV, controlling your little character by mashing plastic buttons on a controller? Now, thanks to the Microsoft Kinect, the controller is your body, and the video game watches you. This isn’t news — the technology has been around since 2010. But the crazy things that you can do if you hack the Kinect are. 

James George and Jonathan Minard are two of the latest pioneers to break conventions using the device. Since last year, the two coder-artists have been developing an innovative filmmaking technique that turns the Kinect sensor into a 3D movie camera. They’re calling the new format RGB+D. Using a DSLR to color the image, the resultant video looks like a cross between a video game and a Richard Linklater experiment. They want the viewing experience to follow suit and are assembling their first film, a self-referential, Kickstarter-funded documentary about software art called Clouds, as a choose-your-own-adventure sort of thing. “The goal of building such a system is to leave the possibility space wide open, to create an experience that feels spontaneous and full of surprises,” the duo told The Verge this week. “We want Clouds to feel like a continuous revelation in a space of ideas.”

It feels like the feature-length Kinect movie was inevitable from the moment that code wizards first hacked the device. Its very design is uniquely positioned for next-gen video capture techniques. The Kinect is, after all, just a camera to begin with. Like a bat using sonar in the dark, the device blasts out a bunch of infrared dots and measures the distance of objects in its field based on how long it takes for the dots to bounce back. The embedded imaging software then reconstructs the objects in the room using a “depth map,” and any changes in position are registered as movement. Unlike a normal video camera, though, the richness of the data that the Kinect collects can allow for multiple perspectives, and while the resultant “image” isn’t quite as sharp as something that a Canon 5D might produce, it’s simply different.

This gets at philosophical underpinnings of Clouds. George and Minard say that they’re drawing from the ideas of sci-fi author Bruce Sterling, who recently said that he imagind cameras of the future would be devices “that absorbed photons from all angles at all times, turning the act of taking a picture into a computational problem of choosing which angle at which moment to visualize.” Sounds kind of like the Kinect, right? It’s proof of that William Gibson motto–“The future is here. It’s just not very well distributed yet.”

Image via Vimeo