Puppet Parade is an interactive installation by Emily Gobeille and Theo Watson of Design I/O that allows children to use their arms to puppeteer larger than life creatures projected on the wall in front of them. This dual interactive setup allows children to perform alongside the puppets, blurring the line between the ‘audience’ and the puppeteers and creating an endlessly playful dialogue between the children in the space and the children puppeteering the creatures.
The setup consists of a stage where two puppeteers can control puppets displayed on a large interactive screen. The audience can stand in front of the screen and interact with the puppets directly, making food for them to eat which transforms their appearance or just engaging in a fun dialogue with the puppet, the performer and themselves.
There are two kinect cameras mounted on the stage which tracks the arm position of the two puppeteers, mapping their arm position and angle to the body of the puppets. The puppeteers can therefore control with their arm exactly where the puppets are and what they are looking at. Tracking of the hand of the puppeteers allows the team to map the angle between the thumb and forefinger to how open or closed the mouth of the puppet is, allowing the puppets to eat food or in some cases breath fire. Another subtle detail is that the creatures in Puppet Parade are modeled in 3D and can rotate towards or away from the audience when the puppeteer rotates their arm towards or away from the Kinect.
For the narrative, Theo explains, they really liked this idea of ‘you are what you eat’ where the food that the creatures ate would transform them to look like the style of the food.The audience can make the food for the creatures by holding out their hands. Once the food forms they can push it towards the creature (or just have the creature eat it out of their hand ). If someone stands with their arms above their heads they can make a cloud which travels up and which colors the creatures with a cloudy pattern when they eat it. There is also a feather bush and a fire bush, which you can see from the video what they do.
In terms of the interaction they really enjoyed this idea that the audience were also performers creating stories together with the puppeteers. Having them interact with the puppeteers through the puppets, as well as with each other. They actually saw a lot of children going back and forth between being at the wall and being on the stage, so most got to experience the project from both perspectives.
The software is made with openFrameworks 007 and runs on a Mac Pro connected to two Xbox Kinects and an IR camera ( Sony M183 ). For the kinect tracking Theo and Emily used ofxKinect which in turn uses the open-source libFreenect. They also decided to do their own joint tracking as both the Microsoft and OpenNi SDKs lack the ability to track fingers and the team needed to be able to detect when someone had opened or closed their thumb and forefinger as that would then be mapped to the mouth of the puppet. To do the arm tracking they use openCv and lots of ofPolyline operations to find the arm, the elbow, writst and fingers and then the hand of the puppeteer. They then run a bunch of other operations on just the hand region to try and determine whether the hand is open or closed. Its all very custom to the problem at hand but they were really happy with the accuracy.
They have also found the Kinect motor control to be incredibly useful as it allowed them to easily adjust for heights of people ranging from 3-7 feet. If the system saw something but wasn’t finding an arm it would get the kinect to look down or up till it found a good candidate.
Puppet Parade premiered at the 2011 Cinekid festival in Amsterdam. Puppet Parade is made with openFrameworks and the ofxKinect addon.
[ via ]