When we were accepted into the Leap Motion Developer Program in early 2013, it allowed us to prototype applications and gain insight into this unique new way to interact with computers.
The Leap Motion controller is a small USB peripheral device which is designed to be placed in front of a computer screen, facing upwards. Using two infrared cameras and three infrared LEDs to capture depth information, the device observes a roughly hemispherical area, to a distance of about 1 meter.
The device streams frames of data (with a maximum of almost 300 frames per second) to the Leap Motion controller software, where it is analyzed and forwarded to client applications. This data accessible to applications mainly consists of hand, fingertip and pointer tool orientation and position information, but it does not provide depth image or point cloud data.
Having this information that the Leap Motion tracks hands, fingers and pointer tools in 3d space in a very fast and accurate way, we decided to build an experimental application, which was not possible before, because of the lack of suitable input sensor. We came up with the idea of a puppeteering application, in which one can control marionette puppets with hand gestures.
First we made a couple of puppet sketches. We tried to make everything as simple as possible, so we chose the bird for our first puppet to experiment with, since it has a quite straightforward skeleton structure compared to other animals.
We also designed the background stage for the bird, a forest with hills. The stage was cut into layers similarly how the scenes look in an original puppet theatre. These layers were organized in a parallel way with gaps between them. There is a game area between the two main layers, in which the puppets move.
When we had the basic idea of the puppets and background scenes, we came up with visual gags, small stories we envisioned to be played in the application. One of them was turned into a short animation.
The 3d world was done in Blender. We have already used Blender in our previous projects, so modelling, texturing and rigging was fairly straightforward. For the physics simulation we chose to work with the Bullet Physics library. Since Blender has also got Bullet included, we intended to plan the physics there as well, in addition to the usual modelling tasks.
The creation of the stages was nothing complicated. The images drawn by the graphic designer were cut to layers, vectorized and imported to Blender from svg format. The layers then got uv mapped and extruded to reach their final look.
All scene layers were arranged to face a fixed camera in space, which was used through the whole gameplay.
The workflow of the puppets consisted of building the 3d mesh with textures and rigging. After having the initial puppet mesh, a low-poly model bounding shape is created. The rigid body of the crossbar and the puppet were connected with soft body ropes. The physics constraints, the mass of body parts, and other parameters were set up in Blender Game Engine, which enabled us to fine tune the simulation and the behaviour of the 3d model.
Built with Cinder using Bullet, and our own Cinder blocks including Cinder-Assimp, Cinder-Leap, Cinder-libpd, Cinder-portaudio.
Marionette Zoo can be downloaded for free from the Leap Motion Airspace Store https://airspace.leapmotion.com/apps/marionette-zoo/