Marionette Zoo | 2014

Leap Motion

In early 2013, we were accepted into the Leap Motion Developer Program, allowing us to prototype applications and gain insight into this unique new way to interact with computers.

The Leap Motion controller is a small USB peripheral device which is designed to be placed in front of a computer screen, facing upwards. Using two infrared cameras and three infrared LEDs to capture depth information, the device observes a roughly hemispherical area, up to a distance of about one meter.

The device streams frames of data (with a maximum of almost 300 frames per second) to the Leap Motion controller software, where it is analyzed and forwarded to client applications. This data, accessible to applications, mainly consists of hand, fingertip and pointer tool orientation and position information, but it does not provide depth image or point cloud data.

Using these capabilities of the Leap Motion to track hands, fingers and pointer tools in 3D space in a very fast and accurate way, we decided to build an experimental application. This had not been possible before, because of the lack of suitable input sensor. We came up with the idea of a puppeteering application, which allows you to control marionette puppets with hand gestures.

Artwork and animations

We started by making a number of puppet sketches. We aimed to make everything as simple as possible, so we chose the bird as our first puppet figure to experiment with, due to its straightforward skeleton structure compared to other animals.

Then, the backdrop, a forest with hills, was designed. It was cut into separate layers in a similar way as scenes are prepared in the original puppet theatres. These layers were placed one after the other, in a parallel way, with some space between each. The puppets move in this inbetween area.

When we had the basic idea of the puppets and background scenes, we came up with visual gags, small stories we envisioned to be played in the application. One of them was turned into a short animation.

Moreover, we came up with visual gags, short narratives to be played in the application. One of these was turned into a short animation.

Puppets and physics

The 3D world was made in Blender. We have already used Blender in our previous projects, so modelling, texturing and rigging was fairly straightforward. For the physics simulation, we chose to work with the Bullet Physics library. Since Blender has also got Bullet included, we intended to plan the physics there as well, in addition to the usual modelling tasks.

In terms of creating the stages, the images drawn by the graphic designer were separated to layers, vectorized and imported to Blender from svg format. The layers were, then, UV-mapped and extruded to reach their final look.

All scene layers were arranged to face a fixed camera in space, which is used through the whole gameplay.

The workflow of the puppets consisted of building the 3D mesh with textures and rigging. Taking the initial puppet mesh, a low-poly model bounding shape is created. The rigid body of the crossbar and the puppet were connected with soft body ropes. The physics constraints, the mass of body parts, and other parameters were set up in Blender Game Engine, which enabled us to fine-tune the simulation and the behaviour of the 3D model.

Built with Cinder using Bullet, and our own Cinder blocks including Cinder-Assimp, Cinder-Leap, Cinder-libpd, Cinder-portaudio.