This project is a real-time interactive game with Kinect boding tracking. It takes the form of an ancient Chinese football game. Players can control the Shadow-play puppets through movement and posture. It tries to digitize the traditional culture and relive it through fun interaction. The exhibited game is a 2-player system, in which 2 players are captured by 1 Kinect.
Interaction designer and developer
Background research, interaction design, game design, prototyping, C++ development, openFrameworks application development
The traditional shadow-play game is played by controlling sticks attached to the puppets. I tried to make it more fun by controlling with full body movement. The controlling is through mapping 3 key points: neck, left hand, right hand.
Below is an example of real-time capturing and tracking of body and limbs. With openFrameworks, it is easy to extract the key points with OpenCV related libraries.
I tried to add another shadow layer in the visuals. A simple approach is to draw a layer of semi-transparent body parts with the position info from the previous frame in animated simulation. With openFrameworks, it is easy to draw anything with OpenGL and simulate puppet movement with Box2D.
After the project, I reflected on the whole process and found several key points to keep in mind:
Kinect is only a 3D camera: I was trying out Kinect with a MacBook, then I realized no official body tracking SDK was provided. All information I could get from the 3D camera is a real-time 2D color image, plus a real-time greyscale image representing the depth.
Heavy computation in each timestep will cause jagged animation: It is easier to find a light-weighted physics engine to do the calculation and render simple images.