The Technical Magic

Technically, this project was lots of fun and frustration, all in the same sentence. The kinect depth map is not a very stable in general. Blob detection tends to be unstable across frames as a result.

The hand detection algorithm is overtly complex but it works like this:

1. For every depth of 15 values or so, it thresholds front and back.
2. For the thresholded image, it uses the openCV contour finder to look for blobs within a certain range.
3. Cull if the blobs are too high or low in the frame.
4. Calculate the ratio between blob and boundingRect.
5. Calculate the distances between the centroid and the perimeter. Cull if there are no more than 4 peaks in the difference in distances. The idea is to start the recognition by looking at an open hand.
6. Iterate through current row of hands. If there is already an existing hand in the same location, assume they are the same and merge them together.
7. If not, add a new hand.
8. There is also a lifetime function to calculate basically a confidence level for the blob. Because the kinect drop frames, it is essential to actually “remember” that there was a hand in the same location.

The closed hand detection uses the above, but looks if the smaller blob overlaps the existing larger area. As we can observe in the video, it is fairly good but not entirely robust. There are a few techniques to improve detection going forward.

1. Use real dimensions to calculate length. We can estimate hand size by using real measurements.
2. Use bounding polygon versus blob instead of bounding rect. This way we can account for the “holes” in the hand and get a more accurate result.
3. Remember the size, depth and location and guess intelligently.

The rest of the sketch uses box2d and various other enhancements such as a gradient drawer that takes in any arbitrary RGB color on each end. Another challenge that we had in the project was to deal with C++ code. C++ is en entirely too verbose language compared even to Java, which says alot. Another bug that I encountered was that the vector class actually uses the copy constructor on the push_back function. This actually led to 2 different copies of the same object at one point in the program. In short, pointers is such a big hairy mess. However, this was solved in the version demoed, hence the animations.

All in all, pretty fun! I’ll continue in working on the robustness of the hand detection and hopefully able to get some hand gestures going on.

Process + Back-story

At first, we were planning to generate the art using code. The first two prototypes were done in Flash and Processing, to get the feel for the physics of the floating and collisions. However, when porting to C++ it became evident that our project’s scope was somewhat large. We cut trying to generate images and having implementing a system that animated the pngs.

As for the story, we wanted to play with the idea of the “Red String of Fate.” Ran out of the time, but ideally, the hands would be tied together with red string and the players would not be able to float unless all participants were clenching their hands (grabbing balloons). There was a lot of thought for all the cases. Here’s a snippet:

2 Hands:
Co-op
Rise: can only rise when both hands are closed
Fall: when EITHER hand is open

1 open, 1 closed: balloon hand goes to top of screen, but balloon bursts due to lack of co-op

3 Hands:
Co-op with “weighted” areas on string
Rise: all 3
Fall: any 1 hand
1 open, 2 closed: can’t even float up
2 open, 1 closed: balloon hand goes to top of screen, but balloon bursts due to lack of co-op