KinectPortal

Kinect Portal from Ward Penney on Vimeo.

Kinect Portal is an interactive installation that allows users to see into an invisible world. Developed in C++ using OpenFrameworks, it enables the participant walks up to a projector and XBox Kinect facing them, but no light is projecting. They then pick up a 2 x 1 foot opaque acrylic panel and hold it up to the projector. The projector then comes to life and displays an image fitted to the corners of the user’s acrylic panel. The user can then rotate the panel freely and watch the corresponding rectangle follow and fit to the panel, providing the experience of holding a custom display.

Concept

For my final project for Interactive Art and Computational Design, I wanted to build something that mixed reality with unreality. On the Comic Kinect earlier this year, I developed an interaction utilizing the OpenNI skeleton via ofxOpenNI with Xbox Kinect. For my final, I wanted to use the raw depth reading form the Kinect.

I drew up a few concept sketches to explore the idea.

Concept sketch for a Kinect rock climbing wall

Concept sketch for Kinect Tetris

Concept sketch for a Kinect "The Wall" game

After talking with our professor, Golan Levin, he helped me decide on an interactive installation that would use the depth values from the Kinect. The idea was to have one or more users holding up opaque acrylic panels while facing a projector and a Kinect. The Kinect would sense the acrylic rectangles, and display a personal image for each user. The display would be decided later, but was intended to utilize the tilting of the rectangles in X, Y and Z spaces. I made this quick sketch to illustrate two users holding panels and gaining different viewpoints of an “invisible” object in 3D.

KinectPortal sketch with two users. Lower is "light" beam users would reflect as a team towards a target.

Data Recording / Playback

In order to make development easier, I wanted to get the recording and playback functionaltiy working. With Comic Kinect, I was using ofxOpenNI and the recording did not work there. Because it took longer to compile, longer to load, longer to get a skeleton, and we needed two users, my test cycle for ComicKinect was almost 5 minutes! That means I really only got about 30 tests with it. By just using the depth pixels with ofxKinect, and the recording / playback, I can now test very quickly (~15 sec).

Depth Histogram to Discover Object Blobs with OpenCV Depth Image Thresholding

Next, using ofxOpenCV Thresholding, I went for getting the depth threshold to target just behind the square plane. First, I had to generate a histogram of the depth map. Then, I did some 3 point simple derivative math to figure out when the slope was increasing, decreasing and in a trough. I had to make sure that it only recognized reasonably-sized peaks in the histogram, to avoid noise from the environment. This took some tweaking:

• smoothing the histogram
• looking for edges larger than some tested amount of values from the histo
• averaging the threshold over a few frames to avoid “jerking”

Control Panel Testing

KinectPortal Process screenshot with OpenCV depth thresholding. Without ofxControlPanel.

I wanted to have more control over my variables, because I was about to start implementing and testing various algorithms to discover the corners of the rectangle. So, I decided to implement Theo‘s ofxControlPanel. The result is a display containing the RGP, Depth, Thresholded Depth pixels, and controls for selecting the rectangle discovery type and smoothing run counts.

The Acrylic Panels

KinectPortal Acrylic Panels with Architect's Vellum glued to one side.

I laser cut two 2 x 1 foot acrylic panels with rounded edges to be comfortable for the user. I then attached to bathroom cabinet handles from Home Depot to the back of the panels, and coated the front side with architect’s vellum. The Vellum was transparents enough to let through a lot of brightness from the projector, but still catch enough light to render a sharp image on the other side.

Kinect Portal Demonstration

For our final demonstration day, we set up at Golan’s office in the STUDIO for Creative Inquiry. My project was stituated with about 5 feet in a semi-circle in front of the projector and Kinect. Users walked by and picked up the acrylic panels and, with a little explanation from me, held them to face the setup with the face of the vellum perpendicular. Using a quad warper in the ofxAppUtils toolkit developed by Dan Wilcox, Kinect Portal then displays an image texture onto the corners of the rectangle discerned from the Kinect sensor.

Reactions

Many people expressed how the project was uniquely interesting because if offered something a lot of people had not seen or touched before. Some executives from Bank of America commented on how it was a really novel interaction and could be used for augmented reality.

Special Thanks

• OpenFrameworks
• Golan Levin
• Dan Wilcox – ofxAppUtils
• STUDIO for Creative Inquiry
• Theo
  • ofxControlPanel
  • Roxulu, and Dan Wilcox ofxKinect

Source Code

Here is a link to the source code ZIP on my DropBox. KinectPortal Source

Next Steps

The following is a list of issues I am working on fixing, with the help of Golan and Dan.

• The rectangle is jittery. I think this is because my contour smoothing of the blob counter from OpenCV is not working properly.
• The quad warper does not take into account the Z-depth from the Kinect. I want to do that to allow the user to look “around” the scene.
• I want to make this multi-user. For each depth threshold registered, I want to look for valid rectangles and throw out anything that is not a known panel.
• The graphic rendered is a simple image put onto a texture. I want to make this a view into a 3D world, or some other interaction.

Thank you for reading! Email me if you have any questions.

Ward