Since I have my performance at Bakery Square coming up at the end of March, I will merge/mod my game event actions sending out OSC (generative sound) to a graphic score for a percussion ensemble to play live. These are my bookmarks for references.

my first quick test in OF

references



guitar hero is a natural graphical score with gaming


http://blog.wfmu.org/freeform/2007/01/gallery_of_musi.html

Neurospasta (Greek word usually translated as “puppets”, which means “string-pulling”, from nervus, meaning either sinew, tendon, muscle, string, or wire, and span, to pull), is a game without any defined objective or goal, but rather a platform for experimentation and play.

Neurospasta is am interactive two-player game in which the player’s physical movements are mapped to a 2D avatar. The game’s graphic interface consists of a set of manipulation functions which enable players to interact with their avatar as well as the other player’s.

In terms of software, we used Open NI for the skeletal tracking and OpenFrameworks for the UI and texture mapping.

*For future consideration, Neurospasta would include a larger selection of manipulation functions and in an ideal environment would be able to capture and generate avatar textures for accommodating new player identities.

The Technical Magic

Technically, this project was lots of fun and frustration, all in the same sentence. The kinect depth map is not a very stable in general. Blob detection tends to be unstable across frames as a result.

The hand detection algorithm is overtly complex but it works like this:

1. For every depth of 15 values or so, it thresholds front and back.
2. For the thresholded image, it uses the openCV contour finder to look for blobs within a certain range.
3. Cull if the blobs are too high or low in the frame.
4. Calculate the ratio between blob and boundingRect.
5. Calculate the distances between the centroid and the perimeter. Cull if there are no more than 4 peaks in the difference in distances. The idea is to start the recognition by looking at an open hand.
6. Iterate through current row of hands. If there is already an existing hand in the same location, assume they are the same and merge them together.
7. If not, add a new hand.
8. There is also a lifetime function to calculate basically a confidence level for the blob. Because the kinect drop frames, it is essential to actually “remember” that there was a hand in the same location.

The closed hand detection uses the above, but looks if the smaller blob overlaps the existing larger area. As we can observe in the video, it is fairly good but not entirely robust. There are a few techniques to improve detection going forward.

1. Use real dimensions to calculate length. We can estimate hand size by using real measurements.
2. Use bounding polygon versus blob instead of bounding rect. This way we can account for the “holes” in the hand and get a more accurate result.
3. Remember the size, depth and location and guess intelligently.

The rest of the sketch uses box2d and various other enhancements such as a gradient drawer that takes in any arbitrary RGB color on each end. Another challenge that we had in the project was to deal with C++ code. C++ is en entirely too verbose language compared even to Java, which says alot. Another bug that I encountered was that the vector class actually uses the copy constructor on the push_back function. This actually led to 2 different copies of the same object at one point in the program. In short, pointers is such a big hairy mess. However, this was solved in the version demoed, hence the animations.

All in all, pretty fun! I’ll continue in working on the robustness of the hand detection and hopefully able to get some hand gestures going on.

Process + Back-story

At first, we were planning to generate the art using code. The first two prototypes were done in Flash and Processing, to get the feel for the physics of the floating and collisions. However, when porting to C++ it became evident that our project’s scope was somewhat large. We cut trying to generate images and having implementing a system that animated the pngs.

As for the story, we wanted to play with the idea of the “Red String of Fate.” Ran out of the time, but ideally, the hands would be tied together with red string and the players would not be able to float unless all participants were clenching their hands (grabbing balloons). There was a lot of thought for all the cases. Here’s a snippet:

2 Hands:
Co-op
Rise: can only rise when both hands are closed
Fall: when EITHER hand is open

1 open, 1 closed: balloon hand goes to top of screen, but balloon bursts due to lack of co-op

3 Hands:
Co-op with “weighted” areas on string
Rise: all 3
Fall: any 1 hand
1 open, 2 closed: can’t even float up
2 open, 1 closed: balloon hand goes to top of screen, but balloon bursts due to lack of co-op

roboScan is a 3D modeler + scanner that uses a Kinect mounted on a ABB4400 robot arm. Motion planning and RAPID code are produced in Robot Studio and Robot Master. This code sends movement commands and positions to the robot as well as the 3D position of the camera. C++ and openFrameworks are used to plot the depth data of Kinect in digital 3D space to produce an accurate model of the environment.

This work was done by Shawn Sims and Karl Willis as the first project of the research group Interactive Robotic Fabrication. The facilities of the Digital Fabrication Lab in the School of Architecture at Carnegie Mellon were used in the making of this project.

There were many technical hurdles along the way. We needed to “teach” the robot the coordinates of the Kinect IR camera in order to accurately move the perspective through 3D space. We also attempted to calibrate the Kinect. We successful did this but it is in fact hard to tell if it made a difference. All-in-all we learned a lot about the robot and about the difficulties of moving a camera around and still trying to extract continuous and accurate data.

Many thanks to dFab, Zach Ali, and Jeremy Ficca

Developed using Cinder + OpenNI + NITE +Xbox Kinect

After seeing what the Kinect was capable of, we were interested in using it to generate a particle system that would respond to user movement and depth placement. By pulling out the depth map, we were able to isolate the user to an extremely high level of accuracy.

The particles flock to the silhouette of the user when he or she is still and exhibits flocking/swarming behavior otherwise, creating mesmerizing shapes that ebb and flow between the recognizable and unknown.

Creation Process

Each point  that comprises my body pulled from the kinect depth map has a gravitational pull on one particle in the simulation. The strength of this force is inversely related to how fast I’m moving, so when I stand perfectly still the particle zooms to the point it corresponds to, and when I move its free to wander and behave on its natural flocking tendency. Thus you get these really compelling visuals as my silhouette breaks and reforms depending on how fast I’m moving.

Ray set out getting the kinect to actually work with Windows and pull the depth map/user data out from it using the OpenNI library and NITE drivers. Almost all of the current kinect support is for Macs, so this was no small feat! We attempted our project in Processing and Openframeworks, but finally settled on Cinder for our developing environment.

early concept sketch

Here’s a short clip of some process sketches I did in Processing in order to explore the flocking behavior before porting it over with some more hard hitting graphics to Cinder. And countless thanks to Robert Hodgin for his fantastic “Hello Cinder” tutorial

more from Alex: alexwolfe.blogspot.com

more from Ray:

playing around testing the kinect @ brillobox pittsburgh with Keeb$, Cutups, and Freddy Todd (detroit)
stackinpaper.com/​

Openframeworks.cc
tuio.org/​
osculator.net/​
OF, FFT OSC, MSA 3D Shape, Kinect, TUIO

Process:

I wanted to create an interaction that was  physically reactive — given that the kinect requires you to engage physically, it made sense to me to bring the interaction off of the screen and into a real space. Golan pointed me towards Marie Sester’s Access project, which does exactly that, engage an inbetween space with a spotlight that automatically follows people.

There were lots of techincal challenge in this project for me — getting the right drivers, learning about working with hardware and sending information over serial, dmx protocol, as well as working with the kinect, (very, very) basic image processing, blob tracking.

I approached the project from two ends, first, working with a strobe light (which has only 4 channels) and blob tracking separately, and then finally meeting in the middle so that the blob position controlled the light position of a moving light.

In the end math got in the way a little bit, instead of calculating the position of where the lights should pan and tilt to using trig, I created a grid over the kinect data, and plotted the light movement to points, which was conceptually easier for me to understand. It works, but there’s some choppy-ness.

Initial graphics for integrating with networked kinect skeletons.

Neurospasta

a Kinect based 2-player interaction game

CONCEPT

For our main concept we will be developing an experimental game in which two players have the ability to manipulate their own avatar as well as the other player’s avatar. The players will have the choice to interact, fight with their avatar against one another, or just simply explore different ways of manipulating the appearance and/or ability of their avatar.
The player’s avatar will consist of a rag-doll/skeletal model which will mimic the actions of the player via kinect video/depth capture.

The graphic interface will provide a menu for selecting a specific body part and a manipulation function menu for changing certain characteristics or abilities of the selected part.

THE GAME

After trying really hard for a week, we implemented Neurospasta (Greek word usually translated as “puppets”, which means “string-pulling”, from nervus, meaning either sinew, tendon, muscle, string, or wire, and span, to pull), a game without any defined objective or goal, but rather a platform for experimentation and play.

Neurospasta is am interactive two-player game in which the player’s physical movements are mapped to a 2D avatar. The game’s graphic interface consists of a set of manipulation functions which enable players to interact with their avatar as well as the other player’s.
In terms of software, we used Open NI for the skeletal tracking and OpenFrameworks for the UI and texture mapping.

FUTURE PLAN

*For future consideration, Neurospasta would include a larger selection of manipulation functions and in an ideal environment would be able to capture and generate avatar textures for accommodating new player identities.

Mix & Match Interactive Toy

Mix and match toys and games are a popular activity for children. It typically appears as an upright cylinder or rectangular prism with multiple standing human or animal figures independently painted or carved into the surface. The shape is divided into thirds that rotate independently, allowing the different heads, torsos, and legs of the painted or carved figures to align. They have existed in many forms of media and targeted different age groups over their two-hundred year history. Changeable Gentlemen, a card game published around 1810 targeting adults, was one of the earliest manufactured mix and match games. Adaptations in base material and figure subjects made it more suitable for a younger audience, and modern incarnations are generally as a classic toy for children.

I chose a mix and match toy as the interaction subject because it’s a ubiquitous toy and gives almost all the players a baseline familiarity. Also, while the media and figures are numerous over the last two hundred years, the method of interaction has always been hand manipulation. The Kinect enables the player to use their entire body to control the figure. The figures are from a card game “Mixies” published in 1956 by Ed-U-Cards. The player can rotate through different heads, torsos, and feet by swiping above, at, or below the player’s midsection, respectively. Swiping to the left will rotate to the next card to appear, and swiping to the right will rotate to the previous card. The figure will follow a player’s horizontal and vertical movement, bending, and if they turn around.

I used OpenFrameworks, ofxKinect, and OpenNI. Once the skeleton is detected, the images are overlayed at the position and angle of the corresponding section. Hands and arms do not directly affect card placement, so they are free to control the card rotation. Sweep/wave detection is simply a matter of the end of the left or right arm moving over a certain horizontal distance within a time threshold. If the user’s back is turned, the back of the card is displayed instead of the obverse.

Magrathea uses the kinect camera to dynamically generate a landscape out of any structure or object. The kinect takes an depth reading of what’s built on the table in front of it, which is then rendered live onscreen as terrain using openFrameworks and OpenGL.

The depth image is used as a heightmap for the terrain. A polygon mesh gradually morphs to match the heightmap, creating a nice rise and fall behavior. Textures are dynamically applied based on the height and slope of the mseh. For example, steep slopes are given a rocky texture, and flatter areas a grassy one. As the user builds and removes, the landscape correspondingly grows and sinks out of the ocean, shifting into a new configuration.

Landscapes can be made from anything, such as blocks, boxes, the human body, and even a giant mound of dough.

We both learned OpenGL and openFrameworks for this project.

If we were to continue this project further, we’d do so by adding more textures with more complex conditions, learning shaders and improving the graphics, populating the world with flora and fauna based on certain conditions, and possibly allowing for color-coded objects that could be recognized and rendered as specific features, say, a statue, or giant Yggdrasil like tree.

What if you could use hand gestures to control an audio visualization? Instead of relying on audio metrics like frequency and volume, you could base the visualization on the user’s interpretation of perceivable audio qualities. The end result would be a better reflection of the way that people feel about music.

To investigate this, I wrote an OpenFrameworks application that uses depth data from the Kinect to identify hands in a scene. The information about the users’ hands – position, velocity, heading, and size – is used to create an interactive visualization with long-exposure motion trails and particle effects.

There were a number of challenges in this project. I started with Processing, but it was too slow to extract hands and render the point sprite effects I wanted. I switched to OpenFrameworks and started using OpenNI to extract a skeleton from the Kinect depth image. OpenNI worked well and extracted a full skeleton with wrists that could be tracked, but it was difficult to test because the skeletal detection took nearly a minute every time the visualization was tested. It got frustrating pretty quickly, and I decided to do hand detection manually.

Detecting Hands in the Depth Image
I chose a relatively straightforward approach to finding hands in the depth image. I made three significant assumptions that made realtime detection possible:

1. The users body intersects the bottom of the frame
2. The user is the closest thing in the scene.
3. The users hands are extended (at least slightly) in front of their body

Assumption 1 is important because it allows for automatic depth thresholding. By assuming that the user intersects the bottom of the frame, we can scan the bottom row of depth pixels to determine the depth of the users body. The hand detection ignores anything further away than the user.

Assumptions 2 and 3 are important for the next step in the process. The application looks for local minima in the depth image and identifies the points nearest the camera. It then uses a breadth-first search algorithm to repeatedly expand the blob to neighboring points and find the boundaries of hands. Each pixel is scored based on it’s depth and distance from the source. Pixels that are scored as part of one hand cannot be scored as part of another hand and this prevents near points in the same hand from generating multiple resulting blobs.

Interpreting Hands
Once pixels in the depth image have been identified as hands, a bounding box is created around each one. The bounding boxes are compared to those found in the previous frame and matched together, so that the user’s two hands are tracked separately.

Once each blob has been associated with the left or right hand, the algorithm determines the heading, velocity and acceleration of the hand. This information is averaged over multiple frames to eliminate noise.

Long-Exposure Motion Trails
The size and location of each hand are used to extend a motion trail from the user’s hand. The motion trail is stored in an array. Each point in the trail has an X and Y position, and a size. To render the motion trail, overlapping, alpha-blended point sprites are drawn along the entire length of the trail. A catmul-rom spline algorithm is used to interpolate between the points in the trail and create a smooth path. Though it might seem best to append a point to the motion trail every frame, this tends to cause noise. In the version below, a point is added to the trail every three frames. This increases the distance between the points in the trail and allows for more smoothing using catmul-rom interpolation.

Hand Centers
One of the early problems with the hand tracking code was the center of the blob bounding boxes were used as the input to the motion trails. When the user held up their forearm perpendicular to the camera, the entire length of their arm was recognized as a hand. To better determine where the center of the hand was, I wrote a midpoint finder based on iterative erosion of the blobs. This provided much more accurate hand centers for the motion trails.

Particle Effects
After the long-exposure motion trails were working properly, I decided that more engaging visuals were needed to create a compelling visualization. It seemed like particles would be a good solution because they could augment the feeling of motion created by the user’s gestures. Particles are created when the hand blobs are in motion, and more particles are created based on the hand velocity. The particles stream off the motion trail in the direction of motion, and curve slightly as they move away from the hand. They fade and disappear after a set number of frames.

Challenges and Obstacles
This is my first use of the open-source ofxKinect framework and OpenFrameworks. It was also my first attempt to do blob detection and blob midpoint finding, so I’m happy those worked out nicely. I investigated Processing and OpenNI but chose not to use them because of performance and debug time implications, respectively.

Live Demo
The video below shows the final visualization. It was generated in real-time from improv hand gestures I performed while listening to “Dare you to Move” by the Vitamin String Quartet.

temporary!

James will be posting the video at his post here.

We created a path mapping system which projects a graphic visualization of a user’s path on the ground behind and on top of them. The setup included a ceiling mounted projector, a 45 degree hanging mirror, two computers (Mac OS and Windows 7), and a Microsoft Kinect Sensor. We used the openframeworks libraries ofxOSC, ofxKinect, and ofxOpenCV to track users, communicate between machines and generate graphics.

Tasks
The code for interpreting video (and later Kinect) data, was a challenge for us because we wanted to include a beginning and end point as well as a unique ID for each path. We accomplished this using vectors in C++ which allowed for continuous creation and destruction of paths. This path data was sent to the graphics machine in packets including the ID, x position, y position, and state (begin, middle, end).

The graphics program took this information and created classes of paths. These paths interpret the direction of the user and create a varied color palette that refreshes upon the creation of each new path. In the wake of the user, circles expand and dissolve in subtle pattern. The memory of the path fades quickly and allows for more users to enter the space.

We also had to design and build a mirror mount to ‘fold’ our projection giving it a slightly larger footprint in our space. The CNC milled mount is made from oriented strand board (OSB) which notches together and has adjustable height and rotation. The mirrored acrylic, screwed and taped to the mount, was donated by Max Hawkins (Thanks Max!).

Challenges
We are both new to C++ and openframeworks so we took it upon ourselves to develop this project exclusively on that platform. There were problems in handling the amount of data that came in and was sent between machines. Getting smooth clean data from the openCV library proved to be a challenge. Also, using depth values from the Kinect remains unresolved.

Sorting through the paths and nodes within those paths was a challenge. We had to properly parse all incoming data to be added to the correct path and pathholder (place holder).

Kinect Tracer from eric brockmeyer on Vimeo.

Room For Improvement
We would like to utilize our start and end functions in some graphical manner. We would like to figure out why our depth data in the Kinect was so imprecise. We also would like to further debug our data transmission and improve our parsing data structures.