creatyde-mocap

Here is a video of the work in action:

First, a load of credits: thabir, ango, zapport, farcar, tesh, avatar, ookey, Golan, Aman, my good friend Maia, Cascada for the music, and everyone in the STUDIO.

My project is a real-time motion capture that plays sound based on the movements of two individuals. In general, as two people get closer together, the volume grows. The movements of their hands also affects the frequency of the noise. Cascada’s “Everytime We Touch” plays in the background, for added atmosphere.

I took input from the Kinect2 using OSCeleton.  With some Processing code (shown below), I was able to stream the skeleton data using OSC, and read the data live in MaxMSP (diagram shown below), where I manipulated sound input.

This week was very challenging.  It took a long time for me to get the pipeline of everything working.  I greatly appreciate all the people who helped me, because they took time off that they could have spent on their own work or project. 
Thank you, thank you, thank you. 

This project was mostly an exercise in engineering/hacking/intense-Googling, and a bunch of things I hoped to gain more experience in.  I definitely learned a lot, and am happy with the work not so much for itself but because of what it took to make it.

In terms of the work itself, there is much that can be improved.  I implemented a simple averaging function for the hand’s effect on the frequency.  I could have designed a more sophisticated or interesting function, with more experimentation.  It would be neat if the song could be better mixed into the sound, and if the cyclic tone generator had a more appealing quality.  Sound is seriously very fascinating, and MaxMSP is really cool: it gives a good reason for the existence of GUI programming.

Processing Code (that sends out OSC skeleton packages from OSC), taken from here:

import oscP5.*;
import netP5.*;
 
OscP5 oscP5;
NetAddress myRemoteLocation;
 
void setup() {
  size(400,400);
  myRemoteLocation = new NetAddress("X.X.X.X",8000); // to
}
 
/* incoming osc message are forwarded to the oscEvent method. */
void oscEvent(OscMessage theOscMessage) {
  /* print the address pattern and the typetag of the received OscMessage */
  print("### received an osc message.");
  print(" addrpattern: "+theOscMessage.addrPattern());
  println(" typetag: "+theOscMessage.typetag());
  oscP5.send(theOscMessage, myRemoteLocation);
}

Max MSP Visual Code: