Abstraction: Browser and iPhone Interaction Experiment

Background: There are many interactions with sensors like Kinect, LeapMotion, Wii. All these interactions require additional sensors with limitation of context and cost. They are either scalable but with high cost or with cheap device but limited with size. Therefore, an idea using ubiquitous devices like a smart phone or iPad as sensors is on the right track. Using browser as the interaction medium also frees the system from heterogeneousness between different mobile operating systems.

Inspired by Boundary Functions (1998) by Scott Snibbe, every person in real world has a physical relationship with others, I am thinking how about relation between people in virtual space. This interaction installation will be set in public spaces, like airports and train stations. People are getting into Internet through devices. Identity of human is represented as identity of digital devices. With interactions between their mobile devices, people are getting sense of relations between others(whether anonymous or not) in the same public spaces.

Verification & Communication

Currently I didn’t implement the verification for the system right now. User can access the system by visit the server ip address with websocket. Once a device is connected to server, the server will generate a new ball for the user. Server can get the device id to identify the user.

Interaction

The browser on the mobile devices will obtain gravity and accelerometer of the current device and send data to the server. The server will normalize the velocity and orientation of x-axis and y-axis values and map them to the position on the browser page.

Tech

I use Node.js as server with now.js and socket.io for live interaction. Apple accelerometer is the sensor on the client device.

Github repo: https://github.com/hhua/xBall

I am interested in using the kinect to manipulate 3d meshes or surfaces in a program like rhinoceros or blender.  There are plenty of motion capture projects out there, but I am more interested in taking hand gestures and mapping them to various 3d modeling commands.  The leap motion might be much more suited to this, but because I have never done anything involving hand tracking, perhaps a good first step would be using the kinect.

I think it would be interesting to track fingertips in 3d space and connect them with a curve in the modeling environment.  As the hand moved around it would sweep out a surface.  I want to computationally alter that surface according to the velocity or even acceleration of the points.  The surface could become more convoluted, spikey or maybe perforated. A sculpture made from this 3d model would therefore contain not only visual information about the path of the fingertips, but also about the motion (velocity and accel.)

kind of like this (but hopefully smoother, and computationally altered):

Of course I also just want to continue working on the truss genetic algorithm.

I have a couple of random ideas for this project. Not sure which one I should do yet.

Face-away

This idea doesn’t really have a purpose. It’s more experimental and artsy I guess. Imagine a panel sticking out of the wall that can rotate on the x- and y-axis. The panel reacts to a person and will rotate away from the person’s head, facing away from the user. For example, if the user goes to the right, the panel will rotate to the left on the y-axis. The panel rotates the other way around if the user goes to the left. Same inverse movement will occur when the user tries to look at the panel from above or below.

**sketch image coming very very soon**

Possible implementation:
– I could either use a webcam or Kinect to track the user. I think the most important part is the ability to track the location of a person’s head. If I use the Kinect, I should be able to get the head position from the skeleton (I’ve never worked with the Kinect). I could also use blob detection with the webcam from a top-down view or something, but I don’t think it would be accurate enough. Perhaps a better and easier method would be to use FaceOSC to track the head. I would have to place the camera in such a way that I would be able to see and capture the face from all angles.
– For rotating the panel I could use 2 servo or stepper motors; 1 for each axis. These shouldn’t be hard to implement.

???
For now there’s nothing to see on the panel. Could be just a plain piece of material, wood, acrylic or something. But I’m not sure if that’s interesting enough or if I should come up with something to display on the panel.

How often?

We often hear statistics about how frequently certain events occur.  One child dies from hunger every five seconds.  Someone buys an iPad every 1.5 seconds.  Someone dies from poor indoor air quality every 15 seconds.  A baby is born every quarter second.  These numbers only let us understand these phenomena on a very cerebral level, though.  Even well-designed infographics only engage the user visually.  I would like to make an installation that cycles through a database of these statistics and allows the user to experience each through a combination of touch, light, or sound.  For example, a light could blink with a period of 1.5 seconds to indicate the frantic pace at which the world is buying up iPads while a gentle burst of compressed air to the back of the hand every five seconds reminds the user how often the world lets a child starve to death.  Approximately five children starved in the time it took to read this paragraph.

QR Code Infobombs

People love to scan QR codes, even if they don’t know what they lead to.  I might like to pepper sidewalks with QR codes made with chalk and stencils that lead to a website that presents highly localized and continuously-updated information on smog and air pollution.  If people scan them while walking along a busy road, I hope I can make the presentation compelling enough to make the link between air quality and traffic stick in their minds.

Secret Keeper

I imagine a tiny black cube with a phone number and instructions on the side, to be placed on a pedestal in some public location.  If you text it a secret (and the text checks out in terms of length and variability to weed out messages like “butts butts butts”), it will store it and reply with an anonymized secret that it has heard before and is most similar to yours.  Each secret gets sent to only one other person after a suitable number have accumulated, so you know that when you tell it a secret, only one other person will receive it.  In a sense, it’s a bit like Post Secret, except for the strange sensation that exactly one stranger will know something deeply personal about you. (Also, the cube may emit a faint red glow when it receives the secret, to indicate some link between the physical object and the process).

Interactivity Project ->

For this project I’m really hoping to make a game for Android tablets/phones that utilizes the touch screen. I’m not sure if that’s too ambitious for the time we’re given but I feel like it’s an area I’m going to need to explore eventually.

My current idea, which I think is definitely too big for this assignment, is to make a wave-based (think Tower Defense / Plants vs Zombies) game wherein you play as a goblin warlock’s apprentice, and your job is to go clear out an old fort that’s infested with humans. Levels would be different rooms, and the waves would consist of different types of people (knights, knaves, whatever). As a warlock apprentice, you know some spells that you can cast onto the oncoming waves by drawing different symbols.

I have some other less time-consuming ideas that I might fall back on in the event I can’t get the barebones version of this running by the due date.

I have a couple of ideas that I am trying to decide between for my interactivity project.  I am interested in doing something that is both screen and touch based using either a phone/tablet, Sifteo, the AR toolkit, or Reactivision.  I’m not really sure what I would do with these later two tools, as I was just introduced to them in class on Monday but I’m keeping them in mind.

My first idea is to continue the Sifteo project that Caroline, Bueno and I worked on for project 1.  I think that we had a really neat idea and I would like to find a way to optimize the clock to alleviate the memory issues we were having as well as create an interface that would allow users to design their own “puzzles” for turning off the alarm clock.

Another idea I have to to use As-Rigid-As-Possible Shape Manipulation (which makes it possible to manipulate and deform 2d shapes without using a skeleton) to create a tool for real-time, interactive story-telling.  I plan to implement this algorithm in C++ for my final project for Technical Animation, and I thought that I could extend upon this to let users draw the characters to be manipulated on a tablet, then, by connecting to a monitor or a projector, tell stories by manipulating the characters.  I see two possible applications of this: 1) as a story-telling tool to create a sort of digital puppetry, and 2) as more of an interactive exhibit where visitors could add to the story by either creating new characters or manipulating the characters that are already there.  I’d also be interested to hear other suggestions of applications of this.

I’m also really interested in the idea of educational software.  For my BCSA capstone project I’m working on an educational game and I really appreciated the iPad app we saw Monday that counts your fingers.  I would like to maybe apply the Shape Manipulation I discussed above to an educational context but I don’t have one definitely in mind yet, so I’d also like to hear ideas of such applications or ideas of interactive educational software in general.

For my third (and possibly fourth) project I’d like to create a room with an overhead tracking camera and and front facing recording camera. Users will walk into the room and will receive a series of commands. User actions and responses will be monitored as commands become increasingly antagonistic and or incoherent. Video and photographic recordings will be made of all participants and streamed on the web.