Category: FinalProposal

The Concept: Travels from the World Wide Web

I plan on completing my manufactory project, which is a chrome extension that allows you to generate physical, mailable postcards from your internet adventures. “We live on the internet, so why not share your travels?”

Early prototypes / ideas:

To do:

• add in direct mailers
• add in paypal so i don’t go b r o k e
• email receipts of transaction
• address book
• address verification
• post to twitter  / have a global homepage
• impose NSFW filter
• make a better documentation video

For my final project, I intend to complete my Rebus Chat app that I began for the telematic project (https://ems.andrew.cmu.edu/2019/jaqaur/04/10/jaqaur-telematic/). It will be basically the same as I described in that post, but I have changed part of the Rebus-ification pipeline and also solidified my plans for implementing the rest of the app.

The new Rebus-ification pipeline will not use Rita or The Noun Project‘s API (though I will still use icons from The Noun Project). I will use The CMU Pronouncing Dictionary to get IPA pronunciation for everything users type. I will also manually create a dictionary of 1000-2000 images, along with their pronunciation (these will be short words whose pronunciation is quite common). Then, I will try to match substrings of the users’s typed syllables with syllables I have in my dictionary. I will insert the images in place of the corresponding words.

In terms of other implementation details, I am using Android Studio along with Firebase to help with authentication and data storage (which I highly recommend). I have big dreams for this project; I hope I can finish it all in time!

Interactive Art Final Project Proposal

I propose to complete my watershed visualization project that I began but did not critique for Project 2. My project’s conceit is to generate a line in AR that visualizes the flow of water out of the given Pittsburgh watershed that you are standing in. It might look something like this:

As I’ve gotten feedback about this project, I’ve come to some important insights:

• Watersheds are interesting because they contain themselves: A watershed is an area in which all the water flows to one point. The Allegheny and the Monongahela watersheds are a part of the Ohio watershed, which is a part of the Mississippi. What areas are too small to be watersheds? Where does this cycle end?
• It’s important to connect this visualization to actionable information about protecting our watersheds. A raw visualization is interesting, but not enough to affect change.
• Some aspects of the visualization can be connected to the viewer so that it feels more actionable. For this reason, I think it’s important that the visualization is grounded in the way that water flows from the point the viewer is standing on, as opposed to defining the borders of the watershed more generally. In this way, the viewer might understand their impact on their environment more easily.

 

My technical accomplishments so far:

1. I have built an AR app to iPhone.
2. I have loaded the Mapbox SDK in Unity, and I can represent my current location on a zoomable, pan-able map.
3. I have loaded the Watershed JSON data from an API, and can represent its geometry on the Mapbox map.

 

What I still need to do:

1. Draw a line of best fit, representing the flow of water, through the watershed containing the viewer’s current location, and add some randomness to it so that it’s a squiggly line, but still within the real bounds of the watershed.
2. Draw a line that connects the current location of the viewer to this line of best fit through their watershed, and add some randomness to that.
3. Create an AR object pointed in the direction of that line.
4. Model that AR object so that it appears to be a line on the groundplane.
5. Model that AR object so that it extends in the direction of the line, and curves based on the curves of the line, at approximately the appropriate distance from the viewer.
6. Test so that the line can be followed and updates accordingly.

I am thinking of using p5.js, webGL and hopefully some other libraries to create shaders that give an abstract representation of places, times and color palettes that I associate with certain places and times in my life.

An important aspect of this project is going to be creating a representation of these places that feels true to me but is also relatable to other peoples’ experiences. I think that I will need to think a lot about sound, velocity and interaction in order to make this a satisfying experience for me as a creator as well as for the audience.

For my final project, I would like to create one or two interactive wall projections. I’ve always imagined my Box2D project displayed on a large wall where people could approach it and interact with the shapes by touching it, so I think expanding on that idea wouldn’t be too bad of an idea. I also just have a large affinity for simple geometries.

The first step would be to get the first Box2D project working as an interactive projection. Next, I would use that as a stepping stone to  create different ones. One thought that I had was a pulsing blob that people could pluck smaller blobs from to play with and merge with other blobs that are on the projection. Another that I thought of was a scene similar to the warp speed animations in Star Wars where particles would converge to a focus point (I can’t think of the correct term at the moment) and more focus points would be generated if more people approached it.

To achieve this I would use a standard projector for the animation and a kinect for body tracking since its implemented rather well in that hardware. I think that my main technical issue would be the calibration of the kinect to the projection on the wall.

I want to make a system that attempts to maximise some bodily response from a viewer.

This requires a parametric image and a way to measure bodily response in real-time.  Given the hardware available, the simplest options seem to be either using heartbeat data, or the Muse EEG headband.

The project works as follows: Modify the parametric image. Evaluate response.  Estimate gradient of emotional response w.r.t parametric parameters of image.  Take a step of gradient ascent in the direction in parameter space that maximises the estimated emotional response function using reinforcement learning or genetic algorithms. Repeat. An alternative route would be to train a neural network to predict emotional response, and optimize using its surrogate world-model gradient, which would enable using stochastic gradient descent to optimize the image much faster.

Given the slow response of heartbeat data, I should use the Muse headband. In addition, we know the approximate timeframe that a given visual signal takes to process in a brain, although it remains to be seen if the noisy data from the EEG headband can be optimized against.

This project parallels work done using biofeedback in therapy and meditation, although with the opposite goal. An example of a project attempting to do this is SOLAR (below), in which the VR environment is designed to guide (using biofeedback, presumable using a Muse-like sensor) the participant into meditation.

For the parametric image, there are a variety of options. Currently, I am leaning towards using either a large colour field, or a generative neural network to provide me with a differentiatable parametric output. It would be awesome to use bigGAN to generate complex imagery, but the simplicity of the colour field is also appealing.  A midway option would be to use something like a CPPN, a neural network architecture that produces interesting abstract patterns that can be optimized into recognizeable shapes.

 

Fall in love, go through rough patches, have it easy and wonderful, get baggage, meet other people. Don’t Let Me Down is a 2 player game about being each other’s foundations.

https://docs.google.com/document/d/1yWvWidi2act0fbU7m9EgSKfP_wI7VF_BAYSFSnvBO1E/edit?usp=sharing