SECRET BASE FINAL AR PROJECT

Description: “It’s a treehouse, but without the tree.”

IMAGE AND GIF BELOW:

 

Using distinctive images (posters, stickers, wall patterns) from real life, users can build “secret bases” that will pop up any time the app detects those images. These bases will be stored and can be sent to the users’ friends through a friend code.

These friends also have the ability to see and modify the same structure if they are within proximity of the same image.

All the furniture was textured with hand-painted watercolor textures, and the “sharing” aspect uses both Firebase authorization for user sign in and Firebase’s database to remember base information. The AR was made using vuforia’s cloud database and extended tracking.

For people who are interested in “”AR sharing”””, I’ve uploaded the relevant code to github here 🙂 —->>> https://github.com/khanniie/Secret_Base_Firebase

THE PROCESS:

Concept art:

Part One: Working on the Firebase auth and database

Above: Testing with Firebase’s quickstart apps! I spent a really long time trying to work out the annoyingly small errors, so it ended up taking me the whole week to figure it out. But, in case anyone else is interested, I’ve documented the errors and fixes here –> /conye/04/06/conye-artech/

Above: Database is working!

Above: User auth is working too!! 🙂

Basic demo: I had a small cube that I updated using a slider. The slider’s value info would be sent to firebase, which then sent the information back and told the app to change the size of the cube.

Part Two: Base Building

Above are thumbnails of my texture making process! I would watercolor them, take a picture of those watercolors, edit them in photoshop, and make a normal map for the texture if I thought it should be bumpy.

Above: Basic grid and block placing implementation. The block has my brick watercolor textures, which was super exciting for me because after I used photoshop to create a normal map for the texture, it looked sooooo good.

Above: Block building, but built into AR!

Part Three: Combining the Two! (and fixing my bad user interface)

This week, I spent a lot of time trying to get the firebase details correct. I was able to send and retrieve data, and authorize users, but I didn’t have a structured way of implementing my idea yet. What I ended up with looked like this:

In Firebase:

The above: is how one unit of the grid keeps its info. If there are multiple items stacked at that square, the items variable/key will store a single string with the furniture names concatenated together with a “-” character in between.

Also, in the last few hours on Thursday, I updated my hideous UI!

Before and after:

Example image targets that I used for my secret bases.

My biggest regret was that I wasn’t able to implement dynamically uploading image targets to Vuforia’s cloud database within the app, but after spending 8 hours debugging I had to give up and move on. I think I was almost there, but every time I just get “error: bad request”, so I think I’m encoding my access keys wrong or something. At least I learned a lot about how HTTP requests work.

Overall, this was a crazy hard and time consuming project for me, but I learned a lot about unity and firebase in the process. I’m really happy with how it turned out and am kind of happily surprised by how functional it is.

BIG THANK YOUS

to

Nitesh for meeting up with me to work, and for staying late on thursday to help me document!!! Also for being a good friend overall.

Rain for being the best 15-251 partner, without Rain I would have been doing proofs instead of this project. Also thank you to Rain for being a good friend in general!!!

Sophia C. for being a Very Good Buddy and for meeting up with me to do work :))

Golan for being a really great teacher!!! And for his advice on the project direction, he told me to add furniture instead of just blocks, and my project is decidedly much better looking because of it.

Claire for being so encouraging all the time, and for her advice on the watercolor textures!

Also: Yixin, Peter, Sophia Q. and everyone else for brightening up my days in Golan’s studio!! I’m always super inspired by you guys.

A mildly obnoxious study buddy.

Initial Sketch

Unity setup

Different states of the buddy: idle, happy, angry. Kinda hard to see in the video, but present in the app.

Idle

Happy

Angry

The original idea was the make a rude buddy that only wanted to talk to you when you didn’t want to talk to it. After hearing feedback for the first week of the project, I decided to keep the obnoxiousness but turn my buddy into a kind of productivity aid. To use it, you just load up the android app while you’re working and it’ll continuously spout vaguely useful advice at you. If you attempt to pick up the phone, potentially to procrastinate, it’ll ask you if you’ve finished your work and then respond in accordance with your reply.

I’m happy with the final look of the animations, although the timing of the animations besides idle could be lengthened in the app. I’m also just happy that I got most of the features working, since using Unity produced a number of hurdles that I never expected to be hurdles. For example, I first rigged and animated my modeled creature with various NURBS controllers, but then soon discovered that either Unity or the FBX exporter of Maya hated the controllers, and thus I had to remove all my controllers and re-animate everything while committing the sin of animating by directly moving joints. Some weaknesses of the project: variety of interaction is pretty limited, creature doesn’t interact much with the environment, a few small bugs in the speech remain.

This was a solo project, but I’d like to thank Miyehn who discussed the original idea with me, Zbeok for some Unity troubleshooting, all my classmates that gave helpful comments when we last critiqued these projects, and my housemate who washed my dishes for me while I pulled an all nighter.

Walk through these portals to reveal layers of other worlds that exist alongside ours.

The Process:

This project is the result of a long struggle I had learning to use Stencil Filters with AR in Unity. I originally started by creating portals that could hide specific objects, and moved on to learn how to edit shaders from unity assets and even the unity default shaders to allow objects to be hidden from sight unless you looked at them through these portals.

Once I figured out how to pass through the portals so that the objects would become visible once you walked through, I was able to start working on their organization and on what the portals would contain.

Originally I was working on layering the portals so that it would create a maze or dungeon game forcing the user to look for the next portal level, however it proved incredibly difficult for more than 2 portals, and also felt too restrictive for the user. I wanted to encourage a sense of exploration and discovery with these portals so I shifted to a method where the user could wander through several different kinds of portals and see what objects each one contained, and see how the different objects exist together.

I also implemented light estimation techniques in order to create virtual objects that would adjust their brightness to match that of the real world around them. This is done using a shader that can access the phone’s camera and read the average brightness value of a camera image in as an input to help adjust that of the in-game objects.

 

Resources:

• This invaluable tutorial on creating Portals for AR
• This tutorial on Light Estimation for ARCore
• Unity Documentation on Stencils and Depth Mask Shaders
• This tutorial for coding a simple particle shader
  

Special Thanks:

to aahdee, farcar, creatyde, sheep, and Aman for teaching me about shaders and the logic behind using multiple stencil shaders at once

to conye for helping with documentation and moral support

And of course to Golan, Claire, and the Studio without which this would not have been possible

 

Description: Ever wanted to summon Satan? Well, this is kind of the closest you can get on the market.

Video:

Screencap:

GIF:

Process shots: 

 

 

 

 

 

 

 

 

 

 

Concept art regarding the initial idea. I made a few digital sketches as well.

An in progress model shot, before animation but after rigging.

Vuforia screengrab of my debugging.

The initial setup I had to test my homography graphics setup.

About this Project:

Because I haven’t seen a technology that allows you to summon demons (yet), I’ve created that demon-raising tech to fill in that niche. It combines AR image target technology,  perspective correction, Google text recognition, and some 3D animation magic to bring Satan from hell. All you need to do is to draw a magic circle on the ground, put the proper symbols down, and call upon him!

This was a uniquely challenging project in the sense that the largest problem was finding the proper method of doing what was needed. I could have executed this project in many different ways, so choosing what was best for the project was hard. In addition, I had to quickly sift through new technology that I didn’t quite understand. For instance, importing 3D models from extraneous programs, or even implementing completely new libraries were topics that were pretty new to me in the context of Unity. I think it all went well, given the very loose nature of the project’s conception, but I’d like to be much more ambitious next time.

Acknowledgements:

Many thanks to the following people, who helped immensely in the creation and success of this project.

• conye, who helped me refine my ideas
• kerjos, who assisted me in my technical difficulties
• Prof. Golan Levin, who provided me with the resources I needed to make this work and also for teaching this wonderful class

CMU Stories reinterprets who we commemorate and how through the stories of Avrey, Leon, Latasha, Tiana, and Nikola, five incredible night-shift custodial staff at Carnegie Mellon University.

 

OVERVIEW

This piece aims to give image and voice to those that contribute to CMU in ways we often overlook by overlaying existing narratives of commemoration with the stories of night-shift custodial staff. I approached this piece thinking about the medium, augmented reality, and how that would translate into project. The seen vs. unseen relationship and the ability to give life to a scene or story gave way to thinking about who in my life is rarely seen, yet very present. Those that work the night shift, often 11pm to 6am, as custodial workers fit into this relationship, as they are often unseen, though incredibly important. In my project, I was interested in learning more from night-shift custodial workers, and not just about their jobs, but who they are. I was interested in stories. Thus, over the course of three days, ranging from 9:30pm to 2am, I circled CMU’s campus meeting and speaking with any working staff I came across. The five videos in this application reflect a small fraction of the more than thirty meetings I had. The hours I spent interacting with people are just as important, if not more important, than the final product. While I was doing this work, I was thinking about where these videos would exist within the framework of augmented reality. I decided on using existing memorializing plaques as images targets and then overlaying the videos of Avrey, Leon, Latasha, Tiana, and Nikola as a way of reinterpreting commemoration. The memorials and images of gratitude that are visible often honor money and power. Yet we don’t take time to honor, not even thank, the individuals that sustain our campus on a daily basis. I hope this app respects the night-shift custodial staff I worked with, and serves as a small piece of gratitude for the work that they do. By extracting the videos form their original contexts, I attempt to begin a conversation about who we should be thanking around our school.

STILL SHOTS

ANIMATED GIF

ACKNOWLEDGMENTS

First and foremost, thank you Avrey, Leon, Latasha, Tiana, and Nikola for being collaborators on this piece. It belongs to you, I was merely an organizer. Additionally, thank you to all the night-shift custodial staff that took time to speak with me over the course of this project. Of course, thank you Golan Levin and Claire Hentschker for guidance, and a wonderful course. Thank you to all my classmates, especially my “buddy” Tyvan, who supported me, not just on this project, but throughout the semester!

Avrey Taylor

Leon Murphy

Latasha Scruggs

Tiana Handesty

Nikola Mirkovic

 

Relax from all of those CMU Fine Arts crits in the Great Hall AR bathhouse.

 

---

 

Using Unity engine and the Vuforia library, I created a bathhouse that can be placed in the Great Hall in CMU’s CFA building. This idea was created from the fact that I felt that there weren’t really a lot of places to feel calm and relaxed. The shape of the model is rather simple, but it uses stencil buffers to create the effect of an in-ground pool. The buffers are applied to the pool itself and the water dropping into it. On the outside of the bathhouse, the reflection is created by an image based lighting technique. A panoramic image of the area was taken and then formatted to a cubemap. Then the shader takes the cubemap and reflects it based on the location of the cubemap relative to the location of the pixel and the normal.

The resources I used:

Image Based Lighting

For assembling the in-ground pool and this too

Also a huge thanks to Golan, Claire, Joe Doyle, and tesh for help on shaders

Word Search

This an app that takes words in real life and uses them to compose Dada-style poetry.

Here are some shots of the app:

Here’s a mostly silent video showing the app in action:

In this app, you go around finding words that you might like to include in your generative poems, which, in the Dada style, are random arrangements of words. You tap your phone screen to save these words. When you are ready to generate a poem using some of these words, you may tap the orange hammer icon and see what the app churns out for you.

This is a project that I have been working on for the last five weeks. It started out as a game in which you would be given a scrambled list of words from an unknown poem from the 19th century and, using your phone, have to go out into the world and find all the words in order to have the list unscrambled and the composition of the poem revealed.

For that project, for which there is a brief documentation video below, I implemented the Google Cloud Vision API, which I allowed me to both read the words captured by the app and get the coordinates of their bounding boxes in the captured image. I also made requests to a poetry API so that upon each load of the game, the player was given a new, random scrambled poem to discover.

 

I was using the images of the words to check off found words in the player’s inventory when I reached a point where I just wasn’t sure if completing the discoverable poem app was the right way to continue this project. At the same time, I had been pushing back against Golan’s suggestions that I do some generative poetry with my found words because I thought that whatever I could generate would make for some bad poetry.

This is an early process flow that I created. I cut some of the elements from this chart early on, including animating the words using OpenCV.

 

But then Golan introduced me to Allison Parish’s approach to conceptual poetry, which is to use generative poetry to explore the nonsensical, in the hope that those poem-writing robots return to us with something valuable. I think there is something worthwhile to making generative poems, even dead-simple ones that only arrange words randomly. And I was able to keep using the bounding boxes for the words, which was an added bonus.

In this last week, I tried to use Google’s ngrams datasets before settling on just a functional, totally random poem generator. Despite this, part of the charm of my app is that the poems it generates are not actually completely random; they are specific to the place and person of its user.

And here’s another page from my sketchbook; I typically work out my ideas with just a crazy amount of words, which I guess is fitting.

Acknowledgements

Thank you Golan Levin, sheep, phiaq, Coco, and my friends in the Studio for your help talking through this project, generating new ideas and new ways of looking at it, and helping me move forward when I felt like I was in a rut. Thank you especially to phiaq for helping me document this.

 

My final project visualizes a ghost-like, imaginary creature that lives in electric outlets and reacts when touched by users.

A screenshot

This was actually quite far from my original idea. At the beginning I brainstormed about fish and / or fireworks, as described in this post. After some time of jumping technical hurdles I got something like this ↓ where user uses the second image target (lighter) to approach the first one (firework) to light it and send it into the air.

However, the image targets were too obvious as image targets (don’t fit in the context well), and I couldn’t scale up the firework because 1) it doesn’t make sense to light huge fireworks indoors and not burn the ceiling, and 2) Vuforia’s image target tracking doesn’t always work perfectly, so when the user looks up, it’s very likely that the firework explodes at somewhere totally unexpected.

So I abandoned most of the physics rules about how a firework should look like (big thanks to everyone’s suggestions during our last critique). The “firework” no longer shoots from ground but from an electric outlet instead, in the form of some fish-like flocking creatures. After the “fish” explodes (which happens when the user uses the lighter to approach it), it doesn’t disappear right away- it splits itself into more “fish”, ready to be lit again by the user and explode.

Unfortunately I lost my screenshots at this stage, but this is how it looked like.

The scale problem remained- fireworks are supposed to be huge and loud and overwhelming, but I can’t make anything huge and loud and overwhelming because iPhone’s viewport is tiny and in order to view something big I need to keep turning the phone and leave the image target, and by doing so Vuforia messes up objects’ scales.

Although I still like fish and fireworks, thanks to Golan’s advice I looked into alternatives to fish and fireworks, but make more sense to be visualized in a small scale.

I even tried a string of stones that can explode. Abandoned that idea quickly, though, because it makes even less sense to have stones dropping out of outlets.

Eventually I went with a transparent, ghost-like creature, which avoids the generic quality of pretty much any primitive 3D model and adds mysteriousness to the creature (I think). The creature can still be seen, though, because wherever it passes by the background is slightly warped, as shown in the video.

The way it’s implemented is very close to the fishworks idea, but instead of having a trail of balls moving around, there’s a trail of particle effects with Unity’s glass shader and a transparent but bump-mapped texture. Input for this program is just touch on screen, which then calls a raycasting function and detects if the ray hits the creature. Output other than the screen is vibration when a creature is touched. For this part I’m inspired by static electricity. Too bad I can’t actually produce that to shock the user.

I want to give great thanks to everyone in this class who gave me generous feedback in our last critique. I especially thank Jackalope and Erin Zhang (not in this class) for helping me brainstorm, and Conye for motivation as well as lending me her iPhone for testing and documentation, and of course Golan for his honest critique and suggestions.

Süleymanname — a “pop-up illuminated manuscript”

I love old illuminated manuscripts. I think they are beautiful. I love the occlusion of patterns, and I think AR might be a good way to highlight the layered geometries.

I went to the rare books collections in Hunt and got some help from the amazing librarian Mary Catharine Johnsen. We started with some religious texts, specifically The Book of Kells, but after receiving critique from class, I decided not to use anything of the religious nature.

Then, after a bit of searching, I found this book.

This book is a study of the illustrations in Arifi’s Süleymanname (lit. “Book of Suleiman”), which recreates many of the events, settings, and personages of Suleiman the Magnificent, longest-reigning Sultan of the Ottoman Empire from 1520 until his death in 1566.

I picked my favorite one.

This image describes the celebration of the circumcision of Süleyman’s three oldest princes. (tbh I didn’t know this was the content…I chose it initially simply because I thought it looked the prettiest…)

Here is a detailed description of the story behind the image.

I began by separating the layers.

I used the bottom layers of the image as my target, so basically when you scan the bottom layers, they become 3D and more content come up.

Here is the way it looks when I tried it on Unity.

Currently this is simply an experimentation with aesthetics. What drew me to the materials and the idea initially was the rich history behind it, but the question that I don’t think I have explored enough is how to bring that history out, and celebrate it in some way.

Continue reading →

“An AR soundtrack to a tarot reading”

video documentation (headphones recommended):

Going into this piece, my main goal was to convey the message that one can get through a tarot reading through sound.  As someone who casually practices tarot, I understand the significance that the images on the card have in conveying meaning.  The arrangement of cards also serves an important part in a reading.  I spent a significant amount of time on this project trying to get Unity to communicate with Ableton Live through Open Sound Control.  In the end, due to time and other circumstances, I was unable to complete this. I decided that my energy was better put into creating an interesting soundscape.

I wanted to accomplish my original goal of having interesting sounds that could fit together in interesting ways while conveying the card’s meaning.  I ended up making an excel document with my concepts for each sound to guide me in the process of composing them.  Although many of these sounds eventually changed, the excel document was a good starting point and a chance for me to look for inspiration.  I often worked with field recordings, altering them to fit the sound I wanted to achieve.  I also took recordings from my own instruments, such as the flute and the omnichord.

Thanks to!

• Frank-Ratchye Fund for Arts @ the Frontier
• Golan Levin and Claire Hentschker
• Inspector J, BristolStories, and fallbackcrush of freesound.org where I obtained some field recordings
• The students of 60212 who have continued to help me throughout the semester, especially my “buddy” Sydney Zheng

Curated compositions trigger image targets to reveal hidden intent in film.

 

For our final project, dechoes and I decided to continue with our last AR Project. The last project involved telling a story through the curation of the scene and having the AR be different peoples’ interpretation of the setting. This was more of a demo for a potential interactive piece where people can input their own perspective onto a curated scene.

In progress AR:

 

 

We have been playing with the idea of seeing what is on screen vs. what we want people to notice that’s not visible on screen. The AR is supposed to reveal and highlight different moments within a curated scene in a specific situation. We are using screenshots from a short film as our image target to reveal new visuals. The different kinds of AR are triggered through the different compositions that naturally occur when a variety of items are juxtaposed together in a scene.

The first part of the project involved filming the actual scene and documenting a moment. Our scene takes place at a diner. The character is doing her usual morning routine and is there for a nice coffee. The AR represents and enhances different parts of the mundane ritual. Dechoes and I talked a lot about why we were using specific animations for each moment.

We wanted to find a way to integrate the AR to the different scenes instead of just pasting something random on top of an existing scene. The image targets are extremely specific moments that trigger the AR to appear on the screen. There were definitely some technical hurdles that we had to figure out. We played and experimented with a lot of different animation and effects on Unity but had trouble with them when we were building the app.

 

complete ridiculous fails that look pretty sick:

Trash Day 

 

Trash Ocean 

 

Some stills from the performance:

 

Behind the Scenes:

Scenes/Storyboardish/Prop list sketch

Sketch of VR

‘AR Typography is a mobile app that delivers real-time text motion tracking to video…’

AR Typography                    

AR Typography places text anywhere based on a simple touch of the screen. Built on AR Core in Unity, people can place multiple layers of 3D text, each with their own unique content, into one environment. The text tracks changes in movement, and will maintain its position and rotation values when moving around.

How It Works                    

AR Typography is able to generate 3D text onto flat surfaces (including walls) by intercepting the point cloud data from AR core and creating independent tracking planes for each object at the point the screen was tapped. When tapping the screen, the app will project the active 3D point cloud data (active referring to points visible to the camera) onto a 2D surface and find the closest point to where the screen was touched. Iterating over the point cloud data, this point will find the next two closest point in order to find the best cross product that represents the orientation of the point. The point now contains information about the position and orientation that can be used to generate the text.

In an effort to be able to film shots better without interruptions, I have effectively removed the point cloud and tracking plane visualizations.

AR Typography loads multiple strings of text into a list upon startup. The active text will modulate over the list of strings and generate a text object with the contents of the active text. This allows for effective storage and placement of multiple pieces of text in one scene.

In order to create a 3D effect for the text, a white text layer is placed at position p with orientation r, and 15 additional iterations of black text is placed at position (p – (i+1)*k*r) and orientation r, where i is the index [0,14] and k is some constant.

Visual Development                    

Rendering Environments                    

Final Product                    

AR Typography is a mobile app that delivers real-time text motion tracking to video, allowing for the creation of immersive, real-life kinetic typography videos for any videographer. To demonstrate this, I put together my own real-life kinetic typography video.

Far                    

In Collaboration With                    

AZScreenRecorder – Mobile Screen Capture

cmuTV – Found Footage

Carnegie Mellon – Logos, Branding, etc.

Friends of the Studio – For Being so Friendly

And to Viewers Like You : )

 

 

Note: This project is also my Engineering & Arts capstone, which I worked on as a part of 60-212.

Sending, Feeling

Sometimes, people need to send a message that can’t be expressed as a text, or emoji. It’s a feeling.

I created a pair of networked scarves that allow romantic partners to send feelings to one another, in the form of heat. When one partner puts a hand in the pocket of their scarf, the other person’s scarf warms up.

Distilling communication down to this simple gesture makes it possible to explore complex attention dynamics in relationships. Does one partner reach out more frequently than the other? Does the other partner respond? I’m interested in investigating the imperfections in relationships, and in codifying those complex dynamics using wearable technology.

Attention Dynamics

I started off this project by researching attention dynamics in romantic relationships. In any relationship, giving and receiving attention is key to both partners level of satisfaction. When each partner gives and receives their attention in a balanced way, they both tend to be more satisfied. But when one person needs more than the other person is willing to give, the disconnect puts strain on both partners. This generally happens to most couples at some point or another for circumstantial reasons. However, in some relationships, it becomes a pattern.

My original inspiration for this project came out of my own history with imbalanced attention dynamics in a previous relationship. Being the anxiously attached partner, I would reach out over and over again and rarely receive a response. One can also empathize with the detached partner, who can feel bothered by the constant need for validation exhibited by their anxious partner.

Physical Form: Scarves

In deciding on the form of the wearables, it was important to consider several factors. I wanted it to be easy to access a hidden compartment, where the wearer would send the signal to their partner. I also wanted it to be quick to take the item on and off, in case either partner needed a break from communication. The wearable needed to be large enough to house all of the electronics, with convenient spots for holding a Raspberry Pi and other technological components.

Looking at all of these requirements, scarves stood out to be an obvious choice. They live close to sensitive parts of the body, and the back of the neck is an ideal space for providing an intimate physical sensation. The front of the scarf has close access to the wearer’s hands, which allow them to interact easily with the pocket on the side. And the electronics can live at the back of the neck, where they won’t weigh down the garment.

Materials

The texture, color, and emotion of the scarves is very important to me. I explored different concepts, and ultimately settled on an aesthetic that I felt represented my goals for the project.

Often, wearable technology is intended to look as flashy and high-tech as possible. I wanted to depart from that, and create something earthy and intimate. I chose a muted color palette with natural tones. The two fabrics I picked play off of one another, with one more muted and one more active color. Both of the scarves are primarily made of a different fabric, with accents of the other color throughout, such as inside the pockets.

I chose a woven material for the fabric, as it was both a light fabric that could be worn in lots of different weather, and it was pleasant to work with and sew to.

Prototyping

Before I started developing the real thing, I made several phases of prototypes.

Prototype #1: Tech in Hoodies

I got all of the tech working, and to work through the challenges of sewing into physical material, I put them in a pair of black hoodies.

Prototype #2: Layout

I decided to do scarves for my final version, and laid out where each of the electronics would go.

Prototype #3: Weight

I put the heaviest circuit components actually inside of the scarf, to test if it would be horribly uncomfortable to wear. It wasn’t.

Prototype #4: Stitching

I made a prototype version just using the sewing machine, to assure myself that I would be able to do it.

 

Technology

Each scarf is powered by a Raspberry Pi computer. When one Raspberry Pi senses that it’s touch sensor was activated, it sends a signal to the other scarf to turn on the heating pad.

Turning on the heating pad involves an extra battery and a transistor.

The hardest part was definitely creating an ad-hoc network between two Raspberry Pi’s so that they can communicate wirelessly without connecting to another network. CMU secure is so picky that I decided to bypass it, and just have the computers talk to each other without wifi.

I originally tried using an ESP8226 with an arduino, and ultimately realized that a raspberry pi was a way better option.

There are two Raspberry Pis, both nodes in this ad-hoc network. I learned that there are special ranges of IP addresses, one of them being the 10.2.x.x range, which is where we placed the Raspberry Pis. Raspberry Pi A is set to 10.2.1.1, and Raspberry Pi B is set to 10.2.1.2. A whole slew of other very particular and specific network settings had to be changed in order to make this a reality, but ultimately we were able to set up an ad-hoc network where A and B are connected to each other, so you can ssh into one from the other.

The server/client I wrote to test are simple. The architecture I set up is: The server is running on B (arbitrary choice). B and A are both running clients, which can update the server based on input to the Raspberry Pi. This models the ultimate architecture of the wearable I’m hoping to create: B is polling for information that A is providing, and A is symmetrically polling for information that B is providing.

 

Making the Scarves

I learned to sew for this project, which was a first for me. I had done some work with wearables before, but it was always on readymade clothing, never on something I created myself.

Working with wearables is a whole other beast than working with ordinary circuits. I had to consider the materiality, spacing, and layout in a way that I had never had to do before. My solution was to make a life-size diagram for myself before I ever started soldering connections. This way, I was able to be sure that I wouldn’t have any connections crossing over one another.

I had a whole fiasco with conductive thread, which caused me to abandon the idea of sewing my connections into the fabric. Instead, I opted to use wires in the electronics pockets, and have conductive tape running longer connections down the entire scarf.

Lots of sewing and hours of fiddling with tiny little pieces of electronics later, the scarves were functional. Pressing the touch pad on one heated up the touch pad on the other, and vice versa. It’s hard to put into words how tricky working with these materials and tiny little pieces can be. The scarves are now robust and trustworthy.

Final Thoughts

By developing wearables that distill attention down to a single gesture, I ask questions about what attention can mean in a romantic relationship. These scarves are partly a conceptual exploration into complex relationship dynamics, and partly an exercise in electrical engineering and physical craft.

Wearables are everywhere, and they are becoming more and more popular. The Apple Watch allows wearers to send one another their heartbeat. The Hug Shirt by Cutecircuit allows partners to hug each other from a distance. In these scenarios, what does it mean to send a heartbeat instead of a text, or a warm gesture instead of a phone call? How does that change the interaction? I ask how we can explore these abstract forms of communication to learn more about our relationships with one another.

Inspiration

I came across other wearables projects in my research, and have pages and pages in my Capstone Blog about various other projects. I’m going to mention a few here that I particularly enjoyed.

Afroditi Psarra: Afroditi led a workshop at Eyeo festival, which was my first exploration into wearables and soft circuits. Her work is so elegant, and a refreshing departure from overly “high tech” looking wearables.

Sophia Brueckner: Sophia Brueckner developed a project called the Empathy Amulet, which explores connecting strangers through warmth. I loved this idea of wordlessly sending feelings through physical sensation.

Cutecircuit: Cutecircuit develops The Hug Shirt, a wearable for hugging people from a distance. It is marketed as a product, though incorporates similar ideas of exchanging attention through clothing.

Acknowledgements

 

Molly Wright Steenson, for advising me

Avi Romanoff, for Raspberry Pi help and for modeling the scarves with me

Claire Hentschker, for conceptual help and sewing machine help and emotional support

Brianna Hudock, for helping a lot with the video

Golan Levin, for giving me space to work on this for his class

Tom Hughes, for knowing where everything is all the time

Joyce Wang, for conceptual help

Nitesh Sridhar, for helping me find my way to conductive tape

Lauren McCarthy, for speaking with me about early ideas

Josh Brown, for taking this project to another level with photography and filming

Malaika Handa, for being in my video even though she was studying for finals

Boeing, for recognizing my project at Meeting of the Minds

 

Bibliography

I am creating the experience of being a sphere.

I was inspired by the concept of spheres and new forms of self-being in the aftermath of an unhappy time in my life.

The project is twofold in its execution.   Ultimately, the experience is both audiovisual and tactile.

The first aspect is the audiovisual experience, and the conceptual meaning tied to it.  As I give the person the experience of being a sphere, I also encase them within a society of spheres with their own type of existence, logic, and dialogue.

The part I will be presenting here is the more tactile, technical side, on how I give a person a new body.   The base of the physical body is a Zorb ball which is 3 meters on the outside and 2 meters on the inside.

the new body

The above picture is the Zorb ball taken within the Studio Theater in Cohon University Center.

 

viewer with straps and headset

Within the ball, the viewer is strapped in.  The straps the ball came with were quite ineffective, and so I designed a new strapping system involving lots of lashing straps, velcro, and a rock-climbing harness.  The person wears a Daydream virtual reality headset, and the virtual reality is communicated through a Pixel phone, generously loaned by the STUDIO.

orientation unit: connecting the physical and the virtual

In order to connect the reality of the physical body with that of the observed virtual reality, I allow the viewer to experience the effects of the rotation of their ball body.  The rotation is communicated from the ball’s movement to VR using an orientation unit, shown on the right and placed in the blue outline within the ball.

controlling the ball

Though the viewer inside the ball cannot, in reality, directly control the ball that they are in, they still perceive a sense of will over the ball.  They use a controller to dictate their desired motion.  Pushers will push the ball in the appropriate direction.  A “coordinator,” highlighted in blue, tells how to push the ball correctly in physical reality given the controller output from the viewer and the orientation of the viewer in physical space.

signal transfer

Here is a summary of how the signal processing works know that we have gone over the ball itself.  The orientation unit within the ball sends ball-rotation information to the VR phone.  The coordinator needs to tell the pushers how to push the ball in the right way, so they use an app on a second phone to help them.  The second phone takes in controller output from the viewer as well as information about where the viewer is looking at.  From those two bits of information, the second phone can compute how the force should be applied to the ball so that it rotates in the desired direction.

 

Here is both a GIF and video showing the physical aspect of my project, involving the ball.

 

 

Contributions [abridged]
Advisors:
Golan Levin
Garth Zeglin
Tom Hughes

Donors:
SURG/CW
FRFAF

Volunteers:
Lisa Lo
Rain Du
Maia Iyer
Anjie Cao
Connie Ye
Sophia Qin
Oscar Dadfar
Nitesh Sridhar
Peter Sheehan
Michael Becker
Winston Grenier
Parmita Bawankule