Author: szh

szh-Final

For the final project I wanted to revamp my Face Super Mario controller to make it more user-friendly. (Not sure if it's there yet... but hopefully.)

Since my original inspiration was from a Youtuber who creates videos of beating classic Mario games in inane ways, I really wanted to be able to package my code as an actual program and sent it to him.

But before that, I needed to improve the user interface, so that it can be used by someone who isn't just the programmer who knows all the bugs and work arounds.

I gave the option to users to "reset" the initial position:

(The center lines indicating where the face moved.)

This allows for further recalibration in case the model didn't start of correctly, or if you want to reposition yourself. This also gives a "hard reset" to all the control incase anything is acting finicky.

I also added in options so you can choose how to move your controls!

There is now moving the body back and forth, moving the face side to side, raising the eyebrows, and opening the mouth. And smiling to sprint and or use your power up!

In addition, the user can click on the black circle on the left to recalibrate the settings (so if the maximum and minimum for something is too far off, it can be adjusted to each user's own needs).

(It was kind of difficult to manually make and place all these buttons? Processing maybe isn't the best place to do all the UI!)

In the future, I want to implement a feature where the user can manually input what they want each action to map to what keys (unlike turning something on or off) so there's even more freedom.

szh-arsculpture

Concept

In our studio desks, we have name tags differentiating which desk belongs to which design student. Another aspect of our studio culture is that we'd often leave notes, snacks, etc. for our friends. For example, when we know that a classmate is going through a stressful time, someone might leave a couple chocolates.

For my AR sculpture concept, I thought about using the name tags as an image target (since almost everyone still has theirs on their desks), and having this be a place where people can leave 3D presents for each other.

In this project, it was a rose.

(The original concept behind this was what if we had people leaving behind bouquets of flowers as a present? I wanted to do something where I used the "language of flowers" and had users put together their own bouquet of flowers (using flowers that convey the meaning they were hoping for), and leaving this on people's desks as a gift.)

As per our typical studio culture, we have a semi-public space, as anyone can see what is on our desks, but at the same time, this desk is our space. It is a common courtesy to not touch what is on it unless given permission to do so. Therefore, I decided to keep my application public, so that anyone with the app can see what someone has left on their desk.  (No login factor.)

Development

For the server side of this application, I decided to use Firebase so that I could share the information across platforms (for anyone who has the phone).

However, I ran into a couple issues of getting Firebase to actually work. The biggest problem was getting the build to actually work, because I ran into an issue with CocoaPods, a library that has a universal dependency that was supposed to be automatically installed when building to Xcode...but did not...

I know that Connie used FireBase for her project from last year, so I viewed her documentation (which was actually really thorough!) and looks like she ran into the same error as me:

And I tried doing the solutions she had listed out in her documentation, and had even found the same forum post she was referencing!

Alas... even after I directly installed CocoaPods to my computer using terminal, and double checked that it was actually in my Finder, I was still getting the same error.

As for the actual interaction with the prefab, I wanted users to be able to add a gift for someone once they tap on the screen (if the image target has been recognized), but for some reason, it loads the prefab before I have even touched it...

It was actually sort of odd because sometimes it would show only the original prefab, which is just the blue square, but then it would flash the rose prefab (that's supposed to only show after you touch the screen) immediately afterwards. I'm assuming this bug is coming from my code assuming that the user has touched when I have actually not..

// If user has touched
if (Input.touchCount > 0 && Input.GetTouch(0).phase == TouchPhase.Began){
Debug.Log("Touched!");
imageAnchorGO = Instantiate<GameObject>(prefabToChange, position, rotation);            

I added a boolean and put it in Update() so it would run each time, but the entire app has been crashing... will look into this later.

Future developments...

Assuming I could get the Firebase server to work, I would've ideally wanted a function where users could add in their own present and gift (can pull from a database) and then compile it together. I think a future consideration would be thinking about how long the present should stay on the desk (maybe the owner of the desk can delete it themselves? This may require having log in information, etc, etc.)

szh-vingu-justaline

Our concept involves showing an ordinary doodle from one angle, but as you turn / see the image from another one, the doodle turns into a rather phallic one.

This could probably work as a prototype before 3d modeling.

szh-UnityEssentials

I didn't know you could export directly from Maya; this would  be helpful for me in the future since previously I would always export an .fbx file from my Maya file. Also interesting how often these two platforms are used that there is a direct transfer...

 

szh-SituatedEye

We were inspired by Google's drawing machine learning, and the ability to play around with the different types of applications that machine learning has in with drawing. In order to most quickly and accurately iterate over and over again, we started our explorations by playing around with the whiteboard. We started off playing around with the program to see if machine learning was able to detect the difference between written text and drawings. From this, we were also thinking of maybe incorporating mathematical graphs and/or equations as a possible third scope; an example that lives between text and drawing.

From our experiments, we saw that computer could usually detect between drawings and text, presumptuously mostly dependent on the text. The diversity of drawings was differed widely, as we literally began to draw just about everything that first came to mind, whereas text was definitely more limited in terms of aesthetic, and was visually more uniform. However, we came upon an interesting discovery when drawing stars, but in a linear form. When only one star was drawn, it was detected as a drawing, but as we continued to draw stars next to each other, in a linear fashion, it was detected as text. This propelled us into thinking about the possible implications for using machine learning to detect the differences between languages.

The stars that sparked off our stars.

Our final exploration dealt with exploring the detecting the difference between western and eastern languages; western being more letter-based, and eastern being more pictorial-based characters.

We decided to map the result out visually through three colors:

  • White indicates that there is no hand written text being shown. (We fed a series of white background images to train this part)
  • Blue indicates that it is more-so western text. (We fed a series of handwritten phrases and words in English, Spanish, French, and Latin to train this part)
  • Red indicates that is more-so eastern text. (We fed a series of handwritten phrases and words in Chinese, Japanese, and Korean to train this part)

From our results, we've discovered a couple things.

The program is relatively good at detecting a "blank background" though a couple times, when our paper was shifted, the program recognized it as "western".

But most importantly, the program was very accurate in detecting western text, but significantly less so with eastern text.

This observation has led us to a couple hypotheses:

  • Our data is lacking. We took about 100 photos for western and eastern each, but this may have been not enough for the machine learning to generative a conclusive enough database.
  • The photos that we took could also have been of not high enough quality.
  • In our sample data, we only wrote horizontally for western text, where as eastern had both horizontal and vertical.

Future thoughts...

To test the machine learning program to see if could simply tell the difference between western and eastern languages, we could do away with the "varied handwriting" completely and use very strict criteria (for handwriting style) in writing our sample text. When we tested the learned program, we could continue to write in that same style between the eastern and western texts. This could help isolate our variables to test out our above hypothesis.

szh

Pix2Pix

I started off with trying out obscure shapes, attempting to figure out exactly what shaped causes what kind of result for the cat.

From my experiments, it looks like the look for certain features first, then allow duplicates. For example I thought at times I would have two legs but no tail but it seems that it tries to put a tail in the space anyways? Or that I thought my feature would be an arm, but it turned out to be an ear. I also thought it was interesting how it worked with my lines that had no volume; my cat simply looks like it has a folded seam overtop of it.

 

GANPaint Studio

After exploring this generator for a bit, I theorized that it used very similar technology to Photoshop's stamp and healing brush tool, where it copies over a certain image, reads the surface below that it is copying over, and tries to match both textures.

Artbreeder "General"

Artbreeder "Portrait"

Looking at both of these generators, it's interesting to see what they take from each image. My first general one wasn't so successful, but as I played around with how much of each image it took, it became much more interesting. I also enjoyed the second one, portrait, which was very similar to a "baby face generator", but then when I inserted a pancake, I got really fun results. I like the texture I created on the lady.

Infinite Patterns

GPT-2

After exploring this generator for a bit, I theorized that it uses very similar technology to xtran, only without a microprocessor. The only difference is the way it handles the input data. xtran uses a binary (or binary tree of binary data) format, and I found an obvious similarity in this way. The tree, in the generator, is basically the size of bytes of the input data, minus any padding on the right side. For instance, a character or character combination of 10 bytes would look like this:

00100000 02102001 00110 000010 00000001 000100 < < < 10 012 < 100 0f 011 < 02 < 00 <

The generator is the same, the only difference is the padding and data on the left side. For this reason, I was able to use the Tree Generator, but only from the following generator:

The generator uses a variable-length (0-255 bytes) binary tree with a single insertion and removal node. The initial node, i, is a random seed, which is derived from a random number generator (also in hex). The insert and remove nodes.

There was no way he could actually like him, after all that he had done. It was insane! No possible way that there would be any chance for romantic feelings to develop.  He had just tried to save him!
"Are you insane," said his girlfriend. "This guy, he is so far away and in such a dark place and he is trying to help you!"
'But it is only possible if you are on some form of medication,' he countered. 'If you are just like in high school all the time. This is ridiculous!'
Serena could have been saying this. It was that last day of school. She was about to go to play basketball at one of the two places where she wanted to go play basketball. The first one had been cancelled for reasons yet to be known to her. And with her being so far away from her old life and the one she lived in, there was no way either place would be welcoming her in at this time.
Her parents were on vacation and would be gone until a couple of days later. They would be home on Wednesday, and she didn

Google AI Experiments

rock paper scissors

I thought this project was really intriguing because of how simple the premise was. I also enjoyed learning about how they made it; the developers had used all technology I have seen before and I thought it was really cool the way they made it work together. I guess the only thing that's rough about this project is that the computer will always win in rock paper scissors.

szh-Telematic

full process on vicky's page

Moood is a collaborative listening(ish) experience that connects multiple users to each other and Spotify, using sockets.io, node.js, p5.js, and Spotify API's.

In this "group chatroom" of sorts, users remotely input information (song tracks) asynchronously, and have equal abilities in leveraging the power (color, or mood) of the room.

This first iteration, as it currently stands, is a bare minimum collaborative Spotify platform. Users type a song track (must be available on Spotify), which would then be sent to a server, to be communicated to Spotify. Spotify would then analyze the track based on six audio features: 1. Valence (a measure describing the musical positiveness), 2. Danceability (a measure describing how suitable a track is for dancing based on a combination of musical elements including tempo, rhythm stability, beat strength, and overall regularity), 3. Energy (a measure representing a perceptual measure of intensity and activity), 4. Acousticness (a confidence measure of whether the track is acoustic) 5. Instrumentalness (a prediction on whether a track contains no vocals), and 6. Liveness (a detection of the presence of an audience in the recording) These six audio features are then mapped onto a color scale, and are the aspects in which dictate the color gradient being represented on the screen, which will then be broadcasted to all clients, including the sender.

At this stage, users would be able to share current songs they are listening to and dictate the way in which the "mood" of the room is represented, by changing the color in which room would be. Color is an extremely expressionistic and emotional visual cue, which has the ability to tie in beautifully with the aspect of music.

Our initial idea is a lot more ambitious, however, we ran into several (an understatement lol), issues. The original play was to create a web player environment that would consist of the 3 RGB colors, and CMY color overlaps, with white in the middle. Users would be able to click onto different colors, and the combination / toggle of the colors would trigger different songs to be played based on our mapping of colors to the Spotify API endpoints used above (in our current iteration). Users would then be able to dictate the visual mood of the room, as well as audio mood of the room, by mixing colors and playing different songs. First, there was the issue of the being able to create user authorization; there are several different types of it, some not being compatible with certain codes, and others having certain time limits. Next, there was the issue of being able to handle playback on Spotify Web API, versus Spotify Playback SDK, versus using Spotify Connect. SDK did not allow for collaboration with node.js, but the other two ended up creating issues in overlapping sockets, listening ports, and so on. We were also unable to manipulate / figure out how to pull apart certain songs from select playlists, but that was an issue that we could only have dip into due to the other issues that were more pressing. Because there is not only server and clients being communicated across here, and instead the entire addition of another party (Spotify), there was often conflicting interests in where that code intersected.

That being said, because we have managed to overcome the main hill of having all these parties communicate to each other, we would want to further work on this project to incorporate music (duh). It is quite sad that it is a project revolving around Spotify and music as a social experience, without the actual audio part.

szh-LookingOutwards04

Emoto

This is a really cool project that an upperclassmen friend did for a studio project that I thought was really inspiring, as it really blends the space between the digital and physical world together.

The robot on its own works in conjunction with the user, which I found to be a very innovative / cool point of the project.

 

I'm really interested in being able to combine my programming skills / assets in technology with my designs and I feel with something such as physical computing, I am able to truly build a bridge between that physical and digital world to achieve a project like this that can help people through the usage of technology, without there being a gaping hole of what that connection could be.

szh-CriticalInterface

9. Can we make the invisible visible? The more present interfaces are in our lives, the less we perceive them.

Something I feel like I work with a lot as a designer in the Environments track with a focus in the digital world is trying to help users / people focus on the almost "invisible world" that is our internet. How can we see this? For example, take an online delivery service -- how can we design a space to help people see this visually rather than just something that exists in the physical world? I do agree that the more that we rely on interfaces, the more we will stop seeing them as physical spaces, though they most definitely still are and these digital worlds are important towards helping us understand / manifest a physical space around us, even if we don't really see that anymore.