ulbrik – 60-423: INTERACTIVE ART

ulbrik-FinalProject

A conversational interaction with the hidden layers of a deep learning model about a drawing.

As an artist draws a dog, a machine gives friendly suggestions and acknowledgement about what parts are missing, have recently been improved, or seem fully drawn. It provides text mixed with visualization icons as well as continuous feedback in the form of a colorful bar graph.

The machine’s understanding of the world is based on the hidden layers of a deep learning model. The model was originally trained to classify whole entities in photos, such as “Labrador retriever” or “tennis ball.” However, during this process, the hidden layers end up representing abstract concepts. The model creates its own understanding of “ears,” “snouts,” “legs,” etc. This allows us to co-opt the model to communicate about dog body parts in half-finished sketches.

I was inspired to create this project based on ideas and code from “The Building Blocks of Interpretability.” I became interested in the idea that the hidden layers of machine learning contain abstractions that are not programmed by humans.

This project investigates the question, “Can the hidden layer help us to develop a deeper understanding of the world?” We can often say what something is or whether we like it or not. However, it is more difficult to explain why. We can say, “This is a Labrador retriever.” But how did we come to that conclusion? If we were forced to articulate it (for example in a drawing) we would have trouble.

Much of machine learning focuses on this exact problem. It takes known inputs and outputs, and approximates a function to fit them. In this case we have the idea of a dog (input) that we could identify if we saw it (output), but we do not know precisely how to articulate what makes it a dog (function) in a drawing.

Can the hidden layers in machine learning assist us in articulating this function that may or may not exist in our subconscious? I believe that their statistical perspective can help us to see connections that are we are not consciously aware of.

This project consists of a Python server (based on code from distill.pub) that uses a pre-trained version of Google’s InceptionV1 (trained on ImageNet) to calculate how much a channel of neurons in the mixed4d layer causes an image to be labeled as a Labrador retriever. This attribution is compared across frames and against complete drawings of dogs. Using this information, the machine talks about what dog parts seem to be missing, to be recently improved, or to be fully fleshed out.

This project is successful at demonstrating that it is possible to use real-time comparison of hidden layers to tell us something about a drawing. On the other hand, it is extremely limited:

The neuron channels are hand-picked, creating an element of human intervention in what might be able to arise purely from the machine.
The neuron channel visualizations are static, so they give little information about what the channels actually represent.
It only tells us very basic things about a dog. People already know dogs have snouts, ears, and legs. I want to try to dig deeper to see if there are interesting pieces of information that can emerge.
It is also not really a “conversation,” because the human has no ability to talk back to the machine to create a feedback loop.

In the future, I would like to create a feedback loop with the machine where the human specifies their deepest understanding as the labels for the machine. After the machine is trained, the human somehow communicates with the hidden layers of the machine (through a bubbling up of flexible visualizations) to find deeper/higher levels of abstraction and decomposition. Then, the human uses those as new labels and retrains the machine learning model with the updated outputs… and repeat. Through this process, the human can learn about ideas trapped in their subconscious or totally outside of their original frame of thinking.

Thanks to code examples and libraries from:

https://colab.research.google.com/

https://www.chartjs.org/

https://github.com/jhuckaby/webcamjs

Sample of the types of information provided by the UI

Summary of major events in the history of the drawing and what the machine detected at each time

Charts showing attributions for different channels over time and with a thresholded subtraction to detect events (both normalized with measurements from a canon of completed drawings)

jaquar-ulbrik-BarcodeProject

Robot Ouija Board

Robots are often thought to be deterministic beings devoid of spiritual practice. However, as physical subjects, they share our timely limit bound to this mortal coil. Who is to say they lack the yearning to communicate beyond the earthly plane to their dead brethren? Not us. In fact, we should support them if they so desire.

Robots should be able to use Ouija boards. As everyone knows, robots read in their native language, barcode. As everyone also knows, Ouija boards operate through the ideomotor phenomenon. This phenomenon expresses our subconscious (our best connection to the dead) and requires that users be able to read the text as they perform the ceremony.

Therefore, we built a robot Ouija board. It is similar to a normal Ouija board except that the letters, 0 (for “no”) and 1 (for “yes”) are in barcode. The planchette carries the start and end characters. It occasionally aligns with characters causing the robot vision sensors (modeled in our video with a bar code scanner) to read the character. Through this action, they build their message.

ulbrik-FinalProposal

Playing with the guts of machine learning models to create a conversational design partner.

For my RA work with the Archaeology of CAD project I am recreating Nicholas Negroponte’s URBAN5 design system. Built in the 1970’s, its purpose was to “study the desirability and feasibility of conversing with a machine about an environmental design project.” For my final project, I would like to revisit this idea with a modern machine (i.e. a machine learning model).

Most applications of ML are focused on automatically classifying, generating, stylizing, completing, etc. I would like to create an artifact that frames the interaction as an open-ended conversation with an intelligent design partner.

In its early stages, machine learning functioned as a black box. It developed an understanding of the world in its subconscious. Just like us, it had trouble articulating it’s intuitions. As we work on explainability, we develop tools that allow the machine learning model to communicate it’s understanding.

“The Building Blocks of Interpretability” https://distill.pub/2018/building-blocks/

This project investigates this area through a drawing program with a chat bot powered by the mixed4d GoogLeNet hidden layer. As you draw, it will calculate the difference in the mixed4d layer between your drawing and a design intent (which could be an image or set of images) and then return the neurons with the greatest difference. Google provides an API for visualizing these neurons. This will produce a set of high level abstractions that represent what your picture might be missing (given your intent). These images will be shown through various Tracery.js prompts. The purpose is to make it feel like a conversation with the machine instead of an insistence that you do what the machine tells you. I could also add some stochasticity or novelty checks to keep the suggestions fresh.

visualizations of GoogLeNet mixed4d neurons with biggest difference when Labrador retriever has no ears

This piece takes new machine learning interpretability technology, applies the idea of comparing high level abstraction vectors, and frames it as a conversation with a machine. It proposes an interaction with machines as partners instead of ‘auto’-bots that do everything for us, make all our decisions, free us from work, and control our fate.

ulbrik-checkin

I am taking a pass on this assignment, but I am working on an idea for the final. It is a system that boils down long passages of text into a series of simple illustrations.

It is somewhat similar to TransProse by Hannah Davis, except for pictures. There are projects that generate illustrations for objects, or translate a sentence into an image or scene. But, this system combines a number of technologies to convert arbitrarily large amounts of text into arbitrarily large illustrative images.

Someone could make an illustration mural of their favorite story, or create a wallpaper from a corpus of Aesop’s fables. They could write a story and see it illustrated live. A newspaper could be converted into comic strips. A room could be completely covered with cryptic stories generated by a machine learning algorithm (something like discovering a secret room in a tomb with a language you vaguely understand).

ulbrik-lookingoutwards03

Augmenting Live Coding with Evolved Patterns – Simon Hickinbotham and Susan Stepney

Hickinbotham and Stepney integrated the ability to evolve code patterns into TidalCycles (a collaborative, live coding system for music). As the coder submits bits of code to be used in the music, it is added to a population of patterns. The fitness of patterns is determined through use and votes. Mutant versions are evolved through smartly scrambling the parsed grammar trees from the code. They can then be incorporated back with the hand crafted portions. This project demonstrates a synthesis of coding and machine intelligence for a creative purpose. It integrates programming into a real-time, real life context and facilitates using it as a material.

According to the authors, the work is a novel addition to the field of music generated through artificial evolution because their algorithm resides within the context of live coding. Previous work has focused on evolving whole pieces of music and has faced many challenges with the size of the design space and facilitating human evaluation of fitness. In contrast, live-coded music is particularly well-suited for evolutionary algorithms. The music exists as small segments of digestible code (representing a smaller design space) and the coder is already sending signals that can be used to propel adaptation (which lessens human fatigue in fitness evaluation).

I would like to see an example of how this project could be used in a collaborative setting. In addition, controls for managing the GA population or creating multiple populations would create a more flexible system.

This work and others like it can be found here: EvoMUSART Conference

Epic Exquisite Corpse – Xavier Barrade

I designed and developed most of the website as a personal project to gain digital experience. It hosted more than 70 000 drawings from 172 countries and is one of the biggest collaborative artwork ever made.

Epic Exquisite Corpse (2011) is an exquisite corpse project built from user drawings tiled infinitely in a 2D plane. Users were given a small rectangular canvas with the edges of adjacent drawings visible. Each drawing they made was added to the whole. The project accumulated 70,000 drawings to make a nearly endless mural.

This project harnesses vast multitudes to work on a single piece of art. There are many variations of online exquisite corpses (such as a comic book, a song (music video), or forest drawing). Usually, they are connected along a single dimension, but Epic Exquisite Corpse is 2D. This lends it an aesthetic quality similar to reddit’s R/Place or the One Million Dollar Homepage.

I would be interested in seeing this idea extended to a non-static canvas. Users could add boids instead of vectors. This would increase the interaction between artists.

It would also be nice to find a way to introduce greater continuity across the plane. I would like to see a narrative arc or coherent structure. This could be imparted through introducing assigned themes through a logic or pattern.

ulbrik-DrawingSoftware

Wood Grain Collage Tool

Production systems are streamlined for homogenous materials. Most technologies ask us to crudely reshape the natural world into the uniform shapes they require (think tractors and factory farms). In contrast, the Wood Grain Collage Maker embraces the irregularity of natural materials.

The Wood Grain Collage Maker is a web-based tool (built with ReactJS, Fabric.js, and P5.js) for planning a collage using the grain in a piece of wood and a sketch. It allows users to drag, rotate, and scale selections and placements of wood to construct a collage. Once finished, the user can export the cut and layout files to make the collage IRL.

Wood Grain Collage Maker Demo

Process:

The collage maker tool is part of a larger work flow:

Overall work flow from source material to artifact, with some ideas that I scratched for now… AI-powered placement and a CNC router work flow ended up being out of scope.

Step 1: Collect the materials: wood with visible grain and a sketch.

A piece of plywood from the hardware store is not a completely natural product, but its wood grain is a heterogenous material and serves the purposes of this project.

Step 2: Make a collage with the Wood Grain Collage Maker.

Sketched plan the major parts of the software

Step 3: Make the artifact.

Work flow after collage maker. I projected the cuts on the original piece of wood, cut them out with a jigsaw, and assembled them on a board.

Cut file from collage maker ready for projection onto the original board. The red is the overlap that also needs to be removed. In the future, a SVG file could be generated for a CNC router.

Tracing out the cut file on the piece of wood.

The physical artifact arranged according to the layout designed by the test user. It is glued together and wiped down with tung oil to bring out the grain.

Evaluation:

Originally, I planned on making a collage design machine that acted as a thoughtful, “creative” partner. It would suggest new, provocative ideas while the user worked. The tool would help the user quickly navigate and make leaps through the design space. However, before I could do this I needed a digital collage tool in which to integrate the assistant.

The design space as an archipelago of islands and the designer as a small boat. Tools that make a process more convenient help the designer cover more ground and expand the search area for design treasure. But what about great leaps to distant new continents? Could a “creative” machine help with this?

I created the Wood Grain Collage Maker to facilitate the collage work flow, calculate overlaps, and produce the documents necessary for the physical realization of the collages. My hope was that the tool would allow me to be efficient enough to find a state of creative flow.

When tested with a small, captive audience of one, I received positive feedback that using the collage tool was fun and soothing, much like a puzzle. In addition to the enjoyment of making the design, it was also exciting to put together at the end. As it turned out, the software showed more potential as a form of puzzle than as a tool for production. Maybe adding an intelligent system is unnecessary…

ulbrik-mask

My Two Other Faces

Glitch Demo

This performance is an investigation of the potential existence of multiple personalities within ourselves. It features an exploration of the two other people generated from the left or right sides of my face in a digital mirror. The program I created for the performance can take one half of my face and reflect it over the centerline to cover the other half and create a new face entirely from one side.

Website for doing something similar with static photos: http://www.pichacks.com

I was inspired by reading about occasional emotional asymmetry of the face and the differences between our right and left brains. I had also seen reflected versions of my face before as pictures and found it fascinating to imagine these other people trapped inside me. I was curious to meet a live action version of them and see if they embodied half of my personality.

Illustration of reflection over pose/orientation line

To create this project, I used the FaceOSC openFrameworks app along with an example Processing app as a jumping off point to create a digital mirror with face tracking. I cut out half the face, then reflected, rotated, scaled, and positioned it back onto the other side of the face. I included controls to switch between the different possible faces and to make minor rotation and scale adjustments for fine tuning for the crookedness of your face. It functions best if you do not twist your face.

ulbrik-lookingoutwards02

dadamachines by Johannes Lohbihler

The dadamachines automat toolkit by Johannes Lohbihler is an open source toolkit for creating instruments from arbitrary objects. A central controller connects a MIDI or other music controlling device to solenoids/motors that can interact with objects.

I like this project because it enables people to hack the physical world. I enjoy programming, but I would like to escape my 12 hours of daily screen time (My parents only let me have an hour a day as a child, so why is this suddenly OK?). Tools like this partially realize my dream for the future that goes something like this:

I am doing things in the tangible world. I encounter a difficult situation. I flip open my computer, type up some code to help deal with it. Then, I go about my merry way.

This toolkit provides a way for the expressive power of programming to be accessible and useful in a lightweight manner in our day-to-day lives.

This project shows the influence of Dadaism through its encouragement of readymade objects in music. It updates this in the context of electronic music and maker culture by connecting readymades to MIDIs through Arduino.

If you buy the large package for $750, you have to wonder, “Is this just an Arduino with fancy packaging? All it does is control solenoids and motors.” However, I think this project is exactly what the creators of Arduino had in mind with their platform: prototype with Arduino, then create a robust, specialized version. The creator partnered with manufacturers, engineers, and software developers from Germany and China to realize a product funded by €150,548 from 488 backers on Kickstarter.

This project inspires me to make things that interact with nature through programming. Nature is complex and it is hard to imagine how solenoids and motors will deal with messy natural substances. Maybe a device closely connected to the person operating it could combine the accuracy and repeatability of machines with the situated knowledge of humans to overcome this hurdle.

Project: https://dadamachines.com/

ulbrik-2Dphysics

Bubble Pop – Pinch and Smash the Bubbles!

I began this project thinking about the satisfaction people get from popping and smashing squishy things. Whether it is popping pimples and blisters or exploding fruit, we love to pop things. Gallagher has made an entire career out of it.

I was pleased with how the LiquidFun physics engine models the behavior of popping. I created bubbles out of a ring of rectangles connected by rotating joints encapsulating loose particles. When pierced or pinched, these bubbles build internal pressure until the skin collapses catastrophically and the contents explode outwards. The effect is mesmerizing and there are endless types of bursts the user can discover through experimentation.

Live Version: http://www.erikulberg.com/projects/bubblePop/index.html

ulbrik-reading1

My goal for my work is to expose and examine dominant values. I believe that art, at its best, speaks in an iconoclastic voice while maintaining compassion for real people living messy lives. One of my favorite books, “The Theory of the Leisure Class” by Thorstein Veblen, shows how creative projects can provide necessary perspective on unexamined patterns.

In his book, Veblen explains the motivation behind the unproductive economics of leisure activities and conspicuous consumption in our society. It reveals common phony behaviors and the necessary role they play in social settings. His work helped me to understand certain unconscious actions by myself and others as concurrently absurd and forgivable.

I hope to also uncover the unseen through my work. I find myself most attracted to overlooked inefficiencies, like the unpicked fruit on a neighbor’s apple tree or the valuable materials in our municipal waste streams.