#8b (Situated Eye, Due 10/30) – 60212: INTERACTIVITY & COMPUTATION

This assignment is due Wednesday, October 30th at the beginning of class.

In this project, you are asked to create a "situated eye" - a "contextualized classifier" - a "purposeful detector" - a "poetic surveillant". You will create a camera-based system:

which is located in a specific place;
which is trained to detect a specific thing (or things);
and which responds to what it sees, in an interesting way.

Design Considerations

Escape the physical context of the laptop. Don't limit yourself to the physical constraints of your laptop's webcam, and the implicit assumptions it imposes on where a camera can be (on a table, in a room, at waist height, with a level pitch). If necessary, borrow a peripheral USB camera and a camera mount.
Further to this point: Give extremely careful consideration to where your camera is located, and/or what it observes. Is your camera on a building? In a refrigerator? Above a pet dish? Part of a microscope? Pointed at the sky, or at the floor? Looking at custom cardboard game pieces on a table? Observing objects on a conveyor belt? This is not a speculative matter; actually do the thing.
Your system might respond in real-time, or it might serve as a system for recording, logging, or counting what it observes. Keep in mind that you can save files (data, images) to disk...
Your system might respond audiovisually (i.e. with graphics and/or sound), and/or it might send a signal over the internet.
Your system might respond to the actions of a live interacting user (i.e. in the manner of game controller), or it might respond to people, vehicles, animals, or other phenomena that are wholly unaware that they are being observed.*

*It is understood that you will not violate anyone's privacy.

Technical Details & Templates

You are strongly encouraged to use one of several provided templates:

ml5.js "Feature Recognizer"-based Image Classifier:

ml5.js "Feature Recognizer"-based Image Regression:

ml5.js KNN Classifier:

Note: The KNN classifier is recommended for point data (posenet, faceapi), whereas the Feature Extractor only uses images. The KNN classifier is less accurate for images.

The Google Teachable Machine is very similar to the above tools. We won't be using it for this assignment, but the following might be helpful to view, FYI:

Notes:

It's expected that you will train your own recognizer (i.e. as opposed to using a pre-trained recognizer, like YOLO or MobileNet). However, if you have good reasons to use a pre-trained system, speak with the professor for an exemption.
It's expected that you will train your system on images. Other media (sound, poses, etc.) may or may not be supported by the Professor and/or TA. If you ask for an exemption to work outside the provided templates, you'll be working more on your own.
Some of the provided templates demonstrate how you can load and save your custom models. I haven't been able to get this to work, but in theory it, it could save you time.
You can load pre-stored images from the p5.js Editor file manager, using code approximately like the following (you'd need to have images named 1.jpg, 2.jpg, etc. in a folder called 'dataset'):
```
let imgPath = "/dataset/" + label + "/"
let filename = imgPath + str(i) + ".jpg"
classifier.addImage(filename, label);
```

List of Deliverables:

A blog post, titled nickname-SituatedEye, and categorized, 08b-SituatedEye
A written description of your project, 100-200 words, including a self-evaluation
A brief demonstration video, embedded in the post (uploaded, or linked to YouTube or Vimeo)
At least 3 pieces of visual documentation of your project, consisting (as appropriate and/or possible) of photos/scans of your notebook sketches; photographs of the project in situ; screenshots of the project; technical diagrams; example images from your training data, etc.
An animated GIF