Romeo and Juliet in Hades (1998)

by Ryohei Nakatsu and Naoko Tosa

Romeo and Juliet in Hades is a bizarre sequel of the Shakespeare play, developed by a team of computer scientists led by Ryohei Nakatsu and Naoko Tosa. In this piece of “interactive computer theater,” participants play the parts of Romeo and Juliet, and go on and adventure in the netherworld.

Since its initial production in 1998, this project was included in technical papers as well as film festivals. I see this project as an extremely intentional hybrid of novel technologies and existing art forms — computer graphics, theater, cinema, video game. The key component that connects all of them and challenges their boundaries is interactivity. The experience is designed so that participants have agency in deciding how the story unfolds and what is presented on the screen. They are writers, players, and actors.

Shown in the demo video, the interactivity in this project is separated into two kinds: story-based interaction, and spontaneous interaction. Story-based interaction is enabled by speech recognition and motion capture. Presented with choices of lines, participants can choose one of them to say out loud. Speech recognition technology then captures the speech and moves the story forward based on it. The function of motion capture is that participants can move their avatar on the screen and “touch” certain objects in the cyber world, which also trigger certain storylines to unfold. Spontaneous interaction is enabled by emotion detection. The faces of participants are captured in real time and mapped to characters on the screen. Also, if a participant talks spontaneously to a character, the character can talk back based on the emotions of the participant.

Romeo and Juliet in Hades is a piece of interactive art work, but putting it in any genre feels like over-simplifying, because it crosses so many different fields of work. It exemplifies early attempts of blending computer graphics with art and entertainment. It is very much a technical experimentation, as well as an experimentation in storytelling. It has the narrative capacity of cinema as well as the intimacy of theater. I especially love the possibility for performance in this piece. Participants are encouraged to move their body and physically experience the journey of their avatars. They can speak out loud in emotional and expressive manners.