RIOT is an immersive, emotionally responsive, live-action film that positions viewers in the middle of a riot in progress. The film responds to participants’ emotional expressions using artificial intelligence, altering the video story journey in real-time.

The RIOT film installation experience allows viewers to consider how they might react in the face of imminent danger. A webcam or camera is used to monitor each viewer’s facial characteristics as they watch the film, and the video narrative responds accordingly.

For example, if the viewer appears agitated, the character in the film responds defensively or impatiently. The same is true for a number of other assessed emotional expressions, such as calmness or fear.

RIOT provides feedback into complex cultural issues surrounding split second decisions in times of actual crisis. It places viewers at the center of a provocative story for which there are a number of potential live time scenarios which may triggered by their authentic reaction.

Development of a New Prototype

During her time at the Thoughtworks Arts Residency, Karen worked with Thoughtworks developers on a new machine learning-based facial expression analysis system, and on enhancing the storyline management and RIOT user experience.

As a result, Karen can add new emotional expressions to the experience, and can add new narrative layers to the RIOT prototype via a custom-built user interface.

The RIOT experience works with a deep neural net toolkit for emotional expression analysis, created by Thoughtworks, named EmoPy. The system has been made open source in order to provide free access beyond existing closed-box commercial implementations, both widening access and encouraging debate.

As of 2019, EmoPy is featured as a Thoughtworks project on the Open Source homepage.

Karen’s previous work with Dr. Hongying Meng of Brunel University, London, was refined and developed as part of the piece. This includes his research and implementation of new facial expression analysis techniques.

During her residency, Karen hosted regular ‘Open Studios’ events at Thoughtworks New York. Stakeholders from across industry, academia and the arts were invited to discuss the implications of AI technology, using the residency and artwork as a focal point for critical discussion.

Karen undertook research on human emotional understanding with Emily Balcetis, Associate Professor of Psychology at the Social Perception, Action & Motivation Lab at New York University. This research continues as Karen moves forward with new versions of the RIOT system.

Exhibitions and publications

In the fall of 2017 Karen spoke at TED New York, telling the story of her work from conception through to the Thoughtworks Arts Residency. Karen goes on to explain the next stages of development for RIOT, which will deal with a greater range of emotions, more narratives and a multiplayer experience.

The RIOT film experience was exhibited at the SPRING/BREAK art fair as part of New York’s Armory Week, alongside other Thoughtworks Arts-incubated artists. It has been exhibited at the Future of Storytelling Festival, the Museum of Modern Art Peru, and Festival of the Mind Sheffield.

The project has also been featured in global publications including Forbes, CBS, The Guardian, Fast Company, NBC, WIRED, Engadget and the New York Times.