Looking at audiovisual archives: understanding AI technology

To celebrate their 20th anniversary, Dutch future-themed TV programme VPRO Tegenlicht asked Richard Vijgen - a Dutch artist/designer whose work focuses on artistic data visualisation - to create an experience that presents their broadcast archive of 20 years to the world, but also gives them a scenario for a future way to archive. Result: 'Through Artificial Eyes', an immersive, interactie installation using computer vision*. The installation was displayed at Het Nieuwe Instituut (HNI) in Rotterdam - national museum for architecture, design and digital culture - and is currently on display at Tresoar in Leeuwarden where it is applied to the Friesian audiovisual archive.

*Computer vision is a field of AI that trains computers to capture and interpret information from image and video data. 

The objectives

To explore what could be the role of computer vision (an AI technology) in audiovisual archiving.

To raise awareness around the workings of a computer vision neural network to a broad audience. To demystify the behaviour of algorithms in a playful manner, enabling visitors to understand this often seen as black box technology where they cannot see the inner workings of an algorithm.

The installation

Visitors experience how the computer views the archive of VPRO Tegenlicht. How it searches for and detects certain patterns in 555 VPRO Tegenlicht's broadcasts. The installation playfully allows visitors to discover processes that are essential to computer vision and neural networks. They can dig deeper into the technologies by analysing how algorithms ‘see’ patterns in images or texts. 

A great collaboration between the physical space and digital design. The installation really invites you to interact: dialogue between man and machine.

Visitors can play around with the sliders and see what it does to the results. They are three categories: people, artifacts and natural objects. These categories are the three 'eyes' through which the algorithm observes the archive.

The visitors can then set the 'confidence' threshold. Meaning how confident the algorithm needs to be for a certain result to appear. By playing with this, visitors can explore the limits of the algorithm.

At the same time, it raises many questions about and exposes the ethical aspects of AI technology. Such as who decides which categories are being used, who labels the training data (through who’s eyes are we really looking?) and what are the implications of using this technology. 

Behind the scenes

Studio Richard Vijgen used ImageNet as a source as it is used in many applications using computer vision. 

ImageNet is a visual database designed for use in visual object recognition software research, for training of large scale object recognition models. More than 14 million images have been hand-annotated to indicate what objects are pictured. 

By using ImageNet the studio wanted to show how the structure of Imagenet and the (cultural) perspectives embedded in it, are finding their way into current applications. They included as many categories as possible to highlight both the obvious and successful labels: recognising an apple or a boat. But also the more difficult ones like recognising a bad person or a Dutch person. 

Even though it is technically possible to train a computer by showing it lots of photos of 'bad people' (this is an actual Imagenet category), it is questionable and sometimes leads to controversial outcomes. Yet it is likely that this training data is used in existing applications, creating a rather biased and, perhaps, not truly realistic representation.