Evaluating Natural User Interfaces for Public Displays in the Wild

by Jörg Müller, Daniel Michelis, Chris Kray
Position Paper, Workshop Natural User Interfaces, CHI 2010


One major application area of Natural User Interfaces are public displays, which are often not intended to help the user fulfill a certain task, but rather to be engaging, inspiring and entertaining. A key issue in this area is how to evaluate these systems and their user interfaces. Based on the case study of Magical Mirrors, we identify challenges that occur with evaluating such systems and propose ways to address these evaluation challenges. Challenges include the lack of a welldefined task, the difficulty of describing and observing audience behavior, the briefness and sparseness of typical interactions, and the public nature of the space.


Public Displays are by definition provided for a very broad user base, many of whom will not have encountered the display before. For such situations, Natural User Interfaces (NUIs), which aim to be learnable very quickly, are ideally suited. Such NUIs can involve touch [5] or gestures [4]. Typically, people pass by such displays at a distance, where gestures are well suited to initiate interaction. Rather than to support the user in a specific task, we argue that the most interesting application of such public displays may be to engage, inspire and entertain the audience. The fun of interaction may be an end in itself. We deployed Magical Mirrors, a set of public displays where passers-by can interact with gestures, and manually observed how the public interacted with the displays. We report on the challenges we encountered while evaluating the system and indicate ways to address these challenges.

Magical Mirrors

Magical Mirrors [4] is an installation of four large public displays in Berlin, Germany. Magical Mirrors are not intended to fulfill a specific purpose, but rather to encourage the engagement of the public audience. The displays show a mirror image of the environment in front of them and apply optical effects reacting to the gestures of the audience. During one hour on a weekend evening, on average 310 passers-by passed the displays. In addition to the Magical Mirror displays, three big projection displays at the edge of the same building showed screenshots from interacting users to audiences across the adjacent junction and down the neighboring streets. The displays were active each evening after sunset from February 2006 until June 2007. The displays show a mirror image of the audience together with one of four different visual effects. The Aura effect shows a white aura around the boundaries of the audiences’ bodies. The Luminary effect shows a cloud of numbers around the fastest moving image region (movement center) (e.g. a hand).

The Flexibility effect shows a ribbon that follows the movement center. The Progression effect consists of a number of flowers which grow towards the movement center. Interaction with the system was evaluated by observing passers-by from a car parked nearby.

Challenges for Evaluation

Magical Mirrors can be seen as prototypical for displays that 1. are installed in public spaces, 2. support
gesture-based interaction and 3. do not support a clear task, but are rather intended to engage the audience. Evaluation of such displays poses the following challenges:

Lack of well-defined primary task

Systems like Magical Mirrors are not meant to help the user in solving a specific, well-defined task. Instead they should engage, inspire and entertain the audience. Traditional usability metrics like effectiveness and efficiency are difficult to apply, because there is no defined goal that the user can achieve. Instead, new metrics need to be found that are suitable to describe how engaged the audience is. Measures of User Experience, like questionnaires, may be used [2]. For the specific case of public displays, duration of interaction, proximity and direction of movement, as well as the occurrence of indicating behaviors may be further measures.

Describing and Observing Audience Behavior

The second challenge is that people use NUIs like Magical Mirrors in unexpected and unpredictable ways. For example, one user interacted with the display by approaching the camera and using the movements his

tongue or others tried to interact by moving their hips only. While this appropriation is what Magical Mirrors was designed for, the identification of interesting audience behaviors that are also measurable proved especially hard. For instance it was difficult to distinguish whether behavior was directed towards the displays or whether users were waving towards a friend. A catalog of typical audience behaviors and methods to observe them would be helpful. This would also allow comparison between systems, as well as provide useful design orientation.

Briefness and Sparseness of Interaction

Interaction with public displays through NUIs may be particularly brief and sparse. Many passers-by merely moved a hand while passing by, leading to interaction times of 1-2 seconds. Also, depending on the location and the attractiveness of the display, only a few people a day may look at them [3]. The combination of briefness and sparseness of interaction makes observation particularly time consuming, and one may miss the few interesting seconds. The selection of deployment locations with a high footfall and high attractiveness to passers-by are crucial for evaluation. Magical Mirrors was installed in a location with more than 300 passers-by during an average hour, and the reaction to audience movements naturally drew attention. Automated audience measurement (like face detection) may help in cases where high footfall cannot be achieved.

Evaluation in Public Space

Public Spaces are spaces which are by definition provide more or less unrestricted access to the general
public. Thus, there is very little control over these spaces, and street musicians performing in front of the

displays or an attention grabbing truck parked next to it can significantly interfere with audience behaviour. Furthermore, public spaces fulfil a number of functions, like taking a walk, shopping, getting to work or home etc. These functions strongly affect audience behaviour, for example people getting to work are less likely to interact than people taking a walk. Finally, people in public spaces play specific roles [1]. For example, a policeman may be less likely to interact. On the other hand, an evaluator without an obvious role usually draws a lot of attention. When we just stood next to Magical Mirrors for example, we attracted more attention than the displays themselves, and people interacting with the displays repeatedly looked at us in suspicion. In order to invite people to interact, the placement of systems is crucial. Places with leisure related functions are well suited. Regarding the role of the observer, carrying a drink while sitting on an electricity cubicle led to much less attention of passers-by.


Evaluating displays with NUIs in the public is challenging. Nonetheless, field studies of such systems
are necessary, because behaviour of the public towards displays can be very different from what the designers intended [3]. Currently, field studies are the only way to evaluate actual audience behaviour. Lab studies that simulate a public situation may be a way to obtain similar results while maintaining control over the setting [6]. We hope that this list of challenges will help researchers interested in evaluating NUIs in the public and foster a discussion how to do such evaluations.


  1. Goffman, E. Presentation of Self in Everyday Life. Doubleday, New York, 1959.
  2. Hassenzahl, M., Tractinsky, N. User Experience – a research agenda. Behavior & Information Technology, 25(2), 91-97.
  3. Huang, E., Koster, A., Borchers, J. Overcoming Assumptions and Uncovering Practices: When Does the Public Really Look at Public Displays? In: Proc. Pervasive 2008, Springer (2008) 228-243.
  4. Michelis, D., Meckel, M. Why do we want to interact with electronic billboards in public space? In: Proc. Workshop on Pervasive Advertising, in conjuction with Pervasive 2009.
  5. Peltonen, P., et al. It’s Mine, Don’t Touch!: interactions at a large multi-touch display in a city centre. In: Proc. CHI 2008, ACM (2008)
  6. Snowdon, C., Kray, C. Exploring the use of landmarks for navigation support in natural environments. In: Proceedings of Mobile HCI 2009. ACM Press, New York, NY, USA (2009

0 Responses to “Evaluating Natural User Interfaces for Public Displays in the Wild”

  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

Enter email and receive new posts.

Join 7 other followers


Pictures & Figures

March 2010
« Jan   Apr »


Blog Stats

  • 41,570 hits

%d bloggers like this: