Swiss Army Knife meets Camera Phone: 
Tool Selection and Interaction using Visual Markers

Christian Kray, Informatics Research Institute, Newcastle University, UK
Michael Rohs, Deutsche Telekom Laboratories, TU Berlin, Germany


Abstract

A key issue in ubiquitous computing in general and public display research
in particular is how to enable interaction. Oftentimes, it is not clear how
users can interact with a system and what functionality it provides. In the
case of public displays, several methods have been suggested such as touch-
enabled surfaces, gesture recognition, voice input, or text messaging.
However, all these methods have some inherent flaws such as being
unreliable, limiting the number of concurrent users or requiring complex
configuration. In this paper, we introduce a novel approach based on visual
markers that users can photograph using their mobile phones. By displaying
the marker snapshots on the screen of their phone and moving them over a
public display, an external camera can track the position of the device and
identify the marker being displayed. We introduce a prototype making use of
this idea, and highlight key benefits of our approach such as no need to
install custom software on the phone, the elimination of network
configuration and exposure of system functionality.