TWEND: A Prototype Framework for Exploring Bending as Interaction Gesture in Mobile Devices

Gero Herkenrath

Media Computing Group

RWTH Aachen University

gero@cs.rwth-aachen.de

ABSTRACT

In this work we propose TWEND, a prototype framework for exploring bending as an interaction design for mobile devices. The system is designed to support different experimental configurations for researching bending gestures. It consists of a hardware device (a bendable block of plastic and foam with embedded sensors) and an extensible software for gesture recognition. A first study demonstrates the applicability of the framework and identifies a set of gestures, which are specifically suited for navigation in e-books.

We then describe an e-book reader application as a refinement of the first prototype. This application illustrates the naturalness of the mapping between bending a device and page turning.

The last section shows how the TWEND project evolved into an interactive exhibition that uses an on-top projection to illustrate how a bendable e-book reader could look like once flexible displays get broadly available.

INTRODUCTION

Over the last decades computers have become smaller, more robust and more powerful. This opens the possibility to enhance people’s daily life with mobile computing devices like personal digital assistants, media players, or cell phones. The interfaces of these devices have to balance the need for good usability and the requirement of not taking up too much space. On cell phones, for example, there is a real estate conflict between the size of the screen and the space the keypad takes. Common solutions are to use context-sensitive soft buttons, combining characters on a single key, or simply limiting the screen size. More recent approaches make use of touch- or multi-touch screens or accelerometers to use the device’s orientation for input. Both ideas can be found for example in the iPhone.

This work explores bending a device as a possible form of input. Like tilting, bending does not require the users to perform any Fitt’s Law tasks using their fingers, but instead uses an additional part of the user’s dexterous system – the wrists.

In many cases, using a mobile device means to hold it in at least one hand and then providing input with a separate motion or movement, sometimes even using the second hand. Bending, in contrast, can free the user from the need to change the grip on the device. If the device is held with both hands in the first place, most movements to deform it are done using the wrists, which are usually not used in traditional interaction techniques.

To start exploring this new way of interaction paradigm, we propose a prototype framework that offers possibilities to experiment with different ways to bend a mobile device.

RELATED WORK

Schwesig et. al. [4, 5] constructed a prototype for a mobile device which is bendable along the horizontal axis, has a mounted non-flexible screen and a touchpad on the back. They investigated how continuous bending input could be used in menus and map navigation. They found out that a gradual transition from one data view to another, for example from a map to a detailed display of information about a certain location, works very well with the analog, continuous input of a bending gesture.

Another project proposing a bendable e-book reader was introduced last year by Watanabe et. al. [7]. Their “Bookisheet” is an interface for browsing content that uses the metaphor of a book. In fact, the approach and used gestures are very similar to the ones described in this work although no collaboration was done. We believe this further supports the results we have found with TWEND.

A different approach was taken by Scott et. al [6] who developed a mobile device that is rigid but measures the forces applied to it as if a user was trying to bend it. Part of their software then visually responded to this input with e.g., windows that bend and deform accordingly.

Balakrishnan et. al. [1] introduced a device called “ShapeTape” which is a long, cable-like, bendable device. Deformations applied to it are directly mapped to a virtual representation of the device, primarily used for 3D modeling. The device is commercially available.

In the work of Harrison et. al. [2] the authors introduce the idea of navigating between digital pages by using touch sensors attached to the corners of a rigid device. Their idea was that stroking the corner of the device closely mimicked the way people skip to the next page in a real world book. TWEND further develops this gesture by including flexibility.

Technically relevant studies on bending sensors were done by Kuang et. al. [3]. They evaluated the kind of optical bending sensors used in the TWEND hardware device and provided a basic description how to construct these sensors by hand.

IMPLEMENTATION

The TWEND prototype framework consists of a hardware device and a software measuring the different bending states of this device and abstracting them into bending events. Because the general idea was to provide a means of conducting more than one experiment, hardware and software were designed to recognize several bending gestures and offer an easy way to create different applications that react to these gestures. This lead to the decision not to create a compact, fully mobile device but to use a connected computer for the processing. Two advantages of this are the larger processing power and being able to use desktop development tools for the creation of applications.

Hardware

The TWEND device itself consists of two layers of plastic with a layer of foam in between. It is packaged in a cloth pocket. Eight optical bending sensors (as presented in [3]) are embedded in the foam body. We used these sensors instead of standard strain gauges because of their greater robustness, their linearity, shorter response time and their ability to measure bidirectional bending. The values provided by the sensors are streamed to the computer by an Arduino microcontroller board.

The TWEND device is approximately 15 cm by 25 cm (6’’ x 10’’) large and 1.3 cm (ca. 0.5’’) thick. The device is shown in Figure 1, a schematic showing the sensor layout is depicted in Figure 2. We chose this layout in accordance with the set of 18 gestures that we designed the device to recognize (see section below).


Figure 1. Shot of the TWEND device in use (wave-like bending).


We decided on a rectangular shape to make the device similar in appearance to the majority of mobile devices today.

Figure 2. Schematic of the TWEND: Layers (left) and sensor layout (right).

The size was a compromise between a real mobile device and the need to build the TWEND device by hand and embed all sensors within it.

The sensors themselves were also handcrafted and each consist of a photocell, fibre cable, an LED, and some resistors and plastic casings. A schematic of one sensor can be seen in Figure 3. Their working principle is quite simple: Light is sent through the fibre cable to the photocell and the resulting resistance is measured. The fibre cable itself has a large abraded section where bending is supposed to be measured. Depending on how far the cable is bent, more or less light is lost at that abraded section, which results in a linear change of the measured resistance. This works for bending the abraded section in a convex and a concave direction, i.e. bending forward and backward is measured.

Software

Figure 3. Circuit for one sensor

The software is a Mac OS X command line tool. The gesture recognition algorithm is straightforward. It uses a k-means to cluster the sensor input in an initial calibration phase into 18 clusters. Each cluster represents a different bending gesture.

When a gesture is recognized, the program converts the bending gesture into a native operating system event according to a configuration file. These can be simple key presses, scroll wheel events, mouse movements or AppleScript events. The source code was designed to be easily extensible with custom events. This way, the TWEND device can be used to control standard applications on Mac OS X that are usually controlled by the mouse and the keyboard. The purpose to enable TWEND to control normal applications is to be able to quickly test ideas for new interaction methods before starting development of a dedicated (mobile) application from scratch.

The software can also use Message Ports to send the raw bending events (gesture ID and degree of bending) to custom applications. This allows for more detailed experiments.

Gestures

A selection of the 18 bending gestures that can be recognized by TWEND can be seen in Figure 4.

The full set of recognized gestures consist of:

•Bending the corners of the device to the back or front (8 gestures, example see Figure 4 A)

•Bending the TWEND along the horizontal, vertical or diagonal axes (8 gestures, example see Figure 4 B)

•Bending it into a wave-like shape along the horizontal axis (2 gestures example see Figure 4 C)

The decision on these gestures was made based on the TWEND device’s form factors described above. Other ways of bending a device are possible, but since there is an “absence of established paradigms for interaction by deformation” [5], we wanted to focus on a manageable set of gestures.

Figure 4. Sketch of three of the bending gestures: bending a corner (A), bending along the horizontal axis (B) and wave-like bending (C).


EXPERIMENT

In our first user study, we wanted to verify the applicability of the prototype framework. At the same time the experiment evaluated the utility of different gestures for navigating a list – a task found in many mobile devices.
We assumed that bending the upper corners forward could be well associated with navigating through a list item-by-item and the two wave-like bending gestures with rapidly scrolling through it. This idea was inspired by the TWEND device’s shape resembling a book.

Figure 5. A person thumbing through a book, bending it in a wave-like shape to do so.

A book page is usually turned by grabbing its corner and then moving it to the other side of the book, in the process bending the page. Deforming a real book into a wave is seen when people rapidly thumb through the pages to get the necessary strain to make them flip, Figure 5 shows this.

Setup

The prototype framework was configured to generate scroll wheel events for the wave-like bending gestures and bending the left and right upper corners. As visual feedback we displayed a list of items in the Mac OS X Finder’s cover flow view. Users were given the task to navigate through the list and look for a marked item while the experimenter observed them. Before the experiment the users were asked to think-aloud while trying out their assumptions on which bending gesture would navigate item-by-item through the list and which would equal a scrolling with the mouse wheel. The experimenter gave hints only when it seemed they would not discover all four gestures after a few tries and take notes on their progress.

Participants

13 people took part in the experiment, four women and nine men. Ten participants were between 20 and 29 years old, the other three were in the age groups 30 to 39, 40 to 49 and 60 to 69. The two oldest users did not work at our chair and the youngest age group completely consisted of students. All but the participant between 40 and 49 worked with a computer on a daily basis.

Results

Only one user did not manage to find the corner gestures in a couple of seconds. All others discovered them very quickly, usually instantly trying out the opposite corner after finding what bending one of the upper two corners did. One of these 12 users mentioned to have expected the lower and not the upper corners to work that way. Only three participants discovered the wave-like gestures without help of the instructor. Five of the others said they expected a complete bending along the horizontal axis to scroll through the list of items, convex bending equaling a forward scroll and vice versa. The rest of the users got the same idea when the instructor hinted at the book metaphor. Nobody got the idea of the wave-like gestures with that hint only. When explaining the metaphor in detail, all users understood the mapping.

Conclusions

The results of the experiment suggested that using the bending of corners for item-by-item navigation through a list is an intuitive, natural mapping. It might, however, also be the case that the gesture is just so simple that it is among the first ones users try out with a new device they know is meant to be bent.

The wave-like gesture showed to be less intuitive in the experiment. A reason might be the stiffness of the prototype, as several users commented after the experiment. Indeed, the users’ comments indicated that they are not consciously aware of how they deform real books when thumbing through them so that the hint alone to think about this did not give them the idea to try out a wave-like bending gesture. After having explained our rationale most participants realized to sometimes bend a book or magazine into a waveform to get the necessary strain to make the pages turn when thumbing through it. Most books, they pointed out, are not too wide so this is not necessary and they can be thumbed through by bending them just along the horizontal axis. Because of this, the first idea most users got when being confronted with the book metaphor was exactly this gesture. They were not aware of an alternative. Since the TWEND can perform a complete horizontal bending as well, we think this result provides useful knowledge about bending gestures as well.

TWEND E-BOOK READER

The results of the experiment motivated the development of an e-book reader. Such devices usually are comparable in size to personal digital assistants and just like these have a rigid casing with buttons. One recent example is the Amazon kindle. Apart from its landscape orientation, the TWEND device looks like a book and is just as bendable as one.

The current implementation of our TWEND e-book reader is a Mac OS X application showing a virtual 3D model of a real book.

Figure 6. Bending the TWEND into wave-like form results in “thumbing through” the e-book (here: backwards).

Using the gestures determined in the experiment plus the horizontal bending suggested by the users, the TWEND device can be used to turn pages. Figures 6 and 7 illustrate how the e-book application reacts to two of the gestures. The reaction to the gesture not shown, the complete horizontal bending (see Figure 4 B) is the same as to the wave-like bending (Figure 6).
Bending the upper right corner (Figure 4 A) lifts the right page currently visible in the reader. The more the corner is bent, the further the page lifts until a threshold is reached and the page completely turns. Consecutive corner bending turns several pages, one page per gesture.
The wave-like bending gesture or alternatively the complete horizontal bending (Figure 4 B) is used to thumb through the book. The animation for this operation consists of several pages fanning up – so more than two or three pages are visible at the same time – and then rapidly turning over. The harder the TWEND is bent, the faster the pages turn and the more pages are fanned up.

Figure 7. Lifting in the e-book by bending the upper left corner of the TWEND.  Stronger bending would finally reach the threshold and make the page turn.


First comments of people informally testing the TWEND e-book reader were very positive. Users who had also taken part in the experiment said that with a visual representation directly mapping the bending state of the device the chosen gestures were even more natural to them.

FUTURE WORK

The next step is to further improve the TWEND e-book reader.
A comprehensive comparison with existing, rigid e-book readers requires the system to be a lot more polished than it currently is. Also, our current setup with a separate display attached to the flexible hardware brings up interesting questions. So far, we have no knowledge about how the deformation of a device-mounted flexible screen changes the user interface.

Figure 8. The redesign of the hardware device.


To address these issues, we rebuilt our hardware from scratch. The new TWEND device currently under construction will optimally support the gestures we identified as suited for e-book navigation. It has a slightly different sensor layout and looks more like a book as can be seen in Figure 8.

Since organic flexible LED displays are still not broadly available, we constructed an aluminum frame around the device that holds a projector for displaying the book pages directly on the device.
As a result the device is fixed right now, but we are confident that we will find out how a shape changing display influences user perception nevertheless. The mounted device with an image projected on it can be seen in Figure 9.

Figure 9. The redesigned hardware with projection on.


The software is also in the process of being improved. From now on, we will use a PCA to train the device for the bending gestures and then recognize them.

Because the whole new setup is completely standalone we will not only conduct further user studies with it, but also use it as a demonstrator for the interaction principle of bending. In fact, the project has been accepted in an interactive exhibition onboard the MS Wissenschaft, a museum ship cruising through Germany from June until October 2009.

References

  1. 1.R. Balakrishnan, G. Fitzmaurice, G. Kurtenbach, and K. Singh. Exploring interactive curve and surface manipulation using a bend and twist sensitive input strip. In Proc. I3D 1999, pages 111–118, New York, NY, USA, 1999. ACM.

  2. 2.B. L. Harrison, K. P. Fishkin, A. Gujar, C. Mochon, and R. Want. Squeeze me, hold me, tilt me! an exploration of manipulative user interfaces. In Proc. CHI 1998, pages 17–24, New York, NY, USA, 1998. ACM Press/Addison-Wesley Publishing Co.

  3. 3.K. S. C. Kuang, W. J. Cantwell, and P. Scully. An evaluation of a novel plastic optical fibre sensor for axial strain and bend measurements. Measurement Science and Technology, 13:1523–1534(12), 2002.

  4. 4.C. Schwesig, I. Poupyrev, and E. Mori. Gummi: user interface for deformable computers. In Ext. Abstracts CHI 2003, pages 954–955, New York, NY, USA, 2003. ACM.

  5. 5.C. Schwesig, I. Poupyrev, and E. Mori. Gummi: a bendable computer. In Proc. CHI 2004, pages 263–270, New York, NY, USA, 2004. ACM.

  6. 6.J. Scott, L. M. Brown, and M. Molloy. I sense a disturbance in the force: Mobile device interaction with force sensing, May 2008.

  7. 7.J. Watanabe, A. Mochizuki, and Y. Horry. Bookisheet: bendable device for browsing content using the metaphor of leafing through the pages. In Proc. UbiComp 2008, pages 360–369, New York, NY, USA, 2008. ACM.