ACM / Crossroads / Xrds13-2 / 

From Chasing Dots to Reading Minds: The Past, Present, and Future of Video Game Interaction

by Damien Marshall, Tomas Ward, and Séamus McLoone

Introduction

It is the calm before the storm. By the end of 2006, Sony, Microsoft, and Nintendo will have released their new wave of gaming hardware, and the next round in the great video game battle will have begun. Capable of displaying photo-realistic images, and acting as the center of your entertainment lifestyle, these machines promise to change the face of gaming. Console games have moved away from the single-screen experiences of old, to multimillion dollar epics, featuring hours and hours of cinematic action. Truly, it is an exciting time to be a gamer.

While visual and audio technology advances toward real-world fidelity, human computer interaction (HCI)—the methods by which users control the simulation—has not received the same degree of attention. But it now seems this aspect of the sense-of-presence problem may undergo a revolution similar to that of its audiovisual counterparts with the next generation of gaming devices. In this article, we discuss the driving forces behind these changes, several devices, and what current research suggests the future may hold for today's gamer.

Background

Many now compare the games industry with its big brother in Hollywood. In the case of a game like Halo 2 for the Microsoft Xbox, shown in Figure 1(a), such comparisons would be justified. Costing approximately $30 million to bring to market, it grossed an impressive $125 million on its first day of release.

But Halo 2 is an exception. The success of its prequel means that it had a ready-made fan base, and it was one of the most highly anticipated titles in recent times. In reality, game development is a very risky venture, with only a very small percentage of titles ever reaching this level of success. A great game does not guarantee stellar sales. Ico, pictured in Figure 1(b), is critically considered one of the finest recently released games, yet it did not perform well at retail. In 2000, 55% of sales in the UK could be attributed to 3.3% of the games released [9]. Due to this, game developers tend to produce video games that conform to standards from popular genres. In 2004, according to the Electronic Software Association, 30.1% of all video games sold could be categorized as an action game, the sports genre accounted for 17.8% of sales, and 9.6% of sales came from shooting games [3].

(a).

(b).

Figure 1(a): Games like Halo 2 tend to sell well, but they are designed to fit a specific genre. 1(b): On the other hand, innovative titles such as Ico, although critically well received, can perform poorly at retail.

Examining these figures, one is painfully reminded of the creative stasis that plagued video games during the early 1980s. Due to the strength of the industry, however, there is very little chance of a repeat of the massive financial losses suffered during this period, but there are two key dangers. First, though players are satisfied with the available selection of games, there is a risk of consumer boredom manifesting itself given the lack of diverse content. Second, potential players—those with currently little or no interest in games—are unlikely to have their curiosity piqued by a plethora of cookie-cutter titles.

To counteract this, new initiatives are directly targeting the reasons behind these trends. This article focuses on one such development. By changing the fundamental ways in which people interact with video games, industry leaders hope to broaden the spectrum of gaming experiences and remove the barriers put in place by current standard control schemes. We will begin by examining the history of game controllers.

Playing the Game

The development of game controllers stems from a number of key advances throughout the relatively short history of the games industry. That history can be divided into a series of generations, each of which lasts approximately 5 years. Each generation begins when a new series of hardware, typically more powerful than its predecessors, is released. A breakdown of game controllers since 1975 is shown in Figure 2 [7].

Figure 2: A time line showing the evolution of game controllers over the past thirty years.

The late seventies and early eighties were a golden time for video games. At its peak, the industry was worth approximately $3 billion in America alone. The two standard input schemes employed at this time were the button and joystick, both of which were featured on the Atari 2600 console. A joystick is a device consisting of a handheld stick that pivots about one end and transmits its angle to the games console. This control scheme proved popular as it suited the dynamics of simple games of the time, such as Pac-Man and Space Invaders, which featured simple two-dimensional single-screen game worlds.

Though many remember these classic titles, these early systems were also home to a slew of shoddy software. Hardware manufacturers of the time did not exercise control over the content that appeared on their systems. They could do nothing to stop the release of underdeveloped content from those eager to cash in on the success of the market. Around 1983, consumers began to tire of these inadequate games and turned instead to relatively inexpensive game-playing personal computers, such as the Commodore C64. The video game market crashed. By 1985, it was worth only $100 million annually worldwide [8].

Soon after, the Nintendo Entertainment System (NES) was launched. Nintendo felt that video game interaction mechanisms had become too complicated for consumers and that the placement and number of buttons were awkward for many players. Subsequently they designed the input device for the NES specifically to resolve these issues in two ways. The number of buttons was significantly reduced to two main action buttons, and the traditional joystick was replaced with a "D-pad" (directional pad). The D-pad allowed the player to control the game using only small movements of their thumb. The thumb has the same degree of freedom as the wrist, meaning that the user can exert the same level of control as they would with a joystick, but it does not tire as easily. The approach worked, and Nintendo quickly dominated the market.

The next generation of consoles, led by the Sega Genesis and the Super Nintendo Entertainment System (SNES), did not result in any major interface innovations other than the introduction of shoulder buttons on the top edge of the SNES controller. Additional buttons also made a reappearance on the face of the controller. In this generation, ergonomics and comfort were considered, replacing the traditional square-cornered design of the NES device with rounded casing. Controller enhancements such as "turbo" were popularized at this time. Turbo removed the need for so-called "button-mashing" required from many titles of this era by sending a rapid stream of button press registers to the system while holding the button down. Previously, each button press registered once; turbo emulated multiple buttons presses.

The fifth generation began in 1995 with the launch of the Sega Saturn, Sony PlayStation, and Nintendo 64 (N64), heralding the arrival of the polygonal age. Cheap dedicated graphics hardware and compact discs (only in the PlayStation and Saturn) provided both the speed and the space to make large three-dimensional worlds both financially and computationally feasible. To cope with the extra functionality required to play the new breed of games, the number of buttons was again increased.

The D-pad, designed to navigate two-dimensional spaces, became unsuitable for the challenges presented by the extra dimension. To remedy this, Nintendo integrated the "analog stick" into the controller for the N64. The analog stick provided players with an unprecedented degree of control in 3D worlds by allowing 360 degrees of control. Players could also vary the speed of their navigation, for example, by varying the pressure applied to the stick [6].

Moving Away from the Game Controller

Current game consoles, namely, the Microsoft Xbox, Sony PlayStation 2, and Nintendo GameCube, have all continued the "more is better" trend of controller design. Game controllers now have up to ten separate buttons, sometimes with confusing labels, and they have three different directional navigation choices that often must be managed in coordination with the buttons.

These control schemes place barriers between the player and the game. For the casual gamer, modern controllers can be very intimidating. Not only must these players learn the intricacies of the game, they also must achieve the dexterity required to use one of these controllers. Such designs also place limits on the technical and amusement possibilities of current video games.

As a result, novel control schemes are now becoming more prevalent. Rather than forcing the player to stoop to the consoles' slow and unwieldy input schemes, these new interfaces allow the player to play in a more natural and intuitive manner: the swing of an arm replaces the push of a joystick, the single spoken command replaces four synchronous button presses. By closing the gap between player and game, these interfaces also bridge the demographic gap between players. Players young and old can compete on a level playing field, regardless of gaming experience.

Most importantly, these new control schemes positively impact the player's experience. As well as increasing the number of innovative game experiences, they increase the player's sense of presence in the virtual world. Rather than pressing a button, a player is usually physically gripping the same tool their in-game hero is using. For example, Guitar Hero from Harmonix comes with a reduced-size replica of a Gibson guitar. Examples of some of these recent non-traditional input schemes are shown in Figure 3.

Figure 3: Current generations have seen an increase in the number of innovative control schemes.

Perhaps the most popular use of these new interfaces has been musical performance titles. For example, in Dance Dance Revolution from Konami, the player steps on an arrow on a dance mat to match an on-screen arrow. The backing song governs the timing of these arrows. Real-time voice control is now also possible. In Singstar on the PlayStation 2, for example, the player sings along to a variety of popular music, and their vocal performance is then rated.

Movement tracking interfaces attempt to remove the controller altogether. Players use natural body movement to control the action on the screen. The EyeToy for the PlayStation 2 has performed admirably in this field. It operates by employing a low-resolution camera that connects via a USB port to the PlayStation 2. Players can then use body movement to carry out a variety of tasks such as scrubbing windows, engaging in a boxing match, or swatting virtual flies. The In2Games GameTrak system operates by attaching two cords to the hands of the player. The movement of the hands is then tracked for punching in the game Dark Wind, for example, or for swinging a golf club in Real World Golf.

Although these schemes allow the player to interact in a more natural way with the game environment, their appeal is limited by the fact that they are not the standard input schemes for consoles. Developers are stuck in a catch-22 situation. Games will not be produced to support a nonstandard input device until it has achieved significant sales, and the device will not usually sell until it has a significant number of games. The result is that these devices are typically seen as novelty items and are only supported by a handful of games made by the creators of the device.

The most recent hardware from Nintendo looks to buck this trend. Launched in late 2004, the Nintendo DS, shown in Figure 4, features two screens (one of which is touch sensitive) and a microphone. Originally, it was perceived to be a small, experimental diversion from the juggernaut that is the Game Boy brand. It has since gone on to become the fastest selling gaming device in Japan. In Europe and America, it has matched the performance of its closest competitor, the Sony PSP, a more traditional device with more power than the DS.

Figure 4: The DS has redefined interactivity and has enjoyed much commercial success.

The success of the DS has been attributed to the new gaming experiences provided by its unusual input scheme and the intuitiveness of control provided by the touch screen. For example, Trauma Centre: Under the Knife places the player in the shoes of a surgeon, performing life-saving operations on the touch screen. The most commercially successful of DS games have been the so-called "touch generation" games, such as Brain Training. These games allow the user to play using voice and handwriting only, and they usually incorporate some degree of educational value. They have attracted players young and old, and nearly all have sold in excess of a million copies. Future releases on the DS will continue this trend, with player activities including painting, cooking, and dating to name a few.

Future Interaction Mechanisms

The commercial success of these nontraditional input schemes in the current generation bodes well for their future development in the next generation. Due to the secretive nature of the games industry, it is difficult to ascertain what exciting developments innovators may have up their sleeves. Instead, we must turn, in the main, to the academic arena in order to catch a glimpse of exciting interaction schemes that may govern how we play games in the future.

We begin with somewhat more traditional, but by no means less innovative, work on control pads, such as that at New York University. Work here has focused on extending the behavior of the regular touch-sensitive screen. The Multi-Touch Interaction Surface can detect multiple contact points on the screen, in comparison to current touch screens that can detect only one touch at any one time [5]. One could envisage the development of a "blank slate" game controller, consisting solely of a touch screen. The controller could then be configured to suit the task at hand. At one moment it could act as a regular game controller, and the next it could act as a virtual piano.

Other work examines how the physical device can be removed completely. Hand- and head-tracking have been extensively developed in the academic arena. They are employed extensively in immersive virtual environments, allowing a user completely surrounded by the virtual world to simply reach out and grab a virtual item [10]. This rich heritage of research is the driving force behind both Nintendo's and Sony's entries in the next-generation console wars.

Figure 5: The Nintendo Wii controller can detect its position in 3D space.

Nintendo hopes that their next console, the "Wii" (pronounced "we"), will change the face of gaming in the same manner as the D-pad did on the NES. The standard controller, shown in Figure 5, looks unassuming enough, featuring two main buttons shaped like a standard television remote control. Its power lies in the fact that both the orientation and position of the controller can be detected in 3D space. Sony's controller for their next generation console, the PlayStation 3, also features sensors that allow the console to track both rotational orientation and acceleration of the controller, but these were recent additions to the design, not a fundamental feature of the system.

This trend of existing academic discoveries inspiring future interface developments is not new. The technology behind the Sony EyeToy has existed for quite some time in the mature field of computer vision, but only recently has it been applied to video games. For example, the EyeToy (at the moment) is limited in that it can only detect changes in what the camera sees, not full real-time tracking. Sony demonstrated what the future of this style of interface holds at a recent trade event: A high-resolution camera, combined with next-generation technology, was shown to support an interface similar to that featured in the futuristic film "Minority Report." The user employs markers attached to their fingers to manipulate images, browse documents, and navigate files. In the PlayStation 3 game Eyedentify, shown in Figure 6, the EyeToy is employed as an integral part of the game. This should sate critics who bemoan the lack of depth found in current EyeToy games.

Figure 6: Eyedentify uses the EyeToy to greater effect in a next-generation game.

The field of Augmented Reality (AR) reverses the camera by superimposing the virtual world onto the user's real world. AR has been employed for years in the military domain. In a fighter jet, for example, a Heads Up Display provides extra tactical information for the pilot. As AR systems can be expensive, they have yet to proliferate in the games market. A few examples of the potential of such systems do, however, exist. ARQuake is an extension of the very popular Quake game from ID Software [11]. Here the player navigates the game world by walking around the real world while wearing a headset connected to a laptop, GPS sensor, and customized toy gun. The enemies within the world are superimposed onto the real world. Bruce Thomas, who initiated the ARQuake project, has since gone on to release a commercial product known as A_RAGE (Augmented Reality Gaming Engine) [1].

An interesting branch of research examines a more subtle level of human communication. Brain Computer Interfaces (BCI) take human computer interaction to an even higher level by translating physiological signals from the brain into a control signal for an external device [2]. Recently a group in Korea demonstrated a game in which the user navigated a maze through imagined hand and leg movement based on brain imaging methods [12]. Although used primarily for medical rehabilitation, a BCI could present some exciting opportunities for future video games. Consider the Nintendo 64 game Tetris 64. The game featured a feedback device that could detect heart rate, which was used as a crude measure of excitement. Consequently, adjustments to game play, speed, and control could be made in response to the player's enjoyment and involvement.

An issue with schemes such as the ones just described is that they deal specifically with a single aspect of interaction, often to the detriment of others. Multimodal interaction mechanisms may well deal with this problem. Here, a number of different interaction modalities are combined to form one interface. In essence, such schemes attempt to capture "the best of all worlds" by combining the benefits of all different input schemes, rather than limiting the player to one.

One example of such development is work being undertaken at the National University of Ireland, Maynooth where a Yoga game experience is being developed [4]. Here, full-body motion-tracking is harnessed via stereo cameras (for depth sensing of body parts) and wireless motion sensors. These are combined with physiological signals such as heart rate, stress response, and breathing rate to produce a hybrid control signal that drives the action in a virtual world. A pattern recognition system is used to determine the closeness between the player's behavior and that of an expert yoga practitioner. Through constant feedback, the player can alter their behavior to improve this closeness of fit, allowing them to progress through the game. This multimodal approach is perhaps what will best model how we interact with the regular world. As more and more natural modes of human computer interaction are tapped into and harnessed, it is their combination that will help fill in the missing parts of our sense of presence.

Conclusion

By the end of 2006, the next generation of game consoles should be available to consumers. Video games found on this new hardware have the potential to deliver unrivaled entertainment experiences to a massive variety of players.

This potential is being threatened by the current condition of the games industry. Spiraling development costs and traditional input mechanisms mean that games must be tailored to fit certain principles or run the risk of being a financial disaster. The result is that store shelves are inundated with sequels and licensed products that appeal only to current gamers and do little to attract outside interest. This is not the way to grow a market.

There is light at the end of the tunnel. Industry leaders are beginning to realize that fundamental changes are required in the way people play games. This approach is helping target those who traditionally ignore games, as well as create new means of play for gamers bored with the current selection of games.

Never one to rest on its laurels, the video game industry has seized the idea and begun to develop it to its full potential. In fact, the next-generation offering from Nintendo is built around the foundation of a single nontraditional input mechanism. Knowing the spotty track record of nontraditional interfacing methods, Nintendo is planning a traditional control device as well.

Work in academia shows the future prospects for input mechanisms. They range from more traditional work, such as developing the scale of touch-sensitive screens, to Brain Computer Interfaces, which interpret signals coming directly from the brain. In an industry notorious for continually confounding expectations, nothing is certain. Indeed, we could all be using something beyond our current conception within a few short years. What is guaranteed, of course, is that we all will have a fun time finding out.

Acknowledgments

Thanks to Dr. Aphra Kerr for her help in researching the background to this paper. This work is supported by Science Foundation Ireland and Enterprise Ireland under grant IRCSET/SC/04/CS0289.

References

1

A_RAGE, "A_RAGE: Augmented Reality Gaming Engine." 19 July 2006, http://www.a-rage.com/ (19 July 2006).

2

Coyle, S., T. Ward, and C. Markham, Brain-Computer Interfaces: A Review. Interdisciplinary Science Reviews, 2003

3

Electronic Software Association, Essential Facts about the Computer and Video Game Industry. 2005 http://www.theesa.com/files/2005EssentialFacts.pdf (19 July 2006)

4

Foody et al., A Prototype Sourceless Kinematic-Feedback Based Video Game for Movement Based Exercise 2006. 28th Annual International Conference of the IEEE Engineering in Medicine and Biology Society: Engineering Revolution In BioMedicine Marriott Marquis at Times Square, New York City, New York, USA, August 31-Sept 3, 2006

5

Han, J, Low-Cost Multi-Touch Sensing through Frustrated Total Internal Reflection. In Proceedings of the 18th Annual ACM Symposium on User Interface Software and Technology, 2005

6

Hinckley, K, Input Technologies and Techniques. To appear in Handbook of Human-Computer Interaction. 2006. Lawrence Erlbaum & Associates

7

Koushik, S, Evolution of Controllers. 2006 http://wii.advancedmn.com/article.php?artid=6355 (19 July 2006)

8

Miller, M, "A History of Home Video Game Consoles." 19 July 2006, http://www.informit.com/articles/article.asp?p=378141&seqNum=3 (19 July 2006).

9

Spectrum Strategy Consultants, From Exuberant Youth to Sustainable Maturity. Competitiveness Analysis of the UK Games Software Sector, 2005 http://www.dti.gov.uk/files/file10663.pdf (19 July 2006)

10

Steed et al., Interaction between Users of Immersion Projection Technology Systems. In 2005.HCI, Las Vegas.

11

Thomas et al., ARQuake: An Outdoor/Indoor Augmented Reality First Person Application. In 4th International Symposium on Wearable Computers, Atlanta, Ga., 2000

12

Yoo et al., Brain-Computer Interface Using fMRI: Spatial Navigation by Thoughts. Neuroreport. 2004 Jul 19;15(10):1591-5

Biography

Damien Marshall graduated with a first class BSc degree (Computer Science and Software Engineering) in 2003. He then joined the Distributed Interactive Applications Group and completed his MSc by research in January 2005. Currently, he is working toward his PhD. His work examines the various factors of consistency in Distributed Interactive Applications.

Tomas Ward is a member of the Distributed Interactive Applications Group at the NUI, Maynooth. His main interests are in assistive technology and data management techniques for distributed shared applications. Spare time is spent on extending his personal best score on Konami's DrumMania V.

Dr. Séamus McLoone graduated with a first class BEng honors degree in Electrical and Electronic Engineering from Queen's University Belfast in 1996. He subsequently joined the Intelligent Systems and Control Group at Queen's and obtained his PhD in September 2000. He is currently employed as a lecturer in the Department of Electronic Engineering in the National University of Ireland, Maynooth. His research interests vary from multiple model approaches, to nonlinear system identification, to reducing and masking the effects of latency in distributed interactive applications.