This article discusses the main differences between multimedia applications and more traditional systems with respect to usability evaluation. Future directions for developing multimedia evaluation methods are then outlined.
FIGURE 1
As one step towards understanding how different media can affect the subjective feeling experienced by the users, ''media biases'' have been used to describe how people experience distinct media types differently [17]. As an example, in one experiment where video was used to present information, the users' reaction was that they felt engaged but they questioned the validity of the information. In contrast, an equivalent textual presentation was seen as more reliable [17].
The lack of conventions in multimedia applications is a potential problem for the field [21]. Because multimedia allows completely new media combinations, there are no genre conventions to ease the perceptual jumps between e.g. colour video and text in black and white. The lack of conventions also prevents users from decoding perceptual cues as in well-known genres like horror movies, where a certain type of music certainly helps the users (the audience) form a mental model of the nature of events to come.
While the above discussion covers some high level issues regarding the usability of multimedia, several researchers have pointed to lower level issues that are vital to consider in the evaluation of multimedia applications.
However, an inherent problem with hypermedia is ``media ghettos''. The inter-linked nature of hypermedia requires that information be divided up into "nodes." This new dynamic medium represents a challenge in identifying nodes that should be linked to other nodes. For instance, how should a node be indicated in a video segment or in a piece of music? This problem has two consequences. First, users may have problems in identifying links in non-textual media, and thus have problems navigating in hypermedia applications. Second, there is a risk that non-textual media become dead ends because of problems in constructing links [11]. In other words, media ghettos will appear [17]. Even if it is technically feasible to provide links in non-textual media, a lot of decisions need to be made. Take video as an example. Should the whole signal be made a link, and what will that tell the user about where the link leads to? Or alternatively, should objects within the video be nodes (this would require some advanced picture processing and pattern recognition)? Further, how should it be indicated what are nodes within the video without interfering too much with the video-presentation itself [22]?
Searching is a widespread means of navigating in hypermedia. However, searching is also impeded by problems referring to non-textual media in general, and time-based media in particular. For instance, how would a user search for a graphical representation of a motorcycle? Or even worse, what if the user is looking for a specific part of a musical play, how should that be indicated?
A lack of control of dynamic media may be a further obstacle for successful use of multimedia applications. Dynamic media raise new demands in terms of controlling dynamic media, e.g. play, stop, rewind, adjust volume, indication of time-duration of media piece and so forth.
In short, there are lots of obstacles in creating useful multimedia applications. Many aspects, such as navigation, need to be considered when evaluating the usability of such systems.
Media illiterate developers of multimedia interfaces are a source of usability problems in multimedia interfaces [1]. As an example of media illiteracy, it has been argued that producing a video sequence for integration in a multimedia interface without knowing anything about the language of video (e.g. a cut between shots indicates a new perspective), is like trying to read before learning the alphabet [1]. Thus the composition and construction of the individual media may cause usability problems in multimedia applications. Finally, there are potential usability problems at the lowest level of the framework presented in figure 1.
As can be seen from the discussion above, if the strengths of new media are used appropriately, multimedia applications can be easier to use, more engaging and fun, easing navigation and support tasks that could not otherwise be supported by computer systems. However, if multimedia systems are created by people with no knowledge about the language of the new media, if human stimulus-response capabilities are not considered, or if no further work is put into answering some of the questions raised by researchers that are outlined in figure 1, multimedia systems may become sparkling, twinkling, colourful and expensive creatures of very limited use. One way to push multimedia systems in the right direction is through evaluations that consider the aspects outlined above.
It should be acknowledged, though, that the problems considered above are in many cases weakly defined events, of which the consequences for usability can be merely predicted. Nevertheless, one can uncover issues that require further research in multimedia evaluation as well as better assess existing multimedia evaluation methods. These will provide the direction for future multimedia evaluation approaches.
In order to address these issues, a few multimedia evaluation methods have thus far been developed. They include some multimedia guidelines [2, 3, 8, 10], as well as some analytic methods [6, 15] and finally, one experimental approach [13]. Most of these approaches consider mainly low level aspects such as information overload or presentation methods. Few assess the means of navigation and interaction. Furthermore, none of the existing multimedia evaluation methods consider how high level issues such as media biases, aesthetics, pleasure and engagement can be assessed. Neither do they point out the importance of these aspects in assessing the usability of multimedia applications. Further, none of the special multimedia evaluation methods consider the problems of navigation in non-textual media, encompassing media ghettos, potential problems in identifying links, nor do they assess the users' control of dynamic media. There seems to be a problem in the evaluation of the higher level issues in figure 1. In the future, perhaps multimedia evaluation methods will target some more of these issues. However, there are a few difficulties to be considered.
On the other hand, for media allocation and media combination, which are factors in the middle levels of the framework, it seems less likely that general guidelines can be made. More commonly, the decisions on media allocation and combination are considerably more complex. As was discussed, different views abound on which factors need consideration. Further obstacles to media allocation are caused by aesthetic considerations as well as media biases, which may influence the choice of media allocation and combination.
With respect to top level issues, it seems even harder to generalize these issues. First of all, it seems very hard to generalize issues such as aesthetic requirements, since they are likely to differ according to personal taste and cultural differences. Further, it is hard to identify when aesthetics should be taken into account in preference to other considerations. In any case, the higher the level of abstraction, the harder it is to generalize usability criteria, and thus the harder it is to evaluate usability factors.
It is thus not surprising that in existing evaluation approaches mostly lower level issues are assessed, since the higher level issues are hard to evaluate. Furthermore, only a few of the usability factors in figure 1 can reasonably be measured quantitatively. It seems that the higher the level, the harder it gets to provide a quantitative measure of usability.
The low level issue of information overload may in some cases be measured quantitatively. Information overload could be measured in terms of the number of video-signals presented simultaneously. However, benchmarks would need to be determined based on the task in question. For example, if the user is expected to listen to speech in two simultaneous video signals, this may cause a problem. On the other hand, if the video signals are used to support some kind of general overview, probably more than two video signals can be attended to at a time. Further, it is possible that some kind of quantitative measurements can be provided in relation to users problems in identifying links. The number of fruitless attempts to click on objects that the user expects to be nodes could provide some kind of measurements of the ease of navigation in non-textual media and thus the usability of the application. Furthermore, media ghettos could be assessed in terms of the number of cross media links, i.e. links between nodes of different media types. Increased cross-media links will then indicate a smaller tendencies toward media ghettos. Apart from these few cases, it is hard to see how the remaining usability factors in figure 1 could be evaluated quantitatively in a reasonable way. Aesthetics can hardly be measured quantitatively. Thus evaluation methods that produce quantitative usability measures do not seem to be the solution to the evaluation of multimedia applications.
Finally, it is remarkable how few of the existing multimedia evaluation methods involve users. It has often been argued that involving users in evaluations is the only way to examine unexpected ways of interaction, which may have serious implications in assessing the usability of the applications [4, 5, 12]. The vast majority of existing multimedia evaluation methods developed thus far consist of sets of guidelines. These first of all assume the generalizability of usability factors and, secondly, fail to reveal how real users will experience and interact with a given design. Thus it cannot be expected that multimedia guidelines can be sufficient for the evaluation of multimedia applications.
The challenge in developing future multimedia applications, then, is to ensure that they are not only visually pleasing but also usable. This will require a better understanding of many of the usability critical issues that are outlined in figure 1, and an investigation of how users can be more actively involved in the assessment of future multimedia applications.
Want more Crossroads articles about User Interfaces? Go to the index or to the next one or to the previous one.
Copyright 1998 Marianne G. Petersen
Last Modified:
Location: www.acm.org/crossroads/xrds4-4/usability.html