Need for Perceptual Display Hierarchies in Visualization

by Amit Prakash Sawant and Christopher G. Healey

---

Abstract

The advent of computers with high processing power has led to the generation of large, multidimensional collections of data. Visualization lends itself well to the challenge of exploring and analyzing these information spaces by harnessing the strengths of the human visual system. Most visualization techniques are based on the assumption that the display device has sufficient resolution, and that our visual acuity is adequate for completing the analysis tasks. However, this may not be true, particularly for specialized display devices (e.g., PDAs or large-format projection walls).

In this article, we propose to: (1) determine the amount of information a particular display environment can encode; (2) design visualizations that maximize the information they represent relative to this upper-limit; and (3) dynamically update a visualization when the display environment changes to continue to maintain high levels of information content. To our knowledge, there are no visualization systems that do this type of information addition/removal based on perceptual guidelines. However, there are systems that attempt to increase or decrease the amount of information based on some level-of-detail or zooming rules. For example, semantic zooming tags objects with "details" and adds or removes them as the user zooms in and out. Furnas's original fisheye lens system [9] used semantic details to determine how much zoom was necessary to include certain details. Thus, while zooming for detail, you see not only a more detailed graphic representation, but also more text details (e.g., more street names on the zoomed-in portion of a map). Level-of-detail hierarchies have also been used in computer graphics to reduce geometric complexity where full resolution models are unnecessary and can be replaced with low-detail models where the resulting error cannot be easily recognized. Our approach is motivated by all these ideas, but our key contribution is that we use human perception constraints to define when to add or remove information.

Introduction

Visualization is an area of computer graphics that deals with the management and presentation of information in a visual form to facilitate rapid, effective, and meaningful analysis of data. Nowadays, visualization is a path from data to understanding [6]. Formally, a dataset D contains n data elements, e_i, such that D = {e₁,..., e_n}. A dataset represents a set of data attributes, A = {A₁,..., A_n}, m > 1. Data elements encode values for each attribute: e_i = {a_i,1, ...,a_i,m}, a_i,j belongs to A_j. A data-feature mapping converts raw data into visual information. Such a mapping is denoted by M(V, θ), where V = {V₁, ..., V_m} is a set of m visual features with V_j selected to represent each attribute A_j, and θ_j : A_j -> V_j maps the domain of A_j to the range of displayable values in V_j. Thus, visualization is the process of selecting an appropriate M. An effective M produces images that support rapid, accurate, and effortless exploration and analysis [8].

Visualization can be used to represent scientific (e.g., land and satellite weather information, geographic information systems, supernova flow, and molecular biology) as well as abstract data (e.g., movie data, performance data, data mining, and network security).

Examples of Visualization

Figure 1: Visualization of a weather dataset using perceptual texture elements

Figure 1 shows an example of scientific visualization representing weather data made up of monthly environmental and weather conditions provided by the Intergovernmental Panel on Climate Change[6]. Individual weather readings (or data elements) are visualized using stroke glyphs (2D rectangular objects) that vary individual color and texture properties according to the data values they represent. Hue represents temperature: blue strokes for cold temperatures to red strokes for hot temperatures. Density represents wind speed: more strokes displayed in a fixed area of screen space for stronger wind speed. Size represents pressure: larger strokes for higher pressure. Orientation represents precipitation: tilted strokes for heavier rainfall. Finally, luminance represents cloud coverage: brighter strokes for heavier cloud coverage.

Figure 2: Visualization of movie data

Figure 2 shows a query result containing recommendations for movies [7]. We represent the following attributes: movie title, genre of the movie, year the movie was released, length in minutes, and predicted user rating. Each movie recommendation is displayed as a red 3D tower glyph and is positioned along a space-filling spiral, such that high ranking values are positioned near the center of the spiral, while elements with low ranking values are positioned near the periphery. Height represents predicted user rating (short for low ratings to tall for high ratings). A greyscale flag at the bottom of the glyph represents year of release ranging from 1921 to 2006 (dark for older movies to white for the most recent movies). Light brown flags wrapped around the glyph at different heights represent genre (the order of the flags from bottom to top represent Action, Comedy, Drama, and Romance, respectively). Luminance represents the length of the movie (dark red for short to bright red for long). Predicted user rating was also used as the ranking attribute to position glyphs along the spiral.

Perception in Visualization

The knowledge of perception can be used to generate visualizations that harness the strengths of the low-level human visual system and display data in ways that accentuate interesting items to the user. Applying perceptual guidelines to "take full advantage of the available bandwidth of the human visual system" has been cited as an important area of current and future research in visualization [1, 2]. Exactly, how a particular visualization technique and the display device used to present the visualization affect the available visual bandwidth is currently unknown. This article identifies which properties of both the visualization and the display environment must be considered to maximize the amount of visual information a user can perceive.

We hypothesize, the available "visual bandwidth" depends, at least in part, on the following criteria:

1. The physical characteristics of the display device (e.g., resolution in terms of the total number of pixels, and the physical size of the display).
2. The acuity of the human visual system (e.g., the viewer's limitations of distinguishing different visual features such as color, orientation and size, and the visual angle subtended by elements on the viewer's eye).
3. The visualization technique, that is, the methods used to map a data element's values to a visual representation.
4. The properties of the data (e.g., its dimensionality and number of elements), and the analysis tasks to be performed by the viewer.

To date, most research has focused on the last two criteria. Much less work has been conducted on understanding how display resolution and visual acuity influence the effectiveness of a visualization. We propose to study the first two issues in detail and then combine the results with the last two criteria. A better understanding of these properties will help us perform important tasks during visualization such as:

1. Provide fundamental knowledge required to construct perceptually salient visualizations.
2. Verify whether the display device properties and the limitations of the human visual system can meet the requirements of a given visualization technique.
3. Characterize to what extent a visualization technique saturates "visual bandwidth".

Most current algorithms assume sufficient physical display properties are available for generating effective visualizations. However, there are now many different types of displays available to users: some with limited resources (e.g., the small size and low pixel count on PDAs and mobile phones) and some with expensive capabilities (e.g., multi-projector powerwalls, responsive workbenches, and high-resolution monitors). These displays are already having a significant impact on visualization techniques, creating a strong need to learn how to use them effectively. The acuity of the human visual system also introduces constraints on a visualization. Improvements in physical display properties alone cannot necessarily fix these limits. For example, an on-screen element must subtend a minimum visual angle on the viewer's retina to be distinguishable. Increasing a display device's pixel resolution (i.e., increasing pixels-per-inch and therefore possibly decreasing the size of the on-screen elements) beyond a certain limit may produce diminishing results in terms of perceiving different visual features of a data element.

Problem Description

Figure 4: Examples of visualizing with different viewing parameters: (top) a close-up of Hudson Bay, each square represents weather conditions on a 0.5-degrees longitude by 0.5-degrees latitude grid, temperature mapped to hue (blue for cold to red for hot), pressure mapped to luminance (brighter for higher), wind speed mapped to size (larger for stronger), cloud coverage mapped to orientation (more tilted for denser), and precipitation mapped to regularity (more irregular for heavier); (left) North America with all five attributes visualized; (right) North America with only temperature and pressure visualized

Consider a simple example of visualizing a large, multidimensional dataset on a typical CRT monitor, and assume that the viewer has zoomed in on a small subset of the dataset. In this scenario a large number of pixels are available to display each data element. Thus, a fully detailed visualization containing as many data attributes as can be shown effectively would be most useful. However, if the viewer zooms out to see an overview of the entire dataset, only a few pixels of screen space will be allocated to each data element, and thus many of the visual features used to represent different data attributes may not be easy to distinguish. This overloading of multiple visual features mapped to a data element with a limited number of pixels could be counterproductive, since it may interfere with our ability to accurately identify any data values at this low per-element resolution.

Figure 4 is a visualization of the same weather dataset in Figure 1. In the bottom-left image the same data is visualized, but for the entire continent of North America. Because only a few pixels are available for each data element, many of the visual feature values are difficult to identify. Moreover, the presence of certain features (e.g., small sizes) interferes with our ability to see other features (e.g., color). In the bottom-right image the same elements are visualized, but the number of attributes is reduced to two: temperature visualized with hue and pressure visualized with luminance. Since both hue and luminance are distinguishable even at small physical resolutions, the underlying data patterns are easy to identify. This type of scenario leads to the following questions: (1) which data attributes should we visualize, and which visual features should we use? and (2) when should attributes be removed from a visualization, and when should they be reintroduced?

We propose one possible solution - build a visualization system that can smoothly reduce or increase the number of attributes it represents based on the number of available pixels per element. For example, as the viewer zooms out, the number of pixels per element decreases and thus attributes should be smoothly removed. Likewise additional attributes can be rendered as the viewer zooms in. The idea is to maximize the utilization of the display's capabilities in an effective and efficient manner, thereby maintaining a balance in the display environment (i.e., more elements with fewer attributes encoded, or fewer elements with more attributes encoded). Zooming is also analogous to moving the visualization to a different display device with a different number of pixels, different physical dimensions, or different viewing distances. Therefore, techniques that vary a visualization based on a viewer's changing perspective on a given display device could be extended to address how a visualization should be updated for different types of displays.

Perceptual Display Hierarchies

This hierarchy may depend on the number of pixels needed for a visual feature to represent information effectively (i.e., what display resolutions) and the physical size needed for our visual system to accurately identify and interpret a visual feature (i.e., what visual acuity). Understanding the limits on display resolution and visual acuity may allow us to better validate a given visualization technique and characterize the extent to which a technique saturates "visual bandwidth".

Two separate sets of experiments can be conducted: (1) display resolution experiments designed to determine how many physical pixels are needed to represent different values of each visual feature, and (2) visual acuity experiments designed to determine the minimum visual angle required to represent different values of each visual feature.

Display Resolution Experiments

The display resolution experiments aim to determine the number of pixels needed to distinguish different values for a particular visual feature. Results from the display resolution experiments will identify the number of physical pixels needed to rapidly and accurately differentiate visual elements on the basis of different visual properties, such as, hue, luminance, size, orientation, flicker, and direction of motion.

The trials from the experiments can be displayed on a standard LCD monitor, such that the size of the elements in each trial subtends a sufficient visual angle on the viewer's retina to be easily distinguishable. If necessary, we can use virtual pixels formed by an array of n x n physical pixels to guarantee each data element is large enough to be properly recognized by the viewer's visual system. This will ensure that results depend on differences in the number of pixels allocated to an element.
The general format of each experiment will be as follows:

1. Select multiple values of a visual feature that are known to be distinguishable in isolation. Display the visual feature's values with different display resolutions (e.g., 1 pixel, a 2 x 2 array of pixels, a 4 x 4 array of pixels, and so on).
2. Run trials to test a viewer's ability to differentiate between the different values of a visual feature. For this purpose we can use a standard target detection task where randomly positioned arrays of elements are displayed to a viewer. Viewers are asked to answer whether a target element different from the background elements was present or absent in the array. Viewer accuracy and response time can be used as a performance parameter.
3. Results can be analyzed for statistically significant differences to determine how different display resolutions affect the speed and accuracy of a viewer's responses.

Visual Acuity Experiments

Following the display resolution experiments, we will investigate the minimum visual acuity required to identify different values of each visual feature. Specifically, we will vary the visual angle formed by the element on the viewer's retina to measure how changes in physical size affect the distinguishability of each visual feature.

In these new experiments we must visualize each element at sizes that straddle the visual system's threshold of distinguishability. If a monitor's physical pixel size is too large to allow this, we must choose a different presentation environment. One simple solution is to increase the viewing distance to decrease an element's visual angle. Another possibility is to use a device with a smaller dot size (e.g., a laser printer) to generate smaller elements. In all cases we will fix the display resolution (i.e., number of pixels per element) to a value that generated rapid and accurate performance during the previous experiments. This ensures that differences in performance are due only to differences in visual angle.

Each acuity experiment will have a pattern similar to the display resolution experiments. We will calculate the visual angle v of the elements in the display resolution experiment and use this as a starting point for our visual acuity experiments. Viewer accuracy and response time will again be used as a performance parameter. Visual acuity will be varied across each display type by dividing trials into four different groups: elements with a visual acuity of v degrees, elements with a visual acuity of ^v/₂ degrees, elements with a visual acuity of ^v/₄ degrees, and elements with a visual acuity of ^v/₈ degrees.

Findings will be analyzed using a multifactor analysis of variance, to identify statistically significant variations in performance. Results from the visual acuity experiments will be combined with the results from the display resolution experiments and integrated with existing guidelines on the use of perception during visualization. This will improve our ability to construct perceptually salient visualizations, and to maximize the utilization of a display's capabilities in an effective and efficient manner.

Distinguishability Graphs

Conclusions

The desire to extract information rapidly and efficiently from large, complex datasets motivates the need for developing effective visualization techniques. The knowledge of perception can be used to generate visualizations that harness the strengths of the low-level visual system. Most visualization techniques assume sufficient display resources and visual acuity are available for generating effective visualizations. However, this may not be true, particularly for specialized display devices. Perceptual display hierarchies is aimed at understanding the limits on display resolution and visual acuity which in turn, may allow us to better validate a given visualization technique and characterize the extent to which a technique saturates "visual bandwidth" and maximizes the utilization of a display's capabilities in an effective and efficient manner. This hierarchy will dynamically add or remove information to ensure visual features can be rapidly detected. Such modifications will be based on a display's resolution properties, our visual abilities, analysis tasks, and the type of data we are trying to visualize. Such a hierarchy would help better our understanding of the physical characteristics of display devices and acuity of the human visual system. It will also provide us the fundamental knowledge required to construct perceptually salient visualizations.

Glossary

Visual Angle:

Visual angle is the angle subtended by an object on the eye of an observer. Visual angles are generally defined in degrees, minutes, and seconds of arc (a minute is ¹/₆₀ degree and a second is ¹/₆₀ minute).

VisualAcuity:

Visual acuities are measurements of our ability to see detail. Acuities are important because they define absolute limits on the information densities that can be perceived.

Display Resolution:

The resolution of an element on a particular display device (i.e., the number of pixels allocated to an element on the screen).

 

References

1

McCormick, B. H., DeFanti, T. A., and M. D. Brown. Visualization in scientific computing. ACM SIGGRAPH Computer Graphics, 21, 6 (1987).

2

Smith, P. H., and Van Rosendale, J. Data and visualization corridors report on the 1998 CVD workshop series (sponsored by DOE and NSF). Technical Report CACR-164, Center for Advanced Computing Research, California Institute of Technology, 1998.

3

Callaghan, T. C. Dimensional interaction of hue and brightness in preattentive field segregation. Perception & Psychophysics 36, 1 (1984), 25-34.

4

Callaghan, T. C. Interference and domination in texture segregation: hue, geometric form, and line orientation. Perception & Psychophysics 46, 4 (1989), 299-311.

5

Callaghan, T. C. Interference and dominance in texture segregation. Visual Search, D. Brogan, Ed. Taylor & Francis, New York, New York, 1990, 81-87.

6

Dennis, B. M., Kocherlakota, S. M., Sawant, A. P., Tateosian, L. G., and Healey, C. G. Designing a Visualization Framework for Multidimensional Data. IEEE Computer Graphics and Applications 25, 6 (2005), 10-15.

7

Sawant, A. P., and Healey, C. G. Visualizing Abstract Data Using Animation. In Conference Compendium of IEEE Visualization, Baltimore, Maryland, 2006, 37-38.

8

Healey, C. G. Formalizing artistic techniques and scientific visualization for painted renditions for complex information spaces. In Proceedings International Joint Conference on Artificial Intelligence, Seattle, Washington, 2001, 371-376.

9

Furnas, G. W. Generalized fisheye views. In Proceedings of the SIGCHI conference on Human factors in computing systems, Boston, Massachusetts, 1986, 16-23.

---

Biography

Amit Prakash Sawant (amit.sawant@ncsu.edu) is a Ph.D. student in the Computer Science department at North Carolina State University. His research interests include computer graphics, scientific visualization, information visualization, visual perception, and databases. Web site: http://www4.ncsu.edu/~apsawant.

Christopher G. Healey (healey@csc.ncsu.edu) is an Associate Professor in the Computer Science department at North Carolina State University. His research interests include computer graphics, scientific visualization, perception and cognitive vision, color, texture, databases, and computational geometry. Web site: http://www.csc.ncsu.edu/faculty/healey.

---