Real-Time Adaptive Radiometric Compensation

Anselm Grundhöfer Oliver Bimber
Bauhaus-University Weimar Bauhaus-University Weimar

Camera-Projector systems enable a compensated projection onto colored and textured surfaces. The camera is used to acquire information about the surface which then can be used to project a compensation image which neutralizes the surface color. The image has to be reduced in its contrast and brightness depending on its intensities and the reflectance properties of the projection surface to avoid clipping errors in the compensation image which would make the underlying surface visible. We propose an adaptive, GPU-based real-time algorithm which adjusts the image intensities globally, as well as locally, to avoid visible clipping errors while preserving a maximum brightness in the projection 
Portions reprinted, with permission, from (Grundhöfer, A. and Bimber, O.,
"Real-Time Adaptive Radiometric Compensation",
IEEE Transactions on Visualization and Computer Graphics 2007).  © 2007 IEEE.
Mobile devices enables a series of new communication forms: Cell phones, PDA's and laptops offer enough power to setup a meeting on arbitrary locations. While the hardware devices like laptops and projectors get smaller and more powerful, a projection screen still has to be carried around to project the content in an adequate quality. Projection onto non-optimized screens leads to an image with geometrical distortions (if the surface is not flat) and wrong colors (if the surface has no uniform white reflectance properties). Geometric correction techniques in combination with radiometric and photometric compensation have been developed in the past years to overcome these limitations. These techniques enable the correct projection onto such non-optimized surfaces. Most of the existing algorithms only analyze the surface properties (geometry and reflectance) in advance and use this information to generate a compensation image for the content projection, but do not consider the current image properties like brightness and contrast in their algorithm. Especially when projecting onto dark surfaces with low reflectance the compensation will fail - clipping occurs due to the limited dynamic range of the projector: The calculated intensity value for the compensation cannot be generated by the device because it lies above its physical maximum or below its black level.  
Figure 1 The projection surface is a natural stone wall (a). In (b) a projected image (e) without radiometric compensation is shown. Comparison between globally pre-adjusted compensation (c) and our automatic adaptation (d). Note the preservation of brightness and the neutralization of clipping.
A few algorithms propose solutions to adjust the image intensities depending on the human visual system, the projection setup and the image content to apply an optimized radiometric compensation [1][7][9]. These algorithms however fail to present interactive, live or real-time content due to their numerical complexity. Our algorithm demonstrates a technique to apply a content dependent radiometric compensation in real-time. This makes it possible to present interactive content like presentations in an optimized way onto arbitrary diffuse surfaces. Our technique is realized by an off-line surface analysis and an on-line image content analysis followed by global and local intensity adjustments to reduce disturbing visible clipping errors while preserving high brightness and contrast values. The calculated parameters for the global and local adjustments are temporally smoothed to reduce the visibility of the ongoing intensity variations of the image content. Figure 1 shows a comparison between a non-adaptive radiometric compensation and our method.

2. Background and Related Work

Radiometric compensation techniques are a field of ongoing research. A series of methods have been developed in the last years [2][3][4][5][6][10] which vary in their complexity, functionality and quality. A common method to generate a radiometrically compensated projection is the usage of a camera-projector system which is used to acquire information about the surface and to project the compensated image content. In an off-line calibration step a series of patterns are projected and captured by the camera to measure the surface's reflectance properties, geometry and environmental lighting. The structured light projection can also be used to generate a mapping between projector and camera pixels which is necessary to adjust the projector intensities on a per pixel basis. This enables the projection of undistorted images (from the camera's view position) onto non-planar surfaces. Usually this step takes much less than a few minutes. As long as the projection setup, the surface and the environment does not change, this initial calibration only has to be done once. Currently only the work presented by Fujii et al. [4] enables the compensated projection onto a non-static surface.
In addition to these techniques a series of more complex algorithms were developed which analyze the input image and adjust its intensities to project a compensated image with optimized brightness and contrast [1][7][9]. While these algorithms are able to project images onto colored surfaces in a good way, interactive content cannot be presented in real-time due to their complexity. Our method combines the advantage of an image dependent radiometric compensation with the ability to present real-time content like animations and TV broadcasts.

3. Algorithm

Our algorithm performs content adaptation and radiometric compensation in real-time and reduces visual artifacts while preserving a maximum brightness and contrast. Although we chose the basic compensation scheme presented in [4], only minor modifications to the adaptation algorithm are necessary to instead use any of the other techniques.
The projection surface as well as the image content is analyzed to collect various information which then is used to adjust the intensities of the current image globally as well as locally to generate a compensation image which does not lead to disturbing clipping errors in the projection on the one hand, but efficiently utilizes the range of possible intensities on the other hand. Thus the perceived image quality can be increased significantly compared to a constant radiometric compensation. The algorithm is implemented entirely on the GPU and will be described in detail below.

Analysis

Generating radiometrically compensated images without clipping errors while simultaneously preserving a high contrast ratio and brightness requires the analysis of both: projection surface and input image. While the captured surface reflectance needs to be analyzed only once, the image content has to be analyzed continuously in real-time. The surface analysis requires a projector device, a camera and a device to measure the physical luminance of the projection system. These values are necessary for parts of the content analysis. The luminance can be measured by a photometer or a calibrated HDR-camera.
Surface Analysis
By applying the basic radiometric compensation technique described in [3], structured light projection and camera feedback delivers several surface properties after an initial offline calibration. For radiometric compensation the method requires two parameters for each pixel:
  1. The contribution of the (uncontrollable) environmental light which is reflected from the surface (including the projector's black-level) - EM
  2. The surface's reflectance and the projector-to-surface form-factors (the fall-off of projected intensity, depending on the projector-to-surface distance and the projection angle) - FM
The intensity range for which radiometric compensation without clipping is possible can now be computed from these two parameters. Figure 2 visualizes the reflection properties for a sample surface. By analyzing the responses in both datasets, we can compute the range of intensities for a conservative compensation (bound by the two green planes) which defines the intensities which can be compensated correctly for each point on the surface without causing clipping artifacts. This range can become very small or even zero depending on the reflection properties of the surface. The maximum range of intensities in which compensation can be carried out is defined by the red planes: This range defines the minimum and maximum intensities which can be compensated at at least one surface point. The surface analysis is accomplished during the system calibration. This step takes a few minutes and has to be done only once as long as the setup is not modified.
     
Figure 2: Three-dimensional view of the intensity range reflected by a striped wall paper. The area between both green planes depicts the range in which a compensated projection without clipping errors is possible for any image content (but only with significant contrast and brightness reduction). The area between the red planes represents the maximum range in which compensation is possible (potentially with clipping errors).       Figure 3: The input image (a) is analyzed for its average luminance value (b), the amount of high spatial frequencies (c) and its luminance threshold map is calculated (d).
Content Analysis
Since pixels outside the displayable range cause clipping artifacts, the input image is analyzed to support subsequent global and local luminance adjustments that ensure an optimized compensation. This analysis focuses on three different parameters which are shown in Figure 3. These will later be used to adjust the intensity of the image content globally as well as locally
  1. The average image luminance is used as an initial guess for the global luminance adjustment. The darker the image the brighter it can be presented and vice versa.
  2. The threshold map presented in [8], which stores the maximum non-perceivable luminance variation at each pixel, is used to constrain the local image variations. Varying the local luminance of the input image by an amount that is below the corresponding values in the threshold map will not lead to perceivable differences in luminance. A compensated projection, however, can be enhanced if this is done for regions in which clipping occurs.
  3. Information about the proportion of high frequencies in the image is used to vary the local image adjustments to reduce the visibility of these manipulations. By taking the amount of high frequencies into account when applying local image variations, the visibility of these manipulations can be kept at a low level.
These image processing steps are calculated in real-time directly on the GPU for each input image.

Adaptation and Compensation

Using the results of the analysis, the input image is adapted in its brightness in an optimized way during runtime. Due to the fact that the radiometric compensation should be adjusted according to the current input image in interactive frame rates, we implemented the adaptive compensation in three steps:
  1. Global scaling of the image's intensities depending on the calculated average image luminance.
  2. Error analysis of the scaled images resulting from step 1.
  3. Global and local intensity adjustments based on the errors determined in step 2. For this the threshold map and also the amount of high spatial frequencies are used to adjust the local adjustments. After these adaptation steps, the radiometric compensation is applied and the result is projected.
1. Pre-Adaptation
In a first adaptation step, the information acquired about the average image luminance and the surface properties is used to apply an approximate global scaling of the image's intensity. Then a compensation image is calculated from the scaled input image. This allows the analysis of the resulting quality of the global intensity adjustments and the identification of local clipping errors. The results are used for calculating the final global and local scaling parameters.
The intensities of the input image are scaled depending on the average image luminance and the maximum and minimum color values of the projection surface (see figure 2) which ensures that the input image will not be projected too dark needlessly, but also not too bright which would lead to a large amount of clipping errors. While images with a low average luminance are upscaled, very bright images are down-scaled in their intensity. However after this step of the algorithm clipping errors still might occur due to the fact that only the input image was taken into account but no detailed information projection surface. To obtain information about possible clipping errors, radiometric compensation according to [3] is applied to the adjusted input image. The result is an initial compensation image which is not used for projection, but is analyzed for errors to obtain information about image regions that have to be adjusted locally to avoid visible clipping. Clipping errors above the maximum intensity of the projector are stored as well as errors due to intensities below the minimum intensity which can be projected. Finally the maximum intensity of each pixel in which no clipping occurred is also marked. In the next step, these generated errors are analyzed and used to re-scale the image globally and locally for achieving optimized compensation results.
2. Error Analysis
A conservative global luminance reduction leads to the full elimination of clipping errors: abrupt alternations in luminance and chrominance within the displayed image can then be neutralized. But it also leads to a significant reduction in contrast and brightness which substantially reduces the perceived image quality. Therefore, our algorithm varies the image intensities locally in addition to neutralize remaining clipping errors while preserving a high overall image brightness and contrast. Studies of human visual perception indicate that abrupt changes in luminance are perceived more intensively than smooth and less frequent modifications. Consequently, we blur the calculated clipping errors with a large Gaussian smoothing kernel which enables the use of this information as smooth local intensity scaling factors to reduce the clipping errors. Attention has to be paid when applying the smoothening operator to the clipping errors: On the one hand, a smooth local modification is required to avoid abrupt intensity variations; on the other hand the image content should not be altered more than necessary. We adjust the sigma parameter of the filter kernel inverse proportionally to the calculated amount of high spatial frequencies of the input image. Thus, local luminance reductions affect a larger region in the image if the image content stores mostly low spatial frequencies. The affected area is decreased for a larger amount of high spatial frequencies. With the error information gathered the global image rescaling factor can now be adjusted more precisely with respect to the largest detected clipping value within the image to avoid large clipping errors, leading to a reduced visual quality of the projection. To avoid a perceivable flickering of the projection due to its continuous adjustment, the scaling factors are smoothened over time.
3. Final Adaptation and Compensation
In the remaining adaptation step, the intensity of the original input image is now adjusted globally as well as locally before radiometric compensation is applied. While the intensities are globally adjusted depending on the adjusted scaling factor, the smoothened error maps are used to adjust the intensities locally. This may lead to unnecessary large local intensity modifications in the original image. To adapt the amount of modification to the actual brightness of a pixel, the clipping errors are weighted by the corresponding luminance values. Clipping errors due to the limited maximum brightness of the projector affect only input pixels with high intensities, while errors resulting out of the black level only have an impact on pixels with low intensities. The calculated threshold map is used in combination with a scaling factor to constrain the visibility of the local adjustments. The adapted input image can now be radiometrically compensated in such a way that the resulting image is displayed with high brightness and contrast while disturbing clipping artifacts are minimized.

4. Uniqueness of the Approach

In contrast to existing image content-depended radiometric compensation algorithms, our solution offers the first real-time approach to generate an optimized trade-off between clipping error minimization and maximization of contrast and brightness. In addition to the real-time image adaptation the global and local adjustment parameters are smoothened over time to avoid abrupt visible intensity variations in the projection.
The complete algorithm is implemented on the GPU which gives a significant speed-up compared to a CPU based solution and is necessary to enable a real-time compensation. While other algorithms create good results with the use of complex numerical algorithms for the image adaptation, our method offers the first solution to apply a content dependent radiometric compensation for real-time content like interactive applications, TV broadcasts or presentations.

5. Results

Figure 4 shows an image (e) projected with the proposed method onto striped wallpaper (a). Note that the automatic global adjustments (c) still lead to clipping errors in the upper regions of the image. The additional local adjustments (d) neutralize these errors while a high overall brightness still can be preserved.
Figure 5 illustrates two different frames from the movie Elephants Dream, projected onto a natural stone wall (a). While (b) contains bright scenes, a dark scene is shown in (e). As demonstrated in (c,f), a basic compensation algorithm (e.g., [3]) will fail in this situation. On the one hand, visible clipping errors occur in image areas with bright intensities (c) due to the physical limitations of the projector. On the other hand, the displayed image becomes too dark (g) due to the static adjustment parameters of the basic method. Similar results will be produced by all other non-adaptive radiometric compensation methods. As illustrated in (d,g), our adaptive approach responds to these situations automatically in an optimized way.

     
Figure 4: Uncompensated projection (b) of an image (e) onto a striped canvas (a). The lower images show the results of out proposed method with global (c) and additional local (d) adjustments [© 2007 IEEE].       Fig. 5. Two frames of an animation (b, e) projected onto a natural stone wall with a static radiometric compensation (c,f) and with our adaptive algorithm (d,g). [© 2007 IEEE].

6. User Study

To get information about an objective improvement in perceived image quality compared to a non-adaptive radiometric compensation a user study was carried out. For the test environment a projection onto a natural stone wall was chosen. The evaluation was separated into three different tasks in which the participants had to compare projected still images and videos which were compensated with a static and the proposed adaptive compensation method. 32 subjects participated to the user study. While the subjects indicated only a small preference for the adaptive algorithm when still images were presented, it was significantly favored for dynamic content. Especially videos with varying contrast and brightness levels were perceived as enhanced. The diagram in figure 6 illustrates the results: A significant preference for the adaptive method was indicated for all four sample videos. It was confirmed that the adaptive approach delivers results that appear more like an ordinary projection than the static compensation method.      
      Figure 6: Result of the comparison of video sequences compensated with a static and the proposed adaptive radiometric compensation projected onto a stone wall. The average preference and its standard deviation are plotted on the vertical axis [© 2007 IEEE].

7. Summary and Future Work

We presented a GPU-based adaptive radiometric compensation technique, which analyzes the input image in real-time and uses these parameters to adjust the image intensity globally as well as locally before applying the radiometric compensation to minimize visible clipping errors. The main advantage lies in its real-time capability which provides a perceivably enhanced quality of animated and live-content. While other adaptive compensation algorithms also take the image content into account, the results cannot be computed in real-time. Temporal adaptation of the global and local scaling parameters is applied to reduce the visual awareness of the continuous image adjustments.
Although the image content is continuously adjusted in its intensity which slightly changes its appearance, a user study confirmed that our adaptive radiometric compensation increases the perceived image quality compared to a static radiometric compensation algorithm which might lead to disturbing clipping errors.

References
[1] M. Ashdown, T. Okabe, I. Sato and Y. Sato, "Robust Content-Dependent Photometric Projector Compensation,"
Proc. IEEE International Workshop on Projector-Camera Systems, pp. 60-68, 2006
[2] O. Bimber, F. Coriand, A. Kleppe, E. Bruns, S. Zollmann and T. Langlotz, "Superimposing Pictorial Artwork with Projected Imagery,"
IEEE MultiMedia volume 12, pp. 16-26, 2005
[3] O. Bimber, A. Emmerling and T. Klemmer, "Embedded entertainment with smart projectors, "
IEEE Computer 38 , pp. 56-63, 2005
[4] K. Fujii, M.D. Grossberg, S.K. Nayar, "A Projector-Camera System with Real-Time Photometric Adaptation for Dynamic Environments, "
IEEE Computer Society Conference on Computer Vision and Pattern Recognition CVPR'05, Volume 01, pp. 814-821, 2005
[5] M.D. Grossberg, P. Harish and S.K. Nayar, "Making One Object Look Like Another: Controlling Appearance Using a Projector-Camera System,"
Proc. Conference on Computer Vision and Pattern Recognition, pp. 452-459, 2004
[6] S. K. Nayar, H. Peri, M. D. Grossberg and P. N. Belhumeur, "A Projection System with Radiometric Compensation for Screen Imperfections,"
Proc. IEEE International Workshop on Projector-Camera Systems, 2003
[7] H. Park, M.-H. Lee, S.-J. Kim and J.-Il Park, "Contrast Enhancement in Direct-Projected Augmented Reality,"
ICME'06: International conference on Multimedia and Expo, 2006
[8] M. Ramasubramanian, S. Pattanaik and D.P. Greenberg, "A perceptually based physical error metric for realistic image synthesis,"
SIGGRAPH '99: Proceedings of the 26th annual conference on Computer graphics and interactive techniques, pp. 7382, 1999)
[9] D.Wang, I Sato, O. Takahiro and Y. Sato, "Radiometric Compensation in a Projector-Camera System Based Properties of Human Vision System,"
Proceedings of the 2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'05) - Workshops, Volume 03, page 100, 2005
[10] G. Wetzstein and O. Bimber, "Radiometric Compensation of Global Illumination Effects with Projector-Camera Systems,"
SIGGRAPH '06: Proceedings of the 33th annual conference on Computer graphics and interactive techniques, Poster, 2006