Life through a lens

CSE 558 Project One – Spring, 2006
Leith Caldwell

The various images taken with different shutter speeds.

"Office" scene. Fourteen pictures of a static scene taken at different exposure times, photographed using a Nikon CoolPix 990 digital camera with aperture f/2.5. The original image resolution is 2048 × 1536. The exposure times are, from darkest to brightest, 1/1000, 1/500, 1/250, 1/125, 1/60, 1/30, 1/15, 1/8, 1/4, 1/2, 1, 2, 4 and 8s.

The challenge

The goal of this project was to provide some experience in photometric and geometric callibration of a digital camera for High Dynamic Range (HDR) scenes and allow exposure to some of the larger problems in the area of 3D Photography.

The response curves for the captured sequence of images, on a logarithmic scale.

The response curves for the captured sequence of images, on a logarithmic scale. Something has clearly gone wrong.

The how

The approach I took was to try and implement the photometric callibration system of Robertson et al.1 I captured a series of images from my office with different shutter speed settings, then resampled them down from ~9MB to ~220KB files.

The challenges I came across were:

For Geometric callibration I used Jean-Yves Bouguet's camera calibration toolbox. Here are the relevant results:

The response curves generated by by the max.hdrgen sequence.

The response curves generated by by the max.hdrgen sequence.

%-- Focal length:
fc = [ 2399.876327121294707 ; 
       2345.810520713514506 ];

%-- Principal point:
cc = [ 1026.063110892636814 ; 
        778.955453792131834 ];

%-- Skew coefficient:
alpha_c = 0.000000000000000;

%-- Distortion coefficients:
kc = [ -0.272253860746257 ; 
        1.744306754823821 ; 
        0.001297212055343 ; 
       -0.002004185676521 ; 
        0.000000000000000 ];

%-- Focal length uncertainty:
fc_error = [ 7.435930729555596 ; 
             6.207799135965808 ];

%-- Principal point uncertainty:
cc_error = [ 14.359341199609478 ; 
             12.650806011654399 ];

%-- Skew coefficient uncertainty:
alpha_c_error = 0.000000000000000;

%-- Distortion coefficients uncertainty:
kc_error = [ 0.036374536223011 ; 
             0.877366757914935 ; 
             0.001104762212628 ; 
             0.001045178942635 ; 
             0.000000000000000 ];

Reality Check

The colorcheck image sequence after my max response curve was applied.

The colorcheck image sequence after my max response curve was applied.

By taking the response curve, shown above on the right, from the max.hdrgen set and applying it to the colorcheck image dataset, the image below on the right was generated. From this, the relative intensities of the 6 greyscale patches (the bottom row) were computed.

Patch Relative Intensity Expected Intensity % difference (actual) % difference (relative)
1 1.0 1.0 N/A
2 0.669 0.657 1.2% 1.8%
3 0.369 0.402 -3.3% -8.2%
4 0.196 0.220 -2.4% -10.9%
5 0.091 0.100 -0.9% -9%
6 0.038 0.035 0.3% 8.6%

Changes to the original plan

During the development of this project, a few things became clear about my original plan that needed to change.

Matlab in action!

Matlab in action!

The Camera
It was not the easiest thing in the world to figure out what to push and how to operate it, even aided by the manual. In addition, all the batteries in the system had died. After replacing the batteries in the remote for the system (the DigiSnap 2000) it was determined that the remote couldn't change the shutter speed anyway, so all of the pictures would need to be manually taken.

The Scope
This was a very challenging project to understand and attempt to translate the right equations into code, so my technical limitations became very apparent from early on in the process.

The (erroneous) HDR image generated by my algorithm.

The (erroneous) HDR image generated by my algorithm.

The result

After all that, I have an implementation of the Robertson et al. photometric callibration system that has several issues with response curve convergence but performs reasonably well against the project benchmark.

There is always more you can do

In addition to doing comparisons with different geometric calibration toolkits, there are a number of things that could be implemented to extend the scope of the project. These include, as mentioned in the project outline, enforcing a monotonic and/or smooth response curve, investigating the influence of the input images on the response curve recovery and much more.

Resources & Related work

X-Win32 X-server, tools - for making the whole Linux-remoting-development process much easier.

1. Mark A. Robertson, Sean Borman, and Robert L. Stevenson. Estimation-Theoretic Approach to Dynamic Range Improvement Using Multiple Exposures. Journal of Electronic Imaging, vol. 12, no. 2, pages 219--228, April 2003.


Thanks to Christian Fuchs for helping me get my brain back into the programming hot seat and for being extremely helpful and constructive even when I was tearing my (not inconsiderable amount of) hair out. Thanks also to Michael Goesele for assigning me an opportunity to extend my capabilities and teaching me something challenging and new.