Life through a lens
CSE 558 Project One – Spring, 2006
by
Leith Caldwell
"Office" scene. Fourteen pictures of a static scene taken at different exposure times, photographed using a Nikon CoolPix 990 digital camera with aperture f/2.5. The original image resolution is 2048 × 1536. The exposure times are, from darkest to brightest, 1/1000, 1/500, 1/250, 1/125, 1/60, 1/30, 1/15, 1/8, 1/4, 1/2, 1, 2, 4 and 8s.
The challenge
The goal of this project was to provide some experience in photometric and geometric callibration of a digital camera for High Dynamic Range (HDR) scenes and allow exposure to some of the larger problems in the area of 3D Photography.
The response curves for the captured sequence of images, on a logarithmic scale. Something has clearly gone wrong.
The how
The approach I took was to try and implement the photometric callibration system of Robertson et al.1 I captured a series of images from my office with different shutter speed settings, then resampled them down from ~9MB to ~220KB files.
The challenges I came across were:
- Understanding the equations and translating them back to the appropriate data structures available to me
- Figuring out how to operate the digital camera (the Nikon CoolPix 990) and being able to adjust the settings that I needed to
- Algorithm taking longer than it should to converge (in some cases over a hundred iterations), not converging or not iterating at all (several cases where less than the full range of input images were used resulted in output HDR images that were completely black)
For Geometric callibration I used Jean-Yves Bouguet's camera calibration toolbox. Here are the relevant results:
The response curves generated by by the max.hdrgen sequence.
%-- Focal length: fc = [ 2399.876327121294707 ; 2345.810520713514506 ]; %-- Principal point: cc = [ 1026.063110892636814 ; 778.955453792131834 ]; %-- Skew coefficient: alpha_c = 0.000000000000000; %-- Distortion coefficients: kc = [ -0.272253860746257 ; 1.744306754823821 ; 0.001297212055343 ; -0.002004185676521 ; 0.000000000000000 ]; %-- Focal length uncertainty: fc_error = [ 7.435930729555596 ; 6.207799135965808 ]; %-- Principal point uncertainty: cc_error = [ 14.359341199609478 ; 12.650806011654399 ]; %-- Skew coefficient uncertainty: alpha_c_error = 0.000000000000000; %-- Distortion coefficients uncertainty: kc_error = [ 0.036374536223011 ; 0.877366757914935 ; 0.001104762212628 ; 0.001045178942635 ; 0.000000000000000 ];
Reality Check
The colorcheck image sequence after my max response curve was applied.
By taking the response curve, shown above on the right, from the max.hdrgen
set and applying it to the colorcheck
image dataset, the image below on the
right was generated. From this, the relative intensities of the 6 greyscale patches
(the bottom row) were computed.
Patch | Relative Intensity | Expected Intensity | % difference (actual) | % difference (relative) |
---|---|---|---|---|
1 | 1.0 | 1.0 | N/A | |
2 | 0.669 | 0.657 | 1.2% | 1.8% |
3 | 0.369 | 0.402 | -3.3% | -8.2% |
4 | 0.196 | 0.220 | -2.4% | -10.9% |
5 | 0.091 | 0.100 | -0.9% | -9% |
6 | 0.038 | 0.035 | 0.3% | 8.6% |
Changes to the original plan
During the development of this project, a few things became clear about my original plan that needed to change.
Matlab in action!
The Camera
It was not the easiest thing in the world to figure out what to push and how to
operate it, even aided by the manual. In addition, all the batteries in the system
had died. After replacing the batteries in the remote for the system (the
DigiSnap
2000) it was determined that the remote couldn't change the shutter speed anyway,
so all of the pictures would need to be manually taken.
The Scope
This was a very challenging project to understand and attempt to translate the
right equations into code, so my technical limitations became very apparent from
early on in the process.
The (erroneous) HDR image generated by my algorithm.
The result
After all that, I have an implementation of the Robertson et al. photometric callibration system that has several issues with response curve convergence but performs reasonably well against the project benchmark.
There is always more you can do
In addition to doing comparisons with different geometric calibration toolkits, there are a number of things that could be implemented to extend the scope of the project. These include, as mentioned in the project outline, enforcing a monotonic and/or smooth response curve, investigating the influence of the input images on the response curve recovery and much more.
Resources & Related work
X-Win32 X-server, SSH.com tools - for making the whole Linux-remoting-development process much easier.
1. Mark A. Robertson, Sean Borman, and Robert L. Stevenson. Estimation-Theoretic Approach to Dynamic Range Improvement Using Multiple Exposures. Journal of Electronic Imaging, vol. 12, no. 2, pages 219--228, April 2003.
Acknowledgements
Thanks to Christian Fuchs for helping me get my brain back into the programming hot seat and for being extremely helpful and constructive even when I was tearing my (not inconsiderable amount of) hair out. Thanks also to Michael Goesele for assigning me an opportunity to extend my capabilities and teaching me something challenging and new.