CS194 Project 1: Colorizing the Prokudin-Gorskii photo collection

Rohan Chitnis

Background and Images

(Skip to bells and whistles section.)

The purpose of this project was to extract and align three images, representing three color channels, to form a single color image. First, I implemented a single-scale version, which uses the SSD metric to score the displacement between images. I then aligned both the red and the green images to the blue one using this error metric. Then, because this exhaustive search method is very slow for high-quality TIF images, I implemented coarse-to-fine pyramid speedup, which continually halved the image size and looped over larger displacements for smaller image sizes. I then added together all the optimal displacements obtained at each level, weighted more strongly for the smaller image sizes.

I then observed that this method doesn't quite work for images such as Emir and selfie, because the images have different brightness values. To remedy this, I decided to apply a canny edge detector to the images before aligning them, which means the alignment is between two binary images. This fixed the problem and made Emir and selfie align properly. The difference for selfie is demonstrated near the bottom.

Furthermore, I implemented the techniques of automatic contrasting, automatic white balancing, and automatic border cropping. The images that follow have the contrasting and white balancing applied, but not the cropping. More explanation of my implementation of these extra tasks is given near the bottom.

antique
Offsets: Red X: 12, Red Y: 1, Green X: 3, Green Y: 1
bridge
Offsets: Red X: 69, Red Y: 5, Green X: 15, Green Y: -2
cathedral
Offsets: Red X: 12, Red Y: 3, Green X: 5, Green Y: 2
emir
Offsets: Red X: 107, Red Y: 40, Green X: 49, Green Y: 23
harvesters
Offsets: Red X: 117, Red Y: 11, Green X: 55, Green Y: 10
lady
Offsets: Red X: 120, Red Y: 13, Green X: 56, Green Y: 10
melons
Offsets: Red X: 182, Red Y: 11, Green X: 89, Green Y: 9
monastery
Offsets: Red X: 3, Red Y: 2, Green X: -3, Green Y: 2
onion_church
Offsets: Red X: 107, Red Y: 34, Green X: 52, Green Y: 24
selfie
Offsets: Red X: 175, Red Y: 37, Green X: 83, Green Y: 29
settlers
Offsets: Red X: 14, Red Y: -1, Green X: 7, Green Y: 0
three_generators
Offsets: Red X: 106, Red Y: 7, Green X: 56, Green Y: 11
tobolsk
Offsets: Red X: 6, Red Y: 3, Green X: 3, Green Y: 2
train
Offsets: Red X: 84, Red Y: 28, Green X: 40, Green Y: 8
tree
Offsets: Red X: 11, Red Y: 9, Green X: 5, Green Y: 5
turkmen
Offsets: Red X: 117, Red Y: 29, Green X: 55, Green Y: 17
village
Offsets: Red X: 138, Red Y: 22, Green X: 67, Green Y: 12

Bells and Whistles

1. Better features: use canny edge detection before SSD displacement calculation, instead of using raw pixel values.

selfie
Image without canny edge detection -- uses raw pixel values instead.
selfie
Image with canny edge detection. Note the improvement in quality of alignment!

2. Automatic contrasting: I determine the 5th and 95th percentile pixel values for each color channel, then set pixels below the 5th percentile to 0.0 and above the 95th percentile to 1.0. I then scale all other pixels appropriately.

church
Image without automatic contrasting.
church
Image with automatic contrasting. Note the more vibrant colors, especially in his clothing!

3. Automatic white balancing: I first tried a white world approach, but that didn't work too well. I then utilized a gray world approach, where I calculated the gain of the red pixels as average green intensity over average red intensity, and gain of the blue pixels as average green intensity over average blue intensity. I then multiplied the red pixel intensities by the red gain, and similarly for the blue pixels.

church
Image without automatic white balancing.
church
Image with automatic white balancing. Note that the blue tint has disappeared, making the image seem more realistic to our brains!

4. Automatic border cropping: I again used a canny edge detector to obtain an edge map of the image. With this, I considered the first 10% of rows/columns from each edge, and found the highest-intensity horizontal/vertical pixel strip closest to the center of the image, and used this as the crop point. Unfortunately, a shortcoming of this algorithm is that if the border noise is not axis-aligned, no corresponding straight-line intensity will be very high. Also, if the noise is not very noticeable, an edge detector might not pick up on it.

church
Image without automatic border cropping.
church
Image with automatic cropping. Note that the left and right border noise has been comepletely removed, and the lower border noise has been significantly reduced. However, my algorithm does not work perfectly at cropping this image.