Retinex Theory

Published on: Feb 08, 2022

Retinex Theory of Color Vision

Retinex Color Theory first proposed by Edwin Land in 1964 aimed to explain the color constancy of the Human Visual System. Retinex theory tries to explain how our visual system works for color changes and perceives object/scene color despite changes in illumination.

Color constancy is a phenomenon in which the human visual system adapts to illumination/lighting changes and identifies the relative color of the object correctly like we can observe the color of an apple as red in both bright light and dark light.

To understand how the human visual system sees color, read my later post about Color Science.

Read more about Color Constancy and adaptation in the previous article on Chromatic Adaptation.

With Retinex, Land challenged the conventional color theory that states color sensation occurs due to the mixing of colors and discount of illumination. Land, his colleagues, and other researchers worked for many decades and still working to improve the Retinex theory of understanding Color vision that speaks about color processing, color sensation, and color constancy in our visual system and also using the Retinex model in image processing applications.

Red and White image experiment

In 1959, Land conducted a series of experiments based on the principles of color formation by mixing 3 primary colors (red, blue, green). A color can be seen with the sensitivity of Cones cells that responds to primary color intensities coming from the surface. The Cones cells have three types of photoreceptor cells which are Long(L), Medium(M), and Short(S) cones that respond to three overlapping bands of wavelengths of the visual spectrum and stimulate the colors red, blue, and green respectively.

An image is a composition of these 3 primary colors and our eye stimulates L, M, and S cone cells to perceive color by interleaving these three color sensations. And Rods are a type of photoreceptor cells that are activated only when there is a low level of illumination threshold(dark light) on the scene below where cones cannot stimulate the cells and rods sense object colors as shades of gray(black and white ranges). As an image is a combination of three colors and if we can separate these three colors and make image color (red, blue, green) channels, and if viewed each channel image, it looks like a black and white image. This black and white channel image tell how much composition does have of each color.

Now, according to classical color theory principles, to form a color, one does have to mix all primary colors in any composition. But Land discovered that with only single primary color, a whole wide gamut of colors of a scene can be formed and showed results in his published paper Experiments in Color Vision.

Experiments in Color Vision Original scene red and green channels black-and-white transparencies recording.

He did this by projecting a red channel transparent black-and-white image through a red filter and a green channel transparent black-and-white image without a filter (that is white light) which were earlier taken from the original scene with various objects in color. By superimposition of light coming from the red filter and no filter (green channel image), it was observed that with increasing intensity of red light, colors appearing by superimposition covered a wide gamut of colors of the original scene but not accurate colors.

Experiments in Color Vision Projecting red image recording through a red filter and green image transparent through no filter (white light). Observe that, even with the only red color channel, we can see some range of colors compared to the original image.

The colors formed are shades of red and a wide range of saturated colors. As this experiment showed different colors formation with only red and white light, Land called this experiment a Red-and-White image.

Before proceeding further reading it's better to have an understanding of terms related to light like illumination, radiation energy, radiant flux, radiance, surface reflectance, etc.

Lightness

Lightness is one of the most important concepts that Land proposed to understand the color vision of humans. Before understanding the colorful world, he started uncovering non-colorful details like black-and-white sensations we observe in dark vision which is when rods are activated and identifies scene color in the range from white to black.

In the dark vision, for color objects, our eyes perceive objects color of red, blue, and black as darker, green and white as lighter. The color of an object in night vision is not determined by the amount of light energy we are getting at the eye, but the nature of objects which have the amount of composition of the above colors. So whether increased or decreased illumination on the scene or radiant energy coming to the eye, the color perception is close to the object's true nature in the range of white-to-black colors. Lighter objects look light and darker objects look dark irrespective of radiation flux reaching the eye. Thus we can say colors of scene/image when only one photoreceptor (here rods) is functioning are the only range of white-to-black colors. And the lightness of a point in a scene is simply how much lighter it is compared to the lightest point in the scene.

So, the lightness value of an image/scene tells the black and white range value when only one photoreceptor is activated. You can imagine lightness as a single image channel that consists of only black and white colors as described in the above section. And this behavior can be extended to cones cells and each L, M, and S cone produce lightness values independently. The lightness of an area would be affected by the surrounding area's lightness like a small area when surrounded by a large area, the small area lightness depends on the large area if it is lighter or darker.

In our visual system, irrespective of illumination and flux reaching the eye, when light falls on the retina and it is observed by L, M, and S visual pigments independently and produce a response to the receptor system that converts these responses to produce lightness of that particular cell sensation which doesn't to relate the amount of flux energy reaching the eye.

Now one can assume this lightness as red, blue, and green sensitivity values of a scene that processed independently by three visual receptor cells. If rods and L cones are activated with an illumination level that cuts off M and S cone's responsivity, then the resultant combination is a composition of black-and-white values from rods and red values from L cone cells.

John McCann and Jeanne Benton reproduced the above Red-and-White image experiment by observing the color sensations of an eye when only rods and L cells are activated. This experiment is similar to projecting a red channel image through a red filter and a green channel image through no filter (white light). Rods cells activated when the scene was illuminated with a 550nm narrow waveband of light and long-wave receptor cells (L) activated with a 656nm narrow band of light. As L cones sense red color and rod cells sense black-and-white, the results observed of the scene were similar to the original experiment and various close colors of the original scene were observed.

Retinex

The location of production of lightness in the visual system is uncertain that is it would be formed either in the retina or cortex or in both. As there is uncertainty about the location, Land coined the term Retinex which is a combination of retina and cortex. And Retinex is a biological mechanism of converting light to lightness that can happen in any part of the visual system.

The experiments above raise questions that what makes the color? From the above two-receptor experiment which produced color even though medium and short receptors haven't observed anything, Land concluded that independent lightness of each photoreceptor cell are compared to each other and the color is formed by comparing lightness values of different regions of the scene which contradicts the then existing belief of color formation by mixing of response by photoreceptors.

For color constancy, Land stated that as the lightness of three independent receptors is independent of flux reaching the eye (and also illumination), no matter what the illumination projected on the scene out visual system produces lightness for each of three wavebands and comparison of these lightness values makes color. So we achieve color constancy because of lightness produced independently of flux. The previous statement is quite convincing to answer the human color constancy behavior. But how one can measure lightness? how to prove color constancy with lightness independent of flux? We will cover these questions in the following experiment involving the color Mondrian.

Retinex system respone

Before proceeding to the color Mondrian experiment, let's first discuss Retinex responses that are equal to the human visual response that converts light energy to lightness for three independent receptor cells. To measure the visual sensitivity response of each cell, a device is developed using film-filter combinations to match the responsiveness of photoreceptor cells.

Spectral sensitivities of visual receptor cells Spectral sensitivities of each photoreceptor cell for each waveband range in spectrum.

In the above diagram, the visual pigment sensitivity of the human eye is represented. The solid gray curve for rods and solid black curves for each cone cell. Broken line curves represent the matching sensitivities of the device for each of the photoreceptor cells. With this device that produces sensitivities like the human visual system, black-and-white images for each cell can be produced, and the resultant formation record/image represents respective lightness for that cell. And whenever there is lightness, Retinex comes in. So, the lightness records are called Retinex records which are black-and-white lightness of each cell responsiveness.

Color Mondrian experiment

To prove that the human visual system adapts to illumination changes and performs color constancy, the Color Mondrian experiment was conducted by Land by preparing two identical boards filled with different color papers of various rectangular sizes. The resultant board formation is called Mondrian because the arrangement of color papers in different area sizes resembles the artwork of the Dutch painter Piete Mondrian.

Color Mondrian

The two identical Mondrian boards are illuminated by three (red, green, blue) light projectors equipped with 670nm long-band, 540nm medium-band, and 450nm short-band wave filters. The illumination settings of these three different projectors are controlled to project any overall illumination conditions by changing light intensity. A photometer is used to read the amount of radiation reflected from any region of Mondrian.

Color Mondrian projection experiment Two identical Mondrians are illuminated focusing green and red areas respectively.

Initially left side illumination is on and the right side is projection is off. Then select an area on the left Mondrian say green, change the illumination settings of the three projectors so that the green area looks pure green on the left side. Read radiance values from the photometer of the light that reaches the eye. Radiance values are measured one by one by turning off the other two projectors. Assume (X, Y, Z) is the radiance values for long, medium, and short waveband projectors coming from the left side.

Now on the right side, select an area other than green say red patch. For each of the projectors on the right side, turn on one projector and keep the other two projectors off. Change illumination so that the radiance value measure is the same as that coming on left side projectors. For example, on the long wave projector and turn-off medium and short waveband projectors. Increase/decrease the intensity of the long-wave projector so that the radiance value measured is equal to X. Repeat the same for other medium and short wavebands so that respective radiance values are equal to Y and Z. After reading values individually, turn on all three projectors and observe the color of the selected patch. It looks the same as red color even though the same quanta catch on the retina.

This observation that despite the same light energy reaching the eye, we can identify the patch color closely as green and red. This result challenges the principle that different colors are recognized by the eye depending on the amount of light intensity for each of the long, medium, and short wavebands contain. Here despite identical light energy (X, Y, and Z radiances) falling on the retina, we can identify different colors.

This makes Land conclude that color sensation in the human visual system processes each of long, medium, and short wavebands individually and forms another informative image called lightness image (as described above) for each of the three cones cells. The lightness images calculated are independent of the amount of light intensity reaching the eye.

Lightness is independent of flux because it can be observed that when only one receptor is active, let's take a long wave, despite the change in intensity of light, red color appears light. For medium and short waveband, red color is observed as darker and very dark respectively. So that for every color, the level of lightness for each of the three receptors is constant despite the change in light energy.

Now we know lightness is independent of light illumination and we have to examine how the reflectance from the surface affects the color sensation under different illuminations.

Color Mondrian and Munsell Chip matching experiment

John J. McCann, Suzanne P. McKee, and Thomas H. Taylor experimented to study responses of observers if asked to match Mondrian areas to respective color in Munsell's book of color (which contains color swatch for hundreds of colors).

Color Mondrian and Munsell chip matching Observers view Mondrian and Munsell chips through left and right eyes resepectively

The color Mondrian is made with 17 different colors and projected under various illuminations and observers are asked to match the corresponding Munsell chip for every Mondrian color. Munsell's book is viewed under constant white illumination so that the white color looks pure white. The color chips are viewed by surrounding large gray paper which has a square cut to look exactly one chip at a time. Observers view Mondrian through the left eye and Munsell chips through the right eye.

First observers are asked to match 17 Munsell chips for each of the Mondrian color patches under illumination settings that lit gray area in Mondrain as pure gray. Now record the triplet of radiance energies coming to the eye from the gray region on Mondrian say (X, Y, Z) for long, medium, and short wavebands, and also read radiance triplet for the gray chip in Munsell chip. After reading, select another patch, take the red region on Mondrian, and change illumination settings that the radiance triplet coming from the red region is the same as (x, Y, Z), Note the matched Munsell chips for each of the Mondrian areas under this illumination that red region sent (X, Y, Z) radiance triplet. Repeat the experiment for adjusting illumination for yellow, blue, and green so that triplet energies coming from each of the respective areas is the same as (X, Y, Z) and note down the matched 17 chips and radiance triplet values. Throughout the experiment, only illumination settings for Mondrian change, and Munsell's book is viewed under constant illumination. Several observers are asked to match colors and matching colors are averaged over all experiments. And results showed that observers matched color chips approximately to several Mondrian color patches even under different illuminations. This shows color constancy under different illuminations by observers.

Radiances are measured by a combination of photomultiplier and the retinex filters which measure different waveband responses that are close to the human visual system. This photomultiplier-retinex combination measures the integrated radiation flux for a given area for different wavebands. The resultant value is called integrated radiance as it gives radiation energy integrated for an area.

Illumination values and light triplet integrated radiance values reaching the eye are known but don't know the reflectance from the surface. The reflectance of a surface is the amount of light reflected divided by the amount of light incident on the surface. Reflectance in general is not a comparison value and doesn't give a relative measurement. As white surface reflects almost all amount light, McCann and his colleagues used white as a comparison for forming scaled reflectance values. First, they illuminated the white surface with illumination that had been used for illuminating Munsell's book of color. They recorded integrated radiance values for each of the wavebands coming from a white surface. Now for any surface, reflectance can be obtained by dividing radiance values with radiance values for the white surface under the same illumination. The reflectance values are expressed as a percentage in the range 0-100 with 100 percent being maximum reflectance that is white surface reflectance.

Lightness Sensation

Observers are asked to arrange different shades of black and white color patches in a range from darkest to lightest, and reflectance values are recorded for every color in the range. A function has been derived based on several arrangements by different observers, and the function gives the relation between black-and-white colors and reflectance value. Based on reflectance values for a single narrow-band wave, lightness sensations are scaled from 0-10 equally spaced intervals as shown in the following figure

Lightness sensation vs Reflectance for shades of black-and-white colors

Reflectance values for various Munsell chips are determined under the same white illumination. It is observed that reflectance values for the Mondrian area and corresponding matched Munsell chips are close.

Now back to the experiment, a very close relation between reflectance and illumination is observed that the light energy reaching the eye is equal to reflectance product illumination.

Reflectance product Illumination

In the above image, the left side charts denote Mondrian color paper values and the right side denotes matched Munsell chip values. The bar chart denoting reflectance has values in scaled reflectance in the range of 0.0-1.0 for each of the three narrow-bands 630nm, 530nm, and 450nm. For both Mondrian and Munsell, light energy at the eye (recorded by photometer) is equal to the product of reflectance and illuminant.

From this data, the relation between surface reflectance, illumination of surface, and color sensation in the eye can be described as

I=R*S

where I is the surface image, R is the reflectance of the surface, and S is the illumination on the surface.

Retinex theory talks about how lightness is crucial for color constancy. But how does our visual system calculate the lightness of a scene for each of the receptor cells? How visual system scales these lightness values like lightness sensation and reflectance relation? Land stated that our visual system compares the lightness images and senses a color. The color of an object also depends on the surrounding colors and edges. This comparison of lightness across each receptor cell and for every region gives the sensation of color. And human visual system obtains these lightness values from only flux reaching the eye from a scene by comparing narrow-waveband response ratios at adjacent edges and surrounding areas that are independent of illumination and reflectance.

From a lightness image, the eye compares different regions of the image and calculates the sequential product of ratios of reflectances to estimate maximum and minimum lightness values for that lightness image. These sequential products also give relative lightness between two regions of the image. Our visual system has evolved over the years to adapt to these changes and maintain color constancy.

Even though Edwin Land's Retinex theory conceptualizes how the human visual system adapts to color constancy, the accuracy of the theory has not been proved yet, and many scientists disproved the relation between lightness and color adaptation. After many decades, there is still research going on Retinex theory and its applications mainly in image processing.