Abstract
Polarization may be sensed by imaging modules. This is done in various engineering systems as well as in biological systems, specifically by insects and some marine species. However, polarization per pixel is usually not the direct variable of interest. Rather, polarizationrelated data serve as a cue for recovering taskspecific scene information. How should polarizationpicture postprocessing (P^{4}) be done for the best scene understanding? Answering this question is not only helpful for advanced engineering (computer vision), but also to prompt hypotheses as to the processing occurring within biological systems. In various important cases, the answer is found by a principled expression of scene recovery as an inverse problem. Such an expression relies directly on a physicsbased model of effects in the scene. The model includes analysis that depends on the different polarization components, thus facilitating the use of these components during the inversion, in a proper, even if nontrivial, manner. We describe several examples for this approach. These include automatic removal of path radiance in haze or underwater, overcoming partial semireflections and visual reverberations; threedimensional recovery and distanceadaptive denoising. The resulting inversion algorithms rely on signalprocessing methods, such as independent component analysis, deconvolution and optimization.
1. Introduction
An expanding array of animals are found to have a visual system that is polarizationsensitive, using several mechanisms [1,2]. This array includes various marine animals [3–17] as well as air and land species [18–22]. It has been hypothesized and demonstrated that such a capacity can help animals in various tasks, such as navigation (exploiting the polarization field of the sky), finding and discriminating mates, finding prey and communication.
Similarly, machine vision systems may benefit from polarization. Thus, computational methods are being developed for polarizationpicture postprocessing (defined here as P^{4}). Some tasks are enhancement of images, and extraction of features useful for higherlevel operations (segmentation and recognition). In this paper, we focus on inverse problems that can be solved using P^{4}. In these problems, a physical model of effects occurring in the scene can be formulated in a closed form. Inversion of the model quantitatively recovers the scene, overcoming various degradations. In the context of polarization, this approach is used in remote sensing from satellites, astronomy and medical imaging. In contrast, this paper surveys several inverse problems relating to objects, distances and tasks that are encountered or applicable to everyday scenarios. In such commonly encountered cases and lifesizes, we show how polarization provides a useful tool.
2. Descattering
In atmospheric haze or underwater, scene visibility is degraded in both brightness contrast and colour. The benefit of seeing better in such media is obvious, for animals, human operators and machines. The mechanisms of image degradation both in haze and underwater are driven by scattering within the medium. The main difference between these environments is the distance scale. Other differences relate to the colour and angular dependency of light scattering. Due to the similarity of the effects, image formation in both environments can be formulated using the same parametric equations: the mentioned differences between the media are expressed in the values taken by the parameters of these equations.
It is often suggested that P^{4} can increase contrast in scattering environments. One approach is based on subtraction of different polarizationfiltered images [23–25], or displaying the degree of polarization (DOP) [26,27]. This is an enhancement approach, rather than an attempt to invert the image formation process and thus recover the objects. Furthermore, this approach associates polarization mainly with the object radiance. However, rather than the object, light scattered by the medium (atmosphere or water) is often significantly polarized [6,28,29] and dominates the polarization of the acquired light.
(a) Model
In this section, we describe a simple model for image formation in haze or water, including polarization. Then, this model is mathematically inverted to recover the object. Polarization plays an important role in this recovery task [30–35].
As depicted in figure 1, an image acquired in a medium has two sources. The first source is the scene object at distance z, the radiance of which is attenuated by absorption and scattering. This component is also somewhat blurred by scattering, but we neglect this optical blur, as we explain later. The image corresponding to this degraded source is the signal 2.1where L_{object} is the object radiance we would have sensed, had there been no scattering and absorption along the line of sight (LOS), and (x,y) are the image coordinates. Here t(z) is the transmissivity of the medium. It monotonically decreases with the distance z.
The second source is ambient illumination. Part of the illumination is scattered towards the camera by particles in the medium. In the literature, this part is termed path radiance [36], veiling light [6,16,37], spacelight [4,6,16,29] and backscatter [38]. In literature dealing with the atmosphere, it is also termed airlight [39]. This component is denoted by B. It monotonically increases with z. The total image irradiance is 2.2Clearly, B is a positive additive component. It does not occlude the object. So, how come B appears to veil the scene? Furthermore, the image formation model (equations (2.1) and (2.2)) neglects any optical blur. Thus, how come hazy/underwater images appear blurred? The answer to these puzzles is given in Treibitz & Schechner [40]. Due to the quantum nature of light (photons), the additive component B induces random photon noise in the image. To understand this, recall that photon flux from the scene and the detection of each photon are Poissonian random processes [41]. This randomness yields an effective noise. The overall noise variance [41–43] of the measured pixel intensity is approximately 2.3Here κ and g are positive constants, which are specific to the sensor. Due to equations (2.2) and (2.3), B increases I_{total} and thus the noise^{1} intensity. The longer the distance to the object, the larger B is. Consequently, the image is more noisy there, making it more difficult to see small object details (veiling), even if contraststretching is applied by image postprocessing. As analysed in Treibitz & Schechner [40], image noise creates an effective blur, despite an absence of blur in the optical process: the recoverable signal has an effective spatial cutoff frequency, induced by noise.
To recover L_{object} by inverting equation (2.1), there is first a need to decouple the two unknowns (per image point) S and B, which are mixed by equation (2.2). This is where P^{4} becomes helpful. Let images be taken through a cameramounted polarizer. As we will see below, polarization provides two independent equations, to solve for the two mentioned unknowns. Typically, the path radiance B is partially polarized [6,16,23,28,39,44]. Hence, polarizationfiltered images can sense smaller or higher intensities of the path radiance B, depending on the orientation of the cameramounted polarizer (figure 1), relative to the polarization vector of the path radiance. There are two orthogonal orientations of the polarizer for which its transmittance of the path radiance reaches extremum values B^{max} and B^{min}, where B = B^{max} + B^{min}. At these orientations, the acquired images are 2.4Equation (2.4) assumes that the polarization of the object signal is negligible relative to that of the path radiance. In some cases this assumption may be wrong, particularly at close distances. However, in many practical cases, the approximation expressed by equation (2.4) leads to effective scene recovery. The two equations (2.4) provide the needed constraints to decouple the two unknowns S and B, as described in Kaftory et al. [31], Namer et al. [32], Schechner et al. [34] and Treibitz & Schechner [35].
(b) Inversion
The above model applies both to natural lighting (which is roughly uniform along horizontal lines of sight) and artificial illumination. An example of the latter is shown in figure 2. Here, a polarizer was mounted on the light source, and another polarizer was mounted on the camera. Rotating one of the polarizers relative to the other yielded two images modelled by equation (2.4). They were processed to recover S.
To recover L_{object} by inverting equation (2.1), there is first a need to know t(z) per point. How can t(z) be assessed? In natural lighting, where the lighting along the LOS is roughly uniform [33,34], 2.5Here B_{∞} is the value of B in an LOS which extends to infinity in the medium. This value can be calibrated in situ [32–35]. Recall that B is recovered using P^{4}. Consequently, based on equations (2.1) and (2.5), the object radiance can be recovered approximately by 2.6This concludes the descattering process.
An example of descattering in haze (dehazing) is shown in figures 3 and 4. The recovered image [31] has strong contrast everywhere and vivid colours. The colours are recovered, since the degradation inversion is done per colour channel (wavelength band). This automatically accommodates the dependency of t, B_{∞} and the DOP on the wavelength band. Despite this, the recovered result is very noisy in the background. This noise is not owing to the mathematical dehazing process. Fundamentally, it is owing to the poor background signal to noise ratio (SNR) of the raw data: owing to the small value of t(z) at large z, the signal (equation (2.1)) is low. Moreover, at these areas the noise is high, owing to increased photonnoise induced by a high B, as described above (equations (2.2) and (2.3)) and in Treibitz & Schechner [40,45]. The noise can be countered in a way described in §3.
3. Threedimensional recovery
(a) Distance map
Descattering based on P^{4} has an important byproduct: a threedimensional mapping of the scene. This outcome is explained in this section. As we now show, the path radiance can be estimated based on two polarizationfiltered frames. Using equation (2.4), 3.1where p is the DOP of the path radiance. This parameter can be calibrated based on images taken in situ [32–35]. Equation (3.1) yields the estimated path radiance 3.2This is an approximate relation, since it relies on the assumption that the object signal is unpolarized.
As an example, figure 5 shows a negative image of the path radiance estimated [31] based on the images shown in figure 3. This appears to represent the distance map of the scene: darker pixels are farther from, while brighter pixels are generally closer to the viewer. Actually, there is indeed an equivalence (up to a couple of scale parameters) between B, the distance z and the medium transmissivity t: the transmissivity monotonically decreases with z, while B monotonically increases with z (see equation (2.5)). In case of a uniform medium, the transmissivity is 3.3where the constant β is the attenuation coefficient of the medium. This constant coefficient is inversely related to the attenuation distance in the medium: the scale of β is approximately (0.1–1) m^{−1} underwater and approximately (10^{−4}–10^{−3}) m^{−1} in haze.
Using equations (2.5), (3.3) and (3.2), the distance map of the scene, z(x, y), can be estimated as 3.4Equation (3.4) shows how P^{4} yields an estimate of the distance per image point. This estimation depends on several global parameters (p, β, B_{∞}) which can be assessed based on the image data [32,34,35]. The distance map z(x, y) expresses the observed threedimensional structure of the scene.
There are additional ways in which P^{4} can help in threedimensional recovery. Shape is often derived using triangulation cues: stereoscopic vision and parallax created by motion. It can be helpful to fuse the polarization cue with triangulation (parallax), when seeking threedimensional recovery of a scene in scattering media. This is done in Sarafraz et al. [46]. Binocular stereo helps the recovery particularly when the path radiance has low DOP. On the other hand, polarization helps the recovery irrespective of object texture or surface markings to which stereoscopic recovery is sensitive. Moreover, a stereoscopic setup can simultaneously acquire two polarizationfiltered images, e.g. using a distinct polarizer orientation per camera. This is helpful in dynamic scenes. Sarafraz et al. [46] models the image formation process by combining stereoscopic viewing geometry and scattering effects. The recovery is then expressed as inversion of this model.
An additional way to exploit triangulation for shape recovery is by using structured illumination. In this method, one of the stereo cameras is replaced by a light source, which projects a narrow beam on the scene. Triangulation between the illuminated object spot and the LOS yields the object distance, per image point, while the beam scans the scene. Triangulation becomes more complicated in scattering media, owing to backscatter. Hence, Gupta et al. [30] incorporated backscatter polarization into analysis of structured illumination.
(b) Distance adaptation of noise suppression
We now describe an additional benefit to the observation that the distance, transmissivity and path radiance can be estimated using P^{4}. In figure 4, the dehazed image is noisy in the background, owing to the high values of B and low values of t there [34,35]. Image denoising can enhance images: it attenuates the noise, but this comes at a cost of resolution loss [40]. Consequently, if denoising is to be properly applied in the recovery, then it should only be mild (or not be applied at all) in the scene foreground, where the SNR is high. This way, foreground areas suffer no resolution loss. On the other hand, background regions are very noisy, and may thus benefit from aggressive noisecountering measures. In short, descattering that accounts for noise should apply denoising measures that are adaptive to the object distance (or medium transmissivity). Fortunately, the distance and transmissivity can be derived from the polarization data, as described above.
This principle is the basis of descattering methods that are described in Kaftory et al. [31], Schechner & Averbuch [47]. There, the recovery is not done by directly using equations (2.6) and (3.2). Instead, the recovery task is expressed as an optimization problem. As commonly done in optimization formulation, first, a cost function Ψ is defined for any potential (and generally wrong) guesses of L_{object} and B, which are denoted by L_{object}^{potential} and B^{potential}, respectively. The sought optimal solution is the one that minimizes Ψ: 3.5The optimal solution should fit the model (2.4) and (3.1) to the data {I^{max}, I^{min}} but at the same time, the sought fields should be spatially smooth, not noisy. The two conflicting requirements: data fitting versus smoothness (regularization), are expressed as terms in the cost function 3.6Until this point the optimization formulation is standard. The novelty in Kaftory et al. [31] and Schechner & Averbuch [47] is that the conflicting requirements (fitting and smoothness) are mutually compromised in a way which depends on the distance: at farther objects, smoothness is more strongly imposed.
For example, in Schechner & Averbuch [47], the following operator is used to measure nonsmoothness, and thus increase the cost function at noisy potential solutions: 3.7Here is the twodimensional Laplacian operator. A more unsmooth result increases the absolute output of the Laplacian. This increases Ψ. Adaptivity to the object distance is achieved by the weighting operator . It depends explicitly on the transmissivity t at each pixel, hence it is implicitly adaptive to the object distance z. Let 3.8Recall from equation (3.3) that t(x, y) ∈ [0,1]. Thus, the weighting emphasizes the regularization (hence smoothness) at points corresponding to distant objects (where t(x, y) → 0), and turns off the regularization at close objects (where t(x, y) → 1). Kaftory et al. [31] uses a more sophisticated regularization term. In any case, the optimal solution (3.5) is found by numerical algorithms that run on a computer. A result [31] is shown in figure 6. Compared with figure 4, the result in figure 6 suppresses noise in the background, without blurring the foreground.
(c) Threedimensional object shape, without a medium
Section 3a shows that a medium that scatters partially polarized light encompasses information about the threedimensional structure of the objects behind it. However, often objects are very close, without a significant medium between them and the camera or eye. Then, other methods should be used to assess the threedimensional shape of objects. Many established methods rely on triangulation, photometric stereo, shading and shadows. Nevertheless, new methods have recently been developed for threedimensional recovery, that rely on partial polarization of reflected light.
Rahmann & Canterakis [48] observed that the polarization of reflected light yields constraints on the shape of opaque, specular objects. Consider figure 7. An incident light ray is specularly reflected from a surface. The incident ray, the specularly reflected ray and the surface normal all reside in the same plane, termed the plane of incidence (POI). Assuming the incident ray to be unpolarized, the specular reflection is partially polarized. The orientation of the polarization vector is perpendicular to the POI. Hence, measuring the orientation of the polarization vector imposes one constraint on the surface normal. Furthermore, the DOP of specularly reflected light depends on the angle of incidence, which equals the angle between the surface normal and the LOS. For a given material, this dependency is known, since it is dictated by the Fresnel coefficients. Hence, measuring the DOP imposes a second constraint on the surface normal. Integrating the constraints on the surface normals of all observed points yields an estimate of the threedimensional shape [48]. A similar approach was developed for recovering the shape of specular transparent objects [49], where internal reflections inside the object had to be accounted for.
Recovering the shape of diffuse objects with the help of P^{4} has been done in Atkinson & Hancock [50]. In diffuse reflection, the orientation of the polarization vector and the DOP provide constraints on the surface normal, which are different from the constraints stemming from specular reflection. Anyway, polarizationbased shape cues [48–50] do not rely on prominent object features, and apply even if the objects are textureless. Hence, they complement triangulationbased methods of shape recovery.
4. Separation of semireflections
Semireflections pose a challenge to vision, either biological or artificial. In nature, such reflections exist in a water–air interface, when looking into or out of water. In manmade environments, they are created by glass windows,^{2} which superimpose a scene behind the window and a reflection of a scene in front of the window, as illustrated in figure 8. This creates confusing images, as the one shown in figure 9. Several methods were developed to attack this problem. Some of them are based on motion and stereo vision [51,52], but require extensive computations. A simple, straightforward approach is to use P^{4}. A semireflecting interface affects the polarization state of reflected and transmitted light in a different manner. This difference can be leveraged to separate [53,54], recover, and label the reflected and transmitted scenes.
The object behind the semireflector is transmitted, thus variables associated with it are denoted by ‘T’. An object on the cameraside of the semireflector is reflected, thus variables associated with it are denoted by ‘R’. Specifically, L_{T} is the radiance of the transmitted object (figure 8). Similarly, L_{R} is the radiance of the reflected object, as measured if the semireflector was replaced by a perfect mirror. The two unknowns L_{T} (x, y) and L_{R} (x, y) affect the sensed image, per point (x, y). Similarly to §2, solving for the two unknowns is facilitated by using two independent equations, which are based on two polarizationfiltered images.
(a) Thin reflector
As we describe later in §4b, semireflections may be accompanied by multiple spatial shifts. For simplicity, let us ignore, for the moment, this effect. Simple reflection (without observed spatial shifts) occurs in an air–water interface. It also occurs in reflection by thin windows. A ray incident on the reflecting surface is partly reflected and partly transmitted. All rays are in the POI (see figure 8). The POI sets two polarization components: parallel and perpendicular to the POI. For each of these components, the semireflector has, respectively, reflectance and transmittance coefficients, r_{}, r_{⊥} and t_{}, t_{⊥}. Using a polarizationsensitive camera, we may take images of the compound scene. The cameramounted polarizer is oriented at the corresponding parallel and perpendicular directions (figure 8), yielding two images, which are modelled by Schechner et al. [54] 4.1Equation (4.1) assumes that light emanating from the scene objects (prior to reaching the semireflector) has negligible partial polarization. In other words, all polarization effects are caused by the semireflector. Accordingly, the light intensity carried by L_{T} is equally split between I_{⊥} and I_{}. The same split applies to L_{R}. In some cases this assumption may be wrong, particularly when the objects are shiny. However, the approximation expressed by equation (4.1) often leads to effective scene recovery.
The two equations (4.1) provide the information needed to decouple the two unknowns L_{T} and L_{R}, as described in Schechner et al. [54]. If the coefficients {r_{}, r_{⊥}, t_{}, t_{⊥}} are properly set, then simply solving the two linear equations (4.1) yields the two unknown scenes: 4.2We now explain how the coefficients are set. The coefficients {r_{}, r_{⊥}, t_{}, t_{⊥}} are derived from the Fresnel coefficients, while accounting for interreflections inside the semireflector (see [54]). They depend on the media interfacing at the semireflecting surface [54] (typically, water, air and glass), which are known.
The coefficients also depend on the angle of incidence ϕ (see figure 8). If this angle is known, then {r_{}, r_{⊥}, t_{}, t_{⊥}} are uniquely set. However, ϕ is usually unknown, and needs to be estimated based on the raw images {I_{⊥}, I_{}}. This is done in Schechner et al. [54] using the following trick. Any wrong estimation of ϕ yields the wrong coefficients {r_{}, r_{⊥}, t_{}, t_{⊥}}. This, in turn, yields a wrong recovery via equation (4.2). A wrong recovery is rather easily spotted, since it exhibits a significant crosstalk [54] between the estimated images {L_{R}, L_{T}}. Crosstalk is detected computationally, using crosscorrelation. Hence, a simple computational algorithm, which seeks to minimize the correlation between the estimated {L_{R}, L_{T}} automatically rejects wrong solutions. This leads to a correct ϕ.
The method we just described, which uses crosscorrelation to find the mixture parameters, implicitly assumes that the original signals L_{R} and L_{T} are independent of each other. Algorithms that make this assumption for the purpose of signal separation are often referred to as independent component analysis (ICA).
(b) Overcoming visual reverberations
Light rays incident on a window are reflected back and forth inside the glass. Such internal reflections affect the contribution of both sources: a spatial effect is created of dimmed and shifted replications. Figure 9 demonstrates this effect in a real photograph taken via a window. In addition to the superimposed scenes (toys of a star versus a tree in the Sun), a shifted and weaker replica of the Sun and tree is clearly seen. This is caused by internal reflections that take place inside a window. In addition to that clear replica, there is also a replica of the other scene (star). Additional higher order replicas exist for both objects, but are often too dim to see. Overall, the acquired photograph contains a superposition not only of the two original scenes, but also of those same scenes displaced to various distances and in different powers. Prior studies (including the one described in §4a) did not account for this effect. There, the model and algorithms focused on the limit case, in which the displacement between the replicas is negligible. This is not a valid situation in general.
Visual spatial displacements created by optical reflections are analogous to temporal displacements created by reflections of temporal sound and radio signals. In analogy to the displaced replica in our study, a sound reflection creates a delayed echo. In the field of acoustics, this effect is generally referred to as reverberations. Hence, we use the term visual reverberations to describe the effect we deal with.
This section generalizes the model and treatment of semireflections to deal with this effect. Consider figure 10. A light ray from the object L_{R} reaches the window. There, it undergoes a series of reflections and refractions. The internal reflections inside the window create a series of rays emerging from the window. For example, let us derive the intensities of some of these rays. First, the ray L_{R} partly reflects from the front air–glass interface, yielding a ray the intensity of which is L_{R}R, where R is the reflectance of the interface. The ray L_{R} is also partly transmitted into the glass, with transmittance coefficient T. The ray in the glass hits the back glass–air interface, and partially reflects there. This internally reflected ray returns to the front air–glass interface, and is partially transmitted through the interface to the air, towards the camera. Thus, before it emerged into the air, this ray underwent a total of two transmissions through the front interface, and one reflection at the back interface (figure 10). Thus, the emerging ray has intensity L_{R}RT^{2}. Similarly, the intensities of other reflected rays in the system can be derived.
A similar analysis applies to a ray from the object L_{T}, as illustrated in figure 10. This ray also undergoes a series of reflections and refractions, resulting in a series of rays emerging from the window. For example, first, the ray L_{T} partly transmits through the back air–glass interface, and then partly transmits through the front glass–air interface. Thus, these two transmissions yield a ray the intensity of which is L_{T}T^{2}. In addition, the transmitted ray in the glass undergoes two internal reflections (one in the front and the other in the back interface), before it returns to the front glass–air interface, where it is partially transmitted to the air (figure 10), towards the camera. Thus, before it emerged into the air, this ray underwent a total of two transmissions (one at each interface) and two internal reflections (one at each interface). Thus, the emerging ray has intensity L_{T}R^{2}T^{2}. The power of successive secondary reflections rapidly tends to zero.
Since the window is flat, all the rays are in the POI. The distance between successive emerging rays (secondary reflections) is d′. The reflectance coefficients per polarization component are R_{⊥} and R_{}, at each interface of the window.^{3} Similarly, the transmittance coefficients per polarization component are T_{⊥} and T_{}, at any one of the two interfaces of the window.^{4} The closedform expressions for {R_{}, R_{⊥}, T_{}, T_{⊥}} are given in Diamant & Schechner [55].
It is clear from figure 10 that the secondary reflections create a spatial effect. Each object point is sensed simultaneously in different image points, since the energy of an object point is dissipated among different reflection orders. Hence, the transmitted scene undergoes a convolution with a particular pointspread function (PSF). As seen in figure 10, the PSF of the transmitted scene is 4.3when measuring only the polarization component parallel to the POI, while δ is the Dirac delta function. Here d is the distance between successive visual echoes of L_{T}, as received by the camera (in pixels). It is given by d = α d′, where d′ is the physical distance (in centimetres) between secondary reflections, depicted in figure 10, and α is the camera magnification. In this model, each object point corresponds to a parallel set of rays, which in turn corresponds to a set of equally interspaced pixels [55].
Similarly, the PSF of the reflected scene is 4.4when measuring only the parallel polarization component. The perpendicular components also undergo convolutions: the corresponding PSFs h_{R}^{⊥} and h_{T}^{⊥} are derived analogously,^{5} by using R_{⊥} and T_{⊥} instead of R_{}, T_{} in equations (4.3) and (4.4).
The acquired image intensity is a linear superposition of the reflected and transmitted scenes. In §4a, this superposition was pointwise, since in the imaging conditions there, spatial effects were not seen. In contrast, here the superposition is of convolved scenes. Generalizing equation (4.1), the acquired images are 4.5where * denotes convolution.
Solving for the two unknowns {L_{R}, L_{T}} is possible using the two equations (4.5), though the solution is more elaborate than equation (4.2). A mathematical way to extract {L_{R}, L_{T}} is described in Diamant & Schechner [55]. A way to solve the problem is using an optimization formulation, similar to the one described in §3b. In analogy to equation (3.5), the optimal sought solution is 4.6where {L_{R}^{potential}, L_{T}^{potential}} are any potential (and generally wrong) guesses for the reflected and transmitted scenes.
The optimal solution should fit the model (4.5) to the data {I_{⊥}, I_{}}, but at the same time, the sought field L_{R} should be spatially smooth, not noisy (there is no practical need to impose smoothness on L_{T}, since it is inherently less noisy [55]). As an example, consider figure 9. This is I_{⊥}. The image I_{} is somewhat similar. Based on these two images [55], the recovered L_{R} is depicted in figure 11, while L_{T} is shown in figure 12. In both images, the recovery has no apparent crosstalk and no reverberations.
(c) Separating specular and diffuse reflections
A related problem to semireflection from transparent surfaces is reflection from opaque objects. Here, too, the image has two components, as illustrated in figure 7. One component is diffuse reflection. It is strongly affected by the inherent colour of the object, but is rather insensitive to change of viewpoint and the DOP of which is usually low. The other component is specular reflection, the colour of which matches the illumination colour. It is highly sensitive to changes of viewpoint, and its DOP is often significant. Separating and recovering each of these components is important, for several reasons: each of these reflection types gives different cues about the object shape, as described in §3c; specular reflection may confuse triangulation methods, hence removing specularities is useful in this context; and the different colours of these components may confuse vision.
P^{4} is helpful for inverting the diffuse/specular mixture, hence separating these reflection components. Umeyama and Godin [56] acquired images at different orientations of a polarizer (figure 7). This way, it changes the relative contribution of two components in the images. The acquired images then undergo ICA to solve for the two unknown fields: the specular image and the diffuse image. Nayar et al. [57] fuse colour and polarization data to recover the components.
5. Discussion
The methods described above have been demonstrated in manmade systems involving computers. An interesting question is, whether these methods indicate that animals may also recover scenes using their polarizationsensitive vision. Do some animals see farther in a medium, by descattering the scene using polarization? Are there animals that use polarization in order to assess distances to objects, or estimate the shape of objects, even roughly, in the context of some task (e.g. preying or courting)? Can animals that prey through the surface of seas or lakes (some birds, fish, crocodiles), upwards or downwards, use polarization in order to remove semireflections from the surface? Perhaps polarization is used for separating specular from diffuse reflection [19] in nature to provide an advantage in seeking food in the open air?
It is also possible to consider exploitation of these effects to confuse predators. Suppose a predator uses pathradiance polarization for distance assessment. Then, a marine animal that has a polarizing skin would confuse this distance assessment, hence gaining an advantage in avoiding attack by the predator. These are fascinating questions and possibilities. Biological systems do not work as the computational algorithms described in this paper. But, the algorithms prove feasibility: some computational systems can recover scenes using polarization. Biological brains are also computational systems. Perhaps they do it too.
Acknowledgements
This paper surveys the work I have done with several coauthors to whom I am grateful: Yuval Averbuch, Yaron Diamant, Ran Kaftory, Nir Karpel, Nahum Kiryati, Mohit Gupta, Einav Namer, Shahriar Negahdaripour, Amin Sarafraz, Srinivas Narasimhan, Shree Nayar, Joseph Shamir, Sarit Shwartz, Tali Treibitz and Yehoshua Zeevi. Thanks are also due to Justin Marshall, David O'Carroll, Nick Roberts and Nadav Shashar, for the useful discussions at the 2008 Heron Island Polarization Conference. Their insights into the potential relevance to animal vision come across in this paper. Yoav Schechner is a Landau Fellow  supported by the Taub Foundation. This work is supported by the Israel Science Foundation (Grant 1031/08) and the USIsrael Binational Science Foundation (BSF) Grant 2006384. This work was conducted in the Ollendorff Minerva Center. Minerva is funded through the BMBF. This work relates to Department of the Navy Grant N629091014056 issued by the office of Naval Research Global. The United States Government has a royaltyfree license throughout the world in all copyrightable material contained herein.
Footnotes

One contribution of 20 to a Theme Issue ‘New directions in biological research on polarized light’.

↵1 The increase in the intensity (variance) of the noise is not only relative to the signal S. Rather, the absolute level of the noise intensity increases with B, owing to equations (2.2) and (2.3).

↵2 Semireflections are also created by plastic windows. However, transparent plastic materials are more prone than glass to the photoelastic effect [33]. In this effect, stress in the transparent material changes the polarization of light propagating inside the window. This may confuse polarizationbased methods for separation of semireflections.

↵3 These are not the coefficients r_{⊥} and r_{} used in §4a. The relation between any r and its corresponding R will become clear in the following discussion.

↵4 These are not the coefficients t_{⊥} and t_{} used in §4a, since t_{⊥} and t_{} express the overall transmittance of light through the combined effect of both window interfaces.

↵5 Note that if d ≈ 0 (thin window), the scene and model analysed in this section degenerate to those discussed in §4a. Specifically, if d = 0, equation (4.3) degenerates to h_{T}^{} = t_{} δ (x), where t_{} is a transmittance used in §4a. In other words, t_{} = T_{}^{2}(1 + R_{}^{2} + R_{}^{4} …). Similarly, the other coefficients {t_{⊥}, r_{}, r_{⊥}} used in §4a are obtained by using d = 0 in h_{T}^{⊥} , h_{R}^{} and h_{R}^{⊥}, respectively.
 Received June 29, 2010.
 Accepted July 16, 2010.
 This journal is © 2011 The Royal Society