Royal Society Publishing

Correlation versus gradient type motion detectors: the pros and cons

Alexander Borst


Visual motion contains a wealth of information about self-motion as well as the three-dimensional structure of the environment. Therefore, it is of utmost importance for any organism with eyes. However, visual motion information is not explicitly represented at the photoreceptor level, but rather has to be computed by the nervous system from the changing retinal images as one of the first processing steps. Two prominent models have been proposed to account for this neural computation: the Reichardt detector and the gradient detector. While the Reichardt detector correlates the luminance levels derived from two adjacent image points, the gradient detector provides an estimate of the local retinal image velocity by dividing the spatial and the temporal luminance gradient. As a consequence of their different internal processing structure, both the models differ in a number of functional aspects such as their dependence on the spatial-pattern structure as well as their sensitivity to photon noise. These different properties lead to the proposal that an ideal motion detector should be of Reichardt type at low luminance levels, but of gradient type at high luminance levels. However, experiments on the fly visual systems provided unambiguous evidence in favour of the Reichardt detector under all luminance conditions. Does this mean that the fly nervous system uses suboptimal computations, or is there a functional aspect missing in the optimality criterion? In the following, I will argue in favour of the latter, showing that Reichardt detectors have an automatic gain control allowing them to dynamically adjust their input–output relationships to the statistical range of velocities presented, while gradient detectors do not have this property. As a consequence, Reichardt detectors, but not gradient detectors, always provide a maximum amount of information about stimulus velocity over a large range of velocities. This important property might explain why Reichardt type of computations have been demonstrated to underlie the extraction of motion information in the fly visual system under all luminance levels.


1. Introduction

In motion vision, two distinct models have been proposed to account for direction selectivity: the Reichardt detector and the gradient detector (figure 1). In the Reichardt detector (also called ‘Hassenstein–Reichardt’ detector or correlation-type motion detector), the luminance levels of two neighbouring image locations are multiplied after being filtered asymmetrically (figure 1a). This operation is performed twice in a mirror-symmetrical fashion, before the outputs of both multipliers are subtracted from each other (Hassenstein & Reichardt 1956; Reichardt 1961, 1987; Borst & Egelhaaf 1989). The spatial or temporal average of such local motion detector signals is proportional to the image velocity within a range set by the detector time constant (Egelhaaf & Reichardt 1987). However, it is one of the hallmarks of this model that the output of the individual velocity detectors depends, in addition to stimulus velocity, in a characteristic way on the spatial structure of the moving pattern; in response to drifting gratings, for example, the local Reichardt detector output consists of two components: (i) a sustained component, which indicates by its sign the direction of the moving stimulus and (ii) a time-varying component, which follows the local intensity modulation and, thus, carries no directional information at all. Since the local intensity modulations are phase shifted with respect to each other, the time-varying components in the local signals are cancelled by the spatial integration of many adjacent detectors. Unlike the time-varying component, the sustained component survives spatial or temporal averaging (integration). The global output signal, therefore, is purely directional. It is predicted to exhibit a distinct optimum as a function of stimulus velocity for each pattern wavelength. The ratio of velocity and spatial wavelength at this optimum corresponds to a certain temporal frequency, which is the number of spatial-pattern periods passing one particular image location each second. Despite a different internal structure, the so-called ‘energy model’ (Adelson & Bergen 1985) is identical to the Reichardt detector with respect to the output signal (van Santen & Sperling 1985) and consequently with respect to all of the above predictions (for review see Borst & Egelhaaf 1993). It has been shown to account for various phenomena of vertebrate vision including that of humans (Borst & Egelhaaf 1989), but shall, for the reasons outlined above, not be treated as a separate model here.

Figure 1

Two competing mechanisms proposed to underlie direction selectivity in fly motion detection. (a) Reichardt detector. It consists of two mirror-symmetrical subunits. In each subunit, the luminance values as measured in two adjacent image locations become multiplied (M) with each other after one of them is delayed by a low-pass filter with time constant τ. The resulting output signals of the multipliers become finally subtracted. (b) Gradient detector. The temporal luminance gradient as measured after one photoreceptor (δI/δt, to the left) is divided by the spatial luminance gradient (δI/δx).

A prominent alternative model for motion detection is called the ‘gradient detector’ (Limb & Murphy 1975; Fennema & Thompson 1979; Hildreth & Koch 1987; Srinivasan 1990). The gradient detector computes the velocity signal by dividing the temporal derivative of local luminance ∂I(x, t)/∂t by its spatial derivative ∂I(x, t)/∂x (figure 1a). Different from the Reichardt detector, the gradient detector provides a signal that is proportional to the image velocity at each point and does not depend on pattern properties. In particular, no modulations are expected in the local signals as long as the velocity is constant, and the velocity dependence of the global signal should not vary with the spatial wavelength of the pattern. The main problem with this computation is that the uncertainty in the computed velocity varies widely. If the spatial derivative ∂I(x, t)/∂x is small, the noise in the temporal derivative is amplified, and for points where the spatial derivative is 0, the velocity is completely undefined. Along these lines, Potters & Bialek (1994) proposed that an ideal motion scheme would be based on the gradient detector only in the high signal-to-noise regime, whereas at low signal-to-noise ratios a Reichardt detector would be superior (Potters & Bialek 1994).

Many studies on the mechanisms underlying direction selectivity have been performed in the fly visual system. This system, therefore, can be said to be the best understood in terms of the mechanism underlying direction selectivity. In summary, all the available evidence supports the Reichardt detector and speaks against the gradient detector. However, neither the actual neurons performing the computations nor the biophysical mechanisms underlying such operations as low-pass filtering or multiplication have been elucidated so far (for review see Borst & Haag 2002). Rather, the studies comprise input–output experiments at the behavioural level as well as the extra- and intracellular recordings from motion-sensitive neurons (Fermi & Reichardt 1963; Götz 1964; Buchner 1976; Single & Borst 1998; Haag et al. 2004). In particular, the pattern dependence of both the local and the global signals of the motion detection system was recently tested at various mean luminances and, hence, at different signal-to-noise levels (Haag et al. 2004); under all conditions, the pattern dependence was evident, showing modulations at the temporal frequency of the pattern in the case of local signals, and characteristic shifts of the optimal velocity with different spatial wavelengths in the case of global signals. This strong experimental support of the Reichardt detector poses the question of the functional advantage this processing scheme offers over the gradient detector. In particular, it suggests that flies, for some reason, have picked a suboptimal solution in evolution since optimal motion detectors, according to Potters & Bialek (1994), switch over from Reichardt to gradient detectors with increasing signal-to-noise, and flies' motion detectors do not. Briefly, the argument presented in the following will be that the major advantage of the Reichardt detector, besides its superior noise suppression at low luminance levels, lies in its intrinsic adaptive properties allowing for an automatic gain adjustment to the statistics of the velocity stimulus and, thus, for a maximum information transmission under all conditions.

2. Material and methods

In all the simulations, an array of 20 detectors covering the two spatial periods of the grating was simulated. The output of all the detectors was added, giving rise to the output signal. The filter in the Reichardt detector was a first-order low pass with a time constant τL of 50 ms. In the gradient detector, the spatial and the temporal derivatives were approximated by the difference between two adjacent input stages of the detector and the difference between two subsequent input signals in time, respectively. In order to avoid division by 0, a small epsilon was added to the absolute value of the spatial derivative in the denominator, and the sign of the spatial gradient was taken care of in the numerator. Calculations were done at 1 ms temporal resolution.

3. Results and discussion

As a meanwhile classical functional difference between the two detector models, the steady-state velocity dependence does not follow the velocity in a linear way in Reichardt detectors, but does so in gradient detectors (figure 2). Moreover, the response of Reichardt detectors is dependent on the pattern wavelength and contrast, and completely independent of pattern properties in gradient detectors. It should be noted that all these statements refer to the steady-state response of the spatial average over an array of detectors. When the output signal of an individual detector is considered, further differences appear: when stimulated by a periodic grating moving at a constant velocity, the output of a single Reichardt detector is not constant but rather modulated by the temporal frequency of the pattern. Owing to phase offsets between the individual detectors, these modulations cancel in the spatial average (Single & Borst 1998; Haag et al. 2004). In contrast, even individual gradient detectors report a smooth signal proportional to local pattern velocity at every point in time, without the need of spatial averaging. In general, thus, Reichardt detectors do not qualify as reporters of local retinal velocity because (i) their output is proportional to the image velocity only within a certain range and only after spatial averaging and (ii) their output depends, in addition to the image velocity, on the spatial structure of the moving pattern.

Figure 2

Steady-state velocity dependence of (a) Reichardt and (b) gradient detectors in response to moving sine gratings (spatial wavelength λ as indicated). The Reichardt detector shows a peaked velocity dependence. For velocities higher than the optimal velocity, the response gradually returns to 0. Furthermore, the optimal velocity is different for different pattern wavelengths. In contrast, the response of the gradient detector follows the pattern velocity in a linear way and is independent of the pattern wavelength.

The above distinctions would suffice to expect that any motion vision system evolved in biology should implement a gradient detector and not a Reichardt detector. However, these statements apply only to ideal noise-free circumstances. And reality is noisy, in particular vision due to the Poisson nature of photon emission. When comparing the response properties of both the detector models under noisy conditions, the major disadvantage of gradient detectors becomes apparent: their superb performance is at the expense of an exquisite sensitivity to input noise. The simulations shown in figure 3 exemplify this point. Here, a sine grating was moved with a white-noise velocity profile, low-pass filtered with a cut-off frequency at 10 Hz. The luminance at each point in time and at each image location was determined by the local luminance of the pattern plus a Poisson-distributed photon noise, corresponding to a luminance level of 1 cd m−2. Such low luminance levels would occur during dusk or dawn, or in a room with dim lights. The same velocity profile was used to move the pattern many times, and the response of the detector array was calculated for each trial. From the resulting response array, the signal and noise power spectra were calculated and are shown in figure 3. It is clearly seen that the Reichardt detector shows a much superior performance under these conditions: here, the signal power is orders of magnitude larger than the noise power, in particular in the low frequency range up to 1 Hz, with the cross-over occurring between 10 and 20 Hz. In contrast, the signal of the gradient detector is drowned in a sea of noise at all frequencies. As a result, the information rate, calculated from the signal-to-noise power ratios, of the Reichardt detector is at approximately 80 bits s−1 under these conditions, whereas the gradient detector's signal is almost 0 bits s−1. An analytical investigation of this point indeed reveals how the internal processing structure of the Reichardt detector explicitly supports noise suppression in the detector, maximizing information transmission by the ‘water-filling principle’ (Lei & Borst 2006).

Figure 3

Signal-to-noise ratios of Reichardt and gradient detector responses at low luminance levels. A sine grating of 100% contrast was moved with a Gaussian velocity fluctuation at a luminance level corrupted by photon noise corresponding to 1 cd m−2. The velocity waveform was created by a white-noise signal with an autocorrelation time constant of 100 ms and a standard deviation σ of 5 Hz. The resulting signal and noise spectra at the output of the detector arrays are shown for both types of detectors. Reichardt detectors perform much better than gradient detectors under such conditions, resulting in high information rates for Reichardt detectors and negligible ones for gradient detectors.

At this point, one would argue that each detector model has its pros and cons: while the Reichardt detector is ideal for operation under noisy conditions, the gradient detector shows superior and uncorrupted velocity dependence even locally. Therefore, the proposal made by Potters & Bialek (1994) made a lot of sense: the ideal motion detector should be of Reichardt type under low luminance conditions where noise is prominent, and switch over to a gradient detector under high luminance conditions when signal-to-noise levels at the photoreceptor input are high. However, as already outlined in §1, the available experimental evidence does not support such a switch over, as plausible as it seems.

The functional considerations dealt with so far have left out an important point in image processing, and this pertains to its adaptive properties. The first experiments addressing this point were performed, again on the fly visual system using the motion-sensitive neuron H1, by Brenner et al. (2000), later confirmed and extended by Fairhall et al. (2001) and Borst (2003). In all the three studies, flies were stimulated with a rigidly moving spatial pattern. The velocity profile had zero mean and a band-limited white-noise spectrum. From repeated stimulations using identical velocity profiles, stimulus–response functions were constructed using the first principle component of the spike-triggered stimulus ensemble extracted from all the responses. Such experiments were performed using identical time courses but different velocity amplitudes. When comparing the resulting stimulus–response functions for the different conditions, the response gain was observed to vary depending on the stimulus used: for high velocity amplitudes (large standard deviations of the velocity distribution), the stimulus–response function was seen to have a small slope or low gain, while for small velocity amplitudes (small standard deviations of the velocity distribution), the stimulus–response function was seen to have a large slope or high gain. Such an adaptive gain seemed to be very useful since it allows the system to dynamically adjust its gain to the prevailing stimulus statistics, making use of its full dynamic range under all circumstances.

The surprising outcome of the model simulations on the Reichardt detector was that such detectors do exactly this, namely to adjust their transfer function automatically to the width of the velocity distribution (Borst et al. 2005). The even bigger surprise was that this adaptation occurs in Reichardt detectors without any parameter adjustment: as an intrinsic property of the nonlinear processing structure together with a temporal filter, no internal parameter needs to be changed in order to exhibit this characteristic. As was shown analytically, the effect relies crucially on the temporal dynamics of the stimulus. While the gain adaptation is always the more pronounced the larger the amplitude of the velocity fluctuations is, another crucial parameter is the ratio of the autocorrelation time of the velocity fluctuations and the time constant in the motion detector filter. With the velocity changing slowly over time (slowly with respect to the detector filter time constant), the gain adaptation is only marginal, whereas with the velocity changing quickly (again with respect to the detector filter time constant), the gain adaptation is pronounced. These considerations could lead to the assumption that the gain is set by image acceleration, but as calculation shows, this is an oversimplification (Borst et al. 2005).

In figure 4, model simulations of an array of Reichardt detectors are shown side by side with experimental results obtained in the H1 neuron. Here, instead of the method reported in Brenner et al. (2000), Fairhall et al. 2001 and Borst (2003), the immediate responses of both the model and the fly neuron are displayed. In order to have comparable output signals, the model response was fed into a leaky integrate-and-fire neuron. This also allowed for an analysis of the information rate using the direct method (de Ruyter et al. 1997) without the prerequisite of Gaussian signal and noise distributions. As can be seen, the adaptive gain makes the signals largely independent of the stimulus variance. Nevertheless, the compensation is not complete in the model responses: to some degree, the information rate still grows with stimulus entropy. This again can be understood when calculating the exact propagation of photon noise through the detector (Lei & Borst 2006) and considering the dependence of the gain adaptation on stimulus dynamics, in addition to the stimulus amplitude.

Figure 4

Dynamic gain control in Reichardt detectors. Using a white-noise velocity stimulus with three different amplitudes (indicated in temporal frequencies corresponding to the number of periods passing one image location per second), the Reichardt detector can be seen to adapt its velocity gain to the stimulus statistics (a). Owing to this adaptive gain control, the information rate rises only by a small amount with increasing stimulus amplitude (c). The same phenomenon is observed in the fly motion-sensitive neuron H1 (b, d; data from Brenner et al. 2000). Note that in this simulation, in order to compare the model and experimental data in a one-to-one fashion and to use identical software to evaluate responses from both sources, the output of the Reichardt detector array was used to drive a leaky integrate-and-fire neuron, resulting in a spike output of the model too. Information was calculated from the spike output according to the method as outlined in de Ruyter et al. (1997) and Borst (2003).

In contrast, gradient detectors do not exhibit such an inherent adaptive velocity gain. As is shown in figure 5, when stimulated with Gaussian velocity distributions of three different amplitudes, the velocity–response curves of an array of gradient detectors are identical for the different conditions: the same linear relationship between velocity and response holds, no matter how large the velocity signals are distributed. While this makes the output of gradient detectors unambiguous with respect to the actual velocity input, it is most unfavourable when the noisiness of the visual input is considered (figure 2). This decisive difference between the two models of motion detection is further exemplified when looking at the distributions of their output signals under different input signal distributions (figure 6). While Reichardt detectors have a rather similar output signal distribution under various conditions (figure 6a), output signals of gradient detectors simply follow the input signal distribution (figure 6b).

Figure 5

Gradient detectors do not show adaptive gain control: when stimulated by white-noise velocity fluctuations with three different amplitudes, identical stimulus–response curves are obtained.

Figure 6

Response histograms of Reichardt and gradient detectors stimulated by white-noise velocity fluctuations with three different amplitudes. Whereas Reichardt detectors exhibit similar response distributions, gradient detector response distributions simply follow the corresponding stimulus distributions.

4. Conclusion

The work summarized above has given a possible explanation for the experimental finding that fly motion vision is based on Reichardt detectors over a wide range of signal-to-noise ratios: the inherent adaptive velocity gain control of Reichardt detectors allows this algorithm to cover a wide range of velocities resulting in a nearly optimal information transmission for different stimulus amplitudes. Nevertheless, the question remains of how a flying animal can deal with the resulting ambiguity of the course control signals with respect to the absolute retinal velocities. Here, it is important to realize that animals work under closed-loop conditions where any steering action is automatically fed back onto the sensory input. When flying straight requires only the zeroing of the rotational input, the absolute values of the error signal might be of less importance. However, a clear answer to this question will require a thorough and quantitative investigation of the transformation of the sensory signal into the motor output, including the biophysics and the aerodynamics of wing motion (Dickinson et al. 1999), as well as the typical innate flight behaviour that flies display under free flight conditions (van Hateren et al. 2005).


  • One contribution of 15 to a Theme Issue ‘The use of artificial neural networks to study perception in animals’.


View Abstract