WO1990001844A1

WO1990001844A1 - Wide dynamic range image sensor

Info

Publication number: WO1990001844A1
Application number: PCT/US1989/003346
Authority: WO
Inventors: Ran Ginosar; Oliver Hilsenrath; Yehoshua Y. Zeevi; Ofra Zinaty; Daniel J. Kligler
Original assignee: Sorex Corporation
Priority date: 1988-08-02
Filing date: 1989-08-02
Publication date: 1990-02-22
Also published as: EP0431010A1; EP0431010A4

Abstract

An abstract sensor array (10) includes adaptive sensitivity control circuitry (12) for obtaining multiple outputs at different exposure levels. These are combined to yield an output with enhanced dynamic range. This enhanced dynamic range may be reduced in a manner which preferentially retains edge information in order to maintain high contrast.

Description

WIDE DYNAMIC RANGE IMAGE SENSOR

FIELD OP THB INVENTION

The present invention relates to video imagery and sore particularly to apparatus and techniques for providing enhanced video images.

BACKGROUND OP THE INVENTION

Various types of video image enhancement apparatus and techniques have been proposed. AGC techniques have been employed, inter alia, in video signals for reducing the dynamic range of the video signal by subtraction of its DC level. Such a technique is described in U.S. Patent 4,719,350, wherein linear AGC is applied to the video image on a pixel by pixel basis.

U.S. Patent 4,601,055 describes an image processor which provides, inter alia, neighborhood transforms.

U.S. Patent 4,471,228 describes a solid state image sensor with exposure controller, The exposure control appears to be applied to the entire frame and is not addressed to individual pixels.

U.S. Patent 4,556,908 describes a solid state image sensor including a photodiode array which is pixel addressable in what appears to be a random-accessible manner. A photometric output from a selected group of pixels to a light meter is provided, but pixel addressable sensitivity control is not suggested or provided.

U.S. Patent 4,472,638 describes a two-dimensional solid state image sensor which is able to directly output image information subjected to two dimensional image processing. U.S. Patent 4,734,776 describes a system for extending the dynamic range of a CID array. The CID array is designed to provide two integration times, one short and one long, for every pixel in the array, and to read out both of the signals thus produced. The appropriate signal is then selected for each pixel, and, if it is the signal produced from the short integration time, it is amplified to compensate for the difference in sensitivity between the long and short integration times. Pixel-addressable sensitivity is not suggested or provided.

U.S. Patent 4,473,836 teaches a sensor or sensor array achieving wide dynamic range by means of a log amplifier integrated with a detector. It does not offer any variation of integration time.

U.S. Patent 4,636,865 discloses a detector design which allows discharge of excess photocarriers. This is an improved anti-blooming scheme.

U.S. Patent 4,656,361 discloses an nMOS photodiode array with improved sensitivity and dynamic range achieved through the use of an integrated scheme for sensing photogenerated current.

U.S. Patent 4,609,825 teaches a line-transfer type CCD array providing variable integration time on a row-by-row basis. The variable integration time is achieved through variation of the time between the reset and the signal transfer pulses.

U.S. Patent 4,583,002 discloses a solid-state image sensor which provides sensitivity adjustment in response to the overall image intensity. U.S. Patent 4,669,688 teaches a CID array with an individual amplifier for each row in order to reduce circuit capacitance and to improve the signal/noise ratio.

U.S. Patent 4,775,799 improves the dynamic range of an image scanner through the techniques of digital and analog signal averaging in a manner which adapts to the condition of the input signal.

U.S. Patent 4,706,123 discloses an apparatus for changing the exposure times of individual photodetectors in relation to the amount of incident light on only the individual photodetector. The exposure times are limited to a small number of distinct integration times, typically 2, and is subject to restrictions as to which pixels can be controlled. Each line of detectors oust be accompanied by at least two additional shift registers which is a prohibitive amount of cicuitry for two-dimensional arrays.

S. P. DeWeerth and C. A. Mead, in an article entitled "A Two-Dimensional Visual Tracking Array", presented at the Fifth MIT Conference on Advanced Research in VLSI, March 1988, MIT Cambridge, Massachusetts, U.S.A. and published in the proceedings thereof, disclose a semiconductor photosensitive device which computes, per pixel, the difference between the incident light intensity and the average intensity over some neighborhood. This device is limited by the dynamic range of the individual photodetector and by the complexity of circuitry required per pixel.

E.P. Patent 233.464 appears to show a photodiode array providing selectable feedback amplification for each photodiode signal.

SUMMARY OP THE INVENTION

The present invention seeks to provide improved video image enhancement apparatus which overcomes limitations of the prior art apparatus and techniques discussed above.

There is thus provided in accordance with a preferred embodiment of the present invention, video imaging apparatus comprising a multiplicity of photosensitive detectors arranged in a staring array for receiving input radiation and detector control apparatus for controlling the response of the photosensitive detectors such that they operate in different operative ranges within the total inherent dynamic range thereof, whereby the output of the staring array at any given time provides image information in respect of input radiation extending over an input radiation dynamic range which is greater than the operative input radiation dynamic range of any given detector.

In accordance with a preferred embodiment of the invention, the apparatus for controlling the response of the photosensitive detectors is addressable simultaneously to a multiplicity of individual pixels or a multiplicity of designated groups of pixels in a field.

In accordance with a preferred embodiment of the invention, the staring array is a generally two-dimensional array. Alternatively, the staring array may comprise a one- dimensional array.

Further in accordance with a preferred embodiment of the invention, the detector control apparatus includes means for selectably and variably determining the integration time of each of the multiplicity of detectors.

Additionally in accordance with a preferred embodiment of the invention, the staring array may be embodied on a unitary integrated chip, such as in the form of an array of diffusion diodes on a MOS chip.

In accordance with one embodiment of the invention, the detector control apparatus comprises apparatus for providing two control signals to each of the multiplicity of detectors. The detector is actuated in response to simultaneous receipt or absence of a predetermined combination of control signals.

According to an alternative embodiment of the invention, a control circuit is provided for each detector of the multiplicity of detectors.

Additionally in accordance with a preferred embodiment of the invention, the outputs of the detectors are supplied to apparatus for applying thereto a neighborhood transform in order to preserve image contents notwithstanding the reduction in overall dynamic range produced by operation of the detectors with variable integration times.

Further in accordance with a preferred embodiment of the present Invention, the detector control apparatus comprises apparatus responsive to radiation incident on the multiplicity of photosensitive detectors for governing the response of the individual ones of the photosensitive detectors.

Still further, in accordance with a preferred embodiment of the present invention, the apparatus responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon and in response to radiation incident thereon and in the vicinity thereof.

Moreover, in accordance with a preferred embodiment of the present invention, the detector control apparatus includes apparatus for storing a value representing the response of each of the individual ones of the photosensitive detectors and apparatus for updating the stored values. The detector control apparatus is also operative to cause the multiplicity of detectors to provide an output having a dynamic range less than the dynamic range of the input radiation or having a selectable dynamic range.

There is provided in accordance with a preferred embodiment of the present invention, video imaging apparatus also comprising image processing apparatus receiving the outputs of the multiplicity of photosensitive detectors and control signals supplied to the multiplicity of photosensitive detectors and providing in response thereto modified outputs of the detectors such as would be obtained were the array composed of sensors having a uniform sensitivity and an extended dynamic range. The image processing apparatus is operative to cause the multiplicity of detectors to provide an output having a dynamic range generally equal to the dynamic range of the input radiation or having a selectable dynamic range.

Further, in accordance with a preferred embodiment of the present invention, the image processing apparatus receives control signals representing integration times of the multiplicity of detectors and provides in response thereto modified outputs of the detectors corresponding to a uniform sensitivity.

Additionally, in accordance with a preferred embodiment of the present invention, the multiplicity of photosensitive detectors includes a plurality of detectors, etch sensitive to A different color. The detector control apparatus is operative to control the response of the plurality of detectors to a plurality of colors.

Moreover, in accordance with a preferred embodiment of the present invention, the color detector control apparatus includes apparatus responsive to achromatic intensity at a given image location for determining the response and for providing a single control output governing it for the plurality of detectors at that image location.

Further, in accordance with a preferred embodiment of the present invention, the video imaging apparatus comprises a multiplicity of detectors, column and row control apparatus together determining the sample timing of the plurality of photosensitive detectors and apparatus for reading out a video signal from the plurality of photosensitive detectors. Additionally, each of the multiplicity of detectors comprises column and row control apparatus together determining the precharge timing of the plurality of photosensitive detectors.

Still further, in accordance with a preferred embodiment of the present invention, the column and row control apparatus include apparatus for providing multiple precharges of at least some of the plurality of photosensitive detectors in order prevent accumulation of excessive charge thereat.

There is provided in accordance with a preferred embodiment of the present invention, image processing apparatus comprising apparatus for receiving a video signal and for providing an output video signal having a dynamic range narrower than the input dynamic range, while preserving edge information and removing some relatively spatially unvarying information.

Additionally, in accordance with a preferred embodiment of the present invention, the apparatus for receiving and providing video signals comprises at least two neighborhood operator apparati and summation apparatus for combining the outputs of the at least two neighborhood operator apparati.

Further in accordance with a preferred embodiment of the present invention, the apparatus for receiving and providing comprises at least one neighborhood operator and at least one look up table which together receive the input video signal and provide a multiplication factor which is multiplied by the input video signal to provide the output video signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention will be understood and appreciated more fully from the following detailed description taken in conjunction with the drawings in which:

Fig. 1 is a block diagram illustration of a video

Imaging system constructed and operative in accordance with a preferred embodiment of the present invention;

Fig. 2A is a simplified block diagram of a detector array and associated detector control apparatus in accordance with one embodiment of the invention;

Figs.2B and 2C are simplified circuit diagrams of two alternative embodiments of detector elements useful in the present invention;

Fig. 20 is an illustration useful in the understanding of the operation of the detector array and control apparatus of Fig.2A;

Fig. 2B is a seri es of t iming di agrams illustrating the operation of the apparatus of Fig. 2A;

Figs. 3A and 3B are simplified block diagrams of a detector array and associated detector control apparatus in accordance with an alternative embodiment of the invention;

Fig. 4 is an electrical diagram of detector control apparatus for a given detector in the embodiment of Figs. 3A and 3B;

Fig. 5 la a pictorial illustration of the relationship between pixels processed in a typical neighborhood transform used in an embodiment of the invention; Fig. 6 is an electrical diagram of circuitry useful in carrying out a neighborhood transform in accordance with an embodiment of the invention;

Fig. 7 is a simplified block diagram of control circuitry suitable for controlling the apparatus of Fig. 2A;

Figs. 8A and 8B are simplified block diagrams of two alternative embodiments of control circuitry suitable for controlling the apparatus of Fig. 2A;

Fig. 8C is a block diagram of circuitry useful in the circuitry of Figs. 8A and 8B;

Figs. 9A and 9B are simplified block diagrams of two alternative embodiments of control circuitry suitable for controlling apparatus similar to that illustrated in Fig. 2A for full color operation; and

Figs. 10A, 10B and 100 are simplified block diagrams of three alternative embodiments of image processing circuitry useful in the present invention.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

Reference is now made to Fig. 1. which is a schematic block diagram illustration of a video imaging system constructed and operative in accordance with one preferred embodiment: or the present invention. The system comprises α generally two-dimensional solid state detector array 10, which is interconnected with adaptive sensitivity control apparatus 12. The output of the detector array 10 is supplied via a suitable bus 14 to video readout control circuitry 16, which may comprise conventional raster scan circuitry or alternatively intelligent scan circuitry as described and claimed in Israel Patent Application 83213.

Fig. 2A illustrates the interconnection between the adaptive sensitivity control apparatus and the detector array 10 in accordance with one preferred embodiment of the invention. It is seen that each detector element 20, corresponding to an Individual pixel, is provided with connections to both a row sample control circuit 22 and a column sample control circuit 24, both of which may employ shift registers.

Fig. 25 illustrates the specific interconnections of each detector element 20. Here it is seen that first and second AND gates 30 and 32 each have a first input connected to column sample control circuit 24 (Fig. 2A) via respective precharge (P) leads 34 and sample (S) leads 36 and have a second input connected to row sample control circuit 22 (Fig. 2A) via a lead 38. The number of leads for each detector element 20 may be reduced to two if a local delay circuit for determining the time between sample and precharge is provided.

The photodetector element such as a photodiode 40 corresponding to each pixel is coupled to a precharging voltage

V_P via a P switch 42. governed by the output of AND gate 30. and is coupled via an S switch 44, governed by the output of AND gate

32, to a buffer 46 and thus to a capacitor 48. Alternatively, it is possible to reverse the location of switch 44 and buffer 46.

It is also possible to reverse the location of buffer 46 and capacitor 48. It is to be appreciated that each pixel cell may include additional circuitry not shown in Fig. 2B. The circuitry of fig.2B, is however, all that is required for an understanding of the present invention.

It may thus be appreciated that the integration time of each photodetector element corresponding to a pixel is governed by the provision of appropriate electrical signals along leads

34, 36 and 38. Initialization of integration is provided when AND gate 30 simultaneously receives inputs along leads 34 and 38, thus precharging the photodiode 4θ. Termination of integration is provided when AND gate 32 simultaneously receives inputs along leads 36 and 38, thus closing switch 44 and coupling the photodiode via buffer 46 to capacitor 48.

An alternative embodiment of the detector element 20 eliminates AND gate 32 and its connection, via load 38. to the row control circuit 22. The S lead 36, connected to column control circuit 24, is directly connected to S switch 44. Thus, all pixels in a given column are simultaneously sampled and their exposure times are controlled by the precharge signals alone. It will be appreciated that in the embodiment of Figs. 2A and 2B, the selection of the integration time for each pixel is governed externally of the detector array 10 by the row and column sample control circuits 22 and 24. Circuits 22 and 24 are preferably embodied as binary data registers, such as shift registers.

It will be appreciated that the embodiment of Figs. 2A and 2B may be applied to groups of pixels as well as to individual pixels.

The embodiment of Figs. 1, 2A and 2B may be applied also to modified CCD devices. The overall chip architecture remains as is shown in Fig. 2A but the detector element 20 is as shown schematically in Fig. 2C. In this alternative embodiment, the pbotodetector element is a MOS depletion capacitor 100, commonly used in COD detector arrays. As in the previous embodiment, the P switch 42 releases a precharging voltage V_P when AND sate 30 simultaneously receives inputs along leads 34 and 38. Precharging voltage V_P typically removes any accumulated photocharges from the capacitor 100;

Unlike the previous embodiment, the sampling circuit comprises a conventional CCD transfer gate 102. A complete line (row or column) of such photo-sites is triggered simultaneously to transfer their charges to a parallel line of a CCD shift register 104. In due course, the CCD shift register 104 transfers the charge out of the photo-site. Thus, each detector element 20 receives its own individual precharge but all detector elements 20 along a line are sampled together.

This embodiment of the detector element 20 is similar to that of conventional CCD arrays with the exception that each detector element 20 is independently precharged by its own precharging circuit controlled by AND gate 30.

It is a feature of the present invention that each pixel la independently precharged or sampled through switching of the signals on the column and row control leads 34, 36 and 33. The switching operation is illustrated in Fig. 20. The signals are typically determined by control words externally loaded into registers 110 and 112 of row and column control circuits 22 and 24 (Pig. 2A), respectively. The control words typically are comprised of a sequence of 1-bit values, typically of either 1 or 0. The control word of column register 112 typically comprises a 1 at a column address j to be precharged and 0's at the remaining column addresses. The row control word loaded into register 110 simultaneously with the control word for column j comprises a 1 in each row address i at which a pixel (i.j) of detector array 10 is to be precharged. In Fig. 2D the letter P is placed at those pixels which would be precharged given the column and row control words shown.

The column control words are typically designed such that the 1 is shifted sequentially through register 112, thereby sequentially activating the columns of the detector array 10. The loaded row control word corresponds to the pixels to be precharged in the activated column. This cycle of the column precharge signal 1 through the column control register 112, while a new row word is loaded for each column, will be known hereinafter as a "column cycle". In order to produce a plurality of integration times, the column cycle is repeated a plurality of times where each successive column cycle corresponds to a shorter integration or exposure time. Since the pixels of a given column have varied integration times, the row control word loaded for a given column changes with each column cycle. The repetitions end once all of the pixels have completed their reset cycle. At that point, the pixel photocharga values are sampled and their photocharge is transferred out of the detector array 10.

The sampling is performed on a column-by-column basis in order to maintain a constant integration time, for a given precharge cycle, over all the columns. Thus, the column cycle is repeated once more to produce the sampling. A separate column sample register is typically included in column control circuitry 24 for this purpose. In the first embodiment, shown in Fig. 2B, a row word of all 1's is loaded into the row control register before sampling commences. For the other embodiments, whose S switch 44 is not connected to row lead 38, a row word generally is not loaded.

Fig. 2E illustrates the precharging process described hereinabove and is an example schematic timing diagram for the pixels (i.j) through (i+2, J+1) of detector array 10. Fig. 2S presents the precharge signals for the two columns, j and j+1, and for the three rows, i, i+1 and i+2, as well as the sample signals for the two columns.

A pixel is precharged when its column and row addresses simultaneously receive the falling edge of their respective precharge signals. Thus, integration of the photon flux for each pixel begins with a precharge signal which is common to both the row and the column in which the pixel resides. Integration ends on the rising edge of the sample signal for the column in which the pixel resides.

Fig. 2E illustrates that each column, receives its column precharge signal once per column cycle and that, for a given choice of integration time, column j is always precharged before column J+1. The timing of the row precharge signals is not as regular as that for the columns and depends on the integration time desired for each pixel. As shown in Fig. 2E. there are three example integration times, T1, T2 and T3 where T3 is the longest and corresponds to the first column cycle.

Although not illustrated in Fig. 2E. it is possible for a pixel to be precharged a multiplicity of times. Each time the pixel la precharged, its accumulated photocharge is discharged. Thus, charge Integration for sampling begins at the final precharge. It will be appreciated by those skilled in the art that this mechanism is operative to prevent blooming in pixels subjected to high photon flux, corresponding generally to bright areas of the optical image, which would tend to cause saturation at those pixels in conventional sensor arrays.

Since video readout is independent of the precharge/sample procreinabove, integration times

are typically of arbitrary length. In the preferred embodiment of the present invention, useful for standard video applications, integration time is equal to or less than the video frame rate of typically 30 frame/sec. The integration time resolution of detector array 10 depends on the length of time required for a single column cycle. For example, detector array 10 could be a 512 × 512 pixel detector array operating at a 10 KHz clock rate with a 16-bit data bus for loading in the control words. The detector elements 20 are the CCD type illustrated in Fig. 2C. The example detector array 10 will take 3.2 usec to load a single 512-bit row word and 1.6 msec to step through all 512 columns of the array once. To increase the speed of the detector array 10, a faster clock rate and/or wider data busses may be utilized.

According to an embodiment of the invention, selection. of the integration time for each pixel is governed Internally of the detector array 10 by adaptive sensitivity control circuitry associated with the photodetector elements corresponding to each pixel. This embodiment is illustrated in Figs. 3A and 3B.

Fig. 3A illustrates a detector array 50 wherein each pixel, module 51 communicates via a bus 52 with a video readout control circuit 54, which may be similar to circuit 16 (Fig. 1). Fig, 3B illustrates the structure of each pixel module. It is seen that each module 51 comprises a photodetector element such as a photodiode 56 which is coupled via a F switch 58 to a precharging voltage source 60 and via an S switch 62 to a pixel buffer 64, which in turn communicates via bus 52 with the video readout control circuitry 54.

The output V_ij of pixel buffer 64 is also supplied to a neighborhood transform processor 66 which is operative to compute a signal N_ij, which is the neighborhood transform of V_ij and of the output of the pixel buffer 64 for the 8 neighbors of pixel ij , as illustrated in Fig. 5. Alternatively another type of nei ghborhood transfers, may be employed, such as one which includes only the two vertical and two horizontal neighbors only or as a further alternative, a neighborhood lar ger than 3 × 3. In the case of a linear scan, a neighborhood incorporating only the neighbor a along the scan line may be employed.

The signal N_ij is supplied to an adaptive sensitivity control circuit 68, which provides control outputs to switches 58 and 62.

A preferred embodiment of neighborhood transform processor 66 is shown in Fig, 6. The circuitry of Fig, 6 indicates that the light intensity values from the eight pixels which abut on a given pixel are averaged by an averaging circuit 90, such as an 8-input differential amplifier or a resistor network as described in the article entitled "A Two-Dimensional Visual Tracking Array" by S. P. DeWeerth and CA. Mead. The output of averaging circuitry 90 is supplied to a differential amplifier 92, which also receives the video output signal V_ij from the nominal pixel (i,j), via a 1 + epsilon amplifier 94. The output of differential amplifier 92 is the neighborhood signal N_ij for pixel (i.j).

According to an alternative embodiment: of the invention, any suitable neighborhood transform may be applied to obtain a neighborhood signal N_ij.

A preferred embodiment of circuit 68 is illustrated in

Fig. 4. As seen in Fig. 4, a differential amplifier 70 receives αignαl N_ij from neighborhood transform processor 66 and a reference voltage V₁ from an external source. The output of differential amplifier 70 is proportional to the difference between N_ij and V₁ and governs the operation of a transistor 72, which in turn governs charging of a capacitor 74. A switch 76 resets capacitor 74 when provided with a P signal.

The capacitor 74 is coupled to one input of a comparator 80 which also receives a reference voltage V_r. The output of comparator 80 is supplied to a first one shot multivibrator 82 which provides a control signal to switch 62

(Fig. 3B). The output of outs shot 82 is also supplied to a second one shot multivibrator 84, which provides a control signal to switch 58 (fig. 3B) which is also operative to reset switch 76.

The circuitry of Fig..4 defines a first order negative feedback loop having a gain less than unity in order to avoid oscillations. Since the neighborhood signal N_ij is monotonic in V_ij, the circuitry of Fig.4 is operative to compress the dynamic range of the image. Thus for a given cycle, the higher the image video signal is, the shorter the integration time for the next cycle will be, producing a lower V_ij in that subsequent cycle. Higher order feedback loops may also be used for this purpose.

Instead of a photodetector which operates cyclically, it is possible to employ a photosensitive transistor, as described in the article entitled "A Two-Dimensional Visual Tracking Array" as referenced hereinabove.

Reference is now made to Fig. 7 which illustrates a second embodiment of the control circuitry. In this embodiment of the present invention, the adaptive sensitivity control circuitry 12 is partially internal to a sensor chip 119 and partially external to it. The internal circuitry of sensor chip 119 comprises the detector array 10 and the elements directly controlling it, inter alia, the row and column control circuitry 22 and 24, respectively, and video readout control circuitry 16. The adaptive control circuitry 12 external to the chip are those elements which determine the integration time for each pixel on the detector array 10 and which determine the wide dynamic range video signal which is typically the output of the video imaging system of the present invention.

The sensor chip 119 typically operates as described hereinabove for the individual elements. A timing control 120 defines several, typically 3, integration times for the array pixels such that a reasonably wide range of integration times is defined. An example set of integration times of 0.1, 1.6 and 25.6 msec yields a 256:1 range within the constraint of a 30 msec frame rate. To achieve these integration times typically requires a 25 MHz clock and a 64-bit data bus.

Row control words corresponding to the set of integration times are defined as follows. A row control memory 122 stores the row control word for each of the H columns of the H × N detector array 10 for each of r repetitions, where r is also the number of integration tides. For the example set of integration times, each pixel has associated with it one of three entries, either [0,0,1] for the shortest integration time, [0,1,0] for the intermediate length integration time and [1,0,0] for the longest integration time. Row control memory 122 additionally stores the chosen integration time for use in processing described hereinbelow.

Timing control 120 causes the column control 22 to cycle through r column cycles while indicating to row control memory 122 the appropriate row control word R_j to load Into row control register 110. It will be appreciated that it may be necessary to have a wait state between column cycles to achieve the requested integration times. Once the repetitions have finished, timing control 120 sends the appropriate sample signal to the column control 24 to cause the array to be sampled.

It will be appreciated that all of the pixels of a given column are simultaneously sampled and that a gap between the time a given column is sampled and the time its lefthand neighbor is sampled is maintained in order to maintain uniformity of pixel integration times across detector array 10. For the example given above, the gap is 3.2 usec.

After sampling, the pixel intensities are read by the video readout control 16. The video output of readout 16 is amplified, in an amplifier 126, and digitized, in A/D converter 128, to produce V_ij for use in an integration time update and extended dynamic range video output calculation operation. The algorithm utilized in the operation is disclosed in Israel Patent Application 87306, commonly owned by the common owners of the present application. Briefly, the algorithm defines the value for each pixel as that which was received when the pixel detector element 20 was operating in the optimal, linear range. The algorithm then multiplies (or shifts) each value by the inverse of the Integration time which produced the value within the linear range of the pixel detector element 20. The pixel values are then combined into a single, expanded dynamic range picture.

Accordingly, a comparator 130 determines whether or not

V_ij is within the linear range of the detector element 20. If it is, the entry for pixel (i,j) stored in row control memory 122 remains unchanged. If V_ij is out of the linear range, T_ij is incremented or decremented to the next higher or lower integration time, depending on whether it was respectively below or above the linear range, by a Change T_ij block 132 and the entry for pixel (i,J) is changed accordingly. If V_ij is very far out of the linear range, the change in T_ij may not be sufficient and therefore, T_ij will need to he adjusted again at the next frame. This will continue until V_ij is within the linear range or until T_ij has assumed an extreme value and cannot be adjusted any further. Thus, there may be a lag of r-1 frames in adjusting the sensitivity to sudden temporal changes in scene brightness.

V_ij is also input to a multiplier 134 which is operative to adjust the video output signal for variations in pixel sensitivities, or integration times, over the detector array 10. Multiplier 134 transforms V_ij into V_ij/ T_ij where T_ij is calculated from the appropriate pixel entry in row control memory

122. For integration times which are multiples of 2, as in the example above, multiplier 134 performs a shift operation by the appropriate number of bits . In the current embodiment, multiplier 134 typically produces a 16-bit digital video signal from the 8- bit video signal produced by detector array 10. Standard video display equipment is generally capable of displaying only analog signals of no more than 6-7 bits dynamic range. Therefore, the preferred embodiment of the present invention also Includes a dynamic range compressor 136. Compressor 136 typically comprises a FIFO serpentine buffer 138 typically of size M*(m-1) + m)*16, three m×m convolvers 140, 142 and 144, an adder 146 and a D/A converter 148 for producing the output analog video signal.

For typical 3x3 neighborhoods, FIFO buffer 138 typically stores 2*M + 3 pi xel s al any lime and is lapped lo read out a given 3×3 neighborhood around a given pixel until the entire array has passed through and been processed. Each of the 16-bit values of the given neighborhood is 'sliced' into three 8-bit values known as the high byte, the low byte and the middle byte. The middle byte is that which is produced by removing the four highest and four lowest bits.

The high bytes of the neighborhood are convolved together in convolver 140, the middle bytes are convolved by convolver 142 and the low bytes are convolved by convolver 144. The kernel for the three convolvers 140, 142, 144 is as follows:

-1 -1 -1

-1 8+eps -1

-1 -1 -1

where ops is a small-valued constant stored in block 150.

The outputs of the convolvers 140, 142 and 144 are summed and truncated by adder 146 to produce the 8-bit value which is passed to D/A 148 which, in turn, produces the analog value of the 8-bit signal.

It will be appreciated by those skilled in the art that the result of the operation described hereinabove is to stretch the inherent input dynamic range or the detector elements 20 in detector array 10 from a range of 7-8 bits to fully 16 hits while maintaining an output dynamic range that is compatible with common display and processing means. In an alternative embodiment, the 16-bit digital value read out of multiplier 134 is furnished directly to a digital computer or other image processing means capable of processing the extended dynamic range Signal. As mentioned hereinabove, the stretch is accomplished by choosing one of r possible integration times for each pixel.

It will further be appreciated that, with the exception of the detector array 10 and its column and row controllers 24 and 22, the necessary components are currently available.

Reference is now made to Fig. 8A which illustrates an alternative embodiment of the adaptive sensitivity control 12 of Fig. 7, In this embodiment, the analog video output is directly produced, thus eliminating the need to produce the 16-bit intensity values and to subsequently compress them. The Integration time calculation is modified to increase the number of integration time intervals such that the integration time can be any value which is an integral multiple of the time required for a column cycle. For a minimum column cycle time of 0.16 msec and a frame rate of 30 msec, integration time may be varied over a 200:1 range. By restricting sampling time in 2×2 pixel neighborhoods to be identical over the neighborhood, a range of integration time of 800:1 can be attained.

Sensor chip 119 produces analog video output which is amplified by an amplifier 172 before output to display equipment. As in the previous embodiment, the integration time processing elements of the adaptive sensitivity control apparatus 12 are external to sensor chip 119 and are comprised of generally digital circuits. The external integration time processing elements are typically comprised of on A/D converter 174 for converting the analog video output to digital intensities V_ij for processing, an update integration time block 175 which produces optimal integration times T_i'_j given both the output intensity

V_ij and the integration time T_ij used to produce that output, and a frame timing memory 178 of size M × N × 8 bits for storing the selected integration times as 8-bit values.

The update block 175 comprises a lookup table 176, a multi-tap delay line 180 and an accumulator 182. Lookup cable 176 chooses a preliminary new value, T_ijnew, for the integration time

V_ij. Many different functions of V_ij and T_ij can be used to generate the values, stored in lookup table 176, which determine

T_ijnew. In the preferred embodiment, the following logic is used:

1. If V_ij is at the saturation level, which is typically a value of 255, then T_ij new is set to T_ij /2: otherwise,

2. V_ij new = int( k₁ * V_ij *exp(- V_ij) ) (1) where k₁ is a normalization constant chosen to generate a smooth spread of values of T_ij over the range desired and the function int produces an integer value from a non-integer value.

In accordance with an alternative embodiment, T_ijnew is defined according to equation 2.

T_ijnew = k₂*(i=exρ(1n E_ij/ E_ijtau)) (2) where k₂ is a normalization constant chosen to produce an E_ij which is in a range which is convenient for computations, such as the range between 0 and 65,335, useful for 16-bit computations. and Where E_{i j} is defined in accord ance with a quation 3.

and where k₃ and tau are normalization constants chosen to produce an approx imately logarithmic curve which stretches the low values and condanses the high values of T_{i j} new.

In order to locally enhance edges and to suppress artifacts, T_{i j}new is modified by the average of the new integration times in its neighborhood. Thus, it is typically stored in multi-tap delay line 180 to wait for the neighborhood values to be produced. The neighborhood is then averaged by accumulator 182 to produce T_i'_j, the hew value of the integration time for the next pass.

T_i'_j is stored as an 8-bit number in frame timing memory

178 for use in acquiring the next image. The appropriate row control words which will produce an integration time of T'_{i j} at pixel (i,j) are generated by row control logic 184 as follows and are loaded into row control circuitry 22 as described hereinabove.

Row control logic 181 is shown generally in Fig. 8C. it generates the row word corresponding to column j at a current integration time T_n by reading the T_{i j} values for that column from memory 178 and comparing each or them, in a comparator 300, to T_n, Acounter 302 provides the value T_n in units of the basic timing gap, which is 0.16 msec in the example. If T_{i j} is less than or equal to T_n, then address i in the current row word receives a value of 1, indicating that pixel (i,J) will be precharged on the current column cycle. Otherwise, address 1 receives a 0 value, indicating that pixel (i,j) will continue integrating charge until it is sampled. It will be appreciated that counter 302 is typically decremented each column cycle until a sampling cycle occurs; thus, T_n begins the Integration cycle at its largest value and ends at its smallest value, decrementing by one column cycle each time. Additionally, it will be appreciated that each pixel may be precharged at every column cycle until T_n reaches its integration rime value.

Timing control 120 causes the column control circuitry

24 to precharge each column once per column cycle.

It will be appreciated that the pixel-by-pixel update of the integration times T_ij occurs independently of the precharge and sample operations of the sensor chip 119. The offchip generation of row words is typically very fast; for example, to achieve a 0.16 msec minimal Integration time, a row word of 512 bits must be generated every 300 nsec. A 64-bit bus operating at 25 MHz is capable of producing row words at such a speed.

An alternative embodiment of the adaptive sensitivity control 12 which directly produces analog video output is shown

In Fig, 8B. and differs from the embodiment illustrated in Fig. 8A in the update block 175 which produces the optimal new integration time value, T'_ij.

In accordance with this alternative embodiment, update block 175 comprises a first and a second lookup table. 190 and

192, respectively, and a multiplexer 194.

First lookup table 190 receives the output intensity V_{i j} and the integration time T_{i j} and chooses a value E_{i j} which corresponds to the signals it rsceives . The values E_{i j} are typically generated according to equation 2.

As in the previous embodiment, local edge enhancement and artifact suppression is provided by delay line 180 and accumulator 182 which together produce the neighborhood average E_{i j}avg.

Second lookup table 192 receives E_{i j}avg and chooses a value for T_{i j}new which corresponds to the input E_{i j}avg. The values stored in second lookup table 192 are typically generated according to equation 4.

T new = k₄*(1 = exp(-1nE_{i j}avg/ E_{i j}avg*tau)) (4) where k₄ and tau are normalization constants as described with respect to equation 3 hereinabove.

If V_{i j} is at the saturation level, then the value of T_{i j}new computed in equation 4 is invalid and multiplexer 194 sets

T'_ijto T_{i j}/2. Otherwise, multiplexer 194 sets T'_ijto T_{i j}new.

It will be appreciated that the embodiment of Fig. 8A first determines T_{i j} new and then performs the local edge enhancement and artifact suppression wheras the embodiment of

Fig.8B first determines the neighborhood average I_{i j}avg and then uses it to compute the integration time T'_ij.

id

Reference is now made to Figs. 9A and 9B which illustrate two embodiments of the adaptive sensitivity control 12 for a system which produces color Images. Three sensor arrays

200, 202 and 204, respectively, separately but synchronously generate the red (R), green (O) and blue (B) components of an image. The R, O and B components are individually digitized by A/D converters 236, 208 and 210 respectively and the digital output is converted into the hue (H), saturation (S) end intensity (I) color representation by color converter 212. It will be appreciated that the operations described above can also be performed on output from a single ROB color detector array.

In accordance with a first embodiment of the adaptive sensitivity control 12 for color, the intensity values I_ij are treated exactly as the video output V_ij is treated in the monochrome adaptive sensitivity embodiment of Fig, 7. I_ij is multiplied (or shifted) by a multiplier 214, compress aed by compressor 136 and then recombined, in a combiner 216, with the corresponding hue and saturation values, H_ij and S_ij respectively. to produce the corrected RGB signal for display.

The digital corrected Has signal is subsequently converted to on analog signal by D/A converters 218, 220 and 222.

The intensity value I_ij is also utilized for computation of the integration time T_ij, as V_ij was utilized in the monochrome embodiment depicted in Fig. 7 and described hereinabove. The operation is performed In update integration t ion block 224 which comprises co mparator 130 (Fig . 7) and change T_ij block 132 (Fig. 7) . The new integration time la identical for all three sensor arrays 200, 202 and 204 and is stored in row control memory 122. Thus, a single set of row control words is read out of row control memory 122 to be used for controlling the three sensor arrays 200. 202 and 204.

Alternatively, three separate control circuits can be employed, for each of the three sensor arrays 200, 202 and 204, each similar to one described in Figs. 7 and 8. A second embodiment of the adaptive sensitivity control for color, shown in Fig.9B, is similar to that of the first color embodiment in that the intensity I_ij is used to calculate the now integration tines. However, update integration time block 224 of Fig. 9A is replaced by either of the update integration time blocks 175 of Figs.8A and Fig.8B and row control memory 122 is replaced by frame timing memory 178. Additionally, there is no 16-bit intensity value calculation nor the attendant compressor 136 since the analog video output is directly produced from sensor arrays 200, 202 and 204.

Reference is now made to Fig. 10 which illustrates three alternative image processing means useful for reducing the dynamic range of a wide range input video signal, V, without losing any details of the image. Fig. 10A comprises a small number, typically 3, of convolvers 310, 312 and 314 each receiving a different consecutive portion of the bits of the original pixel word, and together covering all of the bits of the many bit pixel word. Three selectors 311, 313 and 31? each produce one of the consecutive portions of the original many bit word. The convolvers typically use a 3×3 kernel, as described hereinabove in reference to Fig, 7. The results of the convolvers are added together in an adder 316 to produce a smaller dynamic range signal, v.

An alternative embodiment; shown in Fig. 10B, employs a circuit similar to that used in Fig.8B to compute T_ijnew and id comprises an averaging circuit 322 and a lookup table 324. The averaging circuit 322 is typically comprised of a delay line 320 and an accumulator 321 and produces an average neighborhood intensity. Lookup table 324 implements any of the functions described hereinabove. The output of the lookup table 324 is multiplied, in a multiplier 326, with the extended dynamic range input signal V to produce the reduced dynamic range output signal v.

A further embodiment of the dynamic range compressor is illustrated in Fig. 10C. The circuitry is similar to that shown in Fig. 10B with the exception that the lookup table 324 is Placed before the averaging circuit 322.

It will be appreciated by persons skilled in the art that the present invention is not limited to what has been particularly shown and described hereinabove. Rather the scope of the present invention is defined only by the claims which follow:

Claims

Video imaging apparatus comprising;

a multiplicity of photosensitive detectors arranged in a staring array for receiving input radiation; and

detector control means for controlling the response of individual ones of the photosensitive detectors such that they operate in different operative ranges within the total inherent dynamic range thereof, whereby the output of the staring array at any given time provides image information in respect of input radiation extending over an input radiation dynamic range which is greater than the operative input radiation dynamic range of any given detector.

2. Apparatus according to claim 1 and wherein said means for controlling the response of the photosensitive detectors is addressable simultaneously to a multiplicity of individual pixels or a multiplicity of designated groups of pixels in a field.

3. Apparatus according to either of claims 1 and 2 and whrein said staring array is a generally two-dimensional array.

4. Apparatus according to any of the preceding claims and wherein said detector control means includes means for selectably and variably determining the integration time of each of the multiplicity of detectors.

5. Apparatus according to any of the preceding claims and wherein said staring array is embodied on a unitary integrated chip.

6. Apparatus according to any of the preceding claims and wherein said detector control means comprises means for providing two control signals to each of the multiplicity of detectors whereby each detector is actuated in response to simultaneous receipt or absence of a predetermined combination of control signals.

7• Apparatus according to any of the preceding claims 1 - 5 and wherein said detector control means comprises a control circuit for each detector of the multiplicity of detecto rs .

8. Apparatus according to any of the preceding claims and wherein the outputs of the detectors are supplied to apparatus for applying thereto a neighborhood transform in order to preserve image contents notwithstanding the reduction in overall dynamic range produced by operation of the detectors with variable integration times.

9. Apparatus according to any of the preceding claims and wherein said detector control means comprises means responsive to radiation incident on said multiplicity of photosensitive detectors for governing the response of said individual ones of the photosensitive detectors.

10. Apparatus according to claim 9 and wherein said means responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon.

11. Apparatus according to claim 9 and wherein said means responsive to radiation is operative to govern the response of at least one individual one of the photosensitive detectors in response to radiation incident thereon and in the vicinity thereof.

12. Apparatus according to any of the preceding claims and wherein said detector control means includes means for storing a value representing the response of each of said individual ones of said photosensitive detectors and means for updating the stored values.

13. Apparatus according to any of the preceding claims 1 - 12 and wherein said detector control means is operative to cause said multiplicity of detectors to provide an output having a dynamic range less than the dynamic range of said input radiation.

14. Apparatus according to any of the preceding claims 1 - 12 and wherein said detector control means is operative to cause said multiplicity of detectors to provide an output having a selectable dynamic range.

15. Apparatus according to any of the preceding claims and also comprising image processing means receiving the outputs of the multiplicity of photosensitive detectors and control signlas supplied to said multiplicity of photosensitive detectors and providing in response thereto modified outputs of the detectors corresponding to a uniform sensitivity.

16. Apparatus according to claim 15 and wherein said image processing means is operative to cause said multiplicity of detectors to provide an output having a dynamic range generally equal to the dynamic range of said input radiation.

17. Apparatus according to claim 15 and wherein said image processing means is operative to

multiplicity of detectors to provide an output having a selectable dynamic range.

18. Apparatus according to any of the preceding claims 15 - 17 and wherein said image processing means receives control signals representing integration times of said multiplicity of detectors and provides in response thereto modified outputs of the detectors corresponding to a uniform sensitivity.

19. Apparatus according to any of the preceding claims and wherein said multiplicity of photosensitive detectors includes a plurality of detectors, each sensitive to a different color and wherein said detector control means is operative to control the response of said plurality of detectors to a plurality of colors.

20. Apparatus according to claim 19 and wherein said detector control means includes means responsive to achromatic intensity at a given image location for determining the response of said plurality of detectors at said image location.

21. Apparatus according to claim 20 and wherein said detector control means includes means responsive to achromatic intensity at a given im age loca ti on for providing a single control output governing the response of said plurality of detectors at said image location.

22. Apparatus according to any of the preceding claims and wherein each of said multiplicity of detectors comprises:

a plurality of photosensitive detectors;

column and row control means together determining the sample timing of said plurality of photosensitive detectors; and means for reading out a video signal from said plurality of photosensitive detectors.

23. Apparatus according to any of the preceding claims and wherein each of said multiplicity of detectors comprises:

a plurality of photosensitive detectors;

column and row control means together determining the precharge liming of said plurality of photosensitive detectors; and

means for reading out a video signal from said plurality of photosensitive detectors.

24. Apparatus according to claim 23 and wherein said column and row control means include means for providing multiple precharges of at least some of said plurality of photosensitive detectors in order to prevent accumulation of excessive charge thereat.

25. Image processing apparatus comprising: means for receiving a video signal having a first dynamic range and for providing an output video signal having a second dynamic range narrower than said first dynamic range, while preserving edge information and removing acme relatively spatially unvarying information.

25. Apparatus according to claim 25 and wherein said means for receiving and providing comprises at least two neighborhood operator means and summation means for combining the outputs of said at least two neighborhood operator means.

27. Apparatus according to claim 23 and wherein said means for receiving and providing comprises at least one neighborhood operator and at least one look up table which together receive said input video signal and provide a multiplication factor which is multiplied by the input video signal to provide said output video signal.

28. Apparatus substantially as shown and described hereinabove.

29. Apparatus substantially as illustrated in any of the drawings.