Publication number | US5604545 A |

Publication type | Grant |

Application number | US 08/272,780 |

Publication date | 18 Feb 1997 |

Filing date | 8 Jul 1994 |

Priority date | 8 Jul 1994 |

Fee status | Lapsed |

Publication number | 08272780, 272780, US 5604545 A, US 5604545A, US-A-5604545, US5604545 A, US5604545A |

Inventors | Chanchal Chatterjee, Vwani P. Roychowdhurry |

Original Assignee | Purdue Research Foundation |

Export Citation | BiBTeX, EndNote, RefMan |

Patent Citations (3), Referenced by (8), Classifications (12), Legal Events (6) | |

External Links: USPTO, USPTO Assignment, Espacenet | |

US 5604545 A

Abstract

An image enhancement system that efficiently enhances gray scale images, wherein an analog scene is captured and a resulting analog signal is digitized to a raw gray scale image S. Image S is divided into M disjoint non-empty regions R_{1} . . . R_{M}, where S_{i} identifies region R_{i} of S. The raw digitized signal S_{i} is processed in accordance with an enhancement algorithm to obtain a set of enhancement parameters (offsets P_{i} =(s_{ai}, s_{bi}) and weights (w_{i})). Digital offsets P_{i} =(s_{ai}, s_{bi}) are converted to analog offsets Q_{i} =(v_{ai}, v_{bi}). The analog signal is stretched and clipped and converted to a digital signal that is an enhanced image signal T_{i}. The enhanced image signal is multiplied by weight w_{i} to obtain the final enhanced image U_{i}. Image U_{i} forms the region R_{i} of a final enhanced image U. The multiplied signals are summed to produce an enhanced summed image of the scene.

Claims(21)

1. An image enhancement system that enhances images, said system comprising:

means for capturing an analog image signal V of a scene;

first means coupled to said capturing means for digitizing the image to a raw gray scale image S;

means coupled to said digitizing means for processing the raw gray scale image S in accordance with

t.sub.i =gs.sub.i +k.sub.1 gs.sub.a +k.sub.2 s.sub.a +k.sub.3 s.sub.i +k.sub.4, i=1 . . . M

where t_{i} is a gray scale value, g is a gain, S_{a} is a low offset, S_{i} is a full range gray scale value, k_{1}, k_{2}, k_{3}, k_{4} are constants, and M is a maximum number of regions to be enhanced,

to obtain a set of digital enhancement offset parameters P and weights w to produce a processed signal;

means coupled to said processing means for converting the digital enhancement parameters P into analog enhancement parameters Q;

means coupled to said capturing means and said converting means for stretching and clipping the analog signal V by analog enhancement parameters Q;

second means coupled to said stretching and clipping means for digitizing the stretched and clipped analog signal V to produce an enhanced image T;

means coupled to said second means for multiplying said enhanced image T by a multiplying weight w; and

means coupled to said multiplying weight w and to said enhanced image signal T for producing an enhanced final image U of the scene.

2. An image enhancement system that enhances images, said system comprising:

means for capturing an analog image signal V_{i} of a scene;

first means coupled to said capturing means for digitizing the image to a raw gray scale image S_{i} ;

means coupled to said digitizing means for processing the raw gray scale image S_{i} in accordance with

t.sub.i =g.sub.i s.sub.i +k.sub.1 g.sub.i s.sub.ai +k.sub.2 s.sub.ai +k.sub.3 s.sub.i +k.sub.4, i=1 . . . M

where t_{i} is a gray scale value, g_{i} is a gain, S_{ai} is a low offset, S_{i} is a full range gray scale value, k_{1}, k_{2}, k_{3}, k_{4} are constants, and M is a maximum number of regions to be enhanced,

to obtain a set of digital enhancement offset parameters P_{i} and weights w_{i} ;

means coupled to said processing means for converting the digital enhancement parameters P_{i} into analog enhancement parameters Q_{i} ;

means coupled to said capturing means and said converting means for stretching and clipping the analog signal V_{i} by analog enhancement parameters Q_{i} ;

second means coupled to said stretching and clipping means for digitizing the stretched and clipped analog signal V_{i} to produce an enhanced image signal T_{i} ;

means coupled to said second means for multiplying weight w_{i} to enhanced image signal T_{i} ;

means coupled to said multiplying means for changing the multiplied enhanced image by activation function f() to produce an enhanced image U_{i} of the scene; and

means coupled to said changing means for combining enhanced images U_{i} to generate a final enhanced image U.

3. The system claimed in claim 2, further including first buffering means whose input is coupled to said first digitizing means and whose output is coupled to said processing means for buffering the input to said processing means.

4. The system claimed in claim 1, further including second buffering means whose input is coupled to said second digitizing means and whose output is coupled to said multiplying means for buffering the input to said multiplying means.

5. The system claimed in claim 2, wherein said capturing means is comprised of an electronic camera.

6. The system claimed in claim 2, wherein said capturing means is comprised of a video camera.

7. The system claimed in claim 2, wherein said capturing means is comprised of a laser.

8. The system claimed in claim 2, further including means for producing L copies of an enhanced image T_{i}.

9. The system claimed in claim 8, wherein said means for producing L copies comprises:

means for decomposing weight w_{i} into a sum of L powers of two weights, w_{ij} ;

means for multiplying image T_{i} with each weight w_{ij} ; and

means for summing the L weighted images.

10. The system claimed in claim 2, further including means for creating a desired gray scale value distribution in each region by varying w_{i}.

11. The system claimed in claim 10, further including means for changing a range of gray scale values.

12. The system claimed in claim 2, further including means for adjusting an illumination of the image.

13. The system claimed in claim 2, further including:

first means for selecting statistical measures of desired gray scale values in the enhanced image;

second means for selecting first order statistical measures such as mean, median, weighted average, or random sample and;

third means for selecting to second order statistical measures such as variance, variation, or range of intensities about the mean.

14. A method of producing an enhanced image, said method comprising the steps of:

capturing an analog image signal V_{i} of a scene;

digitizing the captured image signal to a raw gray scale image S_{i} ;

processing the raw gray scale image S_{i} in accordance with

t.sub.i =g.sub.i s.sub.i +k.sub.1 g.sub.i s.sub.ai +k.sub.2 s.sub.ai +k.sub.3 s.sub.i +k.sub.4, i=1 . . . M,

where t_{i} is a gray scale value, g_{i} is a gain, S_{ai} is a low offset, S_{i} is a full range gray scale value, k_{1}, k_{2}, k_{3}, k_{4} are constants, and M is a maximum number of regions to be enhanced,

to obtain a set of enhanced parameter offsets P_{i} and weights W_{i} to produce a processed signal;

converting the processed signal into an analog signal that has analog enhancement parameters Q_{1} by applying a transform function to V_{i} to both stretch and clip V_{i} ;

digitizing the transformed analog signal V_{i} to produce L copies of an enhanced image signal T_{i} ;

multiplying weight w_{i} to enhanced image signal T_{i} ;

changing the multiplied enhanced image by activation function f() to produce an enhanced image U_{i} of the scene; and

combining the enhanced image U_{i} to generate final enhanced image U.

15. An image enhancement system that enhances images, said system comprising:

means for capturing an analog image signal V_{i} of a scene;

first means coupled to said capturing means for digitizing the analog signal V_{i} to a raw gray scale image S_{i} ;

means coupled to said digitizing means for processing the raw gray scale image S_{i} in accordance with an enhancement algorithm to obtain a set of digital enhancement parameters P_{i} =(s_{ai}, s_{bi}), and weights w_{i} ;

means of obtaining a separate set of enhancement parameters P_{i} =(s_{ai}, s_{bi}), and weights w_{i} for each region i of the image;

means coupled to said processing means for converting said digital enhancement parameters into analog enhancement parameters Q_{i} =(v_{ai}, v_{bi});

means coupled to said capturing means and said converting means for stretching and clipping the analog image signal V_{i} in accordance with ##EQU23## where V_{i} is clipped at its low and high levels v_{ai} and v_{bi} respectively by analog enhancement parameters Q_{i} =(v_{ai}, v_{bi});

second means coupled to said stretching and clipping means for digitizing the stretched and clipped analog signal V_{i} to produce an enhanced gray scale image T_{i} ;

means coupled to said second means for multiplying the enhanced gray scale image T_{i} with weights w_{i} to produce a multiplied enhanced image W_{i} T_{i} ;

means coupled to said multiplying means for clipping the said multiplied enhanced image with the activation function f() in accordance with ##EQU24## to produce a final enhanced image U_{i} and; means coupled to said clipping means for combining enhanced images U_{i} in accordance with ##EQU25## to generate a final enhanced image U.

16. The system claimed in claim 14, further including:

means for producing L copies of an enhanced image T_{i}.

17. The system claimed in claim 15, further including

means of selection M regions for enhancement;

means of applying the enhancement algorithm on each of the said regions to obtain an enhanced image U_{i} ;

means of combining enhanced images U_{i} for each region to obtain final enhanced image U in accordance with ##EQU26##

18. The image enhancement system claimed in claim 14, further including:

means for determining the lower and upper bounds of first and second order measures of desired gray scale values and;

means for selecting measures of desired gray scale values within the said bounds to obtain the desired enhanced gray scale image.

19. The image enhancement system claimed in claim 14 further including:

means for determining a feedback to the system in order to obtain a desired enhancement.

20. An image enhancement system that enhances images, said system comprising:

means for capturing an analog image signal V_{i} of a scene;

first means coupled to said capturing means for digitizing the analog signal V_{i} to a raw gray scale image S_{i} ;

means coupled to said digitizing means for processing the raw gray scale image S_{i} in accordance with ##EQU27## where d_{i} is a desired gray scale value for region i, k_{1}, k_{2}, k_{3}, k_{4} are constants, g is a gain, and M is a maximum number of regions to be enhanced,

to obtain a set of digital enhancement parameters P=(s_{a}, s_{b}) and weight w;

means for obtaining said enhancement parameters P=(s_{a}, s_{b}), and weight w for all regions of the image;

means coupled to said processing means for converting said digital enhancement parameters into analog enhancement parameters Q=(v_{a}, v_{b});

means coupled to said capturing means and said converting means for stretching and clipping the analog image signal V_{i} in accordance with a transform function F_{i} : ##EQU28## the analog image signal being clipped at its low and high levels v_{a} and v_{b} respectively;

second means coupled to said stretching and clipping means for digitizing the stretched and clipped analog signal V_{i} to produce an enhanced gray scale image T;

means coupled to said second means for multiplying the enhanced gray scale image T with weight w to produce a multiplied enhanced image and;

means coupled to said multiplying means for transforming the said multiplied enhanced image with an activation function f() in accordance with ##EQU29## to produce a final enhanced image U.

21. The system claimed in claim 20 wherein said system operates to enhance images to desired measures of gray scale values in real-time speeds.

Description

The invention relates generally to the field of electronic circuits, and particularly to image enhancement for visual recognition systems.

Imagery is the pictorial or graphical representation of a subject by sensing quantitatively the patterns of electromagnetic radiation emitted by, reflected from, or transmitted through a scene of interest. Basically there are two types of imagery, chemical and electronic.

Chemical imagery or traditional photography relies on the interaction between exposing light from a scene on a photosensitive material in such a manner as ultimately to render visible an image of the incident light exposure distribution. The interaction is between the individual exposure photons of light and the photosensitive elements of the photosensitive material. The resulting image is composed of microscopic picture elements corresponding in position to those photosensitive elements that have received adequate exposure.

Electronic imagery utilizes the sensitivity of various electronic sensors to different bands of the electromagnetic spectrum. The energy received by the sensors is transduced into an electronic signal or electrical effect so that the signal or effect may be processed to display the sensed information. The most common forms of electronic imagery are: television cameras, electronic still cameras, and charge coupled devices, etc. The raw information that makes up the electronic image is in the form of pixels. In a digitized picture a pixel is one of the dots or resolution elements making up the picture.

A picture is not always a satisfactory representation of the original object or scene. The picture may have a poorly chosen gray scale, i.e., it may be overexposed or underexposed. The picture may also be geometrically distorted, or the picture may be blurred or noisy. Image enhancement is the process by which a scene, or one or more portions of a scene, are enhanced so that the scene has more detail or that the scene is more visually appealing and/or contain additional information. The foregoing process is often used to increase the usefulness of microscopy pictures, satellite pictures or reconnaissance pictures. Image enhancement involves the reworking of the raw data after the data has been received. Electronic manipulation of the received data can increase or decrease emphasis, extract data selectively from the total received data, and examine data characteristics that would not show up by normal imagery.

The pixels can be measured one at a time at a rapid rate of speed for brightness and other quantities and over a wide scale of selections. For example, black may equal zero, medium gray may equal 128, and white may equal 255 so that groupings of input information can be made to provide better contrast in pattern and blackness, when the information is regrouped and reassembled for display. Computers have been utilized to process the pixels and enhance the image.

One of the methods utilized by the prior art to enhance images was contrast enhancement. Contrast enhancement transformed every pixel in the image by a continuous or discontinuous function into a new enhanced image. This caused the stetching or compression of contrast of the image.

Another method utilized by the prior art to enhance images was histogram transformation. The histogram transformation constructed a transfer function from the raw gray scale image to the enhanced image.

One of the disadvantages of the contrast enhancement and histogram transformation methods was that the aforementioned methods required computations for every pixel in the gray scale image. Thus, for normal images, e.g., 512 pixels by 480 pixels a large amount of computations are required. This necessitated the use of expensive custom hardware in order to obtain high computation speeds.

Another disadvantage of the prior art is that most image enhancement techniques do not supply feedback to inform the user of the system that different enhancement results may be obtained under different imaging conditions.

An additional disadvantage of the prior art was that some image enhancement techniques do not allow the user to choose gray scale values over different regions of the image.

The present invention overcomes the disadvantages of the prior art by providing an image enhancement system that efficiently enhances gray scale images. An analog scene is captured and the analog signal is digitized to a raw gray scale image S. Image S is divided into M disjoint non empty regions R_{1} . . . R_{M}. A symbol S_{i} is issued to identify region R_{i} of S. The raw digitized signal S_{i} is processed in accordance with the enhancement algorithm to obtain a set of digital enhancement parameter offsets P_{i} =(s_{ai}, s_{bi}) and weights (w_{i}). Digital offsets P_{i} =(S_{ai}, s_{bi}) are converted to analog offsets Q_{i} =(v_{ai}, v_{bi}). The high and low levels of the analog signal are clipped and converted to a digital signal that is an enhanced image signal T_{i}. The enhanced image signal is multiplied by weight w_{i} to obtain the final enhanced image U_{i}. Images U_{i} form the regions R_{i} of the final enhanced image U.

An advantage of this invention is that the apparatus of this invention may be implemented to execute at very high speeds in almost any central processor.

An additional advantage of this invention is that the apparatus of this invention supplies feedbacks to the operator of the invention so that desired enhancement results may be obtained. The feedbacks include modifications to the camera-lens and illumination conditions, which allows the operator complete control over the final enhancement results.

A further advantage of this invention is that the operator of this invention is able to select a desired gray scale value distribution of the enhanced image, from scene to scene. The foregoing is important in applications where the texture, color and surface qualities of scenes change from sample to sample.

FIG. 1 is a block diagram of the apparatus of this invention;

FIG. 2 is a graph of the errors in gray scale values for different gains; and

FIG. 3A is a graph showing a range R(s_{a}, s_{b})εΦ^{2} of low offset s_{ai} for different values of ADC gain for full range gray scale value s_{i} =100.

FIG. 3B is a graph showing a range R(s_{ai}) of low offsets s_{ai} for different values of ADC gains g_{i} for full range gray scale values S_{i} =200.

FIG. 4A is a drawing of two industrial steel samples with laser etched characters on them.

FIG. 4B is a drawing of some characters in each part enhanced by the enhancement model and segmented by a standard segmentation algorithm.

FIG. 5A is a drawing of desired gray scale values d_{i} that can be obtained for different values of low offset s_{ai} and ADC gain g_{i} for full range gray scale value s_{i} =100.

FIG. 5B is a drawing of desired gray scale values d_{i} that can be obtained for different values of low offset s_{ai} and ADC gain g_{i} for full range gray scale value s_{i} =200.

FIG. 6 is a drawing of full range gray scale values s_{i} for different values of low offset s_{ai} and ADC gain g_{i} for d_{i} =100.

Referring now to the drawings in detail, and more particularly to FIG. 1, the reference character 11 represents a scene that is captured by image sensor 12. Image sensor 12 may be an electronic camera, video camera, laser, etc. The output of sensor 12 is an analog signal (V_{i}), that is coupled to analog to digital converter 14. Analog to digital converter 14 digitizes the signal V_{i} to a raw gray scale image S_{i} which is stored in image buffer 15. Central processing unit 16 receives the raw digitized signal from buffers 15 and processes this signal in accordance with the enhancement algorithm to obtain a set of digital enhancement parameters (offsets P_{i} =(s_{ai}, s_{bi}) and weights (w_{i})). The enhancement algorithm is hereinafter disclosed. The processed signal is converted to analog quantities by digital to analog converter 17. The digital enhancement parameters are P_{i} =(s_{ai}, s_{bi}) and the analog enhancement parameters are Q_{i} =(v_{ai}, v_{bi}). The aforementioned analog parameters Q_{i} are transmitted to offset block 13. Offset block 13 clips the analog signal V_{i} at the aforementioned signals high and low levels by offsets Q_{i} =(V_{ai}, v_{bi}). The clipped analog signal is transmitted to analog to digital converter 18. Analog to digital converter 18 digitizes the clipped analog signal and transmits L copies of the enhanced image T_{i} to the input of image buffers 19. There are L image buffers 19. In this embodiment L will be equal to three. Weight w_{i} is decomposed into a sum of L powers of two weights w_{ij}, j=1 . . . L. Image T_{i} is multiplied by weight w_{ij} by multiplier 20.

The output of multiplier 20 is connected to the input of summer 21. Summer 21 sums the multiplied enhanced image to produce an enhanced summed image of the scene at its output. The enhanced summed image of the scene may now be utilized for any purpose that is known in the art.

The output of summer 21 i.e. the enhanced summed image is passed through an activation function 22 illustrated by the graph to the right of summer 21. Any digitized gray scale value within S_{min} and S_{max} is passed as is. Any value exceeding S_{max} is truncated to S_{max} and any value below S_{min} is truncated to S_{min}.

The enhancing algorithm that is utilized by central processing unit 16 will be presented below in two steps:

Step 1

From a set of representative images, the model constants are estimated. This step is performed in an off-line process where the speed of the implementation is not critical. This step is usually performed once at the beginning of the experiment. However, the implementations presented in this invention are efficient and can be executed at high speeds.

Step 2

In this step, the enhanced image is generated from known model constants, and from user-specified desired gray values. The execution speed of this step is critical, since this step is performed repeatedly for every different image to be enhanced.

An algorithm to estimate model constants will be discussed hereinafter under the subheading Computation of Model Constants. A separate algorithm is discussed hereinafter under the subheading Estimation of Upper Limit g_{max} of gain g.

In our notations, we shall use upper case letters for images or vectors. Lower cases will be used for variable such as gray scale values or offsets. Although functions are defined for scalars, we shall use vectors as parameters for functions, where the functions are applied to each element of the vector.

Let Vε^{N} be an input analog image signal and Sε^{N} be a digitized ray gray scale image obtained by digitizing signal V to image S, where N is the number of digitized pixels. Note that input V can come from many sources such as sensors (cameras, lasers, etc.) or stored analog image signals. Within S, we shall identify M(1≦M≦N) disjoint non empty regions (R_{1} . . . R_{M}) to be enhanced by our algorithm. Each Region R_{i} of S consists of a subset S_{i} .OR right.S of contiguous picture points. We shall define s_{i} as a measure for the set S_{i}, where s_{i} is the mean of S_{i} i=1 . . . M, although, in general, s_{i} can be any other measure such as the median, weighted average, standard deviation or random sample of the set S_{i}.

Let (s_{min}, s_{max}) be the complete range of the gray scale values in S. The permissible input range (v_{min}, v_{max}) is obtained by the transform h (see below) of (s_{min}, s_{max}), i.e. v_{min} =h(s_{min}) and v_{max} =h(s_{max}). Define θ={vε|v_{min}≦v≦v_{max} }, and Φ={sε|s_{min} ≦s≦s_{max} }.

The analog to digital converter (ADC) is a surjective map g:θ→Φ, where g^{-1} does not exist. However, we shall define h:s|→v, where s is a gray scale value in image S and v is the mean of the analog values that are digitized to s. Thus h:Φ→θ is a bijective linear map. Also define V_{i} =h(S_{i}).

For every region R_{i} of S, the proposed algorithm allows the user to select a desired gray scale value d_{i} εΦ. Let U be the final enhanced image such that the mean gray scale value u_{i} for any region R_{i} within U equals d_{i} for i=1 . . . M.

For each region R_{i} of S we shall compute a set of digital parameters P_{i} =(s_{ai}, s_{bi}) and weight w_{i}. P_{i} includes a low offset s_{ai}, and a high offset s_{bi} which are transformed by the digital to analog converter (DAC)(h) to analog offsets Q_{i} =(v_{ai}, v_{bi}). Parameters Q_{i} are used to modify (offset) the input signal V_{i} by affine functions F_{i} ():θ→θ where ##EQU1##

The modified input F_{i} (V_{i}) is digitized by ADC to enhance image T_{i}. Image T_{i} is multiplied by weight w_{i} and passed through the activation function f() to obtain an enhanced image. Note that we have multiplied weight w_{i} to enhanced image T_{i} instead of inputs v_{i} or F_{i} (v_{i}). This is because most ADCs have the hardware to modify the input V_{i} but do not have a method of multiplying V_{i} by weight w_{i}. Approximating g to a linear function g, g(w_{i} F_{i} (V_{i})) =w_{i} g(F_{i} (V_{i}))=w_{i} T_{i}. Therefore, multiplying F_{i} (V_{i}) by w_{i} is the same as multiplying T_{i} by w_{i} under this approximation.

In the proposed approach, our concern is to generate enhanced images from inputs V_{i} irrespective of their source. We shall, therefore, not use the actual measured value of input analog signal V_{i} or its modified value F_{i} (v_{i}). Instead, the raw gray scale image S and enhanced images T_{i}, are used in our computations.

In our application, we shall present an efficient implementation of the analytical models with an ADC. In the framework established above, the ADC performs two functions: (1) affine transformation F_{i} (), and (2) digitization g(). Note that although the analysis is carried out for the ADC, the general method described above can be applied to any hardware which conforms to functions F_{i} () and g() defined in our models.

We shall present an enhancement model to fit the non-idealities found in many common analog to digital converters (ADCs). Considering these non-idealities and ignoring second order error terms, we get the following expression for enhanced images T_{i} (see FIG. 1), where t_{i} εT_{i} :

t.sub.i =g.sub.i s.sub.i +k.sub.i g.sub.i s.sub.ai +k.sub.2 s.sub.ai +k.sub.3 s.sub.i +k.sub.4, i=1 . . . M, t.sub.i εΦ(1)

Here g_{i} is the gain: ##EQU2##

In (1), when s_{ai} =s_{min}, and s_{bi} =s_{max} we call it the full range operation. Output gray scale values s_{i} at full range are called the full range gray scale values. Note that s_{i} and t_{i} represent the ideal digitized values of v_{i} and F_{i} (v_{i}) respectively. The actual gray scale values are a round-off (quantization) of the ideal values.

Next the enhanced images U are obtained by the weighted sum of image T_{i} with weights w_{i}. Here {U_{i}, . . . U_{M} } form regions {R_{i}, . . . , R_{M} } respectively of the final enhanced image U. Thus, ##EQU3## As defined before, s_{i} is a mean (or any other measure) of the raw image S_{i}. Similarly t_{i} and u_{i} are the means (or any other measures) of enhanced images T_{i} and U_{i} respectively.

In (l), (k_{1}, k_{2}, k_{3}, k_{4}) are unknown real-valued model constants. The ideal values are: k_{1} -1, k_{2} =k_{3} =0. k_{4} represents any bias in the system. Ideally for an unbiased system, k_{4} =s_{min} which is usually 0. It is clear from (1), that for unknown (k_{1}, k_{2}, k_{3}, k_{4}), the true values of s_{i} are unknown. Only the measured values m_{i} are obtained at full range operation as:

m.sub.i =(l+k.sub.3)s.sub.i +(k.sub.1 +k.sub.2) s.sub.min +k.sub.4, i=l . . . M (4)

We shall call m_{i} the measured full range gray scale values.

Perhaps it can be demonstrated that (1) can be derived by assuming non-idealities for other parameters or by assuming different non-idealities for the same parameters. However, our experiments show that (1), independent of its derivation, appropriately represents the non ideal system under discussion.

Before estimating (k_{1}, k_{2}, k_{3}, k_{4}) we should compute the upper limit g_{max} of gain g_{i} by the algorithm under the subheading estimation of upper limit g_{max} of Gain g and also identify M regions {R , . . . , R } for enhancement. From the estimate of g_{max} we obtain the following operating range for offsets (s_{ai}, s_{bi}) ( derived from (2)): ##EQU4## It is clear from (1) that the full range gray scale values s_{i}, i=l . . . M, are unknown variables in estimating constants (k_{1}, k_{2}, k_{3}, k_{4}). For this reason, we shall also estimate s_{i} along with model constants (k_{1}, k_{2}, k_{3}, k_{4}) as follows:

(1) For each region R_{i}, i=1 . . . M, choose K offsets (s_{ak}, s_{bk}), k=l . . . K, within constraint (5), and compute g_{k} by (2).

(2) For each (s_{ak}, s_{bk}) acquire an image and estimate t_{ik} by averaging gray scale values within R_{i}. Note that estimates of t_{ik} can be obtained by other methods such as choosing a median, or random sample of gray scale values within R_{i}. Eliminate those t_{ik} that are close to s_{min} and s_{max}.

(3) Equation (1) can be written in the following matrix form for each region R_{i}. ##EQU5## The above equation is solved by the least squares method. (4) Final estimates of (k_{1}, k_{2}) are obtained by averaging M estimates of (k_{1}, k_{2}) from M regions.

(5) From the M estimates of s_{i} and k_{3} s_{i} +k_{4}, i=l . . . M, obtained above (6), we can estimate k_{3} and k_{4} by least squares method from the following equations: ##EQU6##

The above algorithm for the estimation of model constants (k_{1}, k_{2}, k_{3}, k_{4}) is applied on a set of industrial samples of steel. Constants (k_{1}, k_{2}, k_{3}, k_{4}) estimated from 8 different parts are shown in Table 1 below:

TABLE 1______________________________________MODEL CONSTANTS (k.sub.1, k.sub.2, k.sub.3, k.sub.4) ESTIMATED FROM8 DIFFERENT PARTSSamples k.sub.1 k.sub.2 k.sub.3 k.sub.4______________________________________1 -1.0480 -0.0270 0.590 -6.35902 -1.0284 -0.0106 0.0550 -6.00563 -1.0044 -0.0374 0.0550 -6.14884 -1.0210 -0.0210 0.0542 -5.90585 -1.0110 -0.0378 0.0582 -6.01186 -1.0300 -0.0108 0.0496 -5.69767 -1.0354 -0.0056 0.0564 -6.15988 -1.0196 -0.0244 0.0518 -5.2660______________________________________

The table shows:

(1) Estimates of model constants (k_{1}, k_{2}, k_{3}) are close to their ideal values (ideal k_{1} -1, k_{2} =k_{3} =0, k_{4} =s_{min} =0).

(2) Estimate of bias constant k_{4} is different from its ideal value of s_{min} =0. This result demonstrates that the ADC system has a bias that is represented by k_{4}, and the enhancement model (in (1)) has helped us arrive at accurate enhanced images, which can not be obtained without this model.

In order to maintain the linearity assumptions made for (l), we should determine operating ranges of offsets s_{a} and s_{b}. For this reason, we shall first determine the range (g_{min} _{g} _{max}) for gain g. It is clear from (1) and (2) that the lower limit g_{min} of g is obtained for the full range operation (i.e. s_{a=s} _{min}, s_{b} =s_{max}) where g=1.

Before determining the upper limit g_{max} of g, we should establish the experimental conditions to acquire valid data to properly estimate g_{max}. Since setting offsets (s_{a}, s_{b}) decreases the dynamic range of input V, we should make sure that V is well distributed within θ=[v_{min}, v_{max} ] for g=1. If input is distributed close to limits v_{min} or v_{max} for g=1, setting offsets (V_{a}, V_{b}) will further limit V, and reduce the availability of reliable data for accurate estimation of g_{max}. An easy way to check this condition, is to acquire an image S, and measure gray scale values at different regions of interest within the image. If these gray scale values are distributed close to s_{min} or s_{max}, the optical setup conditions such as lens aperture and illumination should be reconfigured so that they are within (s_{min}, s_{max}).

Estimate g_{max} by the following steps:

1. Place a part under the camera with the optical condition established above.

2. Select M (M≧2) regions of interest {R_{1}, . . . R_{M} } within the image.

3. Start from the first region R_{1}, i.e. R_{i} =R_{1}, where index i is the i^{th} estimate of g_{max} from region R_{1}.

4. Assuming a "reasonable" value g_{start} ≧1 of g_{max} (say g_{start} =1.5), estimate model constants (k_{1}, k_{2}, k_{3}, k_{4}).

5. Choose a set of K offsets (s_{ak}, s_{bk}), k=1 . . . K, within constraint (5).

6. For each choice of offset (s_{ak}, s_{bk}), k=1 . . . K, perform the following steps:

6.1 Acquire an image with offsets (s_{ak}, s_{bk}).

6.2 From this acquired image measure gray scale value t_{ik}.sbsb.--_{measured} by averaging gray scale values within region R_{i}. Eliminate values that are close to s_{min} and s_{max}.

6.3 Compute gray scale value t_{ik}.sbsb.--_{computed} from (1).

6.4 Compute the error (e_{k}) between the measured and computed gray scale values i.e. e_{k} =t_{ik}.sbsb.--_{measured} t_{ik}.sbsb.--_{computed}.

6.5 compute gain g_{k} from (2).

7. Choosing a limit e_{max} of error e_{k} k=1 . . . K, estimate the upper limit g_{max} ^{R}.sbsp.1 of gain g from region R_{1}.

8. If g_{max} ^{R}.sbsp.1 <g_{max} then set g_{start} =g_{max} ^{R}.sbsp.i and repeat steps 4-7 till g_{max} ^{R}.sbsp.1 ≧g_{start}.

9. If g_{max} ^{R}.sbsp.i ≧g_{start}, then report g_{max} ^{R}.sbsp.i as the estimate of g_{max} from region R_{i}.

10. Repeat steps 5-9 for all selected regions R_{1}, . . . , R_{M}.

11. Choose the final estimate g_{max} as the minimum value of all estimates: g_{max} =min {g_{max} ^{R}.sbsp.i, i=l . . . M}.

Here we shall first partition the image into M disjoint non empty regions of interest {R_{1}, . . . , R_{M} }. The M regions are usually chosen by pre-defined criteria dependent on the application. For example, in an image with multiple parts on a background, each part and the background form regions {R_{1}, . . . R_{M} }. Instead of using the entire region R_{1} for our computations, a small probing window within R_{1} is chosen to reduce computational complexity.

This analysis involves efficient estimation of offsets (s_{ai}, s_{bi}) and weights w_{i}, i=1 . . . M, from: (1) known model constants (k_{1}, k_{2}, k_{3}, k_{4}) , and (2) user-specified desired gray scale values (d_{1}, . . . , d_{M}) for regions {R_{1}, . . . , R_{M} } respectively. The enhanced gray scale image T_{i} is obtained by modifying input V_{i} by estimated offsets (s_{ai}, s_{bi}). The final enhanced image U_{i} =w_{i} T_{i}, where weights w_{i} are generated by our algorithm. From (1) and (3),

d.sub.i =w.sub.i (g.sub.i s.sub.i +k.sub.i g.sub.i s.sub.ai +k.sub.2 s.sub.ai +k.sub.3 s.sub.i +k.sub.4), i=1 . . . M, where d.sub.i εφ (9)

We shall first determine a range R(s_{ai}) of low offset s_{ai}, R(g_{i}) of gain g_{i} and R(w_{i}) of weight w_{i}, such that when these parameters are within their respective ranges, we shall obtain enhanced gray scale values t_{i} εΦ where t_{i} εT_{i}.R(s_{ai}) is shown below (derived from (5) and t_{i} εΦ in (1)): ##EQU7## A plot of s_{ai} against different values of g_{i} is shown in FIG. 3 for s_{i} =100 (FIG. 3A) and s_{i} =200 (FIG. 3B).

In (2) we can choose (s_{ai}, s_{bi}) such that 1<g_{i} <s_{max} -s_{min}. However, the linearity assumptions of ADC(g) in (1) are invalid for g_{i} >g_{max} (g_{max<s} _{max-s} _{min}) where g_{max} is determined experimentally. We, therefore, have the following permissible range R(g_{i}) for gain g_{i}.

R(g.sub.i):1≦g.sub.i ≦g.sub.max (11)

It is clear from (9) that given any (g_{i}, s_{ai}) we can obtain the desired d_{i} by adjusting weight w_{i}. It is therefore, not so interesting to study the solution to (9) for any arbitrary weight w_{i}. Since there can be several choices of w_{i} ≦0, the optimum choice may be determined from computational considerations. The best choice is 1, which requires no more processing than a simple image acquisition. In one example, it took us 144 seconds to multiply an entire image (512×480 pixel resolution) with a weight, whereas it took, us 33 ms to acquire an image. If w_{i} =1 is not a valid solution for (9), choose w_{i} closest to 1. Multiplying by w_{i} >1(w_{i} <1) may raise (lower) some gray values above (below) s_{max} (s_{min}) which are truncated to s_{max} (s_{min}) by activation function f() in FIG. 1. Choosing w_{i} closest to 1 minimizes this loss of accuracy due to truncation. We shall, therefore, solve (9) for w_{i} closest to 1. An efficient method of multiplying enhanced image T_{i} with w_{i} ≠1 is discussed herein.

Compute (g_{i}, s_{ai}) by the following algorithm:

(1) Choose the starting value of w_{i} =1.

(2) Choose the starting value of g_{i} =1.

(3) Check if s_{ai} satisfies the following inequality (obtained from (9) and (10)): ##EQU8## (4) If s_{ai} does not satisfy (12), increment g_{i} (by say 0.1) and go back to step 3.

(5) If s_{ai} satisfies (12), report g_{i}, s_{ai} and w_{i} and terminate the algorithm.

(6) Continue the search till g_{i} =g_{max}. If g_{max} does not satisfy (12) check the following:

(6.1) If s_{ai} computed from (12) with g_{i} =g_{max} exceeds the upper limit in R(s_{ai}) then increment w_{i} by 2^{-L} where L is determined in a later section.

(6.2) If s_{ai} computed from (12) with g_{i} =g_{max} is below the lower limit in R(s_{ai}) then decrement w_{i} by 2^{-L}.

(6.3) Go back to step 2.

Note that the algorithm chooses the smallest permissible value of g_{i}, because the linearity assumption of the ADC is strongest for g_{i} closest to 1. Parameter s_{bi} is obtained as ##EQU9##

Here we shall study a range R(d_{i}) of desired gray scale values d_{i}, so that when d_{i} are chosen within R(d_{i}), we can obtain an enhanced image by our algorithm. As seen before, we can obtain any value of d_{i} by appropriately adjusting values of weight w_{i}. It is, therefore, not so interesting to study the model for any arbitrary weight w_{i}. It is, however, important to obtain the desired gray scale value d_{i} with w_{i} =1 in order to reduce computation.

If we select d_{i} .epsilon slash.R(d_{i}), we have three choices:

(1) We can choose a weight w_{i} such that d_{i} can be obtained. This is the less preferable option due to computational considerations.

(2) From a plot of R(d_{i}) choose "satisfactory" values of d_{i} within it.

(3) Obtain guidelines to change the optical/illumination setup to attain d_{i} with our algorithm.

Assuming (k_{1}, k_{2}, k_{3}, k_{4}) close to their ideal values, we obtain: ##EQU10## For the ideal case, the above equation can be simplified to the following: ##EQU11## A plot of R (d_{i}) for this case is shown in FIG. 5. Here we observe that for s_{i} =100, we can obtain all values of d_{i} with our algorithm. For s_{i} =200, we can only obtain higher values of d_{i} >123. Thus, low values of s_{i} gives us greater choices of d_{i}.

If our choice of d_{i} .epsilon slash.R(d_{i}), we need to reconfigure the imaging conditions (such as by adjusting the camera-lens or illumination setup) so that d_{i} εR(d_{i}). In (13), R(d_{i}) depends on s_{i} and we observed in FIG. 5 that appropriate choices of s_{i} gives us wider choices of d_{i}. We, therefore, need to find a range R(s_{i}) for s_{i} such that if s_{i} εR(s_{i}) then d_{i} can be obtained with our algorithm. Assuming (k_{1}, k_{2}, k_{3}, k_{4}) close to their ideal values: ##EQU12## A plot of s_{i} against different values of g_{i} and s_{ai} is shown in FIG. 6. To change the imaging conditions, the camera-lens and illumination setup are reconfigured so that s_{i} εR(s_{i}).

Among the computations involved in the algorithms described above, it is the multiplication by weight w_{i} that makes it difficult to implement the method with digital hardware. If w_{i} =1 is not a feasible choice, we have the following disadvantages:

1. It is usually computationally inefficient to multiply an entire image with a number.

2. Multiplication with any number, especially noninteger numbers, can cause loss of gray scale value information due to roundoff or truncation.

In most image processors it is simpler to add or subtract images than to multiply an image with a number. In one example, we added/subtracted two images (512×480 pixel resolution) in 10 ms, while it took us 144 s to multiply the image with a number.

One possible way to solve this problem is to use powers-of-two weights such that the multipliers are replaced by shift registers. This means that the weights can only take values on {0, 1, ±2^{-1}, ±2^{-2}, . . . ±2^{-L} }, where L is determined by the accuracy required. Thus ##EQU13## where w_{ij} are powers-of-two weights. Multiplication of image T_{i} by weight w_{ij} =2^{-k} is realized by left shift of every pixel value by k steps. Clearly, any choice of L gives us an accuracy ≧2^{-L}. Furthermore, L can be chosen from the number of bits used to store each gray scale value. For 256 gray scale values (represented by 8 bits) L can be chosen as 7, and the smallest weight change in step 6 of the enhancement algorithm is 2^{-7}. This method has the following advantages:

1. We may be able to choose w_{ij} =1 which may be otherwise impossible, and thus reduce computation.

2. Adding/subtracting images with w_{ij} =1 causes no loss of information due to round off errors.

3. Adding/subtracting images has the added benefit of noise reduction due to averaging of the noise. The improvement in image quality (signal-to-noise ratio) is proportional to the square root of L.

In this section we shall consider a simplification of the enhancement model described above. Instead of parameters P_{i} =(s_{ai}, s_{bi}) for individual regions, we shall apply the same parameter set P=(s_{a}, s_{b}) and weight w for all regions R_{i}, i=1 . . . M of the image. The advantages are:

(1) This model is computational efficient, and can be optimized to execute at real-time speeds.

(2) The model allows us to compute a single set of parameters P=(s_{a}, s_{b}) and weight w for the entire image.

(3) Since the parameters are the same over the entire image, we do not need any information on the boundaries of regions {R_{1}, . . . , R_{M} }, to be enhanced.

The new model is:

t.sub.i =gs.sub.i +k.sub.1 gs.sub.a +k.sub.2 s.sub.a +k.sub.3 s.sub.i +k.sub.4, and u.sub.i =f(wt.sub.i), and t.sub.i εφ, i=1 . . . M(16)

Here we shall estimate offsets (s_{a}, s_{b}) and weight w from: (1) known model constants (k_{1}, k_{2}, k_{3}, k_{4}) and (2) user-specified desired gray scale values d_{i} for regions R_{i}, i=1 . . . M. The enhanced image T is obtained by modifying input V by estimated offsets (s_{a}, s_{b}). The final enhanced image U=wT. From(16),

d.sub.i =w(gs.sub.i +k.sub.1 gs.sub.a +k.sub.2 s.sub.a +k.sub.3 s.sub.i +k.sub.4), whered.sub.i εφ, i=1 . . . M (17)

From (17), (g and s_{a} are unknown), we obtain the following linear equation: ##EQU14## Since (18) is an overdetermined system, the following theorem describes the conditions on d_{i}, i=1 . . . M, under which solutions for g, s_{a} and w exist.

From (18) the necessary and sufficient condition for which solutions for g, s_{a} and w exist is: ##EQU15##

for all i, j, k, l=1, . . . , M such that i≠j and k≠l(19)

Proof

For any pair of equations indexed by i and j, the solutions to (18) are: ##EQU16## If (18) has a solution then α_{ij} =α_{k} and β_{ij=}β_{k}. Considering only α_{ij} and α_{kl} we get (19). Next we shall show that if (19) is true, then (18) has a solution. In other words, we have to show that β_{ij} =β_{kl} and α_{ij} =β_{kl} if (19) is true. Clearly, from (19) and (19a) α_{ij} =α_{kl}. From (19a) ##EQU17## From this we conclude: β_{ij} -β_{kl} =0. Therefore, β_{ij} =β_{kl}. Q.E.D.

Theorem 1 shows that we can only choose two independent values of d_{i} for a single image, the rest of which are fixed by (19). Since we have only two offsets (s_{a}, s_{b}), to modify input V (giving us only two degrees of freedom), we can independently choose only two regions for simultaneous enhancement, i.e. M=2. Thus, we shall consider only two regions i and j for enhancement, for the rest of this analysis.

Next, we need to determine a range of (w, g, s_{a}) such that t_{i} εΦ when (w, g, s_{a}) belong to their respective ranges. Assuming s_{i} >s_{j}, without loss of generality, the range R(s_{a}) for s_{a} is (from (10)): ##EQU18##

R(g) for gain g is: R(g): 1≦g≦g.sub.max (21)

For the nonideal case, we shall first determine a range R(w) for weight w such that when wεR(w), (g, s_{a}) satisfy (21) and (20) respectively. From R(g) in (21), we get the range R_{1} (w): ##EQU19## From R(s_{a}) in (20) we get the range R_{2} (w): ##EQU20##

The final range R(w) is: R(w)=R.sub.1 (w)∩R.sub.2 (w).(24)

The algorithm for enhancement is as follows:

1. Choose wεR(w). As before the best selection of w is 1. Otherwise choose w closest to 1 to minimize truncation and round-off errors due to activation function f() in FIG. 1.

2. Compute (g, s_{a}) from (18) satisfying (21) and (20) respectively.

3. Compute high offset S_{b} from: ##EQU21##

In all our previous discussions we have only considered the first ordered measures of the desired gray scale values d_{i}. These measures include the mean, median, weighted average and the random sample of d_{i}. All these measures of d_{i} follow a linear equation in (1). In this section, we shall discuss a second order measure of d_{i}. This measure is the standard deviation σ(d_{i}) of d_{i}. From (1) we obtain: σ(d_{i})=w_{i} (g_{i} +k_{3})σ(s_{i}), where σ(s_{i}) is the standard deviation of full range gray scale values s_{i} in raw image S_{i}.

If we need to enhance the image such that both first and second order measures need to be satisfied, we can modify step 2 of the enhancement algorithm such that g_{i} =1 is replaced by ##EQU22## The rest of the algorithm remains the same. Other second order measures that can also be considered by the above methods are a variation of gray scale values or a range of intensities about the mean.

FIG. 2 is a drawing that shows the result of an experiment which demonstrates a typical profile or error in gray scale values vs. gain that is used to estimate g_{max} by the algorithm hereinbefore stated.

Choosing a maximum error of 2 gray scale values, we estimated g_{max} =2.2 from FIG. 2. It is clear from FIG. 2, that the errors increase significantly for g>2.2. The range of offsets s_{a} and s_{b} determined from (5) is: (s_{a} +116)≦s_{b}.

FIG. 3A is a graph showing a range R(s_{a}) of low offset s_{ai} for different values of ADC gain for full range gray scale value s_{i} =100. The lower and upper curves are the lower and upper limits of the low offset s_{ai}.

FIG. 3B is a graph showing a range R(s_{ai}) of low offsets s_{ai} for different values of ADC gains g_{i} for full range gray scale values S_{i} =200. The lower and upper curves are the lower and upper limits of the low offset s_{ai}.

FIG. 4A is a drawing of two industrial steel samples with laser etched characters on them. The figure also shows the difference in contrast between the two parts.

FIG. 4B is a drawing of some characters in each part enhanced by the enhancement model and segmented by a standard segmentation algorithm. FIG. 4B also shows that characters not enhanced by the model are not clearly segmented;

FIG. 5A is a drawing of desired gray scale values d_{i} that can be obtained for different values of low offsets _{ai} and ADC gain g_{i} for full range gray scale value s_{i} =100. The figure also shows that all desired gray scale values can be obtained with our model in this case, for weight w_{i} =1.

FIG. 5B is a drawing of desired gray scale values d_{i} that can be obtained for different values of low offset s_{ai} and ADC gain g_{i} for full range gray scale value s_{i} =200. The figure also shows that only desired gray scale values >123 can be obtained with our model in this case, for weight w_{i} =1;

FIG. 6 is a drawing of full range gray scale values s_{i} for different values of low offset s_{ai} and ADC gain g_{i} for d_{i} =100.

The above specification describes new and improved image enhancement system for visual recognition systems. It is realized that the above description may indicate to those skilled in the art additional ways in which the principles of this invention may be used without departing from the spirit. It is, therefore, intended that this invention be limited only by the scope of the appended claims.

Patent Citations

Cited Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|

US4561104 * | 16 Jan 1984 | 24 Dec 1985 | Honeywell Inc. | Automated inspection of hot steel slabs |

US4823194 * | 30 Jul 1987 | 18 Apr 1989 | Hitachi, Ltd. | Method for processing gray scale images and an apparatus thereof |

US5307425 * | 1 Sep 1992 | 26 Apr 1994 | Rohm Co., Ltd. | Bi-level halftone processing circuit and image processing apparatus using the same |

Referenced by

Citing Patent | Filing date | Publication date | Applicant | Title |
---|---|---|---|---|

US6233011 * | 18 Jun 1997 | 15 May 2001 | Acer Peripherals, Inc. | Apparatus and method for compensating image being sensed |

US6903782 * | 28 Mar 2001 | 7 Jun 2005 | Koninklijke Philips Electronics N.V. | System and method for performing segmentation-based enhancements of a video image |

US8194286 | 14 Nov 2007 | 5 Jun 2012 | Samsung Electronics Co., Ltd. | Image forming apparatus and image forming method capable of revising gray image |

US8422081 | 24 Apr 2012 | 16 Apr 2013 | Samsung Electronics Co., Ltd. | Image forming apparatus and image forming method capable of revising gray image |

US20020140864 * | 28 Mar 2001 | 3 Oct 2002 | Koninklijke Philips Electronics N.V. | System and method for performing segmentation-based enhancements of a video image |

CN101184144B | 12 Nov 2007 | 1 Dec 2010 | 三星电子株式会社 | Image forming apparatus and image forming method capable of revising gray image |

EP0888004A2 * | 29 Jun 1998 | 30 Dec 1998 | Pioneer Electronic Corporation | Brightness controlling apparatus |

EP0888004A3 * | 29 Jun 1998 | 27 Dec 2000 | Pioneer Electronic Corporation | Brightness controlling apparatus |

Classifications

U.S. Classification | 348/671, 348/E05.073, 382/169, 348/28, 382/270 |

International Classification | H04N5/20, G06T5/40 |

Cooperative Classification | H04N5/20, G06T5/50, G06T2207/20221 |

European Classification | G06T5/40, H04N5/20 |

Legal Events

Date | Code | Event | Description |
---|---|---|---|

9 Dec 1996 | AS | Assignment | Owner name: PURDUE RESEARCH FOUNDATION PURDUE UNIVERSITY, INDI Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:CHATTERJEE, CHANCHAL;ROYCHOWDHURY, VWANI P.;REEL/FRAME:008295/0512 Effective date: 19961126 |

17 Aug 2000 | FPAY | Fee payment | Year of fee payment: 4 |

12 Sep 2000 | REMI | Maintenance fee reminder mailed | |

8 Sep 2004 | REMI | Maintenance fee reminder mailed | |

18 Feb 2005 | LAPS | Lapse for failure to pay maintenance fees | |

12 Apr 2005 | FP | Expired due to failure to pay maintenance fee | Effective date: 20050218 |

Rotate