WO2000018111A1

WO2000018111A1 - Efficient compression engine for digital image data

Info

Publication number: WO2000018111A1
Application number: PCT/US1999/021832
Authority: WO
Inventors: John S. Degood
Original assignee: Sarnoff Corporation
Priority date: 1998-09-18
Filing date: 1999-09-20
Publication date: 2000-03-30

Abstract

Embodiments of the present invention provide an optimized image processing system for a digital camera in which memory and computational requirements are reduced over known image processing systems. A digital imaging device as in the figure includes an image capturing device (210) that generates image data in a color space, each color component of the color space being represented by a predetermined data rate. The digital imaging device further includes an image coder (230) that receives the image data at the predetermined data rate and compresses the image data. In the present invention, the pixel interpolation that characterizes known image processing systems is omitted. Optionally, the present invention also may omit color space transformations (220) and downsampling. The resultant image processing system achieves similar coding efficiencies that are achieved by the known image coders and does so with lower computational and memory requirements and, therefore, increased processing speed.

Description

EFFICIENT COMPRESSION ENGINE FOR DIGITAL IMAGE DATA

RELATED APPLICATIONS

This application benefits from the priority of U.S. patent application serial number 60/100,959, filed September 18, 1998.

BACKGROUND OF THE INVENTION

The present invention relates to a computationally efficient compression engine for digital image data, one that finds application in processing systems that have limited processor capacities or memory sizes. For example, the present invention may be advantageously applied to portable imaging devices such as digital cameras and portable scanners.

Digital imaging engines are known. A high-level block diagram of an imaging device is illustrated in FIG. 1. There, the imaging device includes a color filter array ("CFA") 110, a pixel interpolation block 120, a color space transform block 130, a downsampier 140 and an image coder 150.

The CFA 110 captures an image and generates a digital image signal therefrom. Typically, the

CFA 110 comprises an ordered array of pixel sensors, each pixel sensor having the ability to detect one of a predetermined number of colors. The digital image signal from the CFA 110 represents, at each pixel location in the image, the contribution only of the color component detected by the associated pixel sensor.

By way of example, consider the well-known Bayer pattern image shown in FIG. 2. In the

Bayer image pattern, only one color is captured at each pixel location. For the Bayer pattern array, green color components are captured at 50% of the pixel locations, red color components are captured at 25% of the pixel locations and blue color components are captured at 25% of the pixel location. All visible colors can be expressed as some combination of green, red and blue colors. But at a green pixel location, for example, the CFA does not capture information regarding the red or blue components of the image. In this way, the digital image signal is said to be "sparse." The pixel interpolation block 120 corrects this sparseness. At each pixel location, the pixel interpolation block 120 calculates image data for the color components that were not captured by the CFA 110. Thus, at each green pixel location, the pixel interpolation block 120 calculates estimated values for red and for blue color components using measured values of red and blue color components at neighboring pixel locations. FIG. 3 represents a digital image signal output from the pixel interpolation block 120, where three colors are represented per pixel location. The quantity of data used to represent the image data following processing by the pixel interpolation block 120 is three times the size as the image data captured by the CFA 110.

The imaging device 100 typically includes a color space transform 130 that transforms the digital image data to a color space that is used by the image coder 150. Typically, the color space transform 130 may convert the image data from a red-green-blue ("RGB") color space to a luminance- chrominance color space (conventionally represented as "Y-C_r-C_b"). Other color space transforms, of course, are known.

The output of the color space transform 130 may be input to a downsampier 140. The downsampier 140 selectively eliminates certain portions of the image data to reduce the size of the image data to a size that is suitable for the image coder 150. The output of the downsampier 140 is input to the image coder 150.

The image coder 150 performs image compression upon each color component of the image data individually. The image coder 150 may include a multiplexer 152 that separates the image data by color component and also a plurality of image compressors 154-158 that compress the color-separated image data. For JPEG-type image coders 150, for example, the image compressors may perform discrete cosine transformations upon the image data. Other image compression algorithms, of course, are known such as the well-known wavelet coding techniques. The image coder 150 outputs a unitary signal including compressed image data from each of the image compressors 154-158 via a demultiplexer (not shown).

The image processing discussed above with respect to FIG. 1 is computationally complex and requires a large amount of memory. A raw image from a 1 -megapixel CFA array occupies 1 million words of memory. After interpolation, the image size inflates threefold. The computational requirements for the interpolation are dependent on the algorithm used; a range of 10-100 operations per pixel location would not be uncommon. These image sizes, when considered with the resources that are needed to perform JPEG or MPEG compression, require imaging devices to possess powerful processors and multi-megabyte memories. Particularly for portable imaging devices such as hand-held digital cameras and scanners, such requirements increase the cost of these devices and limit their opportunities within the market for domestic goods.

Accordingly, there is a need in the art for an imaging device having reduced memory sizes.

There is a need in the art for computationally efficient and memory efficient image compression in imaging devices.

SUMMARY

Embodiments of the present invention provide an optimized image processing system for a digital camera in which memory and computational requirements are reduced over known image processing systems. A digital imaging device includes an image-capturing device that generates image data in a color space, each color component of the color space being represented by a predetermined data rate. The digital image device further includes an image coder that receives the image data at the predetermined data rate and compresses the image data.

In the present invention, the pixel interpolation that characterizes known image processing systems is omitted from the encoding process. Optionally, the present invention also may omit the downsampling and color space transformations. The resultant image processing system achieves similar coding efficiencies that are achieved by the known image coders and but does so with lower computational and memory requirements. The present invention generates coded data output that may be used with conventional image decoders; such decoders may include their own stage of pixel interpolation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram of processing that may be performed in known digital cameras.

FIG. 2 is a graph representing a Bayer pattern image.

FIG. 3 is a graph representing a pixel interpolated image.

FIG. 4 is a functional diagram of a digital camera according to an embodiment of the present invention. FIG. 5 is a functional diagram of a digital camera according to another embodiment of the present invention.

FIG. 6 is a block diagram of a digital camera according to an embodiment of the present invention.

DETAILED DESCRIPTION

The present invention provides an optimized image processing system for a digital camera in which memory and computational requirements are reduced over known image processing systems. In the present invention, the pixel interpolation that characterizes known image processing systems is omitted. Optionally, the present invention also may omit the downsampling and color space transformations. The resultant image processing system achieves similar coding efficiencies that are achieved by the known image coders and but does so with lower computational and memory requirements.

It is believed that, in the prior art, the downsampling that occurs in most known image coders eliminates much of the image expansion that occurs in the pixel interpolation step. In layman's terms, the downsampling may be thought as "undoing" some of the processing that is done during pixel interpolation. If, for example, the pixel interpolation estimates a likely value for green and blue in a red pixel location, the downsampier simply may discard the estimated value. Of course, the pixel interpolation blocks and downsamplers need not be coordinated in any predetermined manner. Rather than discarding estimated color values, the downsampier may discard measured color values. In so doing, the downsampier may introduce unnecessary errors into the image coding process.

FIG. 4 is a block diagram of an imaging system 200 according to an embodiment of the present invention. The imaging system 200 may be populated by a CFA imager 210, a color space transform 220 and an image coder 230. The image coder 230 may include a multiplexer 232 and a plurality of image compressors 234-238.

The CFA imager 210 may operate as known CFA imagers do -- they generate digital image signals from a captured image. Typically, the image data output by the CFA imager 210 may be populated by pixel data. As was described with respect to FIG. 2, the pixel data represents pixel values for a single color component of the color space to which the CFA applies. For example, the CFA imager 210 may output pixel data in the RGB color space, only one color component per pixel location. Thus, the digital image data may be characterized by the sparseness described above.

By way of representative example, the CFA imager 210 may be based upon a Bayer pattern image but also may be based upon any of the other well-known pattern images, such as the "sparse checkerboard" pattern image, the "diagonal stripes" pattern image, the 3G/2G pattern image, the "horizontal stripes" pattern image or the "vertical stripes" pattern image, among others.

The imaging system 200 may include a color space transform 220 that receives the digital image signal from the CFA imager 210. Based upon the digital image signal from the CFA imager 210, the color space transform 220 may generate a second digital image signal representing the captured image in a second color space. By way of example, the color space transform 220 may generate a digital image signal in a Y-Cr-Cb color space from an input digital image signal in an R-G-B color space. Other known color space transformations are known and may be used in accordance with the spirit and scope of the present invention.

According to an embodiment of the present invention, an R-G-B to Y-Cr-Cb color space transform may be accomplished to retain a single color component at each pixel location of the Bayer pattern CFA shown in FIG 2. At each green pixel location, the color space transform 220 may calculate a luminance component according to the formula:

Y = 0.59 G + 0.30

+0.11 {B +B₂)I2 where R., R₂ refer to the measured values for red and B„ B₂ refer to the measured values for blue at locations adjacent to the green pixel location. At each red pixel location, the color space transform 220 may calculate a chrominance component according to the formula:

Cr = 0.5 R - 0.42 (G₁+G₂+G₃+G₄)/4 - 0.08 (B₁+B₂+B₃+B₄)/4,

where G,-G₄ are the measured values for green and B,-B₄ are the measured values for blue at locations adjacent to the red pixel location. At each blue pixel location, the color space transform 220 may calculate a chrominance component according to the formula:

Cb = 0.5 B + 0.17 (R₁+R₂+R₃+R₄)/4 - 0.33 (G₁+G₂+G₃+G₄)/4

where R,-R₄ are the measured values for red and G,-G₄ are the measured values for green at locations adjacent to the blue pixel location. This color space transform generates transformed image data that retains the one-color-component-per-pixel-location characteristic throughout the imaging device 200. Other color transforms could be used.

The transformed image data is input to the image coder 230. The image coder 230 may include a multiplexer 232 that organizes the image data by color component into a plurality of image planes. Each image plane may be input to one of the image compressors 234-238 for data compression. Again, the image compressors 234-238 may be discrete cosine transform coders in a JPEG embodiment or, alternatively, may be wavelet coders. The image coder 230 outputs a unitary compressed data signal via a demultiplexer (not shown).

The coder 200 also may include a controller 240. The controller 240 may generate an administrative data signal that represents the pixel density of pixel data in each of the image planes.

The administrative data signal may be output from the coder 200 along with the coded image data from the image coder 230. The administrative data signal may be used by a decoder (not shown) as a basis for pixel interpolation during the decoding process.

The image capturing and processing system 200 of FIG. 4 provides advantages that are not available in known systems. The present invention omits the pixel interpolation process entirely and, in doing so, achieves a reduction in computational complexity for such devices. The reduced computational complexity permits the digital camera of the present invention to capture and compress images with greater speed than is available in known digital cameras. As described, pixel interpolation requires, for each pixel location, a computation of typically two color values each based on four or more values from neighboring pixel locations. For an exemplary 1 megapixel image, the pixel interpolation block typically performs 10-100 million computations. These computations are omitted entirely by the present invention. Further, the present invention does not inflate the data size of images that normally accompanies pixel interpolation. For the same megapixel image, a single image occupies 1 million words of memory; it is not inflated threefold as described previously. Accordingly, because there is only one color component at each pixel location, the overall data size of the image is maintained at a reduced level.

The coded data signal output from the coder 200 may be output to a channel (not shown) for later retrieval by a decoder (also not shown). According to an embodiment of the present invention, a channel may include a communications channel that may be established by a communications system or a computer system. A channel also may include any of a number of storage devices such as optical, electrical or magnetic memories.

FIG. 5 illustrates a digital camera processing system according to another embodiment of the present invention. There, the processing system 300 may include a CFA 310 and an image coder 320. The image coder 320 may include a multiplexer 352 and a plurality of image compressors 324-328.

The CFA imager 310 may operate as known CFA imagers do ~ they generate digital image signals from a captured image. Typically, the image data output by the CFA imager 310 may be populated by pixel data. As was described with respect to FIG. 2, the pixel data represents pixel values for a single color component of the color space to which the CFA applies. For example, the CFA imager 310 may output pixel data in the RGB color space, only one color component per pixel location. Thus, the digital image data may be characterized by the sparseness described above.

By way of representative example, the CFA imager 310 may be based upon a Bayer pattern image but also may be based upon any of the other well-known pattern images, such as the "sparse checkerboard" pattern image, the "diagonal stripes" pattern image, the 3G/2G pattern image, the "horizontal stripes" pattern image or the "vertical stripes" pattern image, among others.

The digital image signal output from the CFA 310 may be input to the image coder 320. The image coder 320 may include a multiplexer 322 that organizes the image data by color component into a plurality of image planes. Each image plane may be input to one of the image compressors 324-328 for data compression. Again, the image compressors 324-328 may be discrete cosine transform coders in a JPEG embodiment or, alternatively, may be wavelet coders. The image coder 320 outputs a unitary compressed data signal via a demultiplexer (not shown).

The image coder 320 may include a controller 330. The controller 330 may generate an administrative data signal that represents the pixel density of pixel data in each of the image planes.

The administrative data signal may be output from the image coder 320 along with the coded image data from each image plane. The administrative data signal may be used by a decoder (not shown) as a basis for pixel interpolation during the decoding process.

The embodiment of FIG. 5 achieves further advantages over the prior art by omitting the color space transform of FIG. 4. The pixel data generated by the CFA imager 310 is input directly to the image coder 320 without an interstitial color space transform. This omission reduces even further the complexity of the digital camera. As a consequence of omitting the color space transform, however, the digital camera should include an image coder 320 that is tuned to the color space of the CFA 310. As is known, image coders 320, such as the known discrete cosine transform coders or the wavelet coders, may be tuned to operate in the RGB and Y-Cr-Cb color spaces, for example. The image coder 320 and the CFA 310 should be selected to operate in the same color space.

FIG. 6 is a block diagram of a digital camera 400 according to an embodiment of the present invention. The digital camera 400 may include a processor 410, an image capturing device 420 and a memory 430. Image capturing devices are known per se; they may include the CFAs 210, 310 and associated optical elements such as lenses, apertures and control systems therefor.

The memory 430 may include a static memory 440 storing executable program instructions, a volatile memory 450 and a non-volatile memory 460. The static memory 440 conventionally may be some sort of read only memory ("ROM") provide on an electrical, magnetic or optical storage medium. The volatile memory 450 conventionally may be some sort of random access memory ("RAM") and may be integrated as a cache within the processor 410 or provided externally from the processor 410 as a separate integrated circuit. The non-volatile memory 450 also may be an electrical, magnetic or optical storage medium. In one embodiment, the non-volatile memory 450 may be a storage device that may be physically removed from the digital camera.

The digital camera 400 also may include certain user interface devices 470 ("I/O") that may include buttons to snap pictures, lens controls and user readouts. Such devices are well-known in the art.

Several embodiments of the present invention are specifically illustrated and described herein. However, it will be appreciated that modifications and variations of the present invention are covered by the above teachings and within the purview of the appended claims without departing from the spirit and intended scope of the invention.

Claims

I CLAIM:

1. A digital imaging device comprising: an image capturing device, generating image data in a color space, each color component of the color space being represented by a predetermined data rate, and a image coder, receiving the image data at the predetermined data rate and compressing the image data.

2. The digital imaging device of claim 1 , wherein the image coder comprises: a color component separator, and a plurality of image compressors in communication with the color component separator.

3. The digital imaging device of claim 2, wherein the image compressors are discrete cosine transform encoders.

4. The digital imaging device of claim 2, wherein the image compressors are wavelet encoders.

5. The digital imaging device of claim 1 , further comprising a color space transform coupled to the image capturing device and having an output coupled to the image coder.

6. The digital imaging device of claim 1 , wherein the image capturing device comprises a color filter array.

7. The digital imaging device of claim 1 , further comprising a controller that generates an administrative signal representing a pixel density of data coded by each of the image plane coders.

8. A method of coding an image, comprising: capturing the image as pixel data, the pixel data representing a single component of a color space at each pixel location, separating the pixel data into image planes according to the color space components, wherein each pixel location is represented by pixel data in only one of the image planes, and compressing the image planes of pixel data.

9. The method of claim 8, wherein the compressing step includes coding the data of the image planes by a discrete cosine transform.

10. The method of claim 8, wherein the compressing step includes coding the data of the image planes by wavelet coding.

11. The method of claim 8, further comprising a step of, before the separating step, converting the pixel data from the first color space to a second color space, wherein the separating step causes the pixel data of the second color space to be separated into the image planes.

12. The method of claim 8, further comprising a step of outputting coded image data representing the coded image planes with an administrative signal representing a pixel density of the pixel data in each of the image planes.

13. A data signal constructed according to the method of: capturing an image as pixel data, the pixel data representing a single component of a color space at each pixel location, separating the pixel data into image planes according to the color space components, wherein each pixel location is represented by pixel data in only one of the image planes, and compressing the image planes of pixel data.

14. The data signal of claim 13, wherein the compressing step includes coding the data of the image planes by a discrete cosine transform.

15. The data signal of claim 13, wherein the compressing step includes coding the data of the image planes by wavelet coding.

16. The data signal of claim 13, constructed according to a method that further comprises, prior to the separating step, transforming the pixel data from the first color space to a second color space, and wherein the separating step causes the pixel data of the second color space to be separated into the image planes.

17. The data signal of claim 13, constructed according to a method that further comprises outputting coded image data representing the coded image planes with an administrative signal representing a pixel density of the pixel data in each of the image planes.

18. A computer readable medium having stored thereon instructions that, when executed by a processor, cause the processor to: capture an image as pixel data, the pixel data representing a single component of a color space at each pixel location, separate the pixel data into image planes according to the color space components, wherein each pixel location is represented by pixel data in only one of the image planes, and compress the image planes of pixel data.

19. The computer readable medium of claim 18, wherein the processor performs compression by discrete cosine transform coding.

20. The computer readable medium of claim 18, wherein the processor performs compression by wavelet coding.

21. The computer readable medium of claim 18, wherein the processor further converts, prior to the separating step, the pixel data from the first color space to a second color space, and wherein the processor performs the separating step by causing the pixel data of the second color space to be separated into the image planes.

22. The computer readable medium of claim 18, wherein the processor further outputs coded image data representing the coded image planes with an administrative signal representing a pixel density of the pixel data in each of the image planes.