US20070097207A1 - Image processing method, image processing device and image display apparatus employing the image processing device - Google Patents
Image processing method, image processing device and image display apparatus employing the image processing device Download PDFInfo
- Publication number
- US20070097207A1 US20070097207A1 US11/553,345 US55334506A US2007097207A1 US 20070097207 A1 US20070097207 A1 US 20070097207A1 US 55334506 A US55334506 A US 55334506A US 2007097207 A1 US2007097207 A1 US 2007097207A1
- Authority
- US
- United States
- Prior art keywords
- image
- image data
- distance
- parallax
- imagers
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000003672 processing method Methods 0.000 title claims abstract description 22
- 238000012545 processing Methods 0.000 title claims description 99
- 238000000034 method Methods 0.000 claims abstract description 76
- 230000008569 process Effects 0.000 claims abstract description 59
- 238000006243 chemical reaction Methods 0.000 claims 2
- 239000000284 extract Substances 0.000 claims 1
- 238000010586 diagram Methods 0.000 description 61
- 230000000694 effects Effects 0.000 description 42
- 238000001514 detection method Methods 0.000 description 27
- 238000000605 extraction Methods 0.000 description 17
- 238000003860 storage Methods 0.000 description 13
- 239000011159 matrix material Substances 0.000 description 12
- 230000015654 memory Effects 0.000 description 12
- 238000010606 normalization Methods 0.000 description 10
- 230000003287 optical effect Effects 0.000 description 8
- 239000004973 liquid crystal related substance Substances 0.000 description 6
- 230000006870 function Effects 0.000 description 5
- 239000011521 glass Substances 0.000 description 4
- 238000012986 modification Methods 0.000 description 4
- 230000004048 modification Effects 0.000 description 4
- 238000003491 array Methods 0.000 description 3
- 230000001747 exhibiting effect Effects 0.000 description 3
- 230000010287 polarization Effects 0.000 description 3
- 230000000295 complement effect Effects 0.000 description 2
- 229910044991 metal oxide Inorganic materials 0.000 description 2
- 150000004706 metal oxides Chemical class 0.000 description 2
- 238000005070 sampling Methods 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 230000004075 alteration Effects 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000005520 cutting process Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 230000003467 diminishing effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 239000000758 substrate Substances 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
- H04N13/218—Image signal generators using stereoscopic image cameras using a single 2D image sensor using spatial multiplexing
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/204—Image signal generators using stereoscopic image cameras
- H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
Definitions
- the present disclosure relates to an image processing method using a solid-state image-taking device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) device, an image processing device adopting the image processing method and an image display apparatus employing the image processing device. More particularly, the present disclosure relates to an image processing method for processing an image based on a parallax between the right and left eyes and displaying the processed image as a pseudo three-dimensional image on an ordinary display, an image processing device adopting the image processing method and an image display apparatus employing the image processing device.
- a solid-state image-taking device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) device
- an image processing device adopting the image processing method and an image display apparatus employing the image processing device.
- image data for the left eye of the observer and image data for the right eye of the same observer are alternately output to a display apparatus.
- the observer is capable of reproducing an image from the pieces of image data passing through the glasses capable of switching the shutter from the right glass to the left one and vice versa synchronously with the timing to switch the image data appearing on the display apparatus.
- Japanese Patent Laid-open No. 2004-343290 discloses a method of reproducing a 3D (three-dimensional) image without using glasses and an apparatus adopting the method.
- a three-dimensional display apparatus based on a parallax is employed.
- a 3D (three-dimensional) image is generated by alternately arranging an image for the right eye and an image for the left eye for every dot in the form of rectangles and the images of the two points of view are displayed as a 3D (three-dimensional) image by adoption of a parallax variable method or a lenticular method.
- the display apparatus has a configuration including a slit and a display screen.
- the slit is placed in front of the display screen.
- the observer observes a 3D (three-dimensional) image appearing on the display screen through the slit.
- the left eye of the observer looks at only an image for the left eye while the right eye of the observer is looking at only an image for the right eye.
- the observer is capable of observing a 3D (three-dimensional) image with a feeling of seeing a 3D (three-dimensional) object.
- Japanese Patent Laid-open No. 2004-40445 discloses an example of providing a portable information terminal with a 3D (three-dimensional) image displaying function.
- a liquid-crystal module capable of displaying a 3D (three-dimensional) image includes a backlight, a switching liquid-crystal device, a phase difference board and a TFT liquid crystal.
- the switching liquid-crystal device is a component capable of switching its polarization angle.
- the phase difference board is a component patterned to form a slit shape.
- a 3D (three-dimensional) image is displayed by changing a phase relation with the patterned phase difference board.
- the phase relation with the patterned phase difference board is changed by switching the polarization angle of the switching liquid-crystal device from one value to another.
- a left image is generated by left pixels but prevented from being projected on the right eye by being cut off from the right eye.
- a right image is generated by right pixels but prevented from being projected on the left eye by being cut off from the left eye.
- only the image for the right eye appears on the right eye and only the image for the left eye appears on the left eye.
- an image appearing on the right eye of the observer is different from an image appearing on the left eye of the observer.
- the observer is capable of sensing a 3D (three-dimensional) image having a depth in the inward direction.
- the display apparatus requires special units such as the phase difference board and the switching liquid-crystal device capable of switching its polarization angle.
- the present disclosure provides an image display apparatus capable of carrying out image processing at a speed higher than the speed of the ordinary parallax image configuration by performing parallel processing such as a process to compare a signal output from every horizontal register employed in a main imager with a signal output from the corresponding horizontal register employed in an auxiliary imager integrated with the main imager in a single LSI (Large Scale Integrated) circuit in order to produce an image of high minuteness and a high rate.
- parallel processing such as a process to compare a signal output from every horizontal register employed in a main imager with a signal output from the corresponding horizontal register employed in an auxiliary imager integrated with the main imager in a single LSI (Large Scale Integrated) circuit in order to produce an image of high minuteness and a high rate.
- LSI Large Scale Integrated
- LSI circuit including main and auxiliary imagers as well as an image processing device
- a Camcoder a registered trademark of Sony Corporation
- a digital still camera which are capable of creating a pseudo 3D (three-dimensional) image by also recording information on parallaxes and distances along with image data.
- the method includes a process carried out by a pair of imagers to extract a parallax caused by a distance to an object of photographing as a positional shift between mutually corresponding pixels included in the imagers or between mutually corresponding image blocks included in the imagers as blocks conforming to the mutually corresponding pixels, the parallax is assigned to image data as a distance coefficient found on the basis of the positional shift between the mutually corresponding pixels or the mutually corresponding image blocks in the imagers.
- the device includes a pair of imagers for extracting pieces of image data with different parallaxes caused by a distance to an object of photographing; and an image processing circuit for processing the pieces of image data with different parallaxes generated by the imagers.
- the imagers and the image processing circuit are constructed in the same integrated circuit chip.
- the image processing device is further characterized in that the image processing device generates pieces of image data with different parallaxes.
- a process is carried to assign a distance coefficient to the pieces of image data, the coefficient being obtained by extracting the parallaxes each according to a positional shift between a pixel or image block conforming to the pixel generating the piece of image data on one of the imagers and a corresponding pixel or image block conforming to the corresponding pixel generating the piece of image data on the other one of the imagers, and the image data is converted into output data in accordance with the assigned distance coefficients.
- An image display apparatus includes a pair of imagers for extracting pieces of image data with different parallaxes each caused by a distance to an object of photographing.
- the apparatus further includes a coefficient assignment circuit for carrying out a process on pieces of image data with different parallaxes output by the imagers to assign a distance coefficient to the pieces of image data, the coefficient being obtained by extracting the parallaxes each according to a positional shift between a pixel or image block conforming to the pixel generating the piece of image data on one of the imagers and a corresponding pixel or image block conforming to the corresponding pixel generating the piece of image data on the other one of the imagers.
- the apparatus still further includes an image processing circuit for creating a pseudo three-dimensional image from image data output by the coefficient assignment circuit, and an image display unit for displaying the pseudo three-dimensional image generated by the image processing circuit.
- the image processing device adopting the image processing method and the image display apparatus employing the image display device, a signal output by a horizontal register employed in a main imager and a signal output by a corresponding horizontal register employed in an auxiliary imager constructed in the same (imager) LSI circuit as the main imager are compared with each other in parallel processing so that the image display apparatus is capable of carrying out image processing to produce an image of high minuteness and a high rate at a speed higher than the speed of the ordinary parallax image configuration.
- LSI circuit including such imagers, it is possible to realize a Camcoder (a registered trademark of Sony Corporation) and a digital still camera, which are capable of recording information on parallaxes and distances along with image data.
- a Camcoder a registered trademark of Sony Corporation
- a digital still camera which are capable of recording information on parallaxes and distances along with image data.
- a parallax/distance coefficient based on a parallax by using a parallax/distance coefficient based on a parallax, a 3D (three-dimensional) pseudo effect can be obtained even with only one eye.
- limitation on the visual-field angle is eliminated so that special means such as a both-eye 3D (three-dimensional) viewer is no longer required.
- FIG. 1A is a diagram showing the configuration of a parallax-detection image-taking system
- FIG. 1B is a diagram showing the structure of an LSI chip employed in the parallax-detection image-taking system
- FIG. 2A is a diagram showing the configuration of another parallax-detection image-taking system
- FIG. 2B is a diagram showing the structure of an LSI chip employed in the other parallax-detection image-taking system
- FIG. 3A is a diagram showing the configuration of a parallax-detection image-taking system for producing a 3D (three-dimensional) display;
- FIGS. 3B and 3C are diagrams respectively showing a structure of main and auxiliary imagers employed in the parallax-detection image-taking system
- FIG. 4 is a diagram showing a top view of the parallax-detection image-taking system shown in FIG. 3A ;
- FIG. 5A is a diagram showing the configuration of a typical parallax-detection image-taking system
- FIGS. 5B and 5C are diagrams showing an image respectively created on main and auxiliary imagers employed in the typical parallax-detection image-taking system
- FIG. 6A is a diagram showing the configuration of another typical parallax-detection image-taking system
- FIGS. 6B and 6C are diagrams showing an image respectively created on main and auxiliary imagers employed in the other typical parallax-detection image-taking system;
- FIG. 7A is a diagram showing the configuration of a further typical parallax-detection image-taking system
- FIGS. 7B and 7C are diagrams showing an image respectively created on main and auxiliary imagers employed in the further typical parallax-detection image-taking system;
- FIG. 8 is a diagram showing a typical concrete configuration of an LSI chip integrating main and auxiliary imagers
- FIG. 9 is a diagram showing typical parallel processing to extract parallaxes from image data and assign parallax/distance coefficients found from the parallaxes to the image data;
- FIG. 10 is a diagram showing other typical parallel processing to extract parallaxes from image data and assign parallax/distance coefficients found from the parallaxes to the image data;
- FIG. 11 is a diagram showing further typical parallel processing to extract parallaxes from image data and assign parallax/distance coefficients found from the parallaxes to the image data;
- FIG. 12A is a diagram showing data including parallax/distance coefficients as data of a reproduced image
- FIG. 12B is a diagram showing data led by adopting a method of processing the image data to demonstrate a pseudo 3D (three-dimensional) effect
- FIG. 12C is a diagram showing another data led by adopting method of processing image data to demonstrate a pseudo 3D (three-dimensional) effect
- FIG. 13A is a diagram showing images demonstrating a pseudo 3D (three-dimensional) effect produced by the typical parallel processing explained by referring to FIG. 9 ;
- FIG. 13B is a diagram showing images demonstrating a pseudo 3D (three-dimensional) effect produced by the other typical parallel processing explained by referring to FIG. 10 ;
- FIG. 13C is a diagram showing images demonstrating a pseudo 3D (three-dimensional) effect produced by the further typical parallel processing explained by referring to FIG. 11 ;
- FIG. 14A is a diagram showing data produced with assigned parallax/distance coefficients as data of a reproduced image
- FIG. 14B is a diagram showing a result of extracting image data with the parallax/distance coefficient of IV from the data shown in FIG. 14A and enlarging an image represented by the extracted image data;
- FIG. 14C is a diagram showing an image obtained as a result of extracting only image data with the parallax/distance coefficient of IV from the data shown in FIG. 14A ;
- FIG. 15A is an explanatory diagram to be referred to in description of a function to cut out a portion from an image produced by the typical parallel processing explained by referring to FIG. 9 ;
- FIG. 15B is an explanatory diagram to be referred to in description of a function to cut out a portion from an image produced by the other typical parallel processing explained by referring to FIG. 10 ;
- FIG. 15C is an explanatory diagram to be referred to in description of a function to cut out a portion from an image produced by the further typical parallel processing explained by referring to FIG. 11 .
- FIGS. 1A and 1B are a diagram showing a parallax-extraction image-taking system 10 for extracting a parallax by using a pair of imagers in accordance with an embodiment.
- the parallax-extraction image-taking system 10 shown in FIGS. 1A and 1B has a configuration including a plurality of imager LSI chips.
- a typical configuration shown in the figure includes two imager LSI chips 12 and 14 serving as main and auxiliary imagers respectively.
- An imager image processing device not shown in the figure is connected to the two imager LSI chips 12 and 14 and includes an image processing circuit for carrying out image processing on main image data and auxiliary image data, which are output by the two imager LSI chips 12 and 14 respectively.
- a lens 11 projects an optical signal representing an image of a photographing object 15 located at a first distance from the position of the lens 11 on an image-taking face of the main-imager LSI chip 12 .
- a lens 13 projects an optical signal representing an image of another photographing object 16 located at a second distance from the position of the lens 13 on an image-taking face of the auxiliary-imager LSI chip 14 .
- the two imager LSI chips 12 and 14 are each a solid-state image-taking device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) device.
- the two imager LSI chips 12 and 14 each have a configuration including a number of pixels.
- the main-imager LSI chip 12 converts a pixel optical signal leaving the lens 11 for the main-imager into an electrical signal.
- the auxiliary-imager LSI chip 14 converts a pixel optical signal leaving the lens 13 for the auxiliary-imager into an electrical signal.
- FIG. 1B is a diagram showing a typical main-imager LSI chip 12 (or a typical auxiliary-imager LSI chip 14 ) implemented by a CMOS device.
- RGB pixels are laid out repeatedly to form a pixel array 21 serving as a column of the main-imager LSI chip 12 (or the auxiliary-imager LSI chip 14 ).
- the pixel arrays 21 are laid out repeatedly in the horizontal direction (or the row direction) alternately with vertical registers 22 .
- a predetermined number of pixel arrays 21 are laid out alternately with the same number of vertical registers 22 in the row direction to form the so-called pixel matrix.
- An optical signal hitting a pixel is converted into an electrical signal, which is then transferred to a vertical register 22 adjacent to the pixel array 21 including the pixel synchronously with a clock signal generated by a timing generator.
- An AD (analog-to-digital) converter converts a signal output by a vertical register 22 into a digital signal, which is supplied to a horizontal register 23 before being output from the main-imager LSI chip 12 (or the auxiliary-imager LSI chip 14 ) with a predetermined timing.
- the digital signals output from the main-imager LSI chip 12 and the auxiliary-imager LSI chip 14 are supplied to a signal processing circuit not shown in the figure as main-image data and auxiliary-image data respectively.
- the signal processing circuit then processes the main-image data and the auxiliary-image data in order to generate a pseudo 3D (three-dimensional) image.
- An image processing method for processing the main-image data and the auxiliary-image data will be described later along with an image processing device adopting the image processing method.
- a parallax-extraction image-taking system provided with the two imager LSI chips 12 and 14 forming a pair of main and auxiliary imagers as shown in FIG. 1A needs the lenses 11 and 13 in addition to the two imager LSI chips 12 and 14 forming a pair of imagers each serving as a sensor in order to recognize distances to the objects of photographing.
- the two imager LSI chips 12 and 14 are LSI chips independent of each other.
- the two imager LSI chips 12 and 14 each output data of an image obtained as a result of an image-taking process as a serial electrical signal by way of vertical registers and a horizontal register.
- horizontal registers are placed at positions preceding the position of a vertical register.
- a serial main-image signal output by the main-imager LSI chip 12 and a serial auxiliary-image signal output by the auxiliary-imager LSI chip 14 are compared with each other in order to detect parallaxes in image processing to be described later.
- the figure shows the main-imager LSI chip 12 on the left side and the auxiliary-imager LSI chip 14 on the right side, and the following description assumes these positions of the main-imager LSI chip 12 and the auxiliary-imager LSI chip 14 . It is worth noting, however, that the positions of the main-imager LSI chip 12 and the auxiliary-imager LSI chip 14 can be reversed.
- FIGS. 2A and 2B are a diagram roughly showing a parallax-extraction image-taking system 50 employing a single imager LSI in accordance with another embodiment of the present invention.
- Reference numeral 54 denotes the imager LSI, including a main imager 54 A and an auxiliary imager 54 B.
- the main imager 54 A and the auxiliary imager 54 B are each typically a solid-state image-taking device such as a CCD or a CMOS device.
- the main imager 54 A and the auxiliary imager 54 B are constructed in one LSI chip implemented by a CMOS device.
- the parallax-extraction image-taking system 50 shown in FIG. 2A includes a first photographing object 58 at a position relatively close to lenses 51 and 55 and a second photographing object 59 at a position relatively far from the lenses 51 and 55 .
- Light beams generated by the first photographing object 58 and the second photographing object 59 are subjected to an image-taking process after passing through the lenses 51 and 55 .
- the light beams representing the images of the first photographing object 58 and the second photographing object 59 are reflected by reflection plates 52 and 53 each serving as a mirror after being passed by the lens 51 , and enter the main imager 54 A included in the integrated LSI chip 54 .
- the light beams representing the images of the first photographing object 58 and the second photographing object 59 are reflected by reflection plates 56 and 57 each serving as a mirror after being passed by the lens 55 , and enter the auxiliary imager 54 B included in the integrated LSI chip 54 .
- the first photographing object 58 and the second photographing object 59 can be projected on a light receiving area in the LSI chip 54 even though the main and auxiliary imagers 54 A and 54 B are separated from each other.
- FIG. 2B is a block diagram showing the integrated LSI chip 54 including the main imager 54 A, the auxiliary imager 54 B and an image processing device in accordance with this embodiment.
- the main imager 54 A, the auxiliary imager 54 B and the image processing device are integrated in a single LSI chip 54 implemented by a CMOS device.
- the block diagram is simplified in order to make the explanation or the LSI chip 54 simple.
- the main imager 54 A has pixel rows 61 A, 61 B and 61 C, which are each oriented in the horizontal direction.
- the LSI chip 54 also includes horizontal registers 62 A, 62 B and 62 C respectively.
- the auxiliary imager 54 B has pixel rows 61 A′, 61 B′ and 61 C′, which are each oriented in the horizontal direction.
- the LSI chip 54 also includes horizontal registers 62 A′, 62 B′ and 62 C′ respectively.
- the outputs of the horizontal register 62 A for the main imager 54 A and the horizontal register 62 A′ for the auxiliary imager 54 B are connected to a parallel processing circuit 63 A.
- the outputs of the horizontal register 62 B for the main imager 54 A and the horizontal register 62 B′ for the auxiliary imager 54 B are connected to a parallel processing circuit 63 B.
- the outputs of the horizontal register 62 C for the main imager 54 A and the horizontal register 62 C′ for the auxiliary imager 54 B are connected to a parallel processing circuit 63 C.
- the outputs of the parallel processing circuits 63 A, 63 B and 63 C are connected to a vertical register 64 , for sequentially receiving processing results from the parallel processing circuits 63 A, 63 B and 63 C.
- a vertical register 64 for sequentially receiving processing results from the parallel processing circuits 63 A, 63 B and 63 C.
- three pixel arrays and the three corresponding horizontal registers composing a total of six rows are shown. In actuality, however, more rows can be provided to obtain the required resolution.
- the LSI chip 54 has the main imager 54 A and the auxiliary imager 54 B each serving as a sensor. On each pixel row oriented in the horizontal (or x-axis) direction in the area of the sensors, typically, a predetermined number of pixels are laid out and, in the vertical direction, such pixel rows are laid out alternately with the horizontal registers.
- R, G and B pixels are laid out repeatedly and connected to a horizontal register associated with the pixel row.
- Each of the pixels employs a row select transistor, a reset transistor, an amplification transistor and a photodiode.
- every row is also provided with a noise canceller, a signal read transistor, the horizontal register and an AD (Analog-to-Digital) converter, which are not shown in the figure.
- the output of the AD converter is connected to the horizontal register provided for the row.
- each of the horizontal scanning circuit and the vertical scanning circuit includes a horizontal register, which starts a shift operation to scan columns or rows synchronously with a driving pulse generated by a TG (timing generator) also not shown in the figure.
- the horizontal scanning circuit sequentially generates horizontal scan (select) pulses to be supplied to column read lines and column signal lines.
- a pixel signal read out from a specific pixel selected by a column read line and a row select line is output to the column signal line.
- the vertical scanning circuit sequentially generates vertical scan pulses to be supplied to row select lines to sequentially select pixel rows, which are laid out in the vertical direction.
- An image signal read out from a column signal line is supplied typically to a CDS (Correlation Double Sampling) circuit serving as a difference computation circuit for carrying out a process to find a difference generated right after a pixel reset operation on the basis of sampling pulses generated by the timing generator as a difference between a noise level and a signal level.
- a CDS Correlation Double Sampling
- components such as an AGC (Automatic Gain Control) circuit and an ADC (Analog Digital Converter) circuit are provided.
- Digital signals output by the ADC circuits as signals representing image data are supplied to the horizontal registers 62 A to 62 C and 62 A′ to 62 C′.
- the pieces of image data supplied to the horizontal registers 62 A to 62 C are sequentially passed on to the parallel processing circuits 63 A to 63 C respectively and the image data supplied to the horizontal registers 62 A′ to 62 C′ are also sequentially passed on to the parallel processing circuits 63 A to 63 C respectively.
- the parallel processing circuits 63 A to 63 C carry out a process to detect pixel shifts caused by parallaxes and other processing.
- the parallax-extraction image-taking system 50 based on a monolithic imager as shown in FIGS. 2A and 2B is configured to include the main imager 54 A and the auxiliary imager 54 B on the same substrate of the LSI chip so as to function as an image-taking system for extracting a parallax and a distance.
- the single LSI chip 54 serving as imagers can be constructed into a configuration in which a signal generated by a horizontal register employed in the main imager 54 A and a signal generated by the corresponding horizontal register employed in the auxiliary imager 54 B are compared with each other for every pair of a horizontal register and a corresponding horizontal register in parallel processing carried out at the same time for all the pairs.
- the LSI configuration including the main imager, the auxiliary imager and the parallel comparison/processing circuit, which are integrated in one chip, also has system-configuration merits and manufacturing merits.
- the distance between lenses for extracting a parallax should be normally set at a value greater than the distance between the main and auxiliary imagers integrated in one LSI chip. For this reason, an optical system having the inter-lens distance matching the inter-imager distance is obviously desired.
- FIGS. 3A to 3 C are a diagram showing the configuration of a parallax-extraction image-taking system 100 for displaying a 3D (three-dimensional) image.
- FIG. 4 is a diagram showing a top view of the parallax-extraction image-taking system 100 as a coordinate recognition model of a parallax extraction process and an image-taking process. The following description explains a process to compare image data generated by a main imager 74 with image data generated by an auxiliary imager 75 in the horizontal direction, a process to extract a parallax of image data and a process to assign a parallax/distance coefficient to the image data.
- setting of dimensions is by no means limited to setting at values shown in FIGS. 3A to 3 C and FIG. 4 .
- that shown in FIG. 3A is taken as the setting of dimensions.
- the distance between a lens 51 and a lens 55 is 4d; whereas the distance between the lens pair consisting of the lens 51 and the lens 55 and the imager pair consisting of the main imager 74 and the auxiliary imager 75 is 5d.
- the distance between the lens 51 and a first photographing object 71 is 10d; the distance between the lens 51 and a second photographing object 72 is 20d; and the distance between the lens 51 and a third photographing object 73 is 40d.
- the first photographing object 71 is a rectangular plate, which has a side of 2d and is marked with the capitals X, Y, Z and W.
- the second photographing object 72 is also a rectangular plate, which has a side of 8d and is marked notations ⁇ 1 to ⁇ 1 .
- the third photographing object 73 is also a rectangular plate, which has a side of 24d and is marked with symbols ⁇ 1 to ⁇ 1 .
- the character “d” denotes any arbitrary distance unit and is not limited to a specific value.
- An image of an object of photographing is created by the lens 51 or 55 on one of the third photographing object 73 and the main imager 74 respectively, which each have dimensions of 4d ⁇ 3d.
- the image created by the lens 55 or 51 on the auxiliary imager 75 or the main imager 74 respectively has matrix elements arranged in an order opposite to the order of the arrangement of the same elements in the object of photographing.
- an image created by the lens 51 on the main imager 74 as overlapping images of the first photographing object 71 , the second photographing object 72 and the third photographing object 73 is shown in FIG. 3B .
- FIG. 3C an image created by the lens 55 on the auxiliary imager 75 as overlapping images of the first photographing object 71 , the second photographing object 72 and the third photographing object 73 is shown in FIG. 3C .
- the matrix elements themselves which are the capitals X, Y, Z and W, the notations ⁇ 1 to ⁇ 1 and the symbols ⁇ 1 to ⁇ 1 , are deliberately rotated by 180 degrees to prevent the matrix elements from being shown as upside-down elements in order to make the capitals, the notations and the symbols easy to recognize.
- the created overlapping images are shown in FIGS. 3B and 3C , being placed on a column-row coordinate system.
- the created images are each an 8-column ⁇ 6-row matrix with each column having a width of d/2 and each row having a height of d/2.
- the columns are columns 1 to 9 whereas the rows are rows A to F.
- the horizontal-direction shift of the image created on the auxiliary imager 75 as the image of the first photographing object 71 which locates near the lens 55 is largest among the horizontal-direction shifts of the images created on the auxiliary imager 75 .
- the horizontal-direction shift of the image created on the auxiliary imager 75 as the image of the third photographing object 73 which locates far from the lens 55 is smallest among the horizontal-direction shifts of the images created on the auxiliary imager 75 .
- the image of the third photographing object 73 is created on coordinate blocks starting with the coordinate blocks A 2 to A 7 of the first row and ending with the coordinate blocks F 2 to F 7 of the last row.
- the image of the third photographing object 73 is created as symbols ⁇ 1 to ⁇ 1 respectively
- the image of the third photographing object 73 is created in the same way as the first row and, on the coordinate blocks F 2 to F 7 of the last row, the image of the third photographing object 73 is created as symbols ⁇ 1 to ⁇ 1 respectively.
- the image of the second photographing object 72 is created as an overlapping image on coordinate blocks starting with the coordinate blocks B 3 to B 6 of the first row and ending with the coordinate blocks E 3 to E 6 of the last row.
- the image of the second photographing object 72 is created as notations ⁇ 1 to ⁇ 3 respectively
- the image of the second photographing object 72 is created in the same way as the first row and, on the coordinate blocks E 3 to E 6 , the image of the second photographing object 72 is created as notations ⁇ 4 to ⁇ 1 respectively.
- the image of the first photographing object 71 is created as an overlapping image on coordinate blocks starting with the coordinate blocks C 4 and C 5 of the first row and ending with the coordinate blocks D 4 and D 5 of the last row.
- the image of the first photographing object 71 is created as the capitals W and Z respectively and, on the coordinate blocks D 4 and D 5 , the image of the first photographing object 71 is created as the capitals Y and X respectively.
- the image created on the main imager 74 shown in FIG. 3B as the image of the third photographing object 73 is obtained by rotating the symbol matrix of the uppermost plate shown in FIG. 3A by 180 degrees due to the effect of the lens 51 .
- the image created on the main imager 74 shown in FIG. 3B as the image of the second photographing object 72 is obtained by rotating the notation matrix of the middle plate shown in FIG. 3A by 180 degrees due to the effect of the lens 51 .
- the image created on the main imager 74 shown in FIG. 3B as the image of the first photographing object 71 is obtained by rotating the capital matrix of the lowermost plate shown in FIG. 3A by 180 degrees due to the effect of the lens 51 .
- the images shown in FIG. 3B are not shifted from their references due to distances to the objects of photographing.
- FIG. 3C is a diagram showing the same overlapping images as those shown in FIG. 3B as except that the horizontal coordinates blocks of the coordinate system shown in FIG. 3C each have an offset of +1 with respect to the coordinate system shown in FIG. 3B .
- the image of the third photographing object 73 is created on coordinate blocks starting with the coordinate blocks A 3 to A 8 of the first row and ending with the coordinate blocks F 3 to F 8 of the last row.
- the image of the third photographing object 73 is created as symbols ⁇ 1 to ⁇ 1 respectively
- the image of the third photographing object 73 is created in the same way as the first row and, on the coordinate blocks F 3 to F 8 of the last row, the image of the third photographing object 73 is created as symbols ⁇ 1 to ⁇ 1 respectively.
- the image of the second photographing object 72 is created as an overlapping image on coordinate blocks starting with the coordinate blocks B 5 to B 8 of the first row and ending with the coordinate blocks E 5 to E 8 of the last row.
- the image of the second photographing object 72 is created as notations ⁇ 1 to ⁇ 3 respectively
- the image of the second photographing object 72 is created in the same way as the first row and, on the coordinate blocks E 5 to E 8 , the image of the second photographing object 72 is created as notations ⁇ 4 to ⁇ 1 respectively.
- the image of the first photographing object 71 is created as an overlapping image on coordinate blocks starting with the coordinate blocks C 8 and C 9 of the first row and ending with the coordinate blocks D 8 and D 9 of the last row.
- the image of the first photographing object 71 is created as the capitals W and Z respectively and, on the coordinate blocks D 8 and D 9 , the image of the first photographing object 71 is created as the capitals Y and X respectively.
- the image created on the auxiliary imager 75 shown in FIG. 3C as the image of the third photographing object 73 is obtained by rotating the symbol matrix of the uppermost plate shown in FIG.
- the image created on the auxiliary imager 75 shown in FIG. 3C as the image of the second photographing object 72 is obtained by rotating the notation matrix of the middle plate shown in FIG. 3A by 180 degrees due to the effect of the lens 55 .
- the image created on the auxiliary imager 75 shown in FIG. 3C as the image of the first photographing object 71 is obtained by rotating the capital matrix of the lowermost plate shown in FIG.
- FIG. 4 is a diagram showing a top view 150 of the parallax-extraction image-taking system 100 . The same dimensions as those taken in FIG. 3 are used in FIG. 4 .
- image creation lines for creating the image of an object of photographing on the main imager 74 through the lens 51 are each drawn as a solid line whereas image creation lines for creating the image of an object of photographing on the auxiliary imager 75 through the lens 55 are each drawn as a broken line.
- the center line of the main imager 74 is set as a line passing through the centers of the first photographing object 71 , the second photographing object 72 and the third photographing object 73 .
- the center of the auxiliary imager 75 is separated away in the horizontal direction from the center of the main imager 74 by a distance of 4d.
- the centers of the first photographing object 71 , the second photographing object 72 and the third photographing object 73 are shifted from the center line of the auxiliary imager 75 and the lens 55 by the distance of 4d.
- the position of an image created on the auxiliary imager 75 as the image of an object of photographing is shifted from the position of an image created on the main imager 74 as the image of the same object of photographing. The closer the distance from the lens 55 to the object of photographing, the larger the positional shift of the image of the object of photographing.
- the position of the image of the first photographing object 71 is shifted to the right from the image position on the main imager 74 shown in FIG. 3B .
- the first photographing object 71 is shifted more to the right from the center line of the auxiliary imager 75 and the lens 55 in comparison with the second photographing object 72 and the third photographing object 73 .
- FIG. 5 is a diagram showing a first typical implementation of a parallax-extraction image-taking coordinate-recognition system 200 according to an embodiment.
- FIG. 5A is a diagram of the typical implementation of the principle.
- the block unit d is set at 12 mm.
- Reference numeral 71 denotes a mini car serving as a front scene separated away from the lens 51 by a distance of 240 mm.
- Reference numeral 73 denotes two books A and B, which serve as a rear scene (or a background) separated away from the lens 51 by a distance of 480 mm.
- the two books A and B are each shown in the figure as a hatched block.
- the image of the mini car 71 is shown as a figure enclosed by a solid line whereas the image of the A and B books 73 serving as the background is shown as hatched blocks.
- the image of the mini car 71 occupies an area of coordinate blocks A 3 to A 7 and B 5 to B 6 .
- the image of the mini car 71 occupies an area of coordinate blocks A 5 to A 9 and B 7 to B 8 .
- the image created on the auxiliary imager 75 as the image of the mini car 71 is shifted to the right side in the block coordinate system rotated by 180 degrees from the image created on the main imager 74 shown in FIG. 5B as the image of the mini car 71 by two blocks including the +1 pre-offset.
- the shift of the image created on the auxiliary imager 75 as the image of the mini car 71 from the image created on the main imager 74 as the image of the mini car 71 in the horizontal direction on the block coordinate system is recognized.
- the image created on the auxiliary imager 75 as the image of the A and B books 73 serving as the background is slightly shifted in the right direction from the image created on the main imager 74 as the image of the A and B books 73 by a distance of about one block including even the +1 pre-offset.
- the shift of the image of the A and B books 73 is smaller than the shift of the image of the mini car 71 .
- FIG. 6 is a diagram showing a second typical implementation of a parallax-extraction image-taking coordinate-recognition system 250 according to the embodiment of the present invention.
- the mini car 71 has slightly moved in the horizontal direction.
- the image of the mini car 71 occupies an area of coordinate blocks A 2 to A 6 and B 4 to B 5 .
- the image of the mini car 71 occupies an area of coordinate blocks A 4 to A 8 and B 6 to B 7 .
- the image created on the auxiliary imager 75 as the image of the mini car 71 is shifted to the right side in the block coordinate system from the image created on the main imager 74 shown in FIG. 6B as the image of the mini car 71 by two blocks.
- the image created on the auxiliary imager 75 as the image of the A and B books 73 serving as the background is slightly shifted in the right direction from the image created on the main imager 74 as the image of the A and B books 73 by a distance of about one block.
- FIG. 7 is a diagram showing a third typical implementation of a parallax-extraction image-taking coordinate-recognition system 300 according to the embodiment.
- the mini car 71 has further moved along an abruptly curved road approaching the lens 51 and reached a position separated from the lens 51 by a distance of 120 mm.
- the image of the mini car 71 occupies an area of coordinate blocks A 1 to A 4 and B 2 to B 4 .
- the image of the mini car 71 occupies an area of coordinate blocks A 5 to A 8 and B 6 to B 8 .
- the images created on the auxiliary imager 75 B as the images of the mini car 71 and the A and B books 73 are shifted to the right side from the coordinate blocks A 1 to A 4 and B 2 to B 4 occupied by the images created on the main imager 74 shown in FIG.
- FIGS. 7B and 7C are compared with FIGS. 5B and 5C respectively of FIGS. 6B and 6C respectively, it will become obvious that, as the mini car 71 serving as the front scene approaches the lenses 51 and 55 , the horizontal-direction block shift between the image created on the main imager 74 A or 75 A as the image of the mini car 71 and the image created on the main imager 74 B or 75 B as the image of the mini car 71 increases.
- FIG. 8 is a diagram showing a typical concrete configuration of an LSI chip 350 including a main imager 74 C, an auxiliary imager 75 C and an image processing circuit, which are shown in FIG. 2 .
- the LSI chip 350 is denoted by reference numeral 54 in FIG. 2 .
- the image processing circuit includes horizontal registers 351 A to 351 F provided for the main imager 74 C, horizontal registers 352 A to 352 F provided for the auxiliary imager 75 C, parallax detection comparison circuits 353 A to 353 F, parallax normalization coefficient computation circuits 354 A to 354 F, parallax/distance coefficient assignment circuits 355 A to 355 F, registers 356 A to 356 F, a vertical register 357 and registers 361 to 366 .
- Pieces of image data represented by symbols ⁇ 1 , ⁇ 5 , ⁇ 4 , ⁇ 3 , ⁇ 2 and ⁇ 1 at coordinate blocks A 2 to A 7 of the main imager 74 C are transferred to the horizontal register 351 A.
- pieces of image data represented by symbols ⁇ 5 , ⁇ 1 , ⁇ 5 , ⁇ 4 , ⁇ 3 and ⁇ 5 at coordinate blocks B 2 to B 7 of the main imager 74 C are transferred to the horizontal register 351 B.
- pieces of image data on the subsequent rows of coordinate blocks of the main imager 74 C are transferred to the subsequent horizontal registers.
- pieces of image data represented by symbols ⁇ 1 , ⁇ 5 , ⁇ 4 , ⁇ 3 , ⁇ 2 and ⁇ 1 at coordinate blocks F 2 to F 7 of the main imager 74 C are transferred to the horizontal register 351 F.
- pieces of image data are transferred from rows of blocks on the auxiliary imager 75 C to the horizontal registers 352 A to 352 F corresponding to the block rows.
- the parallax detection comparison circuits 353 A to 353 F, the parallax normalization coefficient computation circuits 354 A to 354 F and the parallax/distance coefficient assignment circuits 355 A to 355 F compose a system for carrying out parallel processing such as parallax extraction and parallax/distance coefficient assignment processes.
- the parallax detection comparison circuits 353 A to 353 F receive the pieces of image data from the horizontal registers 351 A to 351 F of the main imager 74 C and the pieces of image data from the horizontal registers 352 A to 352 F of the auxiliary imager 75 C, comparing the pieces of data received from the main imager 74 C and the pieces of data received from the auxiliary imager 75 C with each other in order to detect a parallax for each coordinate block.
- the image data represented by symbol ⁇ 4 at coordinate block C 2 of the main imager 74 C is located at coordinate block C 3 of the auxiliary imager 75 C, causing a parallax to be detected as a block shift of I.
- the image data represented by notation ⁇ 2 at coordinate block C 3 of the main imager 74 C is located at coordinate block C 5 of the auxiliary imager 75 C, causing a parallax to be detected as a block shift of II.
- the image data represented by the capital W at coordinate block C 4 of the main imager 74 C is located at coordinate block C 8 of the auxiliary imager 75 C, causing a parallax to be detected as a block shift of IV.
- the image data represented by the capital Z at coordinate block C 5 of the main imager 74 C is located at coordinate block C 9 of the auxiliary imager 75 C, causing a parallax to be detected as a block shift of IV.
- the image data represented by notation ⁇ 4 at coordinate block C 6 of the main imager 74 C is located at coordinate block C 10 (not shown in the figure) of the auxiliary imager 75 C, causing a parallax to be detected as a block shift of II.
- the image data represented by symbol ⁇ 4 at coordinate block C 7 of the main imager 74 C is located at coordinate block C 11 (not shown in the figure) of the auxiliary imager 75 C, causing a parallax to be detected as a block shift of I.
- the parallax normalization coefficient computation circuits 354 A to 354 F each carry out a normalization process to find a normalized parallax/distance coefficient for a parallax detected by the parallax detection comparison circuits 353 A to 353 F respectively as block shifts.
- the distance from the main imager 74 C or the auxiliary imager 75 C to the rear scene is normalized to 1.
- a parallax/distance coefficient found from a parallax detected by the parallax detection comparison circuit 353 for a photographing object serving as a front scene is equal to a fraction, which would be obtained as a result of dividing the actual distance from the main imager 74 C or the auxiliary imager 75 C to the object of photographing by the distance from the main imager 74 C or the auxiliary imager 75 C to the rear scene.
- the position of a rear scene is reversely taken as a reference of 0 and the distance from the rear scene to the lens 51 or 55 is normalized to 1.
- a parallax/distance coefficient found from a parallax detected by the parallax detection comparison circuit 353 for a photographing object serving as a front scene is equal to a fraction, which would be obtained as a result of dividing the actual distance from the rear scene to the object of photographing by the distance from the rear scene to the lens 51 or 55 .
- a detected parallax is thus represented by a normalized parallax/distance coefficient, which is equal to a fraction, which would be obtained as a result of dividing the actual distance from a reference to an object of photographing by a distance normalized to 1 as described above. Then, the normalized parallax/distance coefficient found from a parallax detected by the parallax detection comparison circuit 353 is normalized again into another value as described below and assigned to a coordinate block corresponding to the object of photographing as a final parallax/distance coefficient.
- the parallax/distance coefficient assignment circuits 355 A to 355 F each further carry out additional processing on the normalized parallax/distance coefficient coefficients output by the parallax normalization coefficient computation circuits 354 A to 354 F respectively.
- the normalized parallax/distance coefficient is multiplied by a value to result in a product, which is then typically rounded to produce an integer to be assigned as a final parallax/distance coefficient to the corresponding block.
- digital processes of the subsequent processes become easy to carry out. It is needless to say that, in place of an integer, another value or a code can also be used as the parallax/distance coefficient.
- the aforementioned block shifts of I, II and IV each detected by the parallax detection comparison circuits 353 A to 353 F as a value inversely proportional to the distance to an object of photographing can each be assigned to a corresponding block of the main imager 74 C as a parallax/distance coefficient as it is.
- the parallax/distance coefficient can be normalized and assigned to the corresponding block as a coefficient proportional to the distance.
- the block shifts of I, II and IV are assigned to their respective corresponding blocks as they are and the vertical register 357 outputs a serial image signal including attached parallax/distance coefficients.
- parallax/distance coefficients are shown under each of the registers 356 A to 356 F.
- the parallax/distance coefficients each represent a relation between image data of the main imager 74 C and the corresponding image data of the auxiliary imager 75 C.
- the parallax/distance coefficients each representing a block shift are stored in memories associated with the registers 356 A to 356 F.
- the storage method and the storage means are not limited to what is described here.
- the parallax/distance coefficient assignment circuits 355 A to 355 F supply pieces of image data and parallax/distance coefficients assigned to them to the registers 356 A to 356 F respectively.
- the image data ⁇ 1 is stored in sub-register 2 of the register 356 A and the image data ⁇ 5 is stored in sub-register 3 of the register 356 A.
- the subsequent pieces of image data are stored in their respective subsequent sub-registers of the register 356 A and the last image data ⁇ 1 is stored in sub-register 7 of the register 356 A.
- the parallax/distance coefficients of I, I, - - - and I each representing a block shift are stored in a memory by being associated with their respective pieces of image data stored in sub-registers 2 to 7 of the register 356 A.
- pieces of image data are stored in the register 346 B and the parallax/distance coefficients assigned to the pieces of image data are stored in a memory associated with the register 356 B.
- the image data ⁇ 4 is stored in sub-register 2 of the register 356 C
- the image data ⁇ 2 is stored in sub-register 3 of the register 356 C
- the image data W is stored in sub-register 4 of the register 356 C
- the image data Z is stored in sub-register 5 of the register 356 C
- the image data ⁇ 4 is stored in sub-register 6 of the register 356 C
- the image data 64 is stored in sub-register 7 of the register 356 C.
- the parallax/distance coefficients of I, II, IV, IV, II and I each representing a block shift are stored in a memory by being associated with their respective pieces of image data stored in sub-registers 2 to 7 of the register 356 C.
- the process to store pieces of image data and their parallax/distance coefficients is repeated till pieces of image data and their parallax/distance coefficients are stored respectively in the register 356 F and a memory associated with the register 356 F.
- the embodiment shown in FIG. 8 includes only few pixels.
- the auxiliary imager 75 C does not include pixels corresponding to coordinate blocks C 10 and C 11 associated with respectively sub-registers 6 and 7 of the register 356 C and does not include pixels corresponding to coordinate blocks D 10 and D 11 associated with respectively sub-registers 6 and 7 of the register 356 D.
- sub-registers 6 and 7 of the register 356 C are complementarily associated with the parallax/distance coefficients of II and I respectively and, by the same token, sub-registers 6 and 7 of the register 356 D are also complementarily associated with the parallax/distance coefficients of II and I respectively.
- the pieces of image data stored in the registers 356 A to 356 F are supplied to the vertical register 357 and then serially transferred to the registers 361 to 366 before being output to image-data recipients such as a display unit, a storage device and a network.
- FIGS. 9, 10 and 11 are diagrams respectively showing first, second and third embodiments each implementing parallel processing of the parallax extraction and parallax/distance coefficient assignment processes described above.
- the first embodiment shown in FIG. 9 is an embodiment of a process to assign parallax/distance coefficients to image data obtained as a result of the image-taking operation shown in FIG. 5 by adoption of the principle explained earlier by referring to FIG. 8 .
- the parallax/distance coefficient of II is assigned to the mini car serving as the front scene whereas the parallax/distance coefficient of I is assigned to the two books and a screen, which serves as a background.
- the pieces of image data existing in the coordinate blocks B 1 to B 4 are data of the books and the background
- the pieces of image data existing in the coordinate blocks B 5 and B 6 are data of the mini car
- the pieces of image data existing in the coordinate blocks B 7 and B 8 are data of the books and the background.
- pieces of image data of the two books and the background exist as a result of the image-taking process.
- the image data resulting from the image-taking process on the main imager 74 D as the image data of the books is shifted on the auxiliary imager 75 D by a distance not exceeding one block provided that the +1 pre-offset is not taken into consideration.
- the image data resulting from the image-taking process on the main imager 74 D as the image data the mini car is shifted on the auxiliary imager 75 D by a distance of one block provided that the +1 pre-offset is not taken into consideration or a distance not exceeding two blocks provided that the +1 pre-offset is taken into consideration.
- the pieces of image data existing in the coordinate blocks A 2 to A 4 are data of the books and the background whereas the pieces of image data existing in the coordinate blocks A 5 to A 9 are data of the mini car.
- the pieces of image data existing in the coordinate blocks B 2 to B 6 are data of the books and the background
- the pieces of image data existing in the coordinate blocks B 7 and B 8 are data of the mini car whereas the piece of image data existing in the coordinate block B 9 is data of the books and the background.
- the image data stored in the horizontal register 351 A of the main imager 74 D and the image data stored in the horizontal register 352 A of the auxiliary imager 75 D are supplied to the parallax-detection comparison circuit 353 A.
- the pieces of image data stored in the horizontal registers 351 B to 351 F of the main imager 74 D and the pieces of image data stored in the horizontal registers 352 B to 352 F of the auxiliary imager 75 D are supplied to the parallax-detection comparison circuits 353 B to 353 F respectively.
- the parallax-detection comparison circuits 353 A to 353 F each detect image-data shifts by comparing the pieces of image data supplied from the main imager 74 D with the pieces of image data supplied from the auxiliary imager 75 D. Then, the parallax normalization coefficient computation circuits 354 A to 354 F each carry out a normalization process based on the image-data shifts to generate normalized parallax/distance coefficients. Subsequently, the parallax/distance coefficient assignment circuits 355 A to 355 F each assign the normalized parallax/distance coefficients output by parallax normalization coefficient computation circuits 354 A to 354 F to the pieces of image data supplied by the main imager 74 D.
- a parallax/distance coefficient of I is assigned to the image data of the two books and the background whereas a parallax/distance coefficient of II is assigned to the image data of the mini car.
- the pieces of image data supplied by the main imager 74 D are stored in the registers 356 A to 356 F whereas the parallax/distance coefficients assigned to the pieces of image data are stored in a memory or storage unit associated with the registers 356 A to 356 F.
- the parallax/distance coefficient of I assigned to the pieces of image data stored in sub-registers 1 , 2 and 8 of the register 356 A as well as the parallax/distance coefficient of II assigned to the pieces of image data stored in sub-registers 3 to 7 of the register 356 A are stored in a memory or storage unit associated with the register 356 A.
- the parallax/distance coefficient of I assigned to the pieces of image data stored in sub-registers 1 to 4 , 7 and 8 of the register 356 B as well as the parallax/distance coefficient of II assigned to the pieces of image data stored in sub-registers 5 and 6 of the register 356 B are stored in a memory or storage unit associated with the register 356 B.
- the parallax/distance coefficient of I assigned to the pieces of image data stored in all sub-registers of each of the remaining registers 356 C to 356 F are stored in a memory or storage unit associated with each of the registers 356 C to 356 F.
- the pieces of image data and the parallax/distance coefficients assigned to the pieces of image data are output to image-data recipients such as a display unit, a storage device and a network.
- FIG. 10 is a diagram showing the second embodiment of a process to assign parallax/distance coefficients to image data obtained as a result of the image-taking operation shown in FIG. 6 .
- the mini car moves slightly in the horizontal direction.
- the position of the mini car is shifted in a main imager 74 E shown in FIG. 10 from the position of the mini car in the main imager 74 D shown in FIG. 9 in the horizontal direction to the left by one block.
- the image is shifted in an auxiliary imager 75 E from the position of the mini car in the main imager 74 E in the horizontal direction to the right by one block. Since the mini car moves in the horizontal direction, the distance from the mini car to the lenses 51 and 55 changes only a little bit. Thus, changes caused by the parallax between the main imager 74 E and the auxiliary imager 75 E are also small as well.
- the parallax/distance coefficient of II is assigned to the image data of the mini car much like the first embodiment shown in FIG. 9 .
- the distance from the two books and the screen serving as the background to the lenses 51 and 55 does not change.
- the parallax/distance coefficient of I is assigned to the image data of the two books and the background.
- the parallax/distance coefficient of I assigned to the pieces of image data stored in sub-registers 1 , 7 and 8 of the register 356 A as well as the parallax/distance coefficient of II assigned to the pieces of image data stored in sub-registers 2 to 6 of the register 356 A are stored in a memory or storage unit associated with the register 356 A.
- the parallax/distance coefficient of I assigned to the pieces of image data stored in sub-registers 1 to 3 and 6 to 8 of the register 356 B as well as the parallax/distance coefficient of II assigned to the pieces of image data stored in sub-registers 4 and 5 of the register 356 B are stored in a memory or storage unit associated with the register 356 B.
- the parallax/distance coefficient of I assigned to the pieces of image data stored in all sub-registers of each of the remaining registers 356 C to 356 F are stored in a memory or storage unit associated with each of the registers 356 C to 356 F.
- the pieces of image data and the parallax/distance coefficients assigned to the pieces of image data are output to image-data recipients such as a display unit, a storage device and a network.
- FIG. 11 is a diagram showing the third embodiment of a process to assign parallax/distance coefficients to image data obtained as a result of the image-taking operation shown in FIG. 7 .
- the mini car further moves in a direction approaching the lenses 51 and 55 .
- a parallax/distance coefficient of IV is assigned to the image data of the mini car serving as a front scene.
- the parallax/distance coefficient of I is assigned to the image data of the two books and the background.
- the parallax/distance coefficient of IV is assigned to the image data of the mini car because the distance from the lenses 51 and 55 to the approaching mini car has changed substantially so that the parallaxes obtained as differences in image data between the main imager 74 F and the auxiliary imager 75 F also have become larger considerably as well. As described earlier, these parallax/distance coefficient are typically set in accordance with parallaxes caused by the distance from the lenses 51 and 55 to the object of photographing.
- Pieces of image data existing in the coordinate blocks A 1 to A 4 of the main imager 74 F as a result of an image-taking process are data of the mini car whereas pieces of image data existing in the coordinate blocks A 5 to A 8 of the main imager 74 F as a result of the image-taking process are data of the books and the background.
- Pieces of image data existing in all the remaining coordinate blocks of the main imager 74 F as a result of the image-taking process are data of the books and the background.
- pieces of image data existing in the coordinate blocks A 2 to A 4 and A 9 of the auxiliary imager 75 F as a result of the image-taking process are data of the books and the background
- pieces of image data existing in the coordinate blocks A 5 to A 8 of the auxiliary imager 75 F as a result of the image-taking process are data of mini car.
- Pieces of image data existing in the coordinate blocks B 1 to B 5 and B 9 of the auxiliary imager 75 F as a result of the image-taking process are data of the books and the background
- pieces of image data existing in the coordinate blocks B 6 and to B 8 of the auxiliary imager 75 F as a result of the image-taking process are data of mini car.
- Pieces of image data existing in all the remaining coordinate blocks C 2 to F 9 of the main imager 74 F as a result of the image-taking process are data of the books and the background.
- the main imager 74 F supplies the pieces of image data to the parallax-detection comparison circuits 353 A to 353 F by way of the horizontal registers 351 A to 351 F respectively, which are provided for the main imager 74 F.
- the auxiliary imager 75 F supplies the pieces of image data to the parallax-detection comparison circuits 353 A to 353 F by way of the horizontal registers 352 A to 352 F respectively, which are provided for the auxiliary imager 75 F.
- the parallax-detection comparison circuits 353 A to 353 F detects shifts between the pieces of image data.
- parallax/distance coefficients are assigned to the normalized values in the parallax/distance coefficient assignment circuits 355 A to 355 F.
- the parallax/distance coefficient of IV is assigned to the image data of the mini car serving as a front scene whereas the parallax/distance coefficient of I is assigned to the image data of the two books and the background.
- the parallax/distance coefficient of IV is assigned to pieces of image data, which are then stored in sub-registers 1 to 4 of the register 356 A.
- the parallax/distance coefficient of I is assigned to pieces of image data, which are then stored in sub-registers 5 to 8 of the register 356 A.
- the parallax/distance coefficient of I is assigned to pieces of image data, which are then stored in sub-registers 1 and 5 to 8 of the register 356 B.
- the parallax/distance coefficient of IV is assigned to pieces of image data, which are then stored in sub-registers 2 to 4 of the register 356 B.
- the parallax/distance coefficient of I is assigned to all the remaining pieces of image data, which are then stored in the remaining sub-registers, i.e., sub-registers 1 to 8 of each of the registers 356 C to 356 F.
- the pieces of image data and the parallax/distance coefficients assigned to the pieces of image data are output to image-data recipients such as a display unit, a storage device and a network.
- FIG. 12 is diagrams each showing a model of a pseudo 3D (three-dimensional) effect of a reproduced image using assigned parallax/distance coefficients.
- the ordinary 3D (three-dimensional) effect such as a both-eye 3D (three-dimensional) view can also be obtained in a reproduced image.
- the present invention also provides image processing to generate a 3D (three-dimensional) pseudo effect in not only both of the eyes, but also in one eye.
- the pseudo effect can be obtained by using ordinary display means without requiring special display means such as a both-eye 3D (three-dimensional) viewer.
- the parallax/distance coefficients of I, II and IV are inversely proportional to distances of 4, 2 and 1 units respectively.
- image processing is carried out to alternately shift image data of each horizontal-coordinate block (or each horizontal-coordinate group) in the horizontal direction by a shift distance proportional to the positional distance from the lenses to the object of photographing generating the image data.
- FIG. 12A is a diagram showing data of a reproduced image.
- the data includes parallax/distance coefficients each assigned to a piece of block image data.
- the parallax/distance coefficient of I is assigned to pieces of block image data represented by symbols ⁇ 1 , ⁇ 2 , ⁇ 3 , ⁇ 4 , ⁇ 5 , ⁇ 1 , ⁇ 2 , ⁇ 2 , ⁇ 3 , ⁇ 3 , ⁇ 4 , ⁇ 4 , ⁇ 5 , ⁇ 5 , ⁇ 1 , ⁇ 2 , ⁇ 3 , ⁇ 4 , ⁇ 5 and ⁇ 1 to result in pieces of block image data represented by ⁇ 1 I, ⁇ 2 I, ⁇ 3 I, ⁇ 4 I, ⁇ 5 I, ⁇ 1 I, ⁇ 2 I, ⁇ 2 I, ⁇ 3 I, ⁇ 3 I, ⁇ 4 I, ⁇ 4 I, ⁇ 5 I, ⁇ 5 I, ⁇ 1 I, ⁇ 1 I
- FIG. 12B is a diagram showing a method of processing image data to demonstrate a pseudo 3D (three-dimensional) effect.
- the pieces of block image data each having the assigned parallax/distance of I representing a positional distance of four units as expressed by ⁇ 1 I, ⁇ 2 I, ⁇ 3 I, ⁇ 4 I, ⁇ 5 I and ⁇ 1 I on the first row of FIG. 12A are shifted to the left by one coordinate block.
- the pieces of block image data each having the assigned parallax/distance of I as expressed by ⁇ 3 I and ⁇ 3 I on the third row of FIG. 12A are shifted to the left by one coordinate block, the pieces of block image data each having the assigned parallax/distance of II as expressed by ⁇ 5 II and ⁇ 3 II on the third row are shifted to the left by 1 ⁇ 2 coordinate blocks whereas the pieces of block image data each having the assigned parallax/distance of IV representing a positional distance of 1 unit as expressed by XIV and YIV on the third row are shifted to the left by zero coordinate blocks.
- the pieces of block image data on fourth to the sixth rows of FIG. 12A are processed in the same way by shifting each of the pieces of block image data in the right or left direction depending on the row number by a shift distance depending on the assigned parallax/distance coefficient.
- the series of operations carried out on the rows are carried out typically for every frame of the moving picture in image processing to produce the 3D (three-dimensional) pseudo effect even in one eye.
- the result of the image processing is shown in FIG. 12B .
- image data having the I parallax/distance coefficient indicating a small parallax is shifted by a long shift distance but, on the other hand, image data having the IV parallax/distance coefficient indicating a large parallax is shifted by a small shift distance. It is to be noted, however, that image data having the I parallax/distance coefficient indicating a small parallax can be reversely shifted by a short shift distance and, on the other hand, image data having the IV parallax/distance coefficient indicating a large parallax can be shifted reversely by a long shift distance.
- FIG. 12C is a diagram showing an embodiment in which pieces of data are shifted in the right and left directions opposite to those of the embodiment shown in FIG. 12B .
- the embodiment shown in FIG. 12B is referred to as an embodiment with a normal phase
- the embodiment shown in FIG. 12C is known as an embodiment with an inverted phase. If the normal and inverted phases are introduced alternately every frame of the moving picture, the pseudo 3D (three-dimensional) effect is further enhanced.
- the luminance and contrast can be adjusted in accordance with the parallax/distance coefficients in order to further enhance the pseudo 3D (three-dimensional) effect.
- FIGS. 13A to 13 C are diagrams showing typical images reproduced as images each exhibiting a pseudo 3D (three-dimensional) effect resulting from a process to assign parallax/distance coefficients.
- FIGS. 13A, 13B , and 13 C show typical images demonstrating a pseudo 3D (three-dimensional) effect produced as a result of assigning parallax/distance coefficients in the image processing explained earlier by referring respective to FIGS. 9, 10 , and 11 as well as a result of adopting the principle described earlier reference to FIG. 12 .
- parallax/distance coefficients are assigned to pieces of block image data for every granularity even finer than the granularity adopted in the image processing explained earlier by referring to each of FIGS. 9, 10 and 11 as a granularity based on 8 ⁇ 6 coordinate blocks. That is to say, in the image processing explained earlier by referring to each of FIGS. 9, 10 and 11 , the parallax/distance coefficients of I, II and IV are assigned to pieces of block image data for every 8 ⁇ 6 coordinate blocks.
- the block shifts of I, II and IV correspond to 250, 500 and 1,000 pixels respectively.
- the parallax/distance coefficients of I, II and IV correspond to 250, 500 and 1,000 pixels respectively, which are arranged in the horizontal direction.
- a block used as the unit of the parallax/distance coefficients can be reduced to give a high resolution and the parallax/distance coefficients are assigned to pieces of block image data for every much finer granularity.
- FIGS. 13A to 13 C are models each consisting of an upper diagram showing a typical image reproduced without a pseudo 3D (three-dimensional) effect and a lower diagram each showing a typical image reproduced with a pseudo 3D (three-dimensional) effect.
- the rear scene of the image is moved in line units in accordance to parallax/distance coefficients assigned to the line in order to make the mini car serving as the front scene three-dimensionally visible.
- This image can be displayed on an ordinary display unit requiring no special display means.
- the pseudo 3D (three-dimensional) image can be displayed on an ordinary display unit to allow a pseudo 3D (three-dimensional) image to be observed by either both the eyes or any one of the eyes.
- the front scene is fixed and the rear scene serving as the background is shifted in the right or left direction in accordance with parallax/distance coefficients assigned to the rear scene in order to produce a pseudo 3D (three-dimensional) effect.
- the rear scene can also be fixed and the front scene is shifted in the right or left direction by a shift distance inversely proportional to the positional distance to the front scene in order to produce a pseudo 3D (three-dimensional) effect.
- FIG. 14 is diagrams showing models of reproduced images exhibiting an effect of assigned parallax/distance coefficients and other effects.
- image data can be processed to produce a zoom-up effect or a zoom-down effect, which is an effect opposite to the zoom-up effect.
- the zoom-up effect is an effect obtained as a result of applying a zoom-in process to a close image and a zoom-out process to a far image.
- the image processing can be carried out in a manner simpler than the chromakey technique.
- FIG. 14A is a diagram showing data produced with assigned parallax/distance coefficients as data of a reproduced image.
- FIG. 14B is a diagram showing a result of extracting image data with the parallax/distance coefficient of IV from the data shown in FIG. 14A and enlarging an image represented by the extracted image data. In this case, however, image data with the parallax/distance coefficients of I and II is not processed and left as it is. As shown in FIG. 14B , the image portion represented by the pieces of image data shown as the capitals X, Y, Z and W is enlarged but the remaining image portions shown as notations and symbols are left as they are at their original sizes. In this way, an image completing a zoom-in process is obtained.
- FIG. 14C is a diagram showing an image obtained as a result of extracting only image data with the parallax/distance coefficient of IV from the produced image data having assigned parallax/distance coefficients as shown in FIG. 14A and eliminating image data with the parallax/distance coefficients of I and II. That is to say, only, the image portion represented by the pieces of image data shown as the capitals X, Y, Z and W is displayed. In this way, by assigning parallax/distance coefficients to pieces of data representing the image of an object of photographing, the image of a portion cut out from the image of an object of photographing can be obtained.
- FIG. 15 is diagrams showing typical images exhibiting effects other than the pseudo 3D (three-dimensional) effect obtained as a result of a process to assign parallax/distance coefficients to pieces of data representing the images.
- FIGS. 15A, 15B , and 15 C show typical images demonstrating a pseudo 3D (three-dimensional) effect produced as a result of assigning parallax/distance coefficients in the image processing explained earlier by referring respective to FIGS. 9, 10 , and 11 .
- FIGS. 15A to 15 C each show a typical result of applying a zoom-in process to the image data of the mini car on the basis of the parallax/distance coefficient assigned to the image data and extracting the image data completing the zoom-in process.
- desired image processing can be carried out on the basis of the coefficients.
- the processes carried out after the image-taking processes include a process to extract a desired portion from an area of a displayed image and a process to synthesize desired image portions.
- the image processing device adopting the method and the image display apparatus employing the image processing device, main and auxiliary imagers provided by the present disclosure as well as the image processing device are integrated in the configuration of a single LSI circuit and, the LSI circuit carries out parallel processing to compare a signal generated by each horizontal register provided for the main imager with a signal generated by the corresponding horizontal register provided for the auxiliary imager so that the processing can be carried out at a speed higher than the speed of the ordinary parallax image-taking configuration and is suitable for an image process produce an image of high minuteness and a high rate.
- a pseudo 3D (three-dimensional) effect can be realized even by using only one eye so that it is possible to get rid of restrictions on the actual visual-field angle without using special display means such as a both-eye 3D (three-dimensional) viewer.
- an entertaining content such as a private movie or private video produced at home can be emphasized into a 3D (three-dimensional) display and, in addition, a 3D (three-dimensional) attendance feeling and a speed sense can be stressed in the case of a game image.
Abstract
Description
- The present application claims priority to Japanese Patent Application JP 2005-319728, filed in the Japanese Patent Office on Nov. 2, 2005, the entire contents of which is being incorporated herein by reference.
- The present disclosure relates to an image processing method using a solid-state image-taking device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) device, an image processing device adopting the image processing method and an image display apparatus employing the image processing device. More particularly, the present disclosure relates to an image processing method for processing an image based on a parallax between the right and left eyes and displaying the processed image as a pseudo three-dimensional image on an ordinary display, an image processing device adopting the image processing method and an image display apparatus employing the image processing device.
- In the case of a both-eye three-dimensional view generated by two cameras, special display means are required for reproducing the view. That is to say, display means obtained as a result of an optical effort to focus left image and right image respectively into the left eye and right eye of the observer is required. Due to the optical effort, however, a limitation on the visual-field angle is unavoidable. For this reason, it is difficult to allow a number of observers to look at a display shown by only one display apparatus.
- In addition, there is also a technology of extracting a parallax and utilizing the parallax. However, this technology also requires a display in which a right image and a left image are created on the right and left eyes respectively.
- For example, assuming an observer wears glasses, image data for the left eye of the observer and image data for the right eye of the same observer are alternately output to a display apparatus. At that time, the observer is capable of reproducing an image from the pieces of image data passing through the glasses capable of switching the shutter from the right glass to the left one and vice versa synchronously with the timing to switch the image data appearing on the display apparatus. By reproducing an image in this way, a three-dimensional image can be viewed.
- Japanese Patent Laid-open No. 2004-343290 discloses a method of reproducing a 3D (three-dimensional) image without using glasses and an apparatus adopting the method. In accordance with this method, a three-dimensional display apparatus based on a parallax is employed. A 3D (three-dimensional) image is generated by alternately arranging an image for the right eye and an image for the left eye for every dot in the form of rectangles and the images of the two points of view are displayed as a 3D (three-dimensional) image by adoption of a parallax variable method or a lenticular method.
- In the case of the parallax method, the display apparatus has a configuration including a slit and a display screen. The slit is placed in front of the display screen. The observer observes a 3D (three-dimensional) image appearing on the display screen through the slit. At that time, the left eye of the observer looks at only an image for the left eye while the right eye of the observer is looking at only an image for the right eye. In this way, the observer is capable of observing a 3D (three-dimensional) image with a feeling of seeing a 3D (three-dimensional) object.
- As described above, in order to display a 3D (three-dimensional) image, it is necessary to particularly provide the display apparatus with a special component such as the slit.
- On the other hand, Japanese Patent Laid-open No. 2004-40445 discloses an example of providing a portable information terminal with a 3D (three-dimensional) image displaying function. In the portable information terminal, a liquid-crystal module capable of displaying a 3D (three-dimensional) image includes a backlight, a switching liquid-crystal device, a phase difference board and a TFT liquid crystal. The switching liquid-crystal device is a component capable of switching its polarization angle. The phase difference board is a component patterned to form a slit shape. A 3D (three-dimensional) image is displayed by changing a phase relation with the patterned phase difference board. The phase relation with the patterned phase difference board is changed by switching the polarization angle of the switching liquid-crystal device from one value to another. In this way, a left image is generated by left pixels but prevented from being projected on the right eye by being cut off from the right eye. On the other hand, a right image is generated by right pixels but prevented from being projected on the left eye by being cut off from the left eye. In this state, only the image for the right eye appears on the right eye and only the image for the left eye appears on the left eye. Thus, an image appearing on the right eye of the observer is different from an image appearing on the left eye of the observer. As a result, the observer is capable of sensing a 3D (three-dimensional) image having a depth in the inward direction.
- In this case, however, the display apparatus requires special units such as the phase difference board and the switching liquid-crystal device capable of switching its polarization angle.
- For a both-
eye 3D (three-dimensional) view generated by using two cameras, it is necessary to employ a special display apparatus to be used in a process to reproduce the 3D (three-dimensional) image. The structure of the display apparatus is complicated and the visual-field angle is also limited too. Thus, it is difficult to allow a number of observers to look at a display shown by only one display apparatus. - Addressing the problem described above, the present disclosure provides an image display apparatus capable of carrying out image processing at a speed higher than the speed of the ordinary parallax image configuration by performing parallel processing such as a process to compare a signal output from every horizontal register employed in a main imager with a signal output from the corresponding horizontal register employed in an auxiliary imager integrated with the main imager in a single LSI (Large Scale Integrated) circuit in order to produce an image of high minuteness and a high rate.
- By using such an LSI circuit including main and auxiliary imagers as well as an image processing device, it is possible to realize a Camcoder (a registered trademark of Sony Corporation) and a digital still camera, which are capable of creating a pseudo 3D (three-dimensional) image by also recording information on parallaxes and distances along with image data.
- In accordance with an image processing method according to one embodiment, the method includes a process carried out by a pair of imagers to extract a parallax caused by a distance to an object of photographing as a positional shift between mutually corresponding pixels included in the imagers or between mutually corresponding image blocks included in the imagers as blocks conforming to the mutually corresponding pixels, the parallax is assigned to image data as a distance coefficient found on the basis of the positional shift between the mutually corresponding pixels or the mutually corresponding image blocks in the imagers.
- An image processing device according to another embodiment, the device includes a pair of imagers for extracting pieces of image data with different parallaxes caused by a distance to an object of photographing; and an image processing circuit for processing the pieces of image data with different parallaxes generated by the imagers. The imagers and the image processing circuit are constructed in the same integrated circuit chip.
- The image processing device is further characterized in that the image processing device generates pieces of image data with different parallaxes. In the device, a process is carried to assign a distance coefficient to the pieces of image data, the coefficient being obtained by extracting the parallaxes each according to a positional shift between a pixel or image block conforming to the pixel generating the piece of image data on one of the imagers and a corresponding pixel or image block conforming to the corresponding pixel generating the piece of image data on the other one of the imagers, and the image data is converted into output data in accordance with the assigned distance coefficients.
- An image display apparatus according to an embodiment, includes a pair of imagers for extracting pieces of image data with different parallaxes each caused by a distance to an object of photographing. The apparatus further includes a coefficient assignment circuit for carrying out a process on pieces of image data with different parallaxes output by the imagers to assign a distance coefficient to the pieces of image data, the coefficient being obtained by extracting the parallaxes each according to a positional shift between a pixel or image block conforming to the pixel generating the piece of image data on one of the imagers and a corresponding pixel or image block conforming to the corresponding pixel generating the piece of image data on the other one of the imagers. The apparatus still further includes an image processing circuit for creating a pseudo three-dimensional image from image data output by the coefficient assignment circuit, and an image display unit for displaying the pseudo three-dimensional image generated by the image processing circuit.
- In accordance with an embodiment of the image processing method, the image processing device adopting the image processing method and the image display apparatus employing the image display device, a signal output by a horizontal register employed in a main imager and a signal output by a corresponding horizontal register employed in an auxiliary imager constructed in the same (imager) LSI circuit as the main imager are compared with each other in parallel processing so that the image display apparatus is capable of carrying out image processing to produce an image of high minuteness and a high rate at a speed higher than the speed of the ordinary parallax image configuration.
- By using an LSI circuit including such imagers, it is possible to realize a Camcoder (a registered trademark of Sony Corporation) and a digital still camera, which are capable of recording information on parallaxes and distances along with image data.
- In addition, in accordance with the image processing method provided by the present invention, by using a parallax/distance coefficient based on a parallax, a 3D (three-dimensional) pseudo effect can be obtained even with only one eye. Thus, limitation on the visual-field angle is eliminated so that special means such as a both-
eye 3D (three-dimensional) viewer is no longer required. - Therefore, it is possible to provide a number of observers, who have different powers to view a display as well as a difference in visual ability between the two eyes, with a 3D (three-dimensional) effect of an image display in spite of the fact that the image display is a pseudo display.
- By adoption of such an image processing method, entertaining contents such as a private movie and a private video, which are produced at home, can be emphasized into a 3D (three-dimensional) display and, in addition, a 3D (three-dimensional) attendance feeling and a speed sense can be stressed in the case of a game image.
- Additional features and advantages are described herein, and will be apparent from, the following Detailed Description and the figures.
-
FIG. 1A is a diagram showing the configuration of a parallax-detection image-taking system; -
FIG. 1B is a diagram showing the structure of an LSI chip employed in the parallax-detection image-taking system; -
FIG. 2A is a diagram showing the configuration of another parallax-detection image-taking system; -
FIG. 2B is a diagram showing the structure of an LSI chip employed in the other parallax-detection image-taking system; -
FIG. 3A is a diagram showing the configuration of a parallax-detection image-taking system for producing a 3D (three-dimensional) display; -
FIGS. 3B and 3C are diagrams respectively showing a structure of main and auxiliary imagers employed in the parallax-detection image-taking system; -
FIG. 4 is a diagram showing a top view of the parallax-detection image-taking system shown inFIG. 3A ; -
FIG. 5A is a diagram showing the configuration of a typical parallax-detection image-taking system; -
FIGS. 5B and 5C are diagrams showing an image respectively created on main and auxiliary imagers employed in the typical parallax-detection image-taking system; -
FIG. 6A is a diagram showing the configuration of another typical parallax-detection image-taking system; -
FIGS. 6B and 6C are diagrams showing an image respectively created on main and auxiliary imagers employed in the other typical parallax-detection image-taking system; -
FIG. 7A is a diagram showing the configuration of a further typical parallax-detection image-taking system; -
FIGS. 7B and 7C are diagrams showing an image respectively created on main and auxiliary imagers employed in the further typical parallax-detection image-taking system; -
FIG. 8 is a diagram showing a typical concrete configuration of an LSI chip integrating main and auxiliary imagers; -
FIG. 9 is a diagram showing typical parallel processing to extract parallaxes from image data and assign parallax/distance coefficients found from the parallaxes to the image data; -
FIG. 10 is a diagram showing other typical parallel processing to extract parallaxes from image data and assign parallax/distance coefficients found from the parallaxes to the image data; -
FIG. 11 is a diagram showing further typical parallel processing to extract parallaxes from image data and assign parallax/distance coefficients found from the parallaxes to the image data; -
FIG. 12A is a diagram showing data including parallax/distance coefficients as data of a reproduced image; -
FIG. 12B is a diagram showing data led by adopting a method of processing the image data to demonstrate a pseudo 3D (three-dimensional) effect; -
FIG. 12C is a diagram showing another data led by adopting method of processing image data to demonstrate a pseudo 3D (three-dimensional) effect; -
FIG. 13A is a diagram showing images demonstrating a pseudo 3D (three-dimensional) effect produced by the typical parallel processing explained by referring toFIG. 9 ; -
FIG. 13B is a diagram showing images demonstrating a pseudo 3D (three-dimensional) effect produced by the other typical parallel processing explained by referring toFIG. 10 ; -
FIG. 13C is a diagram showing images demonstrating a pseudo 3D (three-dimensional) effect produced by the further typical parallel processing explained by referring toFIG. 11 ; -
FIG. 14A is a diagram showing data produced with assigned parallax/distance coefficients as data of a reproduced image; -
FIG. 14B is a diagram showing a result of extracting image data with the parallax/distance coefficient of IV from the data shown inFIG. 14A and enlarging an image represented by the extracted image data; -
FIG. 14C is a diagram showing an image obtained as a result of extracting only image data with the parallax/distance coefficient of IV from the data shown inFIG. 14A ; -
FIG. 15A is an explanatory diagram to be referred to in description of a function to cut out a portion from an image produced by the typical parallel processing explained by referring toFIG. 9 ; -
FIG. 15B is an explanatory diagram to be referred to in description of a function to cut out a portion from an image produced by the other typical parallel processing explained by referring toFIG. 10 ; and -
FIG. 15C is an explanatory diagram to be referred to in description of a function to cut out a portion from an image produced by the further typical parallel processing explained by referring toFIG. 11 . -
FIGS. 1A and 1B are a diagram showing a parallax-extraction image-takingsystem 10 for extracting a parallax by using a pair of imagers in accordance with an embodiment. The parallax-extraction image-takingsystem 10 shown inFIGS. 1A and 1B has a configuration including a plurality of imager LSI chips. A typical configuration shown in the figure includes two imager LSI chips 12 and 14 serving as main and auxiliary imagers respectively. An imager image processing device not shown in the figure is connected to the two imager LSI chips 12 and 14 and includes an image processing circuit for carrying out image processing on main image data and auxiliary image data, which are output by the two imager LSI chips 12 and 14 respectively. - A
lens 11 projects an optical signal representing an image of a photographingobject 15 located at a first distance from the position of thelens 11 on an image-taking face of the main-imager LSI chip 12. By the same token, alens 13 projects an optical signal representing an image of another photographingobject 16 located at a second distance from the position of thelens 13 on an image-taking face of the auxiliary-imager LSI chip 14. - Typically, the two imager LSI chips 12 and 14 are each a solid-state image-taking device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) device. The two imager LSI chips 12 and 14 each have a configuration including a number of pixels. The main-
imager LSI chip 12 converts a pixel optical signal leaving thelens 11 for the main-imager into an electrical signal. By the same token, the auxiliary-imager LSI chip 14 converts a pixel optical signal leaving thelens 13 for the auxiliary-imager into an electrical signal. -
FIG. 1B is a diagram showing a typical main-imager LSI chip 12 (or a typical auxiliary-imager LSI chip 14) implemented by a CMOS device. In the vertical direction, RGB pixels are laid out repeatedly to form apixel array 21 serving as a column of the main-imager LSI chip 12 (or the auxiliary-imager LSI chip 14). Thepixel arrays 21 are laid out repeatedly in the horizontal direction (or the row direction) alternately withvertical registers 22. A predetermined number ofpixel arrays 21 are laid out alternately with the same number ofvertical registers 22 in the row direction to form the so-called pixel matrix. An optical signal hitting a pixel is converted into an electrical signal, which is then transferred to avertical register 22 adjacent to thepixel array 21 including the pixel synchronously with a clock signal generated by a timing generator. - An AD (analog-to-digital) converter converts a signal output by a
vertical register 22 into a digital signal, which is supplied to ahorizontal register 23 before being output from the main-imager LSI chip 12 (or the auxiliary-imager LSI chip 14) with a predetermined timing. - The digital signals output from the main-
imager LSI chip 12 and the auxiliary-imager LSI chip 14 are supplied to a signal processing circuit not shown in the figure as main-image data and auxiliary-image data respectively. The signal processing circuit then processes the main-image data and the auxiliary-image data in order to generate a pseudo 3D (three-dimensional) image. An image processing method for processing the main-image data and the auxiliary-image data will be described later along with an image processing device adopting the image processing method. - As described above, a parallax-extraction image-taking system provided with the two imager LSI chips 12 and 14 forming a pair of main and auxiliary imagers as shown in
FIG. 1A needs thelenses - In accordance with an embodiment, a serial main-image signal output by the main-
imager LSI chip 12 and a serial auxiliary-image signal output by the auxiliary-imager LSI chip 14 are compared with each other in order to detect parallaxes in image processing to be described later. - The figure shows the main-
imager LSI chip 12 on the left side and the auxiliary-imager LSI chip 14 on the right side, and the following description assumes these positions of the main-imager LSI chip 12 and the auxiliary-imager LSI chip 14. It is worth noting, however, that the positions of the main-imager LSI chip 12 and the auxiliary-imager LSI chip 14 can be reversed. -
FIGS. 2A and 2B are a diagram roughly showing a parallax-extraction image-taking system 50 employing a single imager LSI in accordance with another embodiment of the present invention.Reference numeral 54 denotes the imager LSI, including amain imager 54A and anauxiliary imager 54B. As described earlier, themain imager 54A and theauxiliary imager 54B are each typically a solid-state image-taking device such as a CCD or a CMOS device. In this typical case of the embodiment, themain imager 54A and theauxiliary imager 54B are constructed in one LSI chip implemented by a CMOS device. - The parallax-extraction image-taking system 50 shown in
FIG. 2A includes a first photographingobject 58 at a position relatively close tolenses object 59 at a position relatively far from thelenses object 58 and the second photographingobject 59 are subjected to an image-taking process after passing through thelenses object 58 and the second photographingobject 59 are reflected byreflection plates lens 51, and enter themain imager 54A included in theintegrated LSI chip 54. By the same token, the light beams representing the images of the first photographingobject 58 and the second photographingobject 59 are reflected byreflection plates lens 55, and enter theauxiliary imager 54B included in theintegrated LSI chip 54. - That is to say, since a pair of main and
auxiliary imagers single LSI chip 54, by employing thereflection plates object 58 and the second photographingobject 59 can be projected on a light receiving area in theLSI chip 54 even though the main andauxiliary imagers -
FIG. 2B is a block diagram showing theintegrated LSI chip 54 including themain imager 54A, theauxiliary imager 54B and an image processing device in accordance with this embodiment. As described above, themain imager 54A, theauxiliary imager 54B and the image processing device are integrated in asingle LSI chip 54 implemented by a CMOS device. The block diagram is simplified in order to make the explanation or theLSI chip 54 simple. - The
main imager 54A haspixel rows pixel rows LSI chip 54 also includeshorizontal registers auxiliary imager 54B haspixel rows 61A′, 61B′ and 61C′, which are each oriented in the horizontal direction. For thepixel rows 61A′, 61B′ and 61C′, theLSI chip 54 also includeshorizontal registers 62A′, 62B′ and 62C′ respectively. - The outputs of the
horizontal register 62A for themain imager 54A and thehorizontal register 62A′ for theauxiliary imager 54B are connected to aparallel processing circuit 63A. By the same token, the outputs of the horizontal register 62B for themain imager 54A and the horizontal register 62B′ for theauxiliary imager 54B are connected to a parallel processing circuit 63B. In the same way, the outputs of thehorizontal register 62C for themain imager 54A and thehorizontal register 62C′ for theauxiliary imager 54B are connected to aparallel processing circuit 63C. - The outputs of the
parallel processing circuits vertical register 64, for sequentially receiving processing results from theparallel processing circuits - As described above, the
LSI chip 54 has themain imager 54A and theauxiliary imager 54B each serving as a sensor. On each pixel row oriented in the horizontal (or x-axis) direction in the area of the sensors, typically, a predetermined number of pixels are laid out and, in the vertical direction, such pixel rows are laid out alternately with the horizontal registers. - On each of the pixel rows, typically, R, G and B pixels are laid out repeatedly and connected to a horizontal register associated with the pixel row. Each of the pixels employs a row select transistor, a reset transistor, an amplification transistor and a photodiode.
- In order to read out an output signal, which is generated by a pixel when the pixel is driven, every row is also provided with a noise canceller, a signal read transistor, the horizontal register and an AD (Analog-to-Digital) converter, which are not shown in the figure. The output of the AD converter is connected to the horizontal register provided for the row.
- At places surrounding the R, G and B pixels, a horizontal scanning circuit for selecting a column and a vertical scanning circuit for selecting a row are provided as a horizontal driving system and a vertical driving system respectively, but the horizontal scanning circuit for selecting a column and the vertical scanning circuit are not shown in the figure. Typically, each of the horizontal scanning circuit and the vertical scanning circuit includes a horizontal register, which starts a shift operation to scan columns or rows synchronously with a driving pulse generated by a TG (timing generator) also not shown in the figure.
- The horizontal scanning circuit sequentially generates horizontal scan (select) pulses to be supplied to column read lines and column signal lines. A pixel signal read out from a specific pixel selected by a column read line and a row select line is output to the column signal line.
- On the other hand, the vertical scanning circuit sequentially generates vertical scan pulses to be supplied to row select lines to sequentially select pixel rows, which are laid out in the vertical direction.
- An image signal read out from a column signal line is supplied typically to a CDS (Correlation Double Sampling) circuit serving as a difference computation circuit for carrying out a process to find a difference generated right after a pixel reset operation on the basis of sampling pulses generated by the timing generator as a difference between a noise level and a signal level. At the stage following the CDS circuit, components such as an AGC (Automatic Gain Control) circuit and an ADC (Analog Digital Converter) circuit are provided.
- Digital signals output by the ADC circuits as signals representing image data are supplied to the
horizontal registers 62A to 62C and 62A′ to 62C′. The pieces of image data supplied to thehorizontal registers 62A to 62C are sequentially passed on to theparallel processing circuits 63A to 63C respectively and the image data supplied to thehorizontal registers 62A′ to 62C′ are also sequentially passed on to theparallel processing circuits 63A to 63C respectively. Theparallel processing circuits 63A to 63C carry out a process to detect pixel shifts caused by parallaxes and other processing. - The parallax-extraction image-taking system 50 based on a monolithic imager as shown in
FIGS. 2A and 2B is configured to include themain imager 54A and theauxiliary imager 54B on the same substrate of the LSI chip so as to function as an image-taking system for extracting a parallax and a distance. In this case, thesingle LSI chip 54 serving as imagers can be constructed into a configuration in which a signal generated by a horizontal register employed in themain imager 54A and a signal generated by the corresponding horizontal register employed in theauxiliary imager 54B are compared with each other for every pair of a horizontal register and a corresponding horizontal register in parallel processing carried out at the same time for all the pairs. - The method adopted in this configuration as a technique, in which a parallax in the horizontal direction is extracted and, then, data is output serially through a vertical register, allows the image processing to be carried out at a high speed and results in an image of high minuteness and a high rate in comparison with the configuration of the parallax-extraction image-taking
system 10 shown inFIGS. 1A and 1B . - In addition to merits described later as merits in the image processing, the LSI configuration including the main imager, the auxiliary imager and the parallel comparison/processing circuit, which are integrated in one chip, also has system-configuration merits and manufacturing merits.
- It is to be noted that the distance between lenses for extracting a parallax should be normally set at a value greater than the distance between the main and auxiliary imagers integrated in one LSI chip. For this reason, an optical system having the inter-lens distance matching the inter-imager distance is obviously desired.
-
FIGS. 3A to 3C are a diagram showing the configuration of a parallax-extraction image-takingsystem 100 for displaying a 3D (three-dimensional) image.FIG. 4 is a diagram showing a top view of the parallax-extraction image-takingsystem 100 as a coordinate recognition model of a parallax extraction process and an image-taking process. The following description explains a process to compare image data generated by amain imager 74 with image data generated by anauxiliary imager 75 in the horizontal direction, a process to extract a parallax of image data and a process to assign a parallax/distance coefficient to the image data. - The following description explains a configuration in which a signal generated by a horizontal register employed in the
main imager 74 and a signal generated by the corresponding horizontal register employed in theauxiliary imager 75 are compared with each other for every pair of a horizontal register and a corresponding horizontal register for all pairs. As for the image-taking system itself, however, the configuration does not have to be that shown inFIGS. 1A and 1B orFIGS. 2A and 2B . - In addition, setting of dimensions is by no means limited to setting at values shown in
FIGS. 3A to 3C andFIG. 4 . For the sake of explanation convenience, however, that shown inFIG. 3A is taken as the setting of dimensions. As shown inFIG. 3A , the distance between alens 51 and alens 55 is 4d; whereas the distance between the lens pair consisting of thelens 51 and thelens 55 and the imager pair consisting of themain imager 74 and theauxiliary imager 75 is 5d. The distance between thelens 51 and a first photographingobject 71 is 10d; the distance between thelens 51 and a second photographingobject 72 is 20d; and the distance between thelens 51 and a third photographingobject 73 is 40d. - The first photographing
object 71 is a rectangular plate, which has a side of 2d and is marked with the capitals X, Y, Z and W. The second photographingobject 72 is also a rectangular plate, which has a side of 8d and is marked notations ι1 to μ1. The third photographingobject 73 is also a rectangular plate, which has a side of 24d and is marked with symbols α1 to θ1. In this case, the character “d” denotes any arbitrary distance unit and is not limited to a specific value. - An image of an object of photographing is created by the
lens object 73 and themain imager 74 respectively, which each have dimensions of 4d×3d. The image created by thelens auxiliary imager 75 or themain imager 74 respectively has matrix elements arranged in an order opposite to the order of the arrangement of the same elements in the object of photographing. Thus, an image created by thelens 51 on themain imager 74 as overlapping images of the first photographingobject 71, the second photographingobject 72 and the third photographingobject 73 is shown inFIG. 3B . On the other hand, an image created by thelens 55 on theauxiliary imager 75 as overlapping images of the first photographingobject 71, the second photographingobject 72 and the third photographingobject 73 is shown inFIG. 3C . However, the matrix elements themselves, which are the capitals X, Y, Z and W, the notations ι1 to μ1 and the symbols α1 to θ1, are deliberately rotated by 180 degrees to prevent the matrix elements from being shown as upside-down elements in order to make the capitals, the notations and the symbols easy to recognize. - The created overlapping images are shown in
FIGS. 3B and 3C , being placed on a column-row coordinate system. The created images are each an 8-column×6-row matrix with each column having a width of d/2 and each row having a height of d/2. In the column-row coordinate system, the columns arecolumns 1 to 9 whereas the rows are rows A to F. The horizontal-direction shift of the image created on theauxiliary imager 75 as the image of the first photographingobject 71 which locates near thelens 55 is largest among the horizontal-direction shifts of the images created on theauxiliary imager 75. On the other hand, the horizontal-direction shift of the image created on theauxiliary imager 75 as the image of the third photographingobject 73 which locates far from thelens 55 is smallest among the horizontal-direction shifts of the images created on theauxiliary imager 75. - In particular, on the
main imager 74 shown in the coordinate recognition diagram ofFIG. 3B , the image of the third photographingobject 73 is created on coordinate blocks starting with the coordinate blocks A2 to A7 of the first row and ending with the coordinate blocks F2 to F7 of the last row. For example, on the coordinate blocks A2 to A7 of the first row, the image of the third photographingobject 73 is created as symbols θ1 to η1 respectively, on the subsequent rows, the image of the third photographingobject 73 is created in the same way as the first row and, on the coordinate blocks F2 to F7 of the last row, the image of the third photographingobject 73 is created as symbols β1 to α1 respectively. By the same token, the image of the second photographingobject 72 is created as an overlapping image on coordinate blocks starting with the coordinate blocks B3 to B6 of the first row and ending with the coordinate blocks E3 to E6 of the last row. For example, on the coordinate blocks B3 to B6, the image of the second photographingobject 72 is created as notations μ1 to λ3 respectively, on the subsequent rows, the image of the second photographingobject 72 is created in the same way as the first row and, on the coordinate blocks E3 to E6, the image of the second photographingobject 72 is created as notations ι4 to ι1 respectively. Likewise, the image of the first photographingobject 71 is created as an overlapping image on coordinate blocks starting with the coordinate blocks C4 and C5 of the first row and ending with the coordinate blocks D4 and D5 of the last row. For example, on the coordinate blocks C4 and C5, the image of the first photographingobject 71 is created as the capitals W and Z respectively and, on the coordinate blocks D4 and D5, the image of the first photographingobject 71 is created as the capitals Y and X respectively. The image created on themain imager 74 shown inFIG. 3B as the image of the third photographingobject 73 is obtained by rotating the symbol matrix of the uppermost plate shown inFIG. 3A by 180 degrees due to the effect of thelens 51. By the same token, the image created on themain imager 74 shown inFIG. 3B as the image of the second photographingobject 72 is obtained by rotating the notation matrix of the middle plate shown inFIG. 3A by 180 degrees due to the effect of thelens 51. In the same way, the image created on themain imager 74 shown inFIG. 3B as the image of the first photographingobject 71 is obtained by rotating the capital matrix of the lowermost plate shown inFIG. 3A by 180 degrees due to the effect of thelens 51. The images shown inFIG. 3B are not shifted from their references due to distances to the objects of photographing. - The overlapping images created on the
auxiliary imager 75 are shown inFIG. 3C , which is a diagram showing the same overlapping images as those shown inFIG. 3B as except that the horizontal coordinates blocks of the coordinate system shown inFIG. 3C each have an offset of +1 with respect to the coordinate system shown inFIG. 3B . - In particular, on the
auxiliary imager 75 shown in the coordinate recognition diagram ofFIG. 3C , the image of the third photographingobject 73 is created on coordinate blocks starting with the coordinate blocks A3 to A8 of the first row and ending with the coordinate blocks F3 to F8 of the last row. For example, on the coordinate blocks A3 to A8 of the first row, the image of the third photographingobject 73 is created as symbols θ1 to η1 respectively, on the subsequent rows, the image of the third photographingobject 73 is created in the same way as the first row and, on the coordinate blocks F3 to F8 of the last row, the image of the third photographingobject 73 is created as symbols β1 to α1 respectively. By the same token, the image of the second photographingobject 72 is created as an overlapping image on coordinate blocks starting with the coordinate blocks B5 to B8 of the first row and ending with the coordinate blocks E5 to E8 of the last row. For example, on the coordinate blocks B5 to B8, the image of the second photographingobject 72 is created as notations μ1 to λ3 respectively, on the subsequent rows, the image of the second photographingobject 72 is created in the same way as the first row and, on the coordinate blocks E5 to E8, the image of the second photographingobject 72 is created as notations ι4 to ι1 respectively. Likewise, the image of the first photographingobject 71 is created as an overlapping image on coordinate blocks starting with the coordinate blocks C8 and C9 of the first row and ending with the coordinate blocks D8 and D9 of the last row. For example, on the coordinate blocks C8 and C9, the image of the first photographingobject 71 is created as the capitals W and Z respectively and, on the coordinate blocks D8 and D9, the image of the first photographingobject 71 is created as the capitals Y and X respectively. The image created on theauxiliary imager 75 shown inFIG. 3C as the image of the third photographingobject 73 is obtained by rotating the symbol matrix of the uppermost plate shown inFIG. 3A by 180 degrees due to the effect of thelens 55 in the same way as the image created on themain imager 74 shown inFIG. 3B as the image of the third photographingobject 73 is obtained by rotating the symbol matrix of the uppermost plate shown inFIG. 3A by 180 degrees due to the effect of thelens 51. By the same token, the image created on theauxiliary imager 75 shown inFIG. 3C as the image of the second photographingobject 72 is obtained by rotating the notation matrix of the middle plate shown inFIG. 3A by 180 degrees due to the effect of thelens 55. In the same way, the image created on theauxiliary imager 75 shown inFIG. 3C as the image of the first photographingobject 71 is obtained by rotating the capital matrix of the lowermost plate shown inFIG. 3A by 180 degrees due to the effect of thelens 55. In addition, the images shown inFIG. 3C as the images created on theauxiliary imager 75 are shifted from their positions in the coordinate system shown inFIG. 3B because of parallaxes also due to the fact that the horizontal coordinates blocks of the coordinate system shown inFIG. 3C each have an offset of +1 with respect to the coordinate system shown inFIG. 3B . - It is to be noted that, by taking a very small coordinate block unit, the precision of the recognition of a parallax can be improved. Thus, it is desirable to take the pixel as a coordinate block unit.
-
FIG. 4 is a diagram showing atop view 150 of the parallax-extraction image-takingsystem 100. The same dimensions as those taken inFIG. 3 are used inFIG. 4 . - In the
top view 150, image creation lines for creating the image of an object of photographing on themain imager 74 through thelens 51 are each drawn as a solid line whereas image creation lines for creating the image of an object of photographing on theauxiliary imager 75 through thelens 55 are each drawn as a broken line. - The center line of the
main imager 74 is set as a line passing through the centers of the first photographingobject 71, the second photographingobject 72 and the third photographingobject 73. The center of theauxiliary imager 75 is separated away in the horizontal direction from the center of themain imager 74 by a distance of 4d. Thus, the centers of the first photographingobject 71, the second photographingobject 72 and the third photographingobject 73 are shifted from the center line of theauxiliary imager 75 and thelens 55 by the distance of 4d. The position of an image created on theauxiliary imager 75 as the image of an object of photographing is shifted from the position of an image created on themain imager 74 as the image of the same object of photographing. The closer the distance from thelens 55 to the object of photographing, the larger the positional shift of the image of the object of photographing. - That is to say, as is also obvious from the
top view 150, on theauxiliary imager 75 shown inFIG. 3C , the position of the image of the first photographingobject 71 is shifted to the right from the image position on themain imager 74 shown inFIG. 3B . In addition, the first photographingobject 71 is shifted more to the right from the center line of theauxiliary imager 75 and thelens 55 in comparison with the second photographingobject 72 and the third photographingobject 73. -
FIG. 5 is a diagram showing a first typical implementation of a parallax-extraction image-taking coordinate-recognition system 200 according to an embodiment. - While
FIG. 3 is referred to in the description of the principle of photographing,FIG. 5A is a diagram of the typical implementation of the principle. InFIG. 5A , the block unit d is set at 12 mm.Reference numeral 71 denotes a mini car serving as a front scene separated away from thelens 51 by a distance of 240 mm.Reference numeral 73 denotes two books A and B, which serve as a rear scene (or a background) separated away from thelens 51 by a distance of 480 mm. The two books A and B are each shown in the figure as a hatched block. - In the block coordinate system consisting of
columns 1 to 8 and rows A to F, the image of themini car 71 is shown as a figure enclosed by a solid line whereas the image of the A andB books 73 serving as the background is shown as hatched blocks. - On the
main imager 74 shown inFIG. 5B , the image of themini car 71 occupies an area of coordinate blocks A3 to A7 and B5 to B6. On theauxiliary imager 75 shown inFIG. 5C , on the other hand, the image of themini car 71 occupies an area of coordinate blocks A5 to A9 and B7 to B8. Thus, the image created on theauxiliary imager 75 as the image of themini car 71 is shifted to the right side in the block coordinate system rotated by 180 degrees from the image created on themain imager 74 shown inFIG. 5B as the image of themini car 71 by two blocks including the +1 pre-offset. In this way, the shift of the image created on theauxiliary imager 75 as the image of themini car 71 from the image created on themain imager 74 as the image of themini car 71 in the horizontal direction on the block coordinate system is recognized. On the other hand, the image created on theauxiliary imager 75 as the image of the A andB books 73 serving as the background is slightly shifted in the right direction from the image created on themain imager 74 as the image of the A andB books 73 by a distance of about one block including even the +1 pre-offset. Thus, the shift of the image of the A andB books 73 is smaller than the shift of the image of themini car 71. -
FIG. 6 is a diagram showing a second typical implementation of a parallax-extraction image-taking coordinate-recognition system 250 according to the embodiment of the present invention. In the second implementation shown inFIG. 6A , themini car 71 has slightly moved in the horizontal direction. - On the main imager 74A shown in
FIG. 6B , the image of themini car 71 occupies an area of coordinate blocks A2 to A6 and B4 to B5. On theauxiliary imager 75 shown inFIG. 6C , on the other hand, the image of themini car 71 occupies an area of coordinate blocks A4 to A8 and B6 to B7. Thus, the image created on theauxiliary imager 75 as the image of themini car 71 is shifted to the right side in the block coordinate system from the image created on themain imager 74 shown inFIG. 6B as the image of themini car 71 by two blocks. On the other hand, the image created on theauxiliary imager 75 as the image of the A andB books 73 serving as the background is slightly shifted in the right direction from the image created on themain imager 74 as the image of the A andB books 73 by a distance of about one block. -
FIG. 7 is a diagram showing a third typical implementation of a parallax-extraction image-taking coordinate-recognition system 300 according to the embodiment. In the third implementation shown inFIG. 7A , themini car 71 has further moved along an abruptly curved road approaching thelens 51 and reached a position separated from thelens 51 by a distance of 120 mm. - On the main imager 74B shown in
FIG. 7B , the image of themini car 71 occupies an area of coordinate blocks A1 to A4 and B2 to B4. On theauxiliary imager 75B shown inFIG. 7C , on the other hand, the image of themini car 71 occupies an area of coordinate blocks A5 to A8 and B6 to B8. Thus, the images created on theauxiliary imager 75B as the images of themini car 71 and the A andB books 73 are shifted to the right side from the coordinate blocks A1 to A4 and B2 to B4 occupied by the images created on themain imager 74 shown inFIG. 7B as the images of themini car 71 and the A andB books 73 by four blocks including the +1 pre-offset. IfFIGS. 7B and 7C are compared withFIGS. 5B and 5C respectively ofFIGS. 6B and 6C respectively, it will become obvious that, as themini car 71 serving as the front scene approaches thelenses mini car 71 and the image created on themain imager 74B or 75B as the image of themini car 71 increases. -
FIG. 8 is a diagram showing a typical concrete configuration of an LSI chip 350 including a main imager 74C, an auxiliary imager 75C and an image processing circuit, which are shown inFIG. 2 . The LSI chip 350 is denoted byreference numeral 54 inFIG. 2 . The image processing circuit includeshorizontal registers 351A to 351F provided for the main imager 74C,horizontal registers 352A to 352F provided for the auxiliary imager 75C, parallaxdetection comparison circuits 353A to 353F, parallax normalizationcoefficient computation circuits 354A to 354F, parallax/distancecoefficient assignment circuits 355A to 355F, registers 356A to 356F, avertical register 357 andregisters 361 to 366. - Pieces of image data represented by symbols θ1, η5, η4, η3, η2 and η1 at coordinate blocks A2 to A7 of the main imager 74C are transferred to the
horizontal register 351A. By the same token, pieces of image data represented by symbols ζ5, μ1, λ5, λ4, λ3 and ε5 at coordinate blocks B2 to B7 of the main imager 74C are transferred to thehorizontal register 351B. In the same way, pieces of image data on the subsequent rows of coordinate blocks of the main imager 74C are transferred to the subsequent horizontal registers. Likewise, pieces of image data represented by symbols β1, α5, α4, α3, α2 and α1 at coordinate blocks F2 to F7 of the main imager 74C are transferred to thehorizontal register 351F. - Similarly, pieces of image data are transferred from rows of blocks on the auxiliary imager 75C to the
horizontal registers 352A to 352F corresponding to the block rows. - The parallax
detection comparison circuits 353A to 353F, the parallax normalizationcoefficient computation circuits 354A to 354F and the parallax/distancecoefficient assignment circuits 355A to 355F compose a system for carrying out parallel processing such as parallax extraction and parallax/distance coefficient assignment processes. - The parallax
detection comparison circuits 353A to 353F receive the pieces of image data from thehorizontal registers 351A to 351F of the main imager 74C and the pieces of image data from thehorizontal registers 352A to 352F of the auxiliary imager 75C, comparing the pieces of data received from the main imager 74C and the pieces of data received from the auxiliary imager 75C with each other in order to detect a parallax for each coordinate block. - For example, the image data represented by symbol ε4 at coordinate block C2 of the main imager 74C is located at coordinate block C3 of the auxiliary imager 75C, causing a parallax to be detected as a block shift of I. By the same token, the image data represented by notation λ2 at coordinate block C3 of the main imager 74C is located at coordinate block C5 of the auxiliary imager 75C, causing a parallax to be detected as a block shift of II. In the same way, the image data represented by the capital W at coordinate block C4 of the main imager 74C is located at coordinate block C8 of the auxiliary imager 75C, causing a parallax to be detected as a block shift of IV. Likewise, the image data represented by the capital Z at coordinate block C5 of the main imager 74C is located at coordinate block C9 of the auxiliary imager 75C, causing a parallax to be detected as a block shift of IV. Similarly, the image data represented by notation κ4 at coordinate block C6 of the main imager 74C is located at coordinate block C10 (not shown in the figure) of the auxiliary imager 75C, causing a parallax to be detected as a block shift of II. By the same token, the image data represented by symbol δ4 at coordinate block C7 of the main imager 74C is located at coordinate block C11 (not shown in the figure) of the auxiliary imager 75C, causing a parallax to be detected as a block shift of I.
- The above description holds true for the other
horizontal registers horizontal registers horizontal registers horizontal registers 351E and 352E corresponding to coordinate blocks E1 to E8 and the otherhorizontal registers - The parallax normalization
coefficient computation circuits 354A to 354F each carry out a normalization process to find a normalized parallax/distance coefficient for a parallax detected by the parallaxdetection comparison circuits 353A to 353F respectively as block shifts. To put it concretely, for example, the distance from the main imager 74C or the auxiliary imager 75C to the rear scene is normalized to 1. In this case, a parallax/distance coefficient found from a parallax detected by the parallax detection comparison circuit 353 for a photographing object serving as a front scene is equal to a fraction, which would be obtained as a result of dividing the actual distance from the main imager 74C or the auxiliary imager 75C to the object of photographing by the distance from the main imager 74C or the auxiliary imager 75C to the rear scene. As an alternative, the position of a rear scene is reversely taken as a reference of 0 and the distance from the rear scene to thelens lens - A detected parallax is thus represented by a normalized parallax/distance coefficient, which is equal to a fraction, which would be obtained as a result of dividing the actual distance from a reference to an object of photographing by a distance normalized to 1 as described above. Then, the normalized parallax/distance coefficient found from a parallax detected by the parallax detection comparison circuit 353 is normalized again into another value as described below and assigned to a coordinate block corresponding to the object of photographing as a final parallax/distance coefficient.
- The parallax/distance
coefficient assignment circuits 355A to 355F each further carry out additional processing on the normalized parallax/distance coefficient coefficients output by the parallax normalizationcoefficient computation circuits 354A to 354F respectively. For example, in the case of a normalized parallax/distance coefficient normally not greater than 1, the normalized parallax/distance coefficient is multiplied by a value to result in a product, which is then typically rounded to produce an integer to be assigned as a final parallax/distance coefficient to the corresponding block. By using the final parallax/distance coefficient, digital processes of the subsequent processes become easy to carry out. It is needless to say that, in place of an integer, another value or a code can also be used as the parallax/distance coefficient. - As another alternative, the aforementioned block shifts of I, II and IV each detected by the parallax
detection comparison circuits 353A to 353F as a value inversely proportional to the distance to an object of photographing can each be assigned to a corresponding block of the main imager 74C as a parallax/distance coefficient as it is. Alternatively, the parallax/distance coefficient can be normalized and assigned to the corresponding block as a coefficient proportional to the distance. - In the embodiment shown in the figure, the block shifts of I, II and IV are assigned to their respective corresponding blocks as they are and the
vertical register 357 outputs a serial image signal including attached parallax/distance coefficients. - For the sake of convenience, parallax/distance coefficients are shown under each of the
registers 356A to 356F. The parallax/distance coefficients each represent a relation between image data of the main imager 74C and the corresponding image data of the auxiliary imager 75C. In actuality, the parallax/distance coefficients each representing a block shift are stored in memories associated with theregisters 356A to 356F. However, the storage method and the storage means are not limited to what is described here. - The parallax/distance
coefficient assignment circuits 355A to 355F supply pieces of image data and parallax/distance coefficients assigned to them to theregisters 356A to 356F respectively. - For example, the image data θ1 is stored in
sub-register 2 of theregister 356A and the image data η5 is stored insub-register 3 of theregister 356A. By the same token, the subsequent pieces of image data are stored in their respective subsequent sub-registers of theregister 356A and the last image data η1 is stored insub-register 7 of theregister 356A. In addition, the parallax/distance coefficients of I, I, - - - and I each representing a block shift are stored in a memory by being associated with their respective pieces of image data stored insub-registers 2 to 7 of theregister 356A. - By the same token, pieces of image data are stored in the register 346B and the parallax/distance coefficients assigned to the pieces of image data are stored in a memory associated with the register 356B. Then, the image data ε4 is stored in
sub-register 2 of the register 356C, the image data λ2 is stored insub-register 3 of the register 356C, the image data W is stored insub-register 4 of the register 356C, the image data Z is stored insub-register 5 of the register 356C, the image data κ4 is stored insub-register 6 of the register 356C and theimage data 64 is stored insub-register 7 of the register 356C. In addition, the parallax/distance coefficients of I, II, IV, IV, II and I each representing a block shift are stored in a memory by being associated with their respective pieces of image data stored insub-registers 2 to 7 of the register 356C. The process to store pieces of image data and their parallax/distance coefficients is repeated till pieces of image data and their parallax/distance coefficients are stored respectively in theregister 356F and a memory associated with theregister 356F. - The embodiment shown in
FIG. 8 includes only few pixels. For example, the auxiliary imager 75C does not include pixels corresponding to coordinate blocks C10 and C11 associated with respectively sub-registers 6 and 7 of the register 356C and does not include pixels corresponding to coordinate blocks D10 and D11 associated with respectively sub-registers 6 and 7 of the register 356D. For this reason, sub-registers 6 and 7 of the register 356C are complementarily associated with the parallax/distance coefficients of II and I respectively and, by the same token, sub-registers 6 and 7 of the register 356D are also complementarily associated with the parallax/distance coefficients of II and I respectively. - The pieces of image data stored in the
registers 356A to 356F are supplied to thevertical register 357 and then serially transferred to theregisters 361 to 366 before being output to image-data recipients such as a display unit, a storage device and a network. -
FIGS. 9, 10 and 11 are diagrams respectively showing first, second and third embodiments each implementing parallel processing of the parallax extraction and parallax/distance coefficient assignment processes described above. In particular, the first embodiment shown inFIG. 9 is an embodiment of a process to assign parallax/distance coefficients to image data obtained as a result of the image-taking operation shown inFIG. 5 by adoption of the principle explained earlier by referring toFIG. 8 . In this embodiment, the parallax/distance coefficient of II is assigned to the mini car serving as the front scene whereas the parallax/distance coefficient of I is assigned to the two books and a screen, which serves as a background. - In coordinates blocks A1 to A8 and B1 to B8 of the main imager 74D, pieces of image data of the two books, the background and the mini car exist as a result of an image-taking process. To be more specific, the pieces of image data existing in the coordinate blocks A1 and A2 are data of the books and the background, the pieces of image data existing in the coordinate blocks A3 to A7 are data of the mini car whereas the piece of image data existing in the coordinate block A8 is data of a portion of the books and the background. On the other hand, the pieces of image data existing in the coordinate blocks B1 to B4 are data of the books and the background, the pieces of image data existing in the coordinate blocks B5 and B6 are data of the mini car whereas the pieces of image data existing in the coordinate blocks B7 and B8 are data of the books and the background. In all the remaining coordinates blocks C1 to C8, D1 to D8, E1 to E8 and F1 to F8 of the main imager 74D and all the horizontal registers associated with the remaining coordinates blocks, pieces of image data of the two books and the background exist as a result of the image-taking process.
- On the other hand, since the books hardly move, the image data resulting from the image-taking process on the main imager 74D as the image data of the books is shifted on the auxiliary imager 75D by a distance not exceeding one block provided that the +1 pre-offset is not taken into consideration.
- Since the mini car moves, however, the image data resulting from the image-taking process on the main imager 74D as the image data the mini car is shifted on the auxiliary imager 75D by a distance of one block provided that the +1 pre-offset is not taken into consideration or a distance not exceeding two blocks provided that the +1 pre-offset is taken into consideration.
- To be more specific, the pieces of image data existing in the coordinate blocks A2 to A4 are data of the books and the background whereas the pieces of image data existing in the coordinate blocks A5 to A9 are data of the mini car. On the other hand, the pieces of image data existing in the coordinate blocks B2 to B6 are data of the books and the background, the pieces of image data existing in the coordinate blocks B7 and B8 are data of the mini car whereas the piece of image data existing in the coordinate block B9 is data of the books and the background. In all the remaining coordinates blocks C2 to C9, D2 to D9, E2 to E9 and F2 to F9 of the auxiliary imager 75D and all the horizontal registers associated with the remaining coordinates blocks, pieces of image data of the two books and the background exist as a result of the image-taking process.
- The image data stored in the
horizontal register 351A of the main imager 74D and the image data stored in thehorizontal register 352A of the auxiliary imager 75D are supplied to the parallax-detection comparison circuit 353A. By the same token, the pieces of image data stored in thehorizontal registers 351B to 351F of the main imager 74D and the pieces of image data stored in thehorizontal registers 352B to 352F of the auxiliary imager 75D are supplied to the parallax-detection comparison circuits 353B to 353F respectively. The parallax-detection comparison circuits 353A to 353F each detect image-data shifts by comparing the pieces of image data supplied from the main imager 74D with the pieces of image data supplied from the auxiliary imager 75D. Then, the parallax normalizationcoefficient computation circuits 354A to 354F each carry out a normalization process based on the image-data shifts to generate normalized parallax/distance coefficients. Subsequently, the parallax/distancecoefficient assignment circuits 355A to 355F each assign the normalized parallax/distance coefficients output by parallax normalizationcoefficient computation circuits 354A to 354F to the pieces of image data supplied by the main imager 74D. In the case of this first embodiment, a parallax/distance coefficient of I is assigned to the image data of the two books and the background whereas a parallax/distance coefficient of II is assigned to the image data of the mini car. The pieces of image data supplied by the main imager 74D are stored in theregisters 356A to 356F whereas the parallax/distance coefficients assigned to the pieces of image data are stored in a memory or storage unit associated with theregisters 356A to 356F. - As a result, the parallax/distance coefficient of I assigned to the pieces of image data stored in
sub-registers register 356A as well as the parallax/distance coefficient of II assigned to the pieces of image data stored insub-registers 3 to 7 of theregister 356A are stored in a memory or storage unit associated with theregister 356A. By the same token, the parallax/distance coefficient of I assigned to the pieces of image data stored insub-registers 1 to 4, 7 and 8 of the register 356B as well as the parallax/distance coefficient of II assigned to the pieces of image data stored insub-registers - Finally, the pieces of image data and the parallax/distance coefficients assigned to the pieces of image data are output to image-data recipients such as a display unit, a storage device and a network.
-
FIG. 10 is a diagram showing the second embodiment of a process to assign parallax/distance coefficients to image data obtained as a result of the image-taking operation shown inFIG. 6 . In the case of the second embodiment, the mini car moves slightly in the horizontal direction. - Since the mini car moves, the position of the mini car is shifted in a main imager 74E shown in
FIG. 10 from the position of the mini car in the main imager 74D shown inFIG. 9 in the horizontal direction to the left by one block. The image is shifted in an auxiliary imager 75E from the position of the mini car in the main imager 74E in the horizontal direction to the right by one block. Since the mini car moves in the horizontal direction, the distance from the mini car to thelenses FIG. 9 . On the other hand, the distance from the two books and the screen serving as the background to thelenses - As a result, the parallax/distance coefficient of I assigned to the pieces of image data stored in
sub-registers register 356A as well as the parallax/distance coefficient of II assigned to the pieces of image data stored insub-registers 2 to 6 of theregister 356A are stored in a memory or storage unit associated with theregister 356A. By the same token, the parallax/distance coefficient of I assigned to the pieces of image data stored insub-registers 1 to 3 and 6 to 8 of the register 356B as well as the parallax/distance coefficient of II assigned to the pieces of image data stored insub-registers - Finally, the pieces of image data and the parallax/distance coefficients assigned to the pieces of image data are output to image-data recipients such as a display unit, a storage device and a network.
-
FIG. 11 is a diagram showing the third embodiment of a process to assign parallax/distance coefficients to image data obtained as a result of the image-taking operation shown inFIG. 7 . In the case of the second embodiment, the mini car further moves in a direction approaching thelenses lenses lenses - Pieces of image data existing in the coordinate blocks A1 to A4 of the main imager 74F as a result of an image-taking process are data of the mini car whereas pieces of image data existing in the coordinate blocks A5 to A8 of the main imager 74F as a result of the image-taking process are data of the books and the background. Pieces of image data existing in the coordinate blocks B1 and B5 to B8 of the main imager 74F as a result of the image-taking process are data of the books and the background, whereas pieces of image data existing in the coordinate blocks B2 to B4 of the main imager 74F as a result of the image-taking process are data of mini car. Pieces of image data existing in all the remaining coordinate blocks of the main imager 74F as a result of the image-taking process are data of the books and the background.
- On the other hand, pieces of image data existing in the coordinate blocks A2 to A4 and A9 of the auxiliary imager 75F as a result of the image-taking process are data of the books and the background, whereas pieces of image data existing in the coordinate blocks A5 to A8 of the auxiliary imager 75F as a result of the image-taking process are data of mini car. Pieces of image data existing in the coordinate blocks B1 to B5 and B9 of the auxiliary imager 75F as a result of the image-taking process are data of the books and the background, whereas pieces of image data existing in the coordinate blocks B6 and to B8 of the auxiliary imager 75F as a result of the image-taking process are data of mini car. Pieces of image data existing in all the remaining coordinate blocks C2 to F9 of the main imager 74F as a result of the image-taking process are data of the books and the background.
- The main imager 74F supplies the pieces of image data to the parallax-
detection comparison circuits 353A to 353F by way of thehorizontal registers 351A to 351F respectively, which are provided for the main imager 74F. By the same token, the auxiliary imager 75F supplies the pieces of image data to the parallax-detection comparison circuits 353A to 353F by way of thehorizontal registers 352A to 352F respectively, which are provided for the auxiliary imager 75F. The parallax-detection comparison circuits 353A to 353F detects shifts between the pieces of image data. After the shifts between the pieces of image data are subjected to a normalization process in the parallax normalizationcoefficient computation circuits 354A to 354F and the parallax/distancecoefficient assignment circuits 355A to 355F, parallax/distance coefficients are assigned to the normalized values in the parallax/distancecoefficient assignment circuits 355A to 355F. In this embodiment, the parallax/distance coefficient of IV is assigned to the image data of the mini car serving as a front scene whereas the parallax/distance coefficient of I is assigned to the image data of the two books and the background. - As a result, the parallax/distance coefficient of IV is assigned to pieces of image data, which are then stored in
sub-registers 1 to 4 of theregister 356A. On the other hand, the parallax/distance coefficient of I is assigned to pieces of image data, which are then stored insub-registers 5 to 8 of theregister 356A. By the same token, the parallax/distance coefficient of I is assigned to pieces of image data, which are then stored insub-registers sub-registers 2 to 4 of the register 356B. The parallax/distance coefficient of I is assigned to all the remaining pieces of image data, which are then stored in the remaining sub-registers, i.e., sub-registers 1 to 8 of each of the registers 356C to 356F. - Finally, the pieces of image data and the parallax/distance coefficients assigned to the pieces of image data are output to image-data recipients such as a display unit, a storage device and a network.
-
FIG. 12 is diagrams each showing a model of a pseudo 3D (three-dimensional) effect of a reproduced image using assigned parallax/distance coefficients. By using the parallax/distance coefficients explained with reference toFIG. 8 , the ordinary 3D (three-dimensional) effect such as a both-eye 3D (three-dimensional) view can also be obtained in a reproduced image. In addition, the present invention also provides image processing to generate a 3D (three-dimensional) pseudo effect in not only both of the eyes, but also in one eye. - In this case, the pseudo effect can be obtained by using ordinary display means without requiring special display means such as a both-
eye 3D (three-dimensional) viewer. The parallax/distance coefficients of I, II and IV are inversely proportional to distances of 4, 2 and 1 units respectively. - In accordance with the embodiment, image processing is carried out to alternately shift image data of each horizontal-coordinate block (or each horizontal-coordinate group) in the horizontal direction by a shift distance proportional to the positional distance from the lenses to the object of photographing generating the image data. By focusing both the eyes on the front scene, it is possible to create an effect of showing the background separated away from the front scene by a pseudo positional distance, which exists between the scenes as a distance proportional to the square of the actual positional distance between the scenes, on a reproduced image seen by each one of the eyes.
-
FIG. 12A is a diagram showing data of a reproduced image. The data includes parallax/distance coefficients each assigned to a piece of block image data. For example, the parallax/distance coefficient of I is assigned to pieces of block image data represented by symbols α1, α2, α3, α4, α5, β1, β2, γ2, γ3, δ3, δ4, ε4, ε5, ζ5, η1, η2, η3, η4, η5 and θ1 to result in pieces of block image data represented by α1I, α2I, α3I, α4I, α5I, β1I, β2I, γ2I, γ3I, δ3I, δ4I, ε4I, ε5I, ζ5I, η1I, η1I, η2I, η3I, η4I, η5I and θ1I respectively, the parallax/distance coefficient of II is assigned to pieces of block image data represented by notations ι1, ι2, ι3, ι4, ι5, κ3, κ4, λ2, λ3, λ4, λ5 and μ1 to result in pieces of block image data represented by ι1II, ι21II, ι3II, ι4II, ι5II, κ3II, κ4II, λ2II, λ3II, λ4II, λ5II and μ1II respectively whereas the parallax/distance coefficient of IV is assigned to pieces of block image data represented by the capitals X, Y, Z and W to result in pieces of block image data represented by XIV, YIV, ZIV and WIV respectively. The parallax/distance coefficient each assigned to a piece of block image data is a value found on the basis of a detected parallax, and with the value, a pseudo 3D (three-dimensional) image is produced. -
FIG. 12B is a diagram showing a method of processing image data to demonstrate a pseudo 3D (three-dimensional) effect. For example, the pieces of block image data each having the assigned parallax/distance of I representing a positional distance of four units as expressed by α1I, α2I, α3I, α4I, α5I and β1I on the first row ofFIG. 12A are shifted to the left by one coordinate block. The pieces of block image data each having the assigned parallax/distance of I as expressed by β2I and γ2I on the second row ofFIG. 12A are shifted to the right by one coordinate block, but the pieces of block image data each having the assigned parallax/distance of II representing a positional distance of two units as expressed by ι1II, ι2II, ι3II and ι4II on the second row are shifted to the right by ½ coordinate blocks. - The pieces of block image data each having the assigned parallax/distance of I as expressed by γ3I and δ3I on the third row of
FIG. 12A are shifted to the left by one coordinate block, the pieces of block image data each having the assigned parallax/distance of II as expressed by ι5II and κ3II on the third row are shifted to the left by ½ coordinate blocks whereas the pieces of block image data each having the assigned parallax/distance of IV representing a positional distance of 1 unit as expressed by XIV and YIV on the third row are shifted to the left by zero coordinate blocks. - The pieces of block image data on fourth to the sixth rows of
FIG. 12A are processed in the same way by shifting each of the pieces of block image data in the right or left direction depending on the row number by a shift distance depending on the assigned parallax/distance coefficient. The series of operations carried out on the rows are carried out typically for every frame of the moving picture in image processing to produce the 3D (three-dimensional) pseudo effect even in one eye. The result of the image processing is shown inFIG. 12B . - In the embodiment described above, image data having the I parallax/distance coefficient indicating a small parallax is shifted by a long shift distance but, on the other hand, image data having the IV parallax/distance coefficient indicating a large parallax is shifted by a small shift distance. It is to be noted, however, that image data having the I parallax/distance coefficient indicating a small parallax can be reversely shifted by a short shift distance and, on the other hand, image data having the IV parallax/distance coefficient indicating a large parallax can be shifted reversely by a long shift distance.
-
FIG. 12C is a diagram showing an embodiment in which pieces of data are shifted in the right and left directions opposite to those of the embodiment shown inFIG. 12B . If the embodiment shown inFIG. 12B is referred to as an embodiment with a normal phase, the embodiment shown inFIG. 12C is known as an embodiment with an inverted phase. If the normal and inverted phases are introduced alternately every frame of the moving picture, the pseudo 3D (three-dimensional) effect is further enhanced. In addition to these operations, the luminance and contrast can be adjusted in accordance with the parallax/distance coefficients in order to further enhance the pseudo 3D (three-dimensional) effect. -
FIGS. 13A to 13C are diagrams showing typical images reproduced as images each exhibiting a pseudo 3D (three-dimensional) effect resulting from a process to assign parallax/distance coefficients. To be more specific,FIGS. 13A, 13B , and 13C show typical images demonstrating a pseudo 3D (three-dimensional) effect produced as a result of assigning parallax/distance coefficients in the image processing explained earlier by referring respective toFIGS. 9, 10 , and 11 as well as a result of adopting the principle described earlier reference toFIG. 12 . - In this case, however, parallax/distance coefficients are assigned to pieces of block image data for every granularity even finer than the granularity adopted in the image processing explained earlier by referring to each of
FIGS. 9, 10 and 11 as a granularity based on 8×6 coordinate blocks. That is to say, in the image processing explained earlier by referring to each ofFIGS. 9, 10 and 11, the parallax/distance coefficients of I, II and IV are assigned to pieces of block image data for every 8×6 coordinate blocks. In actuality, an image may typically include 3M (3 mega=2,000×1,500) pixels. In this case, the block shifts of I, II and IV correspond to 250, 500 and 1,000 pixels respectively. In other words, the parallax/distance coefficients of I, II and IV correspond to 250, 500 and 1,000 pixels respectively, which are arranged in the horizontal direction. Thus, a block used as the unit of the parallax/distance coefficients can be reduced to give a high resolution and the parallax/distance coefficients are assigned to pieces of block image data for every much finer granularity. -
FIGS. 13A to 13C are models each consisting of an upper diagram showing a typical image reproduced without a pseudo 3D (three-dimensional) effect and a lower diagram each showing a typical image reproduced with a pseudo 3D (three-dimensional) effect. The rear scene of the image is moved in line units in accordance to parallax/distance coefficients assigned to the line in order to make the mini car serving as the front scene three-dimensionally visible. This image can be displayed on an ordinary display unit requiring no special display means. In addition, since image data is processed in the image processing circuit to generate data of a pseudo 3D (three-dimensional) image as a result of the processing, the pseudo 3D (three-dimensional) image can be displayed on an ordinary display unit to allow a pseudo 3D (three-dimensional) image to be observed by either both the eyes or any one of the eyes. - As explained earlier by referring to FIGS. 9 to 11, the front scene is fixed and the rear scene serving as the background is shifted in the right or left direction in accordance with parallax/distance coefficients assigned to the rear scene in order to produce a pseudo 3D (three-dimensional) effect. It is to be noted, however that, reversely, the rear scene can also be fixed and the front scene is shifted in the right or left direction by a shift distance inversely proportional to the positional distance to the front scene in order to produce a pseudo 3D (three-dimensional) effect. In this case, by focusing both the eyes on the rear scene, it is possible to create an effect of showing the front scene separated away from the rear scene by a pseudo positional distance existing between the scenes as a distance proportional to the square of the actual positional distance between the scenes on a reproduced image seen by each one of the eyes.
-
FIG. 14 is diagrams showing models of reproduced images exhibiting an effect of assigned parallax/distance coefficients and other effects. - In addition to the pseudo 3D (three-dimensional) effect described above, image data can be processed to produce a zoom-up effect or a zoom-down effect, which is an effect opposite to the zoom-up effect. The zoom-up effect is an effect obtained as a result of applying a zoom-in process to a close image and a zoom-out process to a far image. In addition, by cutting out the image of a photographing object located at a specific distance from the entire image, the image processing can be carried out in a manner simpler than the chromakey technique.
- In particular,
FIG. 14A is a diagram showing data produced with assigned parallax/distance coefficients as data of a reproduced image.FIG. 14B is a diagram showing a result of extracting image data with the parallax/distance coefficient of IV from the data shown inFIG. 14A and enlarging an image represented by the extracted image data. In this case, however, image data with the parallax/distance coefficients of I and II is not processed and left as it is. As shown inFIG. 14B , the image portion represented by the pieces of image data shown as the capitals X, Y, Z and W is enlarged but the remaining image portions shown as notations and symbols are left as they are at their original sizes. In this way, an image completing a zoom-in process is obtained. -
FIG. 14C is a diagram showing an image obtained as a result of extracting only image data with the parallax/distance coefficient of IV from the produced image data having assigned parallax/distance coefficients as shown inFIG. 14A and eliminating image data with the parallax/distance coefficients of I and II. That is to say, only, the image portion represented by the pieces of image data shown as the capitals X, Y, Z and W is displayed. In this way, by assigning parallax/distance coefficients to pieces of data representing the image of an object of photographing, the image of a portion cut out from the image of an object of photographing can be obtained. -
FIG. 15 is diagrams showing typical images exhibiting effects other than the pseudo 3D (three-dimensional) effect obtained as a result of a process to assign parallax/distance coefficients to pieces of data representing the images. Much like the diagrams ofFIG. 13 ,FIGS. 15A, 15B , and 15C show typical images demonstrating a pseudo 3D (three-dimensional) effect produced as a result of assigning parallax/distance coefficients in the image processing explained earlier by referring respective toFIGS. 9, 10 , and 11. -
FIGS. 15A to 15C each show a typical result of applying a zoom-in process to the image data of the mini car on the basis of the parallax/distance coefficient assigned to the image data and extracting the image data completing the zoom-in process. In this way, by assigning parallax/distance coefficients to pieces of data representing the image of an object of photographing, desired image processing can be carried out on the basis of the coefficients. Thus, unlike the conventional chromakey processing, after the image-taking process, a variety of processes based on by assigning parallax/distance coefficients can be carried out with ease. The processes carried out after the image-taking processes include a process to extract a desired portion from an area of a displayed image and a process to synthesize desired image portions. - As described above, in accordance with the image processing method provided by the present disclosure, the image processing device adopting the method and the image display apparatus employing the image processing device, main and auxiliary imagers provided by the present disclosure as well as the image processing device are integrated in the configuration of a single LSI circuit and, the LSI circuit carries out parallel processing to compare a signal generated by each horizontal register provided for the main imager with a signal generated by the corresponding horizontal register provided for the auxiliary imager so that the processing can be carried out at a speed higher than the speed of the ordinary parallax image-taking configuration and is suitable for an image process produce an image of high minuteness and a high rate.
- By using such an LSI circuit including imagers and such an image processing device, it is possible to realize a Camcoder (a registered trademark of Sony Corporation) and a digital still camera, which are capable of recording information on parallaxes and distances along with image data.
- In addition, in accordance with the image processing method provided by the present embodiments, by assigning distance coefficients based on parallaxes, a pseudo 3D (three-dimensional) effect can be realized even by using only one eye so that it is possible to get rid of restrictions on the actual visual-field angle without using special display means such as a both-
eye 3D (three-dimensional) viewer. - Thus, it is possible to allow a number of observers to look at a display shown by only one display apparatus. It is also possible to provide a number of observers, who have different powers to view a display as well as a difference in visual ability between the two eyes, with a 3D (three-dimensional) effect of an image display allowing the observers to view a pseudo 3D (three-dimensional) image.
- By virtue of such an image processing method and an image display apparatus employing the image processing device adopting the image processing method, an entertaining content such as a private movie or private video produced at home can be emphasized into a 3D (three-dimensional) display and, in addition, a 3D (three-dimensional) attendance feeling and a speed sense can be stressed in the case of a game image.
- In addition, it should be understood by those skilled in the art that a variety of modifications, combinations, sub-combinations and alterations may occur in dependence on design requirements and other factors insofar as they are within the scope of the appended claims or the equivalents thereof.
- It should be understood that various changes and modifications to the presently preferred embodiments described herein will be apparent to those skilled in the art. Such changes and modifications can be made without departing from the spirit and scope of the present subject matter and without diminishing its intended advantages. It is therefore intended that such changes and modifications be covered by the appended claims.
Claims (17)
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JPJP2005-319728 | 2005-11-02 | ||
JP2005-319728 | 2005-11-02 | ||
JP2005319728A JP4424299B2 (en) | 2005-11-02 | 2005-11-02 | Image processing method, image processing apparatus, and image display apparatus using the same |
Publications (2)
Publication Number | Publication Date |
---|---|
US20070097207A1 true US20070097207A1 (en) | 2007-05-03 |
US8106939B2 US8106939B2 (en) | 2012-01-31 |
Family
ID=37995731
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US11/553,345 Expired - Fee Related US8106939B2 (en) | 2005-11-02 | 2006-10-26 | Image processing method, image processing device and image display apparatus employing the image processing device |
Country Status (2)
Country | Link |
---|---|
US (1) | US8106939B2 (en) |
JP (1) | JP4424299B2 (en) |
Cited By (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090051790A1 (en) * | 2007-08-21 | 2009-02-26 | Micron Technology, Inc. | De-parallax methods and apparatuses for lateral sensor arrays |
US20120070069A1 (en) * | 2010-09-16 | 2012-03-22 | Kabushiki Kaisha Toshiba | Image processing apparatus |
US8885025B2 (en) | 2010-06-29 | 2014-11-11 | Fujitsu Semiconductor Limited | Processor |
US20150341612A1 (en) * | 2014-05-20 | 2015-11-26 | Canon Kabushiki Kaisha | Image generation apparatus and method for controlling the same |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102025544B1 (en) * | 2013-01-02 | 2019-11-04 | 삼성전자주식회사 | Wearable video device and video system having the same |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5581276A (en) * | 1992-09-08 | 1996-12-03 | Kabushiki Kaisha Toshiba | 3D human interface apparatus using motion recognition based on dynamic image processing |
US5726704A (en) * | 1993-08-26 | 1998-03-10 | Matsushita Electric Industrial Co., Ltd. | Stereoscopic image pickup and display apparatus |
US6157733A (en) * | 1997-04-18 | 2000-12-05 | At&T Corp. | Integration of monocular cues to improve depth perception |
Family Cites Families (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH0715748A (en) | 1993-06-24 | 1995-01-17 | Canon Inc | Picture recording and reproducing device |
JPH07220113A (en) | 1994-01-28 | 1995-08-18 | Canon Inc | Image recording/reproducing device |
JPH0955959A (en) | 1995-08-11 | 1997-02-25 | Yamaha Corp | Picture display |
JP3276931B2 (en) | 1996-08-07 | 2002-04-22 | 三洋電機株式会社 | 3D image adjustment method and 3D image adjustment apparatus |
JP3471262B2 (en) | 1999-07-30 | 2003-12-02 | 日本電信電話株式会社 | 3D image processing device |
JP2003143459A (en) | 2001-11-02 | 2003-05-16 | Canon Inc | Compound-eye image pickup system and device provided therewith |
JP2003281503A (en) | 2002-03-20 | 2003-10-03 | Fuji Heavy Ind Ltd | Image recognition device for three-dimensional object |
JP2003304562A (en) | 2002-04-10 | 2003-10-24 | Victor Co Of Japan Ltd | Object encoding method, object encoder, and program for object encoding |
JP2004040445A (en) | 2002-07-03 | 2004-02-05 | Sharp Corp | Portable equipment having 3d display function and 3d transformation program |
JP2004242000A (en) | 2003-02-05 | 2004-08-26 | Sony Corp | Encoding device and method, and decoding device and method |
JP4181446B2 (en) | 2003-05-14 | 2008-11-12 | シャープ株式会社 | Stereoscopic image display device |
-
2005
- 2005-11-02 JP JP2005319728A patent/JP4424299B2/en not_active Expired - Fee Related
-
2006
- 2006-10-26 US US11/553,345 patent/US8106939B2/en not_active Expired - Fee Related
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5581276A (en) * | 1992-09-08 | 1996-12-03 | Kabushiki Kaisha Toshiba | 3D human interface apparatus using motion recognition based on dynamic image processing |
US5726704A (en) * | 1993-08-26 | 1998-03-10 | Matsushita Electric Industrial Co., Ltd. | Stereoscopic image pickup and display apparatus |
US6157733A (en) * | 1997-04-18 | 2000-12-05 | At&T Corp. | Integration of monocular cues to improve depth perception |
Cited By (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20090051790A1 (en) * | 2007-08-21 | 2009-02-26 | Micron Technology, Inc. | De-parallax methods and apparatuses for lateral sensor arrays |
US8885025B2 (en) | 2010-06-29 | 2014-11-11 | Fujitsu Semiconductor Limited | Processor |
US20120070069A1 (en) * | 2010-09-16 | 2012-03-22 | Kabushiki Kaisha Toshiba | Image processing apparatus |
US8908959B2 (en) * | 2010-09-16 | 2014-12-09 | Kabushiki Kaisha Toshiba | Image processing apparatus |
US20150341612A1 (en) * | 2014-05-20 | 2015-11-26 | Canon Kabushiki Kaisha | Image generation apparatus and method for controlling the same |
US9609303B2 (en) * | 2014-05-20 | 2017-03-28 | Canon Kabushiki Kaisha | Image generation apparatus and method for controlling the same |
Also Published As
Publication number | Publication date |
---|---|
US8106939B2 (en) | 2012-01-31 |
JP2007129436A (en) | 2007-05-24 |
JP4424299B2 (en) | 2010-03-03 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP5468676B2 (en) | Image capture device and method | |
EP2008445B1 (en) | Improved plenoptic camera | |
JP5224124B2 (en) | Imaging device | |
US9341935B2 (en) | Image capturing device | |
EP0645659B1 (en) | Three dimensional imaging apparatus, camera, and microscope. | |
JP5468482B2 (en) | Imaging device | |
CN102917235A (en) | Image processing apparatus, image processing method, and program | |
TW201909401A (en) | Imaging element and imaging unit | |
JP2016511562A (en) | Imaging apparatus for imaging using a plurality of microlenses and imaging method therefor | |
CN104885440B (en) | Image processing apparatus, camera device and image processing method | |
JP2010154493A (en) | Image pickup device | |
CN102213893A (en) | Image pickup apparatus and image pickup device | |
US8106939B2 (en) | Image processing method, image processing device and image display apparatus employing the image processing device | |
CN102007762A (en) | Still image and moving image capturing device | |
CN111669483B (en) | Image sensor, imaging device, electronic apparatus, image processing system, and signal processing method | |
JP2016127043A (en) | Solid-state image pickup element and electronic equipment | |
JP2000032354A (en) | Image pickup device | |
JP2013201466A (en) | Stereoscopic image pickup device | |
US20230007191A1 (en) | Image sensor, imaging apparatus, electronic device, image processing system, and signal processing method | |
TW201043017A (en) | Spatial-pixel-offset imaging device | |
US20120212581A1 (en) | Image capture apparatus and image signal processing apparatus | |
CN103621078A (en) | Image processing apparatus and image processing program | |
WO2016190437A1 (en) | Image capturing and displaying system | |
WO2016088565A1 (en) | Solid-state image-capturing device, and electronic apparatus | |
CN103503447A (en) | Imaging device and program for controlling imaging device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SONY CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TORIUMI, YOICHI;REEL/FRAME:018604/0373 Effective date: 20061201 |
|
ZAAA | Notice of allowance and fees due |
Free format text: ORIGINAL CODE: NOA |
|
ZAAB | Notice of allowance mailed |
Free format text: ORIGINAL CODE: MN/=. |
|
FEPP | Fee payment procedure |
Free format text: PAYOR NUMBER ASSIGNED (ORIGINAL EVENT CODE: ASPN); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
FPAY | Fee payment |
Year of fee payment: 4 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 8TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1552); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 8 |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE UNDER 1.28(C) (ORIGINAL EVENT CODE: M1559); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20240131 |