WO2005093652A1 - Image capture device and related method - Google Patents

Abstract

The present invention provides for an image capture device for capturing frames from a photo electric sensor arrangement comprising a plurality of pixels, the image capture device being arranged to capture a plurality of images only from pixels within a central region of the photo electric sensor arrangement and for subsequent assembly into a composite image and wherein the said central region generally comprises a central strip of pixels.

Description

DESCRIPTION

IMAGE CAPTURE DEVICE AND RELATED METHOD The present invention relates to an image capture device, and related method of image capture, for use, in particular, with a digital camera to be used for creating a panoramic image.

Some current available digital cameras have a selectable mode of operation allowing for the creation of a panoramic image through the combination of a plurality of images captured therein. When in such a mode, the camera is held as steadily as possible by the user who then pans the camera smoothly across the desired scenery whilst activating the panoramic mode. A plurality of pictures are taken at a speed relative to the speed with which the camera pans the desired scenery so as to allow for overlap between the separate images. Subsequent image processing is then conducted so as to effectively join the separate images together to form the panoramic image. In addition to correctly aligning the series of separate images, it also becomes necessary to allow for corrections for changes in perspective of each image. This can cause straight lines to appear curved at the edge of each image and it becomes necessary not only to calculate the appropriate two- dimensional position of each image relative to its neighbour. Also the relative angle of alignment and focal length and exposure values need to be taken into account in order to create an overall uniform panoramic image. Thus, with current digital cameras, it becomes necessary to determine five variables for each pair of adjacent frames within the panoramic image and comprising the relative x-axis position, the relative y-axis position, the relative angle, focal length and exposure setting. The processing exhibited by known digital cameras therefore disadvantageously limits quality particularly with images employing a variety of straight lines, which tend to be clearly noticeable when the panoramic image is viewed.

The present invention seeks to provide for an image capture device, and in particular such device for use within a digital camera, and related method, and which exhibits advantages over known such devices and methods.

According to a first aspect of the present invention, there is provided an image capture device for capturing frames from a photo-electric sensor arrangement comprising a plurality of pixels, the image capture device being arranged to capture a plurality of images only from pixels within a central region of the photo electric sensor arrangement and for subsequent assembly into a composite image. The invention is advantageous in that, in taking only a central portion of the image available at the photo electric sensor arrangement, the accuracy with which the plurality of images can be combined is greatly increased whilst also reducing, and in some cases eliminating, the distorting effects that can arise in the region of the join between two neighbouring images. Preferably, the photo-electric sensor arrangement scans only the pixels within the said central region. The said pixels within the said central region can advantageously comprise a central strip of pixels arranged in a vertical axis. Advantageously, the central strip of pixels exhibits an aspect ratio of 1 :8. Further, the central strip of pixels can advantageously comprise in the order of 5-10% of the total frame width. The image capture device is advantageously arranged for assembly of the plurality of images into a panoramic image. The present invention can also provide for a digital camera including an image capture device as defined above. Advantageously, the camera can be arranged to take pictures from which the said plurality of images are captured at a rate of at least two frames per second. According to another aspect of the present invention, there is provided a method of capturing images from a photoelectric sensor arrangement comprising a plurality of pixels and for subsequent assembly into a composite image, the method comprising the steps of capturing a plurality of images only from pixels within a central region of the photo-electric sensor arrangement for the said subsequent assembly. Preferably, the said central region comprises a central strip of pixels advantageously having an aspect ratio in the order of 1 :8, preferably comprising 5-10% of the total frame width. The strip of pixels advantageously provides a vertical strip of pixels and the method can further include temporary storing the said plurality of images in a buffer memory prior to delivery to an assembly processor. In particular, the invention provides for a method of forming a panoramic image and including the capturing images in a manner defined above. The method and device embodying the present invention are particular advantageous in that, while it is found that the composite image still might require some degree of rotation in order to obtain the most accurate horizontal panorama, alignment problems such as those suffered with current devices do not occur. As a particularly advantageous feature, only two variables, that is those of x-axis and y-axis positions, need to be varied as compared with the five separate variables discussed above in relation to the operation of current devices. While it is noted that the width of each central slice of pixels can be in the order of 5-10% of the total frame width, it is nevertheless appreciated that the actual width can vary dependent upon various factors. For example, there will be a maximum speed allowed for the user to pan the camera across the required scenery and which will depend upon the rate at which the camera can capture and process, the series of separate images so as to retain a degree of overlap between images. Also, to some extent there will be a trade off between the focal length of the lens and the desired image resolution since, for example, a wider angle lens may require that the slices be narrow in order to avoid the requirement for perspective correction. The device can further be provided, or associated, with a buffer memory within which each of the plurality of images is temporally stored prior to further processing. Yet further, the device can comprise, or be associated with, an image assembly processor for combining the said plurality of images received from the buffer memory. The buffer advantageously limits the speed with which the assembly processor will receive the plurality of images and this advantageously affords the assembly processor more time to, for example, correlate neighbouring images. In one particular arrangement, a cross-correlation function is employed for arriving at the most appropriate relative alignment between neighbouring images. Advantageously, the cross-correlation function is restricted to searching only a relatively small range of possible offsets between neighbouring images so as to advantageously reduce the processing time. Preferably, the assembly processor is arranged to find a best-fit straight line so as to define upper and lower boundaries of the composite image. Also, such a best-fit straight line is determined by means of linear regression such as by applying a least squares algorithm to the origins of each of the plurality of images with the composite image. The assembly processor is advantageously arranged to determine the alignment of each of the plurality of images by means of a correlation coefficient. Further, the vertical separation between each of the plurality of images and the aforesaid boundary line is advantageously determined and compared with a threshold value. Yet further, the device and method of the present invention are arranged to provide for a warning to the user should a condition leading to potential failure in producing the composite image be identified. Advantageously, such a warning can be generated responsive to the slope of the aforesaid boundary line, the degree of vertical offset of a new image with respect of the said boundary line and/or the magnitude of horizontal offset of a new image relative to the proceeding image. In this manner, it advantageously becomes possible to provide a warning if the camera is either being held at an inappropriate angle; held either two high or two low; or perhaps is being panned too quickly across the required scenery. Advantageously, the algorithm employed for locating the appropriate relative position of neighbouring images is advantageously arranged to commence its position search on or near the zero-vertical offset location in order to advantageously increase the likelihood of obtaining a match with the most appropriate relative position. The device and/or method embodying the present invention can also store at least half of the images for forming the lateral extremes of the composite image. This can advantageously lead to an increase in the width of the composite image but without requiring any additional processing. Advantageously, the camera employed in the present invention can be arranged to employ a constant exposure setting for each of the plurality of images. As an alternative, the exposure setting can be arranged to be set automatically for each of the said plurality of images but only within a restricted range of exposure values for neighbouring frames. It should therefore be appreciated that the invention provides for image capture device and related method that can be employed with the digital camera so as to enhance the quality of composite images, such as panoramic images, that can be created with such a camera. The invention is described further hereinafter, by way of example only, with reference to the accompanying drawings in which: Fig. 1 is a schematic block diagram illustrating the operation of an image capture device embodying the present invention and as found within a digital camera; Fig. 2 is a flow diagram illustrating one aspect of the operation of the device of Fig. 1 ; Fig. 3 is a further flow diagram illustrating another aspect of the operation of the device of Fig. 1 ; Fig. 4 illustrates the creation of a composite image in accordance with an embodiment of the present invention; Figs. 5-8 illustrate the alignment of images combining to form a composite image in accordance with an embodiment of the present invention; and Fig. 9 illustrate the use of images of increased width at the two lateral extents of the composite image.

Turning now to Fig. 1 , there is illustrated an image capture device employed within a digital camera having a mode of operation for creating a panoramic photograph. The camera includes a photo electric sensor arrangement comprising a 2048x1536 pixel CCD device 10 which includes a central strip 12 of pixels which, in the illustrated example, is in the order of 200 pixels wide and which serve to provide an image slice. As is discussed further below, a particular aspect of the present invention is that it is only the pixels within the central strip 12 that are scanned in order to provide an image captured by means of an image capture device 14 which is arranged, in turn, to deliver each of the image slices captured to a slice buffer memory 16 having, in the illustrated example, five image slices 18 temporally stored therein. The slice buffer memory 16 serves as a buffer for slowing the delivery of the image slices 18 from the image capture device 14 to a panorama assembly processor 20. Within the panorama assembly processor an algorithm such as cross- correlation is employed in order to position the image slices so as to create a panoramic image 24. Such assembled images are then delivered to a panorama buffer memory 22 within which one example of a composite image

24 is illustrated. The further handling of the composite image is then controlled by means of a controller 26 for delivery of the composite image either under the control of a user interface device such as a keypad 30, to, for example, a nonvolatile storage card 28, and/or for delivery to a display device such as liquid crystal display 32. The panorama assembly process 20 can also be arranged to provide feedback to a user by means of an audio/visual output device 34 which can likewise comprise the liquid crystal display 32. In order to capture a panoramic image, the user is required to enable the panoramic mode within the camera and then to pan the camera smoothly across the desired scenery. The repeated slices 18 are arranged to be captured by the image capture device as quickly as possible and delivered to the slice buffer memory 16 while the user continues to pan the camera across the desired scenery. The width of each slice 12,18 is advantageously in the order of 5-10% of the total frame width such that, for a 2048x1536 pixel CCD such as that 10 illustrated in Fig. 1 , the width, in pixels of the central slice 12 will be in the order of 200 pixels. However, the exact width of the central slice can be determined having regard to the maximum speed with which the user can pan the camera but which will be limited to some extent on the rate at which the camera can capture and process each of the frames since the camera must maintain the ability to keep up with the movement of the camera by the user so as to ensure that each image slice is able to overlap, even to a small degree, with its neighbouring image slice. Also, characteristics of focal length and the desired resolution/quality can have a bearing on the nature of the image captured from the central slice of the CCD sensor 10 since a wider angle lens may require that the chosen segments are somewhat narrower than the dimension noted above in order to avoid the need for perspective correction. Advantageously within the illustrated embodiment, in real time, and while the user continues to pan the camera across the desired scenery, the assembly processor 20 is arranged to remove image slices from the buffer memory 16 and to then employ a cross-correlation algorithm to construct the panorama 24 within the panorama buffer memory 22. Upon reaching the end of the desired panorama, the desired image formed by the series of cross-correlated images 24, can be stored in the nonvolatile storage card 28 and displayed as required on the LCD display 32. Turning now to Fig . 2, there is provided a flow diagram illustrating the manner in which images are captured from the CCD in accordance with the arrangement of Fig. 1. Step 36 represents the user pressing the shutter button in order to initiate the generation of a panoramic image by way of a series of photographs. Upon determining that the shutter button has been pressed, the process proceeds to step 38 at which a first of the image slices is captured from the central slice 12 of the CCD 10 and a determination is then made at step 40 as to whether the slice buffer memory 16 is full. Assuming the slice buffer memory is not full, the new slice is then stored therein at step 42 and a further determination is made as to whether the user has released the shutter button. If the shutter button has been released, then the process terminates at step 46. Returning to step 40 at which it was determined whether or not the slice buffer memory 16 was full, if it is there determined that the slice buffer memory 16 is full, then an error condition is identified and reported at step 48 and the process brought to a halt at step 50. Returning however to step 44, and the determination as to whether the shutter button remains pressed, if it is determined that the user has not released the shutter button, the processes returns via path 52 to allow for the capture of the next image slice 18 from the CCD 10. This process then continues generally until such time as the user believes they have completed the required panoramic image and released the shutter button. It should be appreciated that the slice buffer memory 16 proves an important feature since the processor 20 may take a variable amount of time to complete the required processing depending generally upon how easy it proves for neighbouring image slices to be aligned. Use of the slice buffer memory 16 then provides the panorama assembly processor 20 with more time to complete the required processing and this advantageously makes the system more tolerant to irregular camera movement patterns by the user. In one particular aspect however, the slice buffer memory 16 is arranged to be sufficiently small so that any user-feedback from the assembly processor 20 can be received in time for the user to take appropriate corrective measures, for example if a warning is provided that the camera is being held at an inappropriate orientation, for example, if it is pointing too high. Turning now to Fig. 3, there is illustrated a flow chart to further clarify operation of the panorama assembly processor 20 of Fig. 1. The processing starts at step 54 and then at step 56 a first image slice is obtained from the slice buffer memory 16. At subsequent step 58, the panorama buffer memory 22 is initialised with the first slice and, assuming there is a further image slice following the said first slice, this further slice is then retrieved from the slice buffer at step 60. At step 62, the best relative alignment between the slice within the panorama buffer and the current slice is then determined on the basis of a cross-correlation function and it is subsequently determined at step 64, whether or not the resulting correlation coefficient is below a threshold value. If the value is not below the said threshold, then the procedure continues to step 66 where it is determined whether the relative alignment between the two slices is small, and generally within a determined range. If the relative alignment is not within a determined range, the process then proceeds to step 68 at which the best linear alignment for slices already assembled is calculated and a further determination then made at step 70 as to whether any slices are out of alignment. Assuming no such slices are out of alignment, the procedure continues to step 72 and which slice obtained from the panorama assembly processor 20 is blended into the panorama buffer memory 22 of Fig. 1. At step 74, it is then determined hether or not the said latest slice was the last slice and, if so, the process continues to step 76 where the composite image 24 within the panorama buffer memory 20 is skewed and cropped to form a rectangular image. The procedure then ends at step 78. If, however, at step 74, it is determined that the said latest image slice was not the last image slice, the process returns via path 80 to step 60 so as to obtain the next slice from the slice buffer memory 16. Further, if, at step 64 it is determined that the correlation coefficient is too low and therefore below a threshold value, the process continues via 82 to allow for an error to be reported to the end user and then proceeds to conclude the procedure at 78. Also, if the relative alignment value is within the small predetermined range noted at step 66, the process continues directly via path 84 to step 74 where it is determined whether or not the said latest slice comprises the last slice. Although, in accordance with the present invention, a two-dimensional cross-correlation algorithm is employed for performing image registration, it should be appreciated that this is merely by way of an example since a wide variety of alternative algorithms exist. However, in order to provide further detail for this illustrated example, the cross-correlation processing consists of effectively sliding one image across the other such that all relative alignments are considered, and a correlation coefficient is then calculated at each respective position. The correlation coefficient provides the measure of how well the two images match and this produces a two-dimensional function wherein the peak of this function serves to identify the most appropriate relative alignment. Advantageously however, within the present invention, the cross- correlation function can be restricted to searching only a relatively small range of possible offsets since it is appreciated that there should always be only a small horizontal movement, generally positive only and a very small vertical movement, generally positive or negative, between neighbouring slices. This will advantageously serve to minimize the amount of processing time required for generating the appropriate image registration. For example, and with reference to Fig. 4, if an extra image slice is to added to the right-hand end of a panorama 86, the cross correlation function might only explore possible alignments where the new slice is shifted between 0% and +75% of the slice width and between -5% and +5% of the slice height with respect to the previous slice. Within Fig. 4, there is indicated a new slice 88 to be cross-correlated with the existing panorama 86 and, within the chain-dotted region 90, there is illustrated a possible range of overlap between the new slice and the existing panorama that should be considered during the cross-correlation in accordance with the restricted searching feature noted above. The correlation coefficient produced will have a value between 0 - which indicates no similarity between the neighbouring images - and 1 - indicating that the images are identical and perfectly aligned. If, however, the actual coefficient obtained is below a predetermined threshold value, this is considered to indicate that there is an unrecoverable problem such as the camera being moved or rotated so fast that neighbouring image slices do not overlap. As illustrated in Fig. 3 at step 64, if the correlation coefficient is too low, the process is aborted via route 82 as discussed. Remaining again with Fig. 3, and the step 66 at which it is determined whether the relative alignment is of a small value within a predetermined range, this can advantageously serve to identify whether the user is moving the camera very slowly such that each new slice will not add any useful image data to the panorama being created. If this relative alignment is within the small range then the new slice can be ignored as illustrated by path 84 so as to continue the process directly at step 74 where it is determined whether or not the said latest slice comprises the last slice. Turning now to Fig. 5, there is illustrated a plurality of slice images which are combined so as to form part of a panoramic image. While, with regard to the panoramic image 92, a user would ideally pan the camera such that there is no vertical offset between the image slices, in practice, some degree of vertical offset is always likely to arise such as that illustrated by the combined image slices 94. While acknowledging that such vertical alignment will arise, it is found appropriate to employ a best-fit straight line to define the upper and lower boundaries of such an irregular panorama. The creation of such a best-fit straight line can be achieved through linear regression such as applying a least squares algorithm to the origins of each image slice within the panorama. Figs. 6 illustrates the creation of such an upper boundary line for image slices forming a composite image 96 and wherein each of the image slices has an origin point 98. The boundary line 100 produced serves to define the least squares fit of all origin points for a series of image slices. Once having determined the boundary line, it then proves readily possible to determine whether or not any of the image slices are out of alignment to such a degree that the panoramic image is likely to be discarded. Initially, the correlation coefficient is checked and if it is below a threshold value this can serve to indicate that the overall alignment of the slices is too random such that the process should be halted and the user notified. However, it can prove advantageous to suppress such a determination until the panoramic image has reached a predetermined width and this serves to prevent the process failing unnecessarily such as if the user pans the camera too slowly during the early stages of creating the panorama such that the origin points are then bunched un-reprehensively closely in an x-axis direction. Secondly, the vertical separation between the origin point of each slice is captured and the boundary checked against a predetermined limit. If the image slices are found to be too far out of alignment with respect to the boundary line, then the process must again be halted and the user notified. Again such as illustrated at step 70 within Fig. 3 Turning to Fig. 7 however, there is illustrated an example in which the user has started to pan the camera with a slight upwards lift which is initially acceptable. However, after the 6th slice has been added, this drift changes to a downwards drift and after the 9th slice has been added, the separation between the origin point of the 6th slice, and the boundary line - indicated by arrow A - becomes greater than the predetermined failure limit such that the process must then be stopped and the user notified. The use of such processing can advantageously be extended so as to provide the user with feedback before any specific limits are reached as discussed further below. Returning to step 76 within the process illustrate in Fig. 3, once the last slice has been blended into the panorama buffer memory 22, the slope of the boundary lines such as 102 and 104 in Fig. 8 can be employed to define the shape of the parallelogram inside the panorama. These boundary lines 102, 104 can then be employed to allow for cropping of the irregular parts from the top and bottom of the panoramic image as illustrated in Fig. 8. The resulting parallelogram can then be skewed so as to produce the final rectangular panoramic image. The operational characteristics such as those discussed above can be further enhanced by appropriate management and monitoring of conditions that may lead to the process being aborted. In particular, the invention can advantageously be arranged to issue an audible and/or visual warning signal to the user that particular operational characteristics are such that, if they continue, the process will have to be aborted. Thus, for example, if it is determined that the slope of the aforementioned boundary line in an upward or downward direction is too great, a warning can be issued to the user that the camera is being held at an inappropriate angle. Also, if the vertical offset of a new image slice with respect to the boundary line is too large, the user can be warned that the camera is being pointed either too high, or too low, as compared with previous image slices. Further, if the horizontal offset of a new image slice with respect to the proceeding slice is too large then the user can be warned that the camera is being paned too quickly having regard to the maximum speed with which the camera can take the required pictures. If, however, it is identified that the camera is being moved too slowly, there is no particular need to issue a warning since no unnecessary data will actually be stored. Any such user feedback can be indicated by way of the liquid crystal display 32 although, as an alternative, or addition, an audible warning signal can also be generated as required. While it is recognised that the picture registration algorithm such as the cross-correlation algorithm discussed above will require a significant amount of the available CPU resources available, the requirement for such resources can be minimised to some extent. For example, the processor can be arranged to terminate the cross-correlation algorithm relatively early if a suitable value of correlation coefficient is identified relatively early. However such early termination of the cross-correlation algorithm would be based on the assumption that, for most of the time, the user is able to pan the camera smoothly and large vertical shifts will therefore be considered unusual. On this basis, the algorithm can be arranged to start searching around the zero vertical offset point and this is likely to increase the speed with which a suitable match between neighbouring image slices can be obtained. A wider search would then only be required occasionally and the slice buffer would then be arranged to cope with the incoming data on those relatively few occasions when a wider search is needed. Also, with regard to Fig. 9, there is illustrated a further improvement allowing for an increased width of composite image without necessarily requiring any further processing. Turning to Fig. 9, there is illustrated a composite image formed from a plurality of image slices 106 bounded at each end by wider slices 108 , 110 which, in the illustrated embodiment, comprise images captured from the left half and right half of the CCD sensor illustrated in Fig. 1. Since these end images 108, 110 define the lateral extent of the composite image, and there is no further need to match further images thereto, the images can naturally define the left and right boarders of the composite image. That is, since each end of the composite image does not need to be aligned nor blended with any other image, it becomes possible to keep the first and last half frames captured from the CCD sensor 10 and without requiring any adjustment for perspective. While increasing the dimension of the composite image, it also makes the complete image somewhat easier for the user to shoot since the image will then include everything that the user sees through the camera's viewfinder. Further advantages can be incorporated so as to provide for a relatively uniform exposure for each of the images forming the composite image. For example, once the camera is switched into a panoramic mode, additional control setting can be incorporated so as to ensure that exposure settings are balanced across the width of the panorama. In one example, the exposure setting could be simply locked at the time of capturing the first frame when in the panoramic mode and then retained at that setting for all subsequent frames. As an alternative, the exposure can be set automatically for each frame but with limitations being imposed on the allowed difference between the exposures between neighbouring frames. This would advantageously allow for a change in exposure across the width of the composite image but would ensure that the change is slow and gradual. Also, in the panoramic mode, the camera can be arranged to increase shutter speed, along with an automatic increase in aperture, when switched into the panoramic mode.

Claims

1. An image capture device for capturing frames from a photo-electric sensor arrangement comprising a plurality of pixels, the image capture device being arranged to capture a plurality of images only from pixels within a central region of the photo electric sensor arrangement and for subsequent assembly into a composite image.

2. A device as claimed in Claim 1, wherein the photo-electric sensors is arranged only to scan pixels within the said central region.

3. A device as claimed in Claim 1 or 2, wherein the said central region comprises a central strip of pixels arranged in a vertical axis.

4. A device as claimed in Claim 3, wherein the central strip of pixels exhibits an aspect ratio of 1 :8.

5. A device as claimed in Claim 3 or 4, wherein the width of the central strip of pixels is in the order of 5-10% of the total frame width.

6. A digital camera including an image capture device as claimed in any one or more of the preceding claims.

7. A method of capturing images from a photo-electric sensor arrangement comprising a plurality of pixels and for subsequent assembly into a composite image, the method comprising the steps of capturing a plurality of images only from pixels within a central region of the photoelectric sensor arrangement for the said subsequent assembly.

8. A method as claimed in Claim 7, and including the step of scanning only pixels within the said central region.

9. A method as claimed in Claim 8, wherein the said central region comprises a central strip of pixels.

10. An image capture device substantially as hereinbefore described with reference to the accompanying drawings.

11. A method of capturing images from a photo-electric sensor arrangement, substantially as hereinbefore described with reference to the accompanying drawings.