WO1992005664A1

WO1992005664A1 - Video image composition

Info

Publication number: WO1992005664A1
Application number: PCT/GB1991/001620
Authority: WO
Inventors: Michael Joseph Kemp
Original assignee: Spaceward Holdings Limited
Priority date: 1990-09-20
Filing date: 1991-09-20
Publication date: 1992-04-02
Also published as: GB9020540D0

Abstract

A video image composition system receives first and second video image signals from respective sources (2, 4). Signals derived from a control image (6) are used to control the combination of the first and second video images. An addressing means (10) is used to read out signals from individual pixels in the three images (2, 4, 6). An interpolation parameter is derived for the combination of the first and second images at an addressed pixel. The interpolation value for that pixel is derived in dependence on the control image values stored in a patch around that pixel. The first and second images are combined in an interpolator (20) in proportions dependent on the interpolation value to give a composite image (8).

Description

VIDEO IMAGE OCMPOSΓΠCN

This invention relates to video image composition systems and in particular to systems in which the images are digital images.

In graphic arts, film, and video production it is often necessary to combine multiple images into a single composite image. A problem in such composition when the images are in digital form is achieving a smooth blend between parts of the cαrposed image which originate from different source images, without a sharp transition and jagged edges along the pixel boundaries.

Figure 1 illustrates a typical prior art composition system. The system coπprises a frame store 2 which stores a background image, a frame store 4 which stores a foreground image and a frame store 6 which stores a control image to be used in combining background and foreground images from stores 2 and 4. If we consider a monochrome image in both foreground -and Cackground then the stores 2 and 4 will both be, typically, of 8 bits per pixel. The control image 6 is a single bit per pixel.

To combine the foreground and background images to produce a composed image in a frame store 8, corresponding pixels are read-out from frame stores 2 and 4 under the control of addressing circuitry 10 and fed to the inputs of a video switch 12. This is controlled by a corresponding bit received from the control image frame store 6 to output either a pixel from the foreground image or a pixel from the background image. If the corresponding pixel in the control image is a 1 nen the foreground image is output to frame store 8 and if it is a 0 then the background image is output.

A sample of the pixels stored in the control image frame store 6 is shown at 14 and tvvo samples from the composed image in frame store 8 are shown at 16 and 18. It can be seen that the form of the control image results in a sharp transition from foreground to background with no blending of the two images taking place.

This system can be arranged to work with video rate inputs and outputs and therefore without the need for any frame stores.

The sharp transitions and jagged edges at 16 and 18 are unattractive in the composed image and do not give a realistic composition.

One solution which has been proposed to the problem of eliminating jagged edges and sharp transitions from foreground to background is described in a paper entitled "Painting Tutorial Notes" by Alvy Ray Smith which was presented at Siggraph 1979. This system is illustrated in Figure 2. The background and foreground images from frame stores 2 and 4 are as in Figure 1 but the control image 6 has multi-bit values for each pixel. Typically there are 8 bits per pixel. These multi bit values are used to represent a degree of cross blending between foreground and background. With an 8 bit control image value there are 256 possible values for each pixel. The background and foreground images are composed in a linear interpolator 20 to give a composed image in frame store 8.

The linear interpolator takes the background and foreground images and cx-mbines them for each pixel under the control of the numerical value stored in the control image frame store 6. It operates according to the equation; PC = alpha x PF + (1 - alpha) x PB

Where PC is the value of the composite pixel and PF and PB are the values of the foreground and background pixels respectively. Alpha is the value stored in the control image. Thus it can be seen that that when there is a 0 in the control plane the composite image will comprise entirely background data, when there is a 1 in the control plane the composite image will comprise entirely foreground image data and when there is an inter-nediate value there will be a blend of background and foreground image data.

A transition from foreground to background in the control image frame store 6 is shown at 22. The data in the store goes from 0 at the left hand side of the patch 22 to 255 (which corresponds to the value of 1) at the right hand side. The resultant composed image is shown at 24. It can be seen that there are intermediate values of shading between the dark background and the light foreground.

A similar method of image composition to this is described in UK patent no. 2113950.

In many image composition systems the control image is never avail-able as a multi-bit word for each pixel. For example simple chroma-key devices do not have this facility, neither do hard key control images recorded along with a foreground image, for example a "superblack" key. Even when a multi value control image is available the degree of blending which it provides may not be visually acceptable to a user, such as an artist using the image composition facility on a video painting system, and he may wish to vary the degree of blending. A preferred ei±iodiment of the invention allows for the creation of a composite image with blended transitions whether or not multi valued control image data is available. It also allows the softness of the edge between foreground and background to be controlled by a user.

The invention is defined in the appended claims to which reference should now be made.

The invention will now be described in detail by way of example with reference to the figures in which:

Figure 1 is a circuit diagram of the first prior cart device described;

Figure 2 is a circuit diagram of the second prior art device described above;

Figure 3 is a circuit diagram of a system embodying the invention;

Figure 4 shows a distortion of an image.

Figure 5 shows a first modification of figure 3 including a hold-out mask; and

Figure 6 shows a second modification of figure 3 including a hold-out mask.

Figure 3 shows a frame store 2 storing a background image, a frame store 4 storing a foreground image, and a frame store 6 storing a control image. The control image conprises a single bit per pixel. Pixels required to produce a composite image of foreground and background are read out from stores 2, 4 and 6 by a pixel address generator 10. The signals from the foreground and background images are fed to a linear interpolator 20 which produces, from them, a composite output which is fed to store 8-

The interpolation parameter alpha for the linear interpolator is generated as follows. When a pixel is addressed a patch of pixels surrounding the corresponding pixel in the control irrage 6 is used to generate the value alpha. The connection from the pixel address generator 10 to the store 6 is via an adder 22. This has its other input connected to a local patch addressing circuit 24. For each pixel address the local patch addressing cirucit 24 addresses a patch of pixels in the store 6 by means of this adder. At the same time it addresses a corresponding pixel in a local weighting store 26. The value read out from the weighting store 26 is multiplied by the value read out from the corresponding pixel in the control image 6 in a multiplier 28. Thus, when the value from the control image is a 1 the value from the weighting store 26 is available at the output of the multiplier 28 and when it is a 0, a 0 is available at the output of the multiplier 28. Preferably the local addressing circuitry 24 is arranged to align the central pixel of the weighting store 26 with the addressed pixel in store 6.

The values for the whole of the local patch are summed together in cumulative adder 30. This is clocked by a further signal received from the local patch addressing cirucit 24 until the whole of the patch has been processed. The cumulative adder 30 then makes its output available and this forms one the input to a divider 32. The value is divided by a scaling factor K to produce a valid value for alpha. The exact value used for K will depend on the size of the weighting store 26 and the values stored therein. It is chosen so that alpha will be in the range 0 to 255 for a linear interpolator which expects an 8 bit alpha value. The alpha output from the divider thus forms the input to the linear interpolator 20.

Once a patch has been processed the cumulative adder 30 is zeroed and the process steps onto the next pixel.

Considering first the case in which the weighting store contains the same value in each of its pixels it will be appreciated that when the patch is centered on the the boundary between foreground and background the resultant pixel value will comprise data from both foreground and background. The proportion of foreground or background will increase in dependence on how far the patch overlaps the boundary edge.

The weighting store 26 can store different arrangements of values, for example preferably a circular array with a peak at the centre and zeroes at the edges. The shape based on the gausian curve is ideal but in practice any shape with a central peak and declining to the edge gives good results. In practice a square array is unlikely to be used.

Tne rate at which a shape stored in the weighting store 26 declines to 0 controls the sharpness of the blend between foreground and background. If the central value is high and all surrounding values are zero then the results will be the same as Figure 1. If the values decline to 0 within two pixels the image is still sharp but the blend should produce very natural looking results without the jagged edges of Figure 1. If the value declines to 0 over 4 pixels then an even softer blend can be achieved.

Thus it can be seen that the softness of the blend can be varied within a weiφting store and different results could be acnieved by using different sizes of weighting store.

Preferably an image composition system will enable a user to load different distributions of weighting values into the weighting store 26. These are selected by standard menu selection techniques.

The local patch addressing circuitry 24 can be arranged to adapt to the size of the shape stored in weighting store 26 so that values known to be 0 are not processed. This will result in faster processing of the images.

Displacement coordinates (X, Y) can be fed to address generator 10 to effect a displacement of the foreground image with respect to the background. This can lead to the pixels addressed by the local patch addressing circuit 24 being outside the stores 2, 4, and 6. In fact this also occurs in normal processing of pixels adjacent the edges of the background and foreground images. When such illegal addresses are encountered the control image store 6 is arranged to set the output at 0 to simulate total transparency of the foreground image outside its defined area.

To enable a user to see the composed image before he permanently commits it to the store 8 the system can be adapted to continually reprocess the image from stores 2, 4, and 6 and continually update the store 8 whilst the user moves the position of the foreground image using a tablet and stylus coupled to the pixel address generator 10. Once he has moved the foreground image to a desired position he can permanently store the image in the frame store 8. Store 8 is arranged so that its contents can be viewed on a monitor as the artist moves the foreground image relative to the background image. Because the composition process can be slow without a fast processor, particularly if a large weighting store 26 is used, the speed of movement can be improved by replacing the weighting store values with a single value in the centre until the desired position has been selected (as in the prior art device of figure 1) . Thus whilst movement is taking place the jagged edges will be seen but once the user has chosen the final position the proper weighting array from weighting store 26 can be used to create a softened edged composition and permanently store this in frame store 8.

If suitably fast hardware is used it is possible to update the final image in store 8 in real time. In such a situation the foreground and background images in stores 2 and 4 can come from real time sources such as video cameras. The control image in store 6 requires temporary storage to allow for the local addressing to take place but otherwise it also can be updated in real time from a source of keying signals. In practice a "double buffered" system is used in accordance with known techniques and stores 4 and 6 are updated together with related foreground and keying signals. In this way a source with a hard key can still give a properly blended final video image.

For optimum speed the scaling factor K is fixed at an appropriate power of 2 e.g. 256 as this represents an 8 bit shift rather than a full divide operation. The values stored in weighting store 26 are then carefully selected so that they sum to 256 x 255, thus resulting in a maximum value fed to the linear interpolator of 255.

Where values in the control image store 6 are available as ulti bit values for each pixel, good results are obtained by disregarding the additional information and simply using the most significant bit to represent 0 or 1. However, these values could be used as inputs to multiplier 28 with a suitable adjustment to the scaling factor K to give an improved blend over the system shown in Figure 2.

It is often desired to distort the foreground image 4 in some way prior to composition. When this is done it is appropriate to perform an identitical distortion on the control image 6 so that the area def ned to be composed remains the same part of the original image. Such distortions are best performed by known techniques in which the original pixels are weighted to form new pixels. This prevents aliasing of the foreground image. It is appropriate to use the same anti-aliasing technique on tr-a-nsforming the control image and this applies whether or not it is a single bit control image. After distortion the new control image is used as described above in the system on Figure 3.

Figure 4 shows a typical distortion of the foreground image. In 4a a source image 30 is distorted into a smaller rotated image at 32. Figure 4b shows how the old pixel array is mapped onto the new pixel array. Pixel 34 can be seen to cσπprise several whole or part pixels from the source image. It is necessary to calculate the area covered by each of the source pixels to create the new pixel 34 by means of a weighted average of the constitutent pixels. This can be done by known techniques.

In the case where the image 30 is a foreground image from store 4 a new foreground image is created directly. Where the image 30 is a control image the values of the control image should be changed from 0 and 1 to 0 and 255 in the case of a simple on/off control image or we can simply use the original values if the control image is already a multi-value image. The weighted average can then be used to produce a new control image in the range 0 to 255. However, to perform the composition according to the present invention it is only necessary to use the most significant bit of this distorted control image as described with reference to Figure 3.

In a preferred e_ιi--odiment of the invention an additional control image or hold out mask is also provided. This represents an area at the background image where the foreground image is not to be applied to the background image. In prior art devices such as that shown in Figure 1 this additional control signal would be used to inhibit switch 12 such that irrespective of the signal received from the control image 6 the pixel from the background would be used in the composed image 8.

In the current invention when a hold-out mask is used as shown in figure 5 it is stored in frame store 40 and addressed in parallel with the background image 2 but with the additional displacement from patch addressing circuit 24 via adder 43. The signal is inverted in inverter 42 and fed through an AND gate 44 along with the signal read out from the control image 6. Thus a value 1 from the hold-out mask will force the value 0 into the cumulative adder 30. This represents the shift of the interpolator in favour of the background image and no foreground image will be applied at all for pixel well within the protected area. Conversely a value 0 from the hold-out mask enables the control image 6 to pass its values through to the cumulative adder 30.

If the hold out mask value is an 8 bit value in the range 0 to 255 and it is desired not to provide a variable softening effect then the arrangement of figure 6 can be used. The hold-out mask value is complemented in inverter 42 and ultiplied in multiplier 46 with the final output of divider 32. This then provides a proportionally transparent protection to the pixels of the background image by varying alpha in depedence on the hold-out mask value from store 40 following the variable blending function derived in cumulative adder 30. In this arrangement the hold-out mask store 40 is addressed directly by addressing cicuitry 10. If desired a small change can be made to the value of K or to the overall total of the weighting store 26 so that the output of the this second multiplier will be kept in the range 0 to 255.

The above ei odiments of the invention have been described with apparatus which would be appropriate to a monochrome TV or graphics system. In practice, colour systems are preferable and there will therefore be three components, typically red, green and blue. Thus some of the circuitry shown will be triplicated. However, a single control image is all that is required for all three components as is a single cumulative adder 30 and divider 32. Thus the output from the divider 32 will be fed three ways to to three linear interpolators, one each for the red, green -and blue components.

The control image can be created in a number of known ways such as by the user drawing with a stylus on a graphics tablet whilst viewing an image of the control plane. It may be created as a single bit value or as a multi-bit value signal. Altenatively the control image may be derived directly from a video image itself by means of, for example, selection of a specified range of lu a values (lu a keying) or of chroma values (chroirø keying) It may also be a separate input to the system. The system has been described herein as a hardware system. However, the invention can be -Lirplemented in software in any general purpose cαrrputi-ng system.

The invention can be applied to any image composition system in which the images an be provided in digital form, for example, film and graphic arts.

Claims

1. A video composition system comprising a first source of video signals representing a first image; a second source of video signals representing a second video image; a source of control image signals; means for addressing pixels in each of the images; means for deriving an interpolation parameter in dependence on the control image values stored in a patch of pixels surrounding the addressed pixel; means for cαrrfoining the first and second video signal values at that pixel in proportions dependent upon the interpolation parameter; and output means for the composite image.

2. A system according to claim 1 in which the deriving means conprises a weighting store corresponding to a patch of pixels, patch addressing means, means for providing a cumulative total of values from the weighting store in dependence on corresponding values from the control image, and means for deriving the interpolation parameter from the cumulative total.

3. A system according to claim 2 in which the patch addressing means centres the patch on the addressed pixel.

4. A system according to claim 2 or 3 in which the weighting store stores values which have a maximum value in the centre of the store and decrease towards the edges of the store.

5. A system according to claim 2, 3 or 4 in which the cu ulative total of values is provided by multiplying values from the weighting store by corresponding values from the control image and storing a cumulative total of the result.

6. A system according claim 5 in which the cumulative total is divided by a scaling factor.

7. A system according to any preceding claim in which the control image conprises a single bit per pixel.

8. A system according to any preceding claim in which the first source of video signals conprises means for storing a frame of video signals.

9. A system according to any preceding claim in which the second source of video signals conprises means for storing a frame of video signals.

10. A system according to any preceding claim in which the source of control image signals conprises means for storing a frame of control image signals.

11. A system according to any preceding claim including means for displacing the control image and a corresponding one of the first and second video images with respect to the other video image.

12. A system according claim 11 including manually operable means supplying a displacement value to the addressing means.

13. A system according to claim 11 or 12 in which a desired displacement can be viewed on a display means before being selected and stored.

14. A system according to claim 13 including means for deriving a single bit control image for use in the composition of the image prior to selection and storage.

15. A system according to any preceding claim including a source of hold-out mask control signals, and means for inhibiting cαπposition of the first and second video images in dependence on the hold out mask signal values.