WO1999024987A1

WO1999024987A1 - Editing compressed signals

Info

Publication number: WO1999024987A1
Application number: PCT/GB1998/003400
Authority: WO
Inventors: Michael James Knee
Original assignee: Snell & Wilcox Limited
Priority date: 1997-11-12
Filing date: 1998-11-12
Publication date: 1999-05-20
Also published as: AU1047499A; GB9723893D0; EP1031157A1

Abstract

The user of portable MPEG based video equipment is able to produce an Edit Decision List (EDL) in the field. Using the EDL and a knowledge of the Group of Pictures (GOP) structure, a selection is made not only of the frames directly required in the EDL, but also the P and I frames relied upon for prediction purposes. By transmitting or storing only these selected frames, rather than the originally recorded MPEG material, considerable savings can be made. In the studio, the selected frames and EDL are used to provide the desired edited MPEG bitstream.

Description

EDITING COMPRESSED SIGNALS

This invention relates to the editing of compressed bitstreams and is particularly suitable for editing in the field using compact equipment.

The invention will find application with any compression system in which the bitstream itself is not easily edited, for example because it involves prediction to or from information at different points in time. A well known example is the MPEG-2 video compression standard (ISO/IEC 131818-2), particularly when operated at low bit-rates with relatively long periods between intra-coded (I) pictures, that is to say pictures which do not rely upon prediction. In MPEG-2, the sequence of picture types (I, P or B) is known as the Group of Pictures or GOP structure.

A method has been proposed for editing MPEG-2 bitstreams which involves decoding, editing in the video domain and subsequent re-encoding, there being no loss arising in the cascading of decoding and re-encoding processes. This is achieved by carrying forward information about an upstream encoding process so that a downstream re-encoding process makes use of the same coding decisions (see for example W095/35268 and WO97/08898).

One problem which this invention addresses will arise with compact, portable picture acquisition and recording systems, such as a camcorder, using MPEG-2 at a low bit-rate as its storage format. This allows very efficient use of the available disk or tape capacity. A difficulty arises, however, if facilities for editing the bitstream are required within the portable equipment. Two approaches using prior art could be considered:

In theory, it would be possible to use the above-described decode/edit/re-encode approach in the portable equipment to provide full editing capability, so that only the final, edited bitstream is transmitted from the portable equipment. This theoretical approach is illustrated in Figure 1. In practice, however, the necessary technology is not yet available in a sufficiently miniaturised form for practical implementation in portable equipment.

Another theoretical approach would be to generate a browse track and to allow an Edit Decision List (EDL) to be created in the portable equipment, but to transmit the originally recorded material along with the EDL to a remote location such as a studio, and use the above described approach to complete or conform the final edit there. This approach is illustrated in Figure 2. The disadvantages of this approach are the high transmission bandwidth or time that might be required to transmit the originally recorded material to the studio and the increased storage demands. It will be understood that the proportion of the originally recorded material that remains in the edited output, may be very small.

It is an object of the present invention to provide an improved method of editing bitstreams, which overcomes or ameliorates the problems in both the theoretical approaches described above.

Accordingly, the present invention consists in one aspect in a method for use in the editing of compressed bitstream sequences which include intra- coded and prediction-coded frames each corresponding with a respective video frame, comprising the steps of identifying video frames to be represented in an edited bitstream sequence; selecting first coded frames which correspond with said identified video frames; selecting second coded frames which do not correspond with said identified video frames but which are required for prediction of any prediction-coded frames among said first coded frames; passing said first and said second coded frames through a channel; utilising said second coded frames to decode said prediction-coded frames among the first coded frames; and re-encoding the decoded frames to provide the edited bitstream sequence.

The channel in question may comprise - as examples - a communication link or a bitstream store. In either case, considerable advantage will be secured from avoiding the need to pass all coded frames. Since all necessary coded frames are passed, the edited bitstream sequence suffers no degradation.

In another aspect, the present invention consists in apparatus for use in the editing of compressed bitstream sequences which include intra-coded and prediction-coded frames each corresponding with a respective video frame, comprising an input bitstream source, an edit decision list terminal for receiving identification of video frames chosen for an edited bitstream output and frame selection means adapted to select in addition to first coded frames which correspond with said chosen video frames, those second coded frames which are required for prediction of any of said first coded frames.

In an embodiment which minimises the number of second coded frames, the frame selection means is adapted to select in addition to first coded frames which correspond with said chosen video frames, those second coded frames and only those second coded frames which are required for prediction of any of said first coded frames. Thus, in the example of MPEG2, it might usefully be arranged that B frames are never selected as second coded frames, since a B frame (whatever its location in the group of pictures) is never required for prediction of a P or B frame. In an embodiment which may require less processing, the frame selection means is adapted to select in addition to first coded frames which correspond with said chosen video frames, contiguous sequences of coded frames which contain those second coded frames which are required for prediction of any of said first coded frames. Each such contiguous sequence may be selected dynamically to be of the minimum length necessary to ensure that all second coded frames are required. Alternatively, a fixed length can be determined - with reference to the known GOP structure - such that inclusion of all second coded frames is guaranteed.

The invention will now be described by way of example with reference to the accompanying drawings, in which:-

Figures 1 and 2 are block diagrams illustrating two different theoretical approaches to the editing of compressed bitstreams, which are generated remotely; and Figure 3 is a block diagram of apparatus according to the present invention.

Before turning to a detailed description with reference to the drawings, it may be helpful to define certain terms used in this specification. Thus:

A browse track is a signal recorded alongside the main compressed signal which enables quick access to recorded segments on a tape or disk. A browse track will have reduced resolution or quality.

An edit decision list (EDL) records the results of an operator's edit decisions made using the browse track. It consists of a list of segments of recorded material together with precise descriptions of the editing operations to be performed.

The process of conforming an edit is to apply the EDL to the main recorded material to generate the final edited sequence in the same compressed format.

Referring initially to Figure 1 , the portable equipment comprises an MPEG2 encoder 10, receiving video material from a source such as a camera. A browse track generator 12 generates a browse track from the video material and the MPEG2 bitstream and browse track are recorded together on a disk, tape or other store 14. A suitable user interface 16 is provided to enable the remote operator, viewing the browse track, to prepare an EDL. This is then stored in the EDL unit 18. In more detail, a browse track is generated and stored in the portable equipment, either at the time of acquisition or off-line. In accordance with prior art, this browse track could be a reduced-resolution version of the picture in which each frame is self-contained so that the usual "trick" modes such as fast-forward, fast rewind, pause and slow-motion replay can be performed as well as viewing of any proposed edit sequence at browse quality. A graphical user interface, again in accordance with prior art, is used to view the browse track and interpret the operator's decisions to form an EDL. The edit conformer 20 receives the EDL and in communication with the store 14, generates an edited, compressed bitstream. The manner in which this achieved is known in itself and need not be described here.

Unfortunately, the hardware represented by the edit conformer 20 is not available in a sufficiently miniaturised form for it to be included in any sensibly portable equipment.

In the second approach shown in Figure 2, the edit conformer 20 is situated not in the portable equipment, but in the studio. This necessitates the transmission of the originally recorded material and the EDL, to the studio. Transmission of the originally recorded material will typically demand high bandwidth (which may not be available in an economic manner) or transmission times that are inconveniently long.

Turning now to the embodiment of this invention which is illustrated in Figure 3, the arrangement of MPEG2 encoder 10, browse track generator 12, store 14, user interface 16 and EDL unit 18, is the same as that of Figures 1 and 2. As in the second approach illustrated in Figure 3, the final edit conformer 20 is located in the studio.

What distinguishes the present invention is how the required material is selected for transmission. This is achieved in the "select required frames" block 22 which uses the EDL and knowledge of the GOP structure of the recorded material to determine precisely which frames of the recorded material need to be transmitted to the studio to enable the final edit to be conformed.

An example will now be given of how material is selected for transmission in the case of a cut between two sequences. The example shows the I, P and B frames of the two sequences in display order. It will be remembered that I frames are intra-coded with no prediction, P frames are forward predicted from preceding I or P frames and B frames are forward and backward predicted. In the following representation, the underlined frames are those that are selected for transmission to the studio. The final edited sequence will consist of frames from Sequence 1 up to the out point, followed by frames from Sequence 2 starting from the in point.

Sequence 1 : ...I B B P B B P B B P B B I B B P out point

Sequence 2: ....P B B I B B P B B P B B J P B B I B B P... in point In Sequence 1 , all the frames up to and including the P-frame after the out point are transmitted to the studio because the MPEG-2 decoder in the studio will require access to that P-frame for prediction purposes. Likewise, in Sequence 2, all the frames from the l-frame immediately before the in point are transmitted to the studio, because that l-frame and all the P-frames between it and the in point are required for prediction purposes. Thus, two additional frames are transmitted from Sequence 1 and nine additional frames are transmitted from Sequence 2.

The selected frames can then be transmitted to the studio in real time at an average bit-rate only slightly higher than that recorded on the tape or disk. Alternatively, they can be transmitted at the recorded bit-rate slightly slower than real-time, or some other trade-off between transmission bit-rate and time can be found.

A refinement of the invention, in which the number of additional frames to be transmitted is reduced further, is illustrated by the following example based on the same sequences as above. Again, the frames to be transmitted are shown underlined.

Sequence 1 : ...I B B P B B P B B P B B I B B P out point

Sequence 2: ....P B B ] B B P B B P B B [ P B B I B B P... in point

This refinement is based on the observation that B-frames are never required for prediction, so no additional B-frames are required. Thus, the numbers of additional frames required for Sequences 1 and 2 are reduced to one and three, respectively. However, the complexity of the processing required to identify and format the remaining additional frames for transmission will be increased slightly.

In a simplified version of the invention, the number of additional frames following the out point and preceding the in point is fixed to a constant, for example the maximum number of frames in a GOP, that is known to embrace all possible cases. This would be simpler to implement but might increase the transmission bandwidth or time significantly if there are many edit points. The underlined coded frames, together with the EDL, are transmitted (for example over a satellite link) to the studio where they are received by the edit conformer. This essentially operates to decode and then re-encode certain coded frames. Particularly, these will be prediction-coded frames which correspond to selected video frames but which rely for prediction on coded frames which do not correspond to selected video frames. In the sequence examples given above, the coded frames which will require to be decoded and re-encoded are shown underlined below:

Sequence 1 : ...I B B P B B P B B P B B I B B P out point

Sequence 2: ....P B B I B B P B B P B B P B B I B B P. in point

Attention is directed to WO95/35268 and WO97/08898 (which are hereby incorporated by reference) where it is shown that the cascaded decode and re-encode processes can be lossless if the re-encode process uses the original coding decisions. These are available in the decoder and can be passed in an information bus to the re-encoder.

The above examples refer to a simple cut between two sequences. The invention is equally applicable to more complex processing such as fades or wipes, or indeed any process in which the frames that will ultimately be required to conform the edit are known to the portable equipment.

The above description has concentrated on MPEG-2 video bitstreams. The invention may equally be applied to any other compressed video, audio or data format where it is advantageous to conform an edit remotely.

Claims

1. A method for use in the editing of compressed bitstream sequences which include intra-coded and prediction-coded frames each corresponding with a respective video frame, comprising the steps of identifying video frames to be represented in an edited bitstream sequence; selecting first coded frames which correspond with said identified video frames; selecting second coded frames which do not correspond with said identified video frames but which are required for prediction of any prediction-coded frames among said first coded frames; passing said first and said second coded frames through a channel; utilising said second coded frames to decode said prediction-coded frames among the first coded frames; and re-encoding the decoded frames to provide the edited bitstream sequence.

2. A method according to Claim 1 , wherein the channel comprises a communication link.

3. A method according to Claim 1 , wherein the channel comprises a bitstream store.

4. A method according to any one of the preceding claims, wherein coding decisions deduced from said decoding are used in said re-encoding.

5. Apparatus for use in the editing of compressed bitstream sequences which include intra-coded and prediction-coded frames each corresponding with a respective video frame, comprising an input bitstream source, an edit decision list terminal for receiving identification of video frames chosen for an edited bitstream output and frame selection means adapted to select in addition to first coded frames which correspond with said chosen video frames, those second coded frames which are required for prediction of any of said first coded frames.

6. Apparatus according to Claim 5, wherein the frame selection means is adapted to select in addition to first coded frames which correspond with said chosen video frames, those second coded frames and only those second coded frames which are required for prediction of any of said first coded frames.

7. Apparatus according to Claim 5, wherein the frame selection means is adapted to select in addition to first coded frames which correspond with said chosen video frames, contiguous sequences of coded frames which contain those second coded frames which are required for prediction of any of said first coded frames.

8. Apparatus according to Claim 7, wherein the frame selection means is so adapted that said contiguous sequences are of minimum length.