EP2025174A1

EP2025174A1 - Use of a feedback channel for image broadcasting

Info

Publication number: EP2025174A1
Application number: EP07858480A
Authority: EP
Inventors: Marc Baillavoine; Joël JUNG; Loïc BAROUX
Original assignee: France Telecom SA
Current assignee: Orange SA
Priority date: 2006-05-29
Filing date: 2007-05-22
Publication date: 2009-02-18
Also published as: FR2901648A1; US20090138933A1; US8505060B2; WO2008043923A1

Abstract

The invention relates to a method for broadcasting images from a video sequence, wherein said method comprises the following steps at a broadcasting equipment (2): including images (14) in an output flow for transmission to at least one restoration equipment; feedback receiving from said restoration equipment error messages related to the restoration of images from the video sequence; analysing (24) the error messages in order to identify images or parts of images not or badly restored; determining (32) answer messages including correction images or image parts. The method is characterised in that it further comprises: a step (26) for determining a gravity level (FGi) for each error message according to the analysis performed; and a decision step (30) for sending back an answer message to an error message according to the gravity level of the error message.

Description

USING A RETURN CHANNEL FOR IMAGE BROADCAST

The present invention relates to techniques for broadcasting video sequence images. It applies to situations where a sequence of images is transmitted to one or more decoders that use a return channel to provide information explicitly or implicitly indicating whether the images of the video signal have been properly rendered.

Many video encoders support an interframe coding mode if inter-coded, in which the current image is encoded with respect to one or more previous images.

Each image can also be coded without reference to others, ie what is called intra-frame coding. For a given transmission rate, the intra coding provides poorer video quality than the inter coding since it does not take advantage of the temporal correlations between the successive images of the video sequence.

Commonly, a portion of a video sequence has its first intra-coded image and then the following inter-coded images.

A problem of inter coding is its behavior in the presence of transmission error or packet loss on the transmission channel. The degradation or loss of an image propagates on subsequent images until a new intra-coded image occurs. It is common that the mode of transmission of the signal causes total or partial losses of certain images. Such losses result, for example, from the loss or the late arrival of certain data packets when the transmission takes place over a network of packets without delivery guarantees such as an IP (Internet Protocol) network. Losses can also result from errors introduced by the transmission channel beyond the correction capabilities of the correction codes used.

In an environment subject to various signal losses, it is necessary to provide mechanisms to improve the quality of the image at decoder level. One of these mechanisms is the use of a return channel, from the decoder to the encoder on which the decoder informs the coder that he has lost all or part of certain images. In some cases, these are the images reconstructed well as the decoder instructs the encoder on channel ^"back, so that the coder can deduce what images were eventually lost.

In response to each of these error messages, most coders systematically return intra-coded correction images, that is, without reference to previous images, thereby refreshing the display and correcting the errors. errors due to transmission losses. The correction images are transmitted in response messages during specific transmissions or by insertion into the stream of the video sequence in lieu of other images.

Such use of the return channel has significant disadvantages when several error messages are transmitted on the return channel. The current principle results in an answer being given to each of these messages. For example, if an image is lost, the decoder may send an error message on the return channel for each of the subsequent images encoded inter with a reference to the lost image. In this case, the encoder is required to transmit several response messages each comprising an intra-coded correction image. This results in decreased quality of playback and high bandwidth usage.

Likewise, in certain multi-broadcast embodiments, such as conference bridge or other systems in which several decoders receive the same information, it happens that each of the decoders emits an error message on a return channel concerning an same image lost and that the coder responds successively to each of these messages retransmitting several times the same correction image. Thus, the use made of the return channels is not optimum especially when several messages are transmitted on the return channel.

One of the advantages of the present invention is to improve the use of the return channel. The invention thus proposes a method of broadcasting images of a video sequence, the method comprising, at the level of a broadcasting equipment, the following steps:

- include images in an output stream for transmission to at least one rendering equipment; receiving feedback from said reproduction equipment error messages relating to the restitution of the images of the video sequence;

- analyze the error messages to identify images or parts of images not or badly restored;

determining response messages including images or portions of correction images, characterized in that the method further comprises:

a step of determining a severity level for each error message according to the analysis performed; and

a decision step of returning a response message to an error message according to the severity level of this error message.

Accordingly, the method of the invention includes an analysis of the error messages and the quantification of the severity of these messages in order to decide whether to respond to them. This quantization makes it possible to optimize the use of the return channel by optimizing the response messages.

In a particular embodiment, said analysis comprises the determination of at least one factor coming from the following group: a level of quality of reproduction taking into account the error; an evaluation of the importance of the content of the images or parts of images that are not or badly reproduced;

- decoding parameters;

- an assessment of the conditions of transmission; and an estimate of the duration between the current image and the last image transmitted without error, said factor being used to determine the severity level associated with an error message.

Advantageously, said level of severity of at least one error message is a maximum waiting time before sending a response message. This makes it possible to ensure the transmission of a response message for certain error messages.

In a variant, the method comprises determining an emergency level from the severity levels associated with the error messages received since the transmission of the last response message and said decision step is performed according to this level of urgency. 'emergency. This embodiment has the advantage of determining an emergency quantizer from several error messages.

Advantageously, said step of determining response messages comprises a grouping of response messages. This reduces the number of response messages compared to the number of error messages.

In a particular embodiment, said analysis comprises an identification of a last image transmitted without error and said step of determining a response message comprises the coding of a correction image with respect to said last image transmitted without error. This embodiment avoids the transmission of an intra-coded image in order to improve the quality of the reproduction.

Alternatively, said step of including the images in a stream comprises formatting several incoming streams to form said stream of output, said step of analyzing the error messages comprises the identification of the images not or badly restored in said incoming flows and said step of determining a response message comprises the modification of correction images from said incoming flows to make them conform to the format of the output stream.

Furthermore, the invention also relates to a computer program or broadcast equipment implementing the method referred to above and an image diffusion system using such broadcast equipment. The invention will be better understood in the light of the description and the drawings in which:

FIG. 1 is a block diagram of a video sequence broadcasting system;

FIG. 2 is a symbolic representation of a combined video image;

FIG. 3 represents the flowchart of the method of the invention;

- Figures 4A and 4B symbolically represent loss of images;

FIG. 5 represents a correction image; and - Figure 6 symbolically represents a loss of image.

The system shown in Figure 1 comprises several terminals labeled T1 to T4 and formed, for example, personal computers. These terminals are connected by a telecommunications network 4 of the IP type to a broadcast equipment 2 which, in the example, is a telecommunications server providing the conference bridge function.

The terminals T1 to T4 are suitable for acquiring and transmitting a video and audio stream by means of, for example, cameras connected directly to each of the computers. The conference bridge 2 comprises means 6 for transmitting and receiving audio and video streams and a memory 8 and a microcontroller or microprocessor 10, these elements being conventional in themselves.

The microprocessor 10 includes in a memory, a specific program comprising software instructions capable of making it carry out the method of the invention when they are executed by this microprocessor.

During an initial configuration phase, it is agreed, between the conference bridge and the various terminals, to broadcast a combined image to the terminals and to use a return channel for the detection of transmission errors.

More precisely, each of the terminals T1 to T4 transmits a video stream F1 to F4 intended for the conference bridge 2, each of these streams comprises images encoded in a conventional manner, for example according to the H263 protocol with a determined size and resolution. In the configuration described, the conference bridge 2 retransmits the incoming flows F1 to F4 by rearranging them, so as to generate an output stream φ that can be decoded by the other terminals. This reorganization corresponds to formatting, or formatting, of the incoming flows F1 to F4, each image of an incoming flow being placed in a specific sector of the corresponding image in the output stream φ. FIG. 2 diagrammatically represents an image of the output stream comprising four sectors, one for each incoming stream F1 to F4.

The output stream is transmitted conventionally to all terminals T1 to T4. With reference to FIG. 3, the method of the invention will now be described.

The method starts with a step 12 of receiving the incoming flows F1 to F4, followed by a step 14 of combining these flows to form the output stream φ which is transmitted simultaneously to the four terminals during a step Each of the terminals T1 to T4 then receives the flux φ, decodes it and analyzes the decoded images.

In the event of packet loss or deterioration of the quality of the images, each of the terminals is able to issue an error message relating to the restitution of the images of the video sequence to the broadcasting equipment formed by the conference bridge 2 .

In the example, the method comprises a step 18 of initialization of a variable UR corresponding to an emergency level of the transmission of a response message, and a step 20 of initialization of variables FGi corresponding to severity levels associated with the error messages received.

After emitting images of a video sequence in step 16, the conference bridge 2 watches for the arrival of any error messages. This is done during a test step 22 to check whether all the received error messages have been processed. This step 22 is repeated as long as no error message is received.

Subsequently, one or more error messages are received by the conference bridge 2.

Following the detection of the reception of a first error in step 22, the method comprises an analysis step 24 which makes it possible to determine a severity level associated with the error, during a step 26. .

In the example, the analysis 24 includes an evaluation of a quality level of the restitution calculated according to the percentage of image fragments or macroblocks lost, that is to say corresponding to the quotient between the number of blocks lost and the total number of blocks of the image. This quality factor evaluation is noted fq and is calculated as follows:

Where Np is the number of blocks lost, and N _{t is} the total number of blocks in the image. Advantageously, the analysis also includes an evaluation of the importance of the content of the lost part to determine the level of severity. Thus, if the lost part corresponds to a uniform bottom, the loss will be considered less important than if the lost area contained a moving object. For example, this content analysis is performed by comparing the lost area between the different previous images available in the memory 10.

Similarly, the location of the lost part can also be taken into account to assess its importance. For example, the central area of the image is considered more important than the peripheral areas.

As a variant, the analysis of the error message includes the evaluation of parameters related to the decoding, such as the sequence or image parameters commonly called SPS (Parameter Set Sequence) and PPS (Picture Parameter Set). Alternatively, the analysis 24 comprises the identification of the last image transmitted without errors and the time elapsed between the current image and the latter image. The more this time is important, the more the correction will induce a degradation of the quality. In the example, this time factor is noted ft and is calculated as follows:

In these equations, l _c is the number of the current image, I _d ic is the number of the last correct image and N _m represents the number of images in the memory 10 of the bridge 2. Fs represents a reaction threshold of bridge, that is to say the maximum duration stored in the memory 10. If the time between the current image and the last correct image is greater than the memory duration, the last correct image is no longer available in the memory. the memory 10 and a fixed value is assigned to Ft. Advantageously, the analysis 24 takes into account the transmission conditions and in particular the evolution of the behavior of the transmission channel between the bridge and the terminals. In the embodiment described, the probability of a loss occurring on another channel is estimated from the loss statistics provided by the network equipment and is taken into account. The higher the probability, the longer the delay before a response is sent because the risk of another error message is high. This statistical factor is noted Fstat and is calculated as follows:

Where Pr (i) represents the percentage of losses on channel i between bridge 2 and terminal Ti, and Nj represents the total number of terminals in conference, ie four in the example.

According to all these factors, the level of gravity FGi is determined for each error message during a step 26. In the example, this level of gravity is obtained by summing the values obtained for each factor with, advantageously , coefficients, denoted λ, weighting the relative importance of the factors. So :

FGi = fiq + λit ^• fit + Mstat • fistat The method then comprises a step 28 for determining the current emergency level, denoted UR, corresponding to the sum of the severity levels FGi of all the messages received since the last transmission. reply message.

This level of urgency corresponds to a quantification of the overall urgency of a response to the various error messages received.

The method then comprises a decision step 30 of returning one or more response messages to the error messages received since the transmission of the last response message. In the embodiment described, this decision is a comparison of the emergency level with the reaction threshold noted Fs.

If step 30 results in the decision that no correction is to be sent at this time, the method returns directly to step 20 of initializing FGi variables.

In the following step 22, the method detects that error messages are waiting for response and the next steps 24 to 28 are repeated. The evolution of certain factors leads to the calculation of a new emergency level UR which is compared again with the reaction threshold Fs. When the emergency level UR reaches the reaction threshold, it is decided to send a response message. The method of the invention then continues with a step 32 of determining one or more error messages.

For example, the losses P1 and P2 occurring between the bridge and the terminals T1 and T2 as shown with reference to FIGS. 4A and 4B are considered. FIG. 4A shows the loss P1 occurring between the bridge and the terminal T1, relative to the upper part of an image identified as the number 9 image of the stream F1 emitted by the terminal T1, the last correct image being the image 8. This loss P1 relates to 20 macroblocks out of the 100 that form the image. FIG. 4B shows the loss P2 occurring between the bridge and the terminal T2, relating to the lower zone of the image 11 of the flux F2, the last correct image being the image 10. This loss P2 relates to 30 macroblocks on the 100 that make up the image.

It will be assumed for the following that the reaction threshold Fs is set to 1 and that the factors λ are set to 1. The memory 10 has, in the example, a capacity of 10 images. The following table shows the evolution of UR over time. The indices 1 and 2 of the factors fq, ft and fstat as well as the level of gravity FG respectively correspond to the losses P1 and P2.

Thus, in this example, it is decided to send a response message during the transmission of the image 14, following the reception of losses P1 and P2.

Different solutions are possible for the implementation of step 32 of determining response messages. Advantageously, this step

The determination 32 includes a grouping 34 of the various error messages to reduce the number of response messages with respect to the number of error messages.

Advantageously, the various corrections are combined during a step 36, so as to provide different error messages with a combined response. An example of a correction image transmitted in a combined response message following the reception of losses P1 and P2 is shown with reference to FIG.

In this figure, the upper and lower parts corresponding to the lost areas are coded intra, that is to say without reference to the previous images while the central part is normally coded inter with reference to previous images. Thus, by replacing the image 14 in the output stream with this image 14 corresponding to the correction image, the conference bridge responds simultaneously to each of the error messages transmitted following the losses P1 and P2.

Alternatively, the error messages are analyzed in order to provide a correction common to all during a step 38. For example, when several successive error messages relate to the same image or part of image badly restored by several terminals a single response message is transmitted to all terminals. Furthermore, in the example, the grouping step 34 also includes a selection of error messages to which no response is provided. In particular, when error messages relate to images prior to the transmission of a fully intra-coded image, it is no longer necessary to respond to them. Alternatively, when several error messages relate to successive images, only the last error message will be selected to receive a response message.

Finally, the step 32 of determining a response message comprises a step 42 of formatting the response message, that is to say formatting of the correction image.

In the example, the four incoming streams are formatted in a step 44 to form the output stream. When an error message on an image of the output stream is to be answered, it is necessary to identify the correction images in the incoming streams. As shown with reference to FIG. 6, the loss P1 relates to both an image of the incoming flow F1 and an image of the incoming flow F2.

The conference bridge must then identify in the incoming flows F1 to F4, the correction images corresponding to these losses and reformat them to integrate them in the response message. The response message will then be a correction image in which correction images from incoming flows F1 and F2 are placed in their respective sectors.

Advantageously, the formatting step 42 also comprises the coding of the correction image with a view to its transmission during a step 46. In the case where the analysis of the error messages makes it possible to determine the last image correctly received by all the terminals, the correction image is encoded inter with respect to this image in order to obtain a better reproduction. In the example described with reference to FIG. 5, the correction image emitted in response to the losses P1 and P2 has a central portion encoded inter with respect to the image 8, the last image correctly received by all the terminals. Finally, the method comprises a step 48 of transmitting the correction image. As indicated above, this transmission can be made by a specific message or the correction image is sent instead of an image of the output stream. In the example, the correction image replaces the image 14 of the output stream.

Subsequently, the method returns to step 18 of initializing the variable UR and the method resumes for the next error messages.

It should be noted that, in the embodiment described, the use of the time factor for the calculation of the severity level and therefore the level of urgency, makes it possible to determine a maximum response time. Indeed, this factor increases with the emission of each new image so that, according to the weighting coefficient associated with it, it is possible to define a maximum time between the reception of an error message and the emission of an reply message. For example, this maximum time is selected equal to the length of the buffer of the conference bridge ^* 2 so that it has locally necessary corrections of the images. In the embodiment described, the value Fs corresponding to the reaction threshold of the bridge is assigned to the time factor when an error message relates to an image that is no longer in memory. The weighting coefficient associated with the time factor being equal to 1, as soon as such an error message is received, a response will be provided.

Of course, other embodiments are also possible.

In a first variant, the broadcast equipment includes a large memory and is adapted to decode the incoming video streams, memorize them in decoded form in its memory and then recode them to the terminals. The broadcast equipment thus becomes in a way the transmitter of the content from the point of view of the decoding terminals.

In such an embodiment, the possibilities of coding the correction image by the broadcasting equipment are very important since a large part of the preceding images is available. According to this variant of the invention, the method comprises a step of storing, by the broadcasting equipment, a decoded version of the images transmitted in the output stream and the images or parts of correction images are derived from the images. stored by broadcast equipment. Thus, when receiving several error messages, the analysis makes it possible to determine which is the last image transmitted without error for each of the terminals then, the conference bridge transmits a correction message by coding the current image relative to to the last image received without error by all the terminals. Thus, this single correction message will allow all terminals to resume the course of the video sequence.

As compared with the example described above, with the losses P1 and P2, such an embodiment allows the transmission of a single response message comprising the correction image 14 coded with respect to the image number 8 of the sequence, this image number 8 has been received correctly by all terminals.

In yet another variant, the broadcast equipment has only a limited memory and must send requests to each of the content transmitters to obtain the images or parts of images forming the correction messages.

Thus, the invention makes it possible to finely analyze the error messages on the return channel and to quantify their severity in order to decide on their processing. This quantization furthermore makes it possible to optimize the response messages by filtering the messages rendered useless by previous processing, by calculating a maximum reaction time according to a plurality of criteria for delaying the response to an error message, by prioritizing the sending of response messages, depending on the severity of the loss or the degradation of quality or other factors.

In this alternative, the method comprises a step of transmitting a request for correction images by the equipment of broadcasting to one or more image transmitters, for receiving back said correction images to be included in the one or more response messages. In this case, the broadcast equipment is a relay of the broadcast chain.

The invention also makes it possible to aggregate several return messages or else to put the message of return in shape according to the formation of the established transmission.

In a particular embodiment, said grouping comprises aggregating at least two corrections to be made to different error messages in order to form a combined response message. In a variant, said grouping comprises determining a correction common to at least two error messages in order to form a common response message.

Alternatively, said grouping comprises the selection of error messages to which no response will be made. More particularly, said analysis comprises an identification, in the various error messages, of images or parts of images that are not or badly rendered, and said step of determining response messages comprises the determination of a common response message for the different error messages related to the same images or parts of images. This embodiment makes it possible to issue only one response message when several error messages relating to the same image are issued by several reproduction equipment.

In addition, the invention can also use any combination of the variants described. Of course, the invention is not limited to a multicast system such as that described but can be implemented in other environments such as a point-to-point broadcast when several error messages on the return channel. are received successively from the same terminal or in any other type of video broadcast system. Furthermore, the invention can be implemented by means of a computer program as described or other means such as electronic boards, programmed components or the like.

Claims

A method of broadcasting images of a video sequence, the method comprising, at a broadcast equipment (2), the following steps:

- include (14) images in an output stream (φ) for transmission to at least one rendering equipment (T1-T4); receiving feedback from said reproduction equipment error messages (P1, P2) relating to the restitution of the images of the video sequence;

analyzing (24) the error messages to identify images or parts of images that are not or badly rendered; determining (32) response messages including images or portions of correction images, characterized in that the method further comprises:

a step (26) for determining a severity level (FGi) for each error message as a function of the analysis performed; and a decision step (30) of returning a response message to an error message according to the severity level of this error message.

2. Method according to claim 1, characterized in that said analysis (24) comprises the determination of at least one factor from the following group:

a level of quality of reproduction (fq) taking into account the error;

an evaluation of the importance of the content of the images or parts of images that are not or badly reproduced;

- decoding parameters;

- an evaluation (fstat) of the conditions of transmission; and

an estimate (ft) of the duration between the current image and the last image transmitted without error, said factor being used to determine the severity level associated with an error message.

3. Method according to any one of claims 1 to 2, characterized in that said level of severity of at least one error message is a maximum waiting time (Fs) before the transmission of a message of reply.

4. Method according to any one of claims 1 to 3, characterized in that it comprises the determination of a level of urgency (UR) from the severity levels associated with the error messages received since remission of the last response message and in that said decision step is performed according to this level of urgency.

5. Method according to any one of claims 1 to 4, characterized in that said step of determining response messages comprises a grouping (34) of response messages.

6. Method according to any one of claims 1 to 5, characterized in that said analysis comprises an identification of a last image transmitted without error and in that said step of determining a response message comprises coding (46). ) a correction image with respect to said last image transmitted without error.

7. Method according to any one of claims 1 to 6, characterized in that said step of including the images in a stream comprises formatting a plurality of incoming streams to form said output stream, in that said step of analyzing error messages include identifying images not or badly rendered in said incoming streams and that said step of determining a response message comprises the modifying (44) correction images from said incoming streams to conform to the format of the output stream.

A computer program for video footage broadcasting equipment, comprising software instructions for implementing the steps of a method according to any one of the preceding claims, when executed by a calculator of said computer.

9. Equipment for broadcasting (2) images of a video sequence, comprising:

means (6) for transmitting an output stream comprising images for transmission to at least one rendering equipment;

means (6) for receiving in return said reproduction equipment for the error messages relating to the restitution of the images of the video sequence;

means (10) for analyzing the error messages to identify images or parts of images that are not or badly rendered;

means (10) for determining response messages including images or parts of correction images, characterized in that it furthermore comprises:

means (10) for determining a severity level for each error message as a function of the analysis performed; and

decision means (10) for returning a response message to an error message as a function of the severity level of this error message.

10. Broadcasting equipment according to claim 9, characterized in that it comprises means adapted to implement a method according to any one of claims 1 to 7.

11. System for broadcasting images of a video sequence, comprising at least one image transmitter (T1-T4), broadcasting equipment (2) to at least one rendering equipment (T1-T4), characterized in said broadcast equipment is an equipment according to any one of claims 9 or 10.