US20050254499A1

US20050254499A1 - Buffer level signaling for rate adaptation in multimedia streaming

Info

Publication number: US20050254499A1
Application number: US10/901,015
Authority: US
Inventors: David Leon; Miska Hannuksela; Emre Aksu; Ye-Kui Wang
Original assignee: Nokia Oyj
Current assignee: Intellectual Ventures I LLC
Priority date: 2004-05-12
Filing date: 2004-07-28
Publication date: 2005-11-17
Also published as: TW200607280A; US7542435B2; TWI357243B; CN1981492B; US20050254427A1; MY138660A; CN1981492A

Abstract

In a multimedia streaming network where a client has a receiver buffer to store a plurality of packets received from a server so as to compensate for the difference between data transmission amount by the server and data usage amount by the client, the server should be able to adapt the data transmission rate based on the status of the receiver buffer. For rate adaptation purposes, server reconstructs a list of packets in the receiver buffer based on information provided by the client. The information signaled to the server is indicative of the next unit to be decoded and the sequence number of the packet to which the next unit to be decoded belongs.

Description

This is a Continuation-In-Part application of and claims priority to a co-pending U.S. patent application Ser. No. 10/844,062, filed May 12, 2004, assigned to the assignee of the present invention.

FIELD OF THE INVENTION

The present invention relates generally to multimedia streaming and, more particularly, to rate adaptation between a server and a client in multimedia streaming services.

BACKGROUND OF THE INVENTION

In a multimedia streaming service, there are three participants involved: a streaming server, a streaming client and a transmission channel or an underlying network. Usually it is the transmission channel that is the bottleneck of the service, both in terms of throughput and in terms of reliability (i.e., if no throughput bitrate guarantee is assumed), but throughput limitations can occur also at the client and/or at the server.
In a real-time streaming system, due to the dynamically changing throughput characteristics of the channel, client and server, the streaming delivery needs to be adaptive in order to maintain a real-time playback experience for the user. The server should adapt the transmission rate to the varying throughput of the system. An example of such a rate adaptation system can be found in Haskell et al. (U.S. Pat. No. 5,565,924, “Encoder/Decoder Buffer Control for Variable Channel”).
The streaming client provides receiver buffering for storing incoming data before passing them to the media decoder for playout. The receiver buffer is used to compensate for the difference between source encoding rate (also referred to as sampling rate) and transmission rate (pre-decoder buffering). It is also used to compensate for the packet transfer delay variation over the channel (jitter buffering). In general, these two functions are assumed to be combined in a single receiver buffer. However, they can also be implemented with two separate buffers in a receiver, although such an implementation is not optimum from a delay point of view. Receiver buffering can also smooth out the adaptation inaccuracies (i.e. if the system throughput is not matched exactly by the server output).
If the receiver buffer becomes empty (i.e. buffer underflow), which means that the decoder is running out of data to decode, the client needs to pause playout and re-buffer incoming data before resuming. On the other hand, if the incoming data rate is faster than the playout rate, then the receiver buffer space can be exhausted (i.e., buffer overflow), which can result in dropping packets from the buffer in order to make room for new incoming packets. When the packets are dropped, the video quality is degraded. To ensure a smooth and flawless playout, the receiver buffer of the client should be kept within a certain fullness range. In order to guarantee that the receiver buffer will not underflow or overflow, the bitrate for transmission and sampling at the server and that for reception and playout at the client must be adequately controlled.
3GPP rate adaptation signaling as defined in 3GPP TS 26.234 is based on feedback sent from the receiver to the sender in the form of an RTCP APP (Application-Defined Real Time Control Protocol) packet. This packet includes the sequence number (SN) of the oldest packet in the receiver buffer. This SN is referred to as OBSN (oldest buffered sequence number).
The signaling of the OBSN allows the sender to perform the necessary adaptation. Yet, if the decoding order and the display order are different, the sender may not be able to derive the status of the buffer and the purpose of the signaling would be defeated. With the PSS (Packet Switched Streaming Service) video codecs supported in Release 5, this is not a problem as their packet transmission order is equal to the decoding order.
In Release 6, H.264 (also known as MPEG-4 AVC) will be added to the list of the PSS codecs. With H.264, the transmission order and the decoding order could be different because of interleaved packetization at the payload level (as specified in the IETF H.264 RTP payload format draft).
The same property also exists for the frame-interleaved transmission of many audio and speech codecs, such as AMR-NB, AMR-WB, AMR-WB+, AAC and AACPlus (for the latter, the interleaving method defined in RFC 3640 is used).
The problem is hereafter illustrated assuming that the server transmits a series of packets whose RTP sequence numbers are denoted x, x+1, x+2, x+3, . . . . Let us further assume that each of these RTP packets carries two units. From the codec and the payload format in use, each of these units can be mapped to a decoding order y. The decoding order y is defined as follows: If a packet has a decoding order y, it is the y^thpacket to be decoded. That is, when the current packet has a decoding order y, it also means that (y−1) packets have already been decoded by the time the current packet is given to the decoder. For example, in the case of H.264, the DON (Decoding order number) defined by the H.264 RTP payload format can be used to derive the decoding order of the NAL units received in each packet

The following example illustrates the sequence of units where for each unit is given the sequence number of the packet to which it belongs, its unit number (i.e. whether it is the first or second unit in the packet) and its decoding order:



SN	Unit number	Decoding order

x	0	y
x	1	y + 1
x + 1	0	y + 2
x + 1	1	y + 3
x + 2	0	y + 4
x + 2	1	y + 5
x + 3	0	y + 6
x + 3	1	y + 7
x + 4	0	y + 100
x + 4	1	y + 8
x + 5	0	y + 9
x + 5	1	y + 10
x + 6	0	y + 11
x + 6	1	y + 12
. . .	. . .	. . .
x + 49	0	y + 97
x + 49	1	y + 98
x + 50	0	y + 99
x + 50	1	y + 101
x + 51	0	y + 102
x + 51	1	y + 103

In the above-given example, the decoding order is increasing by 1 for each unit received from packets with SN from x to x+3. However, this rule is broken for packet x+4. The first unit of packet x+4, for example, belongs to a frame that will be decoded only in the future.
Let us now look at the evolution of the receiver buffer and assume that, at a certain time, the receiver has received packets x, x+1, x+2, x+3. In this situation, the oldest sequence number in the buffer (OBSN) is x, and the highest received sequence number (HSN) signaled in RTCP RR reports is x+3. As time progresses, packet of SN x has been decoded and packet of SN x+4 has been received. Accordingly, the server will signal to the client OBSN=x+1 (the new “oldest” sequence number in the buffer) and HSN=x+4 (the new “most recent” SN received).
As time further progresses, the units of packets x+1, x+2 and x+3 have been played and x+5, x+6 and x+7 have been received, for example. At that point, the state of the buffer is x+4, x+5, x+6 and x+7. Accordingly, the client will signal to the server OBSN=4 and HSN=7. The problem arises around this time because after x+5, the decoding order number for the following packets: x+6, x+7, etc. is smaller than the decoding order number of the first unit of packet x+4. Accordingly, the current rate adaptation signaling OBSN will remain at x+4 until this unit is played and the packet x+50 is received, at which time the OBSN will be updated. The server will thus lose track of the receiver buffer status because OBSN is not updated according to the decoding and the removal of packets from the receiver buffer.
A further limitation stems from the fact that since multiple units can be sent in a single packet, when the receiver signals to the sender that the OBSN is x+2, for example, the sender cannot know determine whether the first unit of this packet is still in the buffer or whether only the second unit of the packet is in the buffer.
In sum, the prior art method of rate adaptation signaling is based on the oldest packet currently in the receiver playout buffer, allowing the sender to estimate both the number of bytes in the receiver buffer and the duration of the playout buffer. This information is used by the sender to perform adaptation so as to avoid receiver underflow (playout interruption) or receiver overflow (packet loss). However, because the decoding order and the transmission order are not the same in some occasions and that multiple units may be in the same packet, the sender may lose track of the receiver buffer.
A typical RTP packet is shown in FIG. 1. The RTP packet includes a multi-time aggregation packet of type MTAP16 and two multi-time aggregation units. The RTP Header in the first row of the packet is shown in FIG. 2. As shown in FIG. 2, the sequence number (SN) of the packet is shown in the first row of the RTP header. As shown in FIG. 1, the aggregation type packet aggregates multiple Network Abstraction Layer (NAL) units into a single RTP payload. In particular, in MTAP16s, the NAL unit payload consists of a 16-bit unsigned decoding number order (DON) base, or DONB (see second row of the packet). DONB contains the value of DON of the first NAL unit, so that the value of DON of all other NALs can be expressed in DOND, or the difference between the value of DON in a certain NAL and DONB.
The RTP payload format for H.264 codec can be found in the IETF Audio Visual Transport Working Group Internet Draft draft-ietf-avt-rtp-h264-05 (April 2004).

SUMMARY OF THE INVENTION

The present invention provides a mechanism for the server device in a multimedia streaming network, which sends streaming data packets to a client device to playout, to reconstruct a list of data packets stored in the receiver buffer in the client device. Based on the reconstruction, the server device adjusts the streaming data amount provided to the client device, so as to control the level of the receiver buffer.
The first aspect of the present invention provides a method for controlling level of a receiver buffer in a client in a multimedia streaming network, the streaming network comprising a server for providing streaming data in a plurality of packets to the client, wherein at least some of the data packets are stored in the receiver buffer so as to compensate for difference between data transmission amount by the server and the data usage amount by the client, and wherein the packets are decoded in a decoding order based on a plurality of decoding order values associated with a playout order the client. The method comprises:

- determining in the client the next packet to be decoded among the packets in the receiver buffer based on the decoding order values; and
- signaling to the server information indicative of said next packet to be decoded, so as to allow the client to adjust the streaming data amount provided to the client based on the information.

According to the present invention, the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs. Each of the units has a unit number and each of the data packets has a sequence number known to both the client and the server, and the information signaled to the server is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.
According to the present invention, the server maintains a list of units that have been sent and a record of unit number and the sequence number of the packet to which the sent units belong and a mapping between said sequence number and unit number to the decoding order for determining the data units in the receiver buffer based on said mapping so as to adjust the streaming data amount provided to the client based on said determination in the server.
According to the present invention, the information signaled to the server is further indicative of a difference between a scheduled playout time of said next unit to be decoded and the decoding time of said next unit.
According to the present invention, the information signaled to the server is further indicative of the highest sequence number received by the client so as to allow the server to determine the data packets in the receiver buffer.
According to the present invention, each of the units has a timestamp and each of the data packets has a sequence number known to both the client and the server, and the information signaled to the server is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.
The second aspect of the present invention provides a multimedia streaming network comprising:

- at least a client; and
- a server for providing streaming data in a plurality of packets to the client, wherein the client comprises:
  - a receiver buffer for storing at least some of the data packets to be decoded so as to compensate for difference between data transmission amount by the server and data usage amount by the client, and wherein the packets are decoded in a decoding order based on a plurality of decoding values associated with a playout order in the client, and
  - a mechanism for signaling to the server information indicative of the next packet to be decoded among the packets in the buffer based on the decoding order values so as to allow the server to adjust the rate of streaming data provided to the client.

According to the present invention, the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs. Each of the units has a unit number and each of the data packets has a sequence number known to both the client and the server, and the information signaled to the server is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.
According to the present invention, the server maintains a list of units that have been sent and a record of unit number and the sequence number of the packet to which the sent units belong and a mapping between said sequence number and unit number to the decoding order for determining the data units in the receiver buffer based on said mapping so as to adjust the streaming data amount provided to the client based on said determination in the server.
The third aspect of the present invention provides a client device in a multimedia streaming network, the streaming network comprising a server device for providing streaming data in a plurality of packets to the client device, wherein the packets are decoded in a decoding order based on a plurality of decoding values associated with a playout order in the client device. The client device comprises:

- a receiver buffer for storing at least some of the data packets to be decoded so as to compensate for difference between data transmission amount by the server device and the data usage amount in the client device; and
- a mechanism for signaling to the server device information indicative of the next packet to be decoded among the packets in the receiver buffer based on the decoding order values so as to allow the server device to adjust the streaming data amount provided to the client device.

According to the present invention, the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs. Each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.
According to the present invention, the client device further comprises a software program having executable codes for determining:

- the decoding order of the data packets in the receiver buffer based on the decoder order values, and
- the next packet to be decoded among the data packets in the receiver buffer based on the decoding order values.

The fourth aspect of the present invention provides a server device for providing streaming data in a multimedia streaming network, the multimedia streaming network comprising at least a client device for receiving the streaming data in a plurality of data packets and decoding the data packets in a decoding order based on a plurality of decoding order values associated with a playout order, wherein the client device has a receiver buffer for storing at least some of the data packets so as to compensate for difference between data transmission amount by the server device and the data usage amount by the client device. The server device comprises:

- a mechanism for receiving information from the client device indicative of the next packet to be decoded among the packets in the receiver buffer based on the decoding order values in the client device; and
- a software program for determining the packets in the receiver buffer based on the information so as to adjust the streaming data amount provided to the client device for controlling level of the receiver buffer.

According to the present invention, the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs. Each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.
The fifth aspect of the present invention further provides a software product embedded in a computer readable media for use in a client device in a multimedia streaming network, the streaming network comprising a server device for providing streaming data in a plurality of packets to the client device, wherein the packets are decoded in a decoding order based on a plurality of decoding values associated with a playout order in the client device, and wherein the client device comprises a receiver buffer for storing at least some of the data packets to be decoded so as to compensate for difference between data transmission. The software product comprises:

- a code for determining the decoding order of the data packets in the receiver buffer based on the decoding order values; and
- a code for determining the next packet to be decoded among the data packets in the receiver buffer based on the decoding order values, so as to provide to the server device information indicative of said next packet to be decoded, allowing the server device to adjust the streaming data amount provided to the client device based on the information for controlling level of receiver buffer.

According to the present invention, the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs. Each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.
The sixth aspect of the present invention provides a software product embedded in a computer readable media for use in a server device providing streaming data in the multimedia streaming network, the multimedia network comprising at least a client device for receiving the streaming data in a plurality of data packets and decoding the data packets in a decoding order based on a plurality of decoding order values associated with a playout order, wherein the client device has a receiver buffer for storing at least some of the data packets so as to compensate for difference between the transmission amount by the server device and the data usage amount by the client device. The software product comprises:

- a code for relating the decoding order to the sequence numbers of the data packets having been sent to the client device; and
- a code for determining the data packets in the receiver buffer based on said relating and on information provided by the client device indicative of the next packet to be decoded in the client device, so as to allow the server device to adjust the streaming data amount provided to the client device for controlling level of the receiver buffer.

BRIEF DESCRIPTION OF THE DRAWING

FIG. 1 is a block diagram showing a typical RTP packet.
FIG. 2 is a block diagram showing a typical RTP header.
FIG. 3 is a block diagram showing a multimedia streaming system having a server device and a client device that can perform the rate adaptation method, according to the present invention.

DETAILED DESCRIPTION OF THE INVENTION

The present invention provides a method of buffer level signaling so as to allow the server in a multimedia streaming network to perform rate adaptation in codecs such as H.264, for which packet transmission order is different from decoding order. The buffer level signaling method, according to the present invention, is based on information about the next unit to be decoded. The sequence number (SN) of the packet to which the next unit to be decoded belongs is herein referred to as NDSN.
In the method of buffer level signaling, according to the present invention, the receiver reports to the sender information of the next unit to be decoded. This unit is an identifiable decoding unit that can be defined in an RTP payload format. More particularly, the identifiable unit can be identified by the NDSN and the unit number (NDU) within the packet. In the case of an audio codec, the identifiable decoding unit is typically a frame. In the case of the H.264 codec, the identifiable decoding unit is a NAL (Network Abstraction Layer) unit. For example, if the next NAL unit to be passed to the decoder is the third NAL unit found in packet with SN=100, the receiver signals the values 100 and 2 to the sender.
The receiver reports to the sender information of the next unit to be decoded based on the smallest y value. Based on the received NDSN and the NDU, the server can identify the unit to be decoded next. As such, the sender can derive the correct status of the receiver buffer. Using the set of x and y values given in the background section, this implementation can be illustrated as follows:

When packets x+4, x+5, x+6 and x+7 have been received, the decoding orders of the units that were carried in these packets are y+100, y+8, y+9, y+10, y+11, y+12, y+13 and y+14. Thus, the second unit of packet x+4 has the smallest decoding order value of y+8. Accordingly, the next unit to be decoded can be signaled unambiguously to the sender through the SN of the packet to which it belongs (x+4) and the corresponding unit number (NDU=1). The receiver also sets HSN=x+7 as always because this is the latest received sequence number. The above-discussed situation is shown in TABLE I:

TABLE I


Decoding	Packets in
Order value	receiver buffer	NDSN	HSN	NDU

. . .	. . .	. . .	. . .	.
. . .	. . .	. . .	. . .	.
. . .	. . .	. . .	. . .	.
y + 5	x + 2, x + 3, x + 4	x + 2	x + 4	1
y + 6	x + 3, x + 4, x + 5	x + 3	x + 5	0
y + 7	x + 3, x + 4, x + 5	x + 3	x + 5	1
y + 8	x + 4, x + 5, x + 6	x + 4	x + 6	1
y + 9	x + 4, x + 5, x + 6	x + 5	x + 6	0
y + 10	x + 4, x + 5, x + 6, x + 7	x + 5	x + 7	1
y + 11	x + 4, x + 6, x + 7, x + 8	x + 6	x + 8	0
. . .	. . .	. . .	. . .	.
. . .	. . .	. . .	. . .	.
. . .	. . .	. . .	. . .	.

Note:
when decoding order value = y + 6, NDSN = x + 3, and NDU = 0; when decoding order value = y + 7, NDSN = x + 3 and NDU = 1.
Note:
Some data units of the packets in the receiver buffer may have been decoded and removed from the buffer. However, each packet in the buffer should have at least one data unit remaining in the buffer.

Based on the received NDSN and the NDU, the sender is able to get an accurate estimation of which packets are in the receiver buffer, and for each packet in the receiver buffer, which data units are in the receiver buffer.
At the receiver side:

- Compute the decoding order (y values) of the coded units that are currently in the buffer;
- Find the next unit to be decoded (unit with the smallest y value); and
- Signal the unit number of the next unit to be decoded (NDU) and the sequence number of the next packet to which this unit belongs (call this NDSN).

At the sender side:

- Keep a list of units (L) that have been sent. For each unit keep a record of the sequence number (SN) of the packet in which the unit was sent, its unit number indicating the order of the unit in the packet, and the mapping of the unit number and the sequence number to the decoding order number (i.e. its y value);
- Look up the NDU, NDSN and the HSN signalled by the receiver in RTCP RR or SR report;
- Reconstruct the list of units in the receiver buffer, by looking up in L all the units whose recorded information conforms to:
  - 1) the decoding number in the list is higher than the decoding order that maps to the signalled NDSN and NDU, and
  - 2) the sequence number is smaller than the highest sequence number received (HSN).

It should be noted that wrap around should be taken into consideration when comparing the decoding number and the sequence number. For example, since the RTP sequence number is represented using 16 bits in the signaling, the maximum value that has been signaled is 65535. Then the next value of 0 actually represents a sequence number of 65536.
Alternatively, a special value may also be used for the NDU field (e.g. 0xFFFF if the field is two bytes) in order to signal that no unit information is provided. In this case, only the sequence number of the packet the unit belongs to is supplied. The server will not be able to estimate as accurately which units are currently in the receiver buffer.
In order to illustrate how the method of buffer level signaling for rate adaptation is carried out, a multimedia streaming system is shown in FIG. 3. As shown, the multimedia streaming system 1 has means for buffer level signaling from a streaming client 60 to a streaming server 10.
The streaming server 10 comprises an application level signaling engine 20, a rate controller 30 and a server buffer 40. The streaming client 60 comprises an application level signaling engine 70, corresponding to, and adapted to communicate with, the application level signaling engine 20 in the streaming server 10. It further comprises a client buffer 80 which, in the embodiment of the invention illustrated in FIG. 3, comprises a jitter buffer 82 and a pre-decoding buffer 84, integrated as a single unit. In other embodiments of the invention, streaming client 60 may include a jitter buffer and a pre-decoding buffer that are implemented separately. The streaming client further comprises a media decoder 90, a post-decoder buffer 100, a buffer controller 110 and a display/play-out device 120.
The system depicted in FIG. 3 is further shown to comprise a “channel buffer” 50 located between streaming server 10 and streaming client 60, representing the varying transfer delays that occur during transmission of data packets from the streaming server to the client.
The server's rate controller 30 is operative to adapt the rate at which media data is transmitted from the streaming server. The server also has a transmission clock 32 to timestamp the packets to be transmitted to the client. It operates by adjusting the transmitted data rate in accordance with the varying bit-rates on the transmission channel, taking into account the client's request for a transmission time-shift, thereby seeking to avoid pauses in play-back at the client due to pre-decoder buffer underflow or dropping packets at the client due to buffer overflow.
Server buffer 40 stores data packets temporarily before they are transmitted from the streaming server across the transmission channel to streaming client 60. In a “live” streaming scenario where data packets are sampled real-time, the server buffer is indeed a physical buffer where data packets are placed at sampling time and are extracted at transmission time. In a “pre-encoded” streaming scenario, where data packets are not sampled real-time but are stored in a pre-encoded file and are read from the file at transmission time, the server buffer is a virtual buffer that represents the difference between sampling time (with reference to a sampling clock started at the streaming server when the first data packet of the pre-encoded file is transmitted) and transmission time of data packets.
At the streaming client, media data is received from the transmission channel and buffered in client buffer 80. The parameters of pre-decoder buffer 84 and jitter buffer 82 are set by the buffer controller 110. The parameters are chosen as an aggregate of the server recommended pre-decoder buffering parameters and the additional buffering required as estimated by the client. The client estimates what is needed to tolerate the expected packet transfer delay variation (i.e. jitter) on the available transmission channel. Such aggregate is constrained by the maximum buffering capabilities of the client. Media decoder 90 extracts media data from the client buffer and decodes the media data in a manner appropriate for media type in question. It should be appreciated that the media data will, in general, comprise a number of different media types. For example, if the media data transmitted from the server is representative of a video sequence, it is likely to comprise at least an audio component in addition to video data. It should therefore be understood that media decoder 90, as illustrated in FIG. 1, may actually comprise more than one decoder, for example a video decoder implemented according to a particular video coding standard and an associated audio decoder. As the media data is decoded by media decoder 90, it is output to post-decoder buffer 100 where it is stored temporarily until its scheduled play-out time, at which point it is passed from the post-decoder buffer to display/play-out device 120 under the control of buffer controller 110.
According to the present invention, buffer controller 110 is adapted to provide to the application level signaling engine 70 an indication of the next unit to be decoded. The application level signaling engine is, in turn, adapted to transmit to the streaming server information about the next unit to be decoded, as denoted by reference numeral 300 in FIG. 3. The transmitted information indicates the unit number of the next unit to be decoded (referred to as NDU) and the sequence number of the packet to which this unit belongs (referred as to NDSN). In the case of an H.264 payload, the unit is a NAL. As shown in FIG. 1, the streaming client 60 has a software program 112 including codes for computing the decoding order (y values) of the coded units that currently exist in the buffer 80 and determining the next unit to be decoded based on the smallest y value. Based on this finding, the application level signaling engine 70 can signal the unit number of the next unit and the sequence number of the packet to which this next unit belongs to the streaming server 10.
At the server 10, a software program 36 is used to determine the list of units in the receiver buffer from the unit SN list 34 and the information provided by the client 60.
It should be noted that, in the embodiment described above, the receiver signals the SN of the packet to which the next unit to be decoded belongs and the unit number. Alternatively, the receiver may signal a timestamp of the next unit to be decoded instead of its unit number. As long as each unit, in the same packet has a different timestamp, the sender is able to identify unambiguously this unit from the signaled timestamp and sequence number. However, it is possible that some units have the same timestamp in the same packet. In that case, the signaling will not allow the sender to derive the status of the receiver buffer status as accurately as the signaling of a unit number, with the maximum estimation error being the data amount of a frame.
Although the invention has been described with respect to one or more embodiments thereof, it will be understood by those skilled in the art that the foregoing and various other changes, omissions and deviations in the form and detail thereof may be made without departing from the scope of this invention.

Claims

1. A method for controlling level of a receiver buffer in a client in a multimedia streaming network, the streaming network comprising a server for providing streaming data in a plurality of packets to the client, wherein at least some of the data packets are stored in the receiver buffer so as to compensate for difference between data transmission amount by the server and the data usage amount by the client, and wherein the packets are decoded in a decoding order based on a plurality of decoding order values associated with a playout order to the client, said method comprising:

determining in the client the next packet to be decoded among the packets in the receiver buffer based on the decoding order values; and

signaling to the server information indicative of said next packet to be decoded, so as to allow the client to adjust the streaming data amount provided to the client based on the information.

2. The method of claim 1, wherein the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs.

3. The method of claim 2, wherein each of the data packets has a sequence number known to both the client and the server, and wherein the information signaled to the server is indicative of the sequence number of next packet to be decoded.

4. The method of claim 2, wherein each of the units has a unit number and each of the data packets has a sequence number known to both the client and the server, and wherein the information signaled to the server is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

5. The method of claim 4, wherein the server maintains a list of units that have been sent and a record of unit number and the sequence number of the packet to which the sent units belong and a mapping between said sequence number and unit number to the decoding order for determining the data units in the receiver buffer based on said mapping so as to adjust the streaming data amount provided to the client based on said determination in the server.

6. The method of claim 4, wherein the information signaled to the server is further indicative of a difference between a scheduled playout time of said next unit to be decoded and the decoding time of said next unit.

7. The method of claim 6, wherein the information signaled to the server is further indicative of the highest sequence number received by the client so as to allow the server to determine the data packets in the receiver buffer.

8. The method of claim 2, wherein each of the units has a timestamp and each of the data packets has a sequence number known to both the client and the server, and wherein the information signaled to the server is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

9. A multimedia streaming network comprising:

at least a client; and

a server for providing streaming data in a plurality of packets to the client, wherein the client comprises:

a receiver buffer for storing at least some of the data packets to be decoded so as to compensate for difference between data transmission amount by the server and data usage amount by the client, and wherein the packets are decoded in a decoding order based on a plurality of decoding values associated with a playout order in the client, and

a mechanism for signaling to the server information indicative of the next packet to be decoded among the packets in the buffer based on the decoding order values so as to allow the server to adjust the rate of streaming data provided to the client.

10. The streaming network of claim 9, wherein the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs.

11. The streaming network of claim 10, wherein each of the data packets has a sequence number known to both the client and the server, and wherein the information signaled to the server is indicative of the sequence number of next packet to be decoded.

12. The streaming network of claim 10, wherein each of the units has a unit number and each of the data packets has a sequence number known to both the client and the server, and wherein the information signaled to the server is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

13. The streaming network of claim 12, wherein the server maintains a list of units that have been sent and a record of unit number and the sequence number of the packet to which the sent units belong and a mapping between said sequence number and unit number to the decoding order for determining the data units in the receiver buffer based on said mapping so as to adjust the streaming data amount provided to the client based on said determination in the server.

14. The streaming network of claim 12, wherein the information signaled to the server is further indicative of a difference between a scheduled playout time of said next unit to be decoded and the decoding time of said next unit.

15. The streaming network of claim 12, wherein the information signaled to the server is further indicative of the highest sequence number received by the client so as to allow the server to determine the data packets in the receiver buffer.

16. The streaming network of claim 10, wherein each of the units has a timestamp and each of the data packets has a sequence number known to both the client and the server, and wherein the information signaled to the server is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

17. A client device in a multimedia streaming network, the streaming network comprising a server device for providing streaming data in a plurality of packets to the client device, wherein the packets are decoded in a decoding order based on a plurality of decoding values associated with a playout order in the client device, said client device comprising:

a receiver buffer for storing at least some of the data packets to be decoded so as to compensate for difference between data transmission amount by the server device and the data usage amount in the client device; and

a mechanism for signaling to the server device information indicative of the next packet to be decoded among the packets in the receiver buffer based on the decoding order values so as to allow the server device to adjust the streaming data amount provided to the client device.

18. The client device of claim 17, wherein the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs.

19. The client device of claim 18, wherein each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

20. The client device of claim 18, wherein each of the units has a timestamp and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

21. The client device of claim 17, further comprising

a software program having executable codes for determining:

the decoding order of the data packets in the receiver buffer based on the decoder order values, and

the next packet to be decoded among the data packets in the receiver buffer based on the decoding order values.

22. A server device for providing streaming data in a multimedia streaming network, the multimedia streaming network comprising at least a client device for receiving the streaming data in a plurality of data packets and decoding the data packets in a decoding order based on a plurality of decoding order values associated with a playout order, wherein the client device has a receiver buffer for storing at least some of the data packets so as to compensate for difference between data transmission amount by the server device and the data usage amount by the client device, said server device comprising:

a mechanism for receiving information from the client device indicative of the next packet to be decoded among the packets in the receiver buffer based on the decoding order values in the client device; and

a software program for determining the packets in the receiver buffer based on the information so as to adjust the streaming data amount provided to the client device for controlling level of the receiver buffer.

23. The server device of claim 22, wherein the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs.

24. The server device of claim 23, wherein each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

25. The server device of claim 23, wherein each of the units has a timestamp and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

26. A software product embedded in a computer readable media for use in a client device in a multimedia streaming network, the streaming network comprising a server device for providing streaming data in a plurality of packets to the client device, wherein the packets are decoded in a decoding order based on a plurality of decoding values associated with a playout order in the client device, and wherein the client device comprises a receiver buffer for storing at least some of the data packets to be decoded so as to compensate for difference between data transmission, the software product comprising:

a code for determining the decoding order of the data packets in the receiver buffer based on the decoding order values; and

a code for determining the next packet to be decoded among the data packets in the receiver buffer based on the decoding order values, so as to provide to the server device information indicative of said next packet to be decoded, allowing the server device to adjust the streaming data amount provided to the client device based on the information for controlling level of receiver buffer.

27. The software product of claim 26, wherein the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs.

28. The software product of claim 27, wherein each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

29. The software product of claim 27, wherein each of the units has a timestamp and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signal to the server device is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

30. A software product embedded in a computer readable media for use in a server device providing streaming data in the multimedia streaming network, the multimedia network comprising at least a client device for receiving the streaming data in a plurality of data packets and decoding the data packets in a decoding order based on a plurality of decoding order values associated with a playout order, wherein the client device has a receiver buffer for storing at least some of the data packets so as to compensate for difference between the transmission amount by the server device and the data usage amount by the client device, said software product comprising:

a code for relating the decoding order to the sequence numbers of the data packets having been sent to the client device; and

a code for determining the data packets in the receiver buffer based on said relating and information provided by the client device indicative of the next packet to be decoded in the client device, so as to allow the server device to adjust the streaming data amount provided to the client device for controlling level of the receiver buffer.

31. The software product of claim 30, wherein the packets are associated with a plurality of units including the next unit to be decoded, and wherein the next packet to be decoded is the packet to which the next unit to be decoded belongs.

32. The software product of claim 30, wherein each of the units has a unit number and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the unit number of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.

32. The software product of claim 30, wherein each of the units has a timestamp and each of the data packets has a sequence number known to both the client device and the server device, and wherein the information signaled to the server device is indicative of the timestamp of said next unit to be decoded and the sequence number of the packet to which said next unit belongs.