DE102005012668B4 - Frame error detection and correction method for digital video - Google Patents

Abstract

Method for detecting missing frames in audio and video data streams that are bound to a system stream for transmission or storage and are in a format of the MPEG protocol family or related formats, as well as for repairing one damaged by missing frames and made from video and audio data streams bound data stream, characterized in that for the detection of missing frames a) the same playback duration for all frames of a data stream of the frames of the data streams to be examined is calculated from the control parameters of the respective data stream, including the playback time in the field "FRAME RATE ”In the“ SEQUENCE HEADER ”of the associated video data stream encoded frame rate is calculated, and for which purpose the playback time for audio frames is calculated using the frame rate permanently assigned to the audio coding format of the data stream rt of the "STREAM-ID" field and, where necessary, the value of the "SUBSTREAM-ID" field in the "PACKETIZED ELEMENTARY STREAM" is within ...

Description

This invention relates to a method of real-time or retrospective detection of portions in which portions of these sequences have been lost in the digital recording or transmission of video and / or audio sequences using digital compressed video system streams consisting of interlinked motion picture and sound tracks are. This invention further relates to a method for improving such digital video system streams in which lost picture or / and sound sequences are detected.

The digital recording of video data including the associated audio data generates large amounts of data. The mass storage devices available for their storage for consumer electronics products (eg hard disk or optical media such as DVDs) usually have neither sufficient access speed nor the necessary storage density to allow uncompressed storage of digital video data and, if necessary, allow the associated audio data at a sufficient resolution. Therefore, digitally recorded video data must be compressed before being stored or transmitted. This compression of both video and audio data is called encoding. Internationally standardized for this purpose were the protocols and formats of the Motion Pictures Expert Group (MPEG, eg MPEG 1, MPEG 2, or MPEG 4). Industry standards derived from these protocols with only minor changes are e.g. DivX (based on MPEG 4) or the DVD Whitebook (standard for DVD's based on MPEG 2). Larger differences exist z. To Quicktime and MJPEG, although these formats are partly based on the MPEG standards or implemented as part of the product improvement properties of the MPEG standards. The names of the said standards or industry standards are partially registered trademarks.

The core of the video data formats is the compression method. In addition, the video data formats also determine how the video and audio data associated with a video are interconnected (multiplexed) in a file (for storage) or in a data stream (for transmission), and define additional control information.

In the MPEG formats and most other digital video formats, the video and audio data are first separated from each other as their own video and audio data streams digitized or digitally recorded (or at least cached in hardware encoders) and encoded. When encoding, the video and audio streams are subdivided into separate sections called frames. A video frame is usually an image of the movie that was recorded as video, while an audio frame is a fixed length sound sequence. The duration of the audio sequence contained in an audio frame differs depending on the audio format used, but it usually ranges in the tens of milliseconds.

Within the frame, the audio and video raw data is contained in compressed form. If the compression was done at a constant data rate, the frame has a fixed length, which can be calculated from the data rate contained in its header. If the compression has been done with variable data rate, the frame has a variable length. In this case, the detection of the start of a frame can be done with the data contained in the header. For example, the in 2 represented Picture Header, which is the header of a video frame, with a 24-bit synchronization bit sequence (0x000001), followed by the identification byte of the Picture Header (0x00). The MPEG format ensures that no bit sequence similar to the synchronization bit string can occur within the data stream. This ensures that if one searches within the video data stream for a synchronization bit sequence, one finds the beginning of a header. The headers of the audio streams also use a sync bit string, but this is different depending on the audio data format. Thus, in the Dolby Digital audio data format, the synchronization bit string is 16 bits long and has the value 0x0B77.

Before the digital video is stored on a mass storage device, broadcast terrestrially or by satellite, or via a transmission medium such. As the Internet or a network is transmitted, the frames of the audio and video streams must be tied back together. This binding of data streams is called multiplexing. When multiplexing additional control information, such. B. the playback time of a frame and added to the data stream. The result of the multiplexing is a so-called System Stream, which is also referred to here as bound digital video, bound data stream or bound video.

The method used for multiplexing operates in principle in the same way in the MPEG formats or the formats derived therefrom, regardless of which transmission or storage medium the video is multiplexed for. However, depending on which transmission or storage medium the video is multiplexed for, the individual parameters of this method may be large Differences occur. So z. For example, in DVD-Video, only one video stream is allowed, while in the DVB-T format used for the terrestrial transmission of digital television, four video streams are typically multiplexed.

In multiplexing, the individual video and audio streams that make up the system stream are first split into packets of the same size. The size of the packets may be selected from certain allowable values depending on the transmission or storage medium. Each packet is preceded by a so-called "packet header", followed by a packetized elementary stream header (PES header). Behind these two headers, data from the video or audio stream is inserted until the packet is full. Then a new package begins. In order to limit the hardware requirements of the device used for digital video output, the packets of the various data streams to be merged are alternately written in a ratio corresponding to their data rate in the multiplexed data stream. As a result, the multiplexing z. B. in hardware encoder encoders in one step with the encoding and done in real time. For the recognition of the headers generated in the multiplexing, the synchronization bit sequence 0x000001 known from the video data streams is used.

The packet header is followed by the identification byte of the packet header (0xBA) on the 24-bit synchronization bit string (0x000001). The packet header ( 3 ) contains among other things the "SYSTEM CLOCK REFERENCE" (SCR). If a player evaluates this field, it will not start playing the first frame in that packet until a 27 MHz counter started at the start of the video has reached the value contained in the SCR. The SCR is a 48-bit field that follows just after the identification byte of the pack header. The first byte of the SCR field is a 0, further bits of the SCR field are pre-assigned a 1 to ensure that no second sync bit sequence can occur within the header. The value of the 27 MHz counter is divided by 300 and written to the front of the SCR, the remainder resulting from the division being written as a 9-bit number at the end of the SCR. The structure of the packet header can 3 be removed.

The Pack Header is followed by a Packetetized Elementary Stream Header (PES header, 6 ). This starts again with the 24-bit synchronization bit sequence (0x000001). This is followed by the 8-bit stream ID. Permitted stream IDs are the values 0xBD to 0xEF. Each stream contained in the bound stream is assigned a unique stream ID at which it can be identified. The stream ID also determines what the data stream is used for. In this case, the stream IDs 0xE0 to 0xEF for video streams and the streams 0xC0 to 0xDF for audio streams are reserved in one of the original audio formats of the MPEG protocol family. Furthermore, a padding data stream (Stream-ID 0xBE) is allowed for inserting stuffing data and two private data streams (Stream-ID's 0xBD and 0xBF). For example, DVDs use Private Stream 0xBD to wrap audio data in DTS (Digital Theater Sound), AC3 (Dolby Digital), and PCM audio formats not supported by the original MPEG standard.

The Packetized Elementary Stream Header allows a number of fields. Which of them actually exists is decided by the bits in a 32-bit bit field called "PES header extension". This bit field directly follows the stream ID.

For the understanding of the method according to the invention only the field "PRESENTATION TIME STAMP" (PTS) of the PES header is important. It is present if the 7th bit of the "PES header extension" has the value 1. In this case, the PTS follows directly on the PES header extension. The PTS indicates the time in milliseconds at which, as measured by the start of the video, a player should begin playing the first frame in the packet that precedes this PES header.

The structure of the PES header can, as far as it is required for the inventive method, 6 be removed.

The invention is based on the following problem:
The MPEG formats are consistent in that the video and audio data streams can be in the same form on any possible transmission or storage medium, as long as one does not change to another format within that format family. On the other hand, system flows, depending on the medium for which they were created, often differ significantly in terms of the information added in the binding.

Take, for example, a digital video with a TV tuner card and a PC, so a program stream is generated. In order to burn this recording to DVD for later archival, it has to be re-bound at least because the format (VOB - Video OBject) used for DVD's for the system stream differs from that of the program stream. In contrast, the video and audio streams tied in the various system streams are compatible with each other. So theoretically you do not have to encode them anymore, at least, if you already have the Settings for recording certain boundary conditions (eg DVD's support only a few resolutions). This is particularly advantageous because the MPEG formats use lossy compression techniques so that quality loss occurs each time the video and audio data streams are re-encoded.

Especially in the MPEG-2 format, there are a whole range of applications in which such format conversion of the system stream by re-multiplexing is required, and in which the video and audio data streams do not necessarily need to be re-encoded. Since the MPEG-2 format is used very frequently, it has a corresponding number of derivatives. Another example of the need for such format conversion is to archive a TV program broadcast digitally in DVB-S or DVB-T format on a DVD. If you have a video that you z. B. as a transport stream on a DVB receiver with built-in hard disk has to burn to DVD, is due to the additional navigation data, which requires the DVD in Private Stream 2 (Stream ID 0xBF), basically a new multiplexing as well generating additional control information required.

If the video is recorded on the PC (eg with an analog TV card) and further processed there, the encoding, cutting and, if it is to be burned to DVD, the menus and the burning often with different programs. Each of these programs internally demultiplexes the system stream and multiplexes it again after further processing. Some of these programs can not even handle system streams tied from video and audio streams. In this case, the video must be demultiplexed before further processing and be multiplexed again for subsequent storage. The only program known to the author that deviates from this is a version of a video editing program that relies on an external hardware encoder.

As long as there are no frames missing from the audio and video streams included in the source video, demultiplexing followed by retransmissions is easy. However, if frames are missing in one of the streams, then after demultiplexing followed by retransmission, audio and video are no longer synchronous for a portion of the video. Already a difference of less than half a second is very disturbing.

This is because all information about when to start playing a frame is only included in the packet headers and PES headers of the multiplexed video. If a video is played with missing frames, the player or player waits until the time for the next frame to be played back is reached. Until then, the picture will stop short, or the sound will be muted for a short time.

In demultiplexing, these timestamps or synchronization information are removed so that they can no longer be properly formed upon re-multiplexing. Instead, the multiplexer calculates the timestamps from the number of frames it counts and some other parameters, such as the number of frames. B. the frame rate. This will cause the video to become out of sync after retuning if frames are lost.

Unfortunately, the lack of frames is very common. The following are some possible causes for loss of frames. This problem is aggravated by the fact that in a video the video data requires considerably more bandwidth than audio data, so that the likelihood of losing video frames is much higher than that for the loss of audio frames. If there is a loss of frames, this effect is exacerbated by the fact that video and audio frames are usually of different lengths.

Real-time compression into digital video in MPEG format is required when digitizing video from an analog source (analog VCR, analog cable TV, analog terrestrial television). Compressing videos in real time requires extra computing power. Moreover, the computation time required to compress an image sequence of the video is highly dependent on the image content, so that it can not be ruled out even with the most powerful processors available that individual images can not be compressed before the next image is received , In this case, frames, usually video frames, can usually be discarded.

If an analog television signal is received via terrestrial antenna or satellite dish, interference, such. B. slightly delayed reflections on the ground or on obstacles, interference signals, and weather, for which especially the satellite reception is sensitive affect the image quality. In two widespread TV decoder analog-to-digital converter chips under study by the inventor, clock and synchronization information (eg, start-up generation of the next frame at the blanking interval) is generated from the video input signal. This is also necessary because using a separate time base already with minimal deviation from the Time base of the television signal is a continuous picture. But it also leads to that, because z. B. the blanking interval is not detected clean, frames may be missing in the output data stream generated by the digitization, or frames are discarded because no clock could be formed for the sampling of the signal. This behavior can be understood from the datasheets of these circuits (eg Conextant Bt829 and Philips SAA7113) and has been confirmed by the inventor's tests with the SAA7113. The data sheets were published on the websites of the respective manufacturers at the time of writing of this patent application.

The o. G. The effect is exacerbated by the fact that the video decoders do not process the audio signal themselves, but only produce an analog baseband audio signal that is either recorded by the sound card of a PC or hardware coders and standalone devices with a specialized audio decoder chip (sometimes also with the CPU of these devices ) is digitized. This often results in missing a different number of frames in the audio signal and the video signal.

Another source of interference in the analog input signal that results in lost frames as described above may be jitter or mechanical damage to the magnetic tape in an analog VTR. These usually lead to the output signal temporarily failing. Even if on a video cassette to be digitized, there are places where recording begins or ends, it may interfere with the video signal of the recorder when playing beyond these locations. The already described peculiarity of video digitiser chips then often results in a loss of frame as a result of these disturbances, since the digitization circuit with the missing video or audio signal and no clock for the recording can be formed and no blanking interval is detected, the generation of a new video frames would kick off. This problem is exacerbated by the fact that the video head and the audio head are placed offset in analog VCRs of the dominant VHS system, so that the frame losses in audio and video even occur in different places and not have the same duration.

Even when receiving digital sources, frames may be lost due to interference during transmission, e.g. B. when digital television via antenna from the satellite (DVB-S) and in some regions now also terrestrial receive (DVB-T). However, the disturbances of the transmission path described for the analogue television reception also apply to the digital transmission. As long as the disturbances are so small that they do not cause any bit errors in the demodulation in the receiver, the reception quality is not affected. In the border area individual macroblocks of the image are disturbed, then it comes to block formation. If the interference exceeds a certain level, the reception is completely disturbed or individual frames are lost, with a loss of audio frames being less likely than the loss of video frames.

The loss of frames occurs quite often, as already pointed out, and can have many causes. Most commonly, it occurs when videos from digital sources are digitized, and the video signal from these sources is inferior in quality.

Digital video recorders (eg, PC's with TV tuner card, DVD recorder) recording videos in MPEG protocol family format typically record them as system streams. Since the system streams also contain synchronization information which is formed with the aid of the time base of the recording device independent of the received video signal, lost frames initially have no effect.

For further processing z. For example, by cutting the video or burning the video to DVD, this synchronization information is usually lost. In addition, the total playing time of the frames lost in the audio data streams is almost always different from the total playing time of the frames lost in the video data streams when frames have been lost in the recording or transmission of digital video. As a result, audio and video are asynchronous for the rest of the video in further processing. Therefore, it is necessary to determine the position of lost frames prior to processing digital video and then replace them.

• – ein Rückkanal benötigt wird, über den das Widerholen von Daten angefordert werden kann,
• – sowohl Sender als auch Empfänger das Verfahren unterstützen müssen,
• – zur Übertragung ein Verbindungsorientiertes Protokoll wie z. B. TCP/IP benötigt wird,
• – vor oder nach der Übertragung entstandene Fehler nicht beseitigt werden können, und
• – selbst bei während der Übertragung entstandenen Fehlern keine nachträgliche Korrektur möglich ist.

One prior art technique for replacing missing frames is in digital video streaming applications, such as video streaming. As video surveillance systems or video conferencing systems for use. Video streaming is added to the MPEG system streams by the sender using real-time transport protocol (RTP) sequence numbers. Furthermore, the video streaming receiver generates an automatic repeat request to the sender when the receiver z. B. in case of missing sequence numbers the loss of a packet finds. Further developments of this method, the amount of data to be repeated in case of error by appropriately selected priorities for the different types of frames (z. I, P or B frames) are in US 5,768,527 ; JP 10 126772 A . EP 0905976 A1 and DE 600 20 672 T2 described. However, applying these methods to digital videos that are not transmitted by RTP has the disadvantage that:

• - a return channel is required, via which the retrieval of data can be requested,
• Both transmitter and receiver must support the procedure,
• - For transmission a connection-oriented protocol such. B. TCP / IP is required,
• - errors arising before or after transmission can not be eliminated, and
• - no subsequent correction is possible even if errors occur during transmission.

As a result, these o. G. Procedure for receivers of digital terrestrial television or digital satellite television just as little as for the subsequent correction of stored digital video or the digitizing and subsequent compression of analog video signals suitable.

Another known method for replacing lost frames is in US 6,137,920 disclosed. US 6,137,920 serves to mitigate the impression of "hopping" images during video transmission over very narrowband media (eg, analog modem links). For this purpose, after the decoding of the data stream, additional intermediate images calculated by means of a complicated process, referred to as "morphing", from two images transmitted one after the other are to be inserted into a digital VIDEO. Furthermore, US 6,137,920 suggests to decide by comparison of the frame rate of the received data stream with a predefined target frame rate when new video frames (pictures) must be inserted into the data stream.

• – ein bei der MPEG-Komprimierung systembedingter Qualitätsverlust durch die De- und anschließende Encodierung entsteht,
• – die De- und anschließende Encodierung einen enormen Aufwand verursacht,
• – die Notwendigkeit besteht, für die Erzeugung der Zwischenbilder auf das Ende der Übertragung eines auf sie folgenden fehlerfreien Bildes zu warten,
• – eine subjektiv störende Wirkung entsteht, wenn sich im korrigierten Video im Bereich zwischen dem letzten Frame vor den verlorenen Frames und dem ersten Frame nach den verlorenen Frames ein Schnitt oder ein Szenenwechsel befunden hatte,
• – Erkennung und Ersetzung fehlender Audioframes nicht behandelt werden, und
• – es auf Grund der bei diesem Verfahren getroffenen Annahme, dass fehlende Frames in regelmäßigen Abständen auftreten, unmöglich ist, zufällige Störungen zu erkennen und zu korrigieren.

This method has the disadvantage that:

• - a system-related quality loss due to the decoding and subsequent encoding occurs during MPEG compression,
• - the de- and subsequent encoding causes a huge effort,
• There is a need to wait for the transmission of an error-free image following for the generation of the intermediate images,
• A subjectively disturbing effect arises if the corrected video had a section or a scene change in the area between the last frame before the lost frames and the first frame after the lost frames,
• - Detection and replacement of missing audio frames can not be handled, and
• - It is impossible to detect and correct random disturbances due to the assumption made in this procedure that missing frames occur at regular intervals.

From the EP 0 618 728 A2 For example, a method is known in which, in order to correct for synchronization errors of audio and video signal due to missing or disturbed video packets in an audio and video data stream detection and correction can be made by the individual packets is assigned a sequential number, so that a Absence of a packet can be detected during decoding.

The invention is therefore based on the problem
to find a method that can be used to detect during recording or later, when a system stream can not be further processed, because frames have been lost in it.

Furthermore, the method according to the invention is based on the task of inserting lost frames into the audio and video data streams of system streams later or in real time or of subsequently correcting their adverse effects in further processing, in particular the shift of picture and sound, or the loss of frames to prevent completely.

Furthermore, the method according to the invention should be applicable to all formats in which digital video of the MPEG protocol family may be present, regardless of which medium the digital video is stored on or via which medium and with which protocol it is transmitted inventive method the above Disadvantages of the prior art method of replacing missing frames in video streams are avoided.

This object is achieved according to the invention by a method hereinafter referred to as a frame error correction method having the characterizing features of claim 1. Thereafter, a stored or received digital video present as a system stream and in a format of the MPEG protocol family or a related format is examined for missing frames by comparing the synchronization information associated with each frame with the sum of the playback times of all the frames of the same stream occurring prior to that frame ,

In the method according to the invention, the digital video is preferably treated as a data stream and read in and evaluated serially. The claims are not on this serial Processing limited. The advantage of serial processing is that the method according to the invention can always be used in the same way, regardless of whether the video is stored as a file, is received by a network or transmitted by broadcasting.

In the following description of the features of the method according to the invention, reference will be made to the description of the function, data structure and nomenclature of the MPEG protocol family. In this case, data streams or files which are bound together from at least one video and optionally one or more audio data streams assigned to each video data stream and encoded in a format of the MPEG protocol family are referred to as digital video.

According to the above description, in the case of digital video in an MPEG format, it can be considered that the start of a frame in a video stream can be identified by the use of the picture header. For constant bit rate encoded audio data streams, the starting position of the frame may be calculated from the fixed length of the frame, otherwise it may also be identified by its header and the compression bit sequence dependent synchronization bit sequence. The frame or picture header is part of its video and audio stream and thus packaged in the packets of the digital video. As a result, the frames can be uniquely assigned the packet header and packetized elementary stream header (PES header) information of the packet in which the header of the frame begins.

A characteristic feature of the method according to the invention is to identify the individual packets contained in a digital video and process them in the order in which they are stored or transmitted. The stream ID from the PES header (see 6 ) of the packet is to check whether it corresponds to the stream ID of a video or audio stream to be checked for missing frames. If this is the case, for the further processing of the packet a register set or a memory area is to be selected in which the information required by the method for processing this data stream is buffered.

Characteristic feature of the method according to the invention is further that from the headers of the packet, the timestamp of the packet is determined and buffered in the register set or memory area, which is associated with the data stream of the packet. In the following, the time stamp of the packet is the time at which the playback of the frame whose data is contained in the packet should be started. If more than one frame is present in a packet, then the time stamp is to be understood as the point in time at which the playback of the first frame starting in the packet should begin.

The timestamp of the packet is calculated from the synchronization information in the fields "PRESENTATION TIME STAMP" (in the "PACKETIZED ELEMENTARM STREAM" header), "SYSTEM CLOCK REFERENCE" (from the "PACKET HEADER"). An optional plausibility comparison between these two values can also be carried out and the more plausible value accepted.

Another characteristic feature of the method according to the invention is the examination of the contents of the packets of the data streams to be examined. If one were to assemble the contents of these packages while maintaining their order, one could recover the data stream contained therein. This so formed data stream is after the headers of frames and for video frames additionally after GOP headers ( 4 ) and to search for headers containing the frame rate (for example, sequence headers).

Also needed for the inventive method is the playback time of the individual frames of the data stream. For the audio data streams, the playing time of each frame has a fixed value that is different for each audio format. In order to determine the playing time of the currently examined frame, it is sufficient to determine the audio format in which the associated data stream exists and to store the frame rate assigned to it in a register. In order to determine the audio format, the stream ID can be used, if necessary also information on subformats (eg in the case of the MPEG-original audio formats) must be evaluated from the header of the audio frame. With video data streams, however, several different frame rates are possible, which are coded in different headers. The refresh rate of the video stream can be z. For example, from the sequence header that is in 5 is shown determined.

In this search, it must also be taken into account that a frame, picture, sequence or GOP header can be distributed over 2 packets. Therefore, the minimum requirement for the device used for the method according to the invention is that at least one shift register of the length of the largest of the headers to be evaluated for the method be present in the respective data stream for each video or audio data stream to be checked. In this register are the sequentially read from the package read data. The selection of the register set of the data stream is thereby controlled by the stream ID from the PES header of the packet in which the frame begins. After each shift operation, check whether the beginning of this register contains the ID of a frame header. For video data streams, it must also be checked whether the beginning of this register contains the identifier of a GOP header or a sequence header. Optionally, a buffer can be provided in which at least 2 packets of each data stream to be examined can be buffered, and search in this buffer for the start of the frame and possibly after the beginning of a GOP header or a sequence header. For audio frames that are coded with constant bit rate, it is possible (with PCM necessary) to calculate the frame start from the frame size, whereby the size of a frame results from the bit rate.

• • Der Zähler für die Anzahl der Frames des Datenstrom ist zu erhöhen.
• • Bei Videodatenströmen ist zusätzlich der Zähler für die Anzahl der Frames des GOP ist zu erhöhen.
• • Bei Videodatenströmen muss das Feld „TEMPORAL SEQUENCE NUMBER” aus dem Picture Header gelesen und im dem gerade untersuchten Datenstrom zugeordneten Registersatz oder Kontext zwischengespeichert werden. Dieses Feld wird benötigt, um die wegen der innerhalb eines GOP möglicherweise von der Übertragungs- oder Aufzeichnungsreihenfolge abweichenden Abspielreihenfolge erforderliche Korrektur der Zeitberechnung aus der Framezahl vorzunehmen.
• • In den meisten Fällen beginnt das erste Frame eines jeden Datenstroms des digitalen Videos mit einem Offset. Dieser Offset dient dazu, Abspielgeräten oder Programmen die Zeit zu geben, sich zu initialisieren. Der Offset wird dadurch realisiert, dass dem ersten Frame jedes Datenstroms mittels der Synchronisationsinformationen in dem Packet- und PES-Header ein Zeitstempel zugeordnet wird, der größer als 0 ist. Ein sinnvoller Wert für diesen Offset sind 300 Millisekunden. Da es Multiplexer gibt, bei denen dieser Wert eingestellt werden kann, hängt es letztendlich von der Person, die das digitale Video erstellt hat, ab, welcher Wert vorgegeben ist. Der Offset der zu untersuchenden Datenströme wird in den dem Datenstrom zugeordneten Registersatz oder Kontext geschrieben. Das Ermitteln des Offsets erfolgt, wenn der Beginn des ersten Frames eines Video- oder Audiodatenstromes detektiert wurde. Zu diesem Zeitpunkt entspricht der Offset des Datenstroms dem Zeitstempel des Paketes, in dem das erste Frame beginnt. Der Offset kann für jeden Datenstrom unterschiedlich sein. Daher ist für jeden zu untersuchenden Video- und Audiodatenstrom eine Speicherzelle erforderlich, in der der Offset des jeweiligen Datenstroms gemerkt wird.
• • Die Prüfung, ob vor dem Frame, dessen Header gerade ausgewertet wird, Frames verloren gegangen sind, ist entsprechend dem erfindungsgemäßen Verfahren beim Auftreten jedes neuen Frame-Headers auszufahren, wenn es sich um den ersten Frame-Header in dem Paket handelt. Dieses Verfahren wird im Anschluss an die Beschreibung der auf einen GOP-Header und einen Sequence-Header anzuwendenden Operationen beschrieben. Abhängig vom Ergebnis dieser Prüfung können weitere Operationen ausgeführt werden.

When locating the start of a frame, the following operations characteristic of the method according to the invention are to be carried out:

• • Increase the counter for the number of frames of the data stream.
• • For video data streams, the counter for the number of frames of the GOP must also be increased.
• • For video streams, the "TEMPORAL SEQUENCE NUMBER" field must be read from the Picture Header and cached in the register set or context associated with the data stream being examined. This field is needed to correct for the time calculation from frame number required because of the play order that may be different within a GOP from the transfer or record order.
• • In most cases, the first frame of each stream of digital video begins with an offset. This offset is used to give players or programs the time to initialize. The offset is realized by assigning to the first frame of each data stream, by means of the synchronization information in the packet and PES headers, a time stamp which is greater than zero. A reasonable value for this offset is 300 milliseconds. Since there are multiplexers where this value can be set, it ultimately depends on the person who made the digital video, which value is given. The offset of the data streams to be examined is written in the register set or context associated with the data stream. The determination of the offset takes place when the beginning of the first frame of a video or audio data stream has been detected. At this time, the offset of the data stream corresponds to the timestamp of the packet in which the first frame begins. The offset can be different for each data stream. Therefore, a memory cell is required for each video and audio data stream to be examined, in which the offset of the respective data stream is noted.
• The check whether frames have been lost before the frame whose header is being evaluated has to be extended according to the inventive method upon the occurrence of each new frame header, if it is the first frame header in the packet. This method will be described following the description of the operations to be applied to a GOP header and a sequence header. Depending on the result of this check, further operations may be performed.

In the MPEG family of video streams, frames are grouped into groups called GOP (Group of Pictures). Within a GOP, it is permissible to save or transfer the frames in a different order than the order in which they should be displayed. This peculiarity of video data streams must be taken into account for the method according to the invention. As will be explained in more detail in the further course of this description, the inventive method compares the runtime formed from the accumulated running time of the frames previously encountered in the data stream with the time stamp of the frame being tested. For video data streams, this summed runtime must be corrected by a value calculated from the playhead position of the frame within the GOP before the comparison can be made with the timestamp of the frame.

In order to later determine the correction value resulting from the playing order of video frames, the transfer or storage position of the frame within the GOP must be known as well as its play position. Various methods can be used for this purpose. The easiest way is to start a counter at the beginning of each GOP, which is incremented by 1 for each picture header, which is the frame header for video streams. This counter must be present once for each video stream that is scanned for missing frames. The beginning of a GOP is indicated in the video streams by the GOP header. The GOP header is in 4 shown. It begins with a 24-bit synchronization bit string (0x000001), followed by the identification byte of the GOP header (0xB8). The GOP header is just like the picture header part of the video data stream, so it is not created until the multiplexing. When searching for the GOP header can, apart from the value of the identification byte, just as in the search for a picture header (this is used in video streams as a frame header) are moved.

• • Der Zähler für die Anzahl der Frames des GOP ist zurückzusetzen.

When locating the start of a GOP in a video stream, perform the following operations:

• • The counter for the number of frames of the GOP must be reset.

The sequence header is in 5 shown. It begins with a 24-bit synchronization bit string (0x000001), followed by the identification byte of the sequence header (0xB3). The sequence header is just like the picture header and the GOP header part of the video data stream, so it is not created until the multiplexing. The search for the sequence header, apart from the value of the identification byte, can be used in the same way as for the search for a picture header or a GOP header.

• • Aus den Codes für die Bildrate des Videodatenstromes im Feld Framerate des Sequence Headers ist gemäß der Tabelle in 5 die Anzahl der pro Sekunde abzuspielenden Bilder zu ermitteln, aus der wiederum die Abspieldauer eines Bildes ermittelt werden kann.

When locating the start of a sequence header in a video stream, perform the following operations:

• • From the frame rate codes of the video stream in the Frame Rate field of the Sequence Header, refer to the table in 5 determine the number of images to be played per second, from which in turn the playing time of an image can be determined.

T_F:

Abspielstartzeitpunkt (Zeitstempel) des Frames (aus dem Feld SCR des Packet-Headers oder aus dem Feld PTS des PES-Header des Paketes berechnet, in den der Beginn des Frames fällt),

Fps:

Framerate bei Audiodatenströmen (abhängig von der Codierung des Datenstroms) oder Bildrate (aus dem Sequence-Header)

T_Offs:

Der Offset (Abspielstartzeitpunkt/Zeitstempel T_F) des ersten Frames des Datenstroms

N_F:

Die Anzahl der bisher aufgetretenen Frame-Header einschließlich des Headers des gerade ausgewerteten Frames

With the information obtained from the various headers during sequential reading of the data streams as described above, the inventive method for checking for missing packets can be executed at the end of each first frame header in a packet. At this point, the following information is available for the frame being examined:

T _F :

Playback start time (timestamp) of the frame (calculated from the field SCR of the packet header or from the field PTS of the PES header of the packet in which the beginning of the frame falls),

fps:

Frame rate for audio data streams (depending on the encoding of the data stream) or frame rate (from the sequence header)

T _Offs :

The offset (playback start time / time stamp T _F ) of the first frame of the data stream

N _F :

The number of frame headers that have occurred so far, including the header of the currently evaluated frame

TSN:

Abspielposition des aktuell bearbeiteten Frames im GOP (aus dem Picture Header des gerade ausgewerteten Frames entnommener Parameter Temporal Sequence Number)

N_GOP:

Die Anzahl der bisher in dem aktuellen GOP gezählten Frames.

For video frames, the following information is also available:

TSN:

Play position of the currently edited frame in the GOP (parameter Temporal Sequence Number taken from the Picture Header of the currently evaluated frame)

N _GOP :

The number of frames counted in the current GOP so far.

T_R:

Rohzeit

bezeichnet. Für Audiodatenströme kann sie wie folgt berechnet werden: T_R = (N_F/Fps) + T_Offs With this information mm can be checked. For this purpose, the time is calculated, which would have been needed to play all previously occurred frames of the data stream in a row. This time will be referred to below

T _R :

Rohzeit

designated. For audio data streams, it can be calculated as follows: T _R = (N _F / Fps) + T _Offs

For video frames, the position of the frame in the GOP must be considered. Here the raw time is calculated as follows: T _R = ((N _F -N _GOP + TSN) / Fps) + T _Offs

Now it can be checked if frames have been lost in front of the position of the frame being checked. This is the case if the raw time T _{R is} greater than the play start time T _F calculated for that frame from the packet header or PES header of the packet in which the beginning of the frame falls.

The above description of the method according to the invention is in the form of a flow chart in 7 for a video stream. In 1 a flow chart for the application of the method according to the invention to an audio data stream is shown.

So that the method according to the invention is still applicable even if more than one frame has been lost, it is determined how many frames have been lost before the frame just examined. This is necessary because disturbances that prevent proper digitization or transmission can also result in multiple frames being lost.

The number N _V of the missing frames can be used for audio data streams with the formula N _V = (T _F - (N _F / Fps) + T _Offs ) · Fps and be determined by rounding to an integer value. For video frames, the position of the frame in the GOP must be considered. Here can N _V with the formula N _V = (T _F - ((N _F -N _GOP + TSN) / Fps) + T _Offs ) · Fps and be determined by rounding to an integer value.

It is necessary, upon detection of one or more missing frames to increase the offset of the data stream by the playing time of the lost frames, or increase the used for the calculation of the raw time offset of the data stream by the playing time of the lost frames after the detection of the first Frames in front of which a single frame or a block of frames has been lost, also detect other lost frames. If one did not correct the registers of the method after detecting such an error, a loss of frames would be detected every other frame. This correction can be made according to the invention by adding the number of lost frames N _V to the counter of the previously occurred frames N _F. This realization requires u. U. a rounding error, but is particularly easy to implement in particular when integrating the method according to the invention in a hardware solution. Alternatively, this goal can also be achieved by increasing the lost-packet playback period or offset variable in the data stream in which the loss of frames was detected. This also achieves that a continuation of the comparison is possible.

After recognizing the lack of frames and determining the number of missing frames as described above, it is necessary to replace the missing frames to e.g. B. to prevent the loss of synchronization between image and sound in the processing of digital video in the data streams are lost frames.

For this purpose, the synchronization information, which is present only in the Packet and PES headers of the bound video, and can be lost during further processing, in the video and audio data streams format compliant z. B. inserted as user data. After the further processing of the digital video then a post-processing is required, which corrects the possibly incorrectly set in the course of processing synchronization information using the information inserted into the video and audio data streams.

For video streams can be z. B. exploit that the MPEG formats allow the insertion of user data. This user data is z. For example, to insert the information with which encoder the video was compressed, used. User data may be written in video streams to any location outside of an area enclosed by an MPEG header. They start with a user data header. The user data header consists of the 24-bit synchronization bit sequence (0x000001) and the 8-bit identifier of the user data header (0xB2). Concerning the content of the user data, there is only the restriction that it can not contain the 24-bit synchronization bit sequence (0x000001) since this signalizes the beginning of the next MPEG header with which the user data block ends. In order to save the synchronization information from the packet header of the system stream, which would be lost during further processing, you can thus generate a user data block. It suffices to insert the contents of the fields PRESENTATION TIME STAMP in the PES header or / and SYSTEM CLOCK REFERENCE in the packet header of each package in which a picture header or a frame header begins as user data. In addition, these two values for the previous frame must be saved in the data stream in order to be able to recognize later whether the video was cut. In addition, an identifier is required at which this user data block can be recognized as a data record generated by the method according to the invention. For this identifier can z. For example, if a string is used, an extra checksum over the string and the saved data may increase the likelihood of detection.

In order to restore synchronization after further processing of the digital video in a second step, the synchronization information in the fields "PRESENTATION TIME STAMP" in the "PACKETIZED ELEMENTARY STREAM" header or / and "SYSTEM CLOCK REFERENCE" from the "PACKET HEADER" of each packet of the system stream generated during further processing is restored using the system information containers generated prior to further processing. In this further processing (for example authoring as a DVD or editing the video), the data streams are generally demultiplexed and then multiplexed again. As a rule, among other things, the timestamps generated when the bound video is generated for the first time are lost. If these were stored in the data stream as recovery information according to the method of the invention, the PRESENTATION TIME STAMP fields in the PES header and / or SYSTEM CLOCK REFERENCE can be restored with you in the packet header of all packets in the data stream from which they were taken. Will be in a video or Audio stream found a block with recovery information is written to the packet headers of the packet in which this block ends, the values contained in this block for the fields PRESENTATION TIME STAMP or / and SYSTEM CLOCK REFERENCE. All subsequent packet headers of the stream with the same stream ID will be similarly corrected until the next block with recovery information is found. However, if the digital video has been edited, this value must be corrected. This can be recognized by the fact that the saved synchronization information of the last frame in the frame being processed deviates from the synchronization information that was saved for the previous frame. Then, the synchronization information written to the further processed system stream must be corrected for the difference between these two values.

Removing the additional correction information inserted into the audio and video streams after binding or multiplexing the video is very expensive. This is especially true if the video has been multiplexed to DVD for burning. In this case, the navigation parameters for the menus of the DVD would have to be adjusted as well as the playheads and hands that were generated for the search or play backwards and forwards at a multiple of the normal speed. However, because the inserted data is packaged in MPEG format compliant headers that are ignored by the player or program, it is not absolutely necessary to remove them.

However, it makes sense to write this correction information into the data stream only if the synchronization information in the PES or / and packet header changes. This often results in significantly less overhead due to the correction information, since, in particular, hardware encoders enter the same time stamp in the packet or PES header of all frames that belong to a GOP. This not only affects the video frames. All audio frames assigned to this GOP also receive the time stamp from the video frames of the GOP.

The overhead of the correction data can be further reduced if the insertion of the additional correction information to reduce the amount of data only occurs when a frame has occurred before frames have been lost. Whether such a frame is present can be determined by the method according to the invention.

If correction information is inserted only before the frame before which frames have been lost, then instead of the absolute information (the timestamp of the frame), the difference between the actual and the expected timestamp can also be entered in the block inserted for later correction. During the post-processing after the further processing of the digital video, the time stamp of the frames of the data stream must then be increased by the correction value written in the data stream. To do this, in the packet headers of the package in which the block with the correction information ends, the values for the fields PRESENTATION TIME STAMP or / and SYSTEM CLOCK REFERENCE contained in this block are written. All following packet headers of this data stream are corrected in the same way until the next block with correction information is found. In this case, however, it must be ensured that the inserted correction data, should the further processing include editing the video, not be deleted.

• – anstatt jedes verlorenen Frames wird ein Frame, der vorzugsweise beim Anzeigen oder Abspielen nicht bemerkbar ist (z. B. ein stiller Audioframe oder ein B-Frame eines Videodatenstroms, in dem ein Standbild encodiert ist), in den Datenstrom eingefügt werden, wobei die eingefügten Frames während der Verarbeitung des zu überprüfenden Datenstroms unter Verwendung von aus diesem Datenstrom entnommenen Parameter erzeugt werden, oder
• – in der für das erfindungsgemäße Verfahren geschaffenen Vorrichtung wird mindestens ein Frame, der vorzugsweise beim Anzeigen nicht bemerkbarer ist, für jede für die von dieser Vorrichtung unterstützten Formate der Audio- und Videodatenströme zulässige Kombination von Parametern, die für das Encodieren relevant sind (wie z. B. Abtastfrequenz oder Auflösung), gespeichert sind, und dass aus diesen vorgehaltenen Frames der zu den Parameter des Datenstroms passende Frame anstatt eines jeden verlorenen Frames in diesen Datenstrom eingefügt.

Another method for correcting lost frames comprises the steps:

• Instead of each lost frame, a frame which is preferably not noticeable in the display or playback (eg a silent audio frame or a B-frame of a video data stream in which a still picture is encoded) is inserted in the data stream inserted frames are generated during processing of the data stream to be checked using parameters taken from this data stream, or
• In the device provided for the method according to the invention, at least one frame, which is preferably not noticeable when displaying, is used for each combination of parameters which are relevant for the encoding (such as, for example), for the formats of the audio and video data streams supported by this device B. sampling frequency or resolution), are stored, and that from these retained frames of the frame corresponding to the parameters of the data stream instead of each lost frame inserted in this data stream.

These steps also serve to prevent the loss of synchronization between image and sound in the further processing of digital video in which data streams have lost frames. However, this method should also be able to be used if post-processing is not possible after further processing. This can be z. For example, if the processing is done with an authoring program for a DVD, which does not allow the caching of the files destined for burning to DVD on hard disk, but burns the data directly to DVD. In contrast to the method described above, it is also possible to carry out a new encoding in the course of further processing (of which However, because of the associated loss of quality must be discouraged).

In accordance with the above steps, pre-encoded frames or frames encoded at runtime with the parameters of the data stream are inserted into the data stream to achieve this goal. This is not a problem for audio streams because there is no dependency between the individual packets in MPEG audio compression formats. The provision of pre-encoded frames in the non-volatile memory of the apparatus used to carry out the method according to the invention lends itself to the fact that there are very few parameters in which audio and video data streams can vary.

For audio data streams, the number of frames to be pre-encoded only affects the sampling rate (32.441, 48 and 44.1 kHz admissible values) and the data rate (16 permissible values). If one takes into account only the variants of the MPEG audio compression methods MPEG 1 Layer 1, MPEG 1 Layer 2, and MPEG 1 Layer 3 found in the field of consumer electronics, the result is a manageable number of frames which have to be precompressed. The same applies to the audio compression methods used in the industry standard derived from the MPEG formats. These include Dolby Digital, PCM and Digital Theater Sound. So z. For example, Dolby Digital allows sampling rates of 32, 48 kHz and 44.1 kHz, as well as 38 different data rate values. Because of these features, inserting precompiled frames requires significantly less effort than recompressing all or part of the data stream, saving you from implementing an encoder. The processing time is also reduced in this way, this in turn possibly reduces the requirements for the performance of the device used for the inventive method.

If pre-encoded frames are inserted, they should not be noticeable to the observer / listener. This can z. For example, a silent frame (i.e., a sound sequence in which no sound is heard) or a repeat of the last image so that the image freezes are used. Since in the analogue source material disturbances which cause the loss of frames during digitalisation are frequently seen as picture dropouts or continuous stripes, a subjective improvement in the quality compared to the analogous starting material can even be achieved frequently. However, other frames can be included, such. For example, a fault icon, an error message, or a beep to report the problem.

For video streams, too, only a few frames need to be pre-compressed, at least, if you restrict yourself to the resolutions that support DVD players. These are 728x576, 704x576, 352x576 and 352x288 at PAL (in Europe and others) such as 720x480, 704x480, 352x480 and 352x240 at NTSC (in the USA et al ). Other parameters of the video data stream, such as. As the aspect ratio or the frame rate, have no effect on the frames to be inserted. The resolution of the video data stream can, for. B. can be read from the system header of the data stream to be corrected.

On the other hand, it seems problematic at first sight that the frames of video data streams depend on each other, since in each GOP only one picture, the I-frame, is present, which contains all the picture information. All other images contain only the information in which they differ from the I-frame or other images of the GOP. To replace a missing image, it makes sense to limit yourself to freezing the image. In this case, you can insert a picture that does not change from the previous picture. If you now compare 2 frames of completely different, but under the same conditions generated videos that were compressed with the same parameters and the same encoder, and takes in each video a section in which nothing moves, you will find that from frames other than frames (B-frames or P-frames) are identical on both videos. It is therefore possible to remove frames from a video in still sequences, thereby replacing a frame in a still sequence of another video without any changes, even though both videos show completely different still images. You can even replace a frame in a moving sequence with the removed frame. In this case, however, the picture freezes. If the removed frame is in a different location in the GOP than the inserted frame, you may need to adjust the reference to the frame or frames to which the inserted frame contains the delta information. In particular, the parameter Temporal Sequence Number in the headers of the frames of the GOP must be corrected.

Thus, to replace missing frames, additional B or P frames, preferably B frames, may be inserted into a video stream without removing other frames. However, this increases the number of frames in the GOP and its composition compared to other GOPs of the video. In particular, if the source material is intended for a DVD in which a GOP must have 15 or 18 frames (for PAL and NTSC, respectively), it will hurt u. U. the specifications. In practice, however, it has been proven that the tested hardware and software DVD players could play such extra-long GOP's without any problems. That such extra-long GOPs are not critical within certain limits, also speaks that on the market rental and purchase DVD's are available that do not comply with the prescribed by the DVD Whitebook GOP lengths. Further, a tested commercial DVD authoring program which checks the DVD files to be burned for DVD for DVD conformity, and only re-encodes if the video file fails in this test, burned video files with inserted B frames to DVD without re-encoding.

The advantages of the method according to the invention are that it expands the possibilities for detecting whether a video has been incorrectly digitized or transmitted digitally. Missing frames in digital video are a common error that, as described earlier, can have a number of causes. Applicant's personal experience is that when digitizing VHS video cassettes, once they reach a certain age, lost frames can not be avoided, no matter how high-quality the devices used for digitization are and how careful they are. In addition, with conventional procedure, missing frames only become noticeable when one processes a digital video.

In particular, due to the different lengths of video and audio frames, conditions that lose frames during digitization almost always result in loss of synchronization of image and sound after further processing. A missing synchronization of picture and sound is very disturbing.

While interference in analog video is short lived, the same video may become unusable after digitizing and further processing, as the lost audio / video synchronization begins at the point where frames are missing and continues to its end , It is particularly annoying when you have digitized an analog video first, then deletes it, and only after the subsequent processing notes that the digitization has failed. This can be avoided with the method according to the invention.

Furthermore, the inventive method has the advantage that can be with him analog video even "save" if no error-free digitization is possible.

Advantageous embodiments of the method according to the invention

An advantageous embodiment of the invention is specified in claim 2. The development of the invention according to this claim is based on the fact that the method in claim 1, feature β1 in conjunction with feature β2.2 is completely standard compliant only when applied to audio frames. The aim of the method described in claim 2 is therefore to correct also missing frames in video data streams standard compliant. In claim 2 it is therefore proposed that to restore the synchronization between image and sound possibly missing audio frames are inserted in the audio data streams and at the points where video frames are missing, the synchronous to the missing video frames audio frames are removed again. Since audio and video frames may be of different lengths, it may be necessary to vary the number of audio frames to be removed per video frame lost in order to make the playing times of the removed audio frames and the lost video frames approximately equal.

A further advantageous embodiment of the invention is mentioned in the patent claim 3. According to claim 3, it makes sense to use the device used for the inventive method simultaneously for demultiplexing the digital video. This is advantageous because the method according to the invention requires the search for the frame and the packet headers as well as the streamwise consideration in the evaluation, and thus only has to be supplemented slightly in order to fulfill the function of a demultiplexer. Furthermore, it is much easier to insert missing frames into a video or audio stream than to include them in a bound video, taking into account the changes in the packets.

For many in the processing of digital video occurring tasks such. For example, cutting, burning to DVD, or encoding must demultiplex the system stream as the digital video is typically present. Therefore, it is also useful according to claim 3, to integrate the inventive method in such an application.

The invention will be explained with reference to two applications of the closer. Show it:

8th : Representation of the use of the method according to the invention for the correction of errors that have arisen during the digitization of a partially dubbed VHS video

9 , The protocol of the error correction of in 8th illustrated use case

10 , Presentation of the use of the method according to the invention for the correction of Errors that occur when recording is started on a digital video recorder before starting to play the video to be digitized.

11 The protocol of the error correction of in 10 illustrated use case

In consumer electronics, the end user has the VHS system for recording video sources such. B. enforced television broadcasts. At times, this system was also used by amateur filmmakers. VHS has been on the market for about 30 years at the time of filing this patent application.

VHS is an analogue video recording method that is now gradually being replaced by digital video recording. First vendors have already removed VHS VCRs from their product range. This results for end users who have pre-recorded VHS tapes, the problem that for the recorded video in the foreseeable future, no players will be available more. Especially when personal memories are recorded on these cartridges, there is a desire to transfer their contents to a digital medium. It also reduces the quality of analog video recorded on magnetic tapes used with VHS video cassettes over time. Since this is not the case with digitally recorded videos, this transmission creates an additional benefit for the consumer. This will increase the demand for solutions that help consumers and service providers to digitize analog video.

As already described elsewhere in this patent application, it can be inferred from the characteristics of the prior art devices used for playing analog video and recording digital video that video digitization is a process involving the short term disruption of video Output material can cause that after further processing of the digital video sound and picture are out of sync. However, even a slight time offset of image and sound is perceived as very disturbing.

In the 8th and 10 Applications shown represent two situations in which the digitization of analog video using the prior art associated with a u. U. qualitatively no longer satisfactory results, and in which the inventive method can be used in conjunction with its subclaims to improve the result. Both applications describe situations in which the output signal of an analog video player is disturbed so much that a digital recording device can no longer decode it cleanly and therefore rejects frames in the output data stream it generates. Such disorders can have many causes, ranging from poor quality of the media to be dubbed due to aging. The two selected use cases have the advantage that they are reproducible.

In the first, in 8th In this case, it demonstrates how the transitional area affects the digitization of a video at the end of a new recording that has been dubbed an older, slightly longer recording. If playback is continued at the end of a recording on a VHS VCR, a disturbed video signal will be output for about one second before the older recording is output smoothly. The reason for this has already been discussed in the section on the advantages of the method according to the invention.

8th shows how the sequence used to evaluate this use case was generated. In step 1) a longer sequence is recorded on a VHS video cassette. Then the cassette is spooled back and then in step 2) the beginning of this sequence is dubbed with a shorter sequence. Subsequently, the video cassette is rewound again in step 3), played back and recorded digitally in MPEG-2 format.

In the evaluation and correction of the recorded video with the method according to the invention, the in 9 generated log generated. At the same time, the digitally recorded video was visually evaluated image by image using a video editing program. In this case, the later recorded recording ends after 1 minute, 1 second and 14 frames a 40 ms (equivalent to a runtime of 61,560 ms). The image of the older recording is stable at 1 minute, 12 seconds and 18 frames (equivalent to a runtime of 72,720 ms). At this point the transition area ends. According to protocol in 9 was discovered by the inventive method, the first missing video frame after 62,083 ms runtime and the last missing video frame after 72,766 ms. The runtimes issued in the protocol are those entered in the PTS by the digital recorder (in this case a TV tuner card for PC's with hardware encoders). Of these runtimes, 400 ms offset and the lost frame time (in this case 40 ms per frame as recorded in PAL) have to be deducted. At the end of the recording is still missing another video frame, but that does not matter here.

Overall, 19 video frames and 16 audio frames were not recorded as a result of the transients in the transition area. Especially since the playback time of the audio frames is 24 ms (they were encoded as MPEG I Layer 2 audio data) and the video frames 40 ms (the video was available as a PAL signal with 25 frames / s), there is a temporal difference between audio and video Offset of 360 ms from this transition range. However, this temporal offset only becomes visible or audible after burning to DVD. To optically control this, in step 5) both the recorded and the corrected file produced by the method β of claim 1 were burned to DVD and then viewed. It was found that even this short offset of 360 ms in the uncorrected video is already noticeable.

For correcting the system current, the method β of claim 1 was chosen because it provides a result that is not fully compliant with the relatively narrow specifications of the DVD-Whitebook and therefore had to be clarified in experiments as to whether the DVD's produced with it different types of DVD players are executable.

In the second, in 10 In the illustrated application, it is demonstrated which effects it can have if the recording with the digital recording device is started in step 1) before the playback of the playback device used as the signal source is started in step 2). In this case, a VHS VCR was used as the signal source whose antenna input had not been connected. Therefore, as long as the VCR has not played anything, it has output a noise signal at its output.

The evaluation of the digitally recorded video for the second application was again as in 10 Step 3) shown with the method according to claim 1. It was in the 11a to 11c generated log generated. At the same time, the digitally recorded video was visually evaluated image by image using a video editing program. In this case, the output of the video played sets at 28 seconds and 9 frames a 40 ms (equivalent to a runtime of 28,360 ms), the picture is interference-free after another 9 frames (equivalent to a runtime of 28,720 ms). According to protocol in 11a to c, the first missing video frame after 521 ms runtime and the last missing video frame after 29.392 ms were discovered by the method according to the invention. The runtimes issued in the protocol are those entered in the PTS by the digital recorder (in this case a TV tuner card for PC's with hardware encoders). Of these runtimes, 400 ms offset and the lost frame time (in this case 40 ms per frame as recorded in PAL) have to be deducted.

Looking closer, you can see that the TV tuner card loses frames because of the lack of input signal during the time nothing is playing. In the area in which the image stabilizes at the start of playback, an accumulation of lost frames is observed. In total, 54 video frames and 40 audio frames are lost in the switch-on phase. Since in addition to the higher number of video frames lost, the playing time of the audio frames is 24ms (they are encoded as MPEG I Layer 2 audio data) and the video frames are 40ms (the video was a PAL signal at 25fps) Audio and video a time offset of 1,200 s from this transitional area. However, this temporal offset only becomes visible or audible after burning to DVD. To visually control this, were in 10 Step 4) both the digitally recorded and the corrected file produced by the method β of claim 1 burned to DVD and then viewed. It was found that the offset of 1200 ms in the uncorrected video is very disturbing.

The DVD burned in step 4 was created with a total of 4 commercial and one open-source DVD authoring program. For all 5 DVDs created in this way, video and audio were asynchronous on the uncorrected files. 3 of the commercial, tested authoring programs had the property to basically recode the material. Also with these programs, video and audio were asynchronous with the uncorrected files. Apparently you encode a frame for the source file for each frame of the input file, without worrying about the synchronization information. In one of the tested programs, which fundamentally re-encodes the video, the non-existent synchronization between video and audio in one of the uncorrected files had even meant that a directly recorded, flawless file also had lost the synchronization between image and sound.

The quality of analog recorded video material, especially if the recordings are already several years old, or if multiple recordings are present on a cassette, may not be sufficient for digitization and subsequent digital processing with the prior art methods.

As shown in the examples, the DVD-making products available on the market and known to the applicant do not sufficiently take into account the particular problem of lost frames in the digital recording of low-quality analogue sources. By contrast, the method according to the invention offers the possibility of "saving" such recordings. As VHS video recorders in the consumer sector have dominated the video recording and reproducing market for more than 30 years and are now being replaced by digital techniques, a large number of videocassette tapes are expected for their owners or owners Digitization is intended. A high proportion of it will meanwhile have aged so much that a treatment of their quality z. B. may be required with the inventive method.

Claims (11)

A method of detecting missing frames in audio and video streams bound for transmission or storage to a system stream and in a format of the MPEG protocol family or related formats, such as for repairing such damaged by missing frames and video frames. and audio data streams bound data stream, characterized in that for detecting missing frames a) the same for all frames of a data stream playing time of the frames of the data streams to be examined from the control parameters of the respective data stream is calculated, for which video frames whose playing time from the in the field "FRAME RATE Is calculated in the "SEQUENCE HEADER" of the associated video data stream encoded frame rate, and for which audio frame whose playback time is calculated using the audio encoding format of the data stream permanently assigned frame rate, this can be determined by the audio encoding format in which ft, whether the value of the field "STREAM-ID" and, where necessary, the value of the field "SUBSTREAM-ID" in the "PACKETIZED ELEMENTARY STREAM" is within the range reserved for the audio encoding format according to the specifications of the present system stream type. b) searching in the system stream packets belonging to one of the data streams to be examined and containing the beginning of one or more frames; and c) in each audio and video stream to be examined the number of frames of the data stream starting from the start as in d) in the case of audio frames on the basis of the previously counted frames and the duration of the frame determined from the audio data format, for video frames on the basis of the previously counted frames, the display position of the picture from the Field "TEMPORAL SEQUENCE NUMBER" from the "Picture Header" and the frame rate is determined the time referred to below as the raw time at which the frame would be output on continuous output of the previous frame, e) added to the raw time of the frame, the offset of its data stream which offset from the synchronization information from the packet headers of the first packet of the system stream in which the first frame of this data stream begins, and this synchronization information from the field "PRESENTATION TIME STAMP" in the "PACKETIZED ELEMENTARY STREAM" header or / and from the field "SYSTEM CLOCK REFERENCE" from the " PACKET HEADER ", and f) for each frame found, the raw frame plus offset is compared to the synchronization information of the frame, this synchronization information being from the" PRESENTATION TIME STAMP "and / or the" SYSTEM CLOCK REFERENCE "which in turn is the g) is preceded by one or more missing frames before the examined frame, if raw time plus offset are smaller than the "PRESENTATION TIME STAMP" or / and the "SYSTEM CLOCK REFERENCE" of the first packet of the currently considered frame p lus, a security that is less than or equal to the time for the output of a frame; and h) upon detection of one or more lost frames in front of the considered frame, the number of lost frames is determined, in which the difference of timestamps determined from the synchronization information and the sum of raw time and offset is divided by the playing time of a frame determined for the data stream and rounded to an integer, and i) on detection of one or more missing frames, the offset of the data stream is increased by the playing time of the lost frames, or If one or more missing frames are detected in a video or audio data stream present in the system stream, optionally one of the following correction methods α and β is performed, the correction method α the following sc This includes: α1) before further processing a system stream where the system stream synchronization information is lost in the "PESETTED TIME STAMP" fields in the "PACKETIZED ELEMENTARY STREAM" header or / and in the "SYSTEM CLOCK REFERENCE" field from the "PACKET HEADER" into the video or audio stream to be corrected α1.1.1 before each frame of the elementary data stream, or α1.1.2 only before each frame of the elementary data stream before which the content of the synchronization information changes, or α1.1.3 only before each frame of the elementary data stream, before one or more lost frames were discovered, α1.2. hereinafter referred to as system information container MPEG format compliant data structure that is ignored by players or programs such. For example, a User Data field is inserted, and in which at least: α1.2.1 the fields "PRESENTATION TIME STAMP" from the "PACKETIZED ELEMENTARY STREAM" header or / and the field "SYSTEM CLOCK REFERENCE" from the "PACKET HEADER "The packet of the system stream in which the frame associated with this system information container begins, and α1.2.2 starting from the system information container of the second frame of the stream the" PRESENTATION TIME STAMP "fields in the" PACKETIZED ELEMENTARY STREAM "header and / or The "SYSTEM CLOCK REFERENCE" field from the "PACKET HEADER" of the system stream packet, in which the frame that preceded the frame to which this system information container is assigned in this elementary data stream, begins, and .alpha.1.2.3 if only frames, against which at least In addition, if a frame is lost, a system information container is mapped to the system information container in addition to the playing time of the frames that are present α2 in a second step, which is only possible after further processing of the digital video, the synchronization information in the fields "PRESENTATION TIME STAMP" in the "PACKETIZED ELEMENTARY STREAM" header or / and "SYSTEM CLOCK REFERENCE "Are recovered from the" PACKET HEADER "of each packet of the system stream generated in the further processing using the system information containers generated prior to further processing, and wherein correction method β comprises the following steps: β1 the encoding parameters of the data stream present in the data stream for these parameters Headers, unique, z. For example, upon detection of the first such header, are read and latched, and β2.1, instead of each lost frame, a frame which is preferably not noticeable on display or play, is newly generated by encoding using the cached encoding parameters and inserted into the data stream , or β2.2 in the device provided for the method according to the invention at least one frame, which is preferably not noticeable when displaying, for each combination of parameters that is permissible for the encoding for the formats of the audio and video data streams supported by this device , is stored; and selecting, from these reserved frames, the frame whose parameters correspond to the cached encoding parameters of the data stream, and is inserted into this data stream instead of each lost frame. Method for detecting missing frames in audio and video data streams and for correcting these data streams according to claim 1, characterized, that Inserting missing audio frames into the audio streams of the system stream according to feature β1, feature β2.1 or feature β2.2, and The audio frames of the system stream are removed from those audio frames whose playback period overlaps with the playing period of missing video frames, and The number of audio frames removed from the audio data stream due to lost video frames is reduced either at the beginning or at the end of the block of audio frames to be removed so that the sum of the playing times of the previously lost video frames is equal to the sum of the playing times of the previously removed audio frames by a maximum of Playing time of a single audio or video frame is different. Method for detecting missing frames in audio and video data streams and for correcting these data streams according to claim 1 or 2, characterized in that it is part of a demultiplexer or an authoring product for DVD's, or a video editing product for digital video, or an encoder whose functionality Demultiplexing of system streams is required. Method according to one of the preceding claims, characterized in that the system stream is a "system stream" according to the known standards, and comprises a "transport stream", a "program stream", or a "DVD video object". Method according to one of the preceding claims, characterized in that the MPEG protocol family in particular the formats MPEG-1, MPEG-2 and MPEG-4 belong. Method according to one of the preceding claims, characterized in that the MPEG protocol family in particular the formats DivX, DVD Whitebook, DVB-T, DVB-S and DVB-C are related. Method according to one of the preceding claims, characterized in that the encoding parameters of the data stream in a video data stream include the resolution. Method according to one of the preceding claims, characterized in that the encoding parameters of the data stream are taken from the headers present for these parameters in the data stream, in particular from the system header in the case of a video data stream. Method according to one of the preceding claims, characterized in that the one-time reading and buffering of the encoding parameters of the data stream is carried out when recognizing the first suitable header in the data stream. Method according to one of the preceding claims, characterized in that the frame, which is preferably not noticeable in the display or playback, a silent audio frame or a B-frame of a video data stream in which a still image is encoded is. Method according to one of the preceding claims, characterized in that the parameters which are relevant for the encoding include in particular the sampling frequency and the resolution.

Images