CN100581246C - Method and device for making virtual image region for motion estimation and compensation of panorama image - Google Patents
Method and device for making virtual image region for motion estimation and compensation of panorama image Download PDFInfo
- Publication number
- CN100581246C CN100581246C CN200580027414A CN200580027414A CN100581246C CN 100581246 C CN100581246 C CN 100581246C CN 200580027414 A CN200580027414 A CN 200580027414A CN 200580027414 A CN200580027414 A CN 200580027414A CN 100581246 C CN100581246 C CN 100581246C
- Authority
- CN
- China
- Prior art keywords
- reference data
- virtual image
- image region
- panoramic picture
- unit
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/563—Motion estimation with padding, i.e. with filling of non-object values in an arbitrarily shaped picture block or region for estimation purposes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/54—Browsing; Visualisation therefor
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
- G06T7/246—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments
- G06T7/248—Analysis of motion using feature-based methods, e.g. the tracking of corners or segments involving reference images or patches
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/513—Processing of motion vectors
- H04N19/517—Processing of motion vectors by encoding
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/55—Motion estimation with spatial constraints, e.g. at image or region borders
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/50—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding
- H04N19/503—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using predictive coding involving temporal prediction
- H04N19/51—Motion estimation or motion compensation
- H04N19/56—Motion estimation with initialisation of the vector search, e.g. estimating a good candidate to initiate a search
Abstract
A method of and apparatus for setting a virtual image region is provided in order to efficiently perform motion estimation and compensation of a panorama image including 360-degree omni-direction image information. The method includes the steps of searching a previous panorama image for reference data corresponding to all or part of a virtual image region of a current patch desired to be motion-estimated in order to determine the presence of the reference data, and selectively mapping the reference data to all or part of the virtual image region. By doing so, the panorama image containing omni-directional image information is coded by using a 2-dimensional moving picture coding method without change, while the coding efficiency in the process of motion estimation and compensation can be greatly improved.
Description
Technical field
The present invention relates to estimation and compensation.More specifically, the present invention relates to a kind of method and apparatus that is used to be provided with virtual image region (virtual image region), so that carry out the estimation and the compensation of the panoramic picture that comprises 360 degree omnirange image informations effectively.
Background technology
The omnirange video camera system is a kind of camera chain that can take 360 degree omnirange images according to fixed viewpoint.The omnirange camera chain by install and use the speculum of the given shape such as hyperboloidal mirror, such as fish-eye lens certain lenses or by using a plurality of video cameras to take the omnirange image.
The example that a kind of three dimensional lifelike broadcast system is used as the omnirange video coding that uses for this system has been proposed.In the three dimensional lifelike broadcast system, for example, will offer beholder's terminal about image information in baseball game etc. from the different points of view view.That is, multiple image information items is offered the beholder, comprise view, from receiver's view, from batter's view and from the view that is sitting near the spectators the initial corner from the pitcher.The beholder can select desirable viewpoint, and watches image from described viewpoint.
The image that is photographed by the omnirange camera chain has and three-dimensional spherical environment facies characteristic of correspondence.Therefore, will be converted to two-dimensional image by the 3-D view that the omnirange camera chain is taken.At this moment, two-dimensional image becomes the panoramic picture that comprises the omnirange image, and carries out the omnirange video coding at the two-dimensional panoramic image.
Substantially, this panoramic picture has the size bigger than conventional two-dimensional image, and in order to transmit qualitative picture, has used too wide bandwidth.In the omnirange image, typically, the user can be not interested in all images part that obtains simultaneously by the omnirange video camera, just wishes at length to watch most interested part based on the information about entire image by the viewpoint of freely handling the user.Therefore, need and can send the coder-decoder with minimum bandwidth image to receiving terminal, wherein said image is that the active user is most interested in.In this, a kind of method that panoramic picture is divided into tiled patch (tiled patch) and sends described image has been proposed.
Fig. 1 shows a kind of method that panoramic picture is divided into tiled patch and sends described image.
In the method, piece unit and storage together with what the panoramic picture of having encoded was divided into small pieces, then according to user's request, the part that will show with request be that some of center are pieced unit together and sent to decoder one side.According to this method, whole panoramic picture is not once sent to decoder one side, but be divided into tiled patch and transmission, make to have reduced the data volume that sends widely and sent bandwidth, and prevented to be used to send the required initial delay of all images information.Equally, ask peripheral part of current being seen area-of-interest to send together and receive the user, make the user can easily change viewpoint.
In order to send tiled patch, the method for sending and receiving that a kind of use does not have the conventional two-dimensional moving image encoding technology of change has been proposed to decoder one side.Such as MPEG-4Part2Visual or the technology H.264 be the example of employed conventional two-dimensional moving image encoding technology.As shown in Figure 2, according to conventional art,, prepared free movement vector (UMV:UnrestrictedMotion Vector) pattern, and be provided with virtual image region for when when estimation or compensation are carried out in the border of reference map picture frame, raising the efficiency.Then, the pixel value of the upper and lower, left and right boundary member by infilled frame is carried out the estimation of present image.Because can not be with reference to the arbitrary image data except the picture frame that will send in the mobile image processing techniques of two dimension, this method be used for obtaining high efficiency under this kind situation.Yet, for bigger panoramic picture is encoded, the patch units of less minute plate shape is encoded, and benchmark image can with respect to each piece together the unit the position and exist.
Therefore, when the conventional motion method for encoding images is improper, need a kind ofly can piece the benchmark image of the position existence of unit, the system and method for more effectively image being encoded together with respect to each by using.
Summary of the invention
Technical scheme
The present invention has solved above and other problem in fact, and a kind of method and apparatus that is used to produce at the virtual image region of the estimation of panoramic picture and compensation has been proposed, described panoramic picture is divided into the patch units that sends and receive, divides plate shape, wherein not at the benchmark image frame boundaries part of frame outside filler pixels unconditionally.When the actual image data that belongs to virtual image region was present in the previous panoramic picture, described method and apparatus did not use described data to produce virtual image region with changing.
Beneficial effect
According to the embodiment of the invention, panoramic picture is divided into tiled patch, and when sending described image, can map directly to virtual image region with respect to each data of piecing together in the previous panoramic picture that unit carries out reference.By operation like this, by the two dimensional motion method for encoding images that does not have to change the panoramic picture that comprises the omnirange image information is encoded, can greatly improve the code efficiency in estimation and compensation process.
Description of drawings
Fig. 1 shows a kind of conventional method that panoramic picture is divided into tiled patch and sends described image;
Fig. 2 shows by using and fills the conventional method that (padding) forms virtual image region;
Fig. 3 shows the virtual image region that is divided into the tiled patch of 8 parts according to the embodiment of the invention;
Fig. 4 shows the method that produces virtual image region according to the embodiment of the invention, with respect to the position of piecing unit together;
Fig. 5 A and Fig. 5 B show according to the embodiment of the invention, when hope encode piece unit together and be positioned at the leftmost edge of panoramic picture or rightmost edges the time, produce the method for virtual image region;
Fig. 6 shows and relates to the additional information of generation according to the virtual image region method of the embodiment of the invention;
Fig. 7 is the flow chart by the operation of carrying out in encoder according to the method for embodiment of the invention generation virtual image region;
Fig. 8 is the block diagram of the equipment that is provided with in decoder according to the embodiment of the invention, is used to produce virtual image region;
Fig. 9 is the block diagram that can use according to the encoder of the virtual image region of the embodiment of the invention;
Figure 10 is by producing the flow chart of the operation of carrying out in the decoder according to the virtual image region of the embodiment of the invention;
Figure 11 is the block diagram that is used for producing the equipment that the decoder according to the virtual image region of the embodiment of the invention is provided with; And
Figure 12 is the block diagram according to the adaptable decoder of virtual image region of the embodiment of the invention.
Embodiment
Best Implementation Modes
According to an aspect of the present invention, a kind of method of virtual image region of the estimation that is used to produce panoramic picture has been proposed, described panoramic picture is divided into the tiled patch unit and sends from transmitter side, described method comprises step: search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation in previous panoramic picture, so that determine the existence of reference data; And, reference data optionally is mapped to all or part of virtual image region based on result for retrieval.
The step of mapping reference data can comprise step: represent to exist under the situation of reference data at result for retrieval, judge whether to upgrade in the given time described reference data; And if judged result represents to have upgraded in the given time described reference data, then described reference data is mapped to all or part of virtual image region.
Here, if judged result represents not upgrade in the given time described reference data, can determine to carry out described reference data is mapped to all or part of virtual image region, still carry out with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels, and determine to carry out mapping or fill according to described.
Described method can also comprise step: so that send additional information, described additional information allows receiver side to produce all or part of virtual image region at piecing unit together in fact in the same manner to record about definite information.
In addition, if result for retrieval represents not exist reference data, can by with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels produce all or part of virtual image region.
In addition, if all or part of virtual image region is arranged in any one outside of the left margin of previous panoramic picture or right margin, the image recognition from another border to preset range can be the reference data at the virtual image region that is present in outside, a described border.
According to a further aspect of the present invention, the equipment of virtual image region that a kind of generation is used for the estimation of panoramic picture has been proposed, described panoramic picture is divided into the tiled patch unit and is sent out from transmitter side, described equipment comprises: retrieval unit, be used in previous panoramic picture search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation, so that determine the existence of reference data; Judging unit represents to exist reference data if be used for the result for retrieval of retrieval unit, judges whether to upgrade in the given time described reference data; Map unit if judged result represents to upgrade in the given time described reference data, optionally is mapped to all or part of virtual image region with described reference data; Determining unit represents not upgrade in the given time described reference data if be used for judged result, determines to carry out mapping and still fills; Filler cells represents not exist reference data if be used for result for retrieval, by with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels, produce all or part of virtual image region.
According to another aspect of the present invention, the method of virtual image region that a kind of generation is used for the motion compensation of panoramic picture has been proposed, described panoramic picture is divided into the tiled patch unit and is received at receiver side, described method comprises step: search for previous panoramic picture at the current corresponding reference data of all or part of virtual image region of piecing unit together of carrying out estimation with hope, so that determine the existence of reference data; And, reference data optionally is mapped to all or part of virtual image region based on result for retrieval.
The step of mapping reference data can comprise step (wherein, result for retrieval represents to exist reference data): judge whether to upgrade in the given time reference data; And if judged result represents to upgrade in the given time described reference data, described reference data is mapped to all or part of virtual image region.
According to a further aspect of the present invention, the equipment of virtual image region that a kind of generation is used for the estimation of panoramic picture has been proposed, described panoramic picture is used to be divided into the tiled patch unit and is sent out from transmitter side, described equipment comprises: retrieval unit, be used in previous panoramic picture search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation, so that determine the existence of reference data; Judging unit represents to exist reference data if be used for the result for retrieval of retrieval unit, judges whether to upgrade in the given time described reference data; Map unit if judged result represents to have upgraded in the given time described reference data, optionally is mapped to all or part of virtual image region with described reference data; The additional information interpretation unit, if wherein the judged result of judging unit represents not upgrade in the given time described reference data, be used for the additional information that decipher receives from transmitter side, described information representation is that described reference data is mapped to all or part of virtual image region in the transmitter side, still with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels.Described equipment also comprises filler cells, represent not exist reference data if be used for the result for retrieval of retrieval unit, by with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels, produce all or part of virtual image region.
According to another aspect of the present invention, a kind of computer readable recording medium storing program for performing has been proposed, the computer program that has specific implementation thereon, be used to carry out the method that a kind of generation is used for the virtual image region of the estimation of panoramic picture and motion compensation, described panoramic picture is divided into the tiled patch unit and is sent out from transmitter side, described method comprises step: search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation and motion compensation in previous panoramic picture, so that determine the existence of reference data; If result for retrieval represents to exist reference data, judge whether to upgrade in the given time described reference data; And if judged result represents to have upgraded in the given time described reference data, described reference data is mapped to all or part of virtual image region.
According to another aspect of the present invention, a kind of method that is used for the estimation of panoramic picture has been proposed, described panoramic picture is divided into the tiled patch unit and sends from transmitter side, described method comprises step: under the current corresponding reference data of all or part of virtual image region of piecing unit together of carrying out estimation with hope is in situation in the previous panoramic picture, reference data is mapped to all or part of virtual image region, and, produce the current motion vector of piecing unit relevant with benchmark image by using all or part of virtual image region about the mapping reference data.
According to another aspect of the present invention, a kind of coding method has been proposed, be used for being divided into the tiled patch unit, and encode from the panoramic picture that transmitter side sends, described method comprises step: under the current corresponding reference data of all or part of virtual image region of piecing unit together of carrying out estimation with hope is in situation in the previous panoramic picture, reference data is mapped to all or part of virtual image region, and, produce the current motion vector of piecing unit relevant with benchmark image by using all or part of virtual image region about the mapping reference data.Described method also comprises step: carry out the motion compensation of piecing unit together at current by using described motion vector.
According to another aspect of the present invention, a kind of method that is used for the motion compensation of panoramic picture has been proposed, described panoramic picture is divided into the tiled patch unit, and receive at receiver side, described method comprises step: receive at the current input of piecing the motion vector of unit together that comprises in current panoramic picture, if and represent by motion vector, and be in the previous panoramic picture with the current corresponding reference data of all or part of virtual image region of piecing unit together, described reference data is mapped to all or part of virtual image region, and by using all or part of virtual image region, produce the unit that pieces together of benchmark image that described motion vector is used about the mapping reference data.
According to another aspect of the present invention, a kind of method that is used for the motion compensation of panoramic picture has been proposed, described panoramic picture is divided into the tiled patch unit, and receive at receiver side, described method comprises step: receive at the current input of piecing the motion vector of unit together that comprises in current panoramic picture, if and represent by motion vector, and be in the previous panoramic picture with the current corresponding reference data of all or part of virtual image region of piecing unit together, described reference data is mapped to all or part of virtual image region, and, carry out the motion compensation of benchmark image by using all or part of virtual image region about the mapping reference data.Described method also comprises step: by use current piece together the differential signal of piecing unit together of unit and motion compensation and motion compensation piece unit together, recover the current unit that pieces together.
Realize mode of the present invention
Describe the present invention more all sidedly referring now to accompanying drawing, there is shown exemplary embodiments of the present invention.
Fig. 3 shows the virtual image region of the tiled patch that is divided into 8 parts (numeral 1 to 8).According to the embodiment of the invention, whole panoramic picture is divided into tiled patch, and therefore with respect to the position of piecing unit together, the method that produces virtual image region can change in each part.This will be described in greater detail below.
Fig. 4 shows the method that produces virtual image region according to the embodiment of the invention, with respect to the position of piecing unit together.
At first, in internal patch is under the situation of A, wish that wherein the current position of piecing unit together of encoding is not on the edge of panoramic picture, if the reference data of recent renewal in the zone of piecing first corresponding position together of each part is mapped to corresponding part with corresponding reference data in existence and the previous panoramic picture.
If do not carry out the renewal of reference data recently, can determine to utilize the border pixel values of piecing the frame (being called " reference patch " hereinafter) in the identical corresponding previous panoramic picture in position of unit together to fill, still instead use corresponding reference data with wishing recently to encode.If comprise and carry out the definite process of this kind, preferably produce additional information, and send to receiver side (that is, the decoding side) about determining, make the decoding side that identical in fact virtual image region can be set.This additional information will be described in greater detail below.Simultaneously, if there is no reference data produces corresponding part by filler pixels on the boundary member of reference patch.
Next, be positioned in the position of piecing unit together that current hope is encoded under the situation at edge of top section of panoramic picture, as with respect to the B of part 5 and the part 6 corresponding virtual image region of Fig. 3, pixel value 128 is mapped to corresponding zone.For remainder, carry out and the identical process of virtual image region essence that is used for producing A.
Equally, the unit that pieces together at other positions carries out said process in the mode that essence is identical.Yet, under the situation of piecing first F, D, H, G, E and I together of the most left or rightmost edges that lays respectively at panoramic picture, can carry out different slightly processes, so that increase estimation and compensation efficient.Now this process will be described in further detail.
Fig. 5 A shows according to the embodiment of the invention, when what hope was encoded and pieces the method that produces virtual image region when unit is positioned at the leftmost edge of panoramic picture together.
Piecing together under the situation of first D, because the part of Fig. 33,5 and 7 is positioned at the outside of panoramic picture, the reference data that is used for this does not exist.
Yet according to the feature of panoramic picture, D has and G, E and the higher correlation of I.Therefore, with reference to G, E and the I rather than the use filling unconditionally of previous panoramic picture, more effectively produce the virtual image region that is used for part 5,3 and 7 by respectively.That is, with a part of data map of G to the part 5 of piecing first D together, with a part of data map of E to part 3 and with a part of data map of I to part 7.Yet, if do not exist, or do not upgrade recently with reference to view data among G, E and the I, will use above-mentioned algorithm.
Fig. 5 b shows according to the embodiment of the invention, when what hope was encoded and pieces the method that produces virtual image region when unit is positioned at the rightmost edges of panoramic picture together.In this case, in the identical mode of essence, F, D and the H of previous panoramic picture is respectively applied for the part 6,4 and 8 of piecing first E together.
Fig. 6 shows and relates to the additional information of generation according to the virtual image region method of the embodiment of the invention.As mentioned above, although there is reference data, if upgrade reference data recently or have low reliability, determine with this reference data use without change, by utilizing padding data to form virtual image region according to reference patch.Determine according to described, carry out mapping or filling.Because also need in the identical mode of essence virtual area be set at receiver side, preferably, transmitter side need send information to receiver side, and described information is about being used for described method each part of virtual image region.When with when one is pieced together the relevant virtual image region of unit and is divided into 8 parts as the embodiment of the invention,, need 8 bits as shown in Figure 6 at this additional information.If pre-defined between transmitter side and receiver side (as shown in the figure) uses the situation of filling to represent with 0, use the situation that reference data is shone upon to represent with 1, can be by sending 8 bit additional informations, in transmitter side and receiver side, all produce the identical virtual image region of essence.For example, if transmitter side does not use the reference data of recent renewal in part 1,4 and 6 with changing, information " 10010100 " need be sent to receiver side.If carry out the filling according to reference patch, transmission information " 00000000 " makes receiver side can set up the identical virtual image region of essence.Although show the example of fixed-length code (FLC) in embodiments of the present invention, also can use the variable length code of the probability distribution of employing information.
Fig. 7 is the flow chart by the operation of carrying out in encoder according to the method for embodiment of the invention generation virtual image region.As shown in Figure 7, in order to determine whether to exist reference data, can in the relevant previous panoramic picture of a current part of piecing unit together of carrying out estimation with hope, consult reference data, in operation S710, carry out search at reference data.At this moment, as mentioned above,, can from the image of opposite edge, retrieve described reference data piecing together under the situation of the most left or rightmost edges that unit is positioned at panoramic picture.If operation have reference data in 720 in operation 730 deterministic retrieval to reference data whether be recent renewal.Be used for determining to be provided with in many ways the fiducial time whether described data are upgraded recently.If determine that described data are recent renewals, in operation 750, reference data is mapped to corresponding part, so that use reference data as the current virtual image region of piecing unit together.
If reference data is not upgraded recently, then the reliability of reference data is lower, so determines to carry out mapping in operation 740 so that variation ground uses this reference data, still carry out the pixel of using on the boundary member of piecing unit together and fill.If determine to carry out mapping, in operation 750, corresponding reference data is mapped to appropriate section, so that use reference data as the current virtual image region of piecing unit together.If determine to carry out filling, in operation S760, carry out and fill, and in operation 795, produce virtual image region with the pixel on the boundary member of reference patch.At this moment, provide, be used for the synchronous of encoder, and can produce described additional information as mentioned above about selecting the additional information of described method.
If there is no reference data is carried out in operation 760 and is utilized the pixel on the boundary member to fill, and is used for a virtual image region of piecing unit together so that produce in operation 795.
In operation 780 and 790, repeat this process, and generation is used for a virtual image region of piecing unit together in operation 795 at whole parts.
Fig. 8 is the block diagram of the equipment that is provided with at the decoder that is used for producing according to the virtual image region of the embodiment of the invention.As shown in Figure 8, the equipment that is used to produce according to the virtual image region of the embodiment of the invention comprises retrieval unit 810, judging unit 815, map unit 820, filler cells 830, additional information generation unit 840 and determining unit 850.
In order to determine in the corresponding previous panoramic picture of each part of the current virtual image region of piecing unit together of carrying out estimation with hope, whether have reference data, retrieval unit 810 search reference datas, and map unit 820 is mapped to corresponding part with the reference data that retrieves.
Judging unit 815 judges whether that recent renewal crosses the data that retrieve, and if not have to upgrade the reference data that retrieves recently, determining unit 850 determines to carry out still filling of mapping.Determine according to this, carry out mapping or filling.In order to produce virtual image region with the identical method of essence in decoder, additional information generation unit 840 produces the information about the method for using when producing virtual image region in the motion estimation process at encoder.
If there is not reference data, if perhaps determining unit 850 is determined to carry out filling, filler cells 830 in previous image with the boundary member of the current corresponding benchmark image in position of piecing unit together on filler pixels so that produce virtual image region.
Fig. 9 is the block diagram according to the adaptable encoder of virtual image region of the embodiment of the invention.
With reference to figure 9, comprise converter unit 910, quantifying unit 915, inverse transformation block 920, inverse quantization unit 925, adder unit 930, amplitude limit unit 940, frame memory 950, panorama image motion estimation unit 960, panorama image 970, subtrator 980 and variable length coder (VLC) 990 according to the panorama image encoder of the embodiment of the invention.
915 pairs of transform coefficient values from converter unit 910 inputs of quantifying unit quantize.The data that quantized are carried out inverse transformation by inverse transformation block 920, and carry out re-quantization, and output sends to the data of decoder end by inverse quantization unit 925.With of previous image in adder unit 930 addition of this view data with motion compensation in panorama image 970, and the final panoramic picture of preparing to send to decoder one side by encoder.To carry out normalization by amplitude limit unit 940 from the panoramic picture of adder unit 930 outputs, be stored in then in the frame memory 950.The image of the panoramic picture of storage before being current encoded image in frame memory 950, and with the estimation of the panoramic picture that acts on new input and the benchmark image of compensation.
Panorama image motion estimation unit 960 uses the reference panorama image of the previous image of storage in frame memory 950 to carry out estimation.Promptly, panorama image motion estimation unit 960 receives current panorama image information, the reference panorama image that use is stored in frame memory 950 is carried out the motion of current panoramic picture and is estimated, produce the motion vector and the additional information of current panoramic picture according to the present invention, and to VLC 990 output movement vector and additional informations.With the tiled patch is that unit carries out estimation and compensation.
Here, in order to produce motion vector, in motion estimation operation, use the method that produces virtual area according to the embodiment of the invention.Therefore, preferably, equipment shown in Figure 8 is arranged in the panorama image motion estimation unit 960.Panorama image 970 utilizes motion vector to carry out motion compensation, and exports the image that has compensated to subtrator 980.Subtrator 980 is to converter unit 910 output present images and by the differential signal between the image of panorama image 970 compensation.Carry out conversion by 910 pairs of described differential signals of conversion fraction, quantize, carry out variable length code by VLC 990 then by quantifying unit 915.
Figure 10 is by producing the flow chart of the operation of carrying out in the decoder according to the virtual image region of the embodiment of the invention.
In decoder one side, receive motion vector, differential signal and additional information, and carry out motion compensation from encoder one side.Be used for producing virtual image region with in the process of the operation 1010 to 1095 of carrying out motion compensation and the operation 710 to 795, the process nature that is used to be created in the virtual image region that encoder one side carries out is identical, so will omit detailed explanation wherein here.
Yet, when the reference data that retrieves, do not have renewal is recently handled, in order to produce and the identical virtual image region of virtual image region essence that produces in encoder one side, the additional information of receiving from encoder one side joint is carried out decipher, and in operation 1100, in each part, carry out mapping or filling according to described decipher.
Figure 11 is the block diagram of the equipment that is provided with at the decoder that is used for producing according to the virtual image region of the embodiment of the invention.
As shown in figure 11, the equipment that is used to produce according to the virtual image region of the embodiment of the invention comprises retrieval unit 200, map unit 210, filler cells 220, judging unit 230 and additional information interpretation unit 240.
In order to determine in the corresponding previous panoramic picture of each part of the current virtual image region of piecing unit together of carrying out estimation with hope, whether have reference data, retrieval unit 200 search reference datas, and map unit 210 is mapped to corresponding part with the reference data that retrieves.Judging unit 230 judges whether upgrade the data that retrieve recently, and if do not have to upgrade the reference data that retrieves recently, in order to determine that carrying out mapping still fills, the additional information interpretation unit carries out decipher to the additional information of receiving from encoder one side joint.According to the content that in additional information, writes down, carry out mapping or filling at each part.
If there is not reference data, if perhaps decoder really order unit 850 determine to carry out filling, filler cells 220 with previous panoramic picture in the current boundary member of the corresponding benchmark image in position of piecing unit together on filler pixels so that produce virtual image region.
Figure 12 is the block diagram that can use according to the decoder of the virtual image region of the embodiment of the invention.
With reference to Figure 12, comprise variable-length decoder (VLD) 310, inverse quantization unit 320, inverse transformation block 330, adder unit 340, panorama image 350, amplitude limit unit 360 and frame memory 370 according to the panoramic picture motion vector decoder of the embodiment of the invention.
VLD310 carries out length-changeable decoding to the bit stream of input.At the output of VLD310, motion vector is outputed to panorama image 350, and differential signal is outputed to inverse quantization unit 320.In frame memory 370, the benchmark image that reproduces by inverse quantization unit 320, inverse transformation block 330 and amplitude limit unit 360 is stored.The benchmark image that to store in frame memory 370 is used for the motion compensation of the panoramic picture of new input.
Can also be with embodiment of the invention written as computer programs, and can in the general purpose digital computer of readable medium recording program performing executive program that uses a computer, carry out.The example of computer readable recording medium storing program for performing comprises magnetic storage medium (for example, ROM, floppy disk, hard disk etc.), optical storage media (for example, CD-ROM or DVD) and such as the storage medium of carrier wave transmission of internet (for example, by) and so on.
Although with reference to exemplary embodiments of the present invention, specifically illustrate and described the present invention, but those of ordinary skills are to be understood that, under the situation of the spirit and scope of the present invention that do not break away from claims and limited, can carry out multiple change on form and the details to these embodiment.This exemplary embodiments is only thought descriptive, and is not the purpose in order to limit.Therefore, scope of the present invention is not to be limited by detailed description of the present invention, but is defined by the following claims, and the whole difference within described scope all will be interpreted as comprising in the present invention.
Claims (16)
1. a generation is used for the method for virtual image region of the estimation of panoramic picture, and described panoramic picture is divided into the tiled patch unit and is sent out from transmitter side, and described method comprises step:
Search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation in previous panoramic picture are so that determine the existence of described reference data; And
Based on described Search Results, described reference data optionally is mapped to described all or part of virtual image region.
2. the step of the method for claim 1, wherein optionally shining upon described reference data comprises:
If Search Results represents to exist reference data, then judge whether to have upgraded in the given time reference data; And
If judged result represents to have upgraded in the given time described reference data, then described reference data is mapped to all or part of virtual image region;
The step of shining upon described reference data also comprises step:
If judged result represents not upgrade in the given time described reference data, then determine to carry out described reference data be mapped to all or part of virtual image region, or carry out with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels; And
According to described definite execution mapping or filling;
The step of optionally shining upon described reference data also comprises step:
If Search Results represents not exist reference data, then by with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels produce all or part of virtual image region.
3. method as claimed in claim 2 also comprises step:
So that send additional information, described additional information allows receiver side to produce in fact in the same manner at all or part of virtual image region of piecing unit together to record about definite information.
4. the method for claim 1, wherein described search step comprises step:
If all or part of virtual image region is arranged in any one outside of the left margin of previous panoramic picture or right margin, then the image recognition from another border to preset range is the reference data at the virtual image region that is present in outside, a described border.
5. a generation is used for the equipment of virtual image region of the estimation of panoramic picture, and described virtual image region is used for being divided into the tiled patch unit and is sent out from transmitter side, and described equipment comprises:
Search unit is used in previous panoramic picture search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation, so that determine the existence of reference data; And
Map unit according to Search Results, optionally is mapped to all or part of virtual image region with described reference data.
6. equipment as claimed in claim 5 also comprises:
Judging unit represents to exist reference data if be used for Search Results, then judges whether to have upgraded in the given time described reference data;
Wherein, if judged result represents to have upgraded within the predetermined time described reference data, then described map unit is carried out mapping;
Determining unit represents not upgrade in the given time described reference data if be used for judged result, then determines to carry out mapping or filling; And
Filler cells, represent not exist reference data if be used for Search Results, then by with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels, produce all or part of virtual image region, if wherein judged result represents not upgrade in the given time described reference data, then according to described determining unit determine carry out mapping or fill.
7. equipment as claimed in claim 6 also comprises:
The additional information generation unit is used to write down about the information determined so that send additional information, and described additional information allows receiver side to produce in fact in the same manner at all or part of virtual image region of piecing unit together.
8. equipment as claimed in claim 5, wherein, if all or part of virtual image region is in the left margin of previous panoramic picture or any one the outside in the right margin, then described search unit is configured to the image recognition from another border to preset range is the reference data at the virtual image region that is present in outside, a described border, and searches for described reference data.
9. a generation is used for the method for virtual image region of the motion compensation of panoramic picture, and described panoramic picture is divided into the tiled patch unit and is received at receiver side, and described method comprises step:
Search and the current corresponding reference data of all or part of virtual image region of piecing unit together of wishing to carry out estimation in previous panoramic picture are so that determine the existence of reference data; And
Based on described Search Results, reference data optionally is mapped to all or part of virtual image region.
10. method as claimed in claim 9, wherein, the step of mapping search reference data comprises step:
If Search Results represents to exist reference data, then judge whether to have upgraded in the given time described reference data; And
If judged result represents to have upgraded in the given time described reference data, then described reference data is mapped to all or part of virtual image region;
If judged result represents not upgrade in the given time described reference data, then determine described reference data to be mapped to all or part of virtual image region by carrying out with reference to the additional information that receives from transmitter side, or carry out with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels; And
According to described definite execution mapping or filling.
11. a generation is used for the equipment of virtual image region of the estimation of panoramic picture, described panoramic picture is divided into the tiled patch unit and is received at receiver side, and described equipment comprises:
Search unit is used in the current corresponding reference data of all or part of virtual image region of piecing unit together of carrying out estimation at previous panoramic picture search and hope, so that determine the existence of reference data; And
Map unit based on Search Results, optionally is mapped to all or part of virtual image region with described reference data;
12. equipment as claimed in claim 11 also comprises:
Judging unit represents to exist reference data if be used for Search Results, then judges whether to upgrade in the given time described reference data;
Wherein, if judged result represents to have upgraded in the given time described reference data, then described map unit is carried out mapping;
The additional information interpretation unit, represent not upgrade in the given time described reference data if be used for the judged result of judging unit, the additional information that receives from transmitter side of decipher then, wherein, described information representation is mapped to all or part of virtual image zone in the transmitter side with described reference data, or with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels; And
Filler cells represents not exist reference data if be used for Search Results, then by with the current corresponding previous panoramic picture in position of piecing unit together in the boundary member of benchmark image on filler pixels, produce all or part of virtual image region,
If wherein the judged result of judging unit represents not upgrade in the given time described reference data, then carry out mapping or filling according to described additional information.
13. the method for the estimation of a panoramic picture, described panoramic picture are divided into the tiled patch unit and are sent out from transmitter side, described method comprises step:
Determine whether be in the previous panoramic picture, and reference data is mapped to all or part of virtual image region with wishing the current corresponding reference data of all or part of virtual image region of piecing unit together of carrying out estimation; And
By having used all or part of virtual image region of reference data mapped, produce the current motion vector of piecing unit relevant with benchmark image.
14. a coding method is used for being divided into the tiled patch unit and encoding from the panoramic picture that transmitter side sends, described method comprises step:
Determine whether be in the previous panoramic picture, and described reference data is mapped to all or part of virtual image region with wishing the current corresponding reference data of all or part of virtual image region of piecing unit together of carrying out estimation;
By having used all or part of virtual image region of reference data mapped, produce the current motion vector of piecing unit relevant with benchmark image; And
Carry out the motion compensation of piecing unit together at current by using described motion vector.
15. the motion compensation process of a panoramic picture that is divided into the tiled patch unit and receives at receiver side, described method comprises step:
Reception is at the current input of piecing the motion vector of unit together that is included in the current panoramic picture;
That determine to represent and whether be in the previous panoramic picture with the current corresponding reference data of all or part of virtual image region of piecing unit together by motion vector;
Described reference data is mapped to all or part of virtual image region; And
Described motion vector has been applied to shine upon all or part of virtual image region of described reference data.
16. the motion compensation process of a panoramic picture, described panoramic picture are divided into the tiled patch unit and are received at receiver side, described method comprises step:
Reception is at the current input of piecing the motion vector of unit together that is included in the current panoramic picture;
That determine to represent and whether be in the previous panoramic picture with the current corresponding reference data of all or part of virtual image region of piecing unit together by motion vector;
Described reference data is mapped to all or part of virtual image region;
Shine upon all or part of virtual image region of described reference data by use, carried out the motion compensation of benchmark image; And
By use current piece together the differential signal of piecing unit together of unit and motion compensation and motion compensation piece unit together, recover the current unit that pieces together.
Applications Claiming Priority (3)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| KR1020040063891 | 2004-08-13 | ||
| KR20040063891 | 2004-08-13 | ||
| KR1020050072969 | 2005-08-09 |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN101002473A CN101002473A (en) | 2007-07-18 |
| CN100581246C true CN100581246C (en) | 2010-01-13 |
Family
ID=37150010
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200580027414A Expired - Fee Related CN100581246C (en) | 2004-08-13 | 2005-08-13 | Method and device for making virtual image region for motion estimation and compensation of panorama image |
Country Status (2)
| Country | Link |
|---|---|
| KR (1) | KR100700218B1 (en) |
| CN (1) | CN100581246C (en) |
Families Citing this family (14)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101521823B (en) * | 2009-03-27 | 2010-12-01 | 北京航空航天大学 | Spatial correlation panoramic data compressing method |
| CN103871038B (en) * | 2014-03-06 | 2014-11-05 | 中国人民解放军国防科学技术大学 | Super-resolution omnidirectional image reconstruction method based on non-uniform measurement matrix |
| CN105554513A (en) * | 2015-12-10 | 2016-05-04 | Tcl集团股份有限公司 | Panoramic video transmission method and system based on H.264 |
| KR20180000279A (en) * | 2016-06-21 | 2018-01-02 | 주식회사 픽스트리 | Apparatus and method for encoding, apparatus and method for decoding |
| WO2017222301A1 (en) * | 2016-06-21 | 2017-12-28 | 주식회사 픽스트리 | Encoding apparatus and method, and decoding apparatus and method |
| CN106101847A (en) | 2016-07-12 | 2016-11-09 | 三星电子（中国）研发中心 | The method and system of panoramic video alternating transmission |
| US10565916B2 (en) | 2016-09-09 | 2020-02-18 | Kt Corporation | Providing streaming of virtual reality contents |
| JP6835397B2 (en) | 2016-09-30 | 2021-02-24 | ホアウェイ・テクノロジーズ・カンパニー・リミテッド | Equipment, methods and programs for encoding and decoding video signals |
| CN110024408B (en) * | 2016-10-04 | 2022-06-24 | 有限公司B1影像技术研究所 | Image data encoding/decoding method and apparatus |
| WO2018066983A1 (en) | 2016-10-04 | 2018-04-12 | 김기백 | Image data encoding/decoding method and apparatus |
| WO2018124822A1 (en) * | 2017-01-02 | 2018-07-05 | 주식회사 케이티 | Method and apparatus for processing video signals |
| WO2018131830A1 (en) * | 2017-01-11 | 2018-07-19 | 주식회사 케이티 | Video signal processing method and device |
| KR102609477B1 (en) * | 2017-02-06 | 2023-12-04 | 삼성전자주식회사 | Electronic Apparatus which generates panorama image or video and the method |
| WO2019027201A1 (en) * | 2017-07-31 | 2019-02-07 | 에스케이텔레콤 주식회사 | Method and device for encoding or decoding 360 image |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR100580876B1 (en) * | 2003-12-08 | 2006-05-16 | 한국전자통신연구원 | Method and Apparatus for Image Compression and Decoding using Bitstream Map, and Recording Medium thereof |
| KR100613752B1 (en) * | 2004-03-11 | 2006-08-22 | 경북대학교 산학협력단 | Streaming method of a panorama video |
-
2005
- 2005-08-09 KR KR1020050072969A patent/KR100700218B1/en active IP Right Grant
- 2005-08-13 CN CN200580027414A patent/CN100581246C/en not_active Expired - Fee Related
Also Published As
| Publication number | Publication date |
|---|---|
| CN101002473A (en) | 2007-07-18 |
| KR100700218B1 (en) | 2007-03-27 |
| KR20060050351A (en) | 2006-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN100581246C (en) | Method and device for making virtual image region for motion estimation and compensation of panorama image | |
| US7627194B2 (en) | Method and device for making virtual image region for motion estimation and compensation of panorama image | |
| US8019169B2 (en) | Image coding apparatus, image decoding apparatus, image processing apparatus and methods thereof | |
| US20180192063A1 (en) | Method and System for Virtual Reality (VR) Video Transcode By Extracting Residual From Different Resolutions | |
| RU2628319C2 (en) | Method and device for determination of supporting images for external prediction | |
| US20060034374A1 (en) | Method and device for motion estimation and compensation for panorama image | |
| US20080205518A1 (en) | Image Coder for Regions of Texture | |
| EP1779662A1 (en) | Method and device for motion estimation and compensation for panorama image | |
| CN104641643A (en) | Decomposition of residual data during signal encoding, decoding and reconstruction in a tiered hierarchy | |
| CN101002471A (en) | Method and apparatus to encode image, and method and apparatus to decode image data | |
| RU2733218C2 (en) | Method, apparatus and a stream for formatting an immersive video image for traditional and immersive playback devices | |
| CN110312138B (en) | High-embedding-capacity video steganography method and system based on time sequence residual convolution modeling | |
| CN112640459B (en) | Image decoding method and apparatus based on motion prediction using merge candidate list in image coding system | |
| US10735766B2 (en) | Point cloud auxiliary information coding | |
| US10997795B2 (en) | Method and apparatus for processing three dimensional object image using point cloud data | |
| CN115396645A (en) | Immersion media data processing method, device, equipment and storage medium | |
| CN104094317A (en) | Method and apparatus for compressing texture information of three-dimensional (3d) models | |
| KR20070008516A (en) | Method and apparatus for image processing in a handheld device | |
| KR20170044028A (en) | Method and apparatus for de-noising an image using video epitome | |
| CN116708793B (en) | Video transmission method, device, equipment and storage medium | |
| EP0911760A2 (en) | Iterated image transformation and decoding apparatus and methods | |
| US20230088144A1 (en) | Data processing method and apparatus for immersive media, related device, and storage medium | |
| Liu et al. | Geometry-guided compact compression for light field image using graph convolutional networks | |
| WO2023081007A1 (en) | Learning-based point cloud compression via adaptive point generation | |
| WO2022271602A1 (en) | Learning-based point cloud compression via unfolding of 3d point clouds |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20100113 Termination date: 20210813 |