CN102222348A - Method for calculating three-dimensional object motion vector - Google Patents

Abstract

The invention discloses a method for calculating a three-dimensional object motion vector. The method comprises the following steps of: detection and matching of object corner: identifying objects in a video steam, labelling the corners of the objects in a basic frame and a key frame, and matching the corners with each other; calculating the three-dimensional motion vector of the key frame in relation to the basic frame based on the matched corners in the basic frame and the key frame; calculating a three-dimensional object model of the key frame based on a basic three-dimensional model and the three-dimensional motion vector, back projecting the three-dimensional object model to a two-dimensional plane, eliminating wrong points and calculating the final three-dimensional object motion vector.

Description

A kind of objective motion vector calculating method

Technical field

The present invention relates to a kind of multi-angle dynamic imaging field of three-dimensional body, the target three-dimensional motion vector computing method during particularly a kind of stereoscopic video with free visual angles dynamically shows.

Background technology

Along with technical development, occurred some in the research field and shown the display terminal of stereoeffect, wherein, free-viewing angle stereo display is active, real dynamic the demonstration, do not rely on parallax and form the stereoscopic vision effect, the observer can independently select observation visual angle and distance.The calculating of target three-dimensional motion vector is a difficult point in the free-viewing angle stereo display, also be stereoscopic vision from static state to dynamic key.By calculating the Three-dimension Target motion vector, can follow the tracks of, the motion of three-dimensional body in the simulation, display video, therefore for video signal processing field, important function and significance is arranged, be necessary procedure crucial in the Stereoscopic Video Presentation.

Summary of the invention

Goal of the invention: technical matters to be solved by this invention is at the Stereoscopic Video Presentation system, and a kind of linear method that calculates the target three-dimensional motion vector is provided.

Technical scheme: the invention discloses the calculating of three-dimensional motion vector in a kind of Stereoscopic Video Presentation, may further comprise the steps:

Step (1), target Corner Detection and coupling: the target in the identification video stream marks out the angle point of target in basic frame and the key frame, and mates;

Step (2), according to the angle point that mates in basic frame and the key frame, the primary Calculation key frame is with respect to the three-dimensional motion of basic frame;

Step (3), according to the target three-dimensional model of motion and basic three-dimensional model calculating key frame, it is overdue that the two dimensional surface rejecting is returned by back projection, calculates final three-dimensional motion vector.

Among the present invention, step (1) may further comprise the steps:

Step (11), the target in identification key frame and the basic frame;

Step (12) marks out the angle point of target in basic frame and the key frame;

Step (13) is mated key frame and basic frame: the corresponding angle point and the two-dimensional coordinate thereof that find key frame and basic frame.

Among the present invention, step (2) may further comprise the steps:

Based on video camera 1 and video camera 2:

Use approximation method to draw system of linear equations:

$\left[\begin{array}{c}{{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">x</figure-callout>}}_1}^{\&prime;}-p_{14}-X_1{p}_{11}-Y_1{p}_{12}-Z_1{p}_{13}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}-X_1{p}_{21}-Y_1{p}_{22}-Z_1{p}_{23}\\ M\\ M\\ {x_n}^{\&prime;}-{\mathrm{pp}}_{14}-X_n{\mathrm{pp}}_{11}-Y_n{\mathrm{pp}}_{12}-Z_n{\mathrm{pp}}_{13}\\ {y_n}^{\&prime;}-{\mathrm{pp}}_{24}-X_n{\mathrm{pp}}_{21}-Y_n{\mathrm{pp}}_{22}-Z_n{\mathrm{pp}}_{23}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_{11}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccc}{Y}_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\\ M\\ M\\ \begin{array}{cccccc}{Y}_n{\mathrm{pp}}_{13}-Z_n{\mathrm{pp}}_{12}& Z_n{\mathrm{pp}}_{11}-X_n{\mathrm{pp}}_{13}& X_n{\mathrm{pp}}_{12}-{\mathrm{<figure-callout\; id="11"\; label="Y\; n\; pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Y</figure-callout>}}_n{\mathrm{pp}}_{}{}_{11}& {\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccc}{Y}_n{\mathrm{pp}}_{23}-Z_n{\mathrm{pp}}_{22}& Z_n{\mathrm{pp}}_{21}-X_n{\mathrm{pp}}_{23}& X_n{\mathrm{pp}}_{22}-Y_n{\mathrm{pp}}_{21}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{\mathrm{\&phi;}}_1\\ {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3\\ t_1\\ t_2\\ {\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3\end{array}\right],$

φ ₁, φ ₂, φ ₃Be respectively the three-dimensional motion vector reduced parameter, t ₁, t ₂, t ₃Be respectively the translation matrix parameter; Form three-dimensional motion vector M:

$M=\left[\begin{array}{cccc}1& -{\mathrm{\&phi;}}_3& {\mathrm{\&phi;}}_2& {\mathrm{<figure-callout\; id="1"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_1\\ {\mathrm{\&phi;}}_3& 1& -{\mathrm{\&phi;}}_1& t_2\\ -{\mathrm{\&phi;}}_2& {\mathrm{\&phi;}}_1& 1& {\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3\\ 0& 0& 0& 1\end{array}\right];$

Or use non-approximation method, draw system of linear equations:

$\left[\begin{array}{c}{{\mathrm{<figure-callout\; id="1"\; label="x"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">x</figure-callout>}}_1}^{\&prime;}-p_{14}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}\\ M\\ M\\ M\\ M\\ M\\ M\\ M\\ {x_n}^{\&prime;}-{\mathrm{pp}}_{14}\\ {y_n}^{\&prime;}-{\mathrm{pp}}_{24}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccccccccc}{X}_1{p}_{11}& X_1{p}_{12}& X_1{p}_{13}& Y_1{p}_{11}& Y_1{p}_{12}& Y_1{p}_{13}& Z_1{p}_{11}& Z_1{p}_{12}& Z_1{p}_{13}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_1{p}_{21}& X_1{p}_{22}& X_1{p}_{23}& Y_1{p}_{21}& Y_1{p}_{22}& Y_1{p}_{23}& Z_1{p}_{21}& Z_1{p}_{22}& Z_1{p}_{23}& p_{21}& p_{22}& p_{23}\end{array}\\ .\\ M\\ M\\ M\\ M\\ M\\ M\\ \begin{array}{cccccccccccc}{X}_n{\mathrm{pp}}_{11}& X_n{\mathrm{pp}}_{12}& X_n{\mathrm{pp}}_{13}& Y_n{\mathrm{pp}}_{11}& Y_n{\mathrm{pp}}_{12}& Y_n{\mathrm{pp}}_{13}& Z_n{\mathrm{pp}}_{11}& Z_n{\mathrm{pp}}_{12}& Z_n{\mathrm{pp}}_{13}& {\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_n{\mathrm{pp}}_{21}& X_n{\mathrm{pp}}_{22}& X_n{\mathrm{pp}}_{23}& Y_n{\mathrm{pp}}_{21}& Y_n{\mathrm{pp}}_{22}& Y_n{\mathrm{pp}}_{23}& Z_n{\mathrm{pp}}_{21}& Z_n{\mathrm{pp}}_{22}& Z_n{\mathrm{pp}}_{23}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{r}_{11}\\ r_{21}\\ r_{31}\\ r_{12}\\ r_{22}\\ r_{32}\\ r_{13}\\ r_{23}\\ r_{33}\\ t_1\\ t_2\\ {\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3\end{array}\right]$

, r wherein ₁₁, r ₂₁, r ₃₁, r ₁₂, r ₂₂, r ₃₂, r ₁₃, r ₂₃, r ₃₃Be the rotation matrix parameter, form three-dimensional motion vector M:

$M=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_1\\ r_{21}& r_{22}& r_{23}& t_2\\ r_{31}& r_{32}& r_{33}& {\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3\\ 0& 0& 0& 1\end{array}\right];$

(X wherein _i, Y _i, Z _i) any some three-dimensional coordinate in the basic frame of video camera 1 or video camera 2 of expression;

(X _i', Y _i', Z _i') represent this three-dimensional coordinate in the key frame of video camera 1 or video camera 2; (x _i, y _i) represent that this projects to the two-dimensional coordinate of two dimensional surface at video camera 1 or video camera 2 in basic frame; (x _i', y _i') represent that this projects to the two-dimensional coordinate of two dimensional surface at video camera 1 or video camera 2 in key frame;

$P=\left[\begin{array}{cccc}{p}_{11}& p_{12}& p_{13}& p_{14}\\ p_{21}& p_{22}& p_{23}& p_{24}\\ 0& 0& 0& 1\end{array}\right]$ Projection matrix for video camera 1;

$\mathrm{PP}=\left[\begin{array}{cccc}{\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}& {\mathrm{pp}}_{14}\\ {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}& {\mathrm{pp}}_{24}\\ 0& 0& 0& 1\end{array}\right]$ Projection matrix for video camera 2;

Find the solution the least mean-square error of system of linear equations and separate, obtain the kinematic matrix parameter, i.e. motion vector.

Among the present invention, step (3) may further comprise the steps:

To two video cameras, described kinematic matrix parameter is additional to basic three-dimensional model, obtain the three-dimensional model of two camera video key frames;

To two video cameras, respectively with the three-dimensional model of two camera video key frames, the two dimensional surface of video camera matrix correspondence separately returns in back projection, with the angle point two-dimensional coordinate of the key frame of corresponding video camera relatively, reject the point of error beyond threshold value, use step 2 to recomputate the kinematic matrix parameter, obtain final three-dimensional motion motion vector.

Beneficial effect: the present invention calculates relative motion fast and obtains the key frame three-dimensional model key frame according to the static three-dimensional model of basic frame, so that obtain the motion video at any visual angle of Same Scene.The present invention does not need camera calibration, and real-time is good, and the smoothness that helps dynamic video shows, and algorithm has certain robustness.In calculating the free-viewing angle display system, can carry out three-dimensional reconstruction consuming time every the long period, and calculate motion in real time with the inventive method, adjust the position of three-dimensional body, make to show that real-time is better, do not sacrifice precision simultaneously.

Description of drawings

Below in conjunction with the drawings and specific embodiments the present invention is done further to specify, above-mentioned and/or otherwise advantage of the present invention will become apparent.

Fig. 1 is video sequence basis frame of the present invention and key frame signal.

The coordinate system perspective view that Fig. 2 is involved in the present invention.

Fig. 3 is the control experiment comparing result of the present invention with existing research.

Embodiment:

The inventive method adopts twin camera (can expand to multiple-camera) on the basis of static three-dimensional reconstruction, according to two angle motion video pictures of synchronous shooting Same Scene, calculate the target three-dimensional motion vector, shows so that realize the dynamic solid video.

During the invention process, the condition that needs to satisfy is:

Video flowing is synchronous: the collection that starts video flowing comes the two-dimensional scene of synchronous different visual angles camera head shooting; Concerning embodiment, be embodied as the picture frame that adopts with the moment two video cameras, as Fig. 1.

During the invention process, Given information has:

Basis frame and basic three-dimensional model: basic three-dimensional model is that the center is set up with basic frame, and basic frame comprises the basic frame of two video cameras.

The projection matrix of two video cameras: rebuild in the process of basic three-dimensional model and can obtain the video camera projection matrix.

Among the present invention, on the completed basis of static three-dimensional reconstruction, the two-dimensional video picture according to two video cameras provide calculates the Three-dimension Target motion vector.As shown in Figure 1, the image sequence that the optional interval of key frame equates, present embodiment choose wherein that a frame calculates its motion vector with respect to basic frame, and other frame method is identical.

Implementation process is as follows:

One, target Corner Detection and coupling: discern two video cameras target in the video flowing separately, mark out in two basic separately frames of video camera and the key frame angle point on the target;

Can be 200910234584.8 in the application number of application on November 23rd, 2009 referring to the applicant, name be called the patent of invention of " a kind of method for displaying stereoscopic video with free visual angles ".Concise and to the point step is:

1.1 with the Harris algorithm angle point in the basic frame being carried out the first time estimates;

1.2 the angle point that estimates is further screened with the SUSAN algorithm, draws final angle point.

1.3 key frame and basic frame are mated: the two-dimensional coordinate (x that finds key frame with the template window matching process ₁, y ₁) and the two-dimensional coordinate (x of the corresponding angle point of basic frame ₁', y ₁').

Use above method, for two video cameras, its key frame is mated to basic frame respectively.

Two, calculate the three-dimensional motion of key frame with respect to basic frame.

The present invention adopts the affine camera model under the homogeneous coordinates, and is suitable when its degree of depth can be ignored relatively in the change in depth of target as shown in Figure 2, realistic video camera.

Consider video camera 1 earlier.For more arbitrarily, establish that three-dimensional coordinate is respectively (X in basic frame and the key frame ₁, Y ₁, Z ₁) and (X ₁', Y ₁', Z ₁'), two-dimensional coordinate is respectively (x ₁, y ₁) and (x ₁', y ₁').

Three-dimensional coordinate is write to the mapping of two-dimensional coordinate:

$\left[\begin{array}{c}{x}_1\\ y_1\\ 1\end{array}\right]=P\left[\begin{array}{c}{X}_1\\ Y_1\\ Z_1\\ 1\end{array}\right]---\left(1\right)$

Wherein $P=\left[\begin{array}{cccc}{p}_{11}& p_{12}& p_{13}& p_{14}\\ p_{21}& p_{22}& p_{23}& p_{24}\\ 0& 0& 0& 1\end{array}\right]$ Be the video camera projection matrix.

Then key frame can be expressed as with respect to the motion of basic frame:

$\left[\begin{array}{c}{{\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">X</figure-callout>}}_1}^{\&prime;}\\ {Y_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1}^{\&prime;}\\ 1\end{array}\right]=M\left[\begin{array}{c}{X}_1\\ Y_1\\ Z_1\\ 1\end{array}\right]---\left(2\right)$

Wherein kinematic matrix M is: $M=\left[\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="R\; T\; O"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">R</figure-callout>}& T\\ O& 1\end{array}\right]=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_1\\ r_{21}& r_{22}& r_{23}& t_2\\ r_{31}& r_{32}& r_{33}& {\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3\\ 0& 0& 0& 1\end{array}\right]---\left(3\right)$

Wherein R is a rotation matrix, and T is a translation matrix, and all elements of 1 * 3 matrix O is zero.Though rotation matrix R has 9 elements, it is an orthogonal matrix, has 6 independent restrainings:

Thereby real independent parameter has only 3.Can be by Roll-Pitch-Yaw (x of right-handed Cartesian coordinate system, y and z axle) method representation rotation matrix, θ is the anglec of rotation:

$R=\left[\begin{array}{ccc}\cos {\mathrm{\&theta;}}_y\cos {\mathrm{\&theta;}}_z& -\cos {\mathrm{\&theta;}}_y\sin {\mathrm{\&theta;}}_z& \sin {\mathrm{\&theta;}}_y\\ \sin {\mathrm{\&theta;}}_x\sin {\mathrm{\&theta;}}_y\cos {\mathrm{\&theta;}}_z+\cos {\mathrm{\&theta;}}_x\sin {\mathrm{\&theta;}}_z& -\sin {\mathrm{\&theta;}}_x\sin {\mathrm{\&theta;}}_y\sin {\mathrm{\&theta;}}_z+\cos {\mathrm{\&theta;}}_x\cos {\mathrm{\&theta;}}_z& -\sin {\mathrm{\&theta;}}_x\cos {\mathrm{\&theta;}}_y\\ -\cos {\mathrm{\&theta;}}_x\sin {\mathrm{\&theta;}}_y\cos {\mathrm{\&theta;}}_z+\sin {\mathrm{\&theta;}}_x\sin {\mathrm{\&theta;}}_z& \cos {\mathrm{\&theta;}}_x\sin {\mathrm{\&theta;}}_y\sin {\mathrm{\&theta;}}_z+\sin {\mathrm{\&theta;}}_x\cos {\mathrm{\&theta;}}_z& \cos {\mathrm{\&theta;}}_x\cos {\mathrm{\&theta;}}_y\end{array}\right]---\left(4\right)$

Finding the solution of motion is divided into 2 steps:

2.1 will finding the solution motion, step 1. is summed up as the group b=Ax that solves an equation

By $\left[\begin{array}{c}{x}_1\\ {\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1\\ 1\end{array}\right]=P\left[\begin{array}{c}{X}_1\\ Y_1\\ {\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1\\ 1\end{array}\right]\left[\begin{array}{c}{x_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}\\ 1\end{array}\right]=\mathrm{<figure-callout\; id="1"\; label="P\; X"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">P</figure-callout>}\left[\begin{array}{c}{X_{}}^{}\end{array}\right]\left[\begin{array}{c}{{}_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Y</figure-callout>}}_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1}^{\&prime;}\\ 1\end{array}\right]\left[\begin{array}{c}{{\mathrm{<figure-callout\; id="1"\; label="X"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">X</figure-callout>}}_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="Y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Y</figure-callout>}}_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1}^{\&prime;}\\ 1\end{array}\right]=M\left[\begin{array}{c}{X}_1\\ Y_1\\ {\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1\\ 1\end{array}\right]$ Three formulas can get:

$\left[\begin{array}{c}{x_1}^{\&prime;}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}\\ 1\end{array}\right]=\mathrm{PM}\left[\begin{array}{c}{X}_1\\ Y_1\\ {\mathrm{<figure-callout\; id="1"\; label="Z"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Z</figure-callout>}}_1\\ 1\end{array}\right]---\left(5\right)$

Each point provides two equations, and is as follows:

$\left\{\begin{array}{c}{x_1}^{\&prime;}=p_{14}+p_{11}{\mathrm{<figure-callout\; id="1"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_1+p_{12}{\mathrm{<figure-callout\; id="2"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_2+p_{13}{\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3+X_1(p_{11}{\mathrm{<figure-callout\; id="11"\; label="r"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">r</figure-callout>}}_{11}+p_{12}{r}_{21}+p_{13}{r}_{31})\\ +Y_1(p_{11}{\mathrm{<figure-callout\; id="12"\; label="r"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">r</figure-callout>}}_{12}+p_{12}{r}_{22}+p_{13}{r}_{32})+Z_1(p_{11}{r}_{13}+p_{12}{r}_{23}+p_{13}{r}_{33})\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}=p_{24}+p_{21}{\mathrm{<figure-callout\; id="1"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_1+p_{22}{\mathrm{<figure-callout\; id="2"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_2+p_{23}{\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3+X_1(p_{21}{\mathrm{<figure-callout\; id="11"\; label="r"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">r</figure-callout>}}_{11}+p_{22}{r}_{21}+p_{23}{r}_{31})\\ +Y_1(p_{21}{\mathrm{<figure-callout\; id="12"\; label="r"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">r</figure-callout>}}_{12}+p_{22}{r}_{22}+p_{23}{r}_{32})+Z_1(p_{21}{r}_{13}+p_{22}{r}_{23}+p_{23}{r}_{33})\end{array}\right.---\left(6\right)$

1. " if make small angle approximation (key frame is separated by when nearer, and it is less move, can make small angle approximation), and the Roll of employing (6) formula, Pitch and Yaw method for expressing, rotation matrix is reduced to, φ ₁, φ ₂, φ ₃Be respectively the three-dimensional motion vector reduced parameter:

$R=\left[\begin{array}{ccc}1& -{\mathrm{<figure-callout\; id="3"\; label="n"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">n</figure-callout>}}_3\mathrm{\&theta;}& {\mathrm{<figure-callout\; id="2"\; label="n"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">n</figure-callout>}}_2\mathrm{\&theta;}\\ {\mathrm{<figure-callout\; id="3"\; label="n"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">n</figure-callout>}}_3\mathrm{\&theta;}& 1& -{\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">n</figure-callout>}}_1\mathrm{\&theta;}\\ -{\mathrm{<figure-callout\; id="2"\; label="n"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">n</figure-callout>}}_2\mathrm{\&theta;}& {\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">n</figure-callout>}}_1\mathrm{\&theta;}& 1\end{array}\right]=\left[\begin{array}{ccc}1& -{\mathrm{\&phi;}}_3& {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3& 1& -{\mathrm{\&phi;}}_1\\ -{\mathrm{\&phi;}}_2& {\mathrm{\&phi;}}_1& 1\end{array}\right]---\left(7\right)$

(n wherein ₁θ) ²+ (n ₂θ) ²+ (n ₃θ) ²=θ ²So, φ ₁ ²+ φ ₂ ²+ φ ₃ ²=θ ²(8)

System of equations (6) is rewritten as:

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}-X_1{p}_{11}-Y_1{p}_{12}-Z_1{p}_{13}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}-X_1{p}_{21}-Y_1{p}_{22}-Z_1{p}_{23}\end{array}\right]=\left[\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_{11}& p_{11}& p_{12}& p_{13}\\ Y_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\right]\left[\begin{array}{c}{\mathrm{\&phi;}}_1\\ {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3\\ t_1\\ t_2\\ t_3\end{array}\right]---\left(9\right)$

As only using the multiple spot of a video camera, can list system of equations:

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}-X_1{p}_{11}-Y_1{p}_{12}-Z_1{p}_{13}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}-X_1{p}_{21}-Y_1{p}_{22}-Z_1{p}_{23}\\ .\\ .\\ {x_n}^{\&prime;}-p_{14}-X_n{p}_{11}-Y_n{p}_{12}-Z_n{p}_{13}\\ {y_n}^{\&prime;}-p_{24}-X_n{p}_{21}-Y_n{p}_{22}-Z_n{p}_{23}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_{11}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccc}{Y}_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\\ .\\ .\\ \begin{array}{cccccc}{Y}_n{p}_{13}-Z_n{p}_{12}& Z_n{p}_{11}-X_n{p}_{13}& X_n{p}_{12}-Y_n{p}_{11}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccc}{Y}_n{p}_{23}-Z_n{p}_{22}& Z_n{p}_{21}-X_n{p}_{23}& X_n{p}_{22}-Y_n{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{\mathrm{\&phi;}}_1\\ {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3\\ t_1\\ t_2\\ t_3\end{array}\right]---\left(10\right)$

First matrix of equation the right is designated as A in the following formula, and its order is 5 to the maximum.This is because a video camera can not provide along the depth information of its optical axis direction.Issued a certificate below:

I. observation matrix A finds that right three row interlacing are identical.This matrix is done line translation, with the first row cancellation 2i-1 capable after three row, with the second row cancellation 2i capable after three row (i=2,3 ... n):

$\left[\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_1& p_{11}& p_{12}& p_{13}\\ Y_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\\ & \mathrm{<figure-callout\; id="0"\; label="M"\; filenames="HDA0000071689490000012.png"\; state="\{\{state\}\}">M</figure-callout>}& & 0& 0& 0\\ & \mathrm{<figure-callout\; id="0"\; label="M\; M\; M"\; filenames="HDA0000071689490000012.png"\; state="\{\{state\}\}">M</figure-callout>}& & & M& \\ & M& & 0& 0& 0\\ Y_n{p}_{13}-Z_n{p}_{12}& Z_n{p}_{11}-X_n{p}_{13}& X_n{p}_{12}-{\mathrm{<figure-callout\; id="11"\; label="Y\; n\; p"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Y</figure-callout>}}_n{p}_{}{}_{11}& 0& 0& 0\\ Y_n{p}_{23}-Z_n{p}_{22}& Z_n{p}_{21}-X_n{p}_{23}& X_n{p}_{22}-Y_n{p}_{21}& 0& 0& 0\end{array}\right]---\left(11\right)$

Ii. in the matrix that obtains, right three column ranks are 2 to the maximum because non-vanishing have only two the row; A left side three column ranks are 3 to the maximum.So this rank of matrix is 5 to the maximum.The elementary row rank transformation does not change rank of matrix, so the order of matrix A is 5 to the maximum.

By above-mentioned proof as can be known, find the solution the rigid body three-dimensional motion, need two video cameras at least.Second video camera projection matrix is:

$\mathrm{PP}=\left[\begin{array}{cccc}{\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}& {\mathrm{pp}}_{14}\\ {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}& {\mathrm{pp}}_{24}\\ 0& 0& 0& 1\end{array}\right]$

So will find the solution motion, the group that need establish an equation to the multiple spot of two video cameras is:

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}-X_1{p}_{11}-Y_1{p}_{12}-Z_1{p}_{13}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}-X_1{p}_{21}-Y_1{p}_{22}-Z_1{p}_{23}\\ M\\ M\\ {x_n}^{\&prime;}-{\mathrm{pp}}_{14}-X_n{\mathrm{pp}}_{11}-Y_n{\mathrm{pp}}_{12}-Z_n{\mathrm{pp}}_{13}\\ {y_n}^{\&prime;}-{\mathrm{pp}}_{24}-X_n{\mathrm{pp}}_{21}-Y_n{\mathrm{pp}}_{22}-Z_n{\mathrm{pp}}_{23}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_{11}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccc}{Y}_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\\ M\\ M\\ \begin{array}{cccccc}{Y}_n{\mathrm{pp}}_{13}-Z_n{\mathrm{pp}}_{12}& Z_n{\mathrm{pp}}_{11}-X_n{\mathrm{pp}}_{13}& X_n{\mathrm{pp}}_{12}-{\mathrm{<figure-callout\; id="11"\; label="Y\; n\; pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Y</figure-callout>}}_n{\mathrm{pp}}_{}{}_{11}& {\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccc}{Y}_n{\mathrm{pp}}_{23}-Z_n{\mathrm{pp}}_{22}& Z_n{\mathrm{pp}}_{21}-X_n{\mathrm{pp}}_{23}& X_n{\mathrm{pp}}_{22}-Y_n{\mathrm{pp}}_{21}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{\mathrm{\&phi;}}_1\\ {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3\\ t_1\\ t_2\\ t_3\end{array}\right]---\left(12\right)$

Note equation left side matrix is b, and the right matrix is followed successively by A and x.This moment, the order of A was 6 to the maximum.. then finding the solution conversion of motion is solving equation group b=Ax.

N point comprises m point of video camera 1, and the k of video camera 2 point.N=m+k (m, k 〉=1). in theory, 3 groups of points that n=3 promptly uses two video cameras to the time, can solve system of equations; When n 〉=3, need separate the overdetermined equation group, obtain a result.

Herein, equation only needs to add newline in the matrix bottom also widenable to multiple-camera, adds a plurality of video camera equations, and equation form is identical.

2. if do not make small angle approximation, need two video cameras equally, system of equations is:

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}\\ .\\ .\\ .\\ .\\ .\\ .\\ .\\ {x_n}^{\&prime;}-{\mathrm{pp}}_{14}\\ {y_n}^{\&prime;}-{\mathrm{pp}}_{24}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccccccccc}{X}_1{p}_{11}& X_1{p}_{12}& X_1{p}_{13}& Y_1{p}_{11}& Y_1{p}_{12}& Y_1{p}_{13}& Z_1{p}_{11}& Z_1{p}_{12}& Z_1{p}_{13}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_1{p}_{21}& X_1{p}_{22}& X_1{p}_{23}& Y_1{p}_{21}& Y_1{p}_{22}& Y_1{p}_{23}& Z_1{p}_{21}& Z_1{p}_{22}& Z_1{p}_{23}& p_{21}& p_{22}& p_{23}\end{array}\\ .\\ .\\ .\\ .\\ .\\ .\\ .\\ \begin{array}{cccccccccccc}{X}_n{\mathrm{pp}}_{11}& X_n{\mathrm{pp}}_{12}& X_n{\mathrm{pp}}_{13}& Y_n{\mathrm{pp}}_{11}& Y_n{\mathrm{pp}}_{12}& Y_n{\mathrm{pp}}_{13}& Z_n{\mathrm{pp}}_{11}& Z_n{\mathrm{pp}}_{12}& Z_n{\mathrm{pp}}_{13}& {\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_n{\mathrm{pp}}_{21}& X_n{\mathrm{pp}}_{22}& X_n{\mathrm{pp}}_{23}& Y_n{\mathrm{pp}}_{21}& Y_n{\mathrm{pp}}_{22}& Y_n{\mathrm{pp}}_{23}& Z_n{\mathrm{pp}}_{21}& Z_n{\mathrm{pp}}_{22}& Z_n{\mathrm{pp}}_{23}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{r}_{11}\\ r_{21}\\ r_{31}\\ r_{12}\\ r_{22}\\ r_{32}\\ r_{13}\\ r_{23}\\ r_{33}\\ t_1\\ t_2\\ t_3\end{array}\right]---\left(13\right)$

For non-approximate data, in order to guarantee the matrix A full rank, need satisfy m 〉=4 and n 〉=4, altogether the minimum 4+4=8 group of feature point that needs.

Equally, equation is also widenable to multiple-camera.

2.2 step 2. is found the solution the least mean-square error of system of linear equations b=Ax and is separated

Adopt QR (orthonomal matrix-upper triangular matrix) decomposition method, solution is as follows:

Since the existence of error, Ax=b+ ε, problem is to find the solution x to make norm || ε || ₂ ²Minimum.

Can find Q to make $\mathrm{QA}=\left(\begin{array}{c}R\\ O\end{array}\right).$ Wherein Q is an orthogonal matrix, and R is nonsingular upper triangular matrix.

${{\left|\right|\mathrm{\&epsiv;}\left|\right|}_2}^2={{\left|\right|\mathrm{Ax}-b\left|\right|}_2}^2={{\left|\right|Q(\mathrm{Ax}-b)\left|\right|}_2}^2={{\left|\right|\mathrm{QAx}-\mathrm{Qb}\left|\right|}_2}^2={{\left|\right|\left[\begin{array}{c}R\\ O\end{array}\right]-\mathrm{Qb}\left|\right|}_2}^2$

Note $\mathrm{Qb}=\left[\begin{array}{c}{b}_1\\ {\mathrm{<figure-callout\; id="2"\; label="b"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">b</figure-callout>}}_2\end{array}\right],$ ${{\left|\right|\mathrm{Ax}-b\left|\right|}_2}^2={{\left|\right|\left[\begin{array}{c}\mathrm{Rx}-\mathrm{<figure-callout\; id="2"\; label="b\; b"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">b</figure-callout>}& \\ b_{}{}_2\end{array}\right]\left|\right|}_2}^2.$ Because be column vector, so=|| Rx-b ₁|| ₂ ²+ || b ₂|| ₂ ²

Notice || b ₂|| ₂ ²Be constant, so former problem || Ax-b|| ₂ ²Min is converted into || Rx-b ₁|| ₂ ²Min, that is: Rx-b ₁=0.

So have:

x＝R ^-1b ₁ (14)

So far, the x vector solves.

To the approximate data of system of equations (12), x=(φ ₁, φ ₂, φ ₃, t ₁, t ₂, t ₃) ^T,

To the non-approximate data of system of equations (13), x=(r ₁₁, r ₂₁, r ₃₁, r ₁₂, r ₂₂, r ₃₂, r ₁₃, r ₂₃, r ₃₃, t ₁, t ₂, t ₃) ^T.

Three, the three-dimensional motion vector of calculating according to step 2 is calculated the target three-dimensional model of key frame, to two video cameras, the two dimensional surface of this video camera matrix correspondence returns in back projection respectively, compare with the key frame angle point two-dimensional coordinate of this video camera, reject the point of error beyond threshold value, two calculate three-dimensional motion vector again set by step then.

The mistake coupling can cause back projection's error very big, and this error directly influences result of calculation, so that the present invention proposes to reject is overdue.Because back projection's error that mistake coupling causes is generally at 10 more than the pixel, even 20, and general normal point is many in 2 pixels, is 10 so establish threshold value.

As long as surpass threshold value in x direction or y deflection error, promptly as overdue rejecting.(the individual point of l≤n) carries out step 2 again, recomputates three-dimensional motion vector by (15) formula or (16) formula for the l that keeps then.

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}-X_1{p}_{11}-Y_1{p}_{12}-Z_1{p}_{13}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}-X_1{p}_{21}-Y_1{p}_{22}-Z_1{p}_{23}\\ M\\ M\\ {x_l}^{\&prime;}-{\mathrm{pp}}_{14}-X_l{\mathrm{pp}}_{11}-Y_l{\mathrm{pp}}_{12}-Z_l{\mathrm{pp}}_{13}\\ {y_l}^{\&prime;}-{\mathrm{pp}}_{24}-X_l{\mathrm{pp}}_{21}-Y_l{\mathrm{pp}}_{22}-Z_l{\mathrm{pp}}_{23}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_{11}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccc}{Y}_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\\ M\\ M\\ \begin{array}{cccccc}{Y}_l{\mathrm{pp}}_{13}-Z_l{\mathrm{pp}}_{12}& Z_l{\mathrm{pp}}_{11}-X_l{\mathrm{pp}}_{13}& X_l{\mathrm{pp}}_{12}-{\mathrm{<figure-callout\; id="11"\; label="Y\; l\; pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">Y</figure-callout>}}_l{\mathrm{pp}}_{}{}_{11}& {\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccc}{Y}_l{\mathrm{pp}}_{23}-Z_l{\mathrm{pp}}_{22}& Z_l{\mathrm{pp}}_{21}-X_l{\mathrm{pp}}_{23}& X_l{\mathrm{pp}}_{22}-Y_l{\mathrm{pp}}_{21}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{\mathrm{\&phi;}}_1\\ {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3\\ t_1\\ t_2\\ t_3\end{array}\right]---\left(15\right)$

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}\\ {{\mathrm{<figure-callout\; id="1"\; label="y"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">y</figure-callout>}}_1}^{\&prime;}-p_{24}\\ .\\ .\\ .\\ .\\ .\\ .\\ .\\ {x_l}^{\&prime;}-{\mathrm{pp}}_{14}\\ {y_l}^{\&prime;}-{\mathrm{pp}}_{24}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccccccccc}{X}_1{p}_{11}& X_1{p}_{12}& X_1{p}_{13}& Y_1{p}_{11}& Y_1{p}_{12}& Y_1{p}_{13}& Z_1{p}_{11}& Z_1{p}_{12}& Z_1{p}_{13}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_1{p}_{21}& X_1{p}_{22}& X_1{p}_{23}& Y_1{p}_{21}& Y_1{p}_{22}& Y_1{p}_{23}& Z_1{p}_{21}& Z_1{p}_{22}& Z_1{p}_{23}& p_{21}& p_{22}& p_{23}\end{array}\\ .\\ .\\ .\\ .\\ .\\ .\\ .\\ \begin{array}{cccccccccccc}{X}_l{\mathrm{pp}}_{11}& X_l{\mathrm{pp}}_{12}& X_l{\mathrm{pp}}_{13}& Y_l{\mathrm{pp}}_{11}& Y_l{\mathrm{pp}}_{12}& Y_l{\mathrm{pp}}_{13}& Z_l{\mathrm{pp}}_{11}& Z_l{\mathrm{pp}}_{12}& Z_l{\mathrm{pp}}_{13}& {\mathrm{<figure-callout\; id="11"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{11}& {\mathrm{<figure-callout\; id="12"\; label="pp"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">pp</figure-callout>}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_l{\mathrm{pp}}_{21}& X_l{\mathrm{pp}}_{22}& X_l{\mathrm{pp}}_{23}& Y_l{\mathrm{pp}}_{21}& Y_l{\mathrm{pp}}_{22}& Y_l{\mathrm{pp}}_{23}& Z_l{\mathrm{pp}}_{21}& Z_l{\mathrm{pp}}_{22}& Z_l{\mathrm{pp}}_{23}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{r}_{11}\\ r_{21}\\ r_{31}\\ r_{12}\\ r_{22}\\ r_{32}\\ r_{13}\\ r_{23}\\ r_{33}\\ t_1\\ t_2\\ {\mathrm{<figure-callout\; id="3"\; label="t"\; filenames="HDA0000071689490000021.png"\; state="\{\{state\}\}">t</figure-callout>}}_3\end{array}\right]$

(16)

Think that the result is final three-dimensional motion vector.

Comparatively ripe Han-Kanade method in the present invention and the existing research has been done the control experiment.The Han-Kanade method is based on the three-dimensional reconstruction of uncalibrated image not, also can recover motion.Experiment embodiment is: real-world object has rotated 5 degree around the x direction, and the control input point is identical, obtains calculating the anglec of rotation to such as Fig. 3.Precision is in same level as can be seen.And on the time, MATLAB realizes that the computing of one time three frame needs 5～10 minutes, and comprehensive 360 degree are rebuild object just needs the more time.5 video cameras needed more than half an hour.And video camera is few more, and is high more to the requirement of wide-angle coupling, so can not reduce video camera simply.Therefore, in the free-viewing angle display system, can carry out three-dimensional reconstruction consuming time every the long period, and calculate motion in real time, adjust the position of three-dimensional body, make to show that real-time is better, not sacrifice precision simultaneously with the inventive method.

The invention provides the thinking and the method for three-dimensional vectors computing method in a kind of Stereoscopic Video Presentation; the method and the approach of this technical scheme of specific implementation are a lot; the above only is a preferred implementation of the present invention; should be understood that; for those skilled in the art; under the prerequisite that does not break away from the principle of the invention, can also make some improvements and modifications, these improvements and modifications also should be considered as protection scope of the present invention.The all available prior art of each ingredient not clear and definite in the present embodiment is realized.

Claims (4)

1. target three-dimensional motion vector computing method is characterized in that, may further comprise the steps:

Step (1), target Corner Detection and coupling: the target in the identification video stream marks out the angle point of target in basic frame and the key frame, and mates;

Step (2) according to the angle point that mates in basic frame and the key frame, is calculated the three-dimensional motion vector of key frame with respect to basic frame;

Step (3) is calculated the target three-dimensional model of key frame according to basic three-dimensional model and described three-dimensional motion vector, and two dimensional surface returns in back projection with the target three-dimensional model, and rejects overduely, calculates the final goal three-dimensional motion vector.

2. a kind of target three-dimensional motion vector computing method according to claim 1 is characterized in that step (1) may further comprise the steps:

Target in the identification video stream in key frame and the basic frame;

Mark out the angle point of target in basic frame and the key frame;

Angle point to basic frame and key frame mates: find the corresponding angle point of key frame and basic frame, obtain the two-dimensional coordinate of angle point.

3. a kind of target three-dimensional motion vector computing method according to claim 2 is characterized in that step (2) may further comprise the steps:

Based on video camera 1 and video camera 2:

Use approximation method to draw system of linear equations:

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}-X_1{p}_{11}-Y_1{p}_{12}-Z_1{p}_{13}\\ {y_1}^{\&prime;}-p_{24}-X_1{p}_{21}-Y_1{p}_{22}-Z_1{p}_{23}\\ M\\ M\\ {x_n}^{\&prime;}-{\mathrm{pp}}_{14}-X_n{\mathrm{pp}}_{11}-Y_n{\mathrm{pp}}_{12}-Z_n{\mathrm{pp}}_{13}\\ {y_n}^{\&prime;}-{\mathrm{pp}}_{24}-X_n{\mathrm{pp}}_{21}-Y_n{\mathrm{pp}}_{22}-Z_n{\mathrm{pp}}_{23}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccc}{Y}_1{p}_{13}-Z_1{p}_{12}& Z_1{p}_{11}-X_1{p}_{13}& X_1{p}_{12}-Y_1{p}_{11}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccc}{Y}_1{p}_{23}-Z_1{p}_{22}& Z_1{p}_{21}-X_1{p}_{23}& X_1{p}_{22}-Y_1{p}_{21}& p_{21}& p_{22}& p_{23}\end{array}\\ M\\ M\\ \begin{array}{cccccc}{Y}_n{\mathrm{pp}}_{13}-Z_n{\mathrm{pp}}_{12}& Z_n{\mathrm{pp}}_{11}-X_n{\mathrm{pp}}_{13}& X_n{\mathrm{pp}}_{12}-Y_n{\mathrm{pp}}_{11}& {\mathrm{pp}}_{11}& {\mathrm{pp}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccc}{Y}_n{\mathrm{pp}}_{23}-Z_n{\mathrm{pp}}_{22}& Z_n{\mathrm{pp}}_{21}-X_n{\mathrm{pp}}_{23}& X_n{\mathrm{pp}}_{22}-Y_n{\mathrm{pp}}_{21}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{\mathrm{\&phi;}}_1\\ {\mathrm{\&phi;}}_2\\ {\mathrm{\&phi;}}_3\\ t_1\\ t_2\\ t_3\end{array}\right],$

φ ₁, φ ₂, φ ₃Be respectively the three-dimensional motion vector reduced parameter, t ₁, t ₂, t ₃Be respectively the translation matrix parameter; Form three-dimensional motion vector M:

$M=\left[\begin{array}{cccc}1& -{\mathrm{\&phi;}}_3& {\mathrm{\&phi;}}_2& t_1\\ {\mathrm{\&phi;}}_3& 1& -{\mathrm{\&phi;}}_1& t_2\\ -{\mathrm{\&phi;}}_2& {\mathrm{\&phi;}}_1& 1& t_3\\ 0& 0& 0& 1\end{array}\right];$

Or use non-approximation method, draw system of linear equations:

$\left[\begin{array}{c}{x_1}^{\&prime;}-p_{14}\\ {y_1}^{\&prime;}-p_{24}\\ M\\ M\\ M\\ M\\ M\\ M\\ M\\ {x_n}^{\&prime;}-{\mathrm{pp}}_{14}\\ {y_n}^{\&prime;}-{\mathrm{pp}}_{24}\end{array}\right]=\left[\begin{array}{c}\begin{array}{cccccccccccc}{X}_1{p}_{11}& X_1{p}_{12}& X_1{p}_{13}& Y_1{p}_{11}& Y_1{p}_{12}& Y_1{p}_{13}& Z_1{p}_{11}& Z_1{p}_{12}& Z_1{p}_{13}& p_{11}& p_{12}& p_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_1{p}_{21}& X_1{p}_{22}& X_1{p}_{23}& Y_1{p}_{21}& Y_1{p}_{22}& Y_1{p}_{23}& Z_1{p}_{21}& Z_1{p}_{22}& Z_1{p}_{23}& p_{21}& p_{22}& p_{23}\end{array}\\ .\\ M\\ M\\ M\\ M\\ M\\ M\\ \begin{array}{cccccccccccc}{X}_n{\mathrm{pp}}_{11}& X_n{\mathrm{pp}}_{12}& X_n{\mathrm{pp}}_{13}& Y_n{\mathrm{pp}}_{11}& Y_n{\mathrm{pp}}_{12}& Y_n{\mathrm{pp}}_{13}& Z_n{\mathrm{pp}}_{11}& Z_n{\mathrm{pp}}_{12}& Z_n{\mathrm{pp}}_{13}& {\mathrm{pp}}_{11}& {\mathrm{pp}}_{12}& {\mathrm{pp}}_{13}\end{array}\\ \begin{array}{cccccccccccc}{X}_n{\mathrm{pp}}_{21}& X_n{\mathrm{pp}}_{22}& X_n{\mathrm{pp}}_{23}& Y_n{\mathrm{pp}}_{21}& Y_n{\mathrm{pp}}_{22}& Y_n{\mathrm{pp}}_{23}& Z_n{\mathrm{pp}}_{21}& Z_n{\mathrm{pp}}_{22}& Z_n{\mathrm{pp}}_{23}& {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}\end{array}\end{array}\right]\left[\begin{array}{c}{r}_{11}\\ r_{21}\\ r_{31}\\ r_{12}\\ r_{22}\\ r_{32}\\ r_{13}\\ r_{23}\\ r_{33}\\ t_1\\ t_2\\ t_3\end{array}\right]$

, r wherein ₁₁, r ₂₁, r ₃₁, r ₁₂, r ₂₂, r ₃₂, r ₁₃, r ₂₃, r ₃₃Be the rotation matrix parameter, form three-dimensional motion vector M:

$M=\left[\begin{array}{cccc}{r}_{11}& r_{12}& r_{13}& t_1\\ r_{21}& r_{22}& r_{23}& t_2\\ r_{31}& r_{32}& r_{33}& t_3\\ 0& 0& 0& 1\end{array}\right];$

(X wherein _i, Y _i, Z _i) any some three-dimensional coordinate in the basic frame of video camera 1 or video camera 2 of expression; (X _i', Y _i', Z _i') represent this three-dimensional coordinate in the key frame of video camera 1 or video camera 2; (x _i, y _i) represent that this projects to the two-dimensional coordinate of two dimensional surface at video camera 1 or video camera 2 in basic frame; (x _i', y _i') represent that this projects to the two-dimensional coordinate of two dimensional surface at video camera 1 or video camera 2 in key frame;

$P=\left[\begin{array}{cccc}{p}_{11}& p_{12}& p_{13}& p_{14}\\ p_{21}& p_{22}& p_{23}& p_{24}\\ 0& 0& 0& 1\end{array}\right]$ Projection matrix for video camera 1;

$\mathrm{PP}=\left[\begin{array}{cccc}{\mathrm{pp}}_{11}& {\mathrm{pp}}_{12}& {\mathrm{pp}}_{13}& {\mathrm{pp}}_{14}\\ {\mathrm{pp}}_{21}& {\mathrm{pp}}_{22}& {\mathrm{pp}}_{23}& {\mathrm{pp}}_{24}\\ 0& 0& 0& 1\end{array}\right]$ Projection matrix for video camera 2;

Find the solution the least mean-square error of system of linear equations and separate, obtain the kinematic matrix parameter, form three-dimensional motion vector.

4. a kind of target three-dimensional motion vector computing method according to claim 3 is characterized in that step (3) may further comprise the steps:

To two video cameras, described kinematic matrix parameter is additional to basic three-dimensional model, obtain the three-dimensional model of two camera video key frames;

Respectively with the three-dimensional model of two camera video key frames, the two dimensional surface of video camera projection matrix correspondence separately returns in back projection, with the angle point two-dimensional coordinate of the key frame of corresponding video camera relatively, reject the point of error beyond threshold value, use step (2) to recomputate the three-dimensional motion matrix parameter, obtain final three-dimensional motion vector.

Images