CN100487568C - Enhanced real natural interactive helmet with sight line follow-up function - Google Patents

Abstract

The invention is a augmented-reality naturally interactive helm with a sight line tracking function, comprising scene camera, earphone and microphone, electromagnetic position tracker and transmitter, pupil space regulating knob, vertical helm regulating knob, horizontal helm regulating knob, helm liner, and helm light path box. As compared with the existing techniques, it is added with the sight line tracking function, and has better 3D visual effect, and a double-channel naturally interactive function, applied to the technical fields of device maintaining, medical treatment and diagnosis, etc.

Description

Augmented reality natural interactive helmet with eye tracking function

Technical field

The present invention obtains the user at the scene area-of-interest by following the tracks of user's direction of visual lines, and the object of target area is discerned, and utilizes the augmented reality technology, and the active user zone information of carrying out is strengthened.Belong to eye tracking, virtual reconstruction and augmented reality technical field.

Background technology

Figure that the augmented reality system utilization is additional or Word message strengthen the scene dynamics ground of real world on every side, and desired user can be as strengthening free activity in the information space at real world ^[1]No matter the register method of researchist's employing both at home and abroad is based on the orientation tracking equipment at present, based on computer vision, also being based on vision-azimuth tracker combined tracking, all is only to follow the tracks of user's head orientation, utilizes the head position change in information to realize the stack of virtual information ^[2]In special applications such as the navigation of the dynamo-electric maintenance of equipment military aircraft of complexity, weapon aiming, medical operatings, the visual channel is a most important information interface between the mankind and the external environment condition, the people is movable except head movement in strengthening information space, eyes are also rotating, remove approximate direction of visual lines with head position, do not consider that into eye movement the registration error maximum of virtual information has 20 degree.Therefore, augmented reality system must detect user's head position and direction of visual lines in real time, follow the tracks of the variation of user of service's sight line,, virtual information is presented at correct position according to the definite in real time mapping position of virtual information in the real space coordinate that will add of these information.At present, the video perspective formula augmented reality helmet of the Nvis company of the U.S. has been equipped with the eye tracking function, and the ISCAN company of Japan is also similarly being studied.Integrated eye tracking function is subject to the ambient lighting influence because of it in the optical perspective helmet, and the hardware difficulty of processing strengthens, and studies less.

Summary of the invention

The objective of the invention is present situation, propose a kind of helmet simple in structure with eye tracking function and binary channels natural interaction ability at above-mentioned prior art.

Augmented reality system with eye tracking function should be followed the tracks of user's direction of visual lines, obtain the area-of-interest of user in scene, simultaneously, embody the characteristics that the three-dimensional registration of augmented reality system, real-time, interactive and actual situation merge again according to the variation of the blinkpunkt of gaze tracking system reflection.Based on the augmented reality system of eye tracking, at first need the position of real-time follow-up user blinkpunkt in real scene; Then according to the scene priori that is kept at lane database in advance, judge user's area-of-interest and objectives by the true coordinate of blinkpunkt in scene in conjunction with the scene image that the scene video camera obtains, thereby obtain needing to show the object that is enhanced of enhancing information; At last, go to inquire about the virtual object data storehouse according to the target information of current identification, the virtual enhancing information of acquisition and real scene coupling, the employing augmented reality is three-dimensional to be registered and the actual situation integration technology, and active user's area-of-interest target is carried out the information enhancing.Augmented reality system with eye tracking function should be followed the tracks of user's direction of visual lines, obtain the area-of-interest of user in scene, simultaneously again according to the variation of the blinkpunkt of gaze tracking system reflection, embodying the characteristics that the three-dimensional registration of augmented reality system, real-time, interactive and actual situation merge, is not the simple addition of both functions.Integrated system need satisfy following principle: 1. can not reduce both performances; 2. can not the interference user notice, increase added burden to the user of service.

The concrete composition that the present invention has the augmented reality natural interactive helmet of eye tracking function comprises scene video camera, earphone and microphone, electromagnetic type position tracker transmitter, interocular distance adjusting knob, helmet vertical adjustment knob, helmet horizontal adjustment knob, helmet liner, helmet light path box.Wherein, the scene video camera is installed in helmet front end, and earphone and microphone are installed on helmet left side inwall or the right inwall by support, and electromagnetic type position tracker transmitter is installed on the helmet inwall at helmet rear portion, and helmet liner is connected with helmet inwall.Helmet light path box places helmet front end to receive the virtual information that real scene that the scene video camera transmits and computing machine generate.The interocular distance adjusting knob is installed on the helmet of the anterior left and right sides of the helmet and with helmet light path box and is connected.Helmet vertical adjustment knob and helmet horizontal adjustment knob are installed in the vertical direction and the horizontal direction of the helmet respectively, and both all link to each other with helmet liner.Light path in the helmet light path box comprises miniscope, the anti-optical filter of infrared height, semi-transparent semi-reflecting combination light microscopic, the saturating optical filter of infrared height, Eye imaging machine, infrared light supply.Wherein, infrared light supply is installed in close semi-transparent semi-reflecting combination light microscopic on the helmet light path box, and this semi-transparent semi-reflecting combination light microscopic is formed square lens by two triangle lens.Anti-optical filter of infrared height and semi-transparent semi-reflecting combination light microscopic are the top that 45 ° of oblique angles are placed in semi-transparent semi-reflecting combination light microscopic.Miniscope and the anti-optical filter of infrared height are 45 ° of oblique angle horizontal directions equally and lay.The saturating optical filter of infrared height places the front end of Eye imaging machine camera, and Eye imaging machine horizontal direction is placed on the locus that semi-transparent semi-reflecting combination light microscopic and the anti-optical filter of infrared height are 45 ° of oblique angles.

The present invention compared with prior art has following characteristics: on the basis of optical perspective helmet, increased the eye tracking function, the target of utilizing direction of visual lines to finish area-of-interest is chosen, carry out 3-dimensional digital by computing machine and rebuild, utilize the binocular imaging system of left and right sides Biscreen display to see the three-dimensional stereoscopic visual effect that three-dimensional body and real scene actual situation merge; By increasing eye tracking and phonetic function, make the augmented reality helmet have binary channels natural interaction function, by the binary channels natural interaction, can make the augmented reality technology under maintenance of equipment, medical diagnosis etc. " hand is busy " condition, have more wide purposes.

Description of drawings

Fig. 1 is the augmented reality natural interactive helmet structural representation with eye tracking function

Fig. 2 is the light path synoptic diagram in the helmet light path box

The label title of Fig. 1 and Fig. 2:

1 scene video camera, 2 earphones and microphone, 3 electromagnetic type position tracker transmitters, 4 interocular distance adjusting knobs, 5 helmet vertical adjustment knob, 6 helmet horizontal adjustment knob, 7 helmet liners, 8 helmet light path boxes, 9 miniscopes, the anti-optical filter of 10 infrared height, 11 semi-transparent semi-reflecting combination light microscopics, the saturating optical filter of 12 infrared height, 13 Eye imaging machines, 14 infrared light supplies.

Fig. 3 is the fundamental diagram with augmented reality natural interactive helmet of eye tracking function

Embodiment

Augmented reality natural interactive helmet with eye tracking function illustrated in figures 1 and 2 comprises compositions such as scene video camera 1, earphone and microphone 2, electromagnetic type position tracker transmitter 3, interocular distance adjusting knob 4, helmet vertical adjustment knob 5, helmet horizontal adjustment knob 6, helmet liner 7.Basic light path comprises miniscope 9, the anti-optical filter 10 of infrared height, semi-transparent semi-reflecting combination light microscopic 11, the saturating optical filter 12 of infrared height, Eye imaging machine 13, infrared light supply 14, mainly finishes user of service's eye tracking and dummy object Presentation Function.

In Fig. 1, scene video camera 1 is finished the natural scene collection, and earphone and microphone 2 are finished interactive voice, and electromagnetic type position tracker transmitter 3 adopts electromagnetic type to follow the tracks of, and is built on the helmet, finishes head position and follows the tracks of.When different users of service wears this helmet, because everyone head size is different with interocular distance, can be by the size of helmet vertical adjustment knob 5 and helmet horizontal adjustment knob 6 adjusting helmet liners 7, distance by interocular distance adjusting knob 4 is regulated between the light path makes virtual image display effect the best.

The inner helmet light path places in the helmet light path box, overlaps independently light path equipment by two and forms, respectively corresponding people's right and left eyes.Distance between the light path equipment can be regulated by interocular distance adjusting knob 4.Fig. 2 is the integrated system index path, the light path of gaze tracking system is to be placed on the infrared light that the infrared light supply 14 of eyes oblique upper sends to shine human eye, the semi-transparent semi-reflecting combination light microscopic 11 that the infrared light of people's eye reflex passes its place ahead is changed the direction horizontal infections by anti-sheet 10 reflections of infrared height afterwards, enters infrared Eye imaging machine 13 imagings through the saturating sheet 12 of infrared height; The optical perspective helmet light path is to be presented at the light that the virtual information on the miniscope 9 sends to shine after lens on the semi-transparent semi-reflecting combination light microscopic 11, received by human eye from the light of this combiner reflection, make the human eye can the perception virtual information, add scene light, the user is perception real scene and virtual information simultaneously, obtains the enhancing effect of real scene.

Narrate principle of work of the present invention below in conjunction with Fig. 3

Based on the augmented reality system of eye tracking, at first need the position of real-time follow-up user blinkpunkt in real scene; Then according to the scene priori that is kept at lane database in advance, judge user's area-of-interest and objectives by the true coordinate of blinkpunkt in scene in conjunction with the scene image that the scene video camera obtains, thereby obtain needing to show the object that is enhanced of enhancing information; At last, go to inquire about the virtual object data storehouse according to the target information of current identification, the virtual enhancing information of acquisition and real scene coupling, the employing augmented reality is three-dimensional to be registered and the actual situation integration technology, and active user's area-of-interest target is carried out the information enhancing.Wherein, eye tracking partly is used to follow the tracks of user's blinkpunkt, corresponding to

Solution procedure; The area-of-interest that virtual enhancing information display section branch is finished the user strengthens, corresponding to

Solution procedure, its principle is as shown in Figure 3.

Eye tracking

Calculating

Blinkpunkt is that the coordinate of E is F at eye coordinates _e=(x _e, y _e, z _e, 1) ^T, the coordinate F under screen coordinate system O _o=(x _o, y _o, z _o, 1) ^T, the relation of being described by Fig. 2 as can be known

$F_o=\underset{O\←B}{\mathrm{\τ}}\·\underset{B\←S}{\mathrm{\τ}}\·\underset{S\←R}{\mathrm{\τ}}\·\underset{R\←E}{\mathrm{\τ}}\·F_e---\left(1\right)$

Wherein:

Obtain in real time by position tracker;

Can obtain by the static gaze tracking system of head;

With

Obtain by the system calibrating technology.

With

Demarcation

Adopt Eulerian angle and translation vector to describe

With

Parameter can reduce to 12, wherein preceding 6 parametric descriptions 6 parametric descriptions in back

Timing signal, the user watches the central round dot of eyes the place ahead diascope all the time attentively, and eye coordinates is that E Position Tracking Systems device induction coordinate system S is constant so, and eye coordinates is that E is that R overlaps with the main apparent coordinates of eye, and blinkpunkt is on the x of E axle.(x wherein _o, y _o, 0,1) ^TBe the screen coordinate of blinkpunkt, (x _e, 0,0,1) ^TFor the eye coordinates of blinkpunkt is a coordinate, I is a unit matrix, has reflected that the timing signal eye coordinates is that E is the fact that R overlaps with the main apparent coordinates of eye.

$\left[\begin{array}{c}{x}_o\\ y_o\\ 0\\ 1\end{array}\right]=\underset{O\←B}{\mathrm{\τ}}\·\underset{B\←S}{\mathrm{\τ}}\·\underset{S\←R}{\mathrm{\τ}}\·I\·\left[\begin{array}{c}{x}_e\\ 0\\ 0\\ 1\end{array}\right]---\left(2\right)$

Formula (2) is launched can cancellation x _eBe rewritten as

g _xy＝F(P，M _B←S) (3)

Wherein: g _XyBe the two-dimensional screen coordinate that is calculated by the blinkpunkt computational algorithm, P is 12 model parameter set, M _{B ← S}Be the value after the calibration of position tracker reading, F is the blinkpunkt computing function.Define average blinkpunkt error D _MeanFor

$D_{\mathrm{mean}}=\underset{i=1}{\overset{n}{\mathrm{\Σ}}}\left|\right|a-b\left|\right|,(a\∈g_{\mathrm{xy}},b\∈G_{\mathrm{xy}})---\left(4\right)$

Wherein: G _XyBe the actual blinkpunkt screen coordinate of gathering in the calibration process, n is total number of blinkpunkt, by (2) and (3)

With

Demarcation be an iteration optimization solution procedure, constantly adjust each parameter of P, make D _MeanMinimum.

$\underset{R\←E}{\mathrm{\τ}}$ Demarcation

Because being the true origin of R, current coordinate system E of eye and the main apparent coordinates of eye be the eyeball rotation center, therefore

The component that only comprises the reflection rotation.This step calibration request user head keeps static and requires the main apparent coordinates of eye is that the x axle of R is vertical with the approximate maintenance of screen plane.Suppose that the screen coordinate that the timing signal master looks blinkpunkt is C (x _c, y _c), the eyes rotation center is D apart from the distance of screen, the reading of position tracker is The screen coordinate that is calculated by gaze tracking system during eye gaze screen arbitrfary point is P (x _g, y _g).From the main apparent coordinates of eye is that R transforms to and watches P (x attentively _g, y _g) time eye coordinates be that E can be decomposed into: around the z of R axle rotation β angle to interim coordinate system; , can get to E around interim coordinate system y axle rotation alpha angle by α and β

β＝arg?tan(|x _c-x _g|/D)

$\mathrm{\&alpha;}=\arg \tan (|y_g-y_c|/\sqrt{D^2+|x_g-{x_{\mathrm{<figure-callout\; id="2"\; label="c"\; filenames="C200710022978E00092.png"\; state="\{\{state\}\}">c</figure-callout>}}}^2|})---\left(5\right)$

Blinkpunkt calculates

After system calibrating is finished, can pass through formula (6) and calculate the coordinate (X, Y, 0,1) of user's blinkpunkt under screen coordinate system ^T

$\left\{\begin{array}{c}X=-\frac{r_1\&CenterDot;t_z}{r_7}+t_x\\ Y=-\frac{r_4\&CenterDot;t_z}{r_7}+t_y\end{array}\right.---\left(6\right)$

R in the formula ₁, r ₄, r ₇, t _x, t _y, t _zFor

Element.

The virtual information registration Calculating

For virtual enhancing information accurately is placed in the scene, need the transformation relation between acquisition current scene and the optical perspective helmet display device, promptly find the solution The relative transformation relation of world coordinate system and optical perspective helmet display device.

$\underset{H\&LeftArrow;O}{\mathrm{\&tau;}}=\underset{H\&LeftArrow;S}{\mathrm{\&tau;}}\&CenterDot;{\underset{B\&LeftArrow;S}{\mathrm{\&tau;}}}^{-1}\&CenterDot;{\underset{O\&LeftArrow;B}{\mathrm{\&tau;}}}^{-1}---\left(7\right)$

Wherein:

Can obtain by the demarcation of gaze tracking system,

Can calculate by the Position Tracking reading.

Demarcating steps be:

Wear optical perspective helmet display device, from the image direct imaging in the real scene on every side on user's retina.SPAAM single-point alignment algorithm is adopted in the demarcation of optical perspective helmet display device, and human eye and optical perspective helmet display device are regarded as a virtual video camera, comes it is demarcated by the video camera pin-hole model, and basic step is as follows:

1. show a true monumented point in the display optional position, and write down the world coordinates coordinate (x of this point _w, y _w, z _w, 1) ^T

2. computing machine generates the individual virtual mark point of n (n 〉=6), is presented on the optical perspective helmet display device by equidistant arrangement, writes down the coordinate (x under the Helmet Mounted Display coordinate system of this n point _i, y _i, 1) ^TUser's moving-head makes that monumented point overlaps fully on virtual mark point and the screen that eyes are seen, and monumented point of every aligning writes down one time the position tracker reading;

3. foundation

Demarcate by gaze tracking system,

Read in real time by position tracker, calculate the coordinate (X under the true monumented point Position Tracking Systems device receiver coordinate system _Mi, Y _Mi, Z _Mi, 1) ^T

4. with (x _i, y _i, 1) ^T(X _Mi, Y _Mi, Z _Mi, 1) ^TSubstitution formula (8) is found the solution with the SVD decomposition method

Claims (1)

1. augmented reality natural interactive helmet with eye tracking function, comprise scene video camera (1), earphone and microphone (2), electromagnetic type position tracker transmitter (3), interocular distance adjusting knob (4), helmet vertical adjustment knob (5), helmet horizontal adjustment knob (6), helmet liner (7), helmet light path box (8), wherein, scene video camera (1) is installed in helmet front end, earphone and microphone (2) are installed on helmet left side inwall or the right inwall by support, electromagnetic type position tracker transmitter (3) is installed on the helmet inwall at helmet rear portion, helmet liner (7) is connected with helmet inwall, helmet light path box (8) places helmet front end to receive the virtual information that real scene that scene video camera (1) transmits and computing machine generate, interocular distance adjusting knob (4) is installed on the left of the helmet front portion or on the helmet on right side and with helmet light path box (8) and is connected, helmet vertical adjustment knob (5) and helmet horizontal adjustment knob (6) are installed in the vertical direction and the horizontal direction of the helmet respectively, both all link to each other with helmet liner (7), light path in the described helmet light path box (8), comprise miniscope (9), the anti-optical filter of infrared height (10), semi-transparent semi-reflecting combination light microscopic (11), the saturating optical filter of infrared height (12), Eye imaging machine (13), infrared light supply (14), wherein, infrared light supply (14) is installed in helmet light path box (8) and goes up near semi-transparent semi-reflecting combination light microscopic (11), this semi-transparent semi-reflecting combination light microscopic (11) is formed square lens by two triangle lens, the anti-optical filter of infrared height (10) is the top that 45 ° of oblique angles are placed in semi-transparent semi-reflecting combination light microscopic (11) with semi-transparent semi-reflecting combination light microscopic (11), miniscope (9) is the top that 45 ° of oblique angle levels are placed in the anti-optical filter of infrared height (10) equally with the anti-optical filter of infrared height (10), the saturating optical filter of infrared height (12) places the front end of Eye imaging machine (13) camera, and Eye imaging machine (13) horizontal direction is placed on the locus that semi-transparent semi-reflecting combination light microscopic (11) and the anti-optical filter of infrared height are 45 ° of oblique angles.

Images