WO2013121052A1

WO2013121052A1 - Method for highlighting at least one moving element in a scene, and portable augmented reality device implementing said method

Info

Publication number: WO2013121052A1
Application number: PCT/EP2013/053216
Authority: WO
Inventors: Nizar Ouarti; Avinoam SAFRAN; Johan LE BRUN; Samuel PINEAU
Original assignee: Universite Pierre Et Marie Curie (Paris 6); Centre National De La Recherche Scientifique
Priority date: 2012-02-16
Filing date: 2013-02-18
Publication date: 2013-08-22
Also published as: FR2987155A1

Abstract

The invention relates to a method for highlighting at least one moving element in a scene, the method comprising the steps of: observing (1) the scene using at least one sensor; analyzing (2) the information generated by the sensor in order to locate the moving element in the scene; generating (3), in real time, processed images of the scene which include a representation of the moving element as located, and in which at least one visual feature of at least the representation of the moving element is determined on the basis of the analyzed information in order to highlight said element relative to other stationary elements of the scene; and projecting (4) the thus-generated images onto a display member.

Description

Method for highlighting at least one moving element in a scene as well as a portable augmented reality device implementing said method

The invention relates to a method for highlighting at least one moving element in a scene. The invention also relates to a portable augmented reality device implementing said method. BACKGROUND OF THE INVENTION

"Augmented reality" is today the techniques used to change the perception of the environment, whether it is related to sight, touch or even smell, through more or less immersive devices.

The devices relating to the vision may especially be called retinal, when they are directly applied to the retina of the user, "head-mounted display" or HMD, when they are worn on the head or face as helmets or goggles, "hand-held display" or HHD, when they are adapted to be worn as hand-held mobile phones or consoles.

The applications of these new techniques are numerous, they are today used in the sector of the personal assistance and the publicity, the navigation, the tourism, the military industry, the sport, the design or the medicine. In the latter sector, several fields are concerned, notably surgery or ophthalmology.

Eyeglasses are of little use for people with low vision due to a rare condition such as retinopathy pigmentosa or juvenile retinoschisis but also to a more common pathology such as Aging Age-related Degeneration or AMD. . The macula is the central part of the retina whose main role is to transmit to the brain information on the contours and colors of the various elements contained in a scene. But with age, the macula tends to deteriorate. Patients with AMD are thus victims of a progressive deterioration of their vision with for example a decrease in the sensitivity to contrasts, a tendency to the deformation of the straight lines and sometimes the appearance of a dark central task still called scotoma. To date, there is no known therapeutic treatment to completely cure a patient with AMD.

In order to assist a patient with a retinal pathology it has been proposed to employ augmented reality techniques. The scene is observed using a sensor and processed images are generated. Typically a scene consists of a set of real elements, which can be for example people or objects, likely to move within an environment: Movements can also appear in non-rigid objects ( eg facial expressions): the moving element is then the moving portion of the non-rigid object. A sensor is a device that transforms the state of a physical quantity into an observable quantity. Several sensors can be associated to measure one or more physical quantities on a spatial window and may vary over time. Thus, for example, an isolated photodiode may constitute a sensor, the combination of several photodiodes may constitute a camera that will also be described as a sensor

In order to assist a patient suffering from a pathology of the retina, it has notably been proposed in US Pat. No. 6,611,618 to employ augmented reality techniques and more specifically a method comprising the steps of: observe a scene taking place in front of the patient using a sensor, in this case a camera;

generating processed images of the scene from the information generated by the camera so that all contours contained in the image are enhanced;

project the images thus generated

display organ.

Another method is to display a miniaturized version of the scene with outlines enhanced on the original image.

The images processed thus comprise a representation of the elements of the scene according to the information generated by the sensor associated with them, the representation of each of the elements is qualified as figuration.

The visual characteristics of each of the figurations are generally determined according to the same principles for all the elements of the same scene. By way of example, when a digital camera having common optical sensors is used to photograph a scene, it makes it possible to obtain digital images in which the figurations of the elements of the scene are, from the point of view of the user and because of their. visual characteristics, very representative of, or similar to, the scene he can observe with the naked eye.

In the case of US Pat. No. 6,611,618, the contours are accentuated in the images worked, which makes it possible to help the patient to better perceive and better understand the scene unfolding in front of him when he is watching the scene through the organ. of visualization. In addition, if the patient has a scotoma, his peripheral vision is thus improved which substantially compensates for its central absence of vision.

However, there is no hierarchy between enhanced outlines, so all outlines are enhanced and highly visible. It turns out that for a scene containing many details, the patient is tired very quickly to visualize his environment from such images worked. On the other hand, it is difficult for the patient to extract the relevant information.

OBJECT OF THE INVENTION

An object of the invention is to propose a method for generating processed images of a real scene taking place in front of a person by processing the information acquired by means of a sensor from the real scene which makes it possible to overcome disadvantages mentioned above.

An object of the invention is also to provide a portable augmented reality device implementing such a method.

BRIEF DESCRIPTION OF THE INVENTION

In order to achieve this goal, a method of highlighting at least one moving element in a scene is proposed, the method comprising the steps of:

- observe the scene with at least one sensor;

analyzing information generated by the sensor to identify the moving element in the scene;

generating in real time worked images of the scene comprising a figuration of the moving element as identified, in which at least one visual characteristic of at least the representation of the moving element is determined according to the analyzed information to highlight said element in relation to other immobile elements of the scene; project the images thus generated onto a display unit.

Thus, only the moving elements are highlighted so that the number of additional information received by the brain, when a person uses the viewing member, is not excessive. The attention of the person is therefore mainly turned to the figurations of the moving elements.

The number of highlighted elements is reduced vis-à-vis worked images that are generated by a method of the prior art. The person can thus visualize the scene by the display member without being as tired as if it viewed said scene through processed images generated by a method of the prior art. In addition, it is important to put the moving elements in evidence so that a person can move towards these elements for example to avoid or catch one of the elements. In a medical setting, a person with a pathology of the retina can thus more easily interact with his environment. By parallax, the elements closest to the person are much more visible.

In addition, the person apprehends in priority the moving elements that are generally the important elements of the scene. With the method of the prior art, the inventors have found that a person has difficulty in understanding the important elements of the scene because too much contours or not enough contours were enhanced. When too many contours were enhanced, the images were difficult to understand for the person watching them. When little outlines were enhanced, important elements of the scene could escape the attention of the person.

Advantageously, highlighting only moving elements and not all elements of the scene, it is possible to perform the different steps of the process in real time. In real time, we mean here that the processed images are projected as quickly as possible when the information was generated by the sensor so that a person can interact with his environment just as naturally as if he were viewing directly his environment. The inventors have thus been able to produce prototypes implementing the method of the invention in which between fifteen and twenty images are projected per second.

In the context of the present application, the term sensor may of course designate an isolated sensor or a combination of sensors. Moreover, the term figuration designates the representation in the processed image of an element, in particular an element in motion.

The invention thus makes it possible to highlight the figurations of the moving elements directly by working on the information generated by the sensor. The worked images thus highlight said figurations by a work on their visual characteristics and not by the addition of fictitious elements in the images worked to highlight said figurations.

According to a particular mode of implementation of the invention, the method comprises an analysis of the matrix data generated by the sensor by calculating the dense optical flows associated with these data. Preferably, the computation of dense optical fluxes uses a dense differential optical flux method with a pyramidal scheme.

The invention also relates to a portable augmented reality device implementing the method according to the invention.

BRIEF DESCRIPTION OF THE DRAWINGS The invention will be better understood in the light of the following description of a particular nonlimiting implementation of the invention with reference to the attached figures among which:

Figure 1 is a diagram showing the different steps of the method of the invention;

FIG. 2 is a perspective view of a portable augmented reality device embodying the method of the invention illustrated in FIG.

DETAILED DESCRIPTION OF THE INVENTION

With reference to FIG. 1, the method of the invention is intended to highlight at least one moving element in a scene.

The method comprises for this purpose a first step 1 during which the scene is observed using at least one sensor. Here, the sensor comprises at least one camera taking pictures of the scene.

In a second step 2, the images taken by the camera are analyzed to identify the moving element in the scene.

In a third step 3, real-time images of the scene comprising a figuration of the moving element as identified, in which at least one visual characteristic of at least the figuration of the moving element, are generated in real time. determined according to the analyzed images to highlight said element with respect to other immobile elements of the scene.

In a fourth step 4, the processed images generated in the third step 3 are projected onto a display member.

Preferably, the visual characteristic is determined as a function of at least one inertial characteristic related to the movement of the element in the scene. The inertial characteristic is for example a speed of displacement of the element, a direction of displacement of the element, an acceleration of displacement of the element ...

According to a first implementation of the method of the invention, during the third step 3, the images processed are generated from a representation of the velocity vectors respectively associated with the pixels of the images taken by the camera.

In a first example, the hue of the figuration of the moving element is determined according to a direction of movement of the element in the scene.

Preferably, a color code is defined, a color being associated with a direction of movement in the images relative to a position of the camera in the images. Then, processed images are generated by assigning to each pixel of the processed image a color which is function, according to the color code, of the direction of the velocity vector associated with the corresponding pixel of the image taken by the camera.

The pixels representing an inert part of the scene being all of the same color and the pixels representing part of the moving element being all of a color different from the color of the pixels representing an inert part of the scene, the moving element is well highlighted. Advantageously, a user can easily identify the directions of movement of the elements of the scene.

In a second example, the luminance of the figuration of the moving element is determined as a function of a speed of movement of the element in the scene.

Preferably, the processed images are then generated by assigning each pixel of the processed image: a first luminance if the velocity vector associated with the corresponding pixel of the image taken by the camera has a norm greater than a given threshold;

a second luminance, less than the first, if the velocity vector associated with the corresponding pixel of the image taken by the camera has a standard lower than the given threshold.

Since a pixel representing an inert part of the scene has a luminance lower than that of a pixel representing a part of a moving element, the moving element is well highlighted on the images worked.

In a variant, generated images are generated by attributing to the different pixels of the processed image a luminance proportional to the norm of the velocity vector associated with the corresponding pixel of the image taken by the camera.

Advantageously, the pixels representing part of a moving element appear much brighter than the pixels representing an inert part of the scene, the moving element is well highlighted. In addition, the higher a moving element has a high speed, the more the representation of said element differs from the figurations of other elements and the user can easily apprehend such a moving element.

In a third example, the luminance of the figuration of the moving element is determined according to a direction of movement of the element in the scene.

Preferably, processed images are generated by assigning to the different pixels of the processed image a luminance which is a function of the direction of movement of the velocity vector associated with the corresponding pixel of the image taken by the camera so that the direction of the a velocity vector is oriented towards a center of the image taken by the camera, plus the luminance of the corresponding pixel on the image worked will be important.

The pixels representing an inert part of the scene being all of the same luminance and the pixels representing part of a moving element being all of a luminance different from that of the pixels representing an inert part of the scene, the element in motion is well highlighted on the images worked. Advantageously, the closer a moving element is to the user, the more the representation of said element differs from the figurations of the other elements and the user can easily apprehend such a moving element.

In a fourth example, the saturation of the figuration of the moving element is determined according to a speed of movement of the element in the scene.

Preferably, processed images are generated by allocating to the different pixels of the processed image a saturation proportional to the norm of the velocity vector associated with the corresponding pixel of the image taken by the camera.

In a fifth example, the saturation of the figuration of the moving element is determined according to a direction of movement of the element in the scene.

Preferably, processed images are generated by attributing to the various pixels of the processed image a saturation which is a function of the direction of movement of the velocity vector associated with the corresponding pixel of the image taken by the camera so that the direction of the image a velocity vector is oriented towards a center of the image taken by the camera, plus the saturation of the corresponding pixel on the image worked will be important.

According to a second implementation of the method of the invention, during the third step 3, the processed images are generated by modifying the images taken by the camera. Compared to the processed images generated by the first implementation of the method of the invention, the processed images generated by the second implementation contain a greater number of information on the scene observed especially in terms of textures and colors of the elements of the scene.

In a sixth example, the luminance of the figuration of the moving element is determined as a function of a speed of movement of the element in the scene.

Preferably, the images processed are generated by modifying the luminance of the pixels of the images taken by the camera in proportion to the norm of the velocity vector of each of said pixels. The following formula is used here for each pixel of an image taken by the sensor:

with

The luminance of the pixel of the processed image;

The luminance of the pixel of the image taken by the camera;

- the norm of the pixel velocity vector of the image taken by the camera;

- nmax the maximum standard among all the standards of the speed vectors of the image taken by the camera;

- dyn the luminance dynamics of the image taken by the camera (usually 255).

Advantageously, the pixels representing part of a moving element appear much brighter than the pixels representing an inert part of the scene, the moving element is well highlighted. In addition, the higher a moving element has a high speed, the more the representation of said element differs from the figurations of other elements and the user can easily apprehend such a moving element. Alternatively, the processed images are generated by keeping the luminance of the pixels of the images taken by the camera that have a velocity vector standard higher than a predetermined threshold and by lowering the luminance of the pixels of the images taken by the camera that have a standard of velocity vector below said threshold.

Advantageously, the pixels representing part of a moving element appear much brighter than the pixels representing an inert part of the scene, the moving element is well highlighted.

According to another variant, the processed images are generated by increasing the luminance of the pixels of the images taken by the camera which have a velocity vector standard higher than a predetermined threshold and by maintaining the luminance of the pixels of the images taken by the camera which have a velocity vector standard below said threshold.

The following formula is used here for each pixel of an image taken by the sensor:

If n> X,

L = L + g * n

if L'≤0 then L "= 0

with -j if the ≥dyn then L "= dyn

if 0 <The ≤ dyn then L "= L '

If n <X

L = L

with

The luminance of the pixel of the processed image;

The luminance of the pixel of the image taken by the camera;

the norm of the pixel velocity vector of the image taken by the camera;

- X the predetermined threshold;

the predetermined gain for increasing the luminance of the pixels; - dyn the luminance dynamics of the image taken by the camera (usually 255).

In a seventh example, the luminance of the figuration of the moving element is determined according to a direction of movement of the element in the scene.

Preferably, the images processed are generated by modifying the luminance of the pixels of the images taken by the camera as a function of the direction of the velocity vector of each of said pixels so that the direction of the velocity vector of a pixel is oriented toward a center images taken by the camera plus the luminance of said pixel in the processed images is important.

The pixels representing an inert part of the scene keeping their luminance and the pixels representing part of a moving element being all of a luminance different from that of the pixels representing an inert part of the scene, the moving element is well. highlighted. Advantageously, the more a moving element goes in the direction of the user, the more the representation of said element differs from the figurations of the other elements and the user can easily apprehend such a moving element.

In an eighth example, the saturation of the figuration of the moving element is determined according to a speed of movement of the element in the scene.

Preferably, the images worked are generated by modifying the saturation of the pixels of the images taken by the camera in proportion to the norm of the velocity vector of each of said pixels. The following formula is used here for each pixel of an image taken by the sensor:

with

Saturation of the pixel of the processed image;

- saturation of the pixel of the image taken by the camera;

the norm of the pixel velocity vector of the image taken by the camera;

- dyn the dynamic saturation of the image taken by the camera (usually 255).

According to a variant, the processed images are generated by increasing the saturation of the pixels of the images taken by the camera which have a velocity vector standard higher than a predetermined threshold and by maintaining the saturation of the pixels of the images taken by the camera which have a standard velocity vector below said threshold.

If n> X,

S '= S + g * n

^' if S'≤ 0 then S "= 0

with <if S'≥ dyn then S "= dyn

if Q≤S'≤dyn then S "= S '

If n <X

S "= S

with

S '' the saturation of the pixel of the processed image; - saturation of the pixel of the image taken by the camera; the norm of the pixel velocity vector of the image taken by the camera;

- X the predetermined threshold;

the predetermined gain for increasing the saturation of the pixels;

- dyn the luminance dynamics of the image taken by the camera (usually 255).

In a ninth example, the saturation of the figuration of the moving element is determined according to a direction of movement of the element in the scene.

Preferably, the images worked are generated by modifying the saturation of the pixels of the images taken by the camera as a function of the direction of the velocity vector of each of said pixels so that the direction of the velocity vector of a pixel is oriented toward a center images taken by the camera plus the saturation of said pixel in the images worked is important.

With reference to FIG. 2, the determination method is here implemented using a portable augmented reality device 10 which is for example worn by a user suffering from a pathology of the retina.

The augmented reality portable device 10 comprises a display member. Preferably, the display member comprises a pair of spectacles 11 comprising on a first lens a first display 15 and on a second lens a second display 16.

The augmented reality portable device 10 further includes at least one sensor for viewing the scene. According to a preferred embodiment, the augmented reality portable device 10 thus comprises means for taking stereoscopic images which comprise a first camera 12 associated with the right eye of the user and the first lens and a second associated camera 13 to the left eye of the user and to the second glass. The augmented reality portable device 10 further comprises a computing device 14 which is connected here wired to the stereoscopic image taking means as well as to the first display 15 and to the second display 16.

According to the invention, the computing device 14 includes means for analyzing the images taken by the cameras 12, 13 to identify at least one moving element in the scene. The computing device 14 further comprises means for generating the processed images of the scene in real time.

To work in real time, the computing unit 14 must deliver a large computing power in a very short time. Preferably, the computing unit 14 comprises a parallel calculation module in particular for generating the processed images in real time. The parallel computing module is for example a graphics processor (GPU or Graphical Processing Unit in English) or a programmable logic circuit (SPGA or System-Programmable Gaste Array). Advantageously, by parallelizing the calculations, it is possible to accelerate them.

The augmented reality portable device 10 here comprises a battery 17 for powering said portable augmented reality device 10.

The computing device 14 implements, for example, the following algorithm.

The computing device 14 estimates the optical flux of the pixels of the images taken by each camera so as to determine velocity vectors which are respectively associated with each of said pixels. In the following, we note:

(x, y) the coordinates of a pixel i of an image taken; (ui, vi) the coordinates of a velocity vector of a pixel i of the image taken; The luminous intensity of a pixel i of the image taken;

f _n (x, y, t) and f _{n +} i (x, y, t + dt) two successive images taken by the sensors.

For each pixel of the second image f _{n +} 1 (x, y, t + dt), the computation unit 14 estimates:

- It which results from the difference between I _n and I _{n + 1} ;

- Ix resulting from the convolution by a wavelet filter having a single zero moment of the derivative of the mean of I _n and I _{n +} i with respect to x _n ;

- Iy resulting from the convolution by a wavelet filter having a single zero moment of the derivative of the mean of I _n and I _{n +} i with respect to y _n .

Then, for each pixel of the second image fn + i (X _/ Yr t + dt), the computation unit 14 successively estimates:

- u _ng resulting from the convolution of u _n and v by a Gaussian _ng resulting from the convolution of _w by a Gaussian;

- k which is defined by k = ((Ix * u _ng ) + (Iy * v _ng ) + It) l (a ² + Ix ² + Iy ² ) with a which is a coefficient of regularization of the image;

- u _{n +} i by the formula u _{n + l} = u - Ix * k and v _{n +} i by the formula v " ₊ i = ^v , _v - fy *.

Thus, the computation member 14 estimates the velocity vector of each pixel of the image f _{n +} 1 (x, y, t + dt), which makes it possible to locate one or more moving elements in the scene.

In a second step, the computing device 14 generates an image f _{n +} i '(x, y, t + dt) by modifying the image fn + i (, Y, t + dt). For this purpose, for each pixel of the image fn ₊ i (x _r y, t + dt), the computation unit 14 estimates for example the norm of the vector (u _{n +} i; v _{n +} i). The image f _n + i '(x, y, t + dt) is then created for example by modifying the luminance of the pixels of the images taken by the camera in proportion to the norm of the thus calculated velocity vector of each of said pixels, as described previously. The processed images of each camera are here processed by the computing device 14 in order to generate worked stereoscopic images of the scene that are projected onto said glasses displays 11.

The user thus enjoys a better vision through the observation of his environment through the glasses 11. In addition, the portable device 10 augmented reality is a small footprint so that the user can wear it on him everyday. In addition, the augmented reality portable device 10 allows the projection of the images processed in real time so that the user can interact in a natural way with his environment.

According to another embodiment of the invention, the computation unit 14 implements the following algorithm based on the analysis of dense optical flows.

The analysis step thus comprises the real-time calculation of optical flows for estimating the inertial characteristic, the optical flux being a vector field representing the displacement of each pixel between two images generated by the sensor.

By successive images, here means images generated by the sensor at successive times (as in the previous example images taken at times t and t + dt). The images considered for the computation of the optical fluxes correspond preferably to consecutive instants although this is not obligatory.

The visual characteristics of the representation of each element of the scene can thus be modified accordingly by taking into account the inertial data of each pixel. This will result in images where the moving areas of the video formed by the succession of images will have a different representation of the static areas. The goal is to determine which objects are moving thanks to the optical flow.

Preferably, the analysis step comprises the real-time calculation of the dense optical fluxes taking into account all the pixels of the successive images generated by the sensor.

It is known to calculate the sparse optical flux in real time by the method of Lucas-Kanade (see in particular "An Iterative image registration technique with an application to stereo vision" Bruce D. Lucas and Takeo Kanade, Proceedings of the 7th International Joint Conference on Artificial Intelligence (IJCAI '81), April, 1981, pp. 674-679.).

An image contains a large number of pixels and finding the correspondence of each pixel between two images can be expensive in computation time. One solution is to follow only certain pixels. This is the sparse optical flow method. Lucas-Kanade's method thus supposes that the displacement of a point of the image between two consecutive instants is small and approximately constant in a neighborhood of the point. The definition of the sparse optical flux is thus opposed to the dense optical flux envisaged in the context of the invention where this time the objective is to match a translation vector for each pixel of the image.

A real-time dense optical flux calculation is thus performed here. The invention thus aims to take advantage of the computation of a dense optical flux in real time and its graphical representation to obtain the images processed from the real scene.

Preferably, the computation of the dense optical fluxes in real time comprises the implementation of a dense differential optical flow method with a pyramidal scheme (see for example Jean-Yves Bouguet, "Pyramidal Implementation of the Lucas Kanade Feature Tracker: Description of tea algorithm ", Intel Corporation - Microprocessor Research Labs, 2002.)

We define a height Lm pyramid (in practice, L _m is 2, 3 or 4) characterizing a number of levels. At each level of the pyramid, each successive image is sub-sampled from the level 0 corresponding to the initial image to the level L _m corresponding to the coarsest sampling. At the level L _m , the optical flux is computed and then propagated at the lower level by translating one of the two images according to this optical flux. The calculation is performed again at the lower level and so on until level 0 which corresponds to the initial image. The final optical flux is then recovered.

Preferably, the method includes the use of a functional F = F _re + fiF _attack where the attachment term F _am is based on a thresholding method and the regularization term F _reg is based on the U + V method according to Meyer's work, β being a weight controlling the ratio between the term of attachment and the term of regulation.

The computation of the dense optical flux in real time at each level of the pyramid comprises a minimization of the aforementioned functional.

The term regulation here is based on Meyer's U + V model (see, for example, Y. Meyer, Oscillating patterns in image processing and nonlinear evolution equations, The Fifteenth Dean Jacquelines B. Lewis Memorial Lectures, American Mathematical Society, 2001). which makes it possible to split an image into two parts: one written p containing the structures, the other written q containing the textures where p belongs to the space BV of the functions with bounded variations and where q belongs to the space G of oscillating functions which is close to the dual of the space BV which has the property that the more a function is oscillating, the more its standard II II will be weak. It is thus proposed to use a regularization on the space BV, thus with the total variations (ROF).

For the q part, non-stationary wavelet packets will preferably be used. It is thus used a Besov space and non-stationary wavelet packets called "PONS" (see "Best basis denoising with non-stationary avelet packets" Ouarti & Peyre, Proceedings of IEEE-ICIP (pp. 3825-3828), 2009) which generalize the notion of wavelet. Thus, all conventional wavelet filters can be used within the scope of the invention.

For each level of the pyramid consider a first image I _n and a second image I _{n +}} . For each pixel n of the second image I _{n + 1,} an iterative loop is implemented to minimize the functional to determine the components (u _n , v _n ) of the dense optical flux of said pixel. For example, the following iterative loop is implemented, the components (u _n , v _n ) being taken to zero at the beginning of the loop:

we calculate the term u _ng resulting from the convolution of u _n by a Gaussian;

we calculate the term v which results from the convolution of v _n by a Gaussian;

the ratio k is calculated which is a ratio defined by k = ((Ix * u _ng ) + (fy * v _ng ) + I _t ) / (a ² + Ix ² + fy ² ) where a is a regularization coefficient of the image, I _t the difference between the images / "and I _{n + l} , (I _x ; I _y ) the components of the derivative of the mean of the images /" and I _{n + 1} which have been convolved by a wavelet filter having a single zero moment;

we calculate the term u _{n + 1} = u _ng - Ix * k and the term v " ₊ , = v, _¾ , -fy * k;

we define u _n = u _{n + 1} and v _n = v _{n + 1} before starting the loop again.

Preferably, the images worked are generated according to a model Luminance Saturation Hue (TSL) or HSL in English. Remember that the hue is the pure form of a color that is to say without adding white, black or gray, the saturation quantifies the amount of gray present in a color and the luminance quantifies the amount of white or black in a color. The visual characteristic determined according to at least the inertial characteristic is then one of the three parameters of the HSL model, namely hue, saturation or luminance.

The determination of the visual characteristic and the generation accordingly of the processed images will therefore have the effect of representing movement information by a modification of the position in the colored space of the pixels of the images worked with respect to their positions in the non-processed images. generated by the sensor. It should be noted, however, that in the case of monochrome images or videos showing only one physical quantity at a time, the change will only be effective on a single parameter such as luminance.

For example, the center of the image worked to generate is initialized to zero. For each pixel of the image to be generated, the norm of (u, v), the angle of (u, v) and the position of the pixel associated with (u, v) are calculated. The scalar product is then calculated between the normed vector of the position of the pixel and the opposite of the normalized vector (w, v). The norm of (u, v) is defined as luminance, the angle of (, v) as hue and S as saturation.

Thus, the more a moving element is closer to the user, the more the representation of said element differs from the figurations of other elements and the user can easily apprehend such a moving element.

Preferably, the inertial characteristic comprises velocity vectors which are each associated with one of the pixels of the images taken by the camera, the time calculation real optical fluxes for estimating said velocity vectors.

Naturally, the invention is not limited to the embodiment described and alternative embodiments can be made without departing from the scope of the invention as defined by the claims.

In particular, the method of the invention may be implemented for applications other than medical aid to persons suffering from a pathology of the retina and / or may be implemented by other devices than that described. . For example, the method according to the invention can be implemented to help a vehicle driver to better understand his environment by highlighting moving elements at the rear of the vehicle as another vehicle. According to a particular embodiment, the display member will be arranged above one of the mirrors as the left mirror. Thus, the driver can both look at the rearview mirror and the display member to apprehend what is happening at the rear of his vehicle.

The step of real-time generation of the images worked may be different from those described. For example, if the determination of the visual characteristic of the figuration of the moving element is performed as a function of at least one inertial characteristic related to the movement of the element, the inertial characteristic may also be related to the acceleration of the element, at a distance from the element with respect to the center of the images taken by the camera ... The inertial characteristic may thus comprise, instead of velocity vectors, accelerating vectors, which are the derivatives of the velocity vectors. The determined visual characteristic may be other than one of the three parameters hue, saturation, luminance as for example a contrast of the contours of the element. It will be possible to determine several visual characteristics of at least the moving figuration as it is located. It will also be possible to determine the visual characteristic (s) as a function of several inertial characteristics related to the movement of the element in the scene. For example, it will be possible to generate processed images of the scene by determining a first visual characteristic of the figuration of the moving element according to the speed of movement of the element and by determining a second visual characteristic of the figuration of the element. moving element according to a direction of movement of the element relative to the sensor and / or the user. As a variant, it will be possible to generate processed images of the scene by determining a unique visual characteristic of the figuration of the moving element according to different inertial characteristics.

According to a preferred embodiment, the images processed are generated by modifying the luminance of the pixels of the images taken by the camera according to the norm and the direction of the velocity vector of each of said pixels so that the direction of the vector the speed of a pixel is oriented towards a center of the images taken by the camera and the higher the standard of the velocity vector of said pixel, the higher the luminance of said pixel in the processed images is important.

As a variant, the images worked are generated by modifying the saturation of the pixels of the images taken by the camera according to the norm and the direction of the velocity vector of each of said pixels so that the direction of the velocity vector of one pixel is center-oriented images taken by the camera and the higher the speed vector standard of said pixel, the higher the saturation of said pixel in the images worked is important. It will be possible to implement the method of the invention via images taken by the camera in gray level or through images taken by the camera in color. Preferably, we will use images taken in color. Indeed, if the representation of a shadow is mobile from one image to another, it constitutes in itself the figuration of an element in motion. For black-and-white images, the shadow figuration is highlighted on the images worked, which is of little interest to the user. On the other hand, with color images, it is easier to detect the representation of a shadow so that one can then remove at least part of said shadow. Preferably, at least part of the shadow of the user will be removed if it is present on the images.

The sensor, the display member and, in general, the portable augmented reality device implemented can be different from that described here by way of example.

The method may be implemented by a different number of sensors than indicated and by other sensors than those described as a laser, a camera, a touch interface, an electron scanner, etc.

In particular, although in the detailed description the sensor is a camera directly generating images, it will more generally be possible to use a sensor adapted to generate digital data organized in matrix form forming the information to be analyzed.

Most sensors acquire information in analog form, for example a wavelength, and convert it into digital form. The observation of a scene using a sensor makes it possible to obtain data relating to physical quantities, for example optical, sound, tactile (touch screen) data, the detection of secondary electrons, etc. These analog physical data are translated into digital data that can be organized in a matrix form. One can thus consider an n dimensional matrix involving spatial coordinates, temporal coordinates and coordinates depending on the physical data.

Then, from these organized data it is possible to generate a digital image of the scene in which the pixels will be determined / coded according to the digital data of interest.

Then, such a digital image will thus include a representation of the elements of the scene according to the physical data associated with them, the representation of each of the elements is qualified as figuration.

When generating a succession of digital images from a sensor it is possible to produce digital video. Typically the matrices associated with the digital images used for the video contain spatial and temporal data that make it possible to follow the evolution of the physical magnitudes of the elements of the scene.

The invention thus relates to real-time analysis of the differences existing between two successive matrix data, in particular in order to determine the movement of the elements present within the scene, and then to generate in real time processed images of the scene comprising a representation of the image. moving element as marked, in which at least one visual characteristic of at least the figuration of the moving element is determined according to the analyzed information to highlight said element relative to other immobile elements of the scene.

In order to show more precisely the interest of the invention a study has been carried out on patients in order to to show its impact on their ability to detect a person who enters their field of vision. For this, a first experiment was performed showing only the optical flux exceeding a certain speed threshold. Patients have been shown to benefit from the use of the invention compared to the original video by testing latencies of recognition (p = 0.002, Wilcoxon paired test).

A second study was conducted to observe the impact of the invention on the recognition of specific movements. The experiment was carried out using the invention or a treatment proposed by Peli (see for example US Patent 6,611,618). The display method used for this second experiment consisted, this time, in changing the luminance of the original image in proportion to the speed of the optical flux in one pixel; an immobile pixel being kept of the same luminance, compared to the original image. This treatment has preserved the context of the image while accentuating the moving regions.

The use of the invention has improved motion recognition for each patient compared to original videos or videos processed with the Peli method. After Mannheim test it was shown that these differences were significant in the recognition time by comparison between a video processed using the invention and the untreated video (p = 2.9811e-013), the difference is also significant if we compare the use of the invention with the Peli protocol (p = 0.028686).

Claims

A method of highlighting at least one moving element in a scene, the method comprising the steps of:

- observe (1) the scene using at least one sensor;

analyzing (2) information generated by the sensor to locate the moving element in the scene;

generating (3) real-time processed images of the scene comprising a figuration of the moving element as identified, in which at least one visual characteristic of at least the figuration of the moving element is determined according to analyzed information to highlight said element relative to other immobile elements of the scene;

projecting (4) the images thus generated on a display member.

2. Method according to claim 1, wherein the sensor is adapted to generate digital data organized in matrix form forming the information to be analyzed.

The method of claim 2, wherein the visual characteristic is determined according to at least one inertial characteristic related to the motion of the element in the scene.

4. The method of claim 3, wherein the analysis step comprises the real-time calculation of optical flows to estimate the inertial characteristic, the flow being a vector field which represents the displacement of each pixel between two images generated from successive matrix data.

5. Method according to claim 4, wherein the analysis step comprises the real-time calculation of the dense optical fluxes taking into account all the pixels of the images generated from successive matrix data.

The method of claim 5, wherein calculating the dense optical flows in real time comprises implementing a dense multi-level pyramidal dense differential optical flux method.

The method of claim 6, wherein the pyramid scheme dense differential optical flux method includes minimizing a functional (F = F _reg + F _atia ) where the attachment term (F _aHa ) is based on a thresholding method and the regularization term (F ₀ ) is based on a U + V method.

8. The method of claim 7, wherein for each level of the pyramid is considered a first image I _n and a second image I _{n + 1} so that for each pixel n of the second image I _{n + [} we implement the following iterative loop until minimizing the functional F to determine the components (u _n , v _n ) of the dense optical flux of said pixel, the components (u _n , v _n ) being taken to zero at the beginning of the loop.

The method of claim 8, wherein the wavelet filter is on non-stationary wavelet packets.

10. The method of claim 4, wherein the inertial characteristic comprises velocity vectors which are respectively associated with each of the pixels of the images, the computation in real time of the optical fluxes for estimating said velocity vectors.

The method according to claim 3, wherein the processed images are generated according to a Luminance Saturation Hue template, the visual characteristic determined as a function of at least one inertial characteristic being one of the three parameters of said model so that the representation of the moving element has a hue, saturation or luminance, respectively which differentiates figuration from other motionless elements of the scene.

The method according to claims 10 and 11, wherein the processed images are generated by assigning each pixel of the processed image:

a first luminance if the velocity vector associated with the corresponding pixel of the image generated from the successive matrix data has a norm greater than a given threshold;

a second luminance, lower than the first, if the velocity vector associated with the corresponding pixel of the image generated from the successive matrix data has a norm lower than the given threshold.

13. The method of claims 10 and 11, wherein the processed images are generated by modifying the images generated from the successive matrix data.

14. The method of claim 13, wherein the images processed are generated by modifying the luminance of the pixels of the images generated from the successive matrix data in proportion to the norm of the velocity vector of each of said pixels.

The method according to claim 13, wherein the images processed are generated by modifying the luminance of the pixels of the images generated from the successive matrix data according to the norm and the direction of the velocity vector of each of said pixels so that more the direction of the velocity vector of a pixel is oriented towards a center of the images generated from the successive matrix data and the higher the standard of the velocity vector of said pixel, the higher the luminance of said pixel in the processed images is important.

16. An augmented reality portable device (10) having at least one sensor (12, 13) for viewing the scene, a calculating member (14) for analyzing the information generated by the sensor, for generating real-time processed images of the scene and projecting the processed images and a display member, wherein the calculating member comprises a calculation module in parallel for implementing the method according to the invention. one of the preceding claims.