US20120077163A1

US20120077163A1 - 3d monocular visual tracking therapy system for the rehabilitation of human upper limbs

Info

Publication number: US20120077163A1
Application number: US13/027,157
Authority: US
Inventors: Luis Enrique Sucar Succar; Roger Luis-Velasquez; Gildardo Azcarate-Hernandez; Ronald Sturat-Leder; David Reinkensmeyer
Original assignee: Instituto Nacional de Astrofisica Optica y Electronica
Current assignee: Instituto Nacional de Astrofisica Optica y Electronica
Priority date: 2010-02-12
Filing date: 2011-02-14
Publication date: 2012-03-29
Also published as: CA2731775C; CA2731775A1; MX2011001698A; US20130165825A1

Abstract

It is described a 3D monocular tracking system, being robust, having low cost, easy to install and use, useful for the upper limbs rehabilitation in a patient in need thereof, as well as a home self-directed therapy method for patients having upper limbs' movement disability. The system comprising a) a handle or gripper; b) a computational vision system comprising a video camera; c) software comprising a set of games; d) a processor; and, e) a display apparatus.

Description

TECHNICAL FIELD

The present invention relates to equipment and systems used in rehabilitation medicine to recover limb motility for patients suffering from cardiovascular diseases or other kinds of disease, and more particularly, it is related to a 3D monocular visual tracking therapy system for the rehabilitation of the upper limbs of patients having suffered from any kind of injury such as a stroke, as well as to the method of carrying out such rehabilitation.

BACKGROUND OF THE INVENTION

Every year millions of people worldwide suffer accidents or diseases which cause the loss of their motor abilities. Cerebrovascular diseases, commonly known as strokes, are clear examples thereof. About 80% of the people which survive to a stroke lose their movement ability in an arm and hand.
After having suffered from a stroke, an intensive activity therapy for several weeks is the most common treatment to recover the movement abilities. However, due to the increased budgetary pressures in the hospitalization system, more often the rehabilitation treatments are reduced and the patients are sent home early without having reached a correct and vital rehabilitation. On the other hand, hiring a professional physiotherapeutic is not an option for most of the patients due to the high cost of the therapy session.
In view of the above, patients try to rehabilitate themselves by making the same exercises assigned at the hospital; however, as they do not have the guides neither have they the knowledge thereto, usually they exercise in an inappropriate manner, resulting in a low or null progress in their rehabilitation. In other cases, due to laziness or lack of motivation, the patients do not exercise at all.
Accordingly, a cost-effective solution for a self-directed therapy system for the home is necessary so a higher number of patients can have it.
In order to help patients to recover their higher limbs' movement, either of the arm or the hand, several systems have been developed. Most of these systems consist in complex robotic arms or gloves having sensors to detect the patient's arm movement, making them expensive and not accessible for most of the patients.
Due to the great advances in computer technology, computational vision has now a wide variety of applications, within we can mention human-machine interface applications, virtual reality, animation and movement capture, including applications of increased reality or of tracking. In this context, rehabilitation systems based on the visual tracking of the human movement have become another alternative for the patients, which may be based on marks, wherein the image is captured by cameras, tagging the body's joints, or alternatively, free of tags using conventional video cameras to capture the movement.
The combination of several factors is required for the use of the systems based on visual tracking for the rehabilitation of patients, i.e., they may have a low cost and, at the same time, have a high accuracy and ability to be executed in real time.
Visual tracking systems can meet only part of these requirements, since its design presents various difficulties, such as depth inconsistencies, feature deformities, complexity in the kinematics' models and occlusions. To simplify these problems, most of the algorithms to carry out the tracking employ tri-dimensional models of the person's shape or multiple cameras to enhance the robustness.
According to the above, several visual tracking systems are found in the prior art, which are focused on different applications, such as the International Publication No. WO 2008/134745 focused in the therapy of patients having some physical or cognitive disability. Said application discloses a portable therapy apparatus (a car) which captures images of the patient using 2 or more cameras (stereo) or depth cameras (depth camera) to estimate the 3D position, and to select and control one of the therapeutic applications based on the gestures recognition of the user, which are automatically detected from the images. The system contemplates the selection of the different activities by the user or by the attendant; however, the activities are not automatically adapted according to the progress of the patient. Another drawback is that the use of several cameras makes the system more expensive and complex.
On the other hand, in the North American Patent Application No. 2008/0085048, a computer controlled system is disclosed, which allows a human to control a robotic apparatus using gestures and movements being recognized by the system, and causing the robotic apparatus to react thereto. The system includes, among other components, a video camera recording the image and a software implemented in a computer allowing the recognition of dynamic gestures and static poses of a user. However, this system is not used in therapeutic applications.
The tracking device disclosed in the US Patent Application No. US 2006/0274032 is used to get information to control the execution of a game program, this device comprises a body mountable on a game control or on the user's body, and an inertial sensor (accelerometer, mechanic gyroscope or laser gyroscope) operating to produce 3D information to quantify a body's movement through the space. The system may further comprise a camera and the controller may include LEDs to facilitate the tracking by video analysis. This device, in addition to being focused for use in video games, can only estimate the movement and not the 3D position since it uses inertial sensors to detect the 3D movement.
Likewise, US Patent Application No. US 2006/0209021, related to an apparatus and a method to move a virtual mouse using a video camera, which traces an input gesture, extracts from the image of the input gesture the region corresponding to the right or left hand, recognizes each hand's gesture, as well as the command corresponding to said gesture, and executes the command. However, this apparatus has as object providing the user with a more convenient interface to devices such as a computer, and not its application in the therapy of patients requiring upper limbs rehabilitation. In addition, the follow-up of the movement is not carried out in 3D.
On the other hand, US Patent Application No. 2005/0255434 discloses a training interactive system comprising computational vision and including, among other components, at least one video camera to get images of the trained person, as well as pattern recognition algorithms and image analysis to recognize features in the images and thereby detecting gestures of the trained person. The system requires of additional elements like LEDs to be able to carry out the follow-up of the people in training. This system is used in training applications but not in therapeutic applications.
The U.S. Pat. No. 7,262,760 claims the use of a 3D pointer apparatus which transforms sensed movement data from a first reference frame into a second reference frame. The system includes at least one sensor to detect the rotation of the pointer apparatus, an accelerometer to detect its acceleration and a processor to receive the sensor outlet and that of the accelerometer, all of which increases the systems costs.
Finally, U.S. Pat. No. 6,256,033 describes a computer implemented method to recognize a person's gestures within an image sequence and executing an operation based on the semantic meaning of the gesture, wherein the subject enters in the vision field of a camera connected to a computer and makes a gesture. The gesture is examined by the system by means of a program, one image at a time, thereby deriving position data and comparing them with previously derived data representing gestures already known by the system. The comparisons are made in real time and the system may be trained to recognize new gestures. The main drawback of this method is that the recognition is made examining one image at a time and comparing the data with previously derived data and already known by the system. On the other hand, this method is not used in therapeutic applications.
As it may be seen from the above, most of the visual tracking systems found in the prior art have the great inconvenience that they were designed and developed for training or to video games, but no so for the rehabilitation of patients who have suffered a stroke. In addition to the above, some systems found in the state of the art used in the rehabilitation of patients include apparatus or devices highly expensive, or require the implementation of various components such as the use of several cameras, among other, representing a big problem for the hospital system due to the decrease in the budget, and therefore, this means that patients are sent to home before completing their rehabilitation. And if this situation moved into the home of the user, wherein said user has to use own resources, the rehabilitation practically becomes impossible.

BRIEF DESCRIPTION OF THE INVENTION

As it was mentioned above, therapy systems found in prior art used in upper limbs, arm and hand, of patients having suffered from stroke use very expensive devices, such as the case of the system using a robotic arm, thereby limiting its accessibility. Moreover, when it is used in a fixed manner on the arm of the patient, impeding the execution of natural movements.
In addition to the above, as well as budget problems of the hospitals that sending patients to their home before completing his rehabilitation, which requires that self-directed therapy systems in home reach a greater number of patients, and it is required that said systems have an affordable cost.
In view of the aforementioned, a low cost visual tracking therapy system has been developed, with suitable accuracy and executable in real time, wherein said tracking is such that allows recovering the three-dimensional (3D) position of an object, thus allowing the rehabilitation of patients having suffered from stroke for restoring their upper limbs movement.
The therapy system of the present invention generally comprises: a conventional digital camera allowing the capture of images, which does not require additional calibration or standards; a display apparatus configured to show patient interacting 3D virtual environments; a handling element in a grip-shaped that includes a colored sphere located at each one of its ends; a processor or CPU allowing the processing of numerical information; and, a software including two computer programs. One program carries out a 3D visual tracking of a patients' hand, by estimating its 3D spatial position every instant, and sends it to the games system. The second program includes a set of games oriented to daily life activities, which include different difficulty levels to evaluate the patient's progress at every game. Whole system is called “Gesture Therapy”.
The therapy system of the present invention is a low cost system, easy to install and operate, using a conventional digital camera which does not require additional supports to determine, through the tracking of the handle spheres, the position and 3D movement of the patient's arm and/or hand, as well as its rotation around 3 orthogonal axis.
Said system is capable of working in various lightning conditions, and to eventually overcome harsh movements. Moreover, it allows to overcome acceptably being out of frame, i.e., when the tracked object gets out of the visibility field of the camera and then comes back, even if the getting out and re-entry points differ.
Additionally, the therapy system of the present invention is a recognizing and tracking system for a specific point in the patient's limb, i.e., it is capable of knowing at any time where the hand is, overcoming the problems regarding depth detection, such that the use of a single digital camera is enough to carry out the detection and the 3D tracking.
The processor is configured for:

- selecting from a set of games, an appropriate game for the patient;
- calling the game;
- providing the required algorithms to determine the position and 3D movement of the patient's hand;
- recognizing the position and 3D movement of the patient's arm and/or hand;
- controlling the game based on the position and 3D movement of the patient's arm and/or hand;
- adapting to the patient according to his (her) condition and therapy progress; automatically alternating, for example, the working space for the limb's movement;
- detecting the pressure when the patient tights/loses the joystick by a pressure sensor incorporated therein, thereby promoting the rehabilitation of the hand's movements.

OBJECTS OF THE INVENTION

Considering the defects of the prior art, it is an object of the present invention to provide a 3D monocular visual tracking therapy system, easy to install and operate, but highly efficient for the rehabilitation of patients having suffered from stroke.
Another object of the present invention is to provide a monocular visual tracking therapy system being robust, having low cost, easy to install and use, useful for the upper limbs rehabilitation in a patient in need thereof.
Moreover, is another object of the present invention to provide a home self-directed therapy method for patients having upper limbs' movement disability.

BRIEF DESCRIPTION OF THE DRAWINGS

The novel aspects considered characteristic of the present invention are established particularly in the appended claims. However, the invention itself, both in its configuration and in its operation method, together with other objects and advantages thereof, will be better understood in the following detailed description of a specific embodiment, when read along with the appended drawings, in which:

FIG. 1 is a schematic representation wherein the monocular visual tracking therapy system is shown, constructed according to a specific embodiment of the present invention.

FIG. 2 represents the use of the 3D monocular tracking system for the rehabilitation of a patient's upper limbs. In left upper box a possible configuration of the system is illustrated, with a screen and a camera, through which the 3D tracking of the ball in the patient's hand is carried out. In the other 3 boxes are illustrated different game examples at the screen, oriented to rehabilitation, for the patient to interact with the movement of his (her) hand.

DETAILED DESCRIPTION OF THE INVENTION

The present invention is related to a 3D monocular tracking system for the rehabilitation of a patient's upper limbs.
During the development of the present invention it was found that this system determines the exact 3D position of the arm and/or hand, based on a single video camera, therefore not requiring additional sensors and its cost is lower than the traditional systems.
Accordingly, in a one aspect of the present invention, a 3D monocular tracking system for the rehabilitation of a patient's upper limbs is described, preferably of a patient having suffered from stroke, which comprises:
a) a handle, joystick or grip-shaped comprising a cylindrical handle and two colored spheres, one at each end of the gripper (FIG. 1);
b) a computational vision system comprising a video camera to determine, through the tracking the handle balls, the position and 3D movement of the patient's arm and/or hand position, as well as its rotation around 3 orthogonal axis;
c) a gesture therapy software comprising a program for 3D tracking of the two colored spheres in the handle, a set of games oriented to daily life activities, which include different difficulty levels to evaluate the patient's progress at every game;
d) a processor configured for:

- selecting between the set of games, an appropriate game to the patient;
- calling the game;
- providing the required algorithms to determine the position and 3D movement of the patient's hand;
- recognizing the position and 3D movement of the patient's arm and/or hand;
- controlling the game based on the position and 3D movement of the patient's arm and/or hand;
- adapting to the patient according to his (her) condition in the therapy progress; automatically alternating, for example, the working space for the limb's movement;
- detecting the pressure when the patient tights/loses the joystick by a pressure sensor incorporated therein, thereby promoting the rehabilitation of the hand's movements.
  e) a display apparatus configured to show 3D virtual environments that the patient can interact.

In a preferred embodiment of the present invention, the handle or gripper further comprises one or more pressure or strength sensors, for determining when the patient tightens the joystick and the force with which is performed, thus measuring the force in the whole hand or in each finger. This measurement may be sent to the processor and incorporated in the virtual environments. Thereby, the patient's hand and fingers may also be exercised, and his (her) ability to grip and lose, which are important rehabilitation elements.
In other preferred embodiment of the present invention, the video camera is selected from a web camera, a camera integrated to the processor (computer), or an infrared camera; with the web camera being particularly preferred.
Likewise, in an additional embodiment of the present invention, the display apparatus is selected from a computer screen, television monitor, digital personal assistant screen (PDA, by its English initials), cellular phone screen; with the computer screen being particularly preferred.
The use of the 3D monocular tracking system of the present invention is shown in FIG. 2. FIG. 2 a represents a computer with the Gesture Therapy software and a web camera to carry out the 3D monocular tracking, as well as the patient's hand holding the handle with one of the colored spheres. Likewise, some useful games for the rehabilitation of the patient's arm and/or hand are shown. FIG. 2 b shows a car racing game allowing the patient to train lateral arm movements to control the car without getting out of the track. FIG. 2 c shows a game consisting in cleaning a stove, being useful to train movements allowing stretching and shrinking the arm. Finally, the game in FIG. 2 d consists in painting specific squares in a cube, thereby training the shoulder and elbow movements in different directions. In this last example the pressure sensor may be incorporated to “take” the paint, thereby training the hand's movements (opening and closing).
In FIG. 1, a therapy system (1000), comprised of a handle or gripper (100), a web camera (200) and a display apparatus (300).
In FIG. 3: a handle or gripper (100), comprised by a gripper (110) and two colored spheres (120).
Further aspects of the present invention consider a home self-directed therapy method for patients having upper limbs' movement disabilities, preferably one having suffered from stroke, characterized by comprising:

- offering a patient a monocular 3D tracking system for the rehabilitation of the arm and/or hand;
- selecting between a set of games, an appropriate game for the rehabilitation of the patient;
- calling the game;
- recognizing the position and 3D movement of the patient's arm and/or hand;
- detecting when the patient tightens or loses the handle;
- showing 3D virtual environments, visible for the patient, wherein the 3D virtual environments respond to the 3D movement of said patient's arm and/or hand;
- repeating the above steps during the necessary period of time. The repetitions number depends on the state of the patient and the severity of the injury; in clinical studies it has been observed that after 10 sessions of one hour with the system, most patients show a significant improvement according to the clinical indexes.

According to the above-described, it may be seen that the system and method of the present invention have been envisioned to provide a robust system, having a low cost, easy to install and use, useful for the rehabilitation of the arm and hand's movement in people in need thereof, and it will be obvious for those skilled in the art that the embodiments of the 3D monocular tracking system for the rehabilitation of a patient's upper limbs, as well as for the home self-directed therapy method for patients having upper limbs' movement disability, as described above and shown in the drawings, should be considered as illustrative and non-limitative of the present invention, since several detail changes are possible without departing from the scope of the invention.
The present invention will be better understood form the following examples, which shall be construed only as illustrative to permit a better understanding of the preferred embodiments of the present invention, without implicating that there are not further embodiments does not illustrated therein capable of being practiced based on the above detailed description of the invention.

EXAMPLES

The system was used in a group of 22 patients having suffered from stroke in a hospital (Rehabilitation Unit of the Neurology and Neurosurgery Institute in Mexico City). The patients used a prototype of the system in the hospital, interacting with different games, guided by a therapist. The therapy lasted 7 weeks, with 3 sessions per week; before and after the therapy each patient was assessed by a therapist using two different clinical ranges (Fugl-Meyer index and motricity index). The results show a statistically significant improvement in both ranges after the sessions with the “Gesture Therapy” system. In addition, a motivation survey was made to all patients after having used the system, the survey results showing a much higher motivation compared to the “traditional” therapy.
Therefore, the present invention shall not be considered as restricted, except for the prior art demands and by the scope of the appended claims.

Claims

1. A 3D monocular tracking system for the rehabilitation of a patient's upper limbs, characterized by comprising:

a) a handle or gripper comprising a cylindrical gripper and two colored spheres, one at each end of the gripper (FIG. 1);

b) a computational vision system comprising a video camera to determine, through the tracking of the handle balls, the position and 3D movement of the patient's arm and/or, as well as its rotation around 3 orthogonal axis,

c) a Gesture Therapy software comprising a program for 3D tracking of the two colored spheres in the handle, and a set of games oriented to daily life activities, which include different difficulty levels to evaluate the patient's progress at every game;

d) a processor configured for:

selecting between the set of games, an appropriate game to the patient;

calling the game;

providing the required algorithms to determine the position and 3D movement of the patient's hand;

recognizing the position and 3D movement of the patient's arm and/or hand;

controlling the game, based on the position and 3D movement of the patient's arm and/or hand;

adapting to the patient according to his (her) condition in the therapy progress; automatically alternating, for example, the working space for the limb's movement;

detecting the pressure when the patient tights/loses the joystick by a pressure sensor incorporated therein, thereby promoting the rehabilitation of the hand's movements.

e) a display apparatus configured to show 3D virtual environments that the patient can interact.

2. A system, according to claim 1, further characterized in that the handle or gripper comprises one or more pressure or strength sensors allowing to determine when the patient tightens the handle and the force with which is performed.

3. A system, according to claim 1, further characterized in that the video camera is selected from a web camera, a processor integrated camera (computer), an infrared camera.

4. A system, according to claim 3, further characterized in that the video camera is a web camera.

5. A system, according to claim 1, further characterized in that the display apparatus is selected from a computer screen, a television monitor, a digital personal assistant screen (PDA, by its English initials), a cellular phone screen.

6. A system, according to claim 5, further characterized in that the display apparatus is a computer screen.

7. A system, according to claim 1, further characterized in that the patient has suffered from stroke.

8. A home self-directed therapy method for patients having upper limbs' movement disability, preferably one having suffered from stroke, characterized by comprising:

offering a patient a monocular 3D tracking system for the rehabilitation of a patient's upper limbs;

selecting, from the set of games, an appropriate game for the rehabilitation of the patient;

calling the game;

recognizing the position and 3D movement of the patient's arm and/or hand;

showing 3D virtual environments, visible for the patient, wherein the 3D virtual environments respond to the 3D movement of said patient's arm and/or hand;

adapting to the patient according to his (her) condition and therapy progress; automatically alternating, for example, the working space for the limb's movement;

repeating the above steps during the necessary period of time.

9. A method, according to claim 8, further characterized in that the system comprises:

a) a handle or gripper comprising a cylindrical gripper and two colored spheres, one at each end of the gripper (FIG. 3);

b) a computational vision system comprising a video camera to determine, through the tracking of the handle balls, the position and 3D movement of the patient's arm and/or hand, as well as its rotation around 3 orthogonal axis (FIG. 1 and FIG. 4);

c) a Gesture Therapy software comprising a set of games oriented to daily life activities, which include different difficulty levels to evaluate the patient's progress at every game;

d) a processor configured for:

selecting, from the set of games, an appropriate game to the patient;

calling the game;

recognizing the position and 3D movement of the patient's arm and/or hand;

10. A method, according to claim 9, further characterized in that the handle or gripper comprises one or more pressure or strength sensors allowing to determine when the patient tightens the handle and the force with which is performed.

11. A method, according to claim 9, further characterized in that the video camera is selected from a web camera, a processor integrated camera (computer), an infrared camera.

12. A method, according to claim 11, further characterized in that the video camera is a web camera.

13. A method, according to claim 9, further characterized in that the display apparatus is selected from a computer screen, television monitor, digital personal assistant screen (PDA, by its English initials), cellular phone screen.

14. A method, according to claim 13, further characterized in that the display apparatus is a computer screen.

15. A method, according to claim 8, further characterized in that the patient has suffered from stroke.