|Publication number||WO2011067469 A1|
|Publication date||9 Jun 2011|
|Filing date||2 Dec 2010|
|Priority date||2 Dec 2009|
|Publication number||PCT/2010/50990, PCT/FI/10/050990, PCT/FI/10/50990, PCT/FI/2010/050990, PCT/FI/2010/50990, PCT/FI10/050990, PCT/FI10/50990, PCT/FI10050990, PCT/FI1050990, PCT/FI2010/050990, PCT/FI2010/50990, PCT/FI2010050990, PCT/FI201050990, WO 2011/067469 A1, WO 2011067469 A1, WO 2011067469A1, WO-A1-2011067469, WO2011/067469A1, WO2011067469 A1, WO2011067469A1|
|Applicant||Virtual Air Guitar Company Oy|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (4), Non-Patent Citations (1), Referenced by (18), Classifications (26), Legal Events (3)|
|External Links: Patentscope, Espacenet|
A METHOD, A SYSTEM AND A COMPUTER PROGRAM FOR INSTRUCTING AND EVALUATING FULL-BODY MOVEMENT
FIELD OF THE INVENTION
The invention relates to human-computer interaction and machine vision. In particular, the invention relates to a method, a system and a computer program for instructing and evaluating movement in a computerized movement instruction system.
BACKGROUND OF THE INVENTION
There are many computerized movement in¬ struction systems and games, e.g., dancing games. How¬ ever, the movements are usually simplified because in¬ structing and judging full body movement is a diffi¬ cult communication and analysis problem.
Many movement instruction systems are based on the concept of a timeline so that visual instruc¬ tions move linearly on screen along an axis that represents time. For example, in many dancing games the time axis is vertical and there are arrows moving upwards from the bottom of the screen and the player has to step on buttons corresponding to the arrows when the arrows reach the top of the screen. There are also other approaches, e.g., the Just Follow Me vir¬ tual reality movement instruction system, where the teacher' s visual 3d representation overlaps with the student body in the virtual reality and the student's goal is to align his or her body with the teacher. However, a timeline enables the user to predict what actions are expected in the future, which is crucial to perform complex choreographed moves. Karaoke games such as Sing Star also provide an example of a simple horizontal timeline. In the games, the main or only input signal is the pitch of the player's voice. This allows the game to display both the player' s desired and actual action as simple 2d line graphs with the horizontal axis representing time and vertical axis representing pitch.
In instructing full-body movement in an interactive manner, e.g., in a dancing game or a choreo¬ graphed martial arts game, the main problem is the multidimensionality of the data. The player's body configuration can be represented as a real-valued vec- tor b, e.g., with each element corresponding to an angle of a body joint. Displaying the desired behavior b(t) of the body as a function of time using simple graphs would require either more than three coordinate axes, or several graphs, which is not visually intui- tive. More efficient and easy to understand visualiza¬ tion systems need to be developed. This invention pro¬ vides a solution to instructing time-varying full body movement precisely, predictably and clearly.
In this document, the term "align" is often used, e.g. in the phrase "controlling the appearance of the preview instruction so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction". By aligning, we mean that predetermined parts of the vis- ual objects of interest (in this case the preview and scrolling instructions) are located on screen within a predetermined distance from each other. Please see further explanation for the term "align" at the detailed description of the invention with references to figures 3 and 4. A first aspect of the present inven¬ tion is a method of instructing the user' s movement and poses in a computerized movement instruction sys¬ tem. The user's visual representation is displayed. A graphical instruction scrolling on a timeline is dis- played. A graphical preview instruction is displayed. The appearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction. The appearance of the preview instruction is controlled so that it aligns with the scrolling instruction essentially at the moment the user should match the scroll- ing instruction. It is determined whether the user matches the instructions within a time interval deter¬ mined in relation to the moment the preview instruc¬ tion and the scrolling instruction align. Feedback indicating whether the user matched the instructions is provided.
A second aspect of the present invention is a computerized movement instruction system for instruct¬ ing the user's movement and poses. The computerized movement instruction system comprises a screen config- ured to display the user's visual representation. The computerized movement instruction system further comprises a computer configured to generate a graphical instruction scrolling on a timeline and a graphical preview instruction, for display on the screen. The computer is further configured to control the appear¬ ance of the preview instruction so that it becomes visible before the moment the user should match the scrolling instruction. The computer is further configured to control the appearance of the preview instruc- tion so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction. The computer is further configured to determine if the user matches the instructions within a time interval determined in relation to the moment the preview instruction and the scrolling instruction align. The computer is further configured to provide feedback that indicates whether the user matched the instructions.
A third aspect of the present invention is a computer program for instructing the user' s movement and poses in a computerized movement instruction sys¬ tem. The computer program is embodied on a computer- readable medium comprising program code means adapted to perform the following steps when the program is executed in a computing device:
displaying the user's visual representation; displaying a graphical instruction scrolling on a timeline;
displaying a graphical preview instruction; controlling the appearance of the preview instruction so that it becomes visible before the moment the user should match the scrolling instruction;
controlling the appearance of the preview instruction so that it aligns with the scrolling instruction essentially at the moment the user should match the scrolling instruction;
determining whether the user matches the instructions within a time interval determined in rela¬ tion to the moment the preview instruction and the scrolling instruction align; and
providing feedback that indicates whether the user matched the instructions.
In an embodiment of the invention, the determining whether the user matches the instructions com¬ prises detecting that the hands and feet of the user' s visual representation essentially align with the loca- tions of hands and feet illustrated by the instruc¬ tions .
In an embodiment of the invention, the pro¬ viding the feedback comprises stopping the scrolling instruction when it aligns with the preview instruc- tion and displaying the stopped instruction until the user has matched it or until a predetermined time has passed .
In an embodiment of the invention, the pro¬ viding the feedback comprises increasing the user' s score and making the stopped or scrolling instruction disappear when the user matches the instruction. In an embodiment of the invention, the determining that the user has matched said instructions comprises detecting that the user' s visual representa¬ tion collides with at least one of said instructions, and that the colliding part of the user' s visual rep¬ resentation moves in a predetermined direction at a predetermined speed.
In an embodiment of the invention, the user' s visual representation comprises a real-time video im- age of the user captured using a video camera.
In an alternative embodiment, the preview in¬ struction may comprise a circle and the scrolling in¬ struction may comprise a human silhouette, and the in¬ structions are considered to align when the center of the circle and the right hand of the silhouette are located on screen within a predetermined distance from each other. In this case, an example of the player matching the instructions is that the player must move so that the center of the palm of his video image moves inside a circle of the preview instruction half a second before or after a hand of a human silhouette in the scrolling instruction moves to the center of the said circle.
In an embodiment of the invention, the ap- pearance of the preview instruction is controlled so that it becomes visible before the moment the user should match the scrolling instruction. In graphics programming terms, a typical example of this is that the preview instruction is a static image of a human silhouette and the opacity of the image is controlled so that the image transforms from transparent to opa¬ que one second before a scrolling instruction with a similar silhouette aligns with it.
Compared to previous movement instruction systems, e.g., dancing games that display moving ar¬ rows on a timeline, this invention provides at the same time more precise and predictable instructions. This invention also provides the user more precise feedback about his or her performance. The combination of dynamically appearing preview instructions, scroll¬ ing instructions and the player' s visual display over- laid with the instructions enable one to represent both time and spatial dimensions on the same visual axes. The combination allows one to fully specify the user's desired body configuration (pose) at a prede¬ termined time, approximating b(t) in a piecewise man- ner. The combination also allows the user to visually compare his or her current pose to the instructions. All these three elements (space, time, feedback about the user' s full body) are necessary for a movement in¬ struction system that does not limit the range of in- structed movements.
BRIEF DESCRIPTION OF THE DRAWINGS
The accompanying drawings, which are included to provide a further understanding of the invention and constitute a part of this specification, illus¬ trate embodiments of the invention and together with the description help to explain the principles of the invention. In the drawings:
Fig. 1 is an illustration of an example im- plementation of a system according to the present invention,
Fig. 2 is an illustration of an example of the graphical output rendered by a system according to the present invention,
Fig. 3 is a further example of the graphical output rendered by a system according to the present invention .
Fig. 4 is a further example of the graphical output rendered by a system according to the present invention. Fig. 5 is a further example of the graphical output rendered by a system according to the present invention . DETAILED DESCRIPTION OF THE INVENTION
Reference will now be made in detail to the embodiments of the present invention, examples of which are illustrated in the accompanying drawings.
In Figure 1, a typical embodiment of the pre- sent invention is illustrated. The user 10 moves in the vicinity of a sensor device 11, which is a camera in the example illustrated. The computer 15 renders on the screen 12 visual output comprising the user' s vis¬ ual representation 13 and other computer graphics 14, for example, the enemies in a fighting game. The us¬ er's visual representation 13 may overlap the computer graphics 14, allowing the user to virtually touch and interact with computer generated objects and agents. The user's visual representation 13 may be the play- er's real-time video image captured using the camera 11, possibly processed with image processing algo¬ rithms, e.g., to remove the background of the user from the image. The user's visual representation 13 may also be a computer generated 2d or 3d avatar that copies the user's movements based on motion tracking algorithms applied to the data provided by the sensor device 11.
The basic principle of the present invention is to show the user of a movement instruction system a plurality of snapshots of the desired body configura¬ tions on a timeline. In an embodiment of the inven¬ tion, the choreographer may choose a number of body configurations from a sequence of movements. The in¬ struction system may then be configured to show visual instructions, e.g., human silhouettes corresponding to the desired body configurations (poses) on a timeline, and also the user' s visual representation superimposed on the timeline. The goal of the user is to match the instructions by aligning or moving his/her visual rep¬ resentation with the instructions at predetermined mo¬ ments .
However, displaying the user and scrolling instruction graphic on a timeline is not enough, be¬ cause one needs to instruct both time and location in space. For example, most martial arts choreographies consist of both steps and poses so that the poses should be matched at the right time and in the correct location in space. Time adds a fourth dimension to 3d space, so that when instructing three-dimensional movement as a function of time with 3d graphics, one has to show both time and some position variable on the same axis. Because of this ambiguity the user may not know if he/she is supposed to move towards a scrolling instruction or to wait until the instruction comes closer. Furthermore, in many cases it would be desirable to use only 2d graphics to make the overall visual appearance simpler.
In a method and system according to this invention, the abovementioned problems are solved so that the visual instructions are split into two parts, a scrolling instruction and a preview instruction that are configured to align at the moment the user should also align with them or otherwise match the instruc¬ tions. From the desired moment of matching we use the term matching time. The user should also move to the place where the scrolling and preview instructions align. This way, the system can instruct both the us¬ er' s body configurations and location with respect to time. In a typical embodiment of the invention, the scrolling instructions depict body poses and move li¬ nearly along the timeline, and the preview instruc- tions stay in the location where the user should match the pose, and only become visible, e.g., one second before the matching time. This minimizes the amount of visual information and maximizes the clarity of pres¬ entation .
In the following, the description is mainly written from the point of view of a movement based game or a movement training game where the computer provides the user pose instructions that he/she must match. The computer gives the user feedback, e.g., with visual and sound effects and by increasing the user's score after successful matching of the poses.
In Figures 2-4, examples of the visual output of a system according to the present invention are illustrated. Figures 2-4 are in chronological order to illustrate the behavior of the system over time. The user's visual representation 20, 30, 40 is shown to- gether with instruction graphics 21, 22, 23, 31, 32, 33, 34, 35, 36, 37, 39, 41, 42, 43, 44, 45, 46, 47. The instruction graphics move on a timeline so that their position varies as a function of time. In Figures 2-4, the time is mapped to the horizontal coordi- nate axis so that the graphics scroll from right to left. However, the user's visual representation 20, 30, 40 and the preview instructions 31, 41 do not typ¬ ically scroll.
In Figure 2, the user is shown scrolling in- structions 21, 23 and a movement trail instruction 22 that indicates the desired motion of the user' s hand after matching the pose depicted by the scrolling instruction 21. Such movement trails may be provided for any body part. The movement trails may also indicate the desired motion before matching the pose.
In Figure 3, time has passed in relation to Figure 2, and the instructions 21, 22, 23 of Figure 2 have scrolled left, now shown as the instructions 32, 37, 33, respectively. As illustrated by the user's visual representation 30, the user has assumed the correct pose instructed by the scrolling instruction 32. To further clarify where and when the user should assume the pose, a preview instruction 31 related to the scrolling instruction 32 is also displayed. The preview instruction shows the location where the user should move, which in the case of Figure 3 means to the left of the user's current position. If the user does not realize that he/she has to move, the system may display additional instructions, e.g., the arrow 39. The system makes it possible to instruct precise rhythm of arbitrary pose sequences - the correct mo- ment to match the pose is when the preview and scroll¬ ing instructions 31 and 32 align. The preview instruc¬ tions typically work so that they fade in slightly be¬ fore the user should match the corresponding scrolling instruction. The preview instructions are not kept visible all the time to reduce visual clutter. If the user is already in the correct location, the preview instruction may not be shown at all.
After aligning with the preview instruction, the scrolling instructions may stop and fade out only after the user has matched them (in case of the pose instruction 32, the instruction fades out when the user matches the pose at the correct time and in the correct location) . The system may reward the user of correct matching of poses, for example, with score bo- nuses and audiovisual effects.
Embodiments of the present invention may utilize different instruction graphics, depending on the desired movements. Examples are given in Figure 3, including textual instructions 34, forbidden areas 35, and movement targets 36. The forbidden areas work so that, for example, the user's score is decreased or the user cannot match pose instructions while touching a forbidden area. In case of the forbidden area 35, the user should jump over the area to reach and match the movement target 36. Movement targets may be speci¬ fied for a particular or any body part, and optionally so that they may only be matched by touching the tar- get from a particular direction on the screen, e.g., using an axe kick where the foot moves downwards. In case of a martial arts training system, the movement target 36 may be depicted as a brick wall that breaks when the user kicks it using a jump kick high enough to avoid the forbidden area 36.
In Figure 4, time has passed in relation to Figure 3. The instructions 41, 42, 43, 44, 45, 46, 47 correspond to instructions 31, 32, 33, 34, 35, 36, 37 in Figure 3, respectively. The preview instruction 41 and scrolling instruction 42 now align and the user 40 has moved into the correct location.
Optionally, the graphical output of a system according to the present invention may also comprise an additional representation 24, 38, 48 of the user. Figure 4 shows how the additional representation 48 may be manipulated in a time-varying manner, e.g., ro¬ tated when instructions are matched to create the il¬ lusion of the user performing flamboyant acrobatic stunts although the user is in real life moving in a less risky manner. In an alternative embodiment of the invention, the additional representation is omitted, but after finishing a movement sequence, the user is provided a replay where his/her recorded and manipu- lated movements are shown, for example a simple jump turning into an air summersault, where the computer moves and rotates the user's visual representation. This way, one can implement a game, where the player becomes a stunt man with the goal of performing cho- reographed stunts for a movie.
In an advanced embodiment of the present in¬ vention, the user performs the movements on a scroll¬ ing, non-rotating timeline. The graphical output also comprises a virtual world that revolves around the us- er, mimicking summersaults where the virtual camera is fixed to the user's torso. After performing the move¬ ments, the sequence is shown in a replay where the virtual camera is fixed to the virtual world so that the user is rotated within the virtual world.
In another embodiment of the invention, the user's visual representation 20, 30, 40, 50 may be ma- nipulated directly. Examples of the suitable manipula¬ tion techniques are time-varying rotating, scaling, translating and mirroring of the user' s visual representation. The instruction graphics may be manipulated as well, e.g., so that the timeline rotates in syn- chronization with the user's visual representation.
The term "align" can be understood more clearly from the following explanation with references to figures 3 and 4. Consider the embodiment of the present invention illustrated in Figures 3 and 4, where the player's visual representation on screen 30 is the player's real-time video image captured by an imaging device, the scrolling instructions 32, 33, 34, 35, 36, 37 comprise the freeze frame silhouettes 32,33 of a human performing the instructed choreography, the preview instructions 31 comprise the same silhouettes displayed one at a time, and both instructions are shown overlaid on top of the player's video image. In this case, the scrolling instruction 32 may be consid¬ ered to align with the similar preview instruction 31 when they are located within a predetermined distance from each other, an example of which is shown in Figure 4, where the said preview instruction is numbered 41 and the said scrolling instruction is numbered 42. The player's visual representation 40 may be consid- ered to align with the scrolling instruction silhouette 42 if the player's hands and feet (or the corre¬ sponding locations inferred by a motion tracking sys¬ tem) in the video image are within a predetermined distance from the hands and feet of the silhouette. Furthermore, the player may be considered to match the instructions and points may be awarded in the game if the player's visual representation aligns with the scrolling or preview instruction within a predetermined time interval from the moment when the scrolling and preview instructions align.
Figure 5 shows the graphical output of an ad- vanced embodiment of the present invention, where the timeline 55 is three-dimensional or isometric. Simi¬ larly as in the embodiments above, both preview in¬ structions 51 and scrolling instructions 52, 53, 54 are displayed. Forbidden areas 54 are typically also three-dimensional in this case. In embodiments of the present invention illustrated in Figures 2-4, the us¬ er' s movements are typically only instructed in a 2d plane (jumping and moving sideways on a line) . In this case, the time axis of the timeline aligns with the user's movements, making location of a graphical in¬ struction ambiguous in time and space. However, using the preview instructions solves this problem. A three- dimensional timeline such as the one in Figure 5 may either be used to instruct 2d movement more clearly or to instruct full 3d movement. The latter case requires preview instructions that are located in the 3d space so that they align with scrolling instructions at the correct location and time.
An advanced embodiment of the invention may also have multiple simultaneous timelines, so that some instructions are, e.g., scrolling from left to right and some are scrolling from top to bottom. This is particularly useful for forbidden areas. Horizontally scrolling forbidden areas can constrain the us- er's movement in vertical direction, e.g., requiring the user to jump or duct. Vertically scrolling forbid¬ den areas can constrain the movement horizontally, in¬ structing the user to step aside.
An advanced embodiment of the invention may also have a content generation system where the user records a video and then marks the video frames from which the system generates pose instructions, e.g., by using computer vision techniques to extract the out¬ line of the user's silhouette.
The preview instructions do not need to visu¬ ally similar to the scrolling instructions. For exam- pie, a preview instruction may only comprise of markers for the desired locations of the user's hands.
The scrolling and preview instructions can also be animated. For example, if the user is in¬ structed to jump up, the scrolling instruction may first show a person standing in the pose preceding the jump, then play a jump animation and stop in midair in the pose that the player should match. In this case, the preview instruction would also show the pose in midair and appear only after the animation has been played.
In an embodiment of the invention, detecting that the user has matched a pose can be implemented either using a body tracking device, or by simple computer vision methods, e.g., detecting that there is motion at the movement targets at the matching time and no motion within the forbidden area. In addition to motion, one may check whether the user' s silhouette overlaps with the target or forbidden areas. A person skilled in computer vision can typically implement various motion detection and silhouette extraction me¬ thods. Complex poses can also be detected with rea¬ sonably accuracy using target and forbidden areas that are not necessarily displayed to the user. In recent years, computer vision human tracking has also im- proved to the point where the player's full body con¬ figuration can be estimated by fitting a skeleton model to the camera view, especially if a 3d camera is used. 3d cameras measure the distance of each pixel in addition to color.
Each of the various elements and embodiments of the invention described above may be implemented in software, in hardware, or as a combination of software and hardware .
The exemplary embodiments can include, for example, any suitable servers, workstations, PCs, lap- top computers, personal digital assistants (PDAs) , In¬ ternet appliances, handheld devices, cellular tele¬ phones, smart phones, wireless devices, game consoles, other devices, and the like, capable of performing the processes of the exemplary embodiments. The devices and subsystems of the exemplary embodiments can commu¬ nicate with each other using any suitable protocol and can be implemented using one or more programmed com¬ puter systems or devices.
One or more interface mechanisms can be used with the exemplary embodiments, including, for example, Internet access, telecommunications in any suit¬ able form (e.g., voice, modem, and the like), wireless communications media, and the like. For example, em¬ ployed communications networks or links can include one or more wireless communications networks, cellular communications networks, 3G communications networks, Public Switched Telephone Network (PSTNs) , Packet Data Networks (PDNs) , the Internet, intranets, a combina¬ tion thereof, and the like.
It is to be understood that the exemplary em¬ bodiments are for exemplary purposes, as many varia¬ tions of the specific hardware used to implement the exemplary embodiments are possible, as will be appre¬ ciated by those skilled in the hardware and/or soft- ware art(s) . For example, the functionality of one or more of the components of the exemplary embodiments can be implemented via one or more hardware and/or software devices.
The exemplary embodiments can store informa- tion relating to various processes described herein. This information can be stored in one or more memo¬ ries, such as a hard disk, optical disk, magneto- optical disk, RAM, and the like. One or more databases can store the information used to implement the exem¬ plary embodiments of the present inventions. The data¬ bases can be organized using data structures (e.g., records, tables, arrays, fields, graphs, trees, lists, and the like) included in one or more memories or sto¬ rage devices listed herein. The processes described with respect to the exemplary embodiments can include appropriate data structures for storing data collected and/or generated by the processes of the devices and subsystems of the exemplary embodiments in one or more databases .
All or a portion of the exemplary embodiments can be conveniently implemented using one or more gen- eral purpose processors, microprocessors, digital sig¬ nal processors, micro-controllers, and the like, pro¬ grammed according to the teachings of the exemplary embodiments of the present inventions, as will be ap¬ preciated by those skilled in the computer and/or software art(s) . Appropriate software can be readily prepared by programmers of ordinary skill based on the teachings of the exemplary embodiments, as will be ap¬ preciated by those skilled in the software art. In ad¬ dition, the exemplary embodiments can be implemented by the preparation of application-specific integrated circuits or by interconnecting an appropriate network of conventional component circuits, as will be appre¬ ciated by those skilled in the electrical art(s) . Thus, the exemplary embodiments are not limited to any specific combination of hardware and/or software.
Stored on any one or on a combination of computer readable media, the exemplary embodiments of the present inventions can include software for control¬ ling the components of the exemplary embodiments, for driving the components of the exemplary embodiments, for enabling the components of the exemplary embodi¬ ments to interact with a human user, and the like. Such software can include, but is not limited to, de¬ vice drivers, firmware, operating systems, development tools, applications software, and the like. Such com¬ puter readable media further can include the computer program product of an embodiment of the present inven¬ tions for performing all or a portion (if processing is distributed) of the processing performed in imple¬ menting the inventions. Computer code devices of the exemplary embodiments of the present inventions can include any suitable interpretable or executable code mechanism, including but not limited to scripts, in¬ terpretable programs, dynamic link libraries (DLLs) , Java classes and applets, complete executable pro¬ grams, Common Object Request Broker Architecture (COR- BA) objects, and the like. Moreover, parts of the processing of the exemplary embodiments of the present inventions can be distributed for better performance, reliability, cost, and the like.
As stated above, the components of the exem- plary embodiments can include computer readable medium or memories for holding instructions programmed ac¬ cording to the teachings of the present inventions and for holding data structures, tables, records, and/or other data described herein. Computer readable medium can include any suitable medium that participates in providing instructions to a processor for execution. Such a medium can take many forms, including but not limited to, non-volatile media, volatile media, trans¬ mission media, and the like. Non-volatile media can include, for example, optical or magnetic disks, mag¬ neto-optical disks, and the like. Volatile media can include dynamic memories, and the like. Transmission media can include coaxial cables, copper wire, fiber optics, and the like. Transmission media also can take the form of acoustic, optical, electromagnetic waves, and the like, such as those generated during radio frequency (RF) communications, infrared (IR) data com- munications, and the like. Common forms of computer- readable media can include, for example, a floppy disk, a flexible disk, hard disk, magnetic tape, any other suitable magnetic medium, a CD-ROM, CD±R, CD±RW, DVD, DVD-RAM, DVD1RW, DVD±R, HD DVD, HD DVD-R, HD DVD- RW, HD DVD-RAM, Blu-ray Disc, any other suitable opti¬ cal medium, punch cards, paper tape, optical mark sheets, any other suitable physical medium with pat¬ terns of holes or other optically recognizable indi- cia, a RAM, a PROM, an EPROM, a FLASH-EPROM, any other suitable memory chip or cartridge, a carrier wave or any other suitable medium from which a computer can read .
It is obvious to a person skilled in the art that with the advancement of technology, the basic idea of the invention may be implemented in various ways. The invention and its embodiments are thus not limited to the examples described above; instead they may vary within the scope of the claims.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|EP1029565A2 *||15 Feb 2000||23 Aug 2000||Konami Co., Ltd.||Music staging game apparatus, music staging game method, and readable storage medium|
|US20020019258 *||15 Jun 2001||14 Feb 2002||Kim Gerard Jounghyun||Methods and apparatus of displaying and evaluating motion data in a motion game apparatus|
|US20020025842 *||17 Aug 2001||28 Feb 2002||Konami Corporation||Game machine, game processing method and information storage medium|
|US20070234885 *||7 Mar 2007||11 Oct 2007||Harmonix Music Systems, Inc.||Game controller simulating a musical instrument|
|1||*||YANG U. ET AL: "Implementation and Evaluation of "Just Follow Me": An Immersive, VR-based, Motion-Training System", PRESENCE: TELEOPERATORS AND VIRTUAL ENVIRONMENTS, vol. 11, no. 3, June 2002 (2002-06-01), MIT PRESS CAMBRIDGE, MA, USA, pages 304 - 323, Retrieved from the Internet <URL:https://svn.v2.nl/andres/Documentation/AR20Behaviour/VR_motion_training.pdf> [retrieved on 20110502]|
|Citing Patent||Filing date||Publication date||Applicant||Title|
|US8439733||16 Jun 2008||14 May 2013||Harmonix Music Systems, Inc.||Systems and methods for reinstating a player within a rhythm-action game|
|US8444464||30 Sep 2011||21 May 2013||Harmonix Music Systems, Inc.||Prompting a player of a dance game|
|US8444486||20 Oct 2009||21 May 2013||Harmonix Music Systems, Inc.||Systems and methods for indicating input actions in a rhythm-action game|
|US8449360||29 May 2009||28 May 2013||Harmonix Music Systems, Inc.||Displaying song lyrics and vocal cues|
|US8465366||29 May 2009||18 Jun 2013||Harmonix Music Systems, Inc.||Biasing a musical performance input to a part|
|US8550908||16 Mar 2011||8 Oct 2013||Harmonix Music Systems, Inc.||Simulating musical instruments|
|US8562403||10 Jun 2011||22 Oct 2013||Harmonix Music Systems, Inc.||Prompting a player of a dance game|
|US8568234||16 Mar 2011||29 Oct 2013||Harmonix Music Systems, Inc.||Simulating musical instruments|
|US8636572||16 Mar 2011||28 Jan 2014||Harmonix Music Systems, Inc.||Simulating musical instruments|
|US8663013||8 Jul 2009||4 Mar 2014||Harmonix Music Systems, Inc.||Systems and methods for simulating a rock band experience|
|US8678895||16 Jun 2008||25 Mar 2014||Harmonix Music Systems, Inc.||Systems and methods for online band matching in a rhythm action game|
|US8678896||14 Sep 2009||25 Mar 2014||Harmonix Music Systems, Inc.||Systems and methods for asynchronous band interaction in a rhythm action game|
|US8686269||31 Oct 2008||1 Apr 2014||Harmonix Music Systems, Inc.||Providing realistic interaction to a player of a music-based video game|
|US8702485||5 Nov 2010||22 Apr 2014||Harmonix Music Systems, Inc.||Dance game and tutorial|
|US8874243||16 Mar 2011||28 Oct 2014||Harmonix Music Systems, Inc.||Simulating musical instruments|
|US9024166||9 Sep 2010||5 May 2015||Harmonix Music Systems, Inc.||Preventing subtractive track separation|
|US9278286||27 Oct 2014||8 Mar 2016||Harmonix Music Systems, Inc.||Simulating musical instruments|
|US9358456||14 Mar 2013||7 Jun 2016||Harmonix Music Systems, Inc.||Dance competition game|
|International Classification||A63B69/00, G06K9/00, A63F13/00, G06T7/20, A63B24/00|
|Cooperative Classification||A63F13/833, A63F13/213, G06F3/017, G06F3/04815, G06F3/011, A63F13/44, A63B24/0006, A63F13/10, A63F2300/6607, A63F2300/1093, A63F2300/64, A63F2300/638, A63F2300/60, A63F2300/6018, G06K9/00342, A63F2300/8029|
|European Classification||A63F13/10, A63B69/00, A63B24/00H, A63B24/00A1, G06K9/00G1|
|27 Jul 2011||121||Ep: the epo has been informed by wipo that ep was designated in this application|
Ref document number: 10834280
Country of ref document: EP
Kind code of ref document: A1
|4 Jun 2012||NENP||Non-entry into the national phase in:|
Ref country code: DE
|26 Dec 2012||122||Ep: pct application non-entry in european phase|
Ref document number: 10834280
Country of ref document: EP
Kind code of ref document: A1