|Publication number||USRE45884 E1|
|Application number||US 13/028,882|
|Publication date||9 Feb 2016|
|Filing date||16 Feb 2011|
|Priority date||30 Jun 2000|
|Also published as||CN1434947A, CN101303622A, CN101303622B, EP1311966A2, EP1311966A4, EP1311966B1, EP2372496A2, EP2372496A3, US7159008, US7493365, US20070033259, WO2002003172A2, WO2002003172A3|
|Publication number||028882, 13028882, US RE45884 E1, US RE45884E1, US-E1-RE45884, USRE45884 E1, USRE45884E1|
|Inventors||Evan F. Wies, Dean C. Chang, Michael P. Ruf|
|Original Assignee||Immersion Corporation|
|Export Citation||BiBTeX, EndNote, RefMan|
|Patent Citations (133), Non-Patent Citations (86), Classifications (9), Legal Events (1)|
|External Links: USPTO, USPTO Assignment, Espacenet|
This application is a continuation of U.S. patent application Ser. No. 09/608,126 entitled “Chat Interface with Haptic Feedback Functionality” filed Jun. 30, 2000, now U.S. Pat. No. 7,159,008 the entirety of which is hereby incorporated by reference.
The present invention relates generally to interface devices for allowing humans to interface with computer systems, and more particularly to computer interface devices that provide input from the user to computer systems and implement force feedback to the user.
Using an interface device, a user can interact with an environment displayed by a computer system to perform functions and tasks on the computer, such as playing a game, experiencing a simulation or virtual reality environment, using a computer aided design system, operating a graphical user interface (GUI), or otherwise influencing events or images depicted on the screen. Common human-computer interface devices used for such interaction include a joystick, mouse, trackball, stylus, tablet, pressure-sensitive ball, or the like, that is connected to the computer system controlling the displayed environment. Typically, the computer updates the environment in response to the user's manipulation of a user-manipulatable physical object such as a joystick handle or mouse, and provides visual and audio feedback to the user utilizing the display screen and audio speakers. The computer senses the user's manipulation of the user object through sensors provided on the interface device that send locative signals to the computer.
In some interface devices, haptic feedback is also provided to the user, also known as “force feedback.” These types of interface devices can provide physical sensations which are felt by the user manipulating a user manipulable object of the interface device. For example, the Wingman Force joystick or the Wingman Force Feedback Mouse from Logitech Inc. may be connected to a computer and provides forces to a user of the controller. One or more motors or other actuators are used in the device and are connected to the controlling computer system. The computer system controls forces on the joystick in conjunction and coordinated with displayed events and interactions by sending control signals or commands to the actuators. The computer system can thus convey physical force sensations to the user in conjunction with other supplied feedback as the user is grasping or contacting the joystick or other object of the interface device. For example, when the user moves the manipulatable object and causes a displayed cursor to interact with a different displayed graphical object, the computer can issue a command that causes the actuator to output a force on the user object, conveying a feel sensation to the user.
Force feedback can be used to communicate ideas and messages as well as effects. Forces can in many instances provide additional information to a recipient of the message that may not be apparent in a text or voice message. For example, a text message sent to another user over the Internet may not include information indicating how strong the user feels about the topic expressed or other message subtext. Users can try to express this subtext using well-known icons or symbols known as “emoticons,” which are iconic representations of emotions or messages, such as the “smiley” to indicate a humorous message, expressed as a colon and right parenthesis mark, :), which resembles a face smiling when viewed from the side. Variations of the smiley emoticon can express a variety of other emotions. However, such emoticons and symbols are limited in the complexity of the messages they convey and the range of different messages possible. Haptic feedback, in contrast, can offer much more complex and direct ways to express such subtext to other users in a more compelling fashion.
The present invention is directed to an interface allowing a user to exchange haptic chat messages with other users over a computer network. The user is able to provide messages that cause haptic sensations to one or more remote users which have a haptic interface device, allowing more diverse and compelling messages to be sent in a chat environment.
More particularly, one method of the present invention provides a chat interface displayed by a local computer, the chat interface capable of providing haptic messages to other users across a network. The chat interface is displayed on a display device of the local computer, and input data from a user of the local computer is received at the chat interface, the input data providing an outgoing chat message which can include sent force information. The outgoing chat message to be sent to a remote computer that is connected to the local host computer via a computer network, and the remote computer can display a chat interface and output a haptic sensation to a user of the remote computer based at least in part on the force information. An incoming message from the remote computer is received at the chat interface, which may also include received force information. The incoming chat message is displayed on a display device to the user of the local computer. A haptic sensation can be output to the user of the local computer using a haptic device coupled to the local computer, where the haptic sensation is based at least in part on the received force information received from the remote computer.
The local computer and remote computer can each be coupled to a server machine via the network, such as an IRC server, or can be coupled to each other via a peer-to-peer connection. The chat interface preferably includes multiple available haptic effects, each selectable by the user to be sent as the force information in the chat message. The chat interface also may allow the user to create a custom haptic sensation to be referenced by the force information sent to the remote computer. The force information is also preferably associated with sound information, such that the remote computer outputs a sound effect in coordination with the output of the haptic sensation. In one embodiment, the received force (or other) information can be processed by a background application running on the local computer simultaneously with the chat interface, the background application controlling the output of the haptic sensation to the user.
The sent force information may include a network address, which is then used by the chat interface on the remote computer as a network location at which to retrieve additional force information required to output a force sensation to the haptic device at the remote computer. For example, the network address can be an address of a web server storing a library of standard and customized haptic sensations which can be output by the haptic device. In addition, custom force information can be uploaded from a client machine to a server at the network address, where the uploaded custom force information can be downloaded by a different client computer to output a haptic sensation based on the custom force information. The force information in the chat message can alternatively includes data characterizing the desired haptic sensation. Preferably, the chat interface allows the user of the local computer to type a text command including text characters to be sent as a force command or cause force information to be sent to the remote computer. The text force command is preferably displayed in a chat interface of the remote computer and includes at least one delimiter character for indicating the nature of the text force command or can be a predetermined character(s), such as those used for emoticons. Chat messages can also be in audio or other formats, and one embodiment allows audio waveforms in chat messages to be analyzed to base haptic sensations on waveform content.
The present invention advantageously provides features in a chat interface on a computer to allow enhancements to chat messages using haptic sensations. The user can select a desired haptic sensation or even customize a haptic sensation to provide with a message to one or more other users in a chat session. The haptic sensations allow a wide variety of emotions and other content and subtext of messages to be conveyed, allowing a user more freedom to express a desired message across a computer network.
These and other advantages of the present invention will become apparent to those skilled in the art upon a reading of the following specification of the invention and a study of the several figures of the drawing.
Host computer system 12 is preferably a personal computer, such as an IBM-compatible or Macintosh personal computer, or a workstation, such as a SUN or Silicon Graphics workstation. Alternatively, host computer system 12 can be one of a variety of home video game systems, such as systems available from Nintendo, Sega, or Sony, a television “set top box” or a “network computer”, etc. Host computer system 12 preferably implements a host application program with which a user 22 is interacting via peripherals and interface device 14. For example, the host application program can be a video game, medical simulation, scientific analysis program, operating system, graphical user interface, or other application program that utilizes force feedback. Typically, the host application provides images to be displayed on a display output device, as described below, and/or other feedback, such as auditory signals.
Host computer system 12 preferably includes a host microprocessor 16, random access memory (RAM) 17, read-only memory (ROM) 19, input/output (I/O) electronics 21, a clock 18, a display screen 20, and an audio output device 21. Display screen 20 can be used to display images generated by host computer system 12 or other computer systems, and can be a standard display screen, CRT, flat-panel display, 3-D goggles, or any other visual interface. Audio output device 21, such as speakers, is preferably coupled to host microprocessor 16 via amplifiers, filters, and other circuitry well known to those skilled in the art (e.g. in a sound card) and provides sound output to user 22 from the host computer 18. Other types of peripherals can also be coupled to host processor 16, such as storage devices (hard disk drive, CD ROM/DVD-ROM drive, floppy disk drive, etc.), printers, and other input and output devices. Data for implementing the interfaces of the present invention can be stored on computer readable media such as memory (RAM or ROM), a hard disk, a CD-ROM or DVD-ROM, etc.
An interface device 14 is coupled to host computer system 12 by a bi-directional bus 24. The bi-directional bus sends signals in either direction between host computer system 12 and the interface device. An interface port of host computer system 12, such as an RS232 or Universal Serial Bus (USB) serial interface port, parallel port, game port, etc., connects bus 24 to host computer system 12.
Interface device 14 includes a local microprocessor 26, local memory 27, sensors 28, actuators 30, a user object 34, optional sensor interface 36, an optional actuator interface 38, and other optional input devices 39. Local microprocessor 26 is coupled to bus 24 and is considered local to interface device 14 and is dedicated to force feedback and sensor I/O of interface device 14. Microprocessor 26 can be provided with software instructions to wait for commands or requests from computer host 12, decode the command or request, and handle/control input and output signals according to the command or request. In addition, processor 26 preferably operates independently of host computer 16 by reading sensor signals and calculating appropriate forces from those sensor signals, time signals, and stored or relayed instructions selected in accordance with a host command. Suitable microprocessors for use as local microprocessor 26 include the MC68HC711E9 by Motorola, the PIC16C74 by Microchip, and the 82930AX by Intel Corp., for example. Microprocessor 26 can include one microprocessor chip, or multiple processors and/or co-processor chips, and/or digital signal processor (DSP) capability.
Microprocessor 26 can receive signals from sensors 28 and provide signals to actuators 30 of the interface device 14 in accordance with instructions provided by host computer 12 over bus 24. For example, in a preferred local control embodiment, host computer system 12 provides high level supervisory commands to microprocessor 26 over bus 24, and microprocessor 26 manages low level force control loops to sensors and actuators in accordance with the high level commands and independently of the host computer 18. The force feedback system thus provides a host control loop of information and a local control loop of information in a distributed control system. This operation is described in greater detail in U.S. Pat. No. 5,739,811 and patent application Ser. Nos. 08/877,114 and 08/050,665 (which is a continuation of U.S. Pat. No. 5,734,373), all incorporated by reference herein. Microprocessor 26 can also receive commands from any other input devices 39 included on interface apparatus 14, such as buttons, and provides appropriate signals to host computer 12 to indicate that the input information has been received and any information included in the input information. Local memory 27, such as RAM and/or ROM, is preferably coupled to microprocessor 26 in interface device 14 to store instructions for microprocessor 26 and store temporary and other data. In addition, a local clock 29 can be coupled to the microprocessor 26 to provide timing data.
Sensors 28 sense the position, motion, and/or other characteristics of a user object 34 of the interface device 14 along one or more degrees of freedom and provide signals to microprocessor 26 including information representative of those characteristics. Rotary or linear optical encoders, potentiometers, optical sensors, velocity sensors, acceleration sensors, strain gauge, or other types of sensors can be used. Sensors 28 provide an electrical signal to an optional sensor interface 36, which can be used to convert sensor signals to signals that can be interpreted by the microprocessor 26 and/or host computer system 12.
Actuators 30 transmit forces to user object 34 of the interface device 14 in one or more directions along one or more degrees of freedom, and/or may apply forces to the housing of the device 14, in response to signals received from microprocessor 26. Actuators 30 can include two types: active actuators and passive actuators. Active actuators include linear current control motors, stepper motors, pneumatic/hydraulic active actuators, a torquer (motor with limited angular range), a voice coil actuators, and other types of actuators that transmit a force to move an object. Passive actuators can also be used for actuators 30, such as magnetic particle brakes, friction brakes, or pneumatic/hydraulic passive actuators. Actuator interface 38 can be optionally connected between actuators 30 and microprocessor 26 to convert signals from microprocessor 26 into signals appropriate to drive actuators 30.
Other input devices 39 can optionally be included in interface device 14 and send input signals to microprocessor 26 or to host processor 16. Such input devices can include buttons, dials, switches, levers, or other mechanisms. For example, in embodiments where user object 34 is a joystick, other input devices can include one or more buttons provided, for example, on the joystick handle or base. Power supply 40 can optionally be coupled to actuator interface 38 and/or actuators 30 to provide electrical power. A safety switch 41 is optionally included in interface device 14 to provide a mechanism to deactivate actuators 30 for safety reasons.
User manipulable object 34 (“user object”) is a physical object, device or article that may be grasped or otherwise contacted or controlled by a user and which is coupled to interface device 14. By “grasp”, it is meant that users may physically contact the object in some fashion, such as by hand, with their fingertips, or even orally in the case of handicapped persons. The user 22 can manipulate or move the object to interface with the host application program the user is viewing on display screen 20. Object 34 can be a joystick, mouse, trackball, keyboard, stylus (e.g. at the end of a linkage), steering wheel, sphere, medical instrument (laparoscope, catheter, etc.), pool cue (e.g. moving the cue through actuated rollers), hand grip, knob, button, or other article.
The haptic feedback interface device 14 can take a variety of forms, including a mouse, joystick, gamepad, steering wheel, chair pads which the user sits on, fishing rod, pool cue, etc. A variety of these types of devices are available commercially. For example, suitable mice for use with the present invention include kinesthetic force and vibrotactile mice, such as those described in copending U.S. application Ser. Nos. 08/965,720, 09/125,711, 09/456,887, and 60/182,868, all incorporated herein by reference. The user object 34 and/or interface device 14 can also be a keyboard having haptic feedback functionality; some embodiments of such a keyboard are described in copending application no. 09/450,361, filed on May 12, 2000, and entitled, “Haptic Feedback Using a Keyboard Device,” which is incorporated herein by reference. One preferred embodiment of a system for use with the present invention is a haptic keyboard and a haptic mouse, where the user may type messages on the haptic keyboard in the chat interface of the present invention, and may use the haptic mouse to move a cursor to select functions within the chat interface. Both devices can output the haptic feedback communicated in the present invention. Thus, when using both devices, the user can experience the haptic feedback at all times, whether the user has one hand on the keyboard and one hand on the mouse, both hands on the keyboard or one hand on the mouse, or no hands on the keyboard and one hand on the mouse.
In the described implementation, clients 72a and 72b each include a chat client functional portion 76 and a haptic control functional portion 78. The chat client 76 interacts with the chat server 70 according to standard protocols to provide chat communications to the user of the client machine from other client machines. For example, server 70 can be an IRC (Internet Relay Chat) server which communicates with clients using well-known IRC protocols. Other types chat protocols that can be used include pure HTML-based chat protocols, Java-based chat protocols, or protocols based in other standards. Some clients connected to server 70 and participating in chat sessions, such as client 73, may only be implementing the chat client portion 76 and thus ignore the haptic implementation of haptic messages.
The haptic control portion 78 interacts with the chat client portion 76 to provide control over haptic sensations of the present invention that are associated with chat messages. For example, received messages which are interpreted as haptic messages or commands can be detected by the haptic control portion 76 and haptic sensations can be commanded from portion 78 to the haptic device 74. Furthermore, in some embodiments, the haptic control portion 78 can communicate with one or more other servers, such as web server 80. For example, force effect data, sound data, or other data can be retrieved by the haptic control portion 78 to implement particular haptic sensations. This operation is described in greater detail below.
Display screen 20 is shown displaying a chat interface 100 of the present invention. The interface 100 can implement and connect to a server running a standard chat program and protocol, such as Internet Relay Chat (IRC), using methods well known to those skilled in the art. IRC simply provides text characters from one client to the chat server 70, which routes the text characters to the other clients participating in the chat session. In some embodiments, the chat protocol used can be a proprietary one that only functions with particular programs.
In the described embodiment, a chat client program handles all the standard chat interfacing, while a haptic functionality program interfaces with the chat program to handle the output of haptic sensations. For example, the chat interface 100 can be displayed within a web browser program, such as Microsoft Internet Explorer or Netscape Navigator, as a web page. For example, in one implementation, the Internet Explorer web browser can make use of the MS Chat ActiveX Control available from Microsoft Corp., which can perform all the functions necessary to communicate with an IRC server and the necessary network protocols; this can be the chat client portion 76 as shown in
Alternatively, the chat interface 100 can be implemented as a separate application program, as a functional part of another program or operating system, a Java applet or other program implemented over the World Wide Web or Internet, or other implementation. Similar embodiments used for force feedback in web pages sent over the World Wide Web are described in copending application Ser. No. 09/244,622, incorporated herein by reference. For example, in one embodiment the haptic chat interface can be a portion of an “instant messaging” program such as ICQ or AOL Instant Messenger, available from America Online, Inc., which allows users to chat using text, send files to each other, connect to each other using a game program, etc. In a different embodiment, a background application, which is always running on the client computer, checks all input being sent and/or received and determines if any of the input qualifies as a haptic message, e.g. if any input has the brackets surround text characters as described below. Thus, a standard chat interface program can be used for providing chat functionality, with the background application enabling the haptic sensations for haptic messages. If such a background application is used, particular application programs, such as chat programs and instant messaging programs, can be designated by the user to be monitored by the background application, so that the background application ignores input and output of other programs not relevant to haptic messaging. A background application used for force feedback functionality in a graphical user interface, which can also be applied to the chat interface of the present invention, is described in copending U.S. patent application Ser. No. 08/970,953, incorporated herein by reference.
In still other embodiments, the interface 100 can portray a 2-D or 3-D graphical environment through which the user can navigate and manipulate. For example, in a virtual or game environment implemented over the Internet or other network (e.g., using VRML protocols), a message can be sent to another 3-D character or avatar, and a haptic component to the message can be provided for the recipient.
Interface 100 includes information fields 102, chat window 104, a user list window 106, and effects window 108. Information fields 102 allow the user to specify connection and naming options. A server name field 112 allows the user to specify a particular server to which to connect. For example, the described implementation allows the user to connect to IRC servers. Alternatively, this can be a client name to connect directly to another client machine, if such functionality is provided in the chat interface. Nickname field 114 allows the user to specify a name that is used within a chat. Room name field 116 allows the user to specify a “chat room” or particular area on the specified server in which the chat is held which allows only the users designated to be in that chat room to communicate with each other in the chat session. The “leave” button 118, when selected by the user (e.g. with keyboard or mouse cursor), causes the user to leave any chat session in which the user is currently participating. Attributes 120 allow the user to designate whether the user will feel the haptic sensations associated with an effect from window 108 when such an effect is received from another user, and/or hear the sound effect associated with that haptic message when it is received (or, in some embodiments, when the haptic or sound effect is sent by the user, to allow the user to haptically and auditorily experience a message in the way that the recipient of the message will experience it). Icon attributes 122 allow the user to designate whether the user will feel the haptic sensations and/or the sound effects associated with “icons” (emoticons) received from a different user, i.e. messages having haptic and auditory content when used in the present invention, each message represented by a single command or icon. For example, a smiley icon (“:)”) can, when received, cause a predefined force sensation and sound to be output to the user if attributes 122 are selected. The user can therefore select whether he or she wants to experience the haptic and/or auditory content of iconic messages received in the chat interface 100.
Chat window 104 displays the text messages typed in (or otherwise input) by the user as well as messages sent from other users that are currently connected to the user's computer in a chat session. The user can type in a text message in the text entry field 126, can send the message to all the users in the chat session by selecting button 128, or can “whisper” the message only to users selected in window 106 by selecting the button 130. In other implementations, each of the users in the chat session may display messages in a separate window or other area of the interface 100 dedicated to that user. User list window 106 displays all the users currently in the chat room or session in which the user is participating or observing (e.g., in a chat room which the user wishes to observe without sending messages of his or her own). The users participating in the chat room are able to type messages to the other users in the chat room, where those messages are displayed in the chat window 104. In some embodiments, a user is able to select one or more names of users displayed in window 106 to call up information about those users and/or send messages directly to those users. In the preferred embodiment, the user can select one or more names and send tactile messages to the selected users.
Effects list 108 provides a number of force effects that can be sent as haptic messages to a particular users selected in the user list window 106 (or to all users if no users in list 106 are selected). Each of the effects listed in list 108 also has one or more sounds associated with it which are played by the recipient's client machine when the haptic message is sent. Effects list 108 preferably includes all the haptic messages which the user can send, including any custom or newly-downloaded messages. The messages are only sent to the selected user(s), so that other unselected users are not sent the messages. Each name in list 108 represents a particular haptic sensation that has been associated with that name or label in list 108. To send a haptic message, the user can select one of the effects in list 108 with a displayed cursor or by using some other selection method (keyboard, etc.). This preferably calls up a menu to allow the user to perform different functions with the selected effect, as detailed below with respect to
The interface 100 of
The haptic control portion of the recipient user's client detects the received command and outputs a force sensation with the same name to the haptic device. Thus, for example, when the effect “pat” is selected (or the command <pat> is entered), an associated haptic message is sent to other chat user(s). The recipient user(s) then feel the haptic message via a haptic feedback interface device that the recipient user is using. The haptic message is delivered to the recipient user as a haptic sensation output by the recipient user's haptic interface device, e.g. a pulse, vibration, jolt, etc. or combination of multiple haptic sensations. Each of the effects in list 108 preferably has a name or label that is appropriate to the haptic sensation associated with that name. For example, the “pat” effect preferably provides a haptic message implementing a small, smooth jolt to the grip or user object of the recipient user's haptic device, like a pat of a hand. The “giggle” effect can provide a low-frequency vibration, the “slap” effect can provide a sharp, high magnitude jolt, the “smile” effect can provide a slow side-to-side motion, etc.
Also, a predefined sound effect is also preferably associated with the sent message to more effectively convey the message, although such sound need not be played in alternate embodiments. The sound effect is synchronized with features of the haptic sensation of the haptic message. For example, the message “slap” can provide a single haptic jolt and sound effect, while the message “slap-slap” can provide two successive jolts, each jolt synchronized with appropriate slapping sound effects. A sound file (which can be in a standardized format such as .wav) can be associated with the haptic command on the recipient client machine, and this sound file is played concurrently with the output of the force sensation. In other embodiments, other types of media data can be output synchronized with the haptic effect instead of or in addition to the sound effect. For example, animated or static graphics or images can be displayed on a display screen in coordination with the output of the haptic effect and with the sound effect. These other types of data can be stored in files and accessed similar to the sound files described above.
A chat user can also preferably send a haptic message by typing a text command in the chat window directly with a keyboard (or inputting the command with another input device) rather than selecting an effect from the list 108. For example, the user could simply type “<slapslap>” to cause the appropriate haptic message to be sent. Furthermore, predefined “emoticons” can be defined to be associated with haptic and sound effects and can be sent as haptic commands. For example, a smiley emoticon, “:)”, when typed into entry field 126 and sent to another user, can cause the same haptic sensation as a “smile” effect selected from effects list 108, or can cause a unique associated haptic sensation to be output to the recipient. Other examples of emoticons include “: (” (frown), “;)” (wink), and “: o” (surprise). In some embodiments, to be used as haptic messages, such emoticons are placed between brackets to indicate that they are haptic commands. Other embodiments can automatically interpret such emoticons as commands, without brackets or other command characters or delimiters. Emoticons can preferably be predefined by a user in a separate list, where each emoticon can be associated with haptic effects and sound effects similarly to the custom effects described below.
In the described embodiment, the command sent to the recipient clients as a haptic message is also displayed in the recipients' chat window 104 (and the sender's window 104, if desired) as the text characters in the message. Thus, a “smile” haptic message 124 is displayed in
In some embodiments, the user can send a normal text message as well as a haptic effect and sound effect, all simultaneously. For example, the user can type in a message in field 126 and can select an option (not shown) in menu 140 such as “send with text message”, which will cause the selected haptic effect (and associated sound effect) to be simultaneously sent with the text message in field 126 to recipient users. Thus, the term “haptic message” as referenced herein can include a haptic effect as well as a sound effect, a text message, and/or other content.
The whisper command 144 in menu 140, when selected, causes the selected haptic effect(s) from list 108 to be sent only to those users selected in window 106, but is otherwise similar to the send command 142. The play command 146 allows the selected haptic effect(s), and their associated sounds, to be output on the user's own client machine so that the user can check how the haptic message will be experienced by recipient users.
The actual haptic message contents that are sent to the recipient client machine(s) can vary in different embodiments. In the described embodiment, the available haptic messages from list 108 are identical for all users in the chat, who are all using the same interface 100. Thus, the only information that needs to be sent to other users in a haptic message is the high level command indicating the type of haptic sensation is being sent as a message, such as the text label of the effect surround by brackets explained above, or some other type of command. This allows the chat interface of the recipient client machine to receive the command as standard text characters or other standardized data, and allows the haptic control portion of the recipient client machine to know which haptic sensation should be output. It should be noted that in some embodiments, a haptic message can be sent without the sending client knowing that it is a haptic message. For example, as explained above, an emoticon without any other special characters can be sent by a non-force feedback standard client in a chat session as a text message, and the emoticon can be considered a haptic message by the receiving client so that a haptic sensation associated with the emoticon is output upon reception. Such an implementation can be considered a “generic” haptic effect that is implemented only at the receiving client.
In other embodiments, more sophisticated haptic messages can be sent or indicated. For example, the haptic message can include force information content and/or additional commands that are sent to the recipient user's client machine and instruct the haptic device of the recipient user to output a haptic sensation. This can allow for customized haptic sensations to be output that are not previously defined within the interface 100. The force information can be provided in several different ways. For example, the force information can be sent as a high level command that indicates a standardized type of haptic sensation to output, where it is assumed that the recipient users all have a standardized library of haptic sensations available on their computer systems which the high level command can reference. In some embodiments, additional information can be sent, such as one or more command parameters that characterize the commanded haptic sensation, e.g., time duration of the sensation, frequency, magnitude, direction, button parameters, rise time and decay time, simulated mass or position, etc. In yet other embodiments, data describing and defining the actual haptic sensation can be sent over, such as a series of force magnitudes and directions. Or, a network address (or other location) can be sent at which the haptic sensation data can be downloaded or retrieved. Many of these methods can allow completely customized haptic sensations to be sent which recipient users do not already have. For example, the first time a customized haptic message is sent, all the data required to implement the haptic sensation is also sent. Any description data need not be sent thereafter when that haptic message is sent successive later times, since the haptic description data is already resident and saved on the recipient computer system. Such sending of description data is obviously more suitable for communication over faster, broadband networks and connections due to the larger amount of data sent.
Custom haptic effects and sound effects can also be composed by users. For example, if the user selects the Compose New Effect command 148 from menu 140, the user preferably accesses a user interface to allow effect creation. One example of such a user interface is shown in
In some embodiments, libraries of standardized as well as customized haptic effects and sound effects can be stored on network servers available on a network having wide distribution, such as the Internet, which recipient users can access to download the needed data to experience received haptic messages. In the described embodiment, web server 80 (shown in
The chat interface 100 (or other separate program or web page) can include features to allow users to connect to a server that lists customized haptic sensations available from multiple other users that have chosen to make their sensations available, and which allows users to connect to various other user client machines or servers to download selected haptic sensation files and data. For example, if the user selects the Import Effect command 150 of the menu 148, the user can preferably import any of several available effects. An example of an interface to allow such selection in shown in
The effects can be categorized on the server according to a number of different criteria. For example, groups of haptic messages can be displayed and organized according to types of messages, emotion of the message, strength of the message, etc. For example, a “sad messages” category can include all those haptic messages conveying such an emotional state, and a “romance” category can include haptic messages conveying an appropriate close, personal message.
In other embodiments, the chat interface can be part of a voice communication program allowing voice communication or telephony over the computer network 60. Voice communication features can be found in existing utility programs or in API's such as DirectX. For example, when saying something to a recipient user, a sending user can select a message effect similar to an effect in list 108 to provide a haptic sensation to the recipient user in conjunction with the spoken message, or independent of any speech. Haptic messages can also be selected to be sent to a recipient user based on spoken message occurrences or content; for example, a haptic message can be sent each time a word is spoken. If voice recognition is implemented on the client machine (e.g., using standard available voice recognition software), a haptic message can be based on a software interpretation of the actual spoken message content. Thus, if the user says, “I hate you” to another user, a “slap” or “punch” message can automatically be sent with or right after the voice message to provide the appropriate haptic sensation.
A haptic message can be “generic,” i.e. the haptic output can be generated on the receiving machine based on user preferences, where the user can associate desired haptic sensations with particular commands that are received in an appropriate interface. Alternatively, a haptic message can be “authored”, where the sending user defines how the haptic sensation is to be felt within the message, by pointing to standardized haptic sensations or providing the data necessary to implement the authored haptic sensation. Such generic and authored implementations are similar to generic and authored force effects sent over networks as described in copending U.S. application Ser. No. 09/244,622.
In another embodiment, haptic sensations can be based on audio speech that is input and transmitted as chat messages to be output by the other client machines in the chat session. In one such embodiment, a process can be running on the receiving client computer which analyzes incoming audio speech data and commands haptic sensations based on the speech data. For example, in a simple embodiment, a wave pattern representing the speech data can be converted to haptic sensations based on the shape of the waveform, where each (or a selected) peak in the waveform can cause the output of a pulse or jolt on the haptic device, and repeating peaks in the waveform can cause a vibration. Other features of waveforms can be designated for other haptic effects, e.g., jumps in amplitude by a predetermined amount can cause a jolt or pulse, or the jolt's magnitude can be proportional to the jump in amplitude of the waveform.
In a more complex embodiment, the speech waveforms of the received message can be analyzed for predetermined, particular characteristics which may indicate the emotional content of the audio chat message, and a haptic sensation appropriate to the emotional content can then be output to the user. For example, a laugh of the sending user might provide distinguishing characteristics in the sent waveform, such as a high frequency, high amplitude oscillation. If such a laugh is detected, a haptic sensation such as an oscillation or wiggling of the mouse (or other user manipulandum) can be output. A shout from the sending user might appear in the waveform as a quick transition from low amplitude to high amplitude, and the associated haptic sensation can be a quick, high frequency vibration or pulse. A sigh from the sending user may cause a long, low frequency, low volume, consistent pitch or amplitude in the waveform, which can be associated with gentle, circular motions of the manipulandum or low frequency vibrations on the haptic device. Other emotions or inherent messages can be similarly analyzed in received waveforms and appropriate haptic sensations output based on the analysis.
While this invention has been described in terms of several preferred embodiments, it is contemplated that alterations, permutations and equivalents thereof will become apparent to those skilled in the art upon a reading of the specification and study of the drawings. For example, many different application programs can use the messaging functions of the present invention, including game programs, virtual reality programs and environments, teleconferencing applications for business meetings, telephone-type voice communications over computer networks or other communication channels, etc. Furthermore, certain terminology has been used for the purposes of descriptive clarity, and not to limit the present invention. It is therefore intended that the following appended claims include all such alterations, permutations, and equivalents as fall within the true spirit and scope of the present invention.
|Cited Patent||Filing date||Publication date||Applicant||Title|
|US3157853||6 Dec 1957||17 Nov 1964||Joseph Hirsch||Tactile communication system|
|US3220121||20 May 1963||30 Nov 1965||Communications Patents Ltd||Ground-based flight training or simulating apparatus|
|US3497668||25 Aug 1966||24 Feb 1970||Joseph Hirsch||Tactile control system|
|US3517446||19 Apr 1967||30 Jun 1970||Singer General Precision||Vehicle trainer controls and control loading|
|US3902687||25 Jun 1973||2 Sep 1975||Robert E Hightower||Aircraft indicator system|
|US3903614||27 Mar 1970||9 Sep 1975||Singer Co||Apparatus for simulating aircraft control loading|
|US4160508||19 Aug 1977||10 Jul 1979||Nasa||Controller arm for a remotely related slave arm|
|US4236325||26 Dec 1978||2 Dec 1980||The Singer Company||Simulator control loading inertia compensator|
|US4513235||24 Jan 1983||23 Apr 1985||British Aerospace Public Limited Company||Control apparatus|
|US4581491||4 May 1984||8 Apr 1986||Research Corporation||Wearable tactile sensory aid providing information on voice pitch and intonation patterns|
|US4599070||29 Jul 1981||8 Jul 1986||Control Interface Company Limited||Aircraft simulator and simulated control system therefor|
|US4708656||4 Feb 1986||24 Nov 1987||Fokker B.V.||Simulator of mechanical properties of a steering system|
|US4713007||11 Oct 1985||15 Dec 1987||Alban Eugene P||Aircraft controls simulator|
|US4891764||11 Dec 1987||2 Jan 1990||Tensor Development Inc.||Program controlled force measurement and control system|
|US4930770||1 Dec 1988||5 Jun 1990||Baker Norman A||Eccentrically loaded computerized positive/negative exercise machine|
|US4934694||9 Mar 1988||19 Jun 1990||Mcintosh James L||Computer controlled exercise system|
|US4938483||4 Nov 1987||3 Jul 1990||M. H. Segan & Company, Inc.||Multi-vehicle interactive toy system|
|US5019761||21 Feb 1989||28 May 1991||Kraft Brett W||Force feedback control for backhoe|
|US5022407||24 Jan 1990||11 Jun 1991||Topical Testing, Inc.||Apparatus for automated tactile testing|
|US5035242||16 Apr 1990||30 Jul 1991||David Franklin||Method and apparatus for sound responsive tactile stimulation of deaf individuals|
|US5038089||28 Oct 1988||6 Aug 1991||The United States Of America As Represented By The Administrator Of The National Aeronautics And Space Administration||Synchronized computational architecture for generalized bilateral control of robot arms|
|US5078152||25 Dec 1988||7 Jan 1992||Loredan Biomedical, Inc.||Method for diagnosis and/or training of proprioceptor feedback capabilities in a muscle and joint system of a human patient|
|US5186695||26 Oct 1990||16 Feb 1993||Loredan Biomedical, Inc.||Apparatus for controlled exercise and diagnosis of human performance|
|US5208745||4 Jun 1990||4 May 1993||Electric Power Research Institute||Multimedia interface and method for computer system|
|US5212473||21 Feb 1991||18 May 1993||Typeright Keyboard Corp.||Membrane keyboard and method of using same|
|US5240417||14 Mar 1991||31 Aug 1993||Atari Games Corporation||System and method for bicycle riding simulation|
|US5271290||14 Apr 1993||21 Dec 1993||United Kingdom Atomic Energy Authority||Actuator assembly|
|US5275174||16 Jul 1992||4 Jan 1994||Cook Jonathan A||Repetitive strain injury assessment|
|US5299810||23 Jun 1992||5 Apr 1994||Atari Games Corporation||Vehicle simulator including cross-network feedback|
|US5309140||26 Nov 1991||3 May 1994||The United States Of America As Represented By The Secretary Of The Navy||Feedback system for remotely operated vehicles|
|US5334027||25 Feb 1991||2 Aug 1994||Terry Wherlock||Big game fish training and exercise device and method|
|US5466213||6 Jan 1994||14 Nov 1995||Massachusetts Institute Of Technology||Interactive robotic therapist|
|US5547382||10 Apr 1995||20 Aug 1996||Honda Giken Kogyo Kabushiki Kaisha||Riding simulation system for motorcycles|
|US5555894||2 May 1994||17 Sep 1996||Matsushita Electric Industrial Co., Ltd.||Force sensation exhibiting device, data input device and data input equipment|
|US5629594||16 Oct 1995||13 May 1997||Cybernet Systems Corporation||Force feedback system|
|US5690582||1 Jun 1995||25 Nov 1997||Tectrix Fitness Equipment, Inc.||Interactive exercise apparatus|
|US5695400||30 Jan 1996||9 Dec 1997||Boxer Jam Productions||Method of managing multi-player game playing over a network|
|US5719561||25 Oct 1995||17 Feb 1998||Gilbert R. Gonzales||Tactile communication device and method|
|US5721763||31 Jan 1996||24 Feb 1998||At&T Corp||Method for providing information during a chat session|
|US5734373||1 Dec 1995||31 Mar 1998||Immersion Human Interface Corporation||Method and apparatus for controlling force feedback interface systems utilizing a host computer|
|US5764916||27 Sep 1996||9 Jun 1998||Ichat, Inc.||Method and apparatus for real time communication over a computer network|
|US5766016||14 Nov 1994||16 Jun 1998||Georgia Tech Research Corporation||Surgical simulator and method for simulating surgical procedure|
|US5785630||6 Nov 1996||28 Jul 1998||Tectrix Fitness Equipment, Inc.||Interactive exercise apparatus|
|US5818836||9 Aug 1995||6 Oct 1998||Duval; Stephen C.||Method and apparatus for anonymous voice communication using an online data service|
|US5844392||21 May 1997||1 Dec 1998||Cybernet Systems Corporation||Haptic browsing|
|US5880731||14 Dec 1995||9 Mar 1999||Microsoft Corporation||Use of avatars with automatic gesturing and bounded interaction in on-line chat session|
|US5884029||14 Nov 1996||16 Mar 1999||International Business Machines Corporation||User interaction with intelligent virtual objects, avatars, which interact with other avatars controlled by different users|
|US5889670||11 Jan 1996||30 Mar 1999||Immersion Corporation||Method and apparatus for tactilely responsive user interface|
|US5907677||23 Aug 1996||25 May 1999||Ecall Inc.||Method for establishing anonymous communication links|
|US5933599||17 Jul 1995||3 Aug 1999||Microsoft Corporation||Apparatus for presenting the content of an interactive on-line network|
|US5943648||25 Apr 1996||24 Aug 1999||Lernout & Hauspie Speech Products N.V.||Speech signal distribution system providing supplemental parameter associated data|
|US5956484||1 Aug 1996||21 Sep 1999||Immersion Corporation||Method and apparatus for providing force feedback over a computer network|
|US5963217||18 Nov 1996||5 Oct 1999||7Thstreet.Com, Inc.||Network conference system using limited bandwidth to generate locally animated displays|
|US5973678||29 Aug 1997||26 Oct 1999||Ford Global Technologies, Inc.||Method and system for manipulating a three-dimensional object utilizing a force feedback interface|
|US5978467||11 Apr 1997||2 Nov 1999||Walker Asset Management Limited Partnership||Method and apparatus for enabling interaction between callers with calls positioned in a queue|
|US5983190||19 May 1997||9 Nov 1999||Microsoft Corporation||Client server animation system for managing interactive user interface characters|
|US5987503||31 Dec 1996||16 Nov 1999||Fujitsu Limited||System and method for displaying an electronic mail containing a keyword detected in a chat session message|
|US6028593||14 Jun 1996||22 Feb 2000||Immersion Corporation||Method and apparatus for providing simulated physical interactions within computer generated environments|
|US6046726||29 Sep 1997||4 Apr 2000||U.S. Philips Corporation||Virtual workspace with user-programmable tactile feedback|
|US6047356||18 Apr 1994||4 Apr 2000||Sonic Solutions||Method of dynamically allocating network node memory's partitions for caching distributed files|
|US6076100||17 Nov 1997||13 Jun 2000||Microsoft Corporation||Server-side chat monitor|
|US6101530||16 Sep 1998||8 Aug 2000||Immersion Corporation||Force feedback provided over a computer network|
|US6108704||25 Sep 1995||22 Aug 2000||Netspeak Corporation||Point-to-point internet protocol|
|US6111577||4 Apr 1996||29 Aug 2000||Massachusetts Institute Of Technology||Method and apparatus for determining forces to be applied to a user through a haptic interface|
|US6154773||15 Apr 1998||28 Nov 2000||Cddb, Inc.||Network delivery of interactive entertainment complementing audio recordings|
|US6161126||2 Feb 1999||12 Dec 2000||Immersion Corporation||Implementing force feedback over the World Wide Web and other computer networks|
|US6166732||24 Feb 1998||26 Dec 2000||Microsoft Corporation||Distributed object oriented multi-user domain with multimedia presentations|
|US6169540||17 Jun 1997||2 Jan 2001||Immersion Corporation||Method and apparatus for designing force sensations in force feedback applications|
|US6175954||30 Oct 1997||16 Jan 2001||Fuji Xerox Co., Ltd.||Computer programming using tangible user interface where physical icons (phicons) indicate: beginning and end of statements and program constructs; statements generated with re-programmable phicons and stored|
|US6219034||23 Feb 1998||17 Apr 2001||Kristofer E. Elbing||Tactile computer interface|
|US6219045||12 Nov 1996||17 Apr 2001||Worlds, Inc.||Scalable virtual world chat client-server system|
|US6249806||2 Oct 1998||19 Jun 2001||Fujitsu Limited||Apparatus and method for providing information about two-way computer communication services|
|US6252579||23 Aug 1997||26 Jun 2001||Immersion Corporation||Interface device and method for providing enhanced cursor control with force feedback|
|US6256663||22 Jan 1999||3 Jul 2001||Greenfield Online, Inc.||System and method for conducting focus groups using remotely loaded participants over a computer network|
|US6281651||3 Nov 1998||28 Aug 2001||Immersion Corporation||Haptic pointing devices|
|US6292174||4 May 2000||18 Sep 2001||Immersion Corporation||Enhanced cursor control using limited-workspace force feedback devices|
|US6329994||14 Mar 1997||11 Dec 2001||Zapa Digital Arts Ltd.||Programmable computer graphic objects|
|US6339754||29 Oct 1997||15 Jan 2002||America Online, Inc.||System for automated translation of speech|
|US6342880||6 Oct 1999||29 Jan 2002||Immersion Corporation||Force feedback system including multiple force processors|
|US6347307||13 Jun 2000||12 Feb 2002||Integral Development Corp.||System and method for conducting web-based financial transactions in capital markets|
|US6349301||24 Feb 1998||19 Feb 2002||Microsoft Corporation||Virtual environment bystander updating in client server architecture|
|US6393461||12 Feb 1999||21 May 2002||Fujitsu Limited||Communication management system for a chat system|
|US6396509||21 Feb 1998||28 May 2002||Koninklijke Philips Electronics N.V.||Attention-based interaction in a virtual environment|
|US6421048||28 Jun 1999||16 Jul 2002||Sensable Technologies, Inc.||Systems and methods for interacting with virtual objects in a haptic virtual reality environment|
|US6422941||23 Sep 1997||23 Jul 2002||Craig Thorner||Universal tactile feedback system for computer video games and simulations|
|US6425764||12 Dec 1997||30 Jul 2002||Ralph J. Lamson||Virtual reality immersion therapy for treating psychological, psychiatric, medical, educational and self-help problems|
|US6429846||19 Jan 2000||6 Aug 2002||Immersion Corporation||Haptic feedback for touchpads and other touch controls|
|US6442590||27 May 1999||27 Aug 2002||Yodlee.Com, Inc.||Method and apparatus for a site-sensitive interactive chat network|
|US6459892||26 Sep 2001||1 Oct 2002||Motorola, Inc.||Method for processing chat messages in a wireless chat device|
|US6496200 *||2 Nov 1999||17 Dec 2002||Interval Research Corp.||Flexible variation of haptic interface resolution|
|US6519771||14 Dec 1999||11 Feb 2003||Steven Ericsson Zenith||System for interactive chat without a keyboard|
|US6523022||7 Jul 1999||18 Feb 2003||Allen Hobbs||Method and apparatus for selectively augmenting retrieved information from a network resource|
|US6539218||28 Mar 2001||25 Mar 2003||Hitachi, Ltd.||Message transmitter and response message transmitting method in cellular mobile telephone apparatus and recording medium recording program for executing the method|
|US6546390||11 Jun 1999||8 Apr 2003||Abuzz Technologies, Inc.||Method and apparatus for evaluating relevancy of messages to users|
|US6559845||11 Jun 1999||6 May 2003||Pulse Entertainment||Three dimensional animation system and method|
|US6563487||17 Dec 1999||13 May 2003||Immersion Corporation||Haptic feedback for directional control pads|
|US6563503||25 Aug 1999||13 May 2003||Nintendo Co., Ltd.||Object modeling for computer simulation and animation|
|US6630944||22 Dec 1999||7 Oct 2003||Fujitsu Limited||Display control method and device|
|US6658010 *||24 Jul 1997||2 Dec 2003||Hybrid Networks, Inc.||High-speed internet access system|
|US6677858||30 May 2000||13 Jan 2004||Reveo, Inc.||Internet-based method of and system for monitoring space-time coordinate information and biophysiological state information collected from an animate object along a course through the space-time continuum|
|US6678720||10 Apr 2000||13 Jan 2004||Fujitsu Limited||Chat system and method for delivering additional information via another independent network|
|US6693626||12 May 2000||17 Feb 2004||Immersion Corporation||Haptic feedback using a keyboard device|
|US6711264||8 Oct 1999||23 Mar 2004||Fujitsu Limited||Security improvement method and security system|
|US6742116||30 Aug 1999||25 May 2004||Fujitsu Limited||Security method, security software and security system for electronic communications|
|US6755743||1 Sep 2000||29 Jun 2004||Kabushiki Kaisha Sega Enterprises||Communication game system and processing method thereof|
|US6760751||17 Dec 1997||6 Jul 2004||Sony Corporation||Method and apparatus for automatic sending of E-mail and automatic sending control program supplying medium|
|US6763371 *||10 May 1999||13 Jul 2004||Telefonaktiebolaget Lm Ericsson (Publ)||Method and apparatus for collaborative communication in a communication network|
|US6859819||31 Jul 2000||22 Feb 2005||Immersion Corporation||Force feedback enabled over a computer network|
|US6965868 *||24 Jan 2000||15 Nov 2005||Michael David Bednarek||System and method for promoting commerce, including sales agent assisted commerce, in a networked economy|
|US7084867 *||31 Mar 2000||1 Aug 2006||Massachusetts Institute Of Technology||Haptic interface system for collision detection and applications therefore|
|US7149208||25 Sep 1996||12 Dec 2006||Net2Phone, Inc.||Method and apparatus for providing caller identification based responses in a computer telephony environment|
|US7222228 *||31 Oct 2000||22 May 2007||Netwolves Corporation||System and method for secure management or remote systems|
|US20010044725||12 Nov 1997||22 Nov 2001||Koichi Matsuda||Information processing apparatus, an information processing method, and a medium for use in a three-dimensional virtual reality space sharing system|
|US20030067908||25 Sep 1996||10 Apr 2003||Shane D. Mattaway||Method and apparatus for providing caller identification based responses in a computer telephony environment|
|US20040125120||8 Jun 2001||1 Jul 2004||Michael Weiner||Method and apparatus for interactive transmission and reception of tactile information|
|EP0326439B1||30 Jan 1989||4 Aug 1993||British Aerospace Public Limited Company||Control apparatus|
|EP0349086A1||27 Jun 1989||3 Jan 1990||Stork Kwant B.V.||Control system|
|EP0655301B1||11 Nov 1994||17 Jun 1998||Asea Brown Boveri Ab||Method and device for calibration of movement axes of an industrial robot|
|EP0997177A2||8 Oct 1999||3 May 2000||Fujitsu Limited||Information processing apparatus and pseudo biological equipment|
|GB2235310A||Title not available|
|GB2235310B||Title not available|
|GB2325766A||Title not available|
|GB2325766B||Title not available|
|JPH048381Y2||Title not available|
|JPH0724147Y2||Title not available|
|JPH02185278A||Title not available|
|JPH05192449A||Title not available|
|JPH09138767A||Title not available|
|JPH10200882A||Title not available|
|WO1991002313A1||3 Aug 1989||21 Feb 1991||International Business Machines Corporation||Data processing network|
|WO1994025923A1||2 May 1994||10 Nov 1994||Novadigm, Inc.||Method and apparatus for enterprise desktop management|
|WO1999040504A9||2 Feb 1999||21 Oct 1999||Inmotion Technologies Ltd||System and method for vibro generations|
|WO2000010099B1||17 Aug 1999||30 Mar 2000||David Jaffe||Computer architecture and process for audio conferencing over local and global networks including internets and intranets|
|1||"Cyberman Technical Specification," Logitech Cyberman Swift Supplement, Apr. 5, 1994.|
|2||1998 IEEE International Conference on robotics and Automation, web page at www.wings.buffalo.edu/academic/department/eng/mae/ieee/icra98/ABST.html, as available via the Internet.|
|3||Adelstein, "A Virtual Environment System For The Study Of Human Arm Tremor," Ph.D. Dissertation, Dept. of Mechanical Engineering, MIT, Jun. 1989.|
|4||Adelstein, "Design and Implementation of a Force Reflecting Manipulandum for Manual Control research," DSC-vol. 42, Advances in Robotics, Edited by H. Kazerooni, pp. 1-12, 1992.|
|5||ATIP98.059, Virtual Reality (VR) Development at SERI (Korea), 1998, web page at www.atip.org/public/atip.reports.98.059r.html, as available via the Internet.|
|6||Aukstakalnis et al., "Silicon Mirage: The Art and Science of Virtual Reality," ISBN 0-938151-82-7, pp. 129-180, 1992.|
|7||Baigrie, "Electric Control Loading-A Low Cost, High Performance Alternative," Proceedings, pp. 247-254, Nov. 6-8, 1990.|
|8||Bejczy et al., "A Laboratory Breadboard System For Dual-Arm Teleoperation," SOAR '89 Workshop, JSC, Houston, TX, Jul. 25-27, 1989.|
|9||Bejczy et al., "Kinesthetic Coupling Between Operator and Remote Manipulator," International Computer Technology Conference, The American Society of Mechanical Engineers, San Francisco, CA, Aug. 12-15, 1980.|
|10||Bejczy, "Generalization of Bilateral Force-Reflecting Control of Manipulators," Proceedings Of Fourth CISM-IFToMM, Sep. 8-12, 1981.|
|11||Bejczy, "Sensors, Controls, and Man-Machine Interface for Advanced Teleoperation," Science, vol. 208, No. 4450, pp. 1327-1335, 1980.|
|12||Bejczy, et al., "Universal Computer Control System (UCCS) For Space Telerobots," CH2413-3/87/0000/0318501.00 1987 IEEE, 1987.|
|13||Bouguila, et al., "Effect of Coupling Haptics and Stereopsis on Depth," web page at www.dcs.gla.ac.uk/~stephen/workshops/haptic/papers/bougilia-paper.pdf, as available via the Internet.|
|14||Bouguila, et al., "Effect of Coupling Haptics and Stereopsis on Depth," web page at www.dcs.gla.ac.uk/˜stephen/workshops/haptic/papers/bougilia-paper.pdf, as available via the Internet.|
|15||Brooks et al., "Hand Controllers for Teleoperation-A State-of-the-Art Technology Survey and Evaluation," JPL Publication 85-11; NASA-CR-175890; N85-28559, pp. 1-84, Mar. 1, 1985.|
|16||Burdea et al., "Distributed Virtual Force Feedback, Lecture Notes for Workshop on Force Display in Virtual Environments and its Application to Robotic Teleoperation," 1993 IEEE International Conference on Robotics and Automation, pp. 25-44, May 2, 1993.|
|17||Caldwell et al., "Enhanced Tactile Feedback (Tele-Taction) Using a Multi-Functional Sensory System " 1050-4729/93, pp. 955-960, 1993.|
|18||Colgate, J. Edward, et al., "Implementation of Stiff Virtual Walls in Force-Reflecting Interfaces," Department of Mechanical Engineering, Northwestern University, Evanston, IL, Sep. 1993.|
|19||Curry, "Supporting Collaborative Interaction in Tele-Immersion," 1998, web page at www.sv.vt.edu/future/cave/pub/curryMs/CurryMS.html, as available via the Internet.|
|20||Eberhardt et al., "Including Dynamic Haptic Perception by The Hand: System Description and Some Results," DSC-vol. 55-1, Dynamic Systems and Control: vol. 1, ASME 1994.|
|21||Eberhardt et al., "OMAR-A Haptic display for speech perception by deaf and deaf-blind individuals," IEEE Virtual Reality Annual International Symposium, Seattle, WA, Sep. 18-22, 1993.|
|22||eRENA, Deliverable 7b.1, Pushing Mixed Reality Boundaries, 1999, web page at www.nada.kth.se/erena/pdf/D7b-1.pdf, as available via the Internet.|
|23||European Patent Office, Communication Pursuant to 94(3) EPC, Application No. 01995050, dated May 25, 2009.|
|24||European Patent Office, European Supplemental Search Report, Application No. 01995050, dated Mar. 18, 2009.|
|25||Gobel et al., "Tactile Feedback Applied to Computer Mice," International Journal of Human-Computer Interaction, vol. 7, No. 1, pp. 1-24, 1995.|
|26||Gotow et al., "Controlled Impedance Test Apparatus for Studying Human Interpretation of Kinesthetic Feedback," WA11-11:00, pp. 332-337, 1989.|
|27||Hansen, "Enhancing Documents with Embedded Programs: How Ness Extends Insets in the Andrew Toolkit," Computer Languages, 1990, IEEE.|
|28||Hayward et al., "Parameter Sensitivity Analysis for Design and Control of Force Transmission Systems," 1995, web page at www.cim.mcgill.ca/~haptic/pub/MC.trans.ps.gz, as available via the Internet.|
|29||Hayward et al., "Parameter Sensitivity Analysis for Design and Control of Force Transmission Systems," 1995, web page at www.cim.mcgill.ca/˜haptic/pub/MC.trans.ps.gz, as available via the Internet.|
|30||Hinckley,K., "Haptic Issues for Virtual Manipulation," Dissertation for PHD at the University of Virginia, pp. 1-200, Dec. 1996.|
|31||Howe, "A Force-Reflecting Teleoperated Hand System for the Study of Tactile Sensing in Precision Manipulation," Proceedings of the 1992 IEEE International Conference on Robotics and Automation, Nice, France, May 1992.|
|32||IBM Technical Disclosure Bulletin, "Mouse Ball-Actuating Device With Force and Tactile Feedback," vol. 32, No. 9B, Feb. 1990.|
|33||IEEE International Conference on Robotics and Automation, May 16-20, 1998, Lueven, Belgium, 41 pages.|
|34||Iwata, "Pen-based Haptic Virtual Environment," 0-7803-1363-1/93 IEEE, pp. 287-292, 1993.|
|35||Jacobsen et al., "High Performance, Dextrous Telerobotic Manipulator With Force Reflection," Intervention/ROV '91 Conference & Exposition, Hollywood, Florida, May 21-23, 1991.|
|36||Japanese Patent Office, Notification of Reasons for Refusal, Application No. 2010-260419, dispatched Aug. 30, 2012.|
|37||Japanese Patent Office, Notification of Reasons for Refusal, Application No. 2010-260419, dispatched Oct. 27, 2011.|
|38||Jones et al., "A perceptual analysis of stiffness," ISSN 0014-4819 Springer International (Springer-Verlag); Experimental Brain Research, vol. 79, No. 1, pp. 150-156, 1990.|
|39||Jones et al., "NSF Workshop on Human-Centered Systems: Breakout Group 2-Communication and Collaboration," web page at www.ifp.uiuc.edu/nsfhcs/bog-reports/bog2.html, as available via the Internet and printed Mar. 1, 2005.|
|40||Kaczmarek et al., "Tactile Displays," Virtual Environment Technologies, May 1995.|
|41||Kontarinis et al., "Display of High-Frequency Tactile Information to Teleoperators," Telemanipulator Technology and Space Telerobotics, Won S. Kim, Editor, Proc. SPIE vol. 2057, pp. 40-50, Sep. 7-9, 1993.|
|42||MacLean, "Designing with Haptic Feedback," 2000, web page at www.cs.ubc.ca/nest/lci/papers/2000/maclean-icra00-DesignWithHaptic-reprint.pdf, as available via the Internet.|
|43||Mania et al., "A Classification for User Embodiment in Collaborative Virtual Environments (extended version)," web page at www.cs.bris.ac.uk/~mania/paper1.htm, as available via the Internet.|
|44||Mania et al., "A Classification for User Embodiment in Collaborative Virtual Environments (extended version)," web page at www.cs.bris.ac.uk/˜mania/paper1.htm, as available via the Internet.|
|45||Marcus, "Touch Feedback in Surgery," Proceedings of Virtual Reality and Medicine The Cutting Edge, Sep. 8-11, 1994.|
|46||McAffee, "Teleoperator Subsystem/Telerobot Demonstrator: Force Reflecting Hand Controller Equipment Manual," JPL D-5172, pp. 1-50, A1-A36, B1-B5, C1-C36, Jan. 1988.|
|47||McLaughlin et al., "The USC Interactive Art Museum: Removing the Barriers between Museums and their Constituencies," web page at http://ascusc.org/jcmc/paperforica.html, as available via the Internet.|
|48||Mine, "ISAAC: A Virtual Environment Tool for the Interactive," 1995, web page at ftp.cs.unc.edu/pub/technical-reports/95-020.ps.Z, as available via the internet.|
|49||Mine, "Virtual Environment Interaction Techniques," 1995, web page at ftp.cs.unc.edu/pub/technical-reports/95-018.ps.Z, as available via the Internet.|
|50||Minsky, "Computational Haptics: The Sandpaper System for Synthesizing Texture for a Force-Feedback Display," Ph.D. Dissertation, MIT, Jun. 1995.|
|51||NPA International Inc., Office Action, Application No. 200810083652, dated Aug. 19, 2010.|
|52||NPA International Inc., Office Action, Application No. 200810083652, dated Dec. 15, 2009.|
|53||Ouhyoung et al., "The Development of a Low-Cost Force Feedback Joystick and Its Use in the Virtual Reality Environment," Proceedings of the Third Pacific Conference on Computer Graphics and Applications, Pacific Graphics '95, Seoul, Korea, Aug. 21-24, 1995.|
|54||Ouh-Young, "A Low-Cost Force Feedback Joystick and Its Use in PC Video Games," IEEE Transactions on Consumer Electronics, vol. 41, No. 3, Aug. 1995.|
|55||Ouh-Young, "Force Display in Molecular Docking," Order No. 9034744, p. 1-369, 1990.|
|56||Pao et al., "Synergistic Visual/Haptic Computer Interfaces," 1998, Hanoi, Vietnam, web page at schof.colorado.edu/~pao/anonftp/vietnam.ps, as available via the Internet.|
|57||Pao et al., "Synergistic Visual/Haptic Computer Interfaces," 1998, Hanoi, Vietnam, web page at schof.colorado.edu/˜pao/anonftp/vietnam.ps, as available via the Internet.|
|58||Patent Cooperation Treaty, International Preliminary Examination Report, International Application No. PCT/US01/41099, dated May 18, 2002.|
|59||Patent Cooperation Treaty, International Search Report, International Application No. PCT/US01/41099, mailed Jan. 2, 2002.|
|60||Patrick et al., "Design and Testing of A Non-reactive, Fingertip, Tactile Display for Interaction with Remote Environments," Cooperative Intelligent Robotics in Space, Rui J. deFigueiredo et al., Editor, Proc. SPIE vol. 1387, pp. 215-222, 1990.|
|61||Picinbono et al., "Extrapolation: A Solution for Force Feedback?" Virtual Reality and Prototyping, 1999, web page at www-sop.inria.fr/epidaure/AISIM/CompteRendu/aisim3/picinobo.pdf, as available via the Internet.|
|62||Pimentel et al., "Virtual Reality: through the new looking glass," 2nd Edition; McGraw-Hill, ISBN 0-07-050167-X, pp. 41-202, 1994.|
|63||Rabinowitz et al., "Multidimensional tactile displays: Identification of vibratory intensity, frequency, and contactor area," Journal of The Acoustical Society of America, vol. 82, No. 4, Oct. 1987.|
|64||Real Time Graphics, Green Bar-Full Page EPS, Aug. 1998, web page at www.cgsd.com/rtqAug98.pdf as available via the Internet.|
|65||Ruspini et al., "The Haptic Display of Complex Graphical Environments," 1997, Computer Graphics Proceedings, Annual Conference Series, www.robotics.stanford.edu/people/krasi/Siggraph97.ps.Z, as available via the Internet.|
|66||Russo, "Controlling Dissipative Magnetic Particle Brakes in Force Reflective Devices," DSC-vol. 42, Advances in Robotics, pp. 63-70, ASME 1992.|
|67||Russo, "The Design and Implementation of a Three Degree of Freedom Force Output Joystick," MIT Libraries Archives Aug. 14, 1990, pp. 1-131, May 1990.|
|68||Scannell, "Taking a Joystick Ride," Computer Currents, Boston Edition, vol. 9, No. 11, Nov. 1994.|
|69||Shimoga, "Finger Force and Touch Feedback Issues in Dexterous Telemanipulation," Proceedings of Fourth Annual Conference on Intelligent Robotic Systems for Space Exploration, Rensselaer Polytechnic Institute, Sep. 30-Oct. 1, 1992.|
|70||Snow et al., "Model-X Force-Reflecting-Hand-Controller," NT Control No. MPO-17851; JPL Case No. 5348, pp. 1-4, Jun. 15, 1989.|
|71||Stanley et al., "Computer Simulation of Interacting Dynamic Mechanical Systems Using Distributed Memory Parallel Processors," DSC-vol. 42, Advances in Robotics, pp. 55-61, ASME 1992.|
|72||State Intellectual Office of the Peoples Republic of China, Notification of the First Office Action, Application No. 01810866, dated Apr. 2, 2004.|
|73||Tadros, "Control System Design for a Three Degree of Freedom Virtual Environment Simulator Using Motor/Brake Pair Actuators", MIT Archive © Massachusetts Institute of Technology, pp. 1-88, Feb. 1990.|
|74||Terry et al., "Tactile Feedback In A Computer Mouse," Proceedings of Fourteenth Annual Northeast Bioengineering Conference, University of New Hampshire, Mar. 10-11, 1988.|
|75||Thompson, II et al., "maneuverable Nurbs Models within a Haptic Virtual Environment," web page available at www.cs.utah.edu/gdc/publications/papers/thompson97b.ps.Z, as available via the Internet.|
|76||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 09/608,129, mailed Aug. 10, 2004.|
|77||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 09/608,129, mailed Mar. 15, 2005.|
|78||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 09/608,129, mailed Mar. 19, 2004.|
|79||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 09/608,129, mailed Nov. 28, 2005.|
|80||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 09/608,129, mailed Sep. 24, 2003.|
|81||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 11/545,739, mailed Mar. 5, 2008.|
|82||United States Patent and Trademark Office, Office Action, U.S. Appl. No. 11/545,739, mailed Oct. 22, 2007.|
|83||Wloka, "Interacting with Virtual Reality," 1995, web page at wilma.cs.brown.edu/research/graphics/research/pub/papers/coimbra.ps, as available via the Internet.|
|84||Wloka, M, Interacting with Virtual Reality, Science and Technology Center for Computer Graphics and Scientific Visualization, Brown University Site, Department of Computer Science, Brown University, 1995, 14 pages.|
|85||Yamakita et al., "Tele-Virtual reality of Dynamic Mechanical Model," Proceedings of the 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC, Jul. 7-10, 1992.|
|86||Yamakita, M. et al., Tele-Virtual Reality of Dynamic Mechanical Model, Proceedings of the 1992 IEEE/RSJ International Conference on Intelligent Robots and Systems, Raleigh, NC Jul. 7-10, 1992.|
|International Classification||G06F15/16, G06F3/01, G06F15/00, G06F13/00, G06F3/00|
|Cooperative Classification||G06F3/016, G06Q10/06311, H04L12/1827, G06F3/01|