WO1995034881A1 - Interactive projected video image display system - Google Patents

Interactive projected video image display system Download PDF

Info

Publication number
WO1995034881A1
WO1995034881A1 PCT/US1995/007487 US9507487W WO9534881A1 WO 1995034881 A1 WO1995034881 A1 WO 1995034881A1 US 9507487 W US9507487 W US 9507487W WO 9534881 A1 WO9534881 A1 WO 9534881A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
computer
projected
video
captured
Prior art date
Application number
PCT/US1995/007487
Other languages
French (fr)
Inventor
Daniel Marcel Platzker
Yoav Agmon
Reuven Ackner
Original Assignee
Daniel Marcel Platzker
Yoav Agmon
Reuven Ackner
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Daniel Marcel Platzker, Yoav Agmon, Reuven Ackner filed Critical Daniel Marcel Platzker
Priority to JP8502407A priority Critical patent/JPH09512656A/en
Priority to AU29440/95A priority patent/AU2944095A/en
Priority to DE69530395T priority patent/DE69530395T2/en
Priority to EP95925249A priority patent/EP0771460B1/en
Publication of WO1995034881A1 publication Critical patent/WO1995034881A1/en

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011—Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03—Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/041—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
    • G06F3/042—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means
    • G06F3/0425—Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means by opto-electronic means using a single imaging device like a video camera for tracking the absolute position of a single or a plurality of objects with respect to an imaged reference surface, e.g. video camera imaging a display or a projection screen, a table or a wall surface, on which a computer generated image is displayed or projected
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B5/00—Electrically-operated educational appliances
    • G09B5/06—Electrically-operated educational appliances with both visual and audible presentation of the material to be studied
    • G09B5/065—Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems

Definitions

  • the present invention relates generally to projected video image display systems.
  • the invention relates to a computer-based projected video system and method of providing interaction directly with a projected video image without the need for an active computer input device.
  • the conventional way to provide interaction with a computer requires use of an "active" element or computer input device to provide cursor movement control.
  • Examples o of common computer input devices include absolute positioning devices, such as light pens and digitizing tablets; and also relative positioning devices, such as joy sticks, mouse devices, and track balls.
  • Such active input devices are usually too limiting for effective use in projected video image display systems.
  • absolute positioning devices such as conventional light pens, require the user to interact directly with the video 5 screen display of the monitor associated with the computer. This disrupts the spontaneity of the presentation as the user must repeatedly go back and forth between the computer monitor and the projection screen when the user desires to change the computer generated video image.
  • the present invention provides an interactive projected image video display system which includes a computer for generating video images, a projection apparatus coupled to the video output port of the computer for projecting computer-generated video images onto a projection screen, and an optical recording device such as a conventional video camera, aimed at the projection surface for visually recording the projected image.
  • the computer includes a frame grabber board or like image capturing means coupled to the video camera for capturing individual frames or series of frames from the video recording of the projected image.
  • the captured images are then compared to the computer generated images to detect for the presence of an object having a particular feature or a recognized pattern/feature created by the combination of the introduced object and the video image.
  • the detected object indicates an instruction to the computer, such as, for example, to modify the video image.
  • the object is introduced onto the projected image by a user during a presentation.
  • the object may include any convenient pointing element having a recognizable feature, such as for example, a user's hand or fingers held in a particular orientation or angle, a pointer, a pen, etc.
  • the means for detecting the pointing element and activating the computer to perform certain tasks is preferably embodied in a software program written in accordance with known image processing algorithms and techniques.
  • the user may simulate "clicking" a button contained in the projected image.
  • the button contains an instruction for signalling the computer to perform a certain task, such as, for example, advance to the next image, save an image to memory, etc.
  • the location of the button is predetermined and known to the computer.
  • the user places the pointing element over the button on the projected image.
  • the video camera is recording this activity.
  • the frame grabber will grab a single frame containing the button and pointer. A search of a block region on the grabbed image corresponding to the button location is then made.
  • an activation signal is generated and the computer will execute the instruction corresponding to the selected button.
  • a grey scale comparison technique is used to check the amount of shadowing present in the projected button by comparison with other portions of the projection surface where the light intensity is known or by comparison with the level of shadowing previously measured at the same button region.
  • the user may "click" the projected button by simply holding his or her hand or some other object to block or cover the projected button for a predetermined length of time.
  • the user may "click" the projected button by shinning a light source, such as a light pen, onto the projected button to induce a change in the light intensity on the projected image for capture by the video camera and comparison with the computer generated image.
  • a light source such as a light pen
  • the Touch detection process described in the above paragraph is algorithmically very simple, but is subject to "false triggering" by a number of events that might occur during the interactive session. Examples of these events include: (1) accidental or unintended obscuration by a user such as may occur when the user walks in front of the projected buttons; and (2) sudden changes in the local illumination of the projected area by external sources (e.g. doors being opened, room lights being turned on or off, etc.).
  • a technique requiring spatial localization and temporal consistency is used. The computer not only examines the target area of the buttons, but also examines other neighboring areas or regions that surround the target area.
  • buttons are also being obscured, then the touch detection is inhibited.
  • the technique can be modified to allow for some of the invisible buttons to be obscured but not others, thus allowing the user to extend his or her arm thereacross to touch the desired button in the target area.
  • this concept “differential” touch.
  • a requirement for the obscuration configuration to be sustained for a fixed amount of time is also used to prevent triggering during the transit of a user across the field of view.
  • the computer may be programmed in accordance with more advanced image processing techniques, such as feature extraction or pattern correlation, in order to look for special patterns or pattern changes present in the captured image as compared with the projected image.
  • special patterns may include, for example, a cross formed by a pen being laid over a projected line contained in a projected button.
  • the computer may also be programmed to look for other special features such as profile, size, color, etc., or special movements of the introduced object, such as, for example, a hand movement corresponding to making a check mark.
  • the user may simulate marking points on the projected video image.
  • the desired mark locations are not already known to the computer.
  • the user marks the location by pointing using a pointing element or pointer (having a pattern or posture recognizable by the computer).
  • the user marks the points by touching the projected screen.
  • the touching of a pointing element or pointer (having a pattern or feature recognizable by the computer) to the projection screen can be verified again by using known grey scale comparison techniques in order to detect when the pointer meets its own shadow.
  • a second video camera oriented at a side angle relative to the projection screen may be used to verify that the pointer actually touches the surface of the projection screen.
  • the pointing element is used in combination with the projected video to generate a predetermined feature for marking points on the image.
  • the projected video image may include a series of equally spaced vertical lines that cover the whole image. The user marks a point by generating a recognizable cross pattern by holding the pointing element over one of the vertical lines. More advanced features can be generated by combining more sophisticated patterns into the video image.
  • the user may use the pointer as a video cursor control means just like a mouse pointing device or light pen in order to enter hand written text or draw lines, curves or other graphical objects directly on the projected image.
  • FIG. 1 is a diagrammatic view of an interactive projected video image display system in accordance with one embodiment of the present invention.
  • Fig. 2 is a diagrammatic view of a system similar to Fig. 1 in accordance with a second embodiment of the present invention.
  • Fig. 3 A is a flow chart which describes and shows the process of clicking a button for initiating a modification of the projected video image.
  • Fig. 3B is an enlarged diagrammatic view of a projected button contained in a projected video image and which shows an object having a particular feature (in this case a user's hand) being introduced over the projected button.
  • Fig. 3C is a schematic view of a projected button similar to Fig. 3B except in this case the projected button includes a projected vertical line and the object having a particular feature is a pointer.
  • Fig. 4 is a schematic view of a projected video image of a menu window which illustrates the general pointing and positioning mode of operation of the present invention.
  • Fig. 5A-6 illustrate a video mouse aspect of the invention.
  • Fig. 5 A diagrammatically shows a projected video image with a first pop up menu window activated and which contains a list of user selectable command items for performing conventional mouse operations.
  • Fig. 5B diagrammatically shows a projected video image with a second pop up menu window activated which contains a single user selectable command item for ending mouse operation.
  • Figs. 6A-6B is a two part series depicting a flow chart which describes and shows the process for simulating video mouse operation on a projected video image.
  • Fig. 1 shows a generalized diagrammatic view of a projected image video display system 10 in accordance with one embodiment of the present invention.
  • the system 10 includes a computer 12 for generating and displaying a video on monitor 16 (optional).
  • the monitor 16 is coupled to the video output port 12a of the computer 12.
  • an apparatus 18 for providing an enlarged projection of the video image generated by the computer 12.
  • the apparatus 18 includes a conventional liquid crystal display (LCD) panel 20 and an overhead projector 22.
  • the LCD panel 20 receives video image signal input from the computer 12 and generates light images which, in turn, are projected by the overhead projector 22 as a projected image 24 on a viewing surface, such as projection screen 26.
  • the projection screen 26 may be any flat, and preferably light colored surface such as, for example, the ubiquitous white board found in most office conference rooms.
  • two projected buttons 32, 34 are contained in the projected image 24.
  • the apparatus 18 is described as including a separate LCD panel 20 and overhead projector 22, it is understood that single machines capable of receiving and projecting video image data are becoming available and may be advantageously used in the present invention.
  • the system 10 further includes an optical sensing device, such as video camera 28a, for visually recording the projected image 24.
  • the video camera 28a is coupled to the video input port 12b of the computer 12.
  • the video input port 12b computer includes a conventional frame grabber board or equivalent image capturing means for selectively grabbing individual frames or groups of frames of the projected image as recorded by the video camera 28a.
  • One or more additional video cameras, indicated generally at 28b,...,28n, may be included for providing enhanced feature detection capability to the system 10. This is discussed in more detail below with reference to the second mode of operation.
  • the computer 12 is programmed to compare the captured images with the original video image. Upon detection of certain recognized objects having a particular feature which are present in the captured image or upon detection of a recognizable pattern which has been formed by the combination of an introduced object and an existing feature contained in the projected image, a signal is generated to activate the computer to perform a specific instruction, such as for example, modify the video image, save a captured image to memory, etc.
  • Fig. 2 shows an alternate embodiment for the system 10 of the present invention.
  • the system components common to both the first embodiment of Fig. 1 and the second embodiment of Fig. 2 are indicated by the same reference numerals.
  • the second embodiment of Fig. 2 differs from the first embodiment of Fig. 1 in that video image 14 is not projected but instead is presented as an enlarged displayed image 24a on large screen video display 26a.
  • the large screen video display 26a is coupled directly to the video output port 12a of the computer 12.
  • two displayed buttons 32a, 34a are contained in the displayed image 24a.
  • the preferred optical sensing device includes at least one video camera 28a which provides a continuous visual recording of the projected or displayed video image.
  • video camera 28a which provides a continuous visual recording of the projected or displayed video image.
  • other optical sensing devices such as, for example, a charged coupled device (CCD) image sensor or camera may be used in place of the video camera with equally good results.
  • CCD charged coupled device
  • appropriate signal processing apparatus or software may be coupled with a CCD image sensor or camera to convert the recorded images in a form usable by the computer for image comparison with the original known video-generated images.
  • the projected image (Fig. 1) or displayed image (Fig. 2) contains “instructions” for the user. These "instructions" are associated with the projected or displayed buttons contained in the projected or displayed image, respectively.
  • the user may simulate a mouse click by merely touching or placing an object, such as a pointer, over the desired projected or displayed button.
  • the process for clicking a single button will be described.
  • the user inputs information onto the projected or displayed image by either writing information or effecting changes by selecting a desired button. All during this time, the video camera 28a continuous records these changes.
  • the frame grabber grabs a single frame of the video recording (step 40).
  • the computer reads (at step 42) a block region of the frame grabbed by the frame grabber corresponding to the location of a first button. This block region has a location which is known to the computer.
  • the block region is checked (step 44) for the presence of a an object having a particular feature or a recognizable pattern.
  • step 46 the computer executes a specified control function associated with the selected button (step 50). If there is no match or if the object or pattern does not exist, then no computer operation is triggered. The above process is repeated for each grabbed frame.
  • Process steps 42-46 are repeated until all the blocks corresponding to all the button locations in the projected or displayed image are checked for recognizable patterns before advancing to the next grabbed frame.
  • the feature or recognition algorithm may use any one of many well known algorithms including but not limited to binary methods, gray scale methods, etc. As will be appreciated by those having ordinary skill in the art, the particular method chosen will depend in large part on the type of pattern or feature to be recognized.
  • the image discrimination is preferably implemented in software in the form of a pattern correlation or feature extraction algorithm.
  • Techniques for detecting features in digital images are well-known in the art. Exemplar ' references which disclose such techniques include: Digital Picture Processing. Azriel Rosenfeld and Avinash Kak, Vol. II, 2nd edition, Academic Press, 1982; Digital Image Processing. Rafael C. Gonzales,
  • Figs. 3B-3C show examples of objects having a particular feature being introduced over a projected button 32.
  • the button 32 represents the instruction "next".
  • the user 30 introduces his or her hand 31 (or other recognized pointing element) over the button 32 to simulate clicking the button 32 to generate a new video image 14.
  • the projected button 34 contains a projected vertical line 36.
  • the user introduces a pointer such as a ruler 38 onto the button to form a cross.
  • the cross represents a special pattem recognizable by the computer for carrying out a desired instruction.
  • the instruction "capture” signals the computer to save the captured image in a file in the memory of the computer.
  • the captured image may be recalled at a later time to reappear as the projected image.
  • several captured images which have been stored in memory may be recalled at once, each superimposed upon the other, to provide a multi-layering effect.
  • Such a multi-layering feature is particularly advantageous when used in combination with a projection screen in the form of a white board whereby one or more users may draw on the board with erasable ink markers of different colors.
  • Fig. 4 illustrates the general pointing and positioning mode of operation of the invention.
  • the projected image 24 includes a menu bar 50 containing menu items 52a, 52b,...,52n and a general positioning and pointing field 54.
  • the computer searches the entire grabbed image to detect the presence of an object anywhere on the image.
  • the computer will perform the task associated with the selected menu item in accordance with the first mode of operation described above. If the object is detected in the general positioning and pointing field, the computer will generate a point or position a cursor at the location corresponding to where the user has placed the object. In the example shown, the computer is looking for an object in the form of two fingers 60 extended in the pointing position. In this case, two marks 56, 58 have been generated at locations on the projected image where the user's two fingers 60 have been placed.
  • the process of marking a point in the general positioning can be implemented by programming the computer to look for a special feature or features that are generated by the combination of the pointing element and the projected image. For example, as seen in
  • the projected image may be provided with equally spaced vertical lines 53a-53m.
  • the user can then quickly mark point at 59 by placing his or her finger 57 over the vertical line 53c to form a cross shape.
  • the cross shape is readily recognizable by the computer using only a minimum of processing for detection of this simple pattem.
  • the computer can be programmed to look for and recognize a more advanced feature or features by including a more sophisticated pattem in the video image.
  • the above described feature recognition process may be implemented in a front end application program to simulate a mouse for accessing information from other currently running programs and for moving around within the different applications on the screen.
  • a front end application program to simulate a mouse for accessing information from other currently running programs and for moving around within the different applications on the screen.
  • Such application is possible in every computer with an operating system that support a mouse or a pen-like type interface.
  • An example of such operating system is WINDOWSTM (WINDOWSTM is a trademark of Microsoft Corporation).
  • WINDOWSTM a trademark of Microsoft Corporation
  • This may be done, for example, by including within the front end application program, a mouse driver written in software, which will provide a series of pop up menu windows containing user selectable command items corresponding to basic mouse operations.
  • the selection of a desired command item for carrying out a particular mouse operation may be done by placing a "recognized" object (e.g. a pointer, the user's hand or fingers held in a particular way, etc.
  • the mouse driver of the present invention preferably simulates the following four basic operations of a mouse that are common to most conventional mouse drivers:
  • the mouse driver will send a message to the operating system indicating that a particular event has occurred when a command item corresponding to the mouse operation associated with that event has been selected from the associated pop up menu window.
  • the (x,y) coordinates of the mouse at the time the event occurred will also be sent to the operating system.
  • a single o click of the mouse is performed as a combination of "Mouse_Down” and “Mouse_Up” operations without moving the mouse.
  • Figs. 5A-5B illustrate, by way of example, a projected image 26 which depicts a screen display associated with a WINDOWSTM Paintbrush program.
  • the projected image 26 is displaying the main Paintbrush window and shows the items in the main Paintbrush window including the top menu bar 62 (having command items such as "File”, "Edit”, etc.) the tool box 64, and the linesize box 66.
  • Fig. 6a-6b show, in flow chart form, the steps of the video mouse driver as it relates to a currently running WINDOWSTM application program. It is understood that the process of the invention is not limited to just WINDOWSTM type applications but may also be implemented in any application which supports a mouse, a 5 pen or like computer input device.
  • the video mouse driver starts (step 100) when a video image generated by the computer 12 is projected by apparatus 18 onto projection surface or screen 26 (see e.g. Fig. 1).
  • the computer determines (in step 102) whether the user has pointed at the projected image. This is done using the feature/pattern detection process as previously described.
  • WINDOWSTM When the driver detects that the user has pointed at the screen it sends WINDOWSTM (in step 104) the Mouse_Move message along with the coordinates of the new point. WINDOWSTM reacts by moving the cursor 61 of the application. The exact shape of the cursor depends on the specific application.
  • step 106 The driver then checks (in step 106) whether the user continues to point at the projected or displayed video image. If the answer is negative the program returns to its starting point (step 100). If the answer is positive, i.e., the user is still pointing at the projected or displayed video image, the driver checks (in step 108) whether the user has moved the pointer. If the user points at a different place, the program returns to step 104 to update the location of the cursor. If the user keeps pointing at the same location for a predetermined period of time, e.g., 2 seconds (step 110), then a first pop up menu window 68 is projected or displayed adjacent to the cursor position 61 (step 112).
  • a predetermined period of time e.g. 2 seconds
  • the first pop up menu 68 contains the four command items: NOP 70 (which stands for "no operation"), CLICK 72, DOWN 74, and DOUBLE CLICK 76. Adjacent each command item 70, 72, 74, and 76 is a respective selection button 71, 73, 75, and 77. As before, the buttons may be selected by the recognized feature/pattern detection process as described above with reference to Figs. 3B-3C and 4.
  • the computer then checks (in step 114) for a selection of command item contained in the first pop up window 68 allowing the user to make such a selection within a predetermined period of time, e.g., 2 seconds (step 116). If no selection is made during the predetermined period of time, the first pop up menu 68 disappears and the program returns to its starting point. If a selection is made, the selected command item is performed.
  • a predetermined period of time e.g. 2 seconds
  • the driver discriminates between two types of selections (step 120).
  • the first selection is the command DOWN, which requires special processing shown in Fig. 6B.
  • the second selection is for the commands NOP, CLICK, and DOUBLE CLICK.
  • the first pop up menu 68 disappears, the driver executes the command item by sending an appropriate message to the operating system, and the program returns to it starting point (step 100) after a small delay (e.g. 2 seconds).
  • the small delay is necessary in order to allow the user enough time to move the pointer to a new desired location.
  • the process by which the driver executes the command in the menu depends on the specific command. For example, to simulate a single mouse click, the user will select button 73 corresponding to the CLICK command item 72. In this case the mouse driver sends to the operating system the messages the "Mouse_Down” followed by "Mouse_Up", which the WINDOWSTM recognizes as a mouse click. In addition, the mouse driver, sends the (x,y) coordinates of the point in order to update the cursor location. The click triggers an event that will depend on the particular application being run at the time, e.g., drawing a curve.
  • buttons 77 corresponding to the DOUBLE CLICK command item 76 This action signals the mouse driver to send to WINDOWSTM the four messages "Mouse_Down”, “MouseJ p", “Mouse_Down” and “Mouse_Up” within a predetermined time frame.
  • the user also has the option of selecting button 71 corresponding to the NOP (no operation) command item 70. This will make the first pop up window 68 disappears without affecting the cursor location, and causes the driver to go back to the starting point (step 100).
  • a user chooses the button 75 corresponding to the Down command 74
  • the driver response by sending a Mouse_Down message to the operating system (step 126) with the additional information of the coordinates of the point.
  • the first pop up menu window 68 disappears and the user is given a short delay (e.g., 2 seconds) to return the pointer from the location of the button 75 to the location of the cursor 61.
  • the computers resumes following the movement of the user's fingers 60.
  • a series of updated "Mouse_Move” messages are sent to the operating system along with the updated (x,y) coordinates (step 128).
  • step 132 One option is to stop pointing, i.e., by withdrawing his or her finger(s) from the projected or displayed image. This will cause the driver to send to WINDOWSTM a MouseJQp command, and then after a small delay, the program will return to the beginning (step 140).
  • a second pop up menu window 78 (see e.g., Fig. 5b) is displayed.
  • the second pop up menu window 78 preferably contains only the single command item Up 80 along with corresponding selection button 81.
  • the mouse driver sends the "MouseJJp" message to WINDOWSTM, and the application (in this case, the
  • WINDOWSTM Paintbrush program responds accordingly (step 140). The process then returns to the start after a predetermined delay. If (at step 136) a selection is not made by the user within a predetermined period of time, e.g., 2 seconds (step 138), then the second pop up menu 78 disappears and the user may continue to move his or her finger(s) 60 along the projected or displayed image 26 as before. This will result in new "Mouse_Move" messages being sent to WINDOWSTM until the user stops finger movement after which steps 132-138 are repeated. SELF TEST
  • the system may include means for performing a self test to verify the calibration of the system.
  • the system may have stored in memory, two images which are identical except that the second of the two images contains an object having a particular feature, such as a pointer, located over a button contained in the image. This particular button is unoccupied in the first image.
  • the system projects the second image containing the pointer located over the particular button and compares this image to the first image.
  • the self test mode will be successful if the computer identifies the particular button.
  • a calibration algorithm may also be provided to calibrate the respective X,Y axis systems of the projected image and the image captured by the camera 28a in order to maintain focus, color balance, distortion and other parameters of the projected video image. This may be done by projecting predetermined images, such as comer markings 27 (see Fig. 1), having known locations, scale dimensions and color or light intensities, and comparing the predetermined comer markings 27 to the corresponding comer markings in the captured image.
  • predetermined images such as comer markings 27 (see Fig. 1), having known locations, scale dimensions and color or light intensities
  • the calibration algorithm serves two purposes: (1) the calibration algorithm "conditions" the captured images by removing undesirable light reflections, stray marks, cracks or like irregularities on the viewing surface which have an adverse effect on the capture image quality (i.e., image distortion, image resolution, image contrast, etc.); and (2) the calibration algorithm is used to determine an alignment function for ensuring the alignment of subsequently captured images with the projected images.
  • the conditioning portion of the calibration algorithm is carried out by first suppressing or "blanking" the video image generated by the computer such that a blank image is illuminated and projected onto the viewing surface.
  • This provides a first point of reference or “reference image” which enables the system to gather information about particular irregularities on the viewing surface which adversely effect image quality, such as, for example, shadows and/or bright reflections on the viewing surface due to uneven surface conditions and ambient lighting, permanent stains, stray marks or cracks on the viewing surface, etc.
  • the system then captures the blank projected image as a first reference image.
  • a plurality of reference markings e.g. crosses or like indicia, are projected onto the viewing surface and are captured by the frame grabber as one or more additional reference images.
  • the X and Y coordinates for the projected reference markings are pre-specified by the computer. It should be noted that the reference markings can be projected at random points on the viewing surface within the boundaries of the projected image and not necessarily just at the four comer regions as is typically done in accordance with prior art calibration techniques.
  • the captured reference images containing the reference markings are then compared with the reference markings generated by the computer.
  • the computer determines an alignment function which represents the relative displacement, in X and Y coordinates values, between the reference markings contained in the captured additional references images and the reference markings in the computer generated video image.
  • the alignment function once determined, can be used to ensure alignment of the captured images with the projected images.
  • the above calibration algorithm may be implemented automatically any time the individual apparatus components of the system have been bumped or moved slightly during use, or when changes in the ambient light conditions occur, or when changes occur in any other parameter which effects image quality.
  • the calibration algorithm may be advantageously implemented in the situation where a user has made annotations to the projected image by making physical changes on the viewing surface. For example, in the situation where the viewing surface is a conventional white board, the user may want to make edits or corrections to the projected image by drawing directly on the white board. The user may initiate the capture command and save the annotations made to a particular projected image. Since the computer already has a record of the projected image stored in memory, only the annotations are saved to memory. This avoids image degradation caused by the recapture of image information which is already known to the computer.

Abstract

An interactive projected image video display system which includes a computer (12) for generating video images, a projection apparatus (20, 22) coupled to the video output port (12a) of the computer for projecting computer-generated video images onto a projection screen (26), and at least one video camera (28a...n) for visually recording the projected image. The computer includes a frame grabber board coupled to the video camera for capturing individual frames or series of frames from the video recording of projected image. The captured images are compared to the computer generated images to detect for the presence of an object introduced onto the projected image by a user to instruct the computer to trigger a computer operation. The system can also simulate a mouse to be used with any of the shelf application that uses a regular mouse and allow annotation of computer generated data on a markable projection or viewing surface such as whiteboard.

Description

INTERACTIVE PROJECTED VIDEO IMAGE DISPLAY SYSTEM BACKGROUND OF THE INVENTION
1. Field of the Invention
5 The present invention relates generally to projected video image display systems.
More particularly, the invention relates to a computer-based projected video system and method of providing interaction directly with a projected video image without the need for an active computer input device.
2. Description of the Prior Art o Projected video image display systems of the kind which use a computer for generating a video image and a projection device for projecting the computer-generated video image onto an enlarged screen are becoming popular for use in making presentations to a group of people in a small to medium size conference room environment. In such projected video display systems, it is desirable for a user standing next to the projection s screen (located a distance away from the computer) to be able to enter information interactively into the computer for modifying or generating new images during the presentation.
The conventional way to provide interaction with a computer requires use of an "active" element or computer input device to provide cursor movement control. Examples o of common computer input devices include absolute positioning devices, such as light pens and digitizing tablets; and also relative positioning devices, such as joy sticks, mouse devices, and track balls. Such active input devices, however, are usually too limiting for effective use in projected video image display systems. For example, absolute positioning devices, such as conventional light pens, require the user to interact directly with the video 5 screen display of the monitor associated with the computer. This disrupts the spontaneity of the presentation as the user must repeatedly go back and forth between the computer monitor and the projection screen when the user desires to change the computer generated video image. In the case of relative positioning devices, such as track balls or mouse devices, the user must be skillful in manipulating the track ball or mouse in space. This can be awkward and usually requires use of both hands (i.e. one hand to hold the track ball or mouse pad in space and the other hand to manipulate the track ball or mouse to move the cursor). Such two handed operation also disrupts the natural flow of the presentation. Projected video image display systems which use a hand held light pen for enabling a user to shine a high intensity light directly onto a projection screen bearing a computer generated image to provide auxiliary information to a computer are known from the prior art. In such prior art systems, an optical sensing device is used for detecting the high intensity light aimed at the projected image on the screen. Signal processing apparatus are used for discriminating the changes in reflected light on the projected images from the computer generated images in order to input information interactively into the computer.
Exemplary systems are disclosed in U.S. Patent Number: 5,138,304, issued to Barry Bronson, and U.S. Patent Number: 5,181,015, issued to Roger L. Marshall et al.
A disadvantage with the above described known systems, however, is that the optical sensing device and related signal processing apparatus are typically of special purpose design and therefore expensive. Also, the requirement of the active input device, i.e. the light pen, is also disadvantageous as the light pen can be easily misplaced or dropped or damaged during a presentation or the batteries can die off.
It would be desirable to provide a more natural and unobtrusive way for enabling a user to interact with a computer of a projected video image display system in order to modify the projected image. It would also be desirable to provide a projected video image display system that does not require an active input device, such as a light pen, mouse device, track ball, etc., for inputting changes to a computer. In particular, it would be desirable to provide an interactive projected video image display system which would optically recognize a particular feature or a pattern created by the combination of the projected video image and an object introduced onto the projected image by the user in order to trigger a desired computer operation. Finally, it would also be desirable to provide such an interactive projected video display system which obviates the need for special use and expensive optical sensing components and related signal processing apparatus. SUMMARY OF THE INVENTION
Methods and apparatus which incorporate the desired features described above and which are effective to function to overcome the shortcomings of the prior art as described above constitute specific objects of this invention. The present invention provides an interactive projected image video display system which includes a computer for generating video images, a projection apparatus coupled to the video output port of the computer for projecting computer-generated video images onto a projection screen, and an optical recording device such as a conventional video camera, aimed at the projection surface for visually recording the projected image. The computer includes a frame grabber board or like image capturing means coupled to the video camera for capturing individual frames or series of frames from the video recording of the projected image. The captured images are then compared to the computer generated images to detect for the presence of an object having a particular feature or a recognized pattern/feature created by the combination of the introduced object and the video image. The detected object indicates an instruction to the computer, such as, for example, to modify the video image.
The object is introduced onto the projected image by a user during a presentation. The object may include any convenient pointing element having a recognizable feature, such as for example, a user's hand or fingers held in a particular orientation or angle, a pointer, a pen, etc. The means for detecting the pointing element and activating the computer to perform certain tasks is preferably embodied in a software program written in accordance with known image processing algorithms and techniques.
In a first mode of operation, the user may simulate "clicking" a button contained in the projected image. The button contains an instruction for signalling the computer to perform a certain task, such as, for example, advance to the next image, save an image to memory, etc. In this mode, the location of the button is predetermined and known to the computer. The user places the pointing element over the button on the projected image. During this time, the video camera is recording this activity. In accordance with the feature detection process, the frame grabber will grab a single frame containing the button and pointer. A search of a block region on the grabbed image corresponding to the button location is then made. Upon detection of the pointing element, an activation signal is generated and the computer will execute the instruction corresponding to the selected button.
In a simple application of the first mode of operation, a grey scale comparison technique is used to check the amount of shadowing present in the projected button by comparison with other portions of the projection surface where the light intensity is known or by comparison with the level of shadowing previously measured at the same button region. In this case, the user may "click" the projected button by simply holding his or her hand or some other object to block or cover the projected button for a predetermined length of time.
Optionally, the user may "click" the projected button by shinning a light source, such as a light pen, onto the projected button to induce a change in the light intensity on the projected image for capture by the video camera and comparison with the computer generated image.
The Touch detection process described in the above paragraph is algorithmically very simple, but is subject to "false triggering" by a number of events that might occur during the interactive session. Examples of these events include: (1) accidental or unintended obscuration by a user such as may occur when the user walks in front of the projected buttons; and (2) sudden changes in the local illumination of the projected area by external sources (e.g. doors being opened, room lights being turned on or off, etc.). In order to desensitize the system to such events, a technique requiring spatial localization and temporal consistency is used. The computer not only examines the target area of the buttons, but also examines other neighboring areas or regions that surround the target area.
If any of these surrounding areas or "invisible buttons" are also being obscured, then the touch detection is inhibited. The technique can be modified to allow for some of the invisible buttons to be obscured but not others, thus allowing the user to extend his or her arm thereacross to touch the desired button in the target area. We term this concept "differential" touch. Furthermore, a requirement for the obscuration configuration to be sustained for a fixed amount of time is also used to prevent triggering during the transit of a user across the field of view.
In a more advance application of the first mode of operation, the computer may be programmed in accordance with more advanced image processing techniques, such as feature extraction or pattern correlation, in order to look for special patterns or pattern changes present in the captured image as compared with the projected image. Such special patterns may include, for example, a cross formed by a pen being laid over a projected line contained in a projected button. The computer may also be programmed to look for other special features such as profile, size, color, etc., or special movements of the introduced object, such as, for example, a hand movement corresponding to making a check mark.
In a second mode of operation, the user may simulate marking points on the projected video image. In this mode, the desired mark locations are not already known to the computer. The user marks the location by pointing using a pointing element or pointer (having a pattern or posture recognizable by the computer). In a preferred embodiment of the invention, the user marks the points by touching the projected screen. The touching of a pointing element or pointer (having a pattern or feature recognizable by the computer) to the projection screen can be verified again by using known grey scale comparison techniques in order to detect when the pointer meets its own shadow. Alternatively, a second video camera oriented at a side angle relative to the projection screen may be used to verify that the pointer actually touches the surface of the projection screen.
In a more advanced application of the second mode of operation the pointing element is used in combination with the projected video to generate a predetermined feature for marking points on the image. For example, the projected video image may include a series of equally spaced vertical lines that cover the whole image. The user marks a point by generating a recognizable cross pattern by holding the pointing element over one of the vertical lines. More advanced features can be generated by combining more sophisticated patterns into the video image. In a further application of the second mode of operation, the user may use the pointer as a video cursor control means just like a mouse pointing device or light pen in order to enter hand written text or draw lines, curves or other graphical objects directly on the projected image. As another application of the second mode of operation, it is possible to implement a full simulation of a mouse that works with any off the shelf application that uses a regular mouse as an inputting device.
Other and further objects of the present invention will be apparent from the following description and claims and are illustrated in the accompanying drawings, which by way of illustration, show preferred embodiments of the present invention and the principles thereof and what are now considered to be the best modes contemplated for applying these principles. Other embodiments of the invention embodying the same or equivalent principles may be used and structural changes may be made as desired by those skilled in the art without departing from the present invention and the purview of the appended claims.
BRIEF DESCRIPTION OF THE DRAWINGS Fig. 1 is a diagrammatic view of an interactive projected video image display system in accordance with one embodiment of the present invention.
Fig. 2 is a diagrammatic view of a system similar to Fig. 1 in accordance with a second embodiment of the present invention.
Fig. 3 A is a flow chart which describes and shows the process of clicking a button for initiating a modification of the projected video image.
Fig. 3B is an enlarged diagrammatic view of a projected button contained in a projected video image and which shows an object having a particular feature (in this case a user's hand) being introduced over the projected button.
Fig. 3C is a schematic view of a projected button similar to Fig. 3B except in this case the projected button includes a projected vertical line and the object having a particular feature is a pointer. Fig. 4 is a schematic view of a projected video image of a menu window which illustrates the general pointing and positioning mode of operation of the present invention. Fig. 5A-6 illustrate a video mouse aspect of the invention.
Fig. 5 A diagrammatically shows a projected video image with a first pop up menu window activated and which contains a list of user selectable command items for performing conventional mouse operations.
Fig. 5B diagrammatically shows a projected video image with a second pop up menu window activated which contains a single user selectable command item for ending mouse operation. Figs. 6A-6B is a two part series depicting a flow chart which describes and shows the process for simulating video mouse operation on a projected video image.
DETAILED DESCRIPTION OF THE BEST MODE OF THE INVENTION:
The following detailed description illustrates the invention by way of example, not by way of limitation of the principles of the invention. This description will clearly enable one skilled in the art to make and use the invention, and describes several embodiments, adaptations, variations, alternatives and uses of the invention, including what we presently believe is the best mode of carrying out the invention.
Fig. 1 shows a generalized diagrammatic view of a projected image video display system 10 in accordance with one embodiment of the present invention. The system 10 includes a computer 12 for generating and displaying a video on monitor 16 (optional). As is conventional, the monitor 16 is coupled to the video output port 12a of the computer 12. Also coupled to the video output port 12a is an apparatus 18 for providing an enlarged projection of the video image generated by the computer 12. In a preferred embodiment, the apparatus 18 includes a conventional liquid crystal display (LCD) panel 20 and an overhead projector 22. The LCD panel 20 receives video image signal input from the computer 12 and generates light images which, in turn, are projected by the overhead projector 22 as a projected image 24 on a viewing surface, such as projection screen 26. The projection screen 26 may be any flat, and preferably light colored surface such as, for example, the ubiquitous white board found in most office conference rooms. In the example shown, two projected buttons 32, 34 are contained in the projected image 24.
While, the apparatus 18 is described as including a separate LCD panel 20 and overhead projector 22, it is understood that single machines capable of receiving and projecting video image data are becoming available and may be advantageously used in the present invention.
The system 10 further includes an optical sensing device, such as video camera 28a, for visually recording the projected image 24. The video camera 28a is coupled to the video input port 12b of the computer 12. The video input port 12b computer includes a conventional frame grabber board or equivalent image capturing means for selectively grabbing individual frames or groups of frames of the projected image as recorded by the video camera 28a. One or more additional video cameras, indicated generally at 28b,...,28n, may be included for providing enhanced feature detection capability to the system 10. This is discussed in more detail below with reference to the second mode of operation.
The computer 12 is programmed to compare the captured images with the original video image. Upon detection of certain recognized objects having a particular feature which are present in the captured image or upon detection of a recognizable pattern which has been formed by the combination of an introduced object and an existing feature contained in the projected image, a signal is generated to activate the computer to perform a specific instruction, such as for example, modify the video image, save a captured image to memory, etc.
Fig. 2 shows an alternate embodiment for the system 10 of the present invention. The system components common to both the first embodiment of Fig. 1 and the second embodiment of Fig. 2 are indicated by the same reference numerals. The second embodiment of Fig. 2 differs from the first embodiment of Fig. 1 in that video image 14 is not projected but instead is presented as an enlarged displayed image 24a on large screen video display 26a. The large screen video display 26a is coupled directly to the video output port 12a of the computer 12. In this example, two displayed buttons 32a, 34a are contained in the displayed image 24a.
As noted above, the preferred optical sensing device includes at least one video camera 28a which provides a continuous visual recording of the projected or displayed video image. However, it is understood that other optical sensing devices such as, for example, a charged coupled device (CCD) image sensor or camera may be used in place of the video camera with equally good results. As will be readily appreciated by those having ordinary skill in the art, appropriate signal processing apparatus or software may be coupled with a CCD image sensor or camera to convert the recorded images in a form usable by the computer for image comparison with the original known video-generated images.
The method and apparatus for providing interaction directly with a computer generated projected image and thereby simulating a video version of a computer input device will now be explained with reference to two example operational modes of the system 10.
MODE ONE: SELECTING ("CLICKING^) A BUTTON
With reference to Figs. 1, 2, and 3A - 3C, a first mode of operation for selecting or "clicking" a button contained in the projected or displayed image will be described. The projected image (Fig. 1) or displayed image (Fig. 2) contains "instructions" for the user. These "instructions" are associated with the projected or displayed buttons contained in the projected or displayed image, respectively. The user may simulate a mouse click by merely touching or placing an object, such as a pointer, over the desired projected or displayed button.
With reference to the generalized flow chart of Fig. 3C, the process for clicking a single button will be described. During a presentation, the user inputs information onto the projected or displayed image by either writing information or effecting changes by selecting a desired button. All during this time, the video camera 28a continuous records these changes. The frame grabber grabs a single frame of the video recording (step 40). Next, the computer reads (at step 42) a block region of the frame grabbed by the frame grabber corresponding to the location of a first button. This block region has a location which is known to the computer. Next, the block region is checked (step 44) for the presence of a an object having a particular feature or a recognizable pattern. If (at step 46) the object or recognized pattern is found in the block, the computer executes a specified control function associated with the selected button (step 50). If there is no match or if the object or pattern does not exist, then no computer operation is triggered. The above process is repeated for each grabbed frame.
For the case where two or more buttons are contained in the projected or displayed image. Process steps 42-46 are repeated until all the blocks corresponding to all the button locations in the projected or displayed image are checked for recognizable patterns before advancing to the next grabbed frame.
The feature or recognition algorithm may use any one of many well known algorithms including but not limited to binary methods, gray scale methods, etc. As will be appreciated by those having ordinary skill in the art, the particular method chosen will depend in large part on the type of pattern or feature to be recognized.
The image discrimination is preferably implemented in software in the form of a pattern correlation or feature extraction algorithm. Techniques for detecting features in digital images are well-known in the art. Exemplar ' references which disclose such techniques include: Digital Picture Processing. Azriel Rosenfeld and Avinash Kak, Vol. II, 2nd edition, Academic Press, 1982; Digital Image Processing. Rafael C. Gonzales,
Richard E. Woods, Addison- Wesley Publishing Co., 1992; and Computer Vision. Dana H. Ballard, Christopher M. Brown, Prentice-Hall, 1982.
Figs. 3B-3C show examples of objects having a particular feature being introduced over a projected button 32. In Fig. 3B, the button 32 represents the instruction "next". In operation, the user 30 introduces his or her hand 31 (or other recognized pointing element) over the button 32 to simulate clicking the button 32 to generate a new video image 14.
In Fig. 3C, the projected button 34 contains a projected vertical line 36. The user introduces a pointer such as a ruler 38 onto the button to form a cross. The cross represents a special pattem recognizable by the computer for carrying out a desired instruction. In the example given, the instruction "capture" signals the computer to save the captured image in a file in the memory of the computer.
Preferably, only those portions of the captured image which contain new or updated information are saved to memory. In other words, in accordance with an advantageous aspect of the invention, it is now possible to store just the difference between the capture image and the projected image. This is done to conserve memory space since graphic files are typically memory intensive. The captured image may be recalled at a later time to reappear as the projected image. Altemately, several captured images which have been stored in memory may be recalled at once, each superimposed upon the other, to provide a multi-layering effect. Such a multi-layering feature is particularly advantageous when used in combination with a projection screen in the form of a white board whereby one or more users may draw on the board with erasable ink markers of different colors. In this way, several versions of the projected image containing different changes in erasable ink can be captured, saved to memory, and then recalled for subsequent multi-layering onto each other to provide a readily understandable visual chronology of changes throughout the projected video presentation. MODE TWO: GENERAL POINTING AND POSITIONING
Fig. 4 illustrates the general pointing and positioning mode of operation of the invention. In this example, the projected image 24 includes a menu bar 50 containing menu items 52a, 52b,...,52n and a general positioning and pointing field 54. In this mode, the computer searches the entire grabbed image to detect the presence of an object anywhere on the image.
If the object is detected over one of the menu items 52a, 52b,..., 52n, the computer will perform the task associated with the selected menu item in accordance with the first mode of operation described above. If the object is detected in the general positioning and pointing field, the computer will generate a point or position a cursor at the location corresponding to where the user has placed the object. In the example shown, the computer is looking for an object in the form of two fingers 60 extended in the pointing position. In this case, two marks 56, 58 have been generated at locations on the projected image where the user's two fingers 60 have been placed.
The process of marking a point in the general positioning can be implemented by programming the computer to look for a special feature or features that are generated by the combination of the pointing element and the projected image. For example, as seen in
Fig. 4, the projected image may be provided with equally spaced vertical lines 53a-53m. The user can then quickly mark point at 59 by placing his or her finger 57 over the vertical line 53c to form a cross shape. The cross shape is readily recognizable by the computer using only a minimum of processing for detection of this simple pattem. The computer can be programmed to look for and recognize a more advanced feature or features by including a more sophisticated pattem in the video image.
In a further application of the second mode of operation, the above described feature recognition process may be implemented in a front end application program to simulate a mouse for accessing information from other currently running programs and for moving around within the different applications on the screen. Such application is possible in every computer with an operating system that support a mouse or a pen-like type interface. An example of such operating system is WINDOWS™ (WINDOWS™ is a trademark of Microsoft Corporation). This may be done, for example, by including within the front end application program, a mouse driver written in software, which will provide a series of pop up menu windows containing user selectable command items corresponding to basic mouse operations. The selection of a desired command item for carrying out a particular mouse operation may be done by placing a "recognized" object (e.g. a pointer, the user's hand or fingers held in a particular way, etc.) over the desired command item displayed in the pop up menu window in the projected image in accordance with the feature recognition process as described earlier.
The mouse driver of the present invention preferably simulates the following four basic operations of a mouse that are common to most conventional mouse drivers:
(1) "Mouse_Down" corresponding to the depression of a mouse button;
(2) "Mouse_Up" corresponding to the release of a mouse button; (3) "Double_Click" corresponding to where a mouse button has been depressed and released twice; and
(4) "Mouse_Move" corresponding to the movement of a mouse to a new location.
5 In accordance with conventional cursor control techniques, the mouse driver will send a message to the operating system indicating that a particular event has occurred when a command item corresponding to the mouse operation associated with that event has been selected from the associated pop up menu window. In addition, the (x,y) coordinates of the mouse at the time the event occurred will also be sent to the operating system. A single o click of the mouse is performed as a combination of "Mouse_Down" and "Mouse_Up" operations without moving the mouse. In this way, the user may simply select the desired mouse operation to simulate a single mouse click, a double mouse click, dragging the mouse, etc., in order to perform the desired function of the currently operating program, such as, for example, clicking, drawing a line, drawing a curve, drag and drop, etc. s Figs. 5A-5B illustrate, by way of example, a projected image 26 which depicts a screen display associated with a WINDOWS™ Paintbrush program. In the example shown, the projected image 26 is displaying the main Paintbrush window and shows the items in the main Paintbrush window including the top menu bar 62 (having command items such as "File", "Edit", etc.) the tool box 64, and the linesize box 66. o The process of the video mouse driver of the invention will be described in conjunction with Fig. 6a-6b, which show, in flow chart form, the steps of the video mouse driver as it relates to a currently running WINDOWS™ application program. It is understood that the process of the invention is not limited to just WINDOWS™ type applications but may also be implemented in any application which supports a mouse, a 5 pen or like computer input device.
The video mouse driver starts (step 100) when a video image generated by the computer 12 is projected by apparatus 18 onto projection surface or screen 26 (see e.g. Fig. 1). The computer then determines (in step 102) whether the user has pointed at the projected image. This is done using the feature/pattern detection process as previously described.
When the driver detects that the user has pointed at the screen it sends WINDOWS™ (in step 104) the Mouse_Move message along with the coordinates of the new point. WINDOWS™ reacts by moving the cursor 61 of the application. The exact shape of the cursor depends on the specific application.
The driver then checks (in step 106) whether the user continues to point at the projected or displayed video image. If the answer is negative the program returns to its starting point (step 100). If the answer is positive, i.e., the user is still pointing at the projected or displayed video image, the driver checks (in step 108) whether the user has moved the pointer. If the user points at a different place, the program returns to step 104 to update the location of the cursor. If the user keeps pointing at the same location for a predetermined period of time, e.g., 2 seconds (step 110), then a first pop up menu window 68 is projected or displayed adjacent to the cursor position 61 (step 112).
The first pop up menu 68 contains the four command items: NOP 70 (which stands for "no operation"), CLICK 72, DOWN 74, and DOUBLE CLICK 76. Adjacent each command item 70, 72, 74, and 76 is a respective selection button 71, 73, 75, and 77. As before, the buttons may be selected by the recognized feature/pattern detection process as described above with reference to Figs. 3B-3C and 4.
The computer then checks (in step 114) for a selection of command item contained in the first pop up window 68 allowing the user to make such a selection within a predetermined period of time, e.g., 2 seconds (step 116). If no selection is made during the predetermined period of time, the first pop up menu 68 disappears and the program returns to its starting point. If a selection is made, the selected command item is performed.
The driver discriminates between two types of selections (step 120). The first selection is the command DOWN, which requires special processing shown in Fig. 6B.
The second selection is for the commands NOP, CLICK, and DOUBLE CLICK. For the latter selection (step 122), the first pop up menu 68 disappears, the driver executes the command item by sending an appropriate message to the operating system, and the program returns to it starting point (step 100) after a small delay (e.g. 2 seconds). The small delay is necessary in order to allow the user enough time to move the pointer to a new desired location.
The process by which the driver executes the command in the menu depends on the specific command. For example, to simulate a single mouse click, the user will select button 73 corresponding to the CLICK command item 72. In this case the mouse driver sends to the operating system the messages the "Mouse_Down" followed by "Mouse_Up", which the WINDOWS™ recognizes as a mouse click. In addition, the mouse driver, sends the (x,y) coordinates of the point in order to update the cursor location. The click triggers an event that will depend on the particular application being run at the time, e.g., drawing a curve.
To stimulate a double mouse click, the user will select button 77 corresponding to the DOUBLE CLICK command item 76. This action signals the mouse driver to send to WINDOWS™ the four messages "Mouse_Down", "MouseJ p", "Mouse_Down" and "Mouse_Up" within a predetermined time frame. The user also has the option of selecting button 71 corresponding to the NOP (no operation) command item 70. This will make the first pop up window 68 disappears without affecting the cursor location, and causes the driver to go back to the starting point (step 100).
Referring now to Fig. 6b, when a user chooses the button 75 corresponding to the Down command 74, the driver response by sending a Mouse_Down message to the operating system (step 126) with the additional information of the coordinates of the point. In addition, the first pop up menu window 68 disappears and the user is given a short delay (e.g., 2 seconds) to return the pointer from the location of the button 75 to the location of the cursor 61. Once the Mouse_Down command message has been received, the computers resumes following the movement of the user's fingers 60. During this time, a series of updated "Mouse_Move" messages are sent to the operating system along with the updated (x,y) coordinates (step 128). This process continues as long as the user changes the pointing location (steps 128-130). The moment the user stops moving the pointer, he or she has two options (step 132). One option is to stop pointing, i.e., by withdrawing his or her finger(s) from the projected or displayed image. This will cause the driver to send to WINDOWS™ a MouseJQp command, and then after a small delay, the program will return to the beginning (step 140). Alternatively, if the user is still pointing to the same location, a second pop up menu window 78 (see e.g., Fig. 5b) is displayed. The second pop up menu window 78 preferably contains only the single command item Up 80 along with corresponding selection button 81. If the user selects the UP command item 80 (i.e., the user places his or her fingers 60 onto the corresponding selection button 81), then the mouse driver sends the "MouseJJp" message to WINDOWS™, and the application (in this case, the
WINDOWS™ Paintbrush program) responds accordingly (step 140). The process then returns to the start after a predetermined delay. If (at step 136) a selection is not made by the user within a predetermined period of time, e.g., 2 seconds (step 138), then the second pop up menu 78 disappears and the user may continue to move his or her finger(s) 60 along the projected or displayed image 26 as before. This will result in new "Mouse_Move" messages being sent to WINDOWS™ until the user stops finger movement after which steps 132-138 are repeated. SELF TEST
The system may include means for performing a self test to verify the calibration of the system. For example, the system may have stored in memory, two images which are identical except that the second of the two images contains an object having a particular feature, such as a pointer, located over a button contained in the image. This particular button is unoccupied in the first image. Upon start up or at any other point in time, the system projects the second image containing the pointer located over the particular button and compares this image to the first image. The self test mode will be successful if the computer identifies the particular button. CALIBRATION
A calibration algorithm may also be provided to calibrate the respective X,Y axis systems of the projected image and the image captured by the camera 28a in order to maintain focus, color balance, distortion and other parameters of the projected video image. This may be done by projecting predetermined images, such as comer markings 27 (see Fig. 1), having known locations, scale dimensions and color or light intensities, and comparing the predetermined comer markings 27 to the corresponding comer markings in the captured image.
In accordance with a preferred embodiment of the invention, the calibration algorithm serves two purposes: (1) the calibration algorithm "conditions" the captured images by removing undesirable light reflections, stray marks, cracks or like irregularities on the viewing surface which have an adverse effect on the capture image quality (i.e., image distortion, image resolution, image contrast, etc.); and (2) the calibration algorithm is used to determine an alignment function for ensuring the alignment of subsequently captured images with the projected images.
The conditioning portion of the calibration algorithm is carried out by first suppressing or "blanking" the video image generated by the computer such that a blank image is illuminated and projected onto the viewing surface. This provides a first point of reference or "reference image" which enables the system to gather information about particular irregularities on the viewing surface which adversely effect image quality, such as, for example, shadows and/or bright reflections on the viewing surface due to uneven surface conditions and ambient lighting, permanent stains, stray marks or cracks on the viewing surface, etc.
The system then captures the blank projected image as a first reference image. Next, a plurality of reference markings, e.g. crosses or like indicia, are projected onto the viewing surface and are captured by the frame grabber as one or more additional reference images. The X and Y coordinates for the projected reference markings are pre-specified by the computer. It should be noted that the reference markings can be projected at random points on the viewing surface within the boundaries of the projected image and not necessarily just at the four comer regions as is typically done in accordance with prior art calibration techniques. The captured reference images containing the reference markings are then compared with the reference markings generated by the computer. Using this information, the computer then determines an alignment function which represents the relative displacement, in X and Y coordinates values, between the reference markings contained in the captured additional references images and the reference markings in the computer generated video image. Thus, the alignment function, once determined, can be used to ensure alignment of the captured images with the projected images.
The above calibration algorithm may be implemented automatically any time the individual apparatus components of the system have been bumped or moved slightly during use, or when changes in the ambient light conditions occur, or when changes occur in any other parameter which effects image quality. In addition, the calibration algorithm may be advantageously implemented in the situation where a user has made annotations to the projected image by making physical changes on the viewing surface. For example, in the situation where the viewing surface is a conventional white board, the user may want to make edits or corrections to the projected image by drawing directly on the white board. The user may initiate the capture command and save the annotations made to a particular projected image. Since the computer already has a record of the projected image stored in memory, only the annotations are saved to memory. This avoids image degradation caused by the recapture of image information which is already known to the computer. This also advantageously allows for the "multi-layering" of stored annotations (the annotations may be captured and saved as separate data) which can then be selectively projected either individually, in groups, or all together onto the projected image. The . alignment function established by the above described calibration algorithm can be implemented to ensure proper alignment of the saved annotations with any particular video image generated by the computer.
It should be understood that various modifications within the scope of this invention can be made by one of ordinary skill in the art without departing from the spirit thereof.
We therefore wish our invention to be defined by the scope of the appended claims as broadly as the prior art will permit, and in view of the specification if need be. What is claimed is:

Claims

1. A method of providing interaction with a projected video image comprising the steps of: using a computer to generate a video image; projecting the video image onto a viewing surface; introducing an object over the projected image on the viewing surface for a predetermined minimum period of time; optically recording the projected image; capturing an image from the optical recording of the projected image which contains the object; comparing the captured image to the computer generated video image to detect the presence of the object; and generating a control signal for actuating the computer in response to the presence of the object.
2. The method according to claim 1 wherein the projected image contains at least one button and the image comparison step further comprises: reading at least one block portion of the captured image corresponding to the location of said at least one button; checking said at least one block portion for the presence of the object; and verifying that other block portions disposed adjacent said at least one block portion do not differ substantially from their corresponding block portions contained in the projected image.
3. The method according to one of claims 1 or 2 wherein the step of generating a control signal further comprises the steps of: triggering a computer operation in response to the location of the object being introduced over the projected image; and wherein the step of modifying the video image further comprises generating a cursor corresponding to the location of the object being introduced over the projected image.
4. The method according to claim 3 wherein an application program, is currently running on the computer, said application program is responsive to a driver for a computer input device and wherein said method further comprises the steps of: selectively generating pop up menu windows adjacent a cursor on the projected image 5 in response to user positioning of an object over the projected image, said menu windows contain user selectable buttons which represent driver options associated with a computer input device for triggering the operating system of the computer to send a desired computer input control message to said application program in order to perform a desired computer operation; and l o selecting one of said buttons by placing an object thereover for a predetermined period of time.
5. The method according to claim 1 wherein the step of comparing the captured image to the computer generated image includes detecting a recognizable pattem created by the combination of the introduced object and the projected image.
6. The method according to one of claims 1 to 5 which includes the step of calibrating captured images to computer generated images, said step of calibrating including the steps of:
5 temporarily suppressing the video image generated by the computer; projecting a blank illuminated image onto the viewing surface; capturing the blank illuminated image as a first reference image, the first reference image containing information on light reflections, stray marks and like irregularities on the viewing surface; and comparing the first reference image with subsequently captured images in order to remove undesirable light reflections, stray marks and like irregularities from the subsequently captured images.
7. The method according to claim 6 wherein the step of calibrating further includes the steps of: projecting a plurality of reference markings generated by the computer onto the viewing surface, the reference markings having X and Y coordinates which are specified by the computer ; capturing the projected reference markings as at least one additional reference image; comparing said at least one captured additional reference image with the reference markings generated by the computer; determining an alignment function which represents X and Y coordinate displacement between the projected image and the capture image; and using the determined alignment function to compare and align captured images with video images generated by the computer.
8. The method according to claim 7 which includes the step of self testing to verify calibration of projected images with captured images, said step of self testing including the steps of: storing a first image and a second image, said first and second stored images each having a common predetermined location associated therewith, said second stored image further including a reference object disposed over its respective predetermined location; projecting said second stored image; capturing the second stored image; and comparing the captured second stored image with said first stored image to verify that the reference object contained in the captured second stored image is in alignment with the predetermined location for the reference object in said first stored image.
9. The method according to one of claims 1 to 7 whereby a user desires to annotate a projected image on the viewing surface and which includes the steps of: saving a first computer generated image; projecting said first computer generated image onto said viewing surface; suppressing the projection of said first computer generated image such that only user made annotations remain on the viewing surface: capturing the user made annotations on the viewing surface; and storing the captured user made annotations for subsequent display of the stored captured user made annotations in alignment with said first computer generated image.
10. The method according to one of claims 1 to 9 wherein the step of generating a control signal further comprises the steps of: saving at least a portion of the captured image to a memory in the computer.
11. An interactive projected video image display system comprising: means for generating a video image; means for projecting the video image; means for optically recording the projected image; means for capturing an image from the optical recording of the projected image; image discrimination means for comparing the captured image with the generated video image and for detecting the presence of an object introduced onto the projected image; and means responsive to said image discrimination means for triggering a computer operation when said introduced object is detected.
12. The invention according to claim 11 which further includes: means for automatically calibrating deformation of the captured image with respect to the projected image.
PCT/US1995/007487 1994-06-15 1995-06-13 Interactive projected video image display system WO1995034881A1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
JP8502407A JPH09512656A (en) 1994-06-15 1995-06-13 Interactive video display system
AU29440/95A AU2944095A (en) 1994-06-15 1995-06-13 Interactive projected video image display system
DE69530395T DE69530395T2 (en) 1994-06-15 1995-06-13 INTERACTIVE PROJECTED VIDEO DISPLAY SYSTEM
EP95925249A EP0771460B1 (en) 1994-06-15 1995-06-13 Interactive projected video image display system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US08/259,887 US5528263A (en) 1994-06-15 1994-06-15 Interactive projected video image display system
US08/259,887 1994-06-15

Publications (1)

Publication Number Publication Date
WO1995034881A1 true WO1995034881A1 (en) 1995-12-21

Family

ID=22986852

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US1995/007487 WO1995034881A1 (en) 1994-06-15 1995-06-13 Interactive projected video image display system

Country Status (6)

Country Link
US (1) US5528263A (en)
EP (1) EP0771460B1 (en)
JP (1) JPH09512656A (en)
AU (1) AU2944095A (en)
DE (1) DE69530395T2 (en)
WO (1) WO1995034881A1 (en)

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
EP0720109A1 (en) * 1994-12-27 1996-07-03 AT&T Corp. Multimedia program indexing system and method
GB2315859A (en) * 1996-07-29 1998-02-11 Mitel Corp Input device simulating touch screen
DE19714109A1 (en) * 1997-04-05 1998-10-08 Moser Hans Video cabin for film presentation system
GB2370395A (en) * 2000-12-19 2002-06-26 Ubinetics Ltd Display apparatus
US6788289B2 (en) 2000-02-08 2004-09-07 Seiko Epson Corporation System and method of pointed position detection system, projector and program
US6829394B2 (en) 2000-02-22 2004-12-07 Seiko Epson Corporation System and method of pointed position detection, presentation system, and program
US7034807B2 (en) 2000-02-21 2006-04-25 Siemens Aktiengesellschaft Method and configuration for interacting with a display visible in a display window
WO2007088430A1 (en) * 2006-02-01 2007-08-09 Nokia Corporation System, device, method and computer program product for using a mobile camera for controlling a computer
EP1821183A1 (en) * 2004-10-05 2007-08-22 Nikon Corporation Electronic device
AT503008B1 (en) * 2005-12-16 2008-09-15 Dobler & Stampfl Audite Oeg INTERACTIVE OPTICAL SYSTEM AND METHOD FOR EXTRACTING AND ASSESSING INTERACTIONS IN AN OPTICAL SYSTEM
WO2009009338A2 (en) * 2007-07-11 2009-01-15 Igt Processing input for computing systems based on the state of execution
WO2009092126A1 (en) * 2008-01-25 2009-07-30 Stumpfl, Reinhold Interactive multimedia presentation apparatus
EP2177976A2 (en) * 2008-10-15 2010-04-21 LG Electronics, Inc. Mobile terminal with image projection
EP2191350A1 (en) * 2007-09-04 2010-06-02 Canon Kabushiki Kaisha Image projection apparatus and control method for same
EP2296080A3 (en) * 2000-07-05 2011-06-29 SMART Technologies ULC Camera-based touch system
EP2383609A1 (en) * 2010-04-30 2011-11-02 Samsung Electronics Co., Ltd. Interactive display apparatus and operating method thereof
WO2013151645A1 (en) * 2012-04-02 2013-10-10 Sony Computer Entertainment Inc. Detection of interaction with virtual object from finger color change
US10417515B2 (en) 2017-01-09 2019-09-17 Microsoft Technology Licensing, Llc Capturing annotations on an electronic display

Families Citing this family (286)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6141000A (en) * 1991-10-21 2000-10-31 Smart Technologies Inc. Projection display system with touch sensing on screen, computer assisted alignment correction and network conferencing
US6281878B1 (en) 1994-11-01 2001-08-28 Stephen V. R. Montellese Apparatus and method for inputing data
JP3153720B2 (en) * 1994-12-20 2001-04-09 富士通株式会社 Video presentation system
JPH08242417A (en) * 1995-03-01 1996-09-17 Olympus Optical Co Ltd Information reproduction system and information recording medium
US5727141A (en) * 1995-05-05 1998-03-10 Apple Computer, Inc. Method and apparatus for identifying user-selectable regions within multiple display frames
US5684968A (en) * 1995-06-16 1997-11-04 International Business Machines Corporation Method and apparatus for processing multiple images concurrently
US6176782B1 (en) * 1997-12-22 2001-01-23 Philips Electronics North America Corp. Motion-based command generation technology
WO1997046995A1 (en) * 1996-06-07 1997-12-11 Haworth, Inc. Table with internal projection system
US5933134A (en) * 1996-06-25 1999-08-03 International Business Machines Corporation Touch screen virtual pointing device which goes into a translucent hibernation state when not in use
US5831614A (en) * 1996-07-01 1998-11-03 Sun Microsystems, Inc. X-Y viewport scroll using location of display with respect to a point
US5992758A (en) * 1996-09-23 1999-11-30 Agro-Mack Enterprises Ltd. Proximity detector for ground-based implements
US5973672A (en) * 1996-10-15 1999-10-26 Raytheon Company Multiple participant interactive interface
US5933135A (en) * 1996-10-24 1999-08-03 Xerox Corporation Pen input device for high resolution displays
US6070247A (en) * 1996-12-03 2000-05-30 Smart Technologies Inc. Method and apparatus for controlling power to a multi-media conferencing system using any one of a system power switch and a computer
US5990865A (en) * 1997-01-06 1999-11-23 Gard; Matthew Davis Computer interface device
KR100288976B1 (en) * 1997-01-08 2001-05-02 윤종용 Method for constructing and recognizing menu commands of television receiver
DE19708240C2 (en) * 1997-02-28 1999-10-14 Siemens Ag Arrangement and method for detecting an object in a region illuminated by waves in the invisible spectral range
US5957559A (en) * 1997-04-29 1999-09-28 Virtek Vision Corporation Laser scanned menu
US6392665B1 (en) * 1997-05-29 2002-05-21 Sun Microsystems, Inc. Capture mechanism for computer generated motion video images
US5864334A (en) * 1997-06-27 1999-01-26 Compaq Computer Corporation Computer keyboard with switchable typing/cursor control modes
KR19990011180A (en) 1997-07-22 1999-02-18 구자홍 How to select menu using image recognition
WO1999007153A1 (en) * 1997-07-31 1999-02-11 Reality Fusion, Inc. Systems and methods for software control through analysis and interpretation of video information
DE19734511A1 (en) * 1997-08-08 1999-02-11 Siemens Ag Communication device
US6750848B1 (en) * 1998-11-09 2004-06-15 Timothy R. Pryor More useful man machine interfaces and applications
DE19843902A1 (en) * 1997-09-26 1999-04-01 Denso Corp Picture information display system e.g. for use in car show rooms
US6388654B1 (en) * 1997-10-03 2002-05-14 Tegrity, Inc. Method and apparatus for processing, displaying and communicating images
GB2330669B (en) * 1997-10-24 2002-09-11 Sony Uk Ltd Data processing
DE19806024A1 (en) * 1998-02-13 1999-08-19 Siemens Nixdorf Inf Syst Method for monitoring a device operation process and self-service device monitored with it
JPH11239237A (en) * 1998-02-24 1999-08-31 Canon Inc Image display device
US6043805A (en) * 1998-03-24 2000-03-28 Hsieh; Kuan-Hong Controlling method for inputting messages to a computer
US6512507B1 (en) * 1998-03-31 2003-01-28 Seiko Epson Corporation Pointing position detection device, presentation system, and method, and computer-readable medium
US6684211B1 (en) * 1998-04-01 2004-01-27 Planetweb, Inc. Multimedia communication and presentation
EP1004070A2 (en) * 1998-05-27 2000-05-31 Koninklijke Philips Electronics N.V. A system for image generation with screen means and means for image-wise mixing with a received user image
US6088018A (en) * 1998-06-11 2000-07-11 Intel Corporation Method of using video reflection in providing input data to a computer system
DE19830968A1 (en) * 1998-07-10 2000-01-13 Siemens Ag device
GB9816463D0 (en) * 1998-07-30 1998-09-23 Soda Creative Technologies Lim Computer communication device
US6266048B1 (en) * 1998-08-27 2001-07-24 Hewlett-Packard Company Method and apparatus for a virtual display/keyboard for a PDA
DE19845030A1 (en) * 1998-09-30 2000-04-20 Siemens Ag Imaging system for reproduction of medical image information
US6690357B1 (en) * 1998-10-07 2004-02-10 Intel Corporation Input device using scanning sensors
JP2000214992A (en) 1999-01-21 2000-08-04 Mitsubishi Electric Corp Direction indication device, direction estimating method and computer readable recording medium which records program on the method
US7212197B1 (en) * 1999-02-01 2007-05-01 California Institute Of Technology Three dimensional surface drawing controlled by hand motion
WO2000058933A1 (en) * 1999-03-17 2000-10-05 Tegrity, Inc. Method and apparatus for visual pointing and computer control
EP1161720B1 (en) * 1999-03-17 2002-09-18 Siemens Aktiengesellschaft Method and device for effecting an interaction with a user
JP4176230B2 (en) * 1999-03-19 2008-11-05 株式会社ルネサステクノロジ Coordinate position control system, coordinate position control method, and computer-readable recording medium storing a program for causing a computer to execute the method
US6292171B1 (en) 1999-03-31 2001-09-18 Seiko Epson Corporation Method and apparatus for calibrating a computer-generated projected image
DE19918633A1 (en) * 1999-04-23 2000-11-02 Siemens Ag Procedure to reduce extraneous light on object to be recognized for gesture operated touchscreen
US6614422B1 (en) * 1999-11-04 2003-09-02 Canesta, Inc. Method and apparatus for entering data using a virtual input device
US6295051B1 (en) * 1999-06-02 2001-09-25 International Business Machines Corporation Intelligent boundless computer mouse system
WO2000075860A1 (en) * 1999-06-08 2000-12-14 Soffix, S.R.L. Electronic writing/display apparatus and respective method of operation
US6346933B1 (en) 1999-09-21 2002-02-12 Seiko Epson Corporation Interactive display presentation system
DE19948896C1 (en) * 1999-10-11 2001-01-18 Siemens Ag Object detecting arrangement in area illuminated by light in motor vehicle
JP4044255B2 (en) * 1999-10-14 2008-02-06 富士通株式会社 Information processing apparatus and screen display method
DE19951320A1 (en) * 1999-10-25 2001-04-26 Siemens Ag Arrangement for detecting an object on and/or in front of a surface comprises a receiving device and a computer to determine reflected waves from the object
DE19951322A1 (en) * 1999-10-25 2001-04-26 Siemens Ag Interactive arrangement of a virtual touch screen type has a wireless connection between an operating module and the virtual touch screen operating surface to improve resistance to vandalism
JP4052498B2 (en) 1999-10-29 2008-02-27 株式会社リコー Coordinate input apparatus and method
US6710770B2 (en) * 2000-02-11 2004-03-23 Canesta, Inc. Quasi-three-dimensional method and apparatus to detect and localize interaction of user-object and virtual transfer device
US20030174125A1 (en) * 1999-11-04 2003-09-18 Ilhami Torunoglu Multiple input modes in overlapping physical space
US6368002B1 (en) * 1999-11-29 2002-04-09 Xerox Corporation Parking mechanism for storing and exchanging end effectors used in a system for performing actions over vertical surfaces
US6457792B1 (en) * 1999-11-29 2002-10-01 Xerox Corporation Method for effecting actions over vertical surfaces
JP4332964B2 (en) * 1999-12-21 2009-09-16 ソニー株式会社 Information input / output system and information input / output method
JP2001184161A (en) 1999-12-27 2001-07-06 Ricoh Co Ltd Method and device for inputting information, writing input device, method for managing written data, method for controlling display, portable electronic writing device, and recording medium
AU2001227797A1 (en) * 2000-01-10 2001-07-24 Ic Tech, Inc. Method and system for interacting with a display
WO2001054110A1 (en) * 2000-01-18 2001-07-26 The Trustees Of The University Of Pennsylvania Vision-based human computer interface system
GB2361146A (en) * 2000-02-04 2001-10-10 Roke Manor Research User interface for a mobile telecommunications system
JP3826659B2 (en) * 2000-03-27 2006-09-27 セイコーエプソン株式会社 Projection display system and projection display device
US6624833B1 (en) * 2000-04-17 2003-09-23 Lucent Technologies Inc. Gesture-based input interface system with shadow detection
US6753927B2 (en) * 2000-05-16 2004-06-22 Texas Instruments Incorporated Television interface for handheld calculator for enhanced television display
GB0012275D0 (en) * 2000-05-22 2000-07-12 Secr Defence Brit Three dimensional human computer interface
EP1316055A4 (en) * 2000-05-29 2006-10-04 Vkb Inc Virtual data entry device and method for input of alphanumeric and other data
ES2396587T3 (en) * 2000-07-05 2013-02-22 Smart Technologies Ulc Touch camera method and device
ATE365942T1 (en) 2000-09-07 2007-07-15 Canesta Inc QUASI-THREE-DIMENSIONAL METHOD AND DEVICE FOR DETECTING AND LOCALIZING THE INTERACTION OF USER AND OBJECT AND VIRTUAL TRANSFER DEVICE
NO20004514A (en) * 2000-09-11 2002-01-14 Njoelstad Tormod Drawing, writing and pointing tools for computer-based presentations and office work
JP2002118738A (en) * 2000-10-10 2002-04-19 Canon Inc Image display device and method, information processing unit using the image display device, and storage medium
US6774889B1 (en) * 2000-10-24 2004-08-10 Microsoft Corporation System and method for transforming an ordinary computer monitor screen into a touch screen
US20020180726A1 (en) * 2000-11-06 2002-12-05 Jianbo Shi Paper-based remote sketching system
US6704000B2 (en) * 2000-11-15 2004-03-09 Blue Iris Technologies Method for remote computer operation via a wireless optical device
US6901407B2 (en) 2001-01-12 2005-05-31 Rick D. Curns System and method for updating project management scheduling charts
US6731330B2 (en) * 2001-01-30 2004-05-04 Hewlett-Packard Development Company, L.P. Method for robust determination of visible points of a controllable display within a camera view
US20020136455A1 (en) * 2001-01-31 2002-09-26 I-Jong Lin System and method for robust foreground and background image data separation for location of objects in front of a controllable display within a camera view
US6542087B2 (en) 2001-01-31 2003-04-01 Hewlett-Packard Company System and method for extracting a point of interest of an object in front of a computer controllable display captured by an imaging device
US6895520B1 (en) 2001-03-02 2005-05-17 Advanced Micro Devices, Inc. Performance and power optimization via block oriented performance measurement and control
US6642918B2 (en) 2001-04-23 2003-11-04 Canon Kabushiki Kaisha Control of digital projection system
EP1689172B1 (en) * 2001-06-05 2016-03-09 Microsoft Technology Licensing, LLC Interactive video display system
US8300042B2 (en) * 2001-06-05 2012-10-30 Microsoft Corporation Interactive video display system using strobed light
US8035612B2 (en) * 2002-05-28 2011-10-11 Intellectual Ventures Holding 67 Llc Self-contained interactive video display system
US7259747B2 (en) * 2001-06-05 2007-08-21 Reactrix Systems, Inc. Interactive video display system
US20040201698A1 (en) * 2001-06-08 2004-10-14 Keenan Vaughn E. Camera-based system for capturing images of a target area
US7701437B2 (en) * 2001-07-03 2010-04-20 Koninklijke Philips Electronics N.V. Interactive display and method of displaying a message
US7830362B2 (en) 2001-07-05 2010-11-09 Michael Cain Finley Laser and digital camera computer pointer device system
JP4250884B2 (en) * 2001-09-05 2009-04-08 パナソニック株式会社 Electronic blackboard system
US20040205477A1 (en) * 2001-09-13 2004-10-14 I-Jong Lin System for recording a presentation
US7356763B2 (en) * 2001-09-13 2008-04-08 Hewlett-Packard Development Company, L.P. Real-time slide presentation multimedia data object and system and method of recording and browsing a multimedia data object
WO2003026153A1 (en) * 2001-09-20 2003-03-27 Exo-Brain, Inc. Input-output device with universal phone port
JP2003108305A (en) * 2001-09-28 2003-04-11 Fuji Photo Optical Co Ltd Presentation system
US20030067537A1 (en) * 2001-10-04 2003-04-10 Myers Kenneth J. System and method for three-dimensional data acquisition
US20030067538A1 (en) * 2001-10-04 2003-04-10 Myers Kenneth J. System and method for three-dimensional data acquisition
EP1302891A1 (en) * 2001-10-12 2003-04-16 Siemens Aktiengesellschaft Apparatus for the detection and display of motion
DE10163648A1 (en) * 2001-12-21 2003-07-03 Klaus-Dieter Frank Interactive projection and input system
EP1456806B1 (en) * 2001-12-21 2005-08-17 BRITISH TELECOMMUNICATIONS public limited company Device and method for calculating a location on a display
US6982697B2 (en) * 2002-02-07 2006-01-03 Microsoft Corporation System and process for selecting objects in a ubiquitous computing environment
US6990639B2 (en) * 2002-02-07 2006-01-24 Microsoft Corporation System and process for controlling electronic components in a ubiquitous computing environment using multimodal integration
CA2479607A1 (en) * 2002-03-22 2003-10-02 British Telecommunications Public Limited Company Interactive video system
US7348963B2 (en) * 2002-05-28 2008-03-25 Reactrix Systems, Inc. Interactive video display system
US7710391B2 (en) 2002-05-28 2010-05-04 Matthew Bell Processing an image utilizing a spatially varying pattern
US20040070616A1 (en) * 2002-06-02 2004-04-15 Hildebrandt Peter W. Electronic whiteboard
US20030226968A1 (en) * 2002-06-10 2003-12-11 Steve Montellese Apparatus and method for inputting data
JP4326753B2 (en) * 2002-06-14 2009-09-09 株式会社リコー Video information indexing support system, program, and storage medium
US7134080B2 (en) * 2002-08-23 2006-11-07 International Business Machines Corporation Method and system for a user-following interface
US7671843B2 (en) * 2002-11-12 2010-03-02 Steve Montellese Virtual holographic input method and device
US6954197B2 (en) 2002-11-15 2005-10-11 Smart Technologies Inc. Size/scale and orientation determination of a pointer in a camera-based touch system
AU2003301043A1 (en) 2002-12-13 2004-07-09 Reactrix Systems Interactive directed light/sound system
US7576727B2 (en) * 2002-12-13 2009-08-18 Matthew Bell Interactive directed light/sound system
JP2004199299A (en) * 2002-12-18 2004-07-15 Casio Comput Co Ltd Handwritten information recording method and projection recording device
DE10260305A1 (en) * 2002-12-20 2004-07-15 Siemens Ag HMI setup with an optical touch screen
US8456447B2 (en) 2003-02-14 2013-06-04 Next Holdings Limited Touch screen signal processing
US8508508B2 (en) 2003-02-14 2013-08-13 Next Holdings Limited Touch screen signal processing with single-point calibration
US7629967B2 (en) 2003-02-14 2009-12-08 Next Holdings Limited Touch screen signal processing
US7532206B2 (en) 2003-03-11 2009-05-12 Smart Technologies Ulc System and method for differentiating between pointers used to contact touch surface
US8745541B2 (en) * 2003-03-25 2014-06-03 Microsoft Corporation Architecture for controlling a computer using hand gestures
US7665041B2 (en) * 2003-03-25 2010-02-16 Microsoft Corporation Architecture for controlling a computer using hand gestures
TWI241400B (en) * 2003-03-27 2005-10-11 Primax Electronics Ltd A testing method used on a display
US20040196400A1 (en) * 2003-04-07 2004-10-07 Stavely Donald J. Digital camera user interface using hand gestures
US7411575B2 (en) * 2003-09-16 2008-08-12 Smart Technologies Ulc Gesture recognition method and touch system incorporating the same
US7274356B2 (en) 2003-10-09 2007-09-25 Smart Technologies Inc. Apparatus for determining the location of a pointer within a region of interest
EP1676442A2 (en) 2003-10-24 2006-07-05 Reactrix Systems, Inc. Method and system for managing an interactive video display system
US7536032B2 (en) 2003-10-24 2009-05-19 Reactrix Systems, Inc. Method and system for processing captured image information in an interactive video display system
US7420540B2 (en) * 2003-12-01 2008-09-02 Olbrich Craig A Determining positioning and/or relative movement of graphical-user interface element based on display images
US7355593B2 (en) 2004-01-02 2008-04-08 Smart Technologies, Inc. Pointer tracking across multiple overlapping coordinate input sub-regions defining a generally contiguous input region
JP2005236421A (en) * 2004-02-17 2005-09-02 Aruze Corp Image display system
US20050227217A1 (en) * 2004-03-31 2005-10-13 Wilson Andrew D Template matching on interactive surface
US20060020486A1 (en) * 2004-04-02 2006-01-26 Kurzweil Raymond C Machine and method to assist user in selecting clothing
JP4708422B2 (en) * 2004-04-15 2011-06-22 ジェスチャー テック，インコーポレイテッド Tracking of two-hand movement
US7394459B2 (en) 2004-04-29 2008-07-01 Microsoft Corporation Interaction between objects and a virtual environment display
US7460110B2 (en) * 2004-04-29 2008-12-02 Smart Technologies Ulc Dual mode touch system
US7538759B2 (en) 2004-05-07 2009-05-26 Next Holdings Limited Touch panel display system with illumination and detection provided from a single edge
US8120596B2 (en) 2004-05-21 2012-02-21 Smart Technologies Ulc Tiled touch system
US20050264545A1 (en) * 2004-05-27 2005-12-01 Walker Ray A Method and system for determining the location of a movable icon on a display surface
US7787706B2 (en) * 2004-06-14 2010-08-31 Microsoft Corporation Method for controlling an intensity of an infrared source used to detect objects adjacent to an interactive display surface
US7593593B2 (en) * 2004-06-16 2009-09-22 Microsoft Corporation Method and system for reducing effects of undesired signals in an infrared imaging system
US7519223B2 (en) 2004-06-28 2009-04-14 Microsoft Corporation Recognizing gestures and using gestures for interacting with software applications
US7724242B2 (en) * 2004-08-06 2010-05-25 Touchtable, Inc. Touch driven method and apparatus to integrate and display multiple image layers forming alternate depictions of same subject matter
US7728821B2 (en) 2004-08-06 2010-06-01 Touchtable, Inc. Touch detecting interactive display
US7719523B2 (en) * 2004-08-06 2010-05-18 Touchtable, Inc. Bounding box gesture recognition on a touch detecting interactive display
US20070046643A1 (en) * 2004-08-06 2007-03-01 Hillis W Daniel State-Based Approach to Gesture Identification
EP1779226B1 (en) * 2004-08-12 2018-10-24 Philips Intellectual Property & Standards GmbH Method and system for controlling a display
US8842186B2 (en) * 2004-10-25 2014-09-23 I-Interactive Llc Control system and method employing identification of a displayed image
DE102004053738A1 (en) * 2004-11-06 2006-05-11 Man Roland Druckmaschinen Ag Printing press control station
US8508710B2 (en) * 2004-12-02 2013-08-13 Hewlett-Packard Development Company, L.P. Display panel
US7555164B2 (en) * 2004-12-23 2009-06-30 Hewlett-Packard Development Company, L.P. Robust estimation of pixel values
EP3693889A3 (en) * 2005-01-07 2020-10-28 QUALCOMM Incorporated Detecting and tracking objects in images
JP4689684B2 (en) * 2005-01-21 2011-05-25 ジェスチャー テック，インコーポレイテッド Tracking based on movement
JP4612853B2 (en) * 2005-03-29 2011-01-12 キヤノン株式会社 Pointed position recognition device and information input device having the same
US8386909B2 (en) * 2005-04-07 2013-02-26 Hewlett-Packard Development Company, L.P. Capturing and presenting interactions with image-based media
US9128519B1 (en) 2005-04-15 2015-09-08 Intellectual Ventures Holding 67 Llc Method and system for state-based control of objects
US7499027B2 (en) * 2005-04-29 2009-03-03 Microsoft Corporation Using a light pointer for input on an interactive display surface
US8487910B2 (en) * 2005-05-02 2013-07-16 Smart Technologies Ulc Large scale touch system and methods for interacting with same
CN101171560A (en) * 2005-05-04 2008-04-30 皇家飞利浦电子股份有限公司 System and method for projecting control graphics
US8081822B1 (en) 2005-05-31 2011-12-20 Intellectual Ventures Holding 67 Llc System and method for sensing a feature of an object in an interactive video display
JP4589828B2 (en) * 2005-06-28 2010-12-01 日本ソフト開発株式会社 Remote control method and remote control apparatus thereof
US7525538B2 (en) * 2005-06-28 2009-04-28 Microsoft Corporation Using same optics to image, illuminate, and project
WO2007003682A1 (en) * 2005-06-30 2007-01-11 Nokia Corporation Camera control means to allow operating of a destined location of the information surface of a presentation and information system
US20070018966A1 (en) * 2005-07-25 2007-01-25 Blythe Michael M Predicted object location
US20070024580A1 (en) * 2005-07-29 2007-02-01 Microsoft Corporation Interactive display device, such as in context-aware environments
US7911444B2 (en) 2005-08-31 2011-03-22 Microsoft Corporation Input method for surface of interactive display
JP5272293B2 (en) * 2005-09-14 2013-08-28 ソニー株式会社 Remote commander and electronic device operation input method
US20070074040A1 (en) * 2005-09-29 2007-03-29 Nokia Corporation Online authorization using biometric and digital signature schemes
US7843449B2 (en) * 2006-09-20 2010-11-30 Apple Inc. Three-dimensional display system
FR2894353A1 (en) * 2005-12-02 2007-06-08 France Telecom METHOD FOR EXTRACTING AN OBJECT ON A PROJECTED BACKGROUND
US8098277B1 (en) 2005-12-02 2012-01-17 Intellectual Ventures Holding 67 Llc Systems and methods for communication between a reactive video system and a mobile communication device
US8060840B2 (en) 2005-12-29 2011-11-15 Microsoft Corporation Orientation free user interface
US20070165007A1 (en) * 2006-01-13 2007-07-19 Gerald Morrison Interactive input system
JP5201999B2 (en) * 2006-02-03 2013-06-05 パナソニック株式会社 Input device and method thereof
US7515143B2 (en) * 2006-02-28 2009-04-07 Microsoft Corporation Uniform illumination of interactive display panel
US7880719B2 (en) * 2006-03-23 2011-02-01 International Business Machines Corporation Recognition and capture of whiteboard markups in relation to a projected image
US7768527B2 (en) * 2006-05-31 2010-08-03 Beihang University Hardware-in-the-loop simulation system and method for computer vision
US8972902B2 (en) * 2008-08-22 2015-03-03 Northrop Grumman Systems Corporation Compound gesture recognition
US9696808B2 (en) * 2006-07-13 2017-07-04 Northrop Grumman Systems Corporation Hand-gesture recognition method
US8395658B2 (en) * 2006-09-07 2013-03-12 Sony Computer Entertainment Inc. Touch screen-like user interface that does not require actual touching
JP4650381B2 (en) * 2006-09-08 2011-03-16 日本ビクター株式会社 Electronics
US8356254B2 (en) * 2006-10-25 2013-01-15 International Business Machines Corporation System and method for interacting with a display
US9442607B2 (en) 2006-12-04 2016-09-13 Smart Technologies Inc. Interactive input system and method
WO2008083205A2 (en) * 2006-12-29 2008-07-10 Gesturetek, Inc. Manipulation of virtual objects using enhanced interactive system
US8212857B2 (en) * 2007-01-26 2012-07-03 Microsoft Corporation Alternating light sources to reduce specular reflection
WO2008128096A2 (en) 2007-04-11 2008-10-23 Next Holdings, Inc. Touch screen system with hover and click input methods
US8022997B2 (en) * 2007-04-19 2011-09-20 Fuji Xerox Co., Ltd. Information processing device and computer readable recording medium
ES2323399B2 (en) * 2007-06-19 2010-02-26 Gat Microencapsulation Ag SUSPENSIONS IN AGRICULTURAL SULFONILE AND COMBINATIONS OILS.
US8094137B2 (en) 2007-07-23 2012-01-10 Smart Technologies Ulc System and method of detecting contact on a display
EP2195726A1 (en) * 2007-08-30 2010-06-16 Next Holdings, Inc. Low profile touch panel systems
AU2008280953A1 (en) 2007-08-30 2009-03-19 Next Holdings Ltd Optical touchscreen with improved illumination
KR101141087B1 (en) 2007-09-14 2012-07-12 인텔렉츄얼 벤처스 홀딩 67 엘엘씨 Processing of gesture-based user interactions
US8159682B2 (en) 2007-11-12 2012-04-17 Intellectual Ventures Holding 67 Llc Lens system
JP4793377B2 (en) * 2007-11-16 2011-10-12 カシオ計算機株式会社 Projection recording apparatus, on-screen information recording method, and program
US20090132926A1 (en) * 2007-11-21 2009-05-21 Samsung Electronics Co., Ltd. Interactive presentation system and authorization method for voice command controlling interactive presentation process
ATE509308T1 (en) * 2007-12-05 2011-05-15 Almeva Ag INTERACTION ARRANGEMENT FOR INTERACTION BETWEEN A SCREEN AND A POINTER OBJECT
US20090160768A1 (en) * 2007-12-21 2009-06-25 Nvidia Corporation Enhanced Presentation Capabilities Using a Pointer Implement
US20090172714A1 (en) * 2007-12-28 2009-07-02 Harel Gruia Method and apparatus for collecting metadata during session recording
US8405636B2 (en) 2008-01-07 2013-03-26 Next Holdings Limited Optical position sensing system and optical position sensor assembly
US20100039500A1 (en) * 2008-02-15 2010-02-18 Matthew Bell Self-Contained 3D Vision System Utilizing Stereo Camera and Patterned Illuminator
US8259163B2 (en) 2008-03-07 2012-09-04 Intellectual Ventures Holding 67 Llc Display with built in 3D sensing
WO2009137355A2 (en) * 2008-05-06 2009-11-12 Next Holdings, Inc. Systems and methods for resolving multitouch scenarios using software filters
US8902193B2 (en) 2008-05-09 2014-12-02 Smart Technologies Ulc Interactive input system and bezel therefor
JP5258399B2 (en) * 2008-06-06 2013-08-07 キヤノン株式会社 Image projection apparatus and control method thereof
US8595218B2 (en) 2008-06-12 2013-11-26 Intellectual Ventures Holding 67 Llc Interactive display management systems and methods
US8427424B2 (en) * 2008-09-30 2013-04-23 Microsoft Corporation Using physical objects in conjunction with an interactive surface
US8339378B2 (en) * 2008-11-05 2012-12-25 Smart Technologies Ulc Interactive input system with multi-angle reflector
US20100134409A1 (en) * 2008-11-30 2010-06-03 Lenovo (Singapore) Pte. Ltd. Three-dimensional user interface
US20100225588A1 (en) * 2009-01-21 2010-09-09 Next Holdings Limited Methods And Systems For Optical Detection Of Gestures
US20100218100A1 (en) * 2009-02-25 2010-08-26 HNTB Holdings, Ltd. Presentation system
US8538367B2 (en) * 2009-06-29 2013-09-17 Qualcomm Incorporated Buffer circuit with integrated loss canceling
US8276077B2 (en) * 2009-07-10 2012-09-25 The Mcgraw-Hill Companies, Inc. Method and apparatus for automatic annotation of recorded presentations
US8692768B2 (en) 2009-07-10 2014-04-08 Smart Technologies Ulc Interactive input system
JP5292210B2 (en) * 2009-07-16 2013-09-18 株式会社エルモ社 Document presentation device
JP5521855B2 (en) * 2009-09-10 2014-06-18 株式会社リコー Projection image area detection device
US20110119638A1 (en) * 2009-11-17 2011-05-19 Babak Forutanpour User interface methods and systems for providing gesturing on projected images
US8549418B2 (en) * 2009-12-23 2013-10-01 Intel Corporation Projected display to enhance computer device use
US20120293555A1 (en) * 2010-01-15 2012-11-22 Akihiro Okano Information-processing device, method thereof and display device
CN102129151A (en) * 2010-01-20 2011-07-20 鸿富锦精密工业（深圳）有限公司 Front projection control system and method
US20110193969A1 (en) * 2010-02-09 2011-08-11 Qisda Corporation Object-detecting system and method by use of non-coincident fields of light
TW201128516A (en) * 2010-02-12 2011-08-16 Utw Technologies Corp An instinctive operating method and an electronic device using the same
JP5693022B2 (en) * 2010-03-04 2015-04-01 キヤノン株式会社 Display control device, display control system, and control method, program, and storage medium thereof
EP2545494A1 (en) * 2010-03-12 2013-01-16 Shafa Wala Position capture input apparatus, system, and method therefor
CN102253573A (en) * 2010-05-18 2011-11-23 鸿富锦精密工业（深圳）有限公司 Front projection device and front projection control method thereof
US20120011465A1 (en) * 2010-07-06 2012-01-12 Marcelo Amaral Rezende Digital whiteboard system
US9195967B2 (en) * 2010-07-08 2015-11-24 Johnathon Smith System using projector to display reminder messages generated remotely
US9660953B2 (en) * 2010-07-08 2017-05-23 Johnathon Smith System for providing continuous viewable indicia for reminders and for inducing viewer actions
US8848054B2 (en) 2010-07-29 2014-09-30 Crestron Electronics Inc. Presentation capture with automatically configurable output
JP2012053532A (en) * 2010-08-31 2012-03-15 Casio Comput Co Ltd Information processing apparatus and method, and program
US9466148B2 (en) * 2010-09-03 2016-10-11 Disney Enterprises, Inc. Systems and methods to dynamically adjust an image on a display monitor represented in a video feed
EP2614650A1 (en) 2010-09-10 2013-07-17 Lemoptix SA A device for projecting an image
TWI423184B (en) * 2010-09-21 2014-01-11 Chiefway Optronics Co Ltd The device and the method of implementing interactive advertising
US20120146904A1 (en) * 2010-12-13 2012-06-14 Electronics And Telecommunications Research Institute Apparatus and method for controlling projection image
US20120223960A1 (en) * 2011-03-01 2012-09-06 Avermedia Information, Inc. Image control method and image control system
JP2012181721A (en) * 2011-03-02 2012-09-20 Seiko Epson Corp Position input device, projector, control method for projector, and display system
CN102156379A (en) * 2011-03-22 2011-08-17 上海学舟信息技术有限公司 Novel interactive recording and playing curriculum system
US8620113B2 (en) 2011-04-25 2013-12-31 Microsoft Corporation Laser diode modes
US10044871B2 (en) 2011-04-29 2018-08-07 Crestron Electronics, Inc. Conference system including automated equipment setup
US10692020B2 (en) 2011-04-29 2020-06-23 Crestron Electronics, Inc. Real-time automatic meeting room reservation based on the number of actual participants
US9294723B2 (en) 2011-04-29 2016-03-22 Creston Electronics, Inc. Meeting management system including automated equipment setup
US8760395B2 (en) 2011-05-31 2014-06-24 Microsoft Corporation Gesture recognition techniques
US9030487B2 (en) * 2011-08-01 2015-05-12 Lg Electronics Inc. Electronic device for displaying three-dimensional image and method of using the same
CN102956123B (en) * 2011-08-25 2015-06-03 亿度慧达教育科技（北京）有限公司 Teaching sensing device
US8635637B2 (en) 2011-12-02 2014-01-21 Microsoft Corporation User interface presenting an animated avatar performing a media reaction
US9100685B2 (en) 2011-12-09 2015-08-04 Microsoft Technology Licensing, Llc Determining audience state or interest using passive sensor data
US8811938B2 (en) 2011-12-16 2014-08-19 Microsoft Corporation Providing a user interface experience based on inferred vehicle state
US20130179811A1 (en) * 2012-01-05 2013-07-11 Visteon Global Technologies, Inc. Projection dynamic icon knobs
US8898687B2 (en) 2012-04-04 2014-11-25 Microsoft Corporation Controlling a media program based on a media reaction
CA2775700C (en) 2012-05-04 2013-07-23 Microsoft Corporation Determining a future portion of a currently presented media program
WO2013175389A2 (en) * 2012-05-20 2013-11-28 Extreme Reality Ltd. Methods circuits apparatuses systems and associated computer executable code for providing projection based human machine interfaces
US20150186039A1 (en) * 2012-08-27 2015-07-02 Citizen Holdings Co., Ltd. Information input device
CN102929409B (en) * 2012-10-12 2016-01-13 明基电通有限公司 Telechiric device, display system and display packing
JP6111682B2 (en) * 2013-01-21 2017-04-12 セイコーエプソン株式会社 Display device and control method of display device
TWI454971B (en) * 2012-12-11 2014-10-01 Pixart Imaging Inc Electronic apparatus controlling method and electronic apparatus utilizing the electronic apparatus controlling method
CN103869959B (en) * 2012-12-18 2017-06-09 原相科技股份有限公司 Electronic apparatus control method and electronic installation
JP2014206634A (en) * 2013-04-12 2014-10-30 船井電機株式会社 Electronic apparatus
US10891003B2 (en) 2013-05-09 2021-01-12 Omni Consumer Products, Llc System, method, and apparatus for an interactive container
US9360888B2 (en) * 2013-05-09 2016-06-07 Stephen Howard System and method for motion detection and interpretation
US20150370441A1 (en) * 2014-06-23 2015-12-24 Infosys Limited Methods, systems and computer-readable media for converting a surface to a touch surface
US20150138085A1 (en) * 2013-07-31 2015-05-21 Bradley Lloyd Wilk Electronic apparatus for simulating or interfacing a backward compatible human input device by means or control of a gesture recognition system
JP2015094768A (en) * 2013-11-08 2015-05-18 セイコーエプソン株式会社 Display device, display system, and control method
US9011246B1 (en) * 2013-11-18 2015-04-21 Scott Kier Systems and methods for immersive backgrounds
JP6398248B2 (en) 2014-01-21 2018-10-03 セイコーエプソン株式会社 Position detection system and method for controlling position detection system
JP6079695B2 (en) * 2014-05-09 2017-02-15 コニカミノルタ株式会社 Image display photographing system, photographing device, display device, image display and photographing method, and computer program
US9740338B2 (en) 2014-05-22 2017-08-22 Ubi interactive inc. System and methods for providing a three-dimensional touch screen
US20160027131A1 (en) * 2014-07-28 2016-01-28 Cognizant Technology Solutions India Pvt. Ltd System and method for projecting an interactive image and processing user interaction
JP6282758B2 (en) * 2014-11-13 2018-02-21 マクセル株式会社 Projection-type image display device and image display method
CN104345888A (en) * 2014-11-28 2015-02-11 重庆欧派信息科技有限责任公司 Somatic sensing interaction knowledge question answering system
TWI554822B (en) * 2015-03-06 2016-10-21 許建清 Briefing system having function of synchronizing indication position and method thereof
JP2018516422A (en) * 2015-05-28 2018-06-21 アイサイト モバイル テクノロジーズ エルティーディー． Gesture control system and method for smart home
US10217283B2 (en) * 2015-12-17 2019-02-26 Google Llc Navigation through multidimensional images spaces
US11086751B2 (en) 2016-03-16 2021-08-10 Asg Technologies Group, Inc. Intelligent metadata management and data lineage tracing
US11847040B2 (en) 2016-03-16 2023-12-19 Asg Technologies Group, Inc. Systems and methods for detecting data alteration from source to target
CN105702107A (en) * 2016-04-25 2016-06-22 苏州恒体体育发展有限公司 VR holographic body-building and dancing course teaching system
US10592010B1 (en) 2017-06-28 2020-03-17 Apple Inc. Electronic device system with input tracking and visual output
JP7009880B2 (en) * 2017-09-26 2022-01-26 カシオ計算機株式会社 Image correction device, image correction method and program
US11057500B2 (en) * 2017-11-20 2021-07-06 Asg Technologies Group, Inc. Publication of applications using server-side virtual screen change capture
US10812611B2 (en) 2017-12-29 2020-10-20 Asg Technologies Group, Inc. Platform-independent application publishing to a personalized front-end interface by encapsulating published content into a container
US11611633B2 (en) 2017-12-29 2023-03-21 Asg Technologies Group, Inc. Systems and methods for platform-independent application publishing to a front-end interface
US10877740B2 (en) 2017-12-29 2020-12-29 Asg Technologies Group, Inc. Dynamically deploying a component in an application
DE102019002885A1 (en) 2018-04-27 2019-10-31 Marquardt Gmbh operating device
US11762634B2 (en) 2019-06-28 2023-09-19 Asg Technologies Group, Inc. Systems and methods for seamlessly integrating multiple products by using a common visual modeler
US11755760B2 (en) 2019-10-18 2023-09-12 Asg Technologies Group, Inc. Systems and methods for secure policies-based information governance
US11886397B2 (en) 2019-10-18 2024-01-30 Asg Technologies Group, Inc. Multi-faceted trust system
US11941137B2 (en) 2019-10-18 2024-03-26 Asg Technologies Group, Inc. Use of multi-faceted trust scores for decision making, action triggering, and data analysis and interpretation
US11269660B2 (en) 2019-10-18 2022-03-08 Asg Technologies Group, Inc. Methods and systems for integrated development environment editor support with a single code base
US11055067B2 (en) 2019-10-18 2021-07-06 Asg Technologies Group, Inc. Unified digital automation platform
WO2022081476A1 (en) 2020-10-13 2022-04-21 ASG Technologies Group, Inc. dba ASG Technologies Geolocation-based policy rules

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4430526A (en) * 1982-01-25 1984-02-07 Bell Telephone Laboratories, Incorporated Interactive graphics transmission system employing an adaptive stylus for reduced bandwidth
US5138304A (en) * 1990-08-02 1992-08-11 Hewlett-Packard Company Projected image light pen
US5181015A (en) * 1989-11-07 1993-01-19 Proxima Corporation Method and apparatus for calibrating an optical computer input system
US5436639A (en) * 1993-03-16 1995-07-25 Hitachi, Ltd. Information processing system

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE3176016D1 (en) * 1980-12-30 1987-04-23 Ibm System for remotely displaying and sensing information using shadow parallax
GB2173079B (en) * 1985-03-29 1988-05-18 Ferranti Plc Cursor display control apparatus
US5239373A (en) * 1990-12-26 1993-08-24 Xerox Corporation Video computational shared drawing space
JPH05303467A (en) * 1991-10-04 1993-11-16 Shigumatsukusu Kk Operation input device

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4430526A (en) * 1982-01-25 1984-02-07 Bell Telephone Laboratories, Incorporated Interactive graphics transmission system employing an adaptive stylus for reduced bandwidth
US5181015A (en) * 1989-11-07 1993-01-19 Proxima Corporation Method and apparatus for calibrating an optical computer input system
US5138304A (en) * 1990-08-02 1992-08-11 Hewlett-Packard Company Projected image light pen
US5436639A (en) * 1993-03-16 1995-07-25 Hitachi, Ltd. Information processing system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP0771460A4 *

Cited By (28)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5729252A (en) * 1994-12-27 1998-03-17 Lucent Technologies, Inc. Multimedia program editing system and method
EP0720109A1 (en) * 1994-12-27 1996-07-03 AT&T Corp. Multimedia program indexing system and method
GB2315859A (en) * 1996-07-29 1998-02-11 Mitel Corp Input device simulating touch screen
GB2315859B (en) * 1996-07-29 2000-10-18 Mitel Corp Input device simulating touch screen
DE19714109A1 (en) * 1997-04-05 1998-10-08 Moser Hans Video cabin for film presentation system
US6788289B2 (en) 2000-02-08 2004-09-07 Seiko Epson Corporation System and method of pointed position detection system, projector and program
US7034807B2 (en) 2000-02-21 2006-04-25 Siemens Aktiengesellschaft Method and configuration for interacting with a display visible in a display window
US6829394B2 (en) 2000-02-22 2004-12-07 Seiko Epson Corporation System and method of pointed position detection, presentation system, and program
EP2296080A3 (en) * 2000-07-05 2011-06-29 SMART Technologies ULC Camera-based touch system
GB2370395A (en) * 2000-12-19 2002-06-26 Ubinetics Ltd Display apparatus
EP1821183A4 (en) * 2004-10-05 2011-01-26 Nikon Corp Electronic device
EP1821183A1 (en) * 2004-10-05 2007-08-22 Nikon Corporation Electronic device
US8330714B2 (en) 2004-10-05 2012-12-11 Nikon Corporation Electronic device
AT503008B1 (en) * 2005-12-16 2008-09-15 Dobler & Stampfl Audite Oeg INTERACTIVE OPTICAL SYSTEM AND METHOD FOR EXTRACTING AND ASSESSING INTERACTIONS IN AN OPTICAL SYSTEM
WO2007088430A1 (en) * 2006-02-01 2007-08-09 Nokia Corporation System, device, method and computer program product for using a mobile camera for controlling a computer
WO2009009338A2 (en) * 2007-07-11 2009-01-15 Igt Processing input for computing systems based on the state of execution
WO2009009338A3 (en) * 2007-07-11 2009-05-14 Igt Reno Nev Processing input for computing systems based on the state of execution
AU2008275379B2 (en) * 2007-07-11 2013-01-10 Igt Processing input for computing systems based on the state of execution
US8118433B2 (en) 2007-09-04 2012-02-21 Canon Kabushiki Kaisha Image projection apparatus and control method for same
EP2191350A4 (en) * 2007-09-04 2010-12-08 Canon Kk Image projection apparatus and control method for same
EP2191350A1 (en) * 2007-09-04 2010-06-02 Canon Kabushiki Kaisha Image projection apparatus and control method for same
WO2009092126A1 (en) * 2008-01-25 2009-07-30 Stumpfl, Reinhold Interactive multimedia presentation apparatus
EP2177976A2 (en) * 2008-10-15 2010-04-21 LG Electronics, Inc. Mobile terminal with image projection
EP2177976A3 (en) * 2008-10-15 2013-12-11 LG Electronics, Inc. Mobile terminal with image projection
EP2383609A1 (en) * 2010-04-30 2011-11-02 Samsung Electronics Co., Ltd. Interactive display apparatus and operating method thereof
US9030425B2 (en) 2011-04-19 2015-05-12 Sony Computer Entertainment Inc. Detection of interaction with virtual object from finger color change
WO2013151645A1 (en) * 2012-04-02 2013-10-10 Sony Computer Entertainment Inc. Detection of interaction with virtual object from finger color change
US10417515B2 (en) 2017-01-09 2019-09-17 Microsoft Technology Licensing, Llc Capturing annotations on an electronic display

Also Published As

Publication number Publication date
EP0771460A4 (en) 1998-07-22
DE69530395T2 (en) 2004-02-19
US5528263A (en) 1996-06-18
AU2944095A (en) 1996-01-05
JPH09512656A (en) 1997-12-16
DE69530395D1 (en) 2003-05-22
EP0771460B1 (en) 2003-04-16
EP0771460A1 (en) 1997-05-07

Similar Documents

Publication Publication Date Title
EP0771460B1 (en) Interactive projected video image display system
CN101278251B (en) Interactive large scale touch surface system
US20010030668A1 (en) Method and system for interacting with a display
EP2332024B1 (en) Using physical objects in conjunction with an interactive surface
US6802611B2 (en) System and method for presenting, capturing, and modifying images on a presentation board
US7463270B2 (en) Physical-virtual interpolation
CN105706028B (en) Projection-type image display device
US9030498B2 (en) Combining explicit select gestures and timeclick in a non-tactile three dimensional user interface
US8456416B2 (en) Image recognition apparatus, and operation determination method and program therefor
JP4991154B2 (en) Image display device, image display method, and command input method
US20070300182A1 (en) Interface orientation using shadows
US20070188518A1 (en) Variable orientation input mode
US6979087B2 (en) Display system with interpretable pattern detection
US20100091110A1 (en) Single camera tracker
JPH07182101A (en) Apparatus and method for input of graphic, operating method of graphic object and supply method of graphic input signal
US6047249A (en) Video camera based computer input system with interchangeable physical interface
CN106325726A (en) A touch control interaction method
JPS5911935B2 (en) How to erase parts of a drawn image
JP4728540B2 (en) Image projection device for meeting support
KR101911676B1 (en) Apparatus and Method for Presentation Image Processing considering Motion of Indicator
WO2000058933A1 (en) Method and apparatus for visual pointing and computer control
JP3186361B2 (en) Data processing device
JPH0749748A (en) Command indicating device
JPH07141108A (en) Touch panel device
JPH04294416A (en) Method for selecting one of objects indicated on display device

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AM AT AU BB BG BR BY CA CH CN CZ DE DK ES FI GB GE HU JP KE KG KP KR KZ LK LT LU LV MD MG MN MW NO NZ PL PT RO RU SD SE SI SK TJ TT UA US UZ VN

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): KE MW SD SZ UG AT BE CH DE DK ES FR GB GR IE IT LU MC NL PT SE BF BJ CF CG CI CM GA GN ML MR NE SN TD TG

DFPE Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed before 20040101)
121 Ep: the epo has been informed by wipo that ep was designated in this application
WWE Wipo information: entry into national phase

Ref document number: 1995925249

Country of ref document: EP

REG Reference to national code

Ref country code: DE

Ref legal event code: 8642

WWP Wipo information: published in national office

Ref document number: 1995925249

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: CA

WWG Wipo information: grant in national office

Ref document number: 1995925249

Country of ref document: EP