WO2005118998A1 - Horizontal perspective simulator - Google Patents

Horizontal perspective simulator Download PDF

Info

Publication number
WO2005118998A1
WO2005118998A1 PCT/US2005/019069 US2005019069W WO2005118998A1 WO 2005118998 A1 WO2005118998 A1 WO 2005118998A1 US 2005019069 W US2005019069 W US 2005019069W WO 2005118998 A1 WO2005118998 A1 WO 2005118998A1
Authority
WO
WIPO (PCT)
Prior art keywords
display
image
horizontal perspective
peripheral device
simulator system
Prior art date
Application number
PCT/US2005/019069
Other languages
French (fr)
Other versions
WO2005118998A8 (en
Inventor
Michael A. Vesely
Nancy Clemens
Original Assignee
Vesely Michael A
Nancy Clemens
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Vesely Michael A, Nancy Clemens filed Critical Vesely Michael A
Priority to EP05757162A priority Critical patent/EP1781893A1/en
Priority to JP2007515486A priority patent/JP2008507006A/en
Publication of WO2005118998A1 publication Critical patent/WO2005118998A1/en
Publication of WO2005118998A8 publication Critical patent/WO2005118998A8/en

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/03Arrangements for converting the position or the displacement of a member into a coded form
    • G06F3/033Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor
    • G06F3/0346Pointing devices displaced or positioned by the user, e.g. mice, trackballs, pens or joysticks; Accessories therefor with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B30/00Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
    • G02B30/40Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images giving the observer of a single two-dimensional [2D] image a perception of depth
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B30/00Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
    • G02B30/50Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images the image being built up from image elements distributed over a 3D volume, e.g. voxels
    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B30/00Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images
    • G02B30/50Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images the image being built up from image elements distributed over a 3D volume, e.g. voxels
    • G02B30/56Optical systems or apparatus for producing three-dimensional [3D] effects, e.g. stereoscopic images the image being built up from image elements distributed over a 3D volume, e.g. voxels by projecting aerial or floating images
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/011Arrangements for interaction with the human body, e.g. for user immersion in virtual reality
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G06F3/04815Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/003D [Three Dimensional] image rendering
    • G06T15/10Geometric effects
    • G06T15/20Perspective computation
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06TIMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00Geometric image transformation in the plane of the image
    • G06T3/40Scaling the whole image or part thereof
    • G06T3/4038Scaling the whole image or part thereof for image mosaicing, i.e. plane images composed of plane sub-images
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20Image signal generators
    • H04N13/275Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals
    • H04N13/279Image signal generators from 3D object models, e.g. computer-generated stereoscopic image signals the virtual viewpoint locations being selected by the viewers or determined by tracking
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/356Image reproducers having separate monoscopic and stereoscopic modes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/361Reproducing mixed stereoscopic images; Reproducing mixed monoscopic and stereoscopic images, e.g. a stereoscopic image overlay window on a monoscopic image background
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/363Image reproducers using image projection screens
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/366Image reproducers using viewer tracking
    • H04N13/383Image reproducers using viewer tracking for tracking with gaze detection, i.e. detecting the lines of sight of the viewer's eyes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/30Image reproducers
    • H04N13/388Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume
    • H04N13/395Volumetric displays, i.e. systems where the image is built up from picture elements distributed through a volume with depth sampling, i.e. the volume being constructed from a stack or sequence of 2D image planes
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/302Electronic adaptation of stereophonic sound system to listener position or orientation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/302Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303Tracking of listener position or orientation
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04RLOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; DEAF-AID SETS; PUBLIC ADDRESS SYSTEMS
    • H04R2499/00Aspects covered by H04R or H04S not otherwise provided for in their subgroups
    • H04R2499/10General applications
    • H04R2499/15Transducers incorporated in visual displaying devices, e.g. televisions, computer displays, laptops
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S2420/00Techniques used stereophonic systems covered by H04S but not provided for in its groups
    • H04S2420/01Enhancing the perception of the sound image or of the spatial distribution using head related transfer functions [HRTF's] or equivalents thereof, e.g. interaural time difference [ITD] or interaural level difference [ILD]
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04SSTEREOPHONIC SYSTEMS 
    • H04S7/00Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30Control circuits for electronic adaptation of the sound field
    • H04S7/302Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303Tracking of listener position or orientation
    • H04S7/304For headphones

Definitions

  • This invention relates to a three-dimensional simulator system, and in particular, to a hands-on computer simulator system capable of operator's interaction.
  • Three dimensional (3D) capable electronics and computing hardware devices and real-time computer-generated 3D computer graphics have been a popular area of computer science for the past few decades, with innovations in visual, audio and tactile systems. Much of the research in this area has produced hardware and software products that are specifically designed to generate greater realism and more natural computer- human interfaces. These innovations have significantly enhanced and simplified the end- user's computing experience.
  • the answer is three dimensional illusions.
  • the two dimensional pictures must provide a numbers of cues of the third dimension to the brain to create the illusion of three dimensional images.
  • This effect of third dimension cues can be realistically achievable due to the fact that the brain is quite accustomed to it.
  • the three dimensional real world is always and already converted into two dimensional (e.g. height and width) projected image at the retina, a concave surface at the back of the eye. And from this two dimensional image, the brain, through experience and perception, generates the depth information to fonn the three dimension visual image from two types of depth cues: monocular (one eye perception) and binocular (two eye perception).
  • binocular depth cues are innate and biological while monocular depth cues are learned and environmental.
  • the major binocular depth cues are convergence and retinal disparity.
  • the brain measures the amount of convergence of the eyes to provide a rough estimate of the distance since the angle between the line of sight of each eye is larger when an object is closer.
  • the disparity of the retinal images due to the separation of the two eyes is used to create the perception of depth.
  • the effect is called stereoscopy where each eye receives a slightly different view of a scene, and the brain fuses them together using these differences to determine the ratio of distances between nearby objects.
  • Binocular cues are very powerful perception of depth. However, there are also depth cues with only one eye, called monocular depth cues, to create an impression of depth on a flat image.
  • the major monocular cues are: overlapping, relative size, linear perspective and light and shadow. When an object is viewed partially covered, this pattern of blocking is used as a cue to determine that the object is farther away. When two objects known to be the same size and one appears smaller than the other, this pattern of relative size is used as a cue to assume that the smaller object is farther away.
  • the cue of relative size also provides the basis for the cue of linear perspective where the farther away the lines are from the observer, the closer together they will appear since parallel lines in a perspective image appear to converge towards a single point. The light falling on an object from a certain angle could provide the cue for the form and depth of an object.
  • the distribution of light and shadow on objects is a powerful monocular cue for depth provided by "the biologically correct assumption that light comes from above.
  • Perspective drawing is most often used to achieve the illusion of three dimension depth and spatial relationships on a flat (two dimension) surface, such as paper or canvas.
  • a flat (two dimension) surface such as paper or canvas.
  • three dimension objects are depicted on a two dimension plane, but "trick" the eye into appearing to be in three dimension space.
  • the first theoretical treatise for constructing perspective, Depictura was published in the early 1400's by the architect, Leone Battista Alberti. Since the introduction of his book, the details behind "general” perspective have been very well documented. However, the fact that there are a number of other types of perspectives is not well known. Some examples are military, cavalier, isometric, and dimetric, as shown at the top of Figure 1.
  • Central perspective also called one-point perspective, is the simplest kind of "genuine" perspective construction, and is often taught in art and drafting classes for beginners.
  • Figure 2 further illustrates central perspective.
  • Central perspective uses central perspective, the chess board and chess pieces look like three dimension objects, even though they are drawn on a two dimensional flat piece of paper.
  • Central perspective has a central vanishing point, and rectangular objects are placed so their front sides are parallel to the picture plane. The depth of the objects is pe ⁇ endicular to the picture plane. All parallel receding edges run towards a central vanishing point. The viewer looks towards this vanishing point with a straight view.
  • an architect or artist creates a drawing using central perspective they must use a single-eye view. That is, the artist creating the drawing captures the image by looking through only one eye, which is pe ⁇ endicular to the drawing surface.
  • 3D computer graphics Central perspective is employed extensively in 3D computer graphics, for a myriad of applications, such as scientific, data visualization, computer-generated prototyping, special effects for movies, medical imaging, and architecture, to name just a few.
  • 3D gaming One of the most common and well-known 3D computing applications is 3D gaming, which is used here as an example, because the core concepts used in 3D gaming extend to all other 3D computing applications.
  • Figure 3 is a simple illustration, intended to set the stage by listing the basic components necessary to achieve a high level of realism in 3D software applications. At its highest level, 3D game development consists of four essential components:
  • a person using a 3D application is in fact running software in the form of a real-time computer-generated 3D graphics engine.
  • One of the engine's key components is the renderer. Its job is to take 3D objects that exist within computer- generated world coordinates x, y, z, and render (draw/display) them onto the computer monitor's viewing surface, which is a flat (2D) plane, with real world coordinates x, y.
  • Figure 4 is a representation of what is happening inside the computer when running a 3D graphics engine. Within every 3D game there exists a computer-generated 3D "world.” This world contains everything that could be experienced during game play.
  • Game play for a typical 3D game might begin with a computer-generated-3D earth and a computer- generated-3D satellite orbiting it.
  • the virtual world coordinate system enables the earth and satellite to be properly positioned in computer-generated x, y, z space.
  • the 3D graphics engine creates a fourth universal dimension for computer-generated time, t. For every tick of time t, the 3D graphics engine regenerates the satellite at its new location and orientation as it orbits the spinning earth. Therefore, a key job for a 3D graphics engine is to continuously synchronize and regenerate all 3D objects within all four computer-generated dimensions x, y, z, and t.
  • Figure 5 is a conceptual illustration of what happens inside the computer when an end-user is playing, i.e. running, a first-person 3D application.
  • First-person means that the computer monitor is much like a window, through which the person playing the game views the computer-generated world.
  • the 3D graphics engine renders the scene from the point of view of the eye of a computer-generated person.
  • the computer-generated person can be thought of as a computer-generated or "virtual" simulation of the "real" person actually playing the game.
  • the boxed-in area in Figure 5 conceptually represents how a 3D graphics engine minimizes the hardware's burden. It focuses computational resources on extremely small areas of information as compared to the 3D applications entire world. In this example, it is a "computer-generated" polar bear cub being observed by a "computer-generated” virtual person. Because the end user is running in first-person everything the computer- generated person sees is rendered onto the end-user's monitor, i.e. the end user is looking through the eye of the computer-generated person.
  • the computer-generated person is looking through only one eye; in other words, an one-eyed view.
  • the area that the computer-generated person sees with a one-eye view is • called the "view volume", and the computer-generated 3D objects within this view volume are what actually get rendered to the computer monitor's 2D viewing surface.
  • FIG. 6 illustrates a view volume in more detail.
  • a view volume is a subset of a "camera model”.
  • a camera model is a blueprint that defines the characteristics of both the hardware and software of a 3D graphics engine. Like a very complex and sophisticated automobile engine, a 3D graphics engine consist of so many parts that their camera models are often simplified to illustrate only the essential elements being referenced.
  • the camera model depicted in Figure 6 shows a 3D graphics engine using central perspective to render computer-generated 3D objects to a computer monitor's vertical, 2D viewing surface.
  • the view volume shown in Figure 6, although more detailed, is the same view volume represented in Figure 5. The only difference is semantics because a 3D graphics engine calls the computer-generated person's one-eye view a camera point (hence camera model).
  • the element called near clip plane is the 2D plane onto which the x, y, z coordinates of the 3D objects within the view volume will be rendered.
  • Each projection line starts at the camera point, and ends at a x, y. z coordinate point of a virtual 3D object within the view volume.
  • the 3D graphics engine determines where the projection line intersects the near clip-plane and the x and y point where this intersection occurs is rendered onto the near clip-plane.
  • the near clip plane is displayed on the 2D viewing surface of the computer monitor, as shown in Figure 6.
  • 3D central perspective projection though offering realistic 3D illusion, has some limitations is allowing the user to have hands-on interaction with the 3D display.
  • horizontal perspective There is a little known class of images that we called it "horizontal perspective" where the image appears distorted when viewing head on, but displaying a three dimensional illusion when viewing from the correct viewing position.
  • the angle between the viewing surface and the line of vision is preferably 45° but can be almost any angle, and the viewing surface is preferably horizontal (wherein the name "horizontal perspective"), but it can be any surface, as long as the line of vision forming a not-pe ⁇ endicular angle to it.
  • Horizontal perspective images offer realistic three dimensional illusion, but are little known primarily due to the narrow viewing location (the viewer's eyepoint has to be coincide precisely with the image projection eyepoint), and the complexity involving in projecting the two dimensional image or the three dimension model into the horizontal perspective image.
  • the generation of horizontal perspective images requires considerably more expertise to create than conventional pe ⁇ endicular images.
  • the conventional pe ⁇ endicular images can be produced directly from the viewer or camera point. One need simply open one's eyes or point the camera in any direction to obtain the images. Further, with much experience in viewing three dimensional depth cues from pe ⁇ endicular images, viewers can tolerate significant amount of distortion generated by the deviations from the camera point.
  • the creation of a horizontal perspective image does require much manipulation. Conventional camera, by projecting the image into the plane pe ⁇ endicular to the line of sight, would not produce a horizontal perspective image. Making a horizontal drawing requires much effort and very time consuming.
  • the present invention recognizes that the personal computer is perfectly suitable for horizontal perspective display. It is personal, thus it is designed for the operation of one person, and the computer, with its powerful microprocessor, is well capable of rendering various horizontal perspective images to the viewer. Further, horizontal perspective offers open space display of 3D images, thus allowing the hands-on interaction of the end users.
  • the present invention discloses a multi-plane hands-on simulator system comprising at least two display surfaces, one of which displaying a three dimensional horizontal perspective images.
  • the other display surfaces can display two dimensional images, or preferably three dimensional central pe ⁇ ective images.
  • the display surfaces can have a curvilinear blending display section to merge the various images.
  • the multi-plane hands-on simulator can comprise various camera eyepoints, one for the horizontal perspective images, one for the central perspective images, and optionally one for the curvilinear blending display surface.
  • the multi-plane display surface can further adjust the various images to accommodate the position of the viewer.
  • the display can accept manual input such as a computer mouse, trackball, joystick, tablet, etc. to re-position the horizontal perspective images.
  • the display can also automatically re-position the images based on an input device automatically providing the viewer's viewpoint location.
  • the multi-plane hands-on simulator system can project horizontal perspective images into the open space and a peripheral device that allow the end user to manipulate the images with hands or hand-held tools.
  • the display is also included three dimensional audio such as binaural simulation to lend realism to the three dimensional display.
  • FIG. 1 shows the various perspective drawings.
  • Figure 2 shows a typical central perspective drawing.
  • Figure 3 shows 3D software application.
  • Figure 4 shows 3D application running on PC.
  • Figure 5 shows 3D application in first person.
  • Figure 6 shows central perspective camera model
  • Figure 7 shows the comparison of central perspective (Image A) and horizontal perspective (Image B).
  • Figure 8 shows the central perspective drawing of three stacking blocks.
  • Figure 9 shows the horizontal perspective drawing of three stacking blocks.
  • Figure 10 shows the method of drawing a horizontal perspective drawing.
  • Figure 11 shows a horizontal perspective display and a viewer input device.
  • Figure 12 shows a horizontal perspective display, a computational device and a viewer input device.
  • Figure 13 shows a computer monitor
  • Figure 14 shows a monitor's phosphor layer indicating of an incorrect location of image.
  • Figure 15 shows a monitor's viewing surface indicating of a correct location of image.
  • Figure 1 shows a reference plane x, y, z coordinates.
  • Figure 17 shows the location of an angled camera point.
  • Figure 18 shows the mapping of the horizontal plane to a reference plane.
  • Figure 19 shows the comfort plane.
  • Figure 20 shows the hands-on volume.
  • Figure 21 shows the inner plane
  • Figure 22 shows the bottom plane
  • Figure 23 shows the inner access volume
  • Figure 24 shows the angled camera mapped to the end-user's eye
  • Figure 25 shows mapping of the 3-d object onto the horizontal plane.
  • Figure 26 shows the two-eye view.
  • Figure 27 shows the simulation time of the horizontal perspective.
  • Figure 28 shows the horizontal plane.
  • Figure 29 shows the 3D peripherals.
  • Figure 30 shows an open-access camera model.
  • Figure 31 shows the concept of object recognition.
  • Figure 32 shows the 3D audio combination with object recognition.
  • Figure 33 shows another open access camera model.
  • Figure 34 shows another open access camera model
  • Figure 35 shows the mapping of virtual attachments to end of tools.
  • Figure 36 shows the multi-plane and multi-view device.
  • Figure 37 shows an open access camera model.
  • Figure 38 shows another multi-plane device.
  • the new and unique inventions described in this document build upon prior art by taking the current state of real-time computer-generated 3D computer graphics, 3D sound, and tactile computer-human interfaces to a whole new level of reality and simplicity. More specifically, these new inventions enable real-time computer-generated 3D simulations to coexist in physical space and time with the end-user and with other real-world physical objects. This capability dramatically improves upon the end-user's visual, auditory and tactile computing experience by providing direct physical interactions with 3D computer-generated objects and sounds.
  • the present invention discloses a multi-plane horizontal perspective hands-on simulator comprising at least two display surfaces, one of which capable of projecting three dimensional illusion based on horizontal perspective projection.
  • the present invention horizontal perspective hands-on simulator can be used to display and interact with three dimensional images and has obvious utility to many industrial applications such as manufacturing design reviews, ergonomic simulation, safety and training, video games, cinematography, scientific 3D viewing, and medical and other data displays.
  • horizontal perspective Normally, as in central perspective, the plane of vision, at right angle to the line of sight, is also the projected plane of the picture, and depth cues are used to give the illusion of depth to this flat image.
  • the plane of vision remains the same, but the projected image is not on this plane. It is on a plane angled to the plane of vision. Typically, the image would be on the ground level surface. This means the image will be physically in the third dimension relative to the plane of vision.
  • horizontal perspective can be called horizontal projection.
  • the object In horizontal perspective, the object is to separate the image from the paper, and fuse the image to the three dimension object that projects the horizontal perspective image.
  • the horizontal perspective image must be distorted so that the visual image fuses to form the free standing three dimensional figure. It is also essential the image is viewed from the correct eye points, otherwise the three dimensional illusion is lost.
  • the horizontal perspective images In contrast to central perspective images which have height and width, and project an illusion of depth, and therefore the objects are usually abruptly projected and the images appear to be in layers, the horizontal perspective images have actual depth and width, and illusion gives them height, and therefore there is usually a graduated shifting so the images appear to be continuous.
  • Figure 7 compares key characteristics that differentiate central perspective and horizontal perspective.
  • Image A shows key pertinent characteristics of central perspective
  • Image B shows key pertinent characteristics of horizontal perspective.
  • Image A the real-life three dimension object (three blocks stacked slightly above each other) was drawn by the artist closing one eye, and viewing along a line of sight pe ⁇ endicular to the vertical drawing plane.
  • the resulting image when viewed vertically, straight on, and through one eye, looks the same as the original image.
  • Image B the real-life three dimension object was drawn by the artist closing one eye, and viewing along a line of sight 45° to the horizontal drawing plane.
  • the resulting image when viewed horizontally, at 45° and through one eye, looks the same as the original image.
  • central perspective showing in Image A and horizontal perspective showing in Image B is the location of the display plane with respect to the projected three dimensional image.
  • the display plane can be adjusted up and down , and therefore the projected image can be displayed in the open air above the display plane, i.e. a physical hand can touch (or more likely pass through) the illusion, or it can be displayed under the display plane, i.e. one cannot touch the illusion because the display plane physically blocks the hand.
  • This is the nature of horizontal perspective, and as long as the camera eyepoint and the viewer eyepoint is at the same place, the illusion is present.
  • the three dimensional illusion is likely to be only inside the display plane, meaning one cannot touch it.
  • the central perspective would need elaborate display scheme such as surround image projection and large volume.
  • Figures 8 and 9 illustrate the visual difference between using central and horizontal perspective. To experience this visual difference, first look at Figure 8, drawn with central perspective, through one open eye. Hold the piece of paper vertically in front of you, as you would a traditional drawing, pe ⁇ endicular to your eye. You can see that central perspective provides a good representation of three dimension objects on a two dimension surface.
  • Figure 10 is an architectural-style illustration that demonstrates a method for making simple geometric drawings on paper or canvas utilizing horizontal perspective.
  • Figure 10 is a side view of the same three blocks used in Figures 9. It illustrates the actual mechanics of horizontal perspective. Each point that makes up the object is drawn by projecting the point onto the horizontal drawing plane. To illustrate this, Figure 10 shows a few of the coordinates of the blocks being drawn on the horizontal drawing plane through projection lines.
  • the horizontal perspective display system promotes horizontal perspective projection viewing by providing the viewer with the means to adjust the displayed images to maximize the illusion viewing experience.
  • the horizontal perspective display is shown in Fig. 11, comprising a real time electronic display 100 capable of re-drawing the projected image, together with a viewer's input device 102 to adjust the horizontal perspective image.
  • the horizontal perspective display can ensure the minimum distortion in rendering the three dimension illusion from the horizontal perspective method.
  • the input device can be manually operated where the viewer manually inputs his or her eyepoint location, or change the projection image eyepoint to obtain the optimum three dimensional illusion.
  • the input device can also be automatically operated where the display automatically tracks the viewer's eyepoint and adjust the projection image accordingly.
  • the horizontal perspective display removes the constraint that the viewers keeping their heads in relatively fixed positions, a constraint that create much difficulty in the acceptance of precise eyepoint location such as horizontal perspective or hologram display.
  • the horizontal perspective display system can further comprise a computation device 110 in addition to the real time electronic display device 100 and projection image input device 112 providing input to the computational device 110 to calculating the projectional images for display to providing a realistic, minimum distortion three dimensional illusion to the viewer by coincide the viewer's eyepoint with the projection image eyepoint.
  • the system can further comprise an image enlargement reduction input device 115, or an image rotation input device 117, or an image movement device 119 to allow the viewer to adjust the view of the projection images.
  • the horizontal perspective display system promotes horizontal perspective projection viewing by providing the viewer with the means to adjust the displayed images to maximize the illusion viewing experience.
  • the horizontal perspective display comprising a real time electronic display capable of re-drawing the projected image, together with a viewer's input device to adjust the horizontal perspective image.
  • the horizontal perspective display of the present invention can ensure the minimum distortion in rendering the three dimension illusion from the horizontal perspective method.
  • the input device can be manually operated where the viewer manually inputs his or her eyepoint location, or change the projection image eyepoint to obtain the optimum three dimensional illusions.
  • the input device can also be automatically operated where the display automatically tracks the viewer's eyepoint and adjust the projection image accordingly.
  • the horizontal perspective display system removes the constraint that the viewers keeping their heads in relatively fixed positions, a constraint that create much difficulty in the acceptance of precise eyepoint location such as horizontal perspective or hologram display.
  • the horizontal perspective display system can further a computation device in addition to the real time electronic display device and projection image input device providing input to the computational device to calculating the projectional images for display to providing a realistic, minimum distortion three dimensional illusion to the viewer by coincide the viewer's eyepoint with the projection image eyepoint.
  • the system can further comprise an image enlargement/reduction input device, or an image rotation input device, or an image movement device to allow the viewer to adjust the view of the projection images.
  • the input device can be operated manually or automatically.
  • the input device can detect the position and orientation of the viewer eyepoint, to compute and to project the image onto the display according to the detection result.
  • the input device can be made to detect the position and orientation of the viewer's head along with the orientation of the eyeballs.
  • the input device can comprise an infrared detection system to detect the position the viewer's head to allow the viewer freedom of head movement.
  • Other embodiments of the input device can be the triangulation method of detecting the viewer eyepoint location, such as a CCD camera providing position data suitable for the head tracking objectives of the invention.
  • the input device can be manually operated by the viewer, such as a keyboard, mouse, trackball, joystick, or the like, to indicate the correct display of the horizontal perspective display images.
  • the head or eye-tracking system can comprise a base unit and a head-mounted sensor on the head of the viewer.
  • the head-mounted sensor produces signals showing the position and orientation of the viewer in response to the viewer's head movement and eye orientation. These signals can be received by the base unit and are used to compute the proper three dimensional projection images.
  • the head or eye tracking system can be infrared cameras to capture images of the viewer's eyes. Using the captured images and other techniques of image processing, the position and orientation of the viewer's eyes can be determined, and then provided to the base unit. The head and eye tracking can be done in real time for small enough time interval to provide continous viewer's head and eye tracking.
  • the Hands-On Simulator employing the open space characteristics of the horizontal perspective, together with a number of new computer hardware and software elements and processes that together to create a "Hands-On Simulator".
  • the Hands-On Simulator generates a totally new and unique computing experience in that it enables an end user to interact physically and directly (Hands-On) with real-time computer-generated 3D graphics (Simulations), which appear in open space above the viewing surface of a display device, i.e. in the end user's own physical space.
  • the computer hardware viewing surface is situated horizontally, such that the end-user's line of sight is at a 45° angle to the surface. Typically, this means that the end user is standing or seated vertically, and the viewing surface is horizontal to the ground. Note that although the end user can experience hands-on simulations at viewing angles other than 45° (e.g. 55°, 30° etc.), it is the optimal angle for the brain to recognize the maximum amount of spatial information in an open space image. Therefore, for simplicity's sake, we use "45°” throughout this document to mean “an approximate 45 degree angle”. Further, while horizontal viewing surface is preferred since it simulates viewers' experience with the horizontal ground, any viewing surface could offer similar three dimensional illusion experience.
  • the horizontal perspective illusion can appear to be hanging from a ceiling by projecting the horizontal perspective images onto a ceiling surface, or appear to be floating from a wall by projecting the horizontal perspective images onto a vertical wall surface.
  • the hands-on simulations are generated within a 3D graphics engines' view volume, creating two new elements, the "Hands-On Volume” and the "Inner-Access Volume.”
  • the Hands-On Volume is situated on and above the physical viewing surface.
  • the Inner- Access Volume is located underneath the viewing surface and simulations within this volume appear inside the physically viewing device.
  • simulations generated within the Inner- Access Volume do not share the same physical space with the end user and the images therefore cannot be directly, physically manipulated by hands or handheld tools. That is, they are manipulated indirectly via a computer mouse or a joystick.
  • This disclosed Hands-On Simulator can lead to the end user's ability to directly, physically manipulate simulations because they co-inhabit the end-user's own physical space.
  • To accomplish this requires a new computing concept where computer-generated world elements have a 1 :1 correspondence with their physical real- world equivalents; that is, a physical element and an equivalent computer-generated element occupy the same space and time. This is achieved by identifying and establishing a common "Reference Plane", to which the new elements are synchronized.
  • Synchronization with the Reference Plane forms the basis to create the 1:1 correspondence between the "virtual" world of the simulations, and the "real” physical world.
  • the 1:1 correspondence insures that images are properly displayed: What is on and above the viewing surface appears on and above the surface, in the Hands-On Volume; what is underneath the viewing surface appears below, in the Inner- Access Volume. Only if this 1:1 correspondence and synchronization to the Reference Plane are present can the end user physically and directly access and interact with simulations via their hands or hand-held tools.
  • the present invention simulator further includes a real-time computer-generated 3D- graphics engine as generally described above, but using horizontal perspective projection to display the 3D images.
  • a real-time computer-generated 3D- graphics engine as generally described above, but using horizontal perspective projection to display the 3D images.
  • One major different between the present invention and prior art graphics engine is the projection display.
  • Existing 3D-graphics engine uses central- perspective and therefore a vertical plane to render its view volume while in the present invention simulator, a "horizontal" oriented rendering plane vs. a "vertical” oriented rendering plane is required to generate horizontal perspective open space images.
  • the horizontal perspective images offer much superior open space access than central perspective images.
  • One of the invented elements in the present invention hands-on simulator is the 1:1 correspondence of the computer-generated world elements and their physical real- world equivalents.
  • this 1 :1 correspondence is a new computing concept that is essential for the end user to physically and directly access and interact with hands-on simulations.
  • This new concept requires the creation of a common physical Reference Plane, as well as, the formula for deriving its unique x, y, z spatial coordinates. To determine the location and size of the Reference Plane and its specific coordinates requires understanding the following.
  • a computer monitor or viewing device is made of many physical layers, individually and together having thickness or depth.
  • Figure 13 contains a conceptual side-view of typical CRT-type viewing device.
  • the top layer of the monitor's glass surface is the physical "View Surface”
  • the phosphor layer, where images are made is the physical "Image Layer”.
  • the View Surface and the Image Layer are separate physical layers located at different depths or z coordinates along the viewing device's z axis.
  • To display an image the CRT's electron gun excites the phosphors, which in turn emit photons. This means that when you view an image on a CRT, you are looking along its z axis through its glass surface, like you would a window, and seeing the light of the image coming from its phosphors behind the glass.
  • Figure 14 we use the same architectural technique for drawing images with horizontal perspective as previously illustrated in Figure 10.
  • the middle block in Figure 14 does not correctly appear on the View Surface.
  • the bottom of the middle block is located correctly on the horizontal drawing/viewing plane, i.e. a piece of paper's View Surface.
  • the phosphor layer i.e. where the image is made, is located behind the CRT's glass surface. Therefore, the bottom of the middle block is incorrectly positioned behind or underneath the View Surface.
  • Figure 15 shows the proper location of the three blocks on a CRT-type viewing device. That is, the bottom of the middle block is displayed correctly on the View Surface and not on the Image Layer. To make this adjustment the z coordinates of the View Surface and Image Layer are used by the Simulation Engine to correctly render the image.
  • the unique task of correctly rendering an open space image on the View Surface vs. the Image Layer is critical in accurately mapping the simulation images to the real world space.
  • a viewing device's View Surface is the correct physical location to present open space images. Therefore, the View Surface, i.e. the top of the viewing device's glass surface, is the common physical Reference Plane. But only a subset of the View Surface can be the Reference Plane because the entire View Surface is larger than the total image area.
  • Figure 16 shows an example of a complete image being displayed on a viewing device's View Surface. That is, the blue image, including the bear cub, shows the entire image area, which is smaller than the viewing device's View Surface.
  • the Image Layer is given a z coordinate of 0.
  • the View Surface is the distance along the z axis from the Image Layer the Reference Plane's z coordinate is equal to the View Surface, i.e. its distance from the Image Layer.
  • the x and y coordinates, or size of the Reference Plane can be determined by displaying a complete image on the viewing device and measuring the length of its x and y axis.
  • Reference Plane Calibration The concept of the common physical Reference Plane is a new inventive concept. Therefore, display manufactures may not supply or even know its coordinates. Thus a "Reference Plane Calibration" procedure might need to be performed to establish the Reference Plane coordinates.
  • This calibration procedure provides the end user with a number of orchestrated images that s/he interacts. The end-user's response to these images provides feedback to the Simulation Engine such that it can identify the correct size and location of the Reference Plane. When the end user is satisfied and completes the procedure the coordinates are saved in the end user's personal profile.
  • One element of the present invention horizontal perspective projection hands-on simulator is a computer-generated "Angled Camera” point, shown in Figure 17.
  • the camera point is initially located at an arbitrary distance from the Horizontal Plane and the camera's line-of-site is oriented at a 45° angle looking through the center.
  • the position of the Angled Camera in relation to the end-user's eye is critical to generating simulations that appear in open space on and above the surface of the viewing device.
  • the computer-generated x, y, z coordinates of the Angled Camera point form the vertex of an infinite "pyramid", whose sides pass through the x, y, z coordinates of the Reference/Horizontal Plane.
  • Figure 18 illustrates this infinite pyramid, which begins at the Angled Camera point and extending through the Far Clip Plane.
  • These unique view volumes are called Hands-On and the Inner-Access Volume, and are not shown in Figure 18.
  • the dimensions of these volumes and the planes that define them are based on their locations within the pyramid.
  • FIG 19 illustrates a plane, called Comfort Plane, together with other display elements.
  • the Comfort Plane is one of six planes that define the new Hands-On Volume, and of these planes it is closest to the Angled Camera point and parallel to the Reference Plane.
  • the Comfort Plane is appropriately named because its location within the pyramid determines the end-user's personal comfort, i.e. how their eyes, head, body, etc. are situated while viewing and interacting with simulations.
  • the end user can adjust the location of the Comfort Plane based on their personal visual comfort through a "Comfort Plane Adjustment" procedure. This procedure provides the end user with orchestrated simulations within the Hands-On Volume, and enables them to adjust the location of the Comfort Plane within the pyramid relative to the Reference Plane.
  • the present invention simulator further defines a "Hands-On Volume", shown in Figure 20.
  • the Hands-On Volume is where you can reach your hand in and physically "touch” a simulation. You can envision this by imagining you are sifting in front of a horizontally oriented computer monitor and using the Hands-On Simulator. If you place your hand several inches above the surface of the monitor, you are putting your hand inside both the physical and computer-generated Hands-On Volume at the same time.
  • the Hands-On Volume exists within the pyramid and are between and inclusive of the Comfort Planes and the Reference Horizontal Planes.
  • the Inner- Access Volume exists below or inside the physical viewing device. For this reason, an end user cannot directly interact with 3D objects located within the Inner- Access Volume via their hand or hand-held tools. But they can interact in the traditional sense with a computer mouse, joystick, or other similar computer peripheral.
  • An "Inner Plane” is further defined, located immediately below and are parallel to the Reference/Horizontal Plane within the pyramid as shown in Figure 21.
  • the Inner Plane, along with the Bottom Plane, is two of the six planes within the pyramid that define the Inner- Access Volume.
  • the Bottom Plane (shown in Figure 22) is farthest away from the Angled Camera point, but it is not to be mistaken for the Far Clip plane.
  • the Bottom Plane is also parallel to the Reference/Horizontal Plane and is one of the six planes that define the Inner-Access Volume (Figure 23).
  • the end-user's preferred viewing distance to the bottom of the viewing pyramid determines the location of these planes.
  • One way the end user can adjust the location of the Bottom Planes is through a "Bottom Plane Adjustment" procedure. This procedure provides the end user with orchestrated simulations within the Inner-Access Volume and enables them to interact and adjust the location of the Bottom Plane relative to the physical Reference/Horizontal Plane. When the end user completes the procedure the Bottom Plane's coordinates are saved in the end-user's personal profiles.
  • the end user For the end user to view open space images on their physical viewing device it must be positioned properly, which usually means the physical Reference Plane is placed horizontally to the ground. Whatever the viewing device's position relative to the ground, the Reference/Horizontal Plane must be at approximately a 45° angle to the end-user's line-of-sight for optimum viewing.
  • One way the end user might perform this step is to position their CRT computer monitor on the floor in a stand, so that the Reference/Horizontal Plane is horizontal to the floor. This example use a CRT-type computer monitor, but it could be any type of viewing device, placed at approximately a 45° angle to the end-user's line-of-sight.
  • the real-world coordinates of the "End-User's Eye” and the computer-generated Angled Camera point must have a 1 :1 correspondence in order for the end user to properly view open space images that appear on and above the Reference/Horizontal Plane ( Figure 24).
  • One way to do this is for the end user to supply the Simulation Engine with their eye's real-world x, yr. z location and line-of-site information relative to the center of the physical Reference/Horizontal Plane. For example, the end user tells the Simulation Engine that their physical eye will be located 12 inches up, and 12 inches back, while looking at the center of the Reference/Horizontal Plane.
  • the Simulation Engine maps the computer-generated Angled Camera point to the End-User's Eye point physical coordinates and line-of-sight.
  • the present invention horizontal perspective hands-on simulator employs the horizontal perspective projection to mathematically projected the 3D objects to the Hands-On and Inner- Access Volumes.
  • the existence of a physical Reference Plane and the knowledge of its coordinates are essential to correctly adjusting the Horizontal Plane's coordinates prior to projection.
  • This adjustment to the Horizontal Plane enables open space images to appear to the end user on the View Surface vs. the Image Layer by taking into account the offset between the Image Layer and the View Surface, which are located at different values along the viewing device's z axis.
  • the three dimensional x, y, z point of the object becomes a two-dimensional x, y point of the Horizontal Plane (see Figure 25).
  • Projection lines often intersect more than one 3D object coordinate, but only one object x, y, z coordinate along a given projection line can become a Horizontal Plane x, y point.
  • the formula to determine which object coordinate becomes a point on the Horizontal Plane is different for each volume. For the Hands-On Volume it is the object coordinate of a given projection line that is farthest from the Horizontal Plane.
  • the Inner- Access Volume it is the object coordinate of a given projection line that is closest to the Horizontal Plane.
  • the Hands-On Volume's 3D object point is used.
  • Figure 25 is an illustration of the present invention Simulation Engine that includes the new computer-generated and real physical elements as described above. It also shows that a real-world element and its computer-generated equivalent are mapped 1 :1 and together share a common Reference Plane.
  • the full implementation of this Simulation Engine results in a Hands-On Simulator with real-time, computer-generated 3D-graphics appearing in open space on and above a viewing device's surface, which is oriented approximately 45° to the end-user's line-of-sight.
  • the Hands-On Simulator further involves adding completely new elements and processes and existing stereoscopic 3D computer hardware.
  • Multi- View provides the end user with multiple and/or separate left-and right-eye views of the same simulation.
  • the simulator further includes a new computer-generated "time dimension" element, called “Sl-time”.
  • SI is an acronym for "Simulation Image” and is one complete image displayed on the viewing device.
  • Si-Time is the amount of time the Simulation Engine uses to completely generate and display one Simulation Image. This is similar to a movie projector where 24 times a second it displays an image. Therefore, 1/24 of a second is required for one image to be displayed by the projector But SI-Time is variable, meaning that depending on the complexity of the view volumes it could take 1/120* or l A a second for the Simulation Engine to complete just one SI.
  • the simulator also includes a new computer-generated "time dimension" element, called "EV-time” and is the amount of time used to generate a one "Eye- View".
  • EV-time a new computer-generated "time dimension” element
  • the simulator also includes a new computer-generated "time dimension" element, called "EV-time” and is the amount of time used to generate a one "Eye- View".
  • EV-time is the amount of time used to generate a one "Eye- View”.
  • the Simulation Engine needs to create one left-eye view and one right-eye view for pu ⁇ oses of providing the end user with a stereoscopic 3D experience. If it takes the Simulation Engine l A a second to generate the left-eye view then the first EV-Time period is V* a second. If it takes another l A second to generate the right-eye view then the second EV-Time period is also ⁇ second. Since the Simulation Engine was generating a separate left and right eye view of the same Simulation Image the total SI- Time is
  • Figure 26 helps illustrate these two new time dimension elements. It is a conceptual drawing of what is occurring inside the Simulation Engine when it is generating a two- eye view of a Simulated Image.
  • the computer-generated person has both eyes open, a requirement for stereoscopic 3D viewing, and therefore sees the bear cub from two separate vantage points, i.e. from both a right-eye view and a left-eye view. These two separate views are slightly different and offset because the average person's eyes are about 2 inches apart. Therefore, each eye sees the world from a separate point in space and the brain puts them together to make a whole image. This is how and why we see the real world in stereoscopic 3D.
  • Figure 27 is a very high-level Simulation Engine blueprint focusing on how the computer-generated person's two eye views are projected onto the Horizontal Plane and then displayed on a stereoscopic 3D capable viewing device.
  • Figure 26 represents one complete SI-Time period. If we use the example from step 3 above, SI-Time takes one second. During this one second of SI-Time the Simulation Engine needs to generate two different eye views, because in this example the stereoscopic 3D viewing device requires a separate left- and right-eye view. There are existing stereoscopic 3D viewing devices that require more than a separate left- and right-eye view. But because the method described here can generate multiple views it works for these devices as well.
  • EV-Time-1 is the time period used by the Simulation Engine to complete the first eye (right-eye) view of the computer- generated person. This is the job for this step, which is within EV-Time-1, and using the Angled Camera at coordinate x, y, z, the Simulation Engine completes the rendering and display of the right-eye view of a given Simulation Image.
  • the Simulation Engine starts the process of rendering the computer-generated person's second eye (left-eye) view.
  • the illustration in the lower left of Figure 27 shows the Angled Camera point for the left eye at time element "EV-Time-2". That is, this second eye view is completed during EV- Time-2.
  • step 5 makes an adjustment to the Angled Camera point. This is illustrated in Figure 27 by the left eye's x coordinate being incremented by two inches. This difference between the right eye's x value and the left eye's x + 2" is what provides the two-inch separation between the eyes, which is required for stereoscopic 3D viewing.
  • the distances between people's eyes vary but in the above example we are using the average of 2 inches. It is also possible for the end user to supply the Simulation Engine with their personal eye separation value. This would make the x value for the left and right eyes highly accurate for a given end user and thereby improve the quality of their stereoscopic 3D view.
  • the Simulation Engine Once the Simulation Engine has incremented the Angled Camera point's x coordinate by two inches, or by the personal eye separation value supplied by the end user, it completes the rendering and display of the second (left-eye) view. This is done by the Simulation Engine within the EV-Time-2 period using the Angled Camera point coordinate x ⁇ 2", y, z and the exact same Simulation Image rendered. This completes one SI-Time period.
  • the Simulation Engine continues to display the left- and right-eye images, as described above, until it needs to move to the next SI-Time period.
  • the job of this step is to determine if it is time to move to a new SI-Time period, and if it is, then increment SI-Time.
  • An example of when this may occur is if the bear cub moves his paw or any part of his body Then a new and second Simulated Image would be required to show the bear cub in its new position. This new Simulated Image of the bear cub, in a slightly different location, gets rendered during a new SI-Time period or SI-Time-2.
  • This new SI-time-2 period will have its own EV-Time-1 and EV-Time-2, and therefore the simulation steps described above will be repeated during Sl-time-2.
  • This process of generating multiple views via the nonstop incrementing of SI-Time and its EV-Times continues as long as the Simulation Engine is generating real-time simulations in stereoscopic 3D.
  • Multi-View provides the end user with multiple and/or separate left- and right-eye views of the same simulation. Multi-View capability is a significant visual and interactive improvement over the single eye view.
  • the present invention also allows the viewer to move around the three dimensional display and yet suffer no great distortion since the display can track the viewer eyepoint and re-display the images correspondingly, in contrast to the conventional prior art three dimensional image display where it would be projected and computed as seen from a singular viewing point, and thus any movement by the viewer away from the intended viewing point in space would cause gross distortion.
  • the display system can further comprise a computer capable of re-calculate the projected image given the movement of the eyepoint location.
  • the horizontal perspective images can be very complex, tedious to create, or created in ways that are not natural for artists or cameras, and therefore require the use of a computer system for the tasks.
  • To display a three-dimensional image of an object with complex surfaces or to create animation sequences would demand a lot of computational power and time, and therefore it is a task well suited to the computer.
  • Three dimensional capable electronics and computing hardware devices and real-time computer-generated three dimensional computer graphics have advanced significantly recently with marked innovations in visual, audio and tactile systems, and have producing excellent hardware and software products to generate realism and more natural computer-human interfaces.
  • the horizontal perspective display system of the present invention are not only in demand for entertainment media such as televisions, movies, and video games but are also needed from various fields such as education (displaying three-dimensional structures), technological training (displaying three-dimensional equipment).
  • entertainment media such as televisions, movies, and video games
  • various fields such as education (displaying three-dimensional structures), technological training (displaying three-dimensional equipment).
  • three-dimensional image displays which can be viewed from various angles to enable observation of real objects using object-like images.
  • the horizontal perspective display system is also capable of substitute a computer-generated reality for the viewer observation.
  • the systems may include audio, visual, motion and inputs from the user in order to create a complete experience of three dimensional illusions.
  • the input for the horizontal perspective system can be two dimensional image, several images combined to form one single three dimensional image, or three dimensional model.
  • the three dimensional image or model conveys much more information than that a two dimensional image and by changing viewing angle, the viewer will get the impression of seeing the same object from different perspectives continuously.
  • the horizontal perspective display can further provide multiple views or "Multi- View” capability.
  • Multi- View provides the viewer with multiple and/or separate left-and right-eye views of the same simulation.
  • Multi- View capability is a significant visual and interactive improvement over the single eye view.
  • Multi- View mode both the left eye and right eye images are fused by the viewer's brain into a single, three-dimensional illusion.
  • the problem of the discrepancy between accommodation and convergence of eyes, inherent in stereoscopic images, leading to the viewer's eye fatigue with large discrepancy, can be reduced with the horizontal perspective display, especially for motion images, since the position of the viewer's gaze point changes when the display scene changes.
  • Multi-View devices that can be used in the present invention include methods with glasses such as anaglyph method, special polarized glasses or shutter glasses, methods without using glasses such as a parallax stereogram, a lenticular method, and mirror method (concave and convex lens).
  • a display image for the right eye and a display image for the left eye are respectively superimpose-displayed in two colors, e.g., red and blue, and observation images for the right and left eyes are separated using color filters, thus allowing a viewer to recognize a stereoscopic image.
  • the images are displayed using horizontal perspective technique with the viewer looking down at an angle.
  • the eyepoint of the projected images has to be coincide with the eyepoint of the viewer, and therefore the viewer input device is essential in allowing the viewer to observe the three dimensional horizontal perspective illusion. From the early days of the anaglyph method, there are much improvements such as the spectrum of the red/blue glasses and display to generate much more realism and comfort to the viewers.
  • the left eye image and the right eye image are separated by the use of mutually extinguishing polarizing filters such as orthogonally linear polarizer, circular polarizer, elliptical polarizer.
  • the images are normally projected onto screens with polarizing filters and the viewer is then provided with corresponding polarized glasses.
  • the left and right eye images appear on the screen at the same time, but only the left eye polarized light is transmitted through the left eye lens of the eyeglasses and only the right eye polarized light is transmitted through the right eye lens.
  • Another way for stereoscopic display is the image sequential system.
  • the images are displayed sequentially between left eye and right eye images rather than superimposing them upon one another, and the viewer's lenses are synchronized with the screen display to allow the left eye to see only when the left image is displayed, and the right eye to see only when the right image is displayed.
  • the shuttering of the glasses can be achieved by mechanical shuttering or with liquid crystal electronic shuttering.
  • display images for the right and left eyes are alternately displayed on a CRT in a time sharing manner, and observation images for the right and left eyes are separated using time sharing shutter glasses which are opened/closed in a time sharing manner in synchronism with the display images, thus allowing an observer to recognize a stereoscopic image.
  • optical method Other way to display stereoscopic images is by optical method.
  • display images for the right and left eyes which are separately displayed on a viewer using optical means such as prisms, mirror, lens, and the like, are superimpose-displayed as observation images in front of an observer, thus allowing the observer to recognize a stereoscopic image.
  • Large convex or concave lenses can also be used where two image projectors, projecting left eye and right eye images, are providing focus to the viewer's left and right eye respectively.
  • a variation of the optical method is the lenticular method where the images form on cylindrical lens elements or two dimensional array of lens elements.
  • Figure 27 is a horizontal perspective display focusing on how the computer-generated person's two eye views are projected onto the Horizontal Plane and then displayed on a stereoscopic 3D capable viewing device.
  • Figure 27 represents one complete display time period. During this display time period, the horizontal perspective display needs to generate two different eye views, because in this example the stereoscopic 3D viewing device requires a separate left- and right-eye view.
  • the illustration in the upper left of Figure 27 shows the Angled Camera point for the right eye after the first (right) eye-view to be generated.
  • the horizontal perspective display starts the process of rendering the computer- generated person's second eye (left-eye) view.
  • the illustration in the lower left of Figure 27 shows the Angled Camera point for the left eye after the completion of this time.
  • the horizontal perspective display makes an adjustment to the Angled Camera point. This is illustrated in Figure 27 by the left eye's x coordinate being incremented by two inches. This difference between the right eye's x value and the left eye's x + 2" is what provides the two-inch separation between the eyes, which is required for stereoscopic 3D viewing.
  • the distances between people's eyes vary but in the above example we are using the average of 2 inches. It is also possible for the view to supply the horizontal perspective display with their personal eye separation value. This would make the x value for the left and right eyes highly accurate for a given viewer and thereby improve the quality of their stereoscopic 3D view.
  • the rendering continues by displaying the second (left-eye) view.
  • the horizontal perspective display continues to display the left- and right-eye images, as described above, until it needs to move to the next display time period.
  • An example of when this may occur is if the bear cub moves his paw or any part of his body. Then a new and second Simulated Image would be required to show the bear cub in its new position.
  • This new Simulated Image of the bear cub in a slightly different location, gets rendered during a new display time period. This process of generating multiple views via the nonstop incrementing of display time continues as long as the horizontal perspective display is generating realtime simulations in stereoscopic 3D.
  • the display rate is the number of images per second that the display uses to completely generate and display one image. This is similar to a movie projector where 24 times a second it displays an image. Therefore, 1/24 of a second is required for one image to be displayed by the projector. But the display time could be a variable, meaning that depending on the complexity of the view volumes it could take 1/12 or l A a second for the computer to complete just one display image. Since the display was generating a separate left and right eye view of the same image, the total display time is twice the display time for one eye image.
  • Figure 28 shows a horizontal plane as related to both central perspective andhorizontal perspective.
  • the present invention hands-on simulator further includes technologies employed in computer "peripherals".
  • Figure 29 shows examples of such Peripherals with six degrees of freedom, meaning that their coordinate system enables them to interact at any given point in an (x, y, z) space.
  • the simulator creates a "Peripheral Open-Access Volume," for each Peripheral the end-user requires, such as the Space Glove in Figure 29.
  • Figure 30 is a high-level illustration of the Hands-On Simulation Tool, focusing on how a Peripheral's coordinate system is implemented within the Hands-On Simulation Tool.
  • the new Peripheral Open-Access Volume which as an example in Figure 30 is labeled "Space Glove," is mapped one-to-one with the "Open- Access Real Volume” and "Open- Access Computer-generated Volume.”
  • the key to achieving a precise one-to-one mapping is to calibrate the Peripheral's volume with the Common Reference, which is the physical View surface, located at the viewing surface of the display device.
  • Some Peripherals provide a mechanism that enables the Hands-On Simulation Tool to perform this calibration without any end-user involvement. But if calibrating the Peripheral requires external intervention than the end-user will accomplish this through an "Open- Access Peripheral Calibration" procedure. This procedure provides the end- user with a series of Simulations within the Hands-On Volume and a user-friendly interface that enables them to adjusting the location of the Peripheral's volume until it is in perfect synchronization with the View surface. When the calibration procedure is complete, the Hands-On Simulation Tool saves the information in the end-user's personal profile.
  • the Hands-On Simulation Tool will continuously track and map the Peripheral's volume to the Open-Access Volumes.
  • the Hands-On Simulation Tool modifies each Hands-On Image it generates based on the data in the Peripheral's volume.
  • the end result of this process is the end-user's ability to use any given Peripheral to interact with Simulations within the Hands-On Volume generated in real-time by the Hands-On Simulation Tool.
  • the peripherals linking to the simulator, the user can interact with the display model.
  • the Simulation Engine can get the inputs from the user through the peripherals, and manipulate the desired action.
  • the simulator can provide proper interaction and display.
  • the invention Hands-On Simulator then can generate a totally new and unique computing experience in that it enables an end user to interact physically and directly (Hands-On) with real-time computer-generated 3D graphics (Simulations), which appear in open space above the viewing surface of a display device, i.e. in the end user's own physical space.
  • the peripheral tracking can be done through camera triangulation or through infrared tracking devices.
  • the simulator can further include 3D audio devices for "SIMULATION RECOGNITION & 3D AUDIO ".
  • 3D audio devices for "SIMULATION RECOGNITION & 3D AUDIO ".
  • Triangulation is a process employing trigonometry, sensors, and frequencies to "receive" data from simulations in order to determine their precise location in space. It is for this reason that triangulation is a mainstay of the cartography and surveying industries where the sensors and frequencies they use include but are not limited to cameras, lasers, radar, and microwave.
  • 3D Audio also uses triangulation but in the opposite way 3D Audio "sends" or projects data in the form of sound to a specific location. But whether you're sending or receiving data the location of the simulation in three-dimensional space is done by triangulation with frequency receiving sending devices.
  • the device can effectively emulate the position of the sound source.
  • the sounds reaching the ears will need to be isolated to avoid interference.
  • the isolation can be accomplished by the use of ea ⁇ hones or the like.
  • Figure 31 shows an end-user looking at a Hands-On Image of a bear cub. Since the cub appears in open space above the viewing surface the end-user can reach in and manipulate the cub by hand or with a handheld tool. It is also possible for the end-user to view the cub from different angles, as they would in real life. This is accomplished though the use of triangulation where the three real- world cameras continuously send images from their unique angle of view to the Hands-On Simulation Tool. This camera data of the real world enables the Hands-On Simulation Tool to locate, track, and map the end-user's body and other real-world simulations positioned within and around the computer monitor's viewing surface ( Figure 32).
  • Figure 33 also shows the end-user viewing and interacting with the bear cub, but it includes 3D sounds emanating from the cub's mouth. To accomplish this level of audio quality requires physically combining each of the three cameras with a separate speaker, as shown in Figure 32.
  • the cameras' data enables the Hands-On Simulation Tool to use triangulation in order to locate, track, and map the end-user's "left and right ear". And since the Hands-On Simulation Tool is generating the bear cub as a computer-generated Hands-On Image it knows the exact location of the cub's mouth.
  • the Hands-On Simulation Tool uses triangulation to sends data, by modifying the spatial characteristics of the audio, making it appear that 3D sound is emanating from the cub's computer-generated mouth.
  • each camera/speaker device Take these new camera/speaker devices and attach or place them nearby a viewing device, such as a computer monitor as previously shown in Figure 32.
  • a viewing device such as a computer monitor as previously shown in Figure 32.
  • x, y, z real-world
  • line- of-sight line- of-sight
  • frequency receiving/sending volume To understand these parameters think of using a camcorder and looking through its view finder When you do this the camera has a specific location in space, is pointed in a specific direction, and all the visual frequency information you see or receive through the view finder is its "frequency receiving volume”.
  • Triangulation works by separating and positioning each camera/speaker device such that their individual frequency receiving/sending volumes overlap and cover the exact same area of space.
  • Figure 34 is a simplified illustration of the complete Open- Access Camera Model and will assist in explaining each of the additional steps required to accomplish the scenarios described in Figures 32 and 33 above.
  • the simulator then performs simulation recognition by continuously locating and tracking the end-user's "left and right eye” and their "line-of-sight", continuously map the real- world left and right eye coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the computer-generated cameras coordinates to match the real-world eye coordinates that are being located, tracked, and mapped.
  • This enables the real-time generation of Simulations within the Hands-On Volume based on the exact location of the end-user's left and right eye. Allowing the enft-user to " freely m ⁇ ve " fhe ⁇ f ' head ahd d ⁇ around the Hands-On Image without distortion.
  • the simulator then perform simulation recognition by continuously locating and tracking the end-user's "left and right ear” and their "line-of-hearing", continuously map the real-world left- and right-ear coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the 3D Audio coordinates to match the real-world ear coordinates that are being located, tracked, and mapped.
  • This enables the real-time generation of Open- Access sounds based on the exact location of the end-user's left and right ears. Allowing the end-user to freely move their head and still hear Open- Access sounds emanating from their correct location.
  • the simulator then perform simulation recognition by continuously locating and tracking the end-user's "left and right hand" and their "digits," i.e. fingers and thumbs, continuously map the real-world left and right hand coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the Hands- On Image coordinates to match the real-world hand coordinates that are being located, tracked, and mapped.
  • This enables the real-time generation of Simulations within the Hands-On Volume based on the exact location of the end-user's left and right hands allowing the end-user to freely interact with Simulations within the Hands-On Volume.
  • the simulator then perform simulation recognition by continuously locating and tracking "handheld tools", continuously map these real-world handheld tool coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the Hands-On Image coordinates to match the real-world handheld tool coordinates that are being located, tracked, and mapped.
  • This enables the real-time generation of Simulations within the Hands-On ' Volume based on the exact ocation ' of the handheld tools allowing the end-user to freely interact with Simulations within the Hands-On Volume.
  • Figure 35 is intended to assist in further explaining unique discoveries regarding the new Open- Assess Camera Model and handheld tools.
  • Figure 35 is a simulation of and end-user interacting with a Hands-On Image using a handheld tool. The scenario being illustrated is the end-user visualizing large amounts of financial data as a number of interrelated Open- Access 3D simulations. The end-user can probe and manipulated the Open- Access simulations by using a handheld tool, which in Figure 35 looks like a pointing device.
  • a "computer-generated attachment” is mapped in the form of an Open- Access computer-generated simulation onto the tip of a handheld tool, which in Figure 35 appears to the end-user as a computer-generated "eraser".
  • the end-user can of course request that the Hands-On Simulation Tool map any number of computer-generated attachments to a given handheld tool. For example, there can be different computer- generated attachments with unique visual and audio characteristics for cutting, pasting, welding, painting, smearing, pointing, grabbing, etc. And each of these computer- generated attachments would act and sound like the real device they are simulating when they are mapped to the tip of the end-user's handheld tool.
  • the present invention further discloses a Multi-Plane display comprising a horizontal perspective display together with a non-horizontal central perspective display.
  • Figure 36 illustrates an example of the present invention Multi-Plane display in which the Multi- Plane display is a computer monitor that is approximately "L" shaped when open. The end-user views the L-shaped computer monitor from ' its concave side and at approximately a 45° angle to the bottom of the "L," as shown in Figure 36. From the end-user's point of view the entire L-shaped computer monitor appears as one single and seamless viewing surface. The bottom L of the display, positioned horizontally, shows horizontal perspective image, and the other branch of the L display shows central perspective image. The edge is the two display segments is preferably smoothly joined and can also having a curvilinear projection to connect the two displays of horizontal perspective and central perspective.
  • the Multi-Plane display can be made with one or more physical viewing surfaces.
  • the vertical leg of the "L” can be one physical viewing surface, such as flat panel display
  • the horizontal leg of the "L” can be a separate flat panel display.
  • the edge of the two display segments can be a non-display segment and therefore the two viewing surface are not continuous.
  • Each leg of a Multi-Plane display is called a viewing plane and as you can see in the upper left of Figure 36 there is a vertical viewing plane and a horizontal viewing plane where a central perspective image is generated on the vertical plane and a horizontal perspective image is generated on the horizontal plane, and then blend the two images where the planes meet, as illustrated in the lower right of Figure 36.
  • Figure 36 also illustrates that a Multi-Plane display is capable of generating multiple views. Meaning that it can display single-view images, i.e. a one-eye perspective like the simulation in the upper left, and/or multi- view images, i.e. separate right and left eye views like the simulation in the lower right. And when the L-shaped computer monitor is not being used ' by the end-user it can be closed "an l ' f ⁇ o- ' lili e the simulation in the lower left.
  • Figure 37 is a simplified illustration of the present invention Multi-Plane display.
  • the upper right of Figure 37 is an example of a single- view image of a bear cub that is displayed on an L-shaped computer monitor.
  • Normally a single-view or one eye image would be generated with only one camera point, but as you can see there are at least two camera points for the Multi-Plabe display even though this is a single-view example.
  • One camera point is for the horizontal perspective image, which is displayed on the horizontal surface, and the other camera point is for the central perspective image, which is displayed on the vertical surface.
  • the vertical viewing plane of the L-shaped monitor is the display surface for the central perspective images, and thus there is a need to define another common reference plane for this surface.
  • the common reference plane is the plane where the images are display, and the computer need to keep track of this plane for the synchronization of the locations of the displayed images and the real physical locations.
  • the Simulation can to generate the three dimansional images, a horizontal perspective image using (OSI) camera eyepoint, and a central perspective image using (CPI) camera eyepoint.
  • the multi-plane display system can further include a curvilinear connection display section to blend the horizontal perspective and the central perspective images together at the location of the seam in the "L," as shown at the bottom of Figure 37.
  • the multi-plane display system can continuously update and display what appears to be a single L-shaped image on the L-shaped Multi-Plane device.
  • the multi-plane display system can comprise multiple display surfaces together with multiple curvilinear blending sections as shown in Fig. 38.
  • the multiple display surfaces can be a flat wall, multiple adjacent flat walls, a dome, and a curved wraparound panel.
  • the present invention multi-plane display system thus can simultaneously projecting a plurality of three dimensional images onto multiple display surfaces, one of which is a horizontal perspective image. Further, it can be a stereoscopic multiple display system allowing viewers to use their stereoscopic vision for three dimensional image presentation.
  • the multi-plane display system comprises at least two display surfaces
  • various requirements need to be addressed to ensure high fidelity in the three dimensional image projection.
  • the display requirements are typically geometric accuracy, to ensure that objects and features of the image to be correctly positioned, edge match accuracy, to ensure continuity between display surfaces, no blending variation, to ensure no variation in luminance in the blending section of various display surfaces, and field of view, to ensure a continuous image from the eyepoint of the viewer.
  • the blending section of the multi-plane display system is preferably a curve surface
  • some distortion correction could be applied in order for the image projected onto the blending section surface to appear correct to the viewer.
  • There are various solutions for providing distortion correction to a display system such as using a test pattern image, designing the image projection system for the specific curved blending display section, using special video hardware, utilizing a piecewise-linear approximation for the curved blending section.
  • Still another distortion correction solution for the curve surface projection is to automatically computes image distortion correction for any given position of the viewer eyepoint and the projector.
  • the multi-plane display system comprises more than one display surface, care should be taken to minimize the seams and gaps between the edges of the respective displays.
  • the overlapped image is calculated by an image processor to ensure that the projected pixels in the overlapped areas are adjusted to form the proper displayed images.
  • Other solutions are to control the degree of intensity reduction in the overlapping to create a smooth transition from the image of one display surface to the next.
  • Binaural simulation offers realism to the three dimensional simulation together with 3D visualization.
  • hearing using one ear is called monoaural and hearing using two ears is called binaural.
  • Hearing can provide the direction of the sound sources but with poorer resolution than vision, the identity and content of a sound source such as speech or music, and the nature of the environment via echoes, reverberation such as a normal room or an open field.
  • the head and ears, and sometime the shoulder function as an antenna system to provide information about the location, distance and environment of the sound sources.
  • the brain can inte ⁇ rete properly the various kinds of sound arriving at the head such as direct sounds, diffractive sounds around the head and by interaction with the outer ears and shoulder, different sound amplitudes and different arrival time of the sounds. These acoustic modifications are called 'sound cues' and serve to provide us the directional acoustis information of the sounds.
  • the sound cues are related to timing, volume, frequency and reflection.
  • timing cues the ears recognize the time the sound arrives and assume that the sound comes from the closest source. Further, with two ears separated about 8 inches apart, the delay of the sound reaching one ear with respect to the other ear can give a cue about the location of the sound source. The timing cue is stronger than the level cue in the sense that the listener locates the sound based on the first wave that reaches the ear, regardless of the loudness of any later arriving waves.
  • volume (or level) cues the ears recognize the volume (or loudness) of the sound and assume that the sound coming from the loudest direction.
  • the amplitude difference between the ears is a strong cue for the localization of the sound source.
  • frequency (or equalization) cues the ears recognize the frequency balance of the sound as it arrives in each ear since frontal sounds are directed into the eardrums, while rear sounds bounce off the external ear and thus having a high frequency roll off.
  • reflection cue the sound bounces off various surfaces and are either dispersed or absorbed in varying degrees before reaching the ears multiple times. This reflections off the walls of the room and the foreknowledge of the difference between the way various floor coverings sound also contribute to localization.
  • the body especially the head, can move relative to the sound source to help in locate the sound.
  • ITDs interaural time differences
  • ILDs interaural level differences
  • HRTFs head-related transfer functions
  • the HRTFs are a collection of spatial cues for a particular listener, including ITDs, ILDs and the reflections, diffractions and damping caused by the listener's body, head, outer ears and shoulder.
  • the external ear, or pinna has a significant contribution to the HRTFs.
  • Higher frequency sounds are filtered by the pinna to provide the brain a way as to perceive the lateral position, or azimuth, and elevation of the sound source since the response of the pinna filter is highly dependent on the overall direction of the sound source.
  • the head can account for a reduced amplitude of various frequencies of the sounds since the sound has to go through or around the head in order to reach the ear.
  • HRTF is the frequency response of the sound waves as received by the ears.
  • the sounds can be delivered to a listener's ears via headphones or loudspeakers to create a virtual sound image in three dimensions.
  • the sound transformation to the ear canal i.e. HRTF frequency response
  • HRTF frequency response can be measured accurately by using small microphones in the ear canals.
  • the measured signal is then processed by a computer to derive the HRTF frequency responses for the left and right ears corresponding to the sound source location.
  • a 3D audio system works by using the measured HRTFs as the audio filters or equalizers.
  • the HRTFs filters When a sound signal is processed by the HRTFs filters, the sound localization cues are reproduced, and the listener should perceive the sound at the location specified by the HRTFs. This method of binaural synthesis works extremely well when the listener's own HRTFs are used to synthesize the localization cues.
  • HRTFs are a complicated procedure, so 3D audio systems typically use a single set of HRTFs previously measured from a particular human or manikin subject. Thus the HRTF sometimes needs to be changed to accurately respond to a perticular listener.
  • the tuning of a HRTF function can be accomplished by providing various sound source locations and environments and asking the listener to identify.
  • a 3D audio system should provide the ability for the listener to define a three- dimensional space, to position multiple sound sources and that listener in that 3D space, and to do it all in real-time, or interactively.
  • Beside 3D audio system other technologies such stereo extension and surround sound could offer some aspects of 3D positioning or interactivity.
  • Extended stereo processes an existing stereo (two channel) soundtrack to add spaciousness and to make it appear to originate from outside the left/right speaker locations through fairly straight-forward methods.
  • Some of the characteristics of the extended stereo technology include the size of the listening area (called sweet spot), the amount of spreading of stereo images, the amount of tonal changes, the amount of lost stereo panning information, and the ability to achieve effect on headphones as well as speakers.
  • the surround sound create a larger-than-stereo sound stage with a surround sound 5- speaker setup.
  • virtual surround sound systems use 3D audio technology to create the illusion of five speakers emanating from a regular set of stereo speakers, therefore enabling a surround sound listening experience without the need for a five speaker setup.
  • the characteristics of the surround sound technology include the presentation accuracy, the clarity of spatial imaging, and the size of the listening area
  • audio technology needs to create a life-like listening experience by replicating the 3D audio cues that the ears hear in the real world for allowing non-interactive and interactive listening and positioning of sounds anywhere in the three-dimensional space surrounding a listener.
  • the head tracker function is also very important to provide perceptual room constancy to the listener. In other words, when the listener move their heads around, the signals would change so that the perceived auditory world maintain its spatial position. To this end, the simulation system needs to know the head position in order to be able to control the binaural impulse responses adequately. Head position sensors have therefore to be provided. The impression of being immersed is of particular relevance for applications in the context of virtual reality.
  • a replica of a sound field can be produced by putting an infinite number of microphones everywhere. After being stored on a recorder with an infinite number of channels, this recording can then be played back through an infinite number of point- source loudspeakers, each placed exactly as its corresponding microphone was placed. As the number of microphones and speakers is reduced, the quality of the sound field being simulated suffers. By the time we are down to two channels, height cues have certainly been lost and instead of a stage that is audible from anywhere in the room we find that sources on the stage are now only localizable if we listen along a line equidistant from the last two remaining speakers and face them.
  • Headphone reproduction is thus differed from loudspeaker reproduction since headphone microphones should be spaced about seven inches apart for a normal ear separation, and loudspeaker microphones separation should be about seven feet apart. Further loudspeakers suffer from crosstalk and therefore some signal conditioning such as crosstalk cancellation will be needed for 3D loudspeaker setup.
  • Loudspeaker 3D audio systems are extremely effective in desktop computing environments. This is because there is usually only a single listener (the computer user) who is almost always centered between the speakers and facing forward towards the monitor. Thus, the primary user gets the full 3D effect because the crosstalk is properly cancelled. In typical 3D audio applications, like video gaming, friends may gather around to watch. In this case, the best 3D audio effects are heard by others when they are also centered with respect to the loudspeakers. Off-center listeners may not get the full effect, but they still hear a high quality stereo program with some spatial enhancements. To achieve 3D audio, the speakers are typically arranged surrounding the listener in about the same horizontal plane, but could be arranged to completely surround the listener, from the ceiling to the floor to the surrounding walls.
  • the speakers can also be put on the ceiling, on the floor, arranged in an overhead dome configuration, or arranged in a vertical wall configuration.
  • beam transmitted speakers can be used instead of headphone. Beam transmitted speaker offers the freedom of movement for the listener and without the crosstalk between speakers since beam transmitted speaker provide a tight beam of sound.
  • the subject is free to turn their head, and even move about within a limited range.
  • more than one subject can listen to the system simultaneously.
  • crosstalk cancellers are based on a highly simplified model of crosstalk, for example modeling crosstalk as a simple delay and attenuation process, or a delay and a lowpass filter.
  • Other crosstalk cancellers have been based on a spherical head model. As with binaural synthesis, crosstalk cancellation performance is ultimately limited by the variation in the size and shape of human heads.
  • 3D audio simulation can be accomplished by the following steps: - Input the characteristics of the acoustic space.
  • Each sound arrival will have the following characteristics: (a) time of arrival, based on the distance travelled by the echo-path, (b) direction of arrival, (c) attenuation (as a function of frequency) of the sound due to the abso ⁇ tion properties of the surfaces encountered by the echo-path.
  • results from the FIR filter are played back to a listener.
  • the results are played over headphones to the listener.
  • the equalisation required for the particular headphones is also applied.
  • the simulation of an acoustic environment involves one or more of the following functions :
  • Binaural simulation is generally carried out using the sound source material free from any unwanted echoes or noise.
  • the sound source material can then be replayed to a subject, using the appropriate HRTF filters, to create the illusion that the source audio is originating from a particular direction.
  • the HRTF filtering is achieved by simply convolving the audio signal with the pair of HRTF responses (one HRTF filter for each channel of the headphone).
  • Improved playback through headphones can be achieved through the use of head tracking.
  • This technique makes use of continuous measurements of the orientation of a subject's head, and adapts the audio signals being fed to the headphones appropriately.
  • Binaural signal should allow a subject to easily discriminate between left and right sound source locations easily, but the ability to discriminate between front and back, and high and low sound sources is generally only possible if head movement is permitted.
  • multiple speaker playback methods solve this problem to a large degree, there are still many applications where headphone playback is preferable, and head tracking can then be used as a valuable tool for improving the quality of the 3-D playback.
  • the simplest form of head tracking binaural system is one which simply simulates anechoic HRTFs, and changes the HRTF functions rapidly in response to the subjects head movements.
  • This HRTF switching can be achieved through a lookup table, with inte ⁇ olation used to resolve angles that are not represented in the HRTF table.
  • An important feature of any head tracking playback system is the delay from the subject head movement to the change in the audio response at the headphones. If this delay is excessive, the subject can experience a form of virtual motion sickness and general disorientation.
  • head motion to resolve ambiguities: a faint, low sound could be either in front or back of us, so we quickly and sub-consciously turn our head a small fraction to the left, and we know if the sound is now off to the right, it is in the front, otherwise it is in the back.
  • One of the reasons why interactive audio is more realistic than pre-recorded audio (soundtracks) is the fact that the listeners head motion can be properly simulated in an interactive system (using inputs from a joystick, mouse, or head-tracking system).
  • the HRTF function are performed using digital signal processing (DSP) hardware for real time performance.
  • DSP digital signal processing
  • Typical feature of DSP are that the direct sound must be processed to give the correct amplitude and perceived direction, the early echoes must arrive at the listener with appropriate time, amplitude and frequency response to give the perception of the size of the spaces (as well as the acoustic nature of the room surfaces), and the late reverberation must be natural and correctly distributed in 3-D around the listener.
  • the relative amplitude of the direct sound compared to the remainder of the room response helps to provide the sensation of distance.
  • 3D audio simulation can provides a binaural gain so that the exact same audio content is more audible and intelligible in the binaural case, because the brain can localize and therefore "single out” the binaural signal, while the non-binaural signal gets washed into the noise. Further the listener would still be able to tune into and understand individual conversations, because they are still spatially separated, and "amplified by" binaural gain, an effect called the cocktail party effct. Binaural simulation also can provide faster reaction time because such a signal mirrors the ones received in the real world.
  • binaural signals can convey positional information: a binaural radar warning sound can warn a user about a specific object that is approaching (with a sound that is unique to that object), and naturally indicate where that object is coming from. Also listening to binaural simulation can beless fatigue since we are used to hearing sounds that originate outside of their heads, as is the case with binaural signals. Mono or stereo signals appear to come from inside a listener's head when using headphones, and produce more strain than a natural sounding, binaural signal. An lastly, 3D binaural simulation can provide an increased perception and immersion in higher quality 3D environment when visuals are shown in synch with binaural sound.

Abstract

The present invention hands-on simulator system discloses a three dimension display system (see figure 1) comprising a three dimensional horizontal perspective display and a 3-D audio system such as binaural simulation to lend realism to the three dimensional display. The three dimensional display system can further comprise a second display, together with a curvilinear blending display section to merge the various images. The multi-plane display surface can accommodate the viewer by adjusting the various images and.3-D sound according to the viewer's eyepoint and earpoint locations. The present invention hands-on simulator system can project horizontal perspective images into the open space and a peripheral device that allow the end user to manipulate the images with hands or hand-held tools.

Description

Horizontal Perspective Simulator
This application claims priority from U.S. provisional applications Ser. No. 60/576, 187 filed 06/01/2004, entitled "Multi plane horizontal perspective display"; Ser. No. 60/576,189 filed 06/01/2004, entitled "Multi plane horizontal perspective hand on simulator"; Ser. No.60/576, 182 filed 06/01/2004, entitled "Binaural horizontal perspective display"; and Ser. No. 60/576,181 filed 06/01/2004, entitled "Binaural horizontal perspective hand on simulator" which are incoφorated herein by reference.
Field of invention
This invention relates to a three-dimensional simulator system, and in particular, to a hands-on computer simulator system capable of operator's interaction.
Background of the invention
Three dimensional (3D) capable electronics and computing hardware devices and real-time computer-generated 3D computer graphics have been a popular area of computer science for the past few decades, with innovations in visual, audio and tactile systems. Much of the research in this area has produced hardware and software products that are specifically designed to generate greater realism and more natural computer- human interfaces. These innovations have significantly enhanced and simplified the end- user's computing experience.
Ever since humans began to communicate through pictures, they faced a dilemma of how to accurately represent the three-dimensional world they lived in. Sculpture was used to successfully depict three-dimensional objects, but was not adequate to communicate spatial relationships between objects and within environments. To do this, early humans attempted to "flatten" what they saw around them onto two-dimensional, vertical planes (e.g. paintings, drawings, tapestries, etc.). Scenes where a person stood upright, surrounded by trees, were rendered relatively successfully on a vertical plane. But how could they represent a landscape, where the ground extended out horizontally from where the artist was standing, as far as the eye could see?
The answer is three dimensional illusions. The two dimensional pictures must provide a numbers of cues of the third dimension to the brain to create the illusion of three dimensional images. This effect of third dimension cues can be realistically achievable due to the fact that the brain is quite accustomed to it. The three dimensional real world is always and already converted into two dimensional (e.g. height and width) projected image at the retina, a concave surface at the back of the eye. And from this two dimensional image, the brain, through experience and perception, generates the depth information to fonn the three dimension visual image from two types of depth cues: monocular (one eye perception) and binocular (two eye perception). In general, binocular depth cues are innate and biological while monocular depth cues are learned and environmental. The major binocular depth cues are convergence and retinal disparity. The brain measures the amount of convergence of the eyes to provide a rough estimate of the distance since the angle between the line of sight of each eye is larger when an object is closer. The disparity of the retinal images due to the separation of the two eyes is used to create the perception of depth. The effect is called stereoscopy where each eye receives a slightly different view of a scene, and the brain fuses them together using these differences to determine the ratio of distances between nearby objects.
Binocular cues are very powerful perception of depth. However, there are also depth cues with only one eye, called monocular depth cues, to create an impression of depth on a flat image. The major monocular cues are: overlapping, relative size, linear perspective and light and shadow. When an object is viewed partially covered, this pattern of blocking is used as a cue to determine that the object is farther away. When two objects known to be the same size and one appears smaller than the other, this pattern of relative size is used as a cue to assume that the smaller object is farther away. The cue of relative size also provides the basis for the cue of linear perspective where the farther away the lines are from the observer, the closer together they will appear since parallel lines in a perspective image appear to converge towards a single point. The light falling on an object from a certain angle could provide the cue for the form and depth of an object. The distribution of light and shadow on objects is a powerful monocular cue for depth provided by "the biologically correct assumption that light comes from above.
Perspective drawing, together with relative size, is most often used to achieve the illusion of three dimension depth and spatial relationships on a flat (two dimension) surface, such as paper or canvas. Through perspective, three dimension objects are depicted on a two dimension plane, but "trick" the eye into appearing to be in three dimension space. The first theoretical treatise for constructing perspective, Depictura, was published in the early 1400's by the architect, Leone Battista Alberti. Since the introduction of his book, the details behind "general" perspective have been very well documented. However, the fact that there are a number of other types of perspectives is not well known. Some examples are military, cavalier, isometric, and dimetric, as shown at the top of Figure 1.
Of special interest is the most common type of perspective, called central perspective, shown at the bottom left of Figure 1. Central perspective, also called one-point perspective, is the simplest kind of "genuine" perspective construction, and is often taught in art and drafting classes for beginners. Figure 2 further illustrates central perspective. Using central perspective, the chess board and chess pieces look like three dimension objects, even though they are drawn on a two dimensional flat piece of paper. Central perspective has a central vanishing point, and rectangular objects are placed so their front sides are parallel to the picture plane. The depth of the objects is peφendicular to the picture plane. All parallel receding edges run towards a central vanishing point. The viewer looks towards this vanishing point with a straight view. When an architect or artist creates a drawing using central perspective, they must use a single-eye view. That is, the artist creating the drawing captures the image by looking through only one eye, which is peφendicular to the drawing surface.
The vast majority of images, including central perspective images, are displayed, viewed and captured in a plane peφendicular to the line of vision. Viewing the images at angle different from 90° would result in image distortion, meaning a square would be seen as a rectangle when the viewing surface is not peφendicular to the line of vision.
Central perspective is employed extensively in 3D computer graphics, for a myriad of applications, such as scientific, data visualization, computer-generated prototyping, special effects for movies, medical imaging, and architecture, to name just a few. One of the most common and well-known 3D computing applications is 3D gaming, which is used here as an example, because the core concepts used in 3D gaming extend to all other 3D computing applications.
Figure 3 is a simple illustration, intended to set the stage by listing the basic components necessary to achieve a high level of realism in 3D software applications. At its highest level, 3D game development consists of four essential components:
1. Design: Creation of the game's story line and game play
2. Content: The objects (figures, landscapes, etc.) that come to life during game play
3. Artificial Intelligence (Al): Controls interaction with the content during game play
4. Real-time computer-generated 3D graphics engine (3D graphics engine):
Manages the design, content, and Al data. Decides what to draw, and how to draw it, then renders (displays) it on a computer monitor
A person using a 3D application, such as a game, is in fact running software in the form of a real-time computer-generated 3D graphics engine. One of the engine's key components is the renderer. Its job is to take 3D objects that exist within computer- generated world coordinates x, y, z, and render (draw/display) them onto the computer monitor's viewing surface, which is a flat (2D) plane, with real world coordinates x, y. Figure 4 is a representation of what is happening inside the computer when running a 3D graphics engine. Within every 3D game there exists a computer-generated 3D "world." This world contains everything that could be experienced during game play. It also uses the Cartesian coordinate system, meaning it has three spatial dimensions x, y, and z. These three dimensions are referred to as "virtual world coordinates". Game play for a typical 3D game might begin with a computer-generated-3D earth and a computer- generated-3D satellite orbiting it. The virtual world coordinate system enables the earth and satellite to be properly positioned in computer-generated x, y, z space.
As they move through time, the satellite and earth must stay properly synchronized. To accomplish this, the 3D graphics engine creates a fourth universal dimension for computer-generated time, t. For every tick of time t, the 3D graphics engine regenerates the satellite at its new location and orientation as it orbits the spinning earth. Therefore, a key job for a 3D graphics engine is to continuously synchronize and regenerate all 3D objects within all four computer-generated dimensions x, y, z, and t.
Figure 5 is a conceptual illustration of what happens inside the computer when an end-user is playing, i.e. running, a first-person 3D application. First-person means that the computer monitor is much like a window, through which the person playing the game views the computer-generated world. To generate this view, the 3D graphics engine renders the scene from the point of view of the eye of a computer-generated person. The computer-generated person can be thought of as a computer-generated or "virtual" simulation of the "real" person actually playing the game.
While running a 3D application the real person, i.e. the end-user, views only a small segment of the entire 3D world at any given time. This is done because it is computationally expensive for the computer's hardware to generate the enormous number of 3D objects in a typical 3D application, the majority of which the end-user is not currently focused on. Therefore, a critical job for the 3D graphics engine is to minimize the computer hardware's computational burden by drawing/rendering as little information as absolutely necessary during each tick of computer-generated time t.
The boxed-in area in Figure 5 conceptually represents how a 3D graphics engine minimizes the hardware's burden. It focuses computational resources on extremely small areas of information as compared to the 3D applications entire world. In this example, it is a "computer-generated" polar bear cub being observed by a "computer-generated" virtual person. Because the end user is running in first-person everything the computer- generated person sees is rendered onto the end-user's monitor, i.e. the end user is looking through the eye of the computer-generated person.
In Figure 5 the computer-generated person is looking through only one eye; in other words, an one-eyed view. This is because the 3D graphics engine's renderer uses central perspective to draw/render 3D objects onto a 2D surface, which requires viewing through only one eye. The area that the computer-generated person sees with a one-eye view is • called the "view volume", and the computer-generated 3D objects within this view volume are what actually get rendered to the computer monitor's 2D viewing surface.
Figure 6 illustrates a view volume in more detail. A view volume is a subset of a "camera model". A camera model is a blueprint that defines the characteristics of both the hardware and software of a 3D graphics engine. Like a very complex and sophisticated automobile engine, a 3D graphics engine consist of so many parts that their camera models are often simplified to illustrate only the essential elements being referenced.
The camera model depicted in Figure 6 shows a 3D graphics engine using central perspective to render computer-generated 3D objects to a computer monitor's vertical, 2D viewing surface. The view volume shown in Figure 6, although more detailed, is the same view volume represented in Figure 5. The only difference is semantics because a 3D graphics engine calls the computer-generated person's one-eye view a camera point (hence camera model).
Every component of a camera model is called an "element". In our simplified camera model, the element called near clip plane is the 2D plane onto which the x, y, z coordinates of the 3D objects within the view volume will be rendered. Each projection line starts at the camera point, and ends at a x, y. z coordinate point of a virtual 3D object within the view volume. The 3D graphics engine then determines where the projection line intersects the near clip-plane and the x and y point where this intersection occurs is rendered onto the near clip-plane. Once the 3D graphics engine's renderer completes all necessary mathematical projections, the near clip plane is displayed on the 2D viewing surface of the computer monitor, as shown in Figure 6.
The basic of prior art 3D computer graphics is the central perspective projection. 3D central perspective projection, though offering realistic 3D illusion, has some limitations is allowing the user to have hands-on interaction with the 3D display.
There is a little known class of images that we called it "horizontal perspective" where the image appears distorted when viewing head on, but displaying a three dimensional illusion when viewing from the correct viewing position. In horizontal perspective, the angle between the viewing surface and the line of vision is preferably 45° but can be almost any angle, and the viewing surface is preferably horizontal (wherein the name "horizontal perspective"), but it can be any surface, as long as the line of vision forming a not-peφendicular angle to it.
Horizontal perspective images offer realistic three dimensional illusion, but are little known primarily due to the narrow viewing location (the viewer's eyepoint has to be coincide precisely with the image projection eyepoint), and the complexity involving in projecting the two dimensional image or the three dimension model into the horizontal perspective image.
The generation of horizontal perspective images requires considerably more expertise to create than conventional peφendicular images. The conventional peφendicular images can be produced directly from the viewer or camera point. One need simply open one's eyes or point the camera in any direction to obtain the images. Further, with much experience in viewing three dimensional depth cues from peφendicular images, viewers can tolerate significant amount of distortion generated by the deviations from the camera point. In contrast, the creation of a horizontal perspective image does require much manipulation. Conventional camera, by projecting the image into the plane peφendicular to the line of sight, would not produce a horizontal perspective image. Making a horizontal drawing requires much effort and very time consuming. Further, since human has limited experience with horizontal perspective images, the viewer's eye must be positioned precisely where the projection eyepoint point is to avoid image distortion. And therefore horizontal perspective, with its difficulties, has received little attention. For realistic three dimensional simulation, binaural or three dimensional audio simulation is also needed.
Summary of the invention
The present invention recognizes that the personal computer is perfectly suitable for horizontal perspective display. It is personal, thus it is designed for the operation of one person, and the computer, with its powerful microprocessor, is well capable of rendering various horizontal perspective images to the viewer. Further, horizontal perspective offers open space display of 3D images, thus allowing the hands-on interaction of the end users.
Thus the present invention discloses a multi-plane hands-on simulator system comprising at least two display surfaces, one of which displaying a three dimensional horizontal perspective images. The other display surfaces can display two dimensional images, or preferably three dimensional central peφective images. Further, the display surfaces can have a curvilinear blending display section to merge the various images. The multi-plane hands-on simulator can comprise various camera eyepoints, one for the horizontal perspective images, one for the central perspective images, and optionally one for the curvilinear blending display surface. The multi-plane display surface can further adjust the various images to accommodate the position of the viewer. By changing the displayed images to keep the camera eyepoints of the horizontal perspective and central perspective images in the same position as the viewer's eye point, the viewer's eye is always positioned at the proper viewing position to perceive the three dimensional illusion, thus minimizing viewer's discomfort and distortion. The display can accept manual input such as a computer mouse, trackball, joystick, tablet, etc. to re-position the horizontal perspective images. The display can also automatically re-position the images based on an input device automatically providing the viewer's viewpoint location. The multi-plane hands-on simulator system can project horizontal perspective images into the open space and a peripheral device that allow the end user to manipulate the images with hands or hand-held tools.
Further, the display is also included three dimensional audio such as binaural simulation to lend realism to the three dimensional display.
Brief description of the drawings
Figure 1 shows the various perspective drawings.
Figure 2 shows a typical central perspective drawing.
Figure 3 shows 3D software application.
Figure 4 shows 3D application running on PC.
Figure 5 shows 3D application in first person.
Figure 6 shows central perspective camera model.
Figure 7 shows the comparison of central perspective (Image A) and horizontal perspective (Image B).
Figure 8 shows the central perspective drawing of three stacking blocks.
Figure 9 shows the horizontal perspective drawing of three stacking blocks.
Figure 10 shows the method of drawing a horizontal perspective drawing. Figure 11 shows a horizontal perspective display and a viewer input device.
Figure 12 shows a horizontal perspective display, a computational device and a viewer input device.
Figure 13 shows a computer monitor.
Figure 14 shows a monitor's phosphor layer indicating of an incorrect location of image.
Figure 15 shows a monitor's viewing surface indicating of a correct location of image.
Figure 1 shows a reference plane x, y, z coordinates.
Figure 17 shows the location of an angled camera point.
Figure 18 shows the mapping of the horizontal plane to a reference plane.
Figure 19 shows the comfort plane.
Figure 20 shows the hands-on volume.
Figure 21 shows the inner plane.
Figure 22 shows the bottom plane.
Figure 23 shows the inner access volume.
Figure 24 shows the angled camera mapped to the end-user's eye
Figure 25 shows mapping of the 3-d object onto the horizontal plane.
Figure 26 shows the two-eye view.
Figure 27 shows the simulation time of the horizontal perspective. Figure 28 shows the horizontal plane.
Figure 29 shows the 3D peripherals.
Figure 30 shows an open-access camera model.
Figure 31 shows the concept of object recognition.
Figure 32 shows the 3D audio combination with object recognition.
Figure 33 shows another open access camera model.
Figure 34 shows another open access camera model
Figure 35 shows the mapping of virtual attachments to end of tools.
Figure 36 shows the multi-plane and multi-view device.
Figure 37 shows an open access camera model.
Figure 38 shows another multi-plane device.
Detailed description of the invention
The new and unique inventions described in this document build upon prior art by taking the current state of real-time computer-generated 3D computer graphics, 3D sound, and tactile computer-human interfaces to a whole new level of reality and simplicity. More specifically, these new inventions enable real-time computer-generated 3D simulations to coexist in physical space and time with the end-user and with other real-world physical objects. This capability dramatically improves upon the end-user's visual, auditory and tactile computing experience by providing direct physical interactions with 3D computer-generated objects and sounds. This unique ability is useful in nearly every conceivable industry including, but not limited to, electronics, computers, biometrics, medical, education, games, movies, science, legal, financial, communication, law enforcement, national security, military, print media, television, advertising, trade show, data visualization, computer-generated reality, animation, CAD/CAE/CAM, productivity software, operating systems, and more.
The present invention discloses a multi-plane horizontal perspective hands-on simulator comprising at least two display surfaces, one of which capable of projecting three dimensional illusion based on horizontal perspective projection.
In general, the present invention horizontal perspective hands-on simulator can be used to display and interact with three dimensional images and has obvious utility to many industrial applications such as manufacturing design reviews, ergonomic simulation, safety and training, video games, cinematography, scientific 3D viewing, and medical and other data displays.
Horizontal perspective is a little-known perspective, of which we found only two books that describe its mechanics: Stereoscopic Drawing (©1990) and How to Make Anaglyphs (©1979, out of print). Although these books describe this obscure perspective, they do not agree on its name. The first book refers to it as a "free-standing anaglyph," and the second, a "phantogram." Another publication called it "projective anaglyph" (U.S. patent 5,795,154 by G. M. Woods, Aug. 18, 1998). Since there is no agreed-upon name, we have taken the liberty of calling it "horizontal perspective." Normally, as in central perspective, the plane of vision, at right angle to the line of sight, is also the projected plane of the picture, and depth cues are used to give the illusion of depth to this flat image. In horizontal perspective, the plane of vision remains the same, but the projected image is not on this plane. It is on a plane angled to the plane of vision. Typically, the image would be on the ground level surface. This means the image will be physically in the third dimension relative to the plane of vision. Thus horizontal perspective can be called horizontal projection.
In horizontal perspective, the object is to separate the image from the paper, and fuse the image to the three dimension object that projects the horizontal perspective image. Thus the horizontal perspective image must be distorted so that the visual image fuses to form the free standing three dimensional figure. It is also essential the image is viewed from the correct eye points, otherwise the three dimensional illusion is lost. In contrast to central perspective images which have height and width, and project an illusion of depth, and therefore the objects are usually abruptly projected and the images appear to be in layers, the horizontal perspective images have actual depth and width, and illusion gives them height, and therefore there is usually a graduated shifting so the images appear to be continuous.
Figure 7 compares key characteristics that differentiate central perspective and horizontal perspective. Image A shows key pertinent characteristics of central perspective, and Image B shows key pertinent characteristics of horizontal perspective.
In other words, in Image A, the real-life three dimension object (three blocks stacked slightly above each other) was drawn by the artist closing one eye, and viewing along a line of sight peφendicular to the vertical drawing plane. The resulting image, when viewed vertically, straight on, and through one eye, looks the same as the original image.
In Image B, the real-life three dimension object was drawn by the artist closing one eye, and viewing along a line of sight 45° to the horizontal drawing plane. The resulting image, when viewed horizontally, at 45° and through one eye, looks the same as the original image.
One major difference between central perspective showing in Image A and horizontal perspective showing in Image B is the location of the display plane with respect to the projected three dimensional image. In horizontal perspective of Image B, the display plane can be adjusted up and down , and therefore the projected image can be displayed in the open air above the display plane, i.e. a physical hand can touch (or more likely pass through) the illusion, or it can be displayed under the display plane, i.e. one cannot touch the illusion because the display plane physically blocks the hand. This is the nature of horizontal perspective, and as long as the camera eyepoint and the viewer eyepoint is at the same place, the illusion is present. In contrast, in central perspective of Image A, the three dimensional illusion is likely to be only inside the display plane, meaning one cannot touch it. To bring the three dimensional illusion outside of the display plane to allow viewer to touch it, the central perspective would need elaborate display scheme such as surround image projection and large volume.
Figures 8 and 9 illustrate the visual difference between using central and horizontal perspective. To experience this visual difference, first look at Figure 8, drawn with central perspective, through one open eye. Hold the piece of paper vertically in front of you, as you would a traditional drawing, peφendicular to your eye. You can see that central perspective provides a good representation of three dimension objects on a two dimension surface.
Now look at Figure 9, drawn using horizontal perspective, by sifting at your desk and placing the paper lying flat (horizontally) on the desk in front of you. Again, view the image through only one eye. This puts your one open eye, called the eye point at approximately a 45° angle to the paper, which is the angle that the artist used to make the drawing. To get your open eye and its line-of-sight to coincide with the artist's, move your eye downward and forward closer to the drawing, about six inches out and down and at a 45° angle. This will result in the ideal viewing experience where the top and middle blocks will appear above the paper in open space.
Again, the reason your one open eye needs to be at this precise location is because both central and horizontal perspective not only defines the angle of the line of sight from the eye point; they also define the distance from the eye point to the drawing. This means that Figures 8 and 9 are drawn with an ideal location and direction for your open eye relative to the drawing surfaces. However, unlike central perspective where deviations from position and direction of the eye point create little distortion, when viewing a horizontal perspective drawing, the use of only one eye and the position and direction of that eye relative to the viewing surface are essential to seeing the open space three dimension horizontal perspective illusion.
Figure 10 is an architectural-style illustration that demonstrates a method for making simple geometric drawings on paper or canvas utilizing horizontal perspective. Figure 10 is a side view of the same three blocks used in Figures 9. It illustrates the actual mechanics of horizontal perspective. Each point that makes up the object is drawn by projecting the point onto the horizontal drawing plane. To illustrate this, Figure 10 shows a few of the coordinates of the blocks being drawn on the horizontal drawing plane through projection lines. These projection lines start at the eye point (not shown in Figure 10 due to scale), intersect a point on the object, then continue in a straight line to where they intersect the horizontal drawing plane, which is where they are physically drawn as a single dot on the paper When an architect repeats this process for each and every point on the blocks, as seen from the drawing surface to the eye point along the line-of-sight the horizontal perspective drawing is complete, and looks like Figure 9.
Notice that in Figure 10, one of the three blocks appears below the horizontal drawing plane. With horizontal perspective, points located below the drawing surface are also drawn onto the horizontal drawing plane, as seen from the eye point along the line-of- site. Therefore when the final drawing is viewed, objects not only appear above the horizontal drawing plane, but may also appear below it as well — giving the appearance that they are receding into the paper. If you look again at Figure 9, you will notice that the bottom box appears to be below, or go into, the paper, while the other two boxes appear above the paper in open space.
The generation of horizontal perspective images requires considerably more expertise to create than central perspective images. Even though both methods seek to provide the viewer the three dimension illusion that resulted from the two dimensional image, central perspective images produce directly the three dimensional landscape from the viewer or camera point. In contrast, the horizontal perspective image appears distorted when viewing head on, but this distortion has to be precisely rendered so that when viewing at a precise location, the horizontal perspective produces a three dimensional illusion.
The horizontal perspective display system promotes horizontal perspective projection viewing by providing the viewer with the means to adjust the displayed images to maximize the illusion viewing experience. By employing the computation power of the microprocessor and a real time display, the horizontal perspective display is shown in Fig. 11, comprising a real time electronic display 100 capable of re-drawing the projected image, together with a viewer's input device 102 to adjust the horizontal perspective image. By re-display the horizontal perspective image so that its projection eyepoint coincides with the eyepoint of the viewer, the horizontal perspective display can ensure the minimum distortion in rendering the three dimension illusion from the horizontal perspective method. The input device can be manually operated where the viewer manually inputs his or her eyepoint location, or change the projection image eyepoint to obtain the optimum three dimensional illusion. The input device can also be automatically operated where the display automatically tracks the viewer's eyepoint and adjust the projection image accordingly. The horizontal perspective display removes the constraint that the viewers keeping their heads in relatively fixed positions, a constraint that create much difficulty in the acceptance of precise eyepoint location such as horizontal perspective or hologram display.
The horizontal perspective display system, shown in Figure 12, can further comprise a computation device 110 in addition to the real time electronic display device 100 and projection image input device 112 providing input to the computational device 110 to calculating the projectional images for display to providing a realistic, minimum distortion three dimensional illusion to the viewer by coincide the viewer's eyepoint with the projection image eyepoint. The system can further comprise an image enlargement reduction input device 115, or an image rotation input device 117, or an image movement device 119 to allow the viewer to adjust the view of the projection images. The horizontal perspective display system promotes horizontal perspective projection viewing by providing the viewer with the means to adjust the displayed images to maximize the illusion viewing experience. By employing the computation power of the microprocessor and a real time display, the horizontal perspective display, comprising a real time electronic display capable of re-drawing the projected image, together with a viewer's input device to adjust the horizontal perspective image. By re-display the horizontal perspective image so that its projection eyepoint coincides with the eyepoint of the viewer, the horizontal perspective display of the present invention can ensure the minimum distortion in rendering the three dimension illusion from the horizontal perspective method. The input device can be manually operated where the viewer manually inputs his or her eyepoint location, or change the projection image eyepoint to obtain the optimum three dimensional illusions. The input device can also be automatically operated where the display automatically tracks the viewer's eyepoint and adjust the projection image accordingly. The horizontal perspective display system removes the constraint that the viewers keeping their heads in relatively fixed positions, a constraint that create much difficulty in the acceptance of precise eyepoint location such as horizontal perspective or hologram display.
The horizontal perspective display system can further a computation device in addition to the real time electronic display device and projection image input device providing input to the computational device to calculating the projectional images for display to providing a realistic, minimum distortion three dimensional illusion to the viewer by coincide the viewer's eyepoint with the projection image eyepoint. The system can further comprise an image enlargement/reduction input device, or an image rotation input device, or an image movement device to allow the viewer to adjust the view of the projection images.
The input device can be operated manually or automatically. The input device can detect the position and orientation of the viewer eyepoint, to compute and to project the image onto the display according to the detection result. Alternatively, the input device can be made to detect the position and orientation of the viewer's head along with the orientation of the eyeballs. The input device can comprise an infrared detection system to detect the position the viewer's head to allow the viewer freedom of head movement. Other embodiments of the input device can be the triangulation method of detecting the viewer eyepoint location, such as a CCD camera providing position data suitable for the head tracking objectives of the invention. The input device can be manually operated by the viewer, such as a keyboard, mouse, trackball, joystick, or the like, to indicate the correct display of the horizontal perspective display images.
The head or eye-tracking system can comprise a base unit and a head-mounted sensor on the head of the viewer. The head-mounted sensor produces signals showing the position and orientation of the viewer in response to the viewer's head movement and eye orientation. These signals can be received by the base unit and are used to compute the proper three dimensional projection images. The head or eye tracking system can be infrared cameras to capture images of the viewer's eyes. Using the captured images and other techniques of image processing, the position and orientation of the viewer's eyes can be determined, and then provided to the base unit. The head and eye tracking can be done in real time for small enough time interval to provide continous viewer's head and eye tracking. The invention described in this document, employing the open space characteristics of the horizontal perspective, together with a number of new computer hardware and software elements and processes that together to create a "Hands-On Simulator". In the simplest terms, the Hands-On Simulator generates a totally new and unique computing experience in that it enables an end user to interact physically and directly (Hands-On) with real-time computer-generated 3D graphics (Simulations), which appear in open space above the viewing surface of a display device, i.e. in the end user's own physical space.
For the end user to experience these unique hands-on simulations the computer hardware viewing surface is situated horizontally, such that the end-user's line of sight is at a 45° angle to the surface. Typically, this means that the end user is standing or seated vertically, and the viewing surface is horizontal to the ground. Note that although the end user can experience hands-on simulations at viewing angles other than 45° (e.g. 55°, 30° etc.), it is the optimal angle for the brain to recognize the maximum amount of spatial information in an open space image. Therefore, for simplicity's sake, we use "45°" throughout this document to mean "an approximate 45 degree angle". Further, while horizontal viewing surface is preferred since it simulates viewers' experience with the horizontal ground, any viewing surface could offer similar three dimensional illusion experience. The horizontal perspective illusion can appear to be hanging from a ceiling by projecting the horizontal perspective images onto a ceiling surface, or appear to be floating from a wall by projecting the horizontal perspective images onto a vertical wall surface. The hands-on simulations are generated within a 3D graphics engines' view volume, creating two new elements, the "Hands-On Volume" and the "Inner-Access Volume." The Hands-On Volume is situated on and above the physical viewing surface. Thus the end user can directly, physically manipulate simulations because they co-inhabit the end- user' s own physical space. This 1:1 correspondence allows accurate and tangible physical interaction by touching and manipulating simulations with hands or hand-held tools. The Inner- Access Volume is located underneath the viewing surface and simulations within this volume appear inside the physically viewing device. Thus simulations generated within the Inner- Access Volume do not share the same physical space with the end user and the images therefore cannot be directly, physically manipulated by hands or handheld tools. That is, they are manipulated indirectly via a computer mouse or a joystick.
This disclosed Hands-On Simulator can lead to the end user's ability to directly, physically manipulate simulations because they co-inhabit the end-user's own physical space. To accomplish this requires a new computing concept where computer-generated world elements have a 1 :1 correspondence with their physical real- world equivalents; that is, a physical element and an equivalent computer-generated element occupy the same space and time. This is achieved by identifying and establishing a common "Reference Plane", to which the new elements are synchronized.
Synchronization with the Reference Plane forms the basis to create the 1:1 correspondence between the "virtual" world of the simulations, and the "real" physical world. Among other things, the 1:1 correspondence insures that images are properly displayed: What is on and above the viewing surface appears on and above the surface, in the Hands-On Volume; what is underneath the viewing surface appears below, in the Inner- Access Volume. Only if this 1:1 correspondence and synchronization to the Reference Plane are present can the end user physically and directly access and interact with simulations via their hands or hand-held tools.
The present invention simulator further includes a real-time computer-generated 3D- graphics engine as generally described above, but using horizontal perspective projection to display the 3D images. One major different between the present invention and prior art graphics engine is the projection display. Existing 3D-graphics engine uses central- perspective and therefore a vertical plane to render its view volume while in the present invention simulator, a "horizontal" oriented rendering plane vs. a "vertical" oriented rendering plane is required to generate horizontal perspective open space images. The horizontal perspective images offer much superior open space access than central perspective images.
One of the invented elements in the present invention hands-on simulator is the 1:1 correspondence of the computer-generated world elements and their physical real- world equivalents. As noted in the introduction above, this 1 :1 correspondence is a new computing concept that is essential for the end user to physically and directly access and interact with hands-on simulations. This new concept requires the creation of a common physical Reference Plane, as well as, the formula for deriving its unique x, y, z spatial coordinates. To determine the location and size of the Reference Plane and its specific coordinates requires understanding the following.
A computer monitor or viewing device is made of many physical layers, individually and together having thickness or depth. To illustrate this, Figure 13 contains a conceptual side-view of typical CRT-type viewing device. The top layer of the monitor's glass surface is the physical "View Surface", and the phosphor layer, where images are made, is the physical "Image Layer". The View Surface and the Image Layer are separate physical layers located at different depths or z coordinates along the viewing device's z axis. To display an image the CRT's electron gun excites the phosphors, which in turn emit photons. This means that when you view an image on a CRT, you are looking along its z axis through its glass surface, like you would a window, and seeing the light of the image coming from its phosphors behind the glass.
With a viewing device's z axis in mind, let's display an image on that device using horizontal perspective. In Figure 14 we use the same architectural technique for drawing images with horizontal perspective as previously illustrated in Figure 10. By comparing Figure 14 and Figure 10 you can see that the middle block in Figure 14 does not correctly appear on the View Surface. In Figure 10 the bottom of the middle block is located correctly on the horizontal drawing/viewing plane, i.e. a piece of paper's View Surface. But in Figure 14, the phosphor layer, i.e. where the image is made, is located behind the CRT's glass surface. Therefore, the bottom of the middle block is incorrectly positioned behind or underneath the View Surface.
Figure 15 shows the proper location of the three blocks on a CRT-type viewing device. That is, the bottom of the middle block is displayed correctly on the View Surface and not on the Image Layer. To make this adjustment the z coordinates of the View Surface and Image Layer are used by the Simulation Engine to correctly render the image. Thus the unique task of correctly rendering an open space image on the View Surface vs. the Image Layer is critical in accurately mapping the simulation images to the real world space. It is now clear that a viewing device's View Surface is the correct physical location to present open space images. Therefore, the View Surface, i.e. the top of the viewing device's glass surface, is the common physical Reference Plane. But only a subset of the View Surface can be the Reference Plane because the entire View Surface is larger than the total image area. Figure 16 shows an example of a complete image being displayed on a viewing device's View Surface. That is, the blue image, including the bear cub, shows the entire image area, which is smaller than the viewing device's View Surface.
Many viewing devices enable the end user to adjust the size of the image area by adjusting its x and y value. Of course these same viewing devices do not provide any knowledge of, or access to, the z axis information because it is a completely new concept and to date only required for the display of open space images. But all three, x, y, z, coordinates are essential to determine the location and size of the common physical Reference Plane. The formula for this is: The Image Layer is given a z coordinate of 0. The View Surface is the distance along the z axis from the Image Layer the Reference Plane's z coordinate is equal to the View Surface, i.e. its distance from the Image Layer. The x and y coordinates, or size of the Reference Plane, can be determined by displaying a complete image on the viewing device and measuring the length of its x and y axis.
The concept of the common physical Reference Plane is a new inventive concept. Therefore, display manufactures may not supply or even know its coordinates. Thus a "Reference Plane Calibration" procedure might need to be performed to establish the Reference Plane coordinates. This calibration procedure provides the end user with a number of orchestrated images that s/he interacts. The end-user's response to these images provides feedback to the Simulation Engine such that it can identify the correct size and location of the Reference Plane. When the end user is satisfied and completes the procedure the coordinates are saved in the end user's personal profile.
With some viewing devices the distance between the View Surface and Image Layer is quite short. But no matter how small or large the distance, it is critical that all Reference Plane x, y, and z coordinates are determined as close as technically possible.
After the mapping of the "computer-generated" horizontal perspective projection display plane (Horizontal Plane) to the "physical" Reference Plane x, y, z coordinates, the two elements coexist and are coincident in time and space; that is, the computer- generated Horizontal Plane now shares the real- world x, y, z coordinates of the physical Reference Plane, and they exist at the same time.
You can envision this unique mapping of a computer-generated element and a physical element occupying the same space and time by imagining you are sitting in front of a horizontally oriented computer monitor and using the Hands-On Simulator. By placing your finger on the surface of the monitor, you would touch the Reference Plane (a portion of the physical View Surface) and the Horizontal Plane (computer-generated) at exactly the same time, In other words, when touching the physical surface of the monitor, you are also "touching" its computer-generated equivalent, the Horizontal Plane, which has been created and mapped by the Simulation Engine to the same location and time.
One element of the present invention horizontal perspective projection hands-on simulator is a computer-generated "Angled Camera" point, shown in Figure 17. The camera point is initially located at an arbitrary distance from the Horizontal Plane and the camera's line-of-site is oriented at a 45° angle looking through the center. The position of the Angled Camera in relation to the end-user's eye is critical to generating simulations that appear in open space on and above the surface of the viewing device.
Mathematically, the computer-generated x, y, z coordinates of the Angled Camera point form the vertex of an infinite "pyramid", whose sides pass through the x, y, z coordinates of the Reference/Horizontal Plane. Figure 18 illustrates this infinite pyramid, which begins at the Angled Camera point and extending through the Far Clip Plane. There are new planes within the pyramid that run parallel to the Reference/Horizontal Plane, which, together with the sides of the pyramid define two new view volumes. These unique view volumes are called Hands-On and the Inner-Access Volume, and are not shown in Figure 18. The dimensions of these volumes and the planes that define them are based on their locations within the pyramid.
Figure 19 illustrates a plane, called Comfort Plane, together with other display elements. The Comfort Plane is one of six planes that define the new Hands-On Volume, and of these planes it is closest to the Angled Camera point and parallel to the Reference Plane. The Comfort Plane is appropriately named because its location within the pyramid determines the end-user's personal comfort, i.e. how their eyes, head, body, etc. are situated while viewing and interacting with simulations. The end user can adjust the location of the Comfort Plane based on their personal visual comfort through a "Comfort Plane Adjustment" procedure. This procedure provides the end user with orchestrated simulations within the Hands-On Volume, and enables them to adjust the location of the Comfort Plane within the pyramid relative to the Reference Plane. When the end user is satisfied and completes the procedure the location of the Comfort Plane is saved in the end-user's personal profiles. The present invention simulator further defines a "Hands-On Volume", shown in Figure 20. The Hands-On Volume is where you can reach your hand in and physically "touch" a simulation. You can envision this by imagining you are sifting in front of a horizontally oriented computer monitor and using the Hands-On Simulator. If you place your hand several inches above the surface of the monitor, you are putting your hand inside both the physical and computer-generated Hands-On Volume at the same time. The Hands-On Volume exists within the pyramid and are between and inclusive of the Comfort Planes and the Reference Horizontal Planes.
Where the Hands-On Volume exists on and above the Reference/Horizontal Plane, the Inner- Access Volume exists below or inside the physical viewing device. For this reason, an end user cannot directly interact with 3D objects located within the Inner- Access Volume via their hand or hand-held tools. But they can interact in the traditional sense with a computer mouse, joystick, or other similar computer peripheral. An "Inner Plane" is further defined, located immediately below and are parallel to the Reference/Horizontal Plane within the pyramid as shown in Figure 21. The Inner Plane, along with the Bottom Plane, is two of the six planes within the pyramid that define the Inner- Access Volume. The Bottom Plane (shown in Figure 22) is farthest away from the Angled Camera point, but it is not to be mistaken for the Far Clip plane. The Bottom Plane is also parallel to the Reference/Horizontal Plane and is one of the six planes that define the Inner-Access Volume (Figure 23). You can envision the Inner- Access Volume by imagining you are sitting in front of a horizontally oriented computer monitor and using the Hands-On Simulator. If you pushed your hand through the physical surface and placed your hand inside the monitor (which of course is not possible), you would be putting your hand inside the Inner- Access Volume.
The end-user's preferred viewing distance to the bottom of the viewing pyramid determines the location of these planes. One way the end user can adjust the location of the Bottom Planes is through a "Bottom Plane Adjustment" procedure. This procedure provides the end user with orchestrated simulations within the Inner-Access Volume and enables them to interact and adjust the location of the Bottom Plane relative to the physical Reference/Horizontal Plane. When the end user completes the procedure the Bottom Plane's coordinates are saved in the end-user's personal profiles.
For the end user to view open space images on their physical viewing device it must be positioned properly, which usually means the physical Reference Plane is placed horizontally to the ground. Whatever the viewing device's position relative to the ground, the Reference/Horizontal Plane must be at approximately a 45° angle to the end-user's line-of-sight for optimum viewing. One way the end user might perform this step is to position their CRT computer monitor on the floor in a stand, so that the Reference/Horizontal Plane is horizontal to the floor. This example use a CRT-type computer monitor, but it could be any type of viewing device, placed at approximately a 45° angle to the end-user's line-of-sight.
The real-world coordinates of the "End-User's Eye" and the computer-generated Angled Camera point must have a 1 :1 correspondence in order for the end user to properly view open space images that appear on and above the Reference/Horizontal Plane (Figure 24). One way to do this is for the end user to supply the Simulation Engine with their eye's real-world x, yr. z location and line-of-site information relative to the center of the physical Reference/Horizontal Plane. For example, the end user tells the Simulation Engine that their physical eye will be located 12 inches up, and 12 inches back, while looking at the center of the Reference/Horizontal Plane. The Simulation Engine then maps the computer-generated Angled Camera point to the End-User's Eye point physical coordinates and line-of-sight.
The present invention horizontal perspective hands-on simulator employs the horizontal perspective projection to mathematically projected the 3D objects to the Hands-On and Inner- Access Volumes. The existence of a physical Reference Plane and the knowledge of its coordinates are essential to correctly adjusting the Horizontal Plane's coordinates prior to projection. This adjustment to the Horizontal Plane enables open space images to appear to the end user on the View Surface vs. the Image Layer by taking into account the offset between the Image Layer and the View Surface, which are located at different values along the viewing device's z axis.
As a projection line in either the Hands-On and Inner-Access Volume intersects both an object point and the offset Horizontal Plane, the three dimensional x, y, z point of the object becomes a two-dimensional x, y point of the Horizontal Plane (see Figure 25). Projection lines often intersect more than one 3D object coordinate, but only one object x, y, z coordinate along a given projection line can become a Horizontal Plane x, y point. The formula to determine which object coordinate becomes a point on the Horizontal Plane is different for each volume. For the Hands-On Volume it is the object coordinate of a given projection line that is farthest from the Horizontal Plane. For the Inner- Access Volume it is the object coordinate of a given projection line that is closest to the Horizontal Plane. In case of a tie, i.e. if a 3D object point from each volume occupies the same 2D point of the Horizontal Plane, the Hands-On Volume's 3D object point is used.
Figure 25 is an illustration of the present invention Simulation Engine that includes the new computer-generated and real physical elements as described above. It also shows that a real-world element and its computer-generated equivalent are mapped 1 :1 and together share a common Reference Plane. The full implementation of this Simulation Engine results in a Hands-On Simulator with real-time, computer-generated 3D-graphics appearing in open space on and above a viewing device's surface, which is oriented approximately 45° to the end-user's line-of-sight.
The Hands-On Simulator further involves adding completely new elements and processes and existing stereoscopic 3D computer hardware. The result in a Hands-On Simulator with multiple views or "Multi- View" capability. Multi- View provides the end user with multiple and/or separate left-and right-eye views of the same simulation.
To provide motion, or time-related simulation, the simulator further includes a new computer-generated "time dimension" element, called "Sl-time". SI is an acronym for "Simulation Image" and is one complete image displayed on the viewing device. Si-Time is the amount of time the Simulation Engine uses to completely generate and display one Simulation Image. This is similar to a movie projector where 24 times a second it displays an image. Therefore, 1/24 of a second is required for one image to be displayed by the projector But SI-Time is variable, meaning that depending on the complexity of the view volumes it could take 1/120* or lA a second for the Simulation Engine to complete just one SI. The simulator also includes a new computer-generated "time dimension" element, called "EV-time" and is the amount of time used to generate a one "Eye- View". For example, let's say that the Simulation Engine needs to create one left-eye view and one right-eye view for puφoses of providing the end user with a stereoscopic 3D experience. If it takes the Simulation Engine lA a second to generate the left-eye view then the first EV-Time period is V* a second. If it takes another lA second to generate the right-eye view then the second EV-Time period is also Α second. Since the Simulation Engine was generating a separate left and right eye view of the same Simulation Image the total SI- Time is one second. That is, the first EV-Time was V% second and the second EV-Time was also lA second making a total Si-Time of one second.
Figure 26 helps illustrate these two new time dimension elements. It is a conceptual drawing of what is occurring inside the Simulation Engine when it is generating a two- eye view of a Simulated Image. The computer-generated person has both eyes open, a requirement for stereoscopic 3D viewing, and therefore sees the bear cub from two separate vantage points, i.e. from both a right-eye view and a left-eye view. These two separate views are slightly different and offset because the average person's eyes are about 2 inches apart. Therefore, each eye sees the world from a separate point in space and the brain puts them together to make a whole image. This is how and why we see the real world in stereoscopic 3D.
Figure 27 is a very high-level Simulation Engine blueprint focusing on how the computer-generated person's two eye views are projected onto the Horizontal Plane and then displayed on a stereoscopic 3D capable viewing device. Figure 26 represents one complete SI-Time period. If we use the example from step 3 above, SI-Time takes one second. During this one second of SI-Time the Simulation Engine needs to generate two different eye views, because in this example the stereoscopic 3D viewing device requires a separate left- and right-eye view. There are existing stereoscopic 3D viewing devices that require more than a separate left- and right-eye view. But because the method described here can generate multiple views it works for these devices as well.
The illustration in the upper left of Figure 27 shows the Angled Camera point for the right eye at time-element "EV-Time-1", which means the first Eye- View time period or the first eye-view to be generated. So in Figure 27, EV-Time-1 is the time period used by the Simulation Engine to complete the first eye (right-eye) view of the computer- generated person. This is the job for this step, which is within EV-Time-1, and using the Angled Camera at coordinate x, y, z, the Simulation Engine completes the rendering and display of the right-eye view of a given Simulation Image.
Once the first eye (right-eye) view is complete, the Simulation Engine starts the process of rendering the computer-generated person's second eye (left-eye) view. The illustration in the lower left of Figure 27 shows the Angled Camera point for the left eye at time element "EV-Time-2". That is, this second eye view is completed during EV- Time-2. But before the rendering process can begin, step 5 makes an adjustment to the Angled Camera point. This is illustrated in Figure 27 by the left eye's x coordinate being incremented by two inches. This difference between the right eye's x value and the left eye's x + 2" is what provides the two-inch separation between the eyes, which is required for stereoscopic 3D viewing.
The distances between people's eyes vary but in the above example we are using the average of 2 inches. It is also possible for the end user to supply the Simulation Engine with their personal eye separation value. This would make the x value for the left and right eyes highly accurate for a given end user and thereby improve the quality of their stereoscopic 3D view.
Once the Simulation Engine has incremented the Angled Camera point's x coordinate by two inches, or by the personal eye separation value supplied by the end user, it completes the rendering and display of the second (left-eye) view. This is done by the Simulation Engine within the EV-Time-2 period using the Angled Camera point coordinate x±2", y, z and the exact same Simulation Image rendered. This completes one SI-Time period.
Depending on the stereoscopic 3D viewing device used, the Simulation Engine continues to display the left- and right-eye images, as described above, until it needs to move to the next SI-Time period. The job of this step is to determine if it is time to move to a new SI-Time period, and if it is, then increment SI-Time. An example of when this may occur is if the bear cub moves his paw or any part of his body Then a new and second Simulated Image would be required to show the bear cub in its new position. This new Simulated Image of the bear cub, in a slightly different location, gets rendered during a new SI-Time period or SI-Time-2. This new SI-time-2 period will have its own EV-Time-1 and EV-Time-2, and therefore the simulation steps described above will be repeated during Sl-time-2. This process of generating multiple views via the nonstop incrementing of SI-Time and its EV-Times continues as long as the Simulation Engine is generating real-time simulations in stereoscopic 3D.
The above steps describe new and unique elements and process that makeup the Hands-On Simulator with Multi-View capability. Multi-View provides the end user with multiple and/or separate left- and right-eye views of the same simulation. Multi-View capability is a significant visual and interactive improvement over the single eye view.
The present invention also allows the viewer to move around the three dimensional display and yet suffer no great distortion since the display can track the viewer eyepoint and re-display the images correspondingly, in contrast to the conventional prior art three dimensional image display where it would be projected and computed as seen from a singular viewing point, and thus any movement by the viewer away from the intended viewing point in space would cause gross distortion.
The display system can further comprise a computer capable of re-calculate the projected image given the movement of the eyepoint location. The horizontal perspective images can be very complex, tedious to create, or created in ways that are not natural for artists or cameras, and therefore require the use of a computer system for the tasks. To display a three-dimensional image of an object with complex surfaces or to create animation sequences would demand a lot of computational power and time, and therefore it is a task well suited to the computer. Three dimensional capable electronics and computing hardware devices and real-time computer-generated three dimensional computer graphics have advanced significantly recently with marked innovations in visual, audio and tactile systems, and have producing excellent hardware and software products to generate realism and more natural computer-human interfaces.
The horizontal perspective display system of the present invention are not only in demand for entertainment media such as televisions, movies, and video games but are also needed from various fields such as education (displaying three-dimensional structures), technological training (displaying three-dimensional equipment). There is an increasing demand for three-dimensional image displays, which can be viewed from various angles to enable observation of real objects using object-like images. The horizontal perspective display system is also capable of substitute a computer-generated reality for the viewer observation. The systems may include audio, visual, motion and inputs from the user in order to create a complete experience of three dimensional illusions.
The input for the horizontal perspective system can be two dimensional image, several images combined to form one single three dimensional image, or three dimensional model. The three dimensional image or model conveys much more information than that a two dimensional image and by changing viewing angle, the viewer will get the impression of seeing the same object from different perspectives continuously.
The horizontal perspective display can further provide multiple views or "Multi- View" capability. Multi- View provides the viewer with multiple and/or separate left-and right-eye views of the same simulation. Multi- View capability is a significant visual and interactive improvement over the single eye view. In Multi- View mode, both the left eye and right eye images are fused by the viewer's brain into a single, three-dimensional illusion. The problem of the discrepancy between accommodation and convergence of eyes, inherent in stereoscopic images, leading to the viewer's eye fatigue with large discrepancy, can be reduced with the horizontal perspective display, especially for motion images, since the position of the viewer's gaze point changes when the display scene changes. In Multi- View mode, the objective is to simulate the actions of the two eyes to create the perception of depth, namely the left eye and the right eye sees slightly different images. Thus Multi-View devices that can be used in the present invention include methods with glasses such as anaglyph method, special polarized glasses or shutter glasses, methods without using glasses such as a parallax stereogram, a lenticular method, and mirror method (concave and convex lens).
In anaglyph method, a display image for the right eye and a display image for the left eye are respectively superimpose-displayed in two colors, e.g., red and blue, and observation images for the right and left eyes are separated using color filters, thus allowing a viewer to recognize a stereoscopic image. The images are displayed using horizontal perspective technique with the viewer looking down at an angle. As with one eye horizontal perspective method, the eyepoint of the projected images has to be coincide with the eyepoint of the viewer, and therefore the viewer input device is essential in allowing the viewer to observe the three dimensional horizontal perspective illusion. From the early days of the anaglyph method, there are much improvements such as the spectrum of the red/blue glasses and display to generate much more realism and comfort to the viewers.
In polarized glasses method, the left eye image and the right eye image are separated by the use of mutually extinguishing polarizing filters such as orthogonally linear polarizer, circular polarizer, elliptical polarizer. The images are normally projected onto screens with polarizing filters and the viewer is then provided with corresponding polarized glasses. The left and right eye images appear on the screen at the same time, but only the left eye polarized light is transmitted through the left eye lens of the eyeglasses and only the right eye polarized light is transmitted through the right eye lens.
Another way for stereoscopic display is the image sequential system. In such a system, the images are displayed sequentially between left eye and right eye images rather than superimposing them upon one another, and the viewer's lenses are synchronized with the screen display to allow the left eye to see only when the left image is displayed, and the right eye to see only when the right image is displayed. The shuttering of the glasses can be achieved by mechanical shuttering or with liquid crystal electronic shuttering. In shuttering glass method, display images for the right and left eyes are alternately displayed on a CRT in a time sharing manner, and observation images for the right and left eyes are separated using time sharing shutter glasses which are opened/closed in a time sharing manner in synchronism with the display images, thus allowing an observer to recognize a stereoscopic image.
Other way to display stereoscopic images is by optical method. In this method, display images for the right and left eyes, which are separately displayed on a viewer using optical means such as prisms, mirror, lens, and the like, are superimpose-displayed as observation images in front of an observer, thus allowing the observer to recognize a stereoscopic image. Large convex or concave lenses can also be used where two image projectors, projecting left eye and right eye images, are providing focus to the viewer's left and right eye respectively. A variation of the optical method is the lenticular method where the images form on cylindrical lens elements or two dimensional array of lens elements. Figure 27 is a horizontal perspective display focusing on how the computer-generated person's two eye views are projected onto the Horizontal Plane and then displayed on a stereoscopic 3D capable viewing device. Figure 27 represents one complete display time period. During this display time period, the horizontal perspective display needs to generate two different eye views, because in this example the stereoscopic 3D viewing device requires a separate left- and right-eye view. There are existing stereoscopic 3D viewing devices that require more than a separate left- and right-eye view, and because the method described here can generate multiple views it works for these devices as well.
The illustration in the upper left of Figure 27 shows the Angled Camera point for the right eye after the first (right) eye-view to be generated. Once the first (right) eye view is complete, the horizontal perspective display starts the process of rendering the computer- generated person's second eye (left-eye) view. The illustration in the lower left of Figure 27 shows the Angled Camera point for the left eye after the completion of this time. But before the rendering process can begin, the horizontal perspective display makes an adjustment to the Angled Camera point. This is illustrated in Figure 27 by the left eye's x coordinate being incremented by two inches. This difference between the right eye's x value and the left eye's x + 2" is what provides the two-inch separation between the eyes, which is required for stereoscopic 3D viewing. The distances between people's eyes vary but in the above example we are using the average of 2 inches. It is also possible for the view to supply the horizontal perspective display with their personal eye separation value. This would make the x value for the left and right eyes highly accurate for a given viewer and thereby improve the quality of their stereoscopic 3D view. Once the horizontal perspective display has incremented the Angled Camera point's x coordinate by two inches, or by the personal eye separation value supplied by the viewer, the rendering continues by displaying the second (left-eye) view.
Depending on the stereoscopic 3D viewing device used, the horizontal perspective display continues to display the left- and right-eye images, as described above, until it needs to move to the next display time period. An example of when this may occur is if the bear cub moves his paw or any part of his body. Then a new and second Simulated Image would be required to show the bear cub in its new position. This new Simulated Image of the bear cub, in a slightly different location, gets rendered during a new display time period. This process of generating multiple views via the nonstop incrementing of display time continues as long as the horizontal perspective display is generating realtime simulations in stereoscopic 3D.
By rapidly display the horizontal perspective images, three dimensional illusion of motion can be realized. Typically, 30 to 60 images per second would be adequate for the eye to perceive motion. For stereoscopy, the same display rate is needed for superimposed images, and twice that amount would be needed for time sequential method.
The display rate is the number of images per second that the display uses to completely generate and display one image. This is similar to a movie projector where 24 times a second it displays an image. Therefore, 1/24 of a second is required for one image to be displayed by the projector. But the display time could be a variable, meaning that depending on the complexity of the view volumes it could take 1/12 or lA a second for the computer to complete just one display image. Since the display was generating a separate left and right eye view of the same image, the total display time is twice the display time for one eye image.
Figure 28 shows a horizontal plane as related to both central perspective andhorizontal perspective.
The present invention hands-on simulator further includes technologies employed in computer "peripherals". Figure 29 shows examples of such Peripherals with six degrees of freedom, meaning that their coordinate system enables them to interact at any given point in an (x, y, z) space. The simulator creates a "Peripheral Open-Access Volume," for each Peripheral the end-user requires, such as the Space Glove in Figure 29. Figure 30 is a high-level illustration of the Hands-On Simulation Tool, focusing on how a Peripheral's coordinate system is implemented within the Hands-On Simulation Tool.
The new Peripheral Open-Access Volume, which as an example in Figure 30 is labeled "Space Glove," is mapped one-to-one with the "Open- Access Real Volume" and "Open- Access Computer-generated Volume." The key to achieving a precise one-to-one mapping is to calibrate the Peripheral's volume with the Common Reference, which is the physical View surface, located at the viewing surface of the display device.
Some Peripherals provide a mechanism that enables the Hands-On Simulation Tool to perform this calibration without any end-user involvement. But if calibrating the Peripheral requires external intervention than the end-user will accomplish this through an "Open- Access Peripheral Calibration" procedure. This procedure provides the end- user with a series of Simulations within the Hands-On Volume and a user-friendly interface that enables them to adjusting the location of the Peripheral's volume until it is in perfect synchronization with the View surface. When the calibration procedure is complete, the Hands-On Simulation Tool saves the information in the end-user's personal profile.
Once the Peripheral's volume is precisely calibrated to the View surface, the next step in the process can be taken. The Hands-On Simulation Tool will continuously track and map the Peripheral's volume to the Open-Access Volumes. The Hands-On Simulation Tool modifies each Hands-On Image it generates based on the data in the Peripheral's volume. The end result of this process is the end-user's ability to use any given Peripheral to interact with Simulations within the Hands-On Volume generated in real-time by the Hands-On Simulation Tool.
With the peripherals linking to the simulator, the user can interact with the display model. The Simulation Engine can get the inputs from the user through the peripherals, and manipulate the desired action. With the peripherals properly matched with the physical space and the display space, the simulator can provide proper interaction and display. The invention Hands-On Simulator then can generate a totally new and unique computing experience in that it enables an end user to interact physically and directly (Hands-On) with real-time computer-generated 3D graphics (Simulations), which appear in open space above the viewing surface of a display device, i.e. in the end user's own physical space. The peripheral tracking can be done through camera triangulation or through infrared tracking devices.
The simulator can further include 3D audio devices for "SIMULATION RECOGNITION & 3D AUDIO ". This results in a new invention in the form of a Hands- On Simulation Tool with its Camera Model, Horizontal Multi-View Device, Peripheral Devices, Frequency Receiving/Sending Devices, and Handheld Devices as described below.
Object Recognition is a technology that uses cameras and/or other sensors to locate simulations by a method called triangulation. Triangulation is a process employing trigonometry, sensors, and frequencies to "receive" data from simulations in order to determine their precise location in space. It is for this reason that triangulation is a mainstay of the cartography and surveying industries where the sensors and frequencies they use include but are not limited to cameras, lasers, radar, and microwave. 3D Audio also uses triangulation but in the opposite way 3D Audio "sends" or projects data in the form of sound to a specific location. But whether you're sending or receiving data the location of the simulation in three-dimensional space is done by triangulation with frequency receiving sending devices. By changing the amplitudes and phase angles of the sound waves reaching the user's left and right ears, the device can effectively emulate the position of the sound source. The sounds reaching the ears will need to be isolated to avoid interference. The isolation can be accomplished by the use of eaφhones or the like.
Figure 31 shows an end-user looking at a Hands-On Image of a bear cub. Since the cub appears in open space above the viewing surface the end-user can reach in and manipulate the cub by hand or with a handheld tool. It is also possible for the end-user to view the cub from different angles, as they would in real life. This is accomplished though the use of triangulation where the three real- world cameras continuously send images from their unique angle of view to the Hands-On Simulation Tool. This camera data of the real world enables the Hands-On Simulation Tool to locate, track, and map the end-user's body and other real-world simulations positioned within and around the computer monitor's viewing surface (Figure 32).
Figure 33 also shows the end-user viewing and interacting with the bear cub, but it includes 3D sounds emanating from the cub's mouth. To accomplish this level of audio quality requires physically combining each of the three cameras with a separate speaker, as shown in Figure 32. The cameras' data enables the Hands-On Simulation Tool to use triangulation in order to locate, track, and map the end-user's "left and right ear". And since the Hands-On Simulation Tool is generating the bear cub as a computer-generated Hands-On Image it knows the exact location of the cub's mouth. By knowing the exact location of the end-user's ears and the cub's mouth the Hands-On Simulation Tool uses triangulation to sends data, by modifying the spatial characteristics of the audio, making it appear that 3D sound is emanating from the cub's computer-generated mouth.
Create a new frequency receiving/sending device by combining a video camera with an audio speaker, as previously shown in Figure 31. Note that other sensors and/or transducers may be used as well.
Take these new camera/speaker devices and attach or place them nearby a viewing device, such as a computer monitor as previously shown in Figure 32. This results in each camera/speaker device having a unique and separate "real-world" (x, y, z) location, line- of-sight, and frequency receiving/sending volume. To understand these parameters think of using a camcorder and looking through its view finder When you do this the camera has a specific location in space, is pointed in a specific direction, and all the visual frequency information you see or receive through the view finder is its "frequency receiving volume". Triangulation works by separating and positioning each camera/speaker device such that their individual frequency receiving/sending volumes overlap and cover the exact same area of space. If you have three widely spaced frequency receiving/sending volumes covering the exact same area of space than any simulation within the space can accurately be located. The next step creates a new element in the Open- Access Camera Model for this real-world space and in Figure 33 it is labeled "real frequency receiving/sending volume".
Now that this real frequency receiving/sending volume exists it must be calibrated to the Common Reference, which of course is the real View Surface. The next step is the automatic calibration of the real frequency receiving/sending volume to the real View Surface. This is an automated procedure that is continuously performed by the Hands-On Simulation Tool in order to keep the camera/speaker devices correctly calibrated even when they are accidentally bumped or moved by the end-user, which is likely to occur.
Figure 34 is a simplified illustration of the complete Open- Access Camera Model and will assist in explaining each of the additional steps required to accomplish the scenarios described in Figures 32 and 33 above.
The simulator then performs simulation recognition by continuously locating and tracking the end-user's "left and right eye" and their "line-of-sight", continuously map the real- world left and right eye coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the computer-generated cameras coordinates to match the real-world eye coordinates that are being located, tracked, and mapped. This enables the real-time generation of Simulations within the Hands-On Volume based on the exact location of the end-user's left and right eye. Allowing the enft-user to" freely mόve"fheϊf 'head ahd dό around the Hands-On Image without distortion.
The simulator then perform simulation recognition by continuously locating and tracking the end-user's "left and right ear" and their "line-of-hearing", continuously map the real-world left- and right-ear coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the 3D Audio coordinates to match the real-world ear coordinates that are being located, tracked, and mapped. This enables the real-time generation of Open- Access sounds based on the exact location of the end-user's left and right ears. Allowing the end-user to freely move their head and still hear Open- Access sounds emanating from their correct location.
The simulator then perform simulation recognition by continuously locating and tracking the end-user's "left and right hand" and their "digits," i.e. fingers and thumbs, continuously map the real-world left and right hand coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the Hands- On Image coordinates to match the real-world hand coordinates that are being located, tracked, and mapped. This enables the real-time generation of Simulations within the Hands-On Volume based on the exact location of the end-user's left and right hands allowing the end-user to freely interact with Simulations within the Hands-On Volume.
The simulator then perform simulation recognition by continuously locating and tracking "handheld tools", continuously map these real-world handheld tool coordinates into the Open-Access Camera Model precisely where they are in real space, and continuously adjust the Hands-On Image coordinates to match the real-world handheld tool coordinates that are being located, tracked, and mapped. This enables the real-time generation of Simulations within the Hands-On' Volume based on the exact ocation'of the handheld tools allowing the end-user to freely interact with Simulations within the Hands-On Volume.
Figure 35 is intended to assist in further explaining unique discoveries regarding the new Open- Assess Camera Model and handheld tools. Figure 35 is a simulation of and end-user interacting with a Hands-On Image using a handheld tool. The scenario being illustrated is the end-user visualizing large amounts of financial data as a number of interrelated Open- Access 3D simulations. The end-user can probe and manipulated the Open- Access simulations by using a handheld tool, which in Figure 35 looks like a pointing device.
A "computer-generated attachment" is mapped in the form of an Open- Access computer-generated simulation onto the tip of a handheld tool, which in Figure 35 appears to the end-user as a computer-generated "eraser". The end-user can of course request that the Hands-On Simulation Tool map any number of computer-generated attachments to a given handheld tool. For example, there can be different computer- generated attachments with unique visual and audio characteristics for cutting, pasting, welding, painting, smearing, pointing, grabbing, etc. And each of these computer- generated attachments would act and sound like the real device they are simulating when they are mapped to the tip of the end-user's handheld tool.
The present invention further discloses a Multi-Plane display comprising a horizontal perspective display together with a non-horizontal central perspective display. Figure 36 illustrates an example of the present invention Multi-Plane display in which the Multi- Plane display is a computer monitor that is approximately "L" shaped when open. The end-user views the L-shaped computer monitor from' its concave side and at approximately a 45° angle to the bottom of the "L," as shown in Figure 36. From the end-user's point of view the entire L-shaped computer monitor appears as one single and seamless viewing surface. The bottom L of the display, positioned horizontally, shows horizontal perspective image, and the other branch of the L display shows central perspective image. The edge is the two display segments is preferably smoothly joined and can also having a curvilinear projection to connect the two displays of horizontal perspective and central perspective.
The Multi-Plane display can be made with one or more physical viewing surfaces. For example, the vertical leg of the "L" can be one physical viewing surface, such as flat panel display, and the horizontal leg of the "L" can be a separate flat panel display. The edge of the two display segments can be a non-display segment and therefore the two viewing surface are not continuous. Each leg of a Multi-Plane display is called a viewing plane and as you can see in the upper left of Figure 36 there is a vertical viewing plane and a horizontal viewing plane where a central perspective image is generated on the vertical plane and a horizontal perspective image is generated on the horizontal plane, and then blend the two images where the planes meet, as illustrated in the lower right of Figure 36.
Figure 36 also illustrates that a Multi-Plane display is capable of generating multiple views. Meaning that it can display single-view images, i.e. a one-eye perspective like the simulation in the upper left, and/or multi- view images, i.e. separate right and left eye views like the simulation in the lower right. And when the L-shaped computer monitor is not being used' by the end-user it can be closed "an l'fόo- 'lili e the simulation in the lower left.
Figure 37 is a simplified illustration of the present invention Multi-Plane display. In the upper right of Figure 37 is an example of a single- view image of a bear cub that is displayed on an L-shaped computer monitor. Normally a single-view or one eye image would be generated with only one camera point, but as you can see there are at least two camera points for the Multi-Plabe display even though this is a single-view example. This is because each viewing plane of a Multi-Plane device requires its own rendering perspective. One camera point is for the horizontal perspective image, which is displayed on the horizontal surface, and the other camera point is for the central perspective image, which is displayed on the vertical surface.
To generate both the horizontal perspective and central perspective images requires the creation of two camera eyepoints (which can be the same or different) as shown in Figure 37 for two different and separate camera points labeled OSI and CPI. The vertical viewing plane of the L-shaped monitor, as shown at the bottom of Figure 37, is the display surface for the central perspective images, and thus there is a need to define another common reference plane for this surface. As discussed above, the common reference plane is the plane where the images are display, and the computer need to keep track of this plane for the synchronization of the locations of the displayed images and the real physical locations. With the L-shaped Multi-Plane device and the two display surfaces, the Simulation can to generate the three dimansional images, a horizontal perspective image using (OSI) camera eyepoint, and a central perspective image using (CPI) camera eyepoint. The multi-plane display system can further include a curvilinear connection display section to blend the horizontal perspective and the central perspective images together at the location of the seam in the "L," as shown at the bottom of Figure 37. The multi-plane display system can continuously update and display what appears to be a single L-shaped image on the L-shaped Multi-Plane device.
Furthermore, the multi-plane display system can comprise multiple display surfaces together with multiple curvilinear blending sections as shown in Fig. 38. The multiple display surfaces can be a flat wall, multiple adjacent flat walls, a dome, and a curved wraparound panel.
The present invention multi-plane display system thus can simultaneously projecting a plurality of three dimensional images onto multiple display surfaces, one of which is a horizontal perspective image. Further, it can be a stereoscopic multiple display system allowing viewers to use their stereoscopic vision for three dimensional image presentation.
Since the multi-plane display system comprises at least two display surfaces, various requirements need to be addressed to ensure high fidelity in the three dimensional image projection. The display requirements are typically geometric accuracy, to ensure that objects and features of the image to be correctly positioned, edge match accuracy, to ensure continuity between display surfaces, no blending variation, to ensure no variation in luminance in the blending section of various display surfaces, and field of view, to ensure a continuous image from the eyepoint of the viewer.
Since the blending section of the multi-plane display system is preferably a curve surface, some distortion correction could be applied in order for the image projected onto the blending section surface to appear correct to the viewer. There are various solutions for providing distortion correction to a display system such as using a test pattern image, designing the image projection system for the specific curved blending display section, using special video hardware, utilizing a piecewise-linear approximation for the curved blending section. Still another distortion correction solution for the curve surface projection is to automatically computes image distortion correction for any given position of the viewer eyepoint and the projector.
Since the multi-plane display system comprises more than one display surface, care should be taken to minimize the seams and gaps between the edges of the respective displays. To avoid seams or gaps problem, there could be at least two image generators generating adjacent overlapped portions of an image. The overlapped image is calculated by an image processor to ensure that the projected pixels in the overlapped areas are adjusted to form the proper displayed images. Other solutions are to control the degree of intensity reduction in the overlapping to create a smooth transition from the image of one display surface to the next.
The three dimensional simulator would not be complete without a three dimensional audio or binaural simulation. Binaural simulation offers realism to the three dimensional simulation together with 3D visualization.
Similar to vision, hearing using one ear is called monoaural and hearing using two ears is called binaural. Hearing can provide the direction of the sound sources but with poorer resolution than vision, the identity and content of a sound source such as speech or music, and the nature of the environment via echoes, reverberation such as a normal room or an open field. The head and ears, and sometime the shoulder, function as an antenna system to provide information about the location, distance and environment of the sound sources. The brain can inteφrete properly the various kinds of sound arriving at the head such as direct sounds, diffractive sounds around the head and by interaction with the outer ears and shoulder, different sound amplitudes and different arrival time of the sounds. These acoustic modifications are called 'sound cues' and serve to provide us the directional acoustis information of the sounds.
Basically, the sound cues are related to timing, volume, frequency and reflection. In timing cues, the ears recognize the time the sound arrives and assume that the sound comes from the closest source. Further, with two ears separated about 8 inches apart, the delay of the sound reaching one ear with respect to the other ear can give a cue about the location of the sound source. The timing cue is stronger than the level cue in the sense that the listener locates the sound based on the first wave that reaches the ear, regardless of the loudness of any later arriving waves. In volume (or level) cues, the ears recognize the volume (or loudness) of the sound and assume that the sound coming from the loudest direction. With the binaural (two ears) effect, the amplitude difference between the ears is a strong cue for the localization of the sound source. In frequency (or equalization) cues, the ears recognize the frequency balance of the sound as it arrives in each ear since frontal sounds are directed into the eardrums, while rear sounds bounce off the external ear and thus having a high frequency roll off. In reflection cue, the sound bounces off various surfaces and are either dispersed or absorbed in varying degrees before reaching the ears multiple times. This reflections off the walls of the room and the foreknowledge of the difference between the way various floor coverings sound also contribute to localization. In addition, the body, especially the head, can move relative to the sound source to help in locate the sound.
The above various sound cues are scientifically classified into three types of spatial hearing cues: interaural time differences (ITDs), interaural level differences (ILDs), and head-related transfer functions (HRTFs). ITDs relate to the time for a sound to reach the ears and the time difference for reaching both ears. ILDs refer to the amplitude in the frequency spectrum of sound reaching the ears and also the amplitude differences of the sound frequencies as heard in both ears. HRTFs can provide the perception of distance by the changes in the timbre and distance dependencies, the time delay and directions of direct sound and reflections in echoic environments.
The HRTFs are a collection of spatial cues for a particular listener, including ITDs, ILDs and the reflections, diffractions and damping caused by the listener's body, head, outer ears and shoulder. The external ear, or pinna, has a significant contribution to the HRTFs. Higher frequency sounds are filtered by the pinna to provide the brain a way as to perceive the lateral position, or azimuth, and elevation of the sound source since the response of the pinna filter is highly dependent on the overall direction of the sound source. The head can account for a reduced amplitude of various frequencies of the sounds since the sound has to go through or around the head in order to reach the ear. The overall effects of head shadowing contribute to the perception of linear distance and direction of a sound source. Further, sound frequencies in the range of l-3kHz are reflected from the shoulder to produce echoes representing a time delay dependent on the elevation of the sound source. The reflections from surfaces in the world and the reverberation also seem to affect the localization judgement of sound distance and direction.
In addition to these cues, the movement of the head to help in locate the location of a sound source is a key factor, together with the vision to confirm the sound direction. For a 3D immersion, all mechanisms to localize the sounds are always in play and should normally agree. If not, there would be some discomfort and confusion.
Although we can hear with one ear, hearing with two ears is clearly better. Many of the sound cues are related to the binaural perception depending on both the relative loudness of sound and the relative time of arrival of sound at each ear. And thus the binaural performance is clear superior for the localization of single or multiple sound sources and for the formation of the room environment, for the separation of signals coming from multiple incoherent and coherent sound sources; and the enhancement of a chosen signal in a reverberant environment.
Mathematically speaking, HRTF is the frequency response of the sound waves as received by the ears. By measuring the HRTF of a particular listener, and by synthesised electronically using digital signal processing, the sounds can be delivered to a listener's ears via headphones or loudspeakers to create a virtual sound image in three dimensions.
The sound transformation to the ear canal, i.e. HRTF frequency response, can be measured accurately by using small microphones in the ear canals. The measured signal is then processed by a computer to derive the HRTF frequency responses for the left and right ears corresponding to the sound source location. Thus a 3D audio system works by using the measured HRTFs as the audio filters or equalizers. When a sound signal is processed by the HRTFs filters, the sound localization cues are reproduced, and the listener should perceive the sound at the location specified by the HRTFs. This method of binaural synthesis works extremely well when the listener's own HRTFs are used to synthesize the localization cues. However, measuring HRTFs is a complicated procedure, so 3D audio systems typically use a single set of HRTFs previously measured from a particular human or manikin subject. Thus the HRTF sometimes needs to be changed to accurately respond to a perticular listener. The tuning of a HRTF function can be accomplished by providing various sound source locations and environments and asking the listener to identify.
A 3D audio system should provide the ability for the listener to define a three- dimensional space, to position multiple sound sources and that listener in that 3D space, and to do it all in real-time, or interactively. Beside 3D audio system, other technologies such stereo extension and surround sound could offer some aspects of 3D positioning or interactivity.
Extended stereo processes an existing stereo (two channel) soundtrack to add spaciousness and to make it appear to originate from outside the left/right speaker locations through fairly straight-forward methods. Some of the characteristics of the extended stereo technology include the size of the listening area (called sweet spot), the amount of spreading of stereo images, the amount of tonal changes, the amount of lost stereo panning information, and the ability to achieve effect on headphones as well as speakers. The surround sound create a larger-than-stereo sound stage with a surround sound 5- speaker setup. Additionally, virtual surround sound systems use 3D audio technology to create the illusion of five speakers emanating from a regular set of stereo speakers, therefore enabling a surround sound listening experience without the need for a five speaker setup. The characteristics of the surround sound technology include the presentation accuracy, the clarity of spatial imaging, and the size of the listening area
For better 3D audio system, audio technology needs to create a life-like listening experience by replicating the 3D audio cues that the ears hear in the real world for allowing non-interactive and interactive listening and positioning of sounds anywhere in the three-dimensional space surrounding a listener.
The head tracker function is also very important to provide perceptual room constancy to the listener. In other words, when the listener move their heads around, the signals would change so that the perceived auditory world maintain its spatial position. To this end, the simulation system needs to know the head position in order to be able to control the binaural impulse responses adequately. Head position sensors have therefore to be provided. The impression of being immersed is of particular relevance for applications in the context of virtual reality.
A replica of a sound field can be produced by putting an infinite number of microphones everywhere. After being stored on a recorder with an infinite number of channels, this recording can then be played back through an infinite number of point- source loudspeakers, each placed exactly as its corresponding microphone was placed. As the number of microphones and speakers is reduced, the quality of the sound field being simulated suffers. By the time we are down to two channels, height cues have certainly been lost and instead of a stage that is audible from anywhere in the room we find that sources on the stage are now only localizable if we listen along a line equidistant from the last two remaining speakers and face them.
However, only two channels should be adequate, since if we deliver the exact sound required to simulate a live performance at the entrance to each ear canal, then since we only have two ear canals, we should only need to generate two such sound fields. In other words, since we can hear three-dimensionally in the real world using just two ears, it must be possible to achieve the same effect from just two speakers or a set of headphones.
Headphone reproduction is thus differed from loudspeaker reproduction since headphone microphones should be spaced about seven inches apart for a normal ear separation, and loudspeaker microphones separation should be about seven feet apart. Further loudspeakers suffer from crosstalk and therefore some signal conditioning such as crosstalk cancellation will be needed for 3D loudspeaker setup.
Loudspeaker 3D audio systems are extremely effective in desktop computing environments. This is because there is usually only a single listener (the computer user) who is almost always centered between the speakers and facing forward towards the monitor. Thus, the primary user gets the full 3D effect because the crosstalk is properly cancelled. In typical 3D audio applications, like video gaming, friends may gather around to watch. In this case, the best 3D audio effects are heard by others when they are also centered with respect to the loudspeakers. Off-center listeners may not get the full effect, but they still hear a high quality stereo program with some spatial enhancements. To achieve 3D audio, the speakers are typically arranged surrounding the listener in about the same horizontal plane, but could be arranged to completely surround the listener, from the ceiling to the floor to the surrounding walls. Optionally, the speakers can also be put on the ceiling, on the floor, arranged in an overhead dome configuration, or arranged in a vertical wall configuration. Further, beam transmitted speakers can be used instead of headphone. Beam transmitted speaker offers the freedom of movement for the listener and without the crosstalk between speakers since beam transmitted speaker provide a tight beam of sound.
Generally, a minimum of four loudspeakers are required to achieve a convincing 3-D audio experience, while some researchers are using twenty or more speakers in an anechoic chamber to recreate acoustic environments with much greater precision.
The main advantages of multi-speaker playback are:
- There is no dependence on the individual subject's HRTF, since the sound field is created without any reference to individual listeners.
- The subject is free to turn their head, and even move about within a limited range.
- In some cases, more than one subject can listen to the system simultaneously.
Many crosstalk cancellers are based on a highly simplified model of crosstalk, for example modeling crosstalk as a simple delay and attenuation process, or a delay and a lowpass filter. Other crosstalk cancellers have been based on a spherical head model. As with binaural synthesis, crosstalk cancellation performance is ultimately limited by the variation in the size and shape of human heads.
3D audio simulation can be accomplished by the following steps: - Input the characteristics of the acoustic space.
- Determine the sequence of sound arrivals that occur at the listening position. Each sound arrival will have the following characteristics: (a) time of arrival, based on the distance travelled by the echo-path, (b) direction of arrival, (c) attenuation (as a function of frequency) of the sound due to the absoφtion properties of the surfaces encountered by the echo-path.
- Compute the impulse response of the acoustic space incoφorating the multiple sound arrivals.
- The results from the FIR filter are played back to a listener. In the case where the impulse responses were computed using a dummy head response, the results are played over headphones to the listener. In this case, the equalisation required for the particular headphones is also applied.
The simulation of an acoustic environment involves one or more of the following functions :
- Processing an audio source input and presenting it to the subject through a number of loudspeakers (or headphones) with the intention of making the sound source appear to be located at a particular position in space.
- Processing multiple input audio sources in such a way that each source is independently located in space around the subject.
- Enhanced processing to simulate some aspects of the room acoustics, so that the user can acoustically sense the size of the room and the nature of the floor and wall coverings. - The capability for the subject to move (perhaps within a limited range) and turn his/her head so as to focus attention on some aspects of the sound source characteristics or room acoustics.
Binaural simulation is generally carried out using the sound source material free from any unwanted echoes or noise. The sound source material can then be replayed to a subject, using the appropriate HRTF filters, to create the illusion that the source audio is originating from a particular direction. The HRTF filtering is achieved by simply convolving the audio signal with the pair of HRTF responses (one HRTF filter for each channel of the headphone).
The eyes and ears often perceive an event at the same time. Seeing a door close, and hearing a shutting sound, are inteφreted as one event if they happen synchronously. If we 'See a door shut without a sound, or we see a door shut in front of us, and hear a shutting sound to the left, we get alarmed and confused. In another scenario, we might hear a voice in front of us, and see a hallway with a corner; the combination of audio and visual cues allows us to figure out that a person might be standing around the corner. Together, synchronized 3D audio and 3D visual cues provide a very strong immersion experience. Both 3D audio and 3D graphics systems can be greatly enhanced by such synchronization.
Improved playback through headphones can be achieved through the use of head tracking. This technique makes use of continuous measurements of the orientation of a subject's head, and adapts the audio signals being fed to the headphones appropriately. Binaural signal should allow a subject to easily discriminate between left and right sound source locations easily, but the ability to discriminate between front and back, and high and low sound sources is generally only possible if head movement is permitted. Whilst multiple speaker playback methods solve this problem to a large degree, there are still many applications where headphone playback is preferable, and head tracking can then be used as a valuable tool for improving the quality of the 3-D playback.
The simplest form of head tracking binaural system is one which simply simulates anechoic HRTFs, and changes the HRTF functions rapidly in response to the subjects head movements. This HRTF switching can be achieved through a lookup table, with inteφolation used to resolve angles that are not represented in the HRTF table.
Simulation of room acoustics over headphones with head tracking becomes more difficult because the direction of arrival of the early reflections is also important in making the result sound realistic. Many researchers believe that the echoes in the reverberant tail of the room response are generally so diffuse that there is no requirement for this part of the room response to be tracked with the subject's head movements.
An important feature of any head tracking playback system is the delay from the subject head movement to the change in the audio response at the headphones. If this delay is excessive, the subject can experience a form of virtual motion sickness and general disorientation.
Audio cues change dramatically when a listener tilts or rotates his or her head. For example, quickly turning the head 90 degrees to look to the side is the equivalent of a sound traveling from the listener's side to the front in a split second. We often use head motion to track sounds or to search for them. The ears alert the brain about an event outside of the area that the eyes are currently focused on, and we automatically turn to redirect our attention. Additionally, we use head motion to resolve ambiguities: a faint, low sound could be either in front or back of us, so we quickly and sub-consciously turn our head a small fraction to the left, and we know if the sound is now off to the right, it is in the front, otherwise it is in the back. One of the reasons why interactive audio is more realistic than pre-recorded audio (soundtracks) is the fact that the listeners head motion can be properly simulated in an interactive system (using inputs from a joystick, mouse, or head-tracking system).
The HRTF function are performed using digital signal processing (DSP) hardware for real time performance. Typical feature of DSP are that the direct sound must be processed to give the correct amplitude and perceived direction, the early echoes must arrive at the listener with appropriate time, amplitude and frequency response to give the perception of the size of the spaces (as well as the acoustic nature of the room surfaces), and the late reverberation must be natural and correctly distributed in 3-D around the listener. The relative amplitude of the direct sound compared to the remainder of the room response helps to provide the sensation of distance.
Thus 3D audio simulation can provides a binaural gain so that the exact same audio content is more audible and intelligible in the binaural case, because the brain can localize and therefore "single out" the binaural signal, while the non-binaural signal gets washed into the noise. Further the listener would still be able to tune into and understand individual conversations, because they are still spatially separated, and "amplified by" binaural gain, an effect called the cocktail party effct. Binaural simulation also can provide faster reaction time because such a signal mirrors the ones received in the real world. In addition, binaural signals can convey positional information: a binaural radar warning sound can warn a user about a specific object that is approaching (with a sound that is unique to that object), and naturally indicate where that object is coming from. Also listening to binaural simulation can beless fatigue since we are used to hearing sounds that originate outside of their heads, as is the case with binaural signals. Mono or stereo signals appear to come from inside a listener's head when using headphones, and produce more strain than a natural sounding, binaural signal. An lastly, 3D binaural simulation can provide an increased perception and immersion in higher quality 3D environment when visuals are shown in synch with binaural sound.

Claims

What is claimed is:
1. A 3-D horizontal perspective simulator system comprising a first horizontal perspective display using horizontal perspective to display a 3-D image onto an open space; a second display showing information related to the 3-D image; and a peripheral device to manipulate the display image by touching the 3-D image.
2. A simulator system as in claim 1 wherein the second display displays 2D images or central perspective images.
3. A simulator system as in claim 1 further comprising a third curvilinear display blending the first and the second displays.
4. A simulator system as in claim 1 further comprising a processing unit taking the input from the peripheral device and providing output to the first horizontal perspective display.
5. A simulator system as in claim 1 further comprising a processing unit taking the input from the second display and providing output to the first horizontal perspective display.
6. A simulator system as in claim 1 further comprising a means to track the physical peripheral device to the 3-D image.
7. A simulator system as in claim 1 further comprising a means to calibrate the physical peripheral device to the 3-D image.
8. A 3-D horizontal perspective simulator system comprising a processing unit; a first horizontal perspective display using horizontal perspective to display a 3-D image onto an open space; a second display showing information related to the 3-D image; a peripheral device to manipulate the display image by touching the 3-D image; and a peripheral device tracking unit for mapping the peripheral device to the 3-D image.
9. A simulator system as in claim 8 fiirther comprising a third curvilinear display blending the first and the second displays.
10. A simulator system as in claim 8 further comprising a processing unit taking the input from the second display and providing output to the first horizontal perspective display.
11. A simulator system as in claim 8 wherein the first horizontal perspective display further display a portion of the 3-D image onto an inner-access volume, whereby the image portion in the inner-access volume cannot be touched by the peripheral device.
12. A simulator system as in claim 8 wherein the first horizontal perspective display further comprises automatic or manual eyepoint tracking to synchronize the camera eyepoint of the horizontal perspective display with an user's eyepoint.
13. A simulator system as in claim 8 wherein the first horizontal perspective display further comprises a means to zoom, rotation or movement of the 3-D image.
14. A simulator system as in claim 8 wherein the peripheral device is a tool, a handheld tool, a space glove or a pointing device.
15. A simulator system as in claim 8 wherein the peripheral device comprises a tip wherein the manipulation corresponds to the tip of the peripheral device.
16. A simulator system as in claim 8 wherein the manipulation comprises the action of modifying the display image or the action of generating a different image.
17. A simulator system as in claim 8 wherein the peripheral device mapping comprises inputting the position of the peripheral device to the processing unit.
18. A simulator system as in claim 8 wherein the peripheral device tracking unit comprises a triangulation or infrared tracking system.
19. A simulator system as in claim 18 further comprising a means to calibrate the coordinate of the display image to the peripheral device.
20. A multi- view 3-D horizontal perspective simulator system comprising a processing unit; a first stereoscopic horizontal perspective display using horizontal perspective to display a stereoscopic 3-D image onto an open space; a second display showing information related to the 3-D image; a peripheral device to manipulate the display image by touching the 3-D image; and a peripheral device tracking unit for mapping the peripheral device to the 3-D image.
21. A method for 3-D horizontal perspective simulation by horizontal perspective projection, the horizontal perspective projection comprising displaying horizontal perspective images according to a predetermined projection eyepoint, the method comprising the steps of: displaying a 3-D image onto an open space of a first display surface using horizontal perspective; display a second image onto a second display; and manipulating the display image on the first display surface by touching the 3- D image with a peripheral device.
22. A method as in claim 21 further comprising the step of taking an input from the second display and providing output to the first horizontal perspective display.
23. A method as in claim 21 further comprising a step of tracking the physical peripheral device to the 3-D image.
24. A method as in claim 23 wherein tracking the peripheral device comprises tracking a tip of the peripheral device.
25. A method as in claim 23 wherein the peripheral device tracking comprises inputting the position of the peripheral device to the processing unit.
26. A method as in claim 23 wherein the peripheral device tracking comprises a step of triangulation or infrared tracking.
27. A method as in claim 21 further comprising a step of calibrating the physical peripheral device to the 3-D image.
28. A method as in claim 27 wherein the calibration step comprises a manual inputting a reference coordinate.
29. A method as in claim 27 wherein the calibration step comprises an automatic inputting a reference coordinate through a calibration procedure.
30. A method as in claim 21 further comprising a step of display a third image onto a third curvilinear display, the curvilinear display blending the first display and the second display.
31. A method as in claim 21 wherein the horizontal perspective display is a stereoscopic horizontal perspective display using horizontal perspective to display a stereoscopic 3-D image.
32. A method as in claim 21 wherein the horizontal perspective display further display a portion of the 3-D image onto an inner-access volume, whereby the image portion in the inner-access volume cannot be touched by the peripheral device.
33. A method as in claim 21 further comprising a step of automatic or manual eyepoint tracking for the horizontal perspective display.
34. A method as in claim 21 further comprising a step of zooming, rotating or moving the 3-D image.
35. A method as in claim 21 wherein manipulating the display image by the peripheral device comprises tracking a tip of the peripheral device.
36. A method as in claim 35 wherein the manipulation comprises the action of modifying the display image or the action of generating a different image.
37. A 3-D simulation method using a 3-D horizontal perspective simulator system, the 3-D horizontal perspective simulator system comprising a processing unit; a first horizontal perspective display using horizontal perspective to display a 3-D image onto an open space; a second display showing information related to the 3-D image; a peripheral device to manipulate the display image by touching the 3-D image; and a peripheral device tracking unit for mapping the peripheral device to the 3-D image; the method comprising calibrating the peripheral device; displaying a first 3-D image onto an open space of the first display surface using horizontal perspective; displaying a second image onto the second display; tracking the peripheral device; and manipulating the display image by touching the 3-D image with the peripheral device.
38. A method as in claim 37 further comprising a step of display a third image onto a third curvilinear display, the curvilinear display blending the first display and the second display.
39. A 3-D simulation method using a multi-view 3-D horizontal perspective simulator system, the multi- view 3-D horizontal perspective simulator system comprising a processing unit; a first stereoscopic horizontal perspective display using horizontal perspective to display a stereoscopic 3-D image onto an open space; and a second display showing information related to the 3-D image; a peripheral device to manipulate the display image by touching the 3-D image; and a peripheral device tracking unit for mapping the peripheral device to the 3-D image; the method comprising displaying a first stereoscopic 3-D image onto an open space of the first display surface using horizontal perspective; displaying a second image onto the second display; tracking the peripheral device; and manipulating the display image by touching the 3-D image with a peripheral device.
40. A method as in claim 39 further comprising a step of display a third image onto a third curvilinear display, the curvilinear display blending the first display and the second display.
41. A 3-D horizontal perspective simulator system comprising a first horizontal perspective display using horizontal perspective to display a 3-D image onto an open space according to a predetermined projection eyepoint; a peripheral device to manipulate the display image by touching the 3-D image; and a 3-D audio simulation system providing 3-D sound to a predetermined projection eaφoint, the 3-D sound conesponded to the horizontal perspective 3-D images.
42. A 3-D horizontal perspective simulator system comprising a first horizontal perspective display using horizontal perspective to display a 3-D image onto an open space according to a predetermined projection eyepoint; a peripheral device to manipulate the display image by touching the 3-D image; a 3-D audio simulation system providing 3-D sound to a predetermined projection eaφoint, the 3-D sound corresponded to the horizontal perspective images; and an input device for accepting an input location for controlling the 3-D image or the 3-D sound.
43. A simulator system as in claim 42 wherein the 3-D audio simulation system comprises two sound channels and a HRTF (head related transfer function) filter.
44. A simulator system as in claim 42 wherein the 3-D audio simulation system comprises a 3-D loudspeaker audio system or a 3-D headphone audio system.
45. A simulator system as in claim 42 wherein the input device functions as an eyepoint input device for accepting an input eyepoint location wherein the 3-D image can be adjusted using the input eyepoint as the projection eyepoint
46. A simulator system as in claim 42 wherein the input device functions as an eaφoint input device for accepting an input eaφoint location wherein the 3-D sound can be adjusted using the input eaφoint as the projection eaφoint
47. A simulator system as in claim 42 wherein the input device is an automatic input device whereby the automatic input device automatically extracts the eyepoint location or the ear point location from the viewer.
48. A simulator system as in claim 47 wherein the automatic input device is selected from a group consisted of radio-frequency tracking device, infrared tracking device, camera tracking device.
49. A simulator system as in claim 42 further comprising an image input device for accepting an image command; wherein the computer system further accepts an image command from the image input device, calculating a horizontal perspective projection image according to the image command using the input eyepoint location as the projection eyepoint before outputting the image to the display.
50. A simulator system as in claim 49 wherein the image command includes image magnification, image movement, image rotation command and command to display another predetermined image.
51. A simulator system as in claim 42 further comprising a second display positioned at an angle to the first display.
52. A simulator system as in claim 51 further comprising a third curvilinear display blending the first and the second displays.
53. A simulator system as in claim 51 further comprising a processing unit taking the input from the second display and providing output to the first horizontal perspective display.
54. A simulator system as in claim 42 wherein the peripheral device is a tool, a handheld tool, a space glove or a pointing device.
55. A simulator system as in claim 42 wherein the peripheral device comprises a tip wherein the manipulation corresponds to the tip of the peripheral device.
56. A simulator system as in claim 42 wherein the manipulation comprises the action of modifying the display image or the action of generating a different image.
57. A simulator system as in claim 42 further comprising a means to track the physical peripheral device to the 3-D image.
58. A simulator system as in claim 42 wherein the peripheral device tracking unit comprises a triangulation or infrared tracking system.
59. A simulator system as in claim 42 further comprising a means to calibrate the coordinate of the display image to the peripheral device.
60. A simulator system as in claim 42 wherein the horizontal perspective display is a stereoscopic horizontal perspective display using horizontal perspective to display a stereoscopic 3-D image.
61. A method for 3-D horizontal perspective simulation by horizontal perspective projection, the horizontal perspective projection comprising displaying horizontal perspective images according to a predetermined projection eyepoint, the method comprising the steps of: displaying a 3-D image onto an open space of a first display surface using horizontal perspective; presenting 3-D sound to a predetermined projection eaφoint corresponding to the 3-D image; and manipulating the display image on the first display surface by touching the 3- D image with a peripheral device.
62. A method for 3-D horizontal perspective simulation by horizontal perspective projection, the horizontal perspective projection comprising displaying horizontal perspective images according to a predetermined projection eyepoint, the method comprising the steps of: displaying a 3-D image onto an open space of a first display surface using horizontal perspective; display a second image onto a second display; presenting 3-D sound to a predetermined projection eaφoint corresponding to the 3-D image; and manipulating the display image on the first display surface by touching the 3- D image with a peripheral device.
63. A method as in claim 62 wherein presenting 3-D sound comprises outputting two channel sound through a HRTF (head related transfer function) filter.
64. A method as in claim 62 wherein presenting 3-D sound comprises outputting sound through a 3-D loudspeaker audio system or a 3-D headphone audio system.
65. A method as in claim 62 further comprising the step of taking an input from the second display and providing output to the first horizontal perspective display.
66. A method as in claim 62 further comprising a step of tracking the physical peripheral device to the 3-D image.
67. A method as in claim 66 wherein tracking the peripheral device comprises tracking a tip of the peripheral device.
68. A method as in claim 66 wherein the peripheral device tracking comprises inputting the position of the peripheral device to the processing unit.
69. A method as in claim 66 wherein the peripheral device tracking comprises a step of triangulation or infrared tracking.
70. A method as in claim 62 further comprising a step of display a third image onto a third curvilinear display, the curvilinear display blending the first display and the second display.
71. A method as in claim 62 wherein the horizontal perspective display is a stereoscopic horizontal perspective display using horizontal perspective to display a stereoscopic 3-D image.
72. A method as in claim 62 further comprising a step of automatic or manual eyepoint tracking for the horizontal perspective display.
73. A method as in claim 72 further wherein the eyepoint tracking further acts as an eaφoint tracking.
74. A method as in claim 62 further comprising a step of automatic or manual eaφoint tracking for the 3-D sound projection.
75. A method as in claim 62 further comprising a step of zooming, rotating or moving the 3-D image.
76. A method as in claim 62 wherein manipulating the display image by the peripheral device comprises tracking a tip of the peripheral device.
77. A method as in claim 76 wherein the manipulation comprises the action of modifying the display image or the action of generating a different image.
78. A 3-D simulation method using a 3-D horizontal perspective simulator system, the 3-D horizontal perspective simulator system comprising a processing unit; a first horizontal perspective display using horizontal perspective to display a 3-D image onto an open space; a second display showing information related to the 3-D image; a 3-D audio simulation system providing 3-D sound to a predetermined projection eaφoint; a peripheral device to manipulate the display image by touching the 3-D image; and a peripheral device tracking unit for mapping the peripheral device to the 3-D image;
the method comprising calibrating the peripheral device; displaying a first 3-D image onto an open space of the first display surface using horizontal perspective; displaying a second image onto the second display; presenting 3-D sound corresponding to the 3-D image; tracking the peripheral device; and manipulating the display image by touching the 3-D image with the peripheral device.
A method as in claim 78 wherein the 3-D audio simulation system comprises two sound channels and a HRTF (head related transfer function) filter. A method as in claim 78 further comprising a step of display a third image onto a third curvilinear display, the curvilinear display blending the first display and the second display.
PCT/US2005/019069 2004-06-01 2005-05-31 Horizontal perspective simulator WO2005118998A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
EP05757162A EP1781893A1 (en) 2004-06-01 2005-05-31 Horizontal perspective simulator
JP2007515486A JP2008507006A (en) 2004-06-01 2005-05-31 Horizontal perspective simulator

Applications Claiming Priority (8)

Application Number Priority Date Filing Date Title
US57618704P 2004-06-01 2004-06-01
US57618904P 2004-06-01 2004-06-01
US57618104P 2004-06-01 2004-06-01
US57618204P 2004-06-01 2004-06-01
US60/576,181 2004-06-01
US60/576,182 2004-06-01
US60/576,187 2004-06-01
US60/576,189 2004-06-01

Publications (2)

Publication Number Publication Date
WO2005118998A1 true WO2005118998A1 (en) 2005-12-15
WO2005118998A8 WO2005118998A8 (en) 2006-04-27

Family

ID=35462954

Family Applications (2)

Application Number Title Priority Date Filing Date
PCT/US2005/019068 WO2005119376A2 (en) 2004-06-01 2005-05-31 Horizontal perspective display
PCT/US2005/019069 WO2005118998A1 (en) 2004-06-01 2005-05-31 Horizontal perspective simulator

Family Applications Before (1)

Application Number Title Priority Date Filing Date
PCT/US2005/019068 WO2005119376A2 (en) 2004-06-01 2005-05-31 Horizontal perspective display

Country Status (5)

Country Link
US (8) US7796134B2 (en)
EP (2) EP1759379A2 (en)
JP (2) JP2008507006A (en)
KR (2) KR20070052261A (en)
WO (2) WO2005119376A2 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011044936A1 (en) * 2009-10-14 2011-04-21 Nokia Corporation Autostereoscopic rendering and display apparatus

Families Citing this family (145)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7796134B2 (en) 2004-06-01 2010-09-14 Infinite Z, Inc. Multi-plane horizontal perspective display
CA2482240A1 (en) * 2004-09-27 2006-03-27 Claude Choquet Body motion training and qualification system and method
JP4046121B2 (en) * 2005-03-24 2008-02-13 セイコーエプソン株式会社 Stereoscopic image display apparatus and method
EP1889171A4 (en) * 2005-04-07 2012-11-28 Visionsense Ltd Method for reconstructing a three- dimensional surface of an object
US8717423B2 (en) 2005-05-09 2014-05-06 Zspace, Inc. Modifying perspective of stereoscopic images based on changes in user viewpoint
WO2006121957A2 (en) 2005-05-09 2006-11-16 Michael Vesely Three dimensional horizontal perspective workstation
US20060285832A1 (en) * 2005-06-16 2006-12-21 River Past Corporation Systems and methods for creating and recording digital three-dimensional video streams
KR101309176B1 (en) * 2006-01-18 2013-09-23 삼성전자주식회사 Apparatus and method for augmented reality
JP4940671B2 (en) * 2006-01-26 2012-05-30 ソニー株式会社 Audio signal processing apparatus, audio signal processing method, and audio signal processing program
US9104195B2 (en) 2006-12-20 2015-08-11 Lincoln Global, Inc. Welding job sequencer
US10994358B2 (en) 2006-12-20 2021-05-04 Lincoln Global, Inc. System and method for creating or modifying a welding sequence based on non-real world weld data
US9937577B2 (en) 2006-12-20 2018-04-10 Lincoln Global, Inc. System for a welding sequencer
GB2447060B (en) 2007-03-01 2009-08-05 Magiqads Sdn Bhd Method of creation of a virtual three dimensional image to enable its reproduction on planar substrates
US8665180B2 (en) 2007-03-02 2014-03-04 Nec Corporation Image display device
WO2008108389A1 (en) * 2007-03-07 2008-09-12 Nec Corporation Image display
JP2008226400A (en) * 2007-03-15 2008-09-25 Sony Computer Entertainment Inc Audio reproducing system and audio reproducing method
US8269822B2 (en) * 2007-04-03 2012-09-18 Sony Computer Entertainment America, LLC Display viewing system and methods for optimizing display view based on active tracking
US8400493B2 (en) * 2007-06-25 2013-03-19 Qualcomm Incorporated Virtual stereoscopic camera
US8355019B2 (en) * 2007-11-02 2013-01-15 Dimension Technologies, Inc. 3D optical illusions from off-axis displays
US20090125801A1 (en) * 2007-11-10 2009-05-14 Cherif Atia Algreatly 3D windows system
US8452052B2 (en) * 2008-01-21 2013-05-28 The Boeing Company Modeling motion capture volumes with distance fields
JP4991621B2 (en) * 2008-04-17 2012-08-01 キヤノン株式会社 Imaging device
CN101610360A (en) * 2008-06-19 2009-12-23 鸿富锦精密工业(深圳)有限公司 The camera head of automatically tracking sound source
KR101908033B1 (en) 2008-07-10 2018-12-10 리얼 뷰 이미징 리미티드 Broad viewing angle displays and user interfaces
US8884177B2 (en) 2009-11-13 2014-11-11 Lincoln Global, Inc. Systems, methods, and apparatuses for monitoring weld quality
US9280913B2 (en) 2009-07-10 2016-03-08 Lincoln Global, Inc. Systems and methods providing enhanced education and training in a virtual reality environment
US9318026B2 (en) 2008-08-21 2016-04-19 Lincoln Global, Inc. Systems and methods providing an enhanced user experience in a real-time simulated virtual reality welding environment
US8915740B2 (en) 2008-08-21 2014-12-23 Lincoln Global, Inc. Virtual reality pipe welding simulator
US8851896B2 (en) 2008-08-21 2014-10-07 Lincoln Global, Inc. Virtual reality GTAW and pipe welding simulator and setup
US8911237B2 (en) 2008-08-21 2014-12-16 Lincoln Global, Inc. Virtual reality pipe welding simulator and setup
US9330575B2 (en) 2008-08-21 2016-05-03 Lincoln Global, Inc. Tablet-based welding simulator
US9483959B2 (en) 2008-08-21 2016-11-01 Lincoln Global, Inc. Welding simulator
US9196169B2 (en) 2008-08-21 2015-11-24 Lincoln Global, Inc. Importing and analyzing external data using a virtual reality welding system
US8657605B2 (en) 2009-07-10 2014-02-25 Lincoln Global, Inc. Virtual testing and inspection of a virtual weldment
US8747116B2 (en) 2008-08-21 2014-06-10 Lincoln Global, Inc. System and method providing arc welding training in a real-time simulated virtual reality environment using real-time weld puddle feedback
US8834168B2 (en) * 2008-08-21 2014-09-16 Lincoln Global, Inc. System and method providing combined virtual reality arc welding and three-dimensional (3D) viewing
US20110187844A1 (en) * 2008-09-12 2011-08-04 Kabushiki Kaisha Toshiba Image irradiation system and image irradiation method
US20100128112A1 (en) * 2008-11-26 2010-05-27 Samsung Electronics Co., Ltd Immersive display system for interacting with three-dimensional content
US20100156907A1 (en) * 2008-12-23 2010-06-24 Microsoft Corporation Display surface tracking
FR2942096B1 (en) * 2009-02-11 2016-09-02 Arkamys METHOD FOR POSITIONING A SOUND OBJECT IN A 3D SOUND ENVIRONMENT, AUDIO MEDIUM IMPLEMENTING THE METHOD, AND ASSOCIATED TEST PLATFORM
US8274013B2 (en) * 2009-03-09 2012-09-25 Lincoln Global, Inc. System for tracking and analyzing welding activity
TW201039099A (en) * 2009-04-22 2010-11-01 Song-Yuan Hu Electronic device
US8279269B2 (en) * 2009-04-29 2012-10-02 Ke-Ou Peng Mobile information kiosk with a three-dimensional imaging effect
US20100306825A1 (en) * 2009-05-27 2010-12-02 Lucid Ventures, Inc. System and method for facilitating user interaction with a simulated object associated with a physical location
KR101647536B1 (en) * 2009-05-29 2016-08-10 휴렛-팩커드 디벨롭먼트 컴퍼니, 엘.피. Multi-projector system and method
US8269902B2 (en) * 2009-06-03 2012-09-18 Transpacific Image, Llc Multimedia projection management
JP4681690B2 (en) * 2009-06-11 2011-05-11 新日本製鐵株式会社 Manufacturing method of thick high strength steel plate with excellent heat input heat affected zone toughness, and thick high strength steel plate with high heat input heat affected zone toughness
CN102804789B (en) 2009-06-23 2015-04-29 Lg电子株式会社 Receiving system and method of providing 3D image
KR20120039703A (en) 2009-07-07 2012-04-25 엘지전자 주식회사 Method for displaying three-dimensional user interface
US9773429B2 (en) 2009-07-08 2017-09-26 Lincoln Global, Inc. System and method for manual welder training
US9221117B2 (en) 2009-07-08 2015-12-29 Lincoln Global, Inc. System for characterizing manual welding operations
US10748447B2 (en) 2013-05-24 2020-08-18 Lincoln Global, Inc. Systems and methods providing a computerized eyewear device to aid in welding
US9011154B2 (en) 2009-07-10 2015-04-21 Lincoln Global, Inc. Virtual welding system
US9294751B2 (en) 2009-09-09 2016-03-22 Mattel, Inc. Method and system for disparity adjustment during stereoscopic zoom
EP2478492B1 (en) * 2009-09-16 2019-03-20 Sydac Pty Ltd Visual presentation system
US8569655B2 (en) 2009-10-13 2013-10-29 Lincoln Global, Inc. Welding helmet with integral user interface
KR101370925B1 (en) 2009-10-16 2014-03-07 엘지전자 주식회사 Method for indicating a 3d contents and apparatus for processing a signal
US8569646B2 (en) 2009-11-13 2013-10-29 Lincoln Global, Inc. Systems, methods, and apparatuses for monitoring weld quality
US9468988B2 (en) 2009-11-13 2016-10-18 Lincoln Global, Inc. Systems, methods, and apparatuses for monitoring weld quality
WO2011071929A2 (en) * 2009-12-07 2011-06-16 Photon-X, Inc. 3d visualization system
US8532962B2 (en) 2009-12-23 2013-09-10 Honeywell International Inc. Approach for planning, designing and observing building systems
US8717360B2 (en) 2010-01-29 2014-05-06 Zspace, Inc. Presenting a view within a three dimensional scene
DE102010009737A1 (en) * 2010-03-01 2011-09-01 Institut für Rundfunktechnik GmbH Method and arrangement for reproducing 3D image content
US20110234591A1 (en) * 2010-03-26 2011-09-29 Microsoft Corporation Personalized Apparel and Accessories Inventory and Display
US8581905B2 (en) * 2010-04-08 2013-11-12 Disney Enterprises, Inc. Interactive three dimensional displays on handheld devices
US9733699B2 (en) * 2010-04-13 2017-08-15 Dean Stark Virtual anamorphic product display with viewer height detection
TWI407773B (en) * 2010-04-13 2013-09-01 Nat Univ Tsing Hua Method and system for providing three dimensional stereo image
US8315443B2 (en) 2010-04-22 2012-11-20 Qualcomm Incorporated Viewpoint detector based on skin color area and face area
US8995678B2 (en) 2010-04-30 2015-03-31 Honeywell International Inc. Tactile-based guidance system
US8990049B2 (en) 2010-05-03 2015-03-24 Honeywell International Inc. Building structure discovery and display from various data artifacts at scene
US8538687B2 (en) 2010-05-04 2013-09-17 Honeywell International Inc. System for guidance and navigation in a building
JP2011259373A (en) * 2010-06-11 2011-12-22 Sony Corp Stereoscopic image display device and stereoscopic image display method
JP5573426B2 (en) * 2010-06-30 2014-08-20 ソニー株式会社 Audio processing apparatus, audio processing method, and program
JP5488306B2 (en) * 2010-07-29 2014-05-14 船井電機株式会社 projector
US20120056875A1 (en) * 2010-08-11 2012-03-08 Lg Electronics Inc. Method for operating image display apparatus
KR101695819B1 (en) * 2010-08-16 2017-01-13 엘지전자 주식회사 A apparatus and a method for displaying a 3-dimensional image
US8704879B1 (en) 2010-08-31 2014-04-22 Nintendo Co., Ltd. Eye tracking enabling 3D viewing on conventional 2D display
US20120062551A1 (en) * 2010-09-13 2012-03-15 Lg Electronics Inc. Image display apparatus and method for operating image display apparatus
CN103098459B (en) * 2010-09-22 2016-11-09 富士胶片株式会社 Stereo photographic device and shadow correction method
US9001053B2 (en) 2010-10-28 2015-04-07 Honeywell International Inc. Display system for controlling a selector symbol within an image
WO2012082105A1 (en) 2010-12-13 2012-06-21 Edison Welding Institute, Inc. Welding training system
KR20120065774A (en) * 2010-12-13 2012-06-21 삼성전자주식회사 Audio providing apparatus, audio receiver and method for providing audio
ES2768033T3 (en) 2010-12-22 2020-06-19 Zspace Inc Three-dimensional tracking of a user control device in a volume
US8773946B2 (en) 2010-12-30 2014-07-08 Honeywell International Inc. Portable housings for generation of building maps
US8982192B2 (en) 2011-04-07 2015-03-17 Her Majesty The Queen In Right Of Canada As Represented By The Minister Of Industry, Through The Communications Research Centre Canada Visual information display on curvilinear display surfaces
US9766698B2 (en) 2011-05-05 2017-09-19 Nokia Technologies Oy Methods and apparatuses for defining the active channel in a stereoscopic view by using eye tracking
JP5746908B2 (en) * 2011-05-06 2015-07-08 株式会社東芝 Medical image processing device
US8786529B1 (en) 2011-05-18 2014-07-22 Zspace, Inc. Liquid crystal variable drive voltage
US9560314B2 (en) 2011-06-14 2017-01-31 Microsoft Technology Licensing, Llc Interactive and shared surfaces
JP5846777B2 (en) 2011-06-28 2016-01-20 株式会社東芝 Medical image processing device
US9342928B2 (en) 2011-06-29 2016-05-17 Honeywell International Inc. Systems and methods for presenting building information
DE102011112618A1 (en) * 2011-09-08 2013-03-14 Eads Deutschland Gmbh Interaction with a three-dimensional virtual scenario
CN103828359B (en) * 2011-09-29 2016-06-22 杜比实验室特许公司 For producing the method for the view of scene, coding system and solving code system
US9606992B2 (en) * 2011-09-30 2017-03-28 Microsoft Technology Licensing, Llc Personal audio/visual apparatus providing resource management
AU2012326218B2 (en) 2011-10-17 2017-03-09 Butterfly Network, Inc. Transmissive imaging and related apparatus and methods
US9106903B2 (en) 2011-11-18 2015-08-11 Zspace, Inc. Head tracking eyewear system
US9292184B2 (en) 2011-11-18 2016-03-22 Zspace, Inc. Indirect 3D scene positioning control
FR2983330B1 (en) * 2011-11-24 2014-06-20 Thales Sa METHOD AND DEVICE FOR REPRESENTING SYNTHETIC ENVIRONMENTS
ITTO20111150A1 (en) 2011-12-14 2013-06-15 Univ Degli Studi Genova PERFECT THREE-DIMENSIONAL STEREOSCOPIC REPRESENTATION OF VIRTUAL ITEMS FOR A MOVING OBSERVER
JP6017795B2 (en) * 2012-02-10 2016-11-02 任天堂株式会社 GAME PROGRAM, GAME DEVICE, GAME SYSTEM, AND GAME IMAGE GENERATION METHOD
JP2014006674A (en) * 2012-06-22 2014-01-16 Canon Inc Image processing device, control method of the same and program
US20160093233A1 (en) 2012-07-06 2016-03-31 Lincoln Global, Inc. System for characterizing manual welding operations on pipe and other curved structures
US9767712B2 (en) 2012-07-10 2017-09-19 Lincoln Global, Inc. Virtual reality pipe welding simulator and setup
US20140125763A1 (en) * 2012-11-07 2014-05-08 Robert Nathan Cohen 3d led output device and process for emitting 3d content output for large screen applications and which is visible without 3d glasses
US9265458B2 (en) 2012-12-04 2016-02-23 Sync-Think, Inc. Application of smooth pursuit cognitive testing paradigms to clinical drug development
KR101916663B1 (en) * 2012-12-18 2018-11-08 삼성전자주식회사 Device of displaying 3d image using at least one of gaze direction of user or gravity direction
US9380976B2 (en) 2013-03-11 2016-07-05 Sync-Think, Inc. Optical neuroinformatics
US9667889B2 (en) 2013-04-03 2017-05-30 Butterfly Network, Inc. Portable electronic devices with integrated imaging capabilities
US10019130B2 (en) 2013-04-21 2018-07-10 Zspace, Inc. Zero parallax drawing within a three dimensional display
EP2806404B1 (en) 2013-05-23 2018-10-10 AIM Sport AG Image conversion for signage
US10930174B2 (en) 2013-05-24 2021-02-23 Lincoln Global, Inc. Systems and methods providing a computerized eyewear device to aid in welding
US9426598B2 (en) 2013-07-15 2016-08-23 Dts, Inc. Spatial calibration of surround sound systems including listener position estimation
US20150072323A1 (en) 2013-09-11 2015-03-12 Lincoln Global, Inc. Learning management system for a real-time simulated virtual reality welding training environment
US10083627B2 (en) 2013-11-05 2018-09-25 Lincoln Global, Inc. Virtual reality and real welding training system and method
JP2017505933A (en) * 2013-11-26 2017-02-23 シェフィ,ヨァヴ Method and system for generating a virtual image fixed on a real object
US9836987B2 (en) 2014-02-14 2017-12-05 Lincoln Global, Inc. Virtual reality pipe welding simulator and setup
US9667951B2 (en) 2014-02-18 2017-05-30 Cisco Technology, Inc. Three-dimensional television calibration
WO2015142936A1 (en) 2014-03-17 2015-09-24 Meggitt Training Systems Inc. Method and apparatus for rendering a 3-dimensional scene
JP6687543B2 (en) 2014-06-02 2020-04-22 リンカーン グローバル,インコーポレイテッド System and method for hand welding training
US9841844B2 (en) * 2014-06-20 2017-12-12 Funai Electric Co., Ltd. Image display device
JP2016012889A (en) * 2014-06-30 2016-01-21 株式会社リコー Image projection system
WO2016018929A1 (en) 2014-07-28 2016-02-04 Beam Authentic, LLC Mountable display devices
WO2016025853A1 (en) 2014-08-15 2016-02-18 Beam Authentic, LLC Systems for displaying media on display devices
US10235807B2 (en) 2015-01-20 2019-03-19 Microsoft Technology Licensing, Llc Building holographic content using holographic tools
US20170084084A1 (en) * 2015-09-22 2017-03-23 Thrillbox, Inc Mapping of user interaction within a virtual reality environment
CN106067967B (en) * 2016-06-29 2018-02-09 汇意设计有限公司 Deformable Volumetric hologram implementation method
EP3319066A1 (en) 2016-11-04 2018-05-09 Lincoln Global, Inc. Magnetic frequency selection for electromagnetic position tracking
US10878591B2 (en) 2016-11-07 2020-12-29 Lincoln Global, Inc. Welding trainer utilizing a head up display to display simulated and real-world objects
US10913125B2 (en) 2016-11-07 2021-02-09 Lincoln Global, Inc. Welding system providing visual and audio cues to a welding helmet with a display
US10503456B2 (en) 2017-05-05 2019-12-10 Nvidia Corporation Method and apparatus for rendering perspective-correct images for a tilted multi-display environment
CN107193372B (en) * 2017-05-15 2020-06-19 杭州一隅千象科技有限公司 Projection method from multiple rectangular planes at arbitrary positions to variable projection center
JP2018197844A (en) * 2017-05-23 2018-12-13 ヘ−ヨン・チョイ Apparatus for spatially separated images
US10373536B2 (en) 2017-05-26 2019-08-06 Jeffrey Sherretts 3D signage using an inverse cube illusion fixture
US10997872B2 (en) 2017-06-01 2021-05-04 Lincoln Global, Inc. Spring-loaded tip assembly to support simulated shielded metal arc welding
CN107333121B (en) * 2017-06-27 2019-02-26 山东大学 The immersion solid rendering optical projection system and its method of moving view point on curve screens
US11170563B2 (en) * 2018-01-04 2021-11-09 8259402 Canada Inc. Immersive environment with digital environment to enhance depth sensation
US11475792B2 (en) 2018-04-19 2022-10-18 Lincoln Global, Inc. Welding simulator with dual-user configuration
US11557223B2 (en) 2018-04-19 2023-01-17 Lincoln Global, Inc. Modular and reconfigurable chassis for simulated welding training
CN110858464A (en) * 2018-08-24 2020-03-03 财团法人工业技术研究院 Multi-view display device and control simulator
CN109407329B (en) * 2018-11-06 2021-06-25 三亚中科遥感研究所 Space light field display method and device
US11113879B2 (en) 2018-11-16 2021-09-07 Geomni, Inc. Systems and methods for generating augmented reality environments from two-dimensional drawings
IL264032B (en) * 2018-12-30 2020-06-30 Elbit Systems Ltd Systems and methods for reducing image artefacts in binocular displays
US11367361B2 (en) * 2019-02-22 2022-06-21 Kyndryl, Inc. Emulating unmanned aerial vehicle (UAV)
US11508131B1 (en) * 2019-11-08 2022-11-22 Tanzle, Inc. Generating composite stereoscopic images
CN112929636A (en) * 2019-12-05 2021-06-08 北京芯海视界三维科技有限公司 3D display device and 3D image display method

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4763280A (en) * 1985-04-29 1988-08-09 Evans & Sutherland Computer Corp. Curvilinear dynamic image generation system
US5400177A (en) * 1993-11-23 1995-03-21 Petitto; Tony Technique for depth of field viewing of images with improved clarity and contrast
US5652617A (en) * 1995-06-06 1997-07-29 Barbour; Joel Side scan down hole video tool having two camera
US5945985A (en) * 1992-10-27 1999-08-31 Technology International, Inc. Information system for interactive access to geographic information
US20040066376A1 (en) * 2000-07-18 2004-04-08 Max Donath Mobility assist device

Family Cites Families (126)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US1592034A (en) * 1924-09-06 1926-07-13 Macy Art Process Corp Process and method of effective angular levitation of printed images and the resulting product
US4182053A (en) 1977-09-14 1980-01-08 Systems Technology, Inc. Display generator for simulating vehicle operation
US4291380A (en) * 1979-05-14 1981-09-22 The Singer Company Resolvability test and projection size clipping for polygon face display
US4677576A (en) * 1983-06-27 1987-06-30 Grumman Aerospace Corporation Non-edge computer image generation system
US4795248A (en) * 1984-08-31 1989-01-03 Olympus Optical Company Ltd. Liquid crystal eyeglass
GB8701288D0 (en) * 1987-01-21 1987-02-25 Waldern J D Perception of computer-generated imagery
US5079699A (en) * 1987-11-27 1992-01-07 Picker International, Inc. Quick three-dimensional display
US5361386A (en) 1987-12-04 1994-11-01 Evans & Sutherland Computer Corp. System for polygon interpolation using instantaneous values in a variable
US4853592A (en) * 1988-03-10 1989-08-01 Rockwell International Corporation Flat panel display having pixel spacing and luminance levels providing high resolution
JP2622620B2 (en) * 1989-11-07 1997-06-18 プロクシマ コーポレイション Computer input system for altering a computer generated display visible image
US5537144A (en) * 1990-06-11 1996-07-16 Revfo, Inc. Electro-optical display system for visually displaying polarized spatially multiplexed images of 3-D objects for use in stereoscopically viewing the same with high image quality and resolution
US5327285A (en) * 1990-06-11 1994-07-05 Faris Sadeg M Methods for manufacturing micropolarizers
US5502481A (en) * 1992-11-16 1996-03-26 Reveo, Inc. Desktop-based projection display system for stereoscopic viewing of displayed imagery over a wide field of view
US5276785A (en) * 1990-08-02 1994-01-04 Xerox Corporation Moving viewpoint with respect to a target in a three-dimensional workspace
US6392689B1 (en) * 1991-02-21 2002-05-21 Eugene Dolgoff System for displaying moving images pseudostereoscopically
US5168531A (en) 1991-06-27 1992-12-01 Digital Equipment Corporation Real-time recognition of pointing information from video
US5381158A (en) 1991-07-12 1995-01-10 Kabushiki Kaisha Toshiba Information retrieval apparatus
US5264964A (en) 1991-12-18 1993-11-23 Sades Faris Multi-mode stereoscopic imaging system
US5287437A (en) * 1992-06-02 1994-02-15 Sun Microsystems, Inc. Method and apparatus for head tracked display of precomputed stereo images
JPH07325934A (en) 1992-07-10 1995-12-12 Walt Disney Co:The Method and equipment for provision of graphics enhanced to virtual world
US5574835A (en) * 1993-04-06 1996-11-12 Silicon Engines, Inc. Bounding box and projections detection of hidden polygons in three-dimensional spatial databases
GB2288523B (en) * 1993-09-28 1998-04-01 Namco Ltd Clipping processing device, three-dimensional simulator device, and clipping processing method
US5438623A (en) 1993-10-04 1995-08-01 The United States Of America As Represented By The Administrator Of National Aeronautics And Space Administration Multi-channel spatialization system for audio signals
US5686975A (en) 1993-10-18 1997-11-11 Stereographics Corporation Polarel panel for stereoscopic displays
US6111598A (en) * 1993-11-12 2000-08-29 Peveo, Inc. System and method for producing and displaying spectrally-multiplexed images of three-dimensional imagery for use in flicker-free stereoscopic viewing thereof
US5381127A (en) 1993-12-22 1995-01-10 Intel Corporation Fast static cross-unit comparator
US5684460A (en) * 1994-04-22 1997-11-04 The United States Of America As Represented By The Secretary Of The Army Motion and sound monitor and stimulator
JPH08163603A (en) 1994-08-05 1996-06-21 Tomohiko Hattori Stereoscopic video display device
AUPN003894A0 (en) 1994-12-13 1995-01-12 Xenotech Research Pty Ltd Head tracking system for stereoscopic display apparatus
US5795154A (en) * 1995-07-07 1998-08-18 Woods; Gail Marjorie Anaglyphic drawing device
US6331856B1 (en) * 1995-11-22 2001-12-18 Nintendo Co., Ltd. Video game system with coprocessor providing high speed efficient 3D graphics and digital audio signal processing
US6028593A (en) * 1995-12-01 2000-02-22 Immersion Corporation Method and apparatus for providing simulated physical interactions within computer generated environments
US6252707B1 (en) * 1996-01-22 2001-06-26 3Ality, Inc. Systems for three-dimensional viewing and projection
US5574836A (en) 1996-01-22 1996-11-12 Broemmelsiek; Raymond M. Interactive display apparatus and method with viewer position compensation
US5880733A (en) * 1996-04-30 1999-03-09 Microsoft Corporation Display system and method for displaying windows of an operating system to provide a three-dimensional workspace for a computer system
JPH1063470A (en) * 1996-06-12 1998-03-06 Nintendo Co Ltd Souond generating device interlocking with image display
US6100903A (en) * 1996-08-16 2000-08-08 Goettsche; Mark T Method for generating an ellipse with texture and perspective
US6108005A (en) * 1996-08-30 2000-08-22 Space Corporation Method for producing a synthesized stereoscopic image
JP4086336B2 (en) * 1996-09-18 2008-05-14 富士通株式会社 Attribute information providing apparatus and multimedia system
US6139434A (en) * 1996-09-24 2000-10-31 Nintendo Co., Ltd. Three-dimensional image processing apparatus with enhanced automatic and user point of view control
US6317127B1 (en) 1996-10-16 2001-11-13 Hughes Electronics Corporation Multi-user real-time augmented reality system and method
US6115022A (en) * 1996-12-10 2000-09-05 Metavision Corporation Method and apparatus for adjusting multiple projected raster images
JP3034483B2 (en) 1997-04-21 2000-04-17 核燃料サイクル開発機構 Object search method and apparatus using the method
US6226008B1 (en) * 1997-09-04 2001-05-01 Kabushiki Kaisha Sega Enterprises Image processing device
DE59805342D1 (en) * 1997-10-30 2002-10-02 Baldeweg Ag Dr METHOD AND DEVICE FOR PROCESSING IMAGE OBJECTS
US5956046A (en) * 1997-12-17 1999-09-21 Sun Microsystems, Inc. Scene synchronization of multiple computer displays
GB9800397D0 (en) 1998-01-09 1998-03-04 Philips Electronics Nv Virtual environment viewpoint control
US6047201A (en) * 1998-04-02 2000-04-04 Jackson, Iii; William H. Infant blood oxygen monitor and SIDS warning device
US6529210B1 (en) * 1998-04-08 2003-03-04 Altor Systems, Inc. Indirect object manipulation in a simulation
US6466185B2 (en) * 1998-04-20 2002-10-15 Alan Sullivan Multi-planar volumetric display system and method of operation using psychological vision cues
US20020163482A1 (en) 1998-04-20 2002-11-07 Alan Sullivan Multi-planar volumetric display system including optical elements made from liquid crystal having polymer stabilized cholesteric textures
US6211848B1 (en) * 1998-05-15 2001-04-03 Massachusetts Institute Of Technology Dynamic holographic video with haptic interaction
US6064354A (en) * 1998-07-01 2000-05-16 Deluca; Michael Joseph Stereoscopic user interface method and apparatus
US6552722B1 (en) 1998-07-17 2003-04-22 Sensable Technologies, Inc. Systems and methods for sculpting virtual objects in a haptic virtual reality environment
US6351280B1 (en) * 1998-11-20 2002-02-26 Massachusetts Institute Of Technology Autostereoscopic display system
US6373482B1 (en) * 1998-12-23 2002-04-16 Microsoft Corporation Method, system, and computer program product for modified blending between clip-map tiles
US6215403B1 (en) * 1999-01-27 2001-04-10 International Business Machines Corporation Wireless monitoring system
US6614427B1 (en) * 1999-02-01 2003-09-02 Steve Aubrey Process for making stereoscopic images which are congruent with viewer space
US6452593B1 (en) * 1999-02-19 2002-09-17 International Business Machines Corporation Method and system for rendering a virtual three-dimensional graphical display
US6198524B1 (en) * 1999-04-19 2001-03-06 Evergreen Innovations Llc Polarizing system for motion visual depth effects
US6346938B1 (en) * 1999-04-27 2002-02-12 Harris Corporation Computer-resident mechanism for manipulating, navigating through and mensurating displayed image of three-dimensional geometric model
US6417867B1 (en) * 1999-05-27 2002-07-09 Sharp Laboratories Of America, Inc. Image downscaling using peripheral vision area localization
US6690337B1 (en) 1999-06-09 2004-02-10 Panoram Technologies, Inc. Multi-panel video display
US6898307B1 (en) * 1999-09-22 2005-05-24 Xerox Corporation Object identification method and system for an augmented-reality display
US6593924B1 (en) * 1999-10-04 2003-07-15 Intel Corporation Rendering a non-photorealistic image
US6476813B1 (en) * 1999-11-30 2002-11-05 Silicon Graphics, Inc. Method and apparatus for preparing a perspective view of an approximately spherical surface portion
WO2001095061A2 (en) * 1999-12-07 2001-12-13 Frauenhofer Institut Fuer Graphische Datenverarbeitung The extended virtual table: an optical extension for table-like projection systems
WO2001059749A1 (en) * 2000-02-07 2001-08-16 Sony Corporation Multiple-screen simultaneous displaying apparatus, multiple-screen simultaneous displaying method, video signal generating device, and recorded medium
EP1264281A4 (en) * 2000-02-25 2007-07-11 Univ New York State Res Found Apparatus and method for volume processing and rendering
US6669346B2 (en) * 2000-05-15 2003-12-30 Darrell J. Metcalf Large-audience, positionable imaging and display system for exhibiting panoramic imagery, and multimedia content featuring a circularity of action
US6956576B1 (en) * 2000-05-16 2005-10-18 Sun Microsystems, Inc. Graphics system using sample masks for motion blur, depth of field, and transparency
US7227526B2 (en) * 2000-07-24 2007-06-05 Gesturetek, Inc. Video-based image control system
US6643124B1 (en) * 2000-08-09 2003-11-04 Peter J. Wilk Multiple display portable computing devices
US6680735B1 (en) * 2000-10-04 2004-01-20 Terarecon, Inc. Method for correcting gradients of irregular spaced graphic data
US6553256B1 (en) * 2000-10-13 2003-04-22 Koninklijke Philips Electronics N.V. Method and apparatus for monitoring and treating sudden infant death syndrome
GB2370738B (en) * 2000-10-27 2005-02-16 Canon Kk Image processing apparatus
US6593934B1 (en) * 2000-11-16 2003-07-15 Industrial Technology Research Institute Automatic gamma correction system for displays
US20020180727A1 (en) * 2000-11-22 2002-12-05 Guckenberger Ronald James Shadow buffer control module method and software construct for adjusting per pixel raster images attributes to screen space and projector features for digital warp, intensity transforms, color matching, soft-edge blending, and filtering for multiple projectors and laser projectors
JP3705739B2 (en) 2000-12-11 2005-10-12 株式会社ナムコ Information storage medium and game device
US6774869B2 (en) 2000-12-22 2004-08-10 Board Of Trustees Operating Michigan State University Teleportal face-to-face system
US6987512B2 (en) 2001-03-29 2006-01-17 Microsoft Corporation 3D navigation techniques
GB2375699B (en) * 2001-05-16 2003-08-13 Nibble Ltd Information management system and method
JP4722379B2 (en) 2001-05-22 2011-07-13 シエフイ,ヨアブ Method and system for displaying visible content in a virtual 3D space
US7259747B2 (en) 2001-06-05 2007-08-21 Reactrix Systems, Inc. Interactive video display system
TW579019U (en) 2001-06-13 2004-03-01 Eturbotouch Technology Inc Flexible current type touch film
JP2003085586A (en) 2001-06-27 2003-03-20 Namco Ltd Image display, image displaying method, information storage medium, and image displaying program
US6478432B1 (en) 2001-07-13 2002-11-12 Chad D. Dyner Dynamically generated interactive real imaging device
US20040135744A1 (en) * 2001-08-10 2004-07-15 Oliver Bimber Virtual showcases
US6715620B2 (en) * 2001-10-05 2004-04-06 Martin Taschek Display frame for album covers
JP3576521B2 (en) 2001-11-02 2004-10-13 独立行政法人 科学技術振興機構 Stereoscopic display method and apparatus
US6700573B2 (en) * 2001-11-07 2004-03-02 Novalogic, Inc. Method for rendering realistic terrain simulation
US7466307B2 (en) * 2002-04-11 2008-12-16 Synaptics Incorporated Closed-loop sensor on a solid-state object position detector
CA2386702A1 (en) 2002-05-17 2003-11-17 Idelix Software Inc. Computing the inverse of a pdt distortion
US20040196359A1 (en) * 2002-05-28 2004-10-07 Blackham Geoffrey Howard Video conferencing terminal apparatus with part-transmissive curved mirror
US7190331B2 (en) * 2002-06-06 2007-03-13 Siemens Corporate Research, Inc. System and method for measuring the registration accuracy of an augmented reality system
US6943805B2 (en) * 2002-06-28 2005-09-13 Microsoft Corporation Systems and methods for providing image rendering using variable rate source sampling
JP4115188B2 (en) 2002-07-19 2008-07-09 キヤノン株式会社 Virtual space drawing display device
WO2004021151A2 (en) * 2002-08-30 2004-03-11 Orasee Corp. Multi-dimensional image system for digital image input and output
JP4467267B2 (en) * 2002-09-06 2010-05-26 株式会社ソニー・コンピュータエンタテインメント Image processing method, image processing apparatus, and image processing system
US6943754B2 (en) * 2002-09-27 2005-09-13 The Boeing Company Gaze tracking system, eye-tracking assembly and an associated method of calibration
US7321682B2 (en) * 2002-11-12 2008-01-22 Namco Bandai Games, Inc. Image generation system, image generation method, program, and information storage medium
US20040130525A1 (en) * 2002-11-19 2004-07-08 Suchocki Edward J. Dynamic touch screen amusement game controller
JP2006513503A (en) 2002-11-29 2006-04-20 ブラッコ イメージング ソチエタ ペル アチオニ Apparatus and method for managing a plurality of locations in a three-dimensional display
JP4100195B2 (en) * 2003-02-26 2008-06-11 ソニー株式会社 Three-dimensional object display processing apparatus, display processing method, and computer program
KR100526741B1 (en) * 2003-03-26 2005-11-08 김시학 Tension Based Interface System for Force Feedback and/or Position Tracking and Surgically Operating System for Minimally Incising the affected Part Using the Same
US7495638B2 (en) 2003-05-13 2009-02-24 Research Triangle Institute Visual display with increased field of view
US7324121B2 (en) * 2003-07-21 2008-01-29 Autodesk, Inc. Adaptive manipulators
US20050093859A1 (en) * 2003-11-04 2005-05-05 Siemens Medical Solutions Usa, Inc. Viewing direction dependent acquisition or processing for 3D ultrasound imaging
US7667703B2 (en) * 2003-12-19 2010-02-23 Palo Alto Research Center Incorporated Systems and method for turning pages in a three-dimensional electronic document
US7312806B2 (en) * 2004-01-28 2007-12-25 Idelix Software Inc. Dynamic width adjustment for detail-in-context lenses
JP4522129B2 (en) * 2004-03-31 2010-08-11 キヤノン株式会社 Image processing method and image processing apparatus
US20050219693A1 (en) * 2004-04-02 2005-10-06 David Hartkop Scanning aperture three dimensional display device
US20050219240A1 (en) * 2004-04-05 2005-10-06 Vesely Michael A Horizontal perspective hands-on simulator
KR20070044394A (en) 2004-04-05 2007-04-27 마이클 에이 베슬리 Horizontal perspective hands-on simulator
US20050248566A1 (en) 2004-04-05 2005-11-10 Vesely Michael A Horizontal perspective hands-on simulator
US7796134B2 (en) 2004-06-01 2010-09-14 Infinite Z, Inc. Multi-plane horizontal perspective display
US20060126925A1 (en) * 2004-11-30 2006-06-15 Vesely Michael A Horizontal perspective representation
WO2006081198A2 (en) 2005-01-25 2006-08-03 The Board Of Trustees Of The University Of Illinois Compact haptic and augmented virtual reality system
US7843470B2 (en) * 2005-01-31 2010-11-30 Canon Kabushiki Kaisha System, image processing apparatus, and information processing method
US20060250390A1 (en) 2005-04-04 2006-11-09 Vesely Michael A Horizontal perspective display
US20060221071A1 (en) 2005-04-04 2006-10-05 Vesely Michael A Horizontal perspective display
JP4738870B2 (en) 2005-04-08 2011-08-03 キヤノン株式会社 Information processing method, information processing apparatus, and remote mixed reality sharing apparatus
WO2006121956A1 (en) 2005-05-09 2006-11-16 Infinite Z, Inc. Biofeedback eyewear system
WO2006121957A2 (en) 2005-05-09 2006-11-16 Michael Vesely Three dimensional horizontal perspective workstation
US20070040905A1 (en) 2005-08-18 2007-02-22 Vesely Michael A Stereoscopic display using polarized eyewear
US20070043466A1 (en) * 2005-08-18 2007-02-22 Vesely Michael A Stereoscopic display using polarized eyewear

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4763280A (en) * 1985-04-29 1988-08-09 Evans & Sutherland Computer Corp. Curvilinear dynamic image generation system
US5945985A (en) * 1992-10-27 1999-08-31 Technology International, Inc. Information system for interactive access to geographic information
US5400177A (en) * 1993-11-23 1995-03-21 Petitto; Tony Technique for depth of field viewing of images with improved clarity and contrast
US5652617A (en) * 1995-06-06 1997-07-29 Barbour; Joel Side scan down hole video tool having two camera
US20040066376A1 (en) * 2000-07-18 2004-04-08 Max Donath Mobility assist device

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2011044936A1 (en) * 2009-10-14 2011-04-21 Nokia Corporation Autostereoscopic rendering and display apparatus
US8970478B2 (en) 2009-10-14 2015-03-03 Nokia Corporation Autostereoscopic rendering and display apparatus

Also Published As

Publication number Publication date
US7796134B2 (en) 2010-09-14
US20050264858A1 (en) 2005-12-01
KR20070052260A (en) 2007-05-21
EP1781893A1 (en) 2007-05-09
US20050264558A1 (en) 2005-12-01
US20050264857A1 (en) 2005-12-01
JP2008506140A (en) 2008-02-28
WO2005118998A8 (en) 2006-04-27
US20050275913A1 (en) 2005-12-15
US20050275915A1 (en) 2005-12-15
US20050275914A1 (en) 2005-12-15
US20050264559A1 (en) 2005-12-01
EP1759379A2 (en) 2007-03-07
WO2005119376A2 (en) 2005-12-15
WO2005119376A3 (en) 2006-04-27
US20050281411A1 (en) 2005-12-22
KR20070052261A (en) 2007-05-21
JP2008507006A (en) 2008-03-06

Similar Documents

Publication Publication Date Title
US20050275914A1 (en) Binaural horizontal perspective hands-on simulator
US9684994B2 (en) Modifying perspective of stereoscopic images based on changes in user viewpoint
US7907167B2 (en) Three dimensional horizontal perspective workstation
US20050219240A1 (en) Horizontal perspective hands-on simulator
WO2005098516A2 (en) Horizontal perspective hand-on simulator
US20060126927A1 (en) Horizontal perspective representation
CN101006492A (en) Hrizontal perspective display
US20070291035A1 (en) Horizontal Perspective Representation
US20050248566A1 (en) Horizontal perspective hands-on simulator
JP2023168544A (en) Low-frequency interchannel coherence control
JP2010199739A (en) Stereoscopic display controller, stereoscopic display system, and stereoscopic display control method
TW202329684A (en) Autostereoscopic display device presenting 3d-view and 3d-sound

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KM KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NG NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SM SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
WR Later publication of a revised version of an international search report
WWE Wipo information: entry into national phase

Ref document number: 2007515486

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Ref document number: DE

WWE Wipo information: entry into national phase

Ref document number: 8032/DELNP/2006

Country of ref document: IN

WWE Wipo information: entry into national phase

Ref document number: 2005757162

Country of ref document: EP

Ref document number: 1020077000038

Country of ref document: KR

WWE Wipo information: entry into national phase

Ref document number: 200580025729.3

Country of ref document: CN

WWP Wipo information: published in national office

Ref document number: 2005757162

Country of ref document: EP