WO2004019752A2

WO2004019752A2 - Method and device for the generation of intersensory perception-associations

Info

Publication number: WO2004019752A2
Application number: PCT/EP2003/009524
Authority: WO
Inventors: Rolf Eckmiller; Thomas Schieder; Carsten Wilks
Original assignee: Rheinische Friedrich-Wilhelms-Uni Versität Bonn
Priority date: 2002-08-30
Filing date: 2003-08-28
Publication date: 2004-03-11
Also published as: EP1398012A8; AU2003273395A1; WO2004019752A3; AU2003273395A8; EP1398012A1

Abstract

The invention, comprising device and method, relates to a learning, user-specific signal transducer and a storage of sensory stimulus pairs for the conversion of sensory stimuli (SIV) for a sense organ, e.g. S1, into sensory stimuli (SOV) which are each subjectively sensed or associated as sufficiently similar, for another sense organ, e.g. S2, to produce psychophysical percepts or associations by a vector translator module (TM) and a coupled learning module (LM), where initially the user person subjectively compares in a learning process a percept produced by the SIV, with P2 produced by the SOV and signalizes to the LM the psychophysical comparison result for an iterative optimization of the SOV generated in the TM, and where subsequently the TM charged with many pairs of sensory stimulus data, which are evaluated as similar, can be operated in co-operation with the user or autonomously.

Description

Method and Device for the Generation of Intersensory Perception- Associations

The invention relates to a device and a method for the generation of intersensory perception-associations. In addition the invention relates to a method for the operation of an Intersensory Perception Associator (IPA).

State of the art

1. Systems for transformation of sensory stimuli for a sense organ onto sensory stimuli for another sense organ are known from

US 2002/0067271 (22AUG00) US 5,097,326 (27JUL89)

US 5,636,038 (24JUN96)

2. Systems whose functions are modified on the basis of perception are known from US 6,093,953 (04AUG98)

US 6,400,989 (21FEB97) US 6,425,764 (12DEZ97) US 6,408,293 (09JUN99)

3. Learning systems are known from

Computational Intelligence - Imitating Life,

Eds. Zurada - Marks - Robinson - IEEE Press, New York, 1994

An Introduction to Neural and Electronic Networks Eds. Zornetzer - Davis - Lau

Academic Press, New York, 1990 4. Systems for unobtrusive signal transmission unnoticeable for nearby persons are known: vibration signals of a mobile phone or a pager mini loudspeakers integrated in the auditory organ - projected display fed onto glasses (Sony, Olympus)

Disadvantages of background art

1. The spectrum of unobtrusively transmittable messages is very limited. 2. For most of the sense organs it is impossible to find sensory stimuli which psychophysically correspond to other sense organs, e.g. smell as surrogate for sound, tactile event as surrogate for image. 3. Sensory stimulus patterns and their psychophysically corresponding stimulus patterns cannot be stored, related, or retrieved. 4. It is impossible, so far, to adequately fathom the psychophysical perception space which is required for the development of various devices with man-machine interface and of medical technology products.

5. Tactile signal transmitters like pagers or vibration signalers of mobile phones are limited to pure indication signals, which signals are unable to transmit more complex messages.

Invention

The said disadvantages are intended to be removed or at least substantially reduced by means of the herein disclosed invention, according to claim 1, 4 and 10. The sub-claims are related to preferred embodiments of the invention.

Definitions

Percept: A percept is that which is perceived. It is neither the physical object

(distal stimulus) nor its image in a receptor (proximal stimulus). Rather it concerns the experienced (phenomenal) result of the entire perception process.

Spatio-temporal: Spatio-temporal functions are understood as functions depending on time and space.

Physical pattern space: A physical pattern space is in the following understood as the space of all physical sensory stimulus patterns.

Physical sensory stimulus pattern: A physical sensory stimulus pattern is in the following understood as a stimulus pattern adequate to at least one sense organ, which produces a sensory perception or a percept of either an object or a term after being converted (in the assigned receptors).

SIV (sensory input vector): SIV is in the following understood as a physical sensory stimulus pattern.

SOV (sensory output vector): SOV is understood as a physical sensory stimulus pattern generated by the TM and output via the signalers ST.

SR (signal receiver): SR is in the following understood as a sense organ- specific sensor, which converts, e.g. electronically, a part of a physical sensory stimulus pattern, so that the predemonant part of the information of the physical sensory stimulus pattern remains for later reproduction of the sensory stimulus pattern for the specific sense organ, e.g. a microphone, a video camera, an electric nose, or the like. A SR can also comprise processing elements like filters, amplifiers, analog-digital transducers, or the like.

ST (signal transmitter) : A ST is understood as a sense organ-specific transducer, which converts, for example, electrical signals into physical sensory stimulus patterns, where it is possible to fill a part as large as possible of the space receivable by the assigned sense organ. A ST can also comprise preprocessing elements, e.g. digital-analog transducers, amplifiers, filters, or the like. Examples for STs are loudspeakers, displays, tactile actuators, or the like.

Psychophysical perception space: Psychophysical perception space is understood as perception space.

Aims of the invention

1. Person-specific transduction of sensory stimulus patterns with the receiving sense organ being changed.

2. Person-specific detection of sensory stimulus-pattern pairs, which are predominantly or exclusively offered over different sense organs but which produce sufficiently similar psychophysical perceptions or associations.

3. Person-specific learning of perception associations for biomedical applications.

4. Person-specific learning of perception associations in industry and technology.

5. Unobtrusive communication unnoticeable for nearby persons by means of body-near signal transmitters in industry and technology.

6. Person-specific reproduction of multimodal sensory stimulus patterns.

7. Person-specific and automatic exploration of perception space and association space. Advantages of the invention

In contrast to conventional systems for transmission of sensory channel- specific information onto another sense organ the herein disclosed system (IPA) has the advantage that the output signals (SOV) are optimally adapted to the user person's perception due to the adjustment by means of a perception-based learning process.

After the learning process, the user person is instantaneously able to make correct associations and interpretations of the output signals (SOV) because they have been composed according to his association. The advantage is that the user is able to acquire very fast the information being transmitted by the signals, and to "understand" a great many number of SOVs via another sense organ.

Furthermore, the receptivity of the sense organs is optimally used and new sensory stimuli are generated, which stimuli have been unimaginable until now and which therefore have not been used with conventional methods.

Still another advantage in contrast to known methods is that not only relative distances of perceptions of one sensory channel can be translated into relative distances of perceptions of another sensory channel, but that a kind of mapping function between percepts of different sensory channels can be found. Thus, it is possible to intersensorily transmit information via a mapping rule found by means of a learning system.

Preferred embodiments

An preferred embodiment of the IPA is that in a person-specific, perception- based learning process the SOV is found for sensory channel S2, whose percept P2 the user person associates with percept PI being received with another sense organ so far. An preferred embodiment of the IPA is that in a person-specific, perception- based learning process the SOV is found for sensory channel S2, whose percept P2 the user person also associates with P' stored in his memories.

An preferred embodiment of the SIV is that the SIV can be chosen either by the learning module (LM) or by the user of the IPA himself.

Preferred embodiments of a sense organ-specific sensory receptor SR is that the SR comprises a sensor which can record, for example, auditory (microphone), visual (video camera), olfactive, or haptic events, and that signal processing, e.g. preamplification or digitalization, is provided.

Preferred embodiments of the storage module (SM) in the TM are that the SM comprises storage common in electronic data processing, e.g. digital chip or magnetic data storage.

Preferred embodiments of a sense organ-specific sensory stimulus signaler (ST) is that the ST comprises, for example, a loudspeaker, a LCD display, an array of molecular dispensers to compose olfactory patterns, or a tactile actuator array.

An additional preferred embodiment is that the mapping module (MM) can be realized in the TM, for example, as a lookup table connecting the parameter sets of the FM found in the learning process, whose SOV produces a perception P2 sufficiently similar to PI with the signals of the SIV acquired by the SRs.

In another preferred embodiment the MM can comprise an additional preprocessing structure which, for example, classifies or filters the input signals coming from the SR. In another preferred embodiment the MM can also represent the identity- function for parts of the input space or the entire space, so that the signals provided by the SR are instantaneously passed on to the FM.

An preferred embodiment of the said function generator module (FM) is that the FM includes a combination of different signal generator components, e.g. recurrent neural networks, programable digital function generators, or backcoupled filters, which can be adjusted by a parameter set to generate various time functions as function generator vectors (FGV).

Another preferred embodiment is that the function generator vectors (FGV) generated by the FM are lead to the signalers ST in the form of time functions where they are converted into sense organ-specific signal output vectors (SOV).

An preferred embodiment is that FM has access to the storage module (SM) for storing data (parameter sets or FGVs).

Another preferred embodiment is that the FM receives parameter sets from the MM, or that the FM receives the address of a parameter set in the SM from the MM, and that the FM transmits information about the actual parameter set to the mapping module MM, if necessary.

An preferred embodiment of the learning process (LP) is that the LP is based upon evolutionary genetic algorithms or reinforcement learning.

Still another preferred embodiment of the LP can be that the SOV is iteratively optimized with the help of the user person's feedback, with the aim being that the association of percept P2 generated by the SOV to percept PI is sufficiently similar. In another preferred embodiment the learning module (LM) can also instantaneously influence the parameter set via the TMC (Translator Module Controller), in still another embodiment the learning module (LM) can only propose directions for the change of parameter sets to the MM or the FM via the TMC.

An preferred embodiment of the choice of a sense organ for S2 is that the user person explicitly specifies the sense organ S2 before the learning phase.

Another preferred embodiment can be that the user person specifies the sense organ S2 during the learning phase.

Another advantages embodiment can be that the LM specifies the sense organ S2 during the learning phase.

An preferred embodiment of the motor output vector (MOV) is that the assigned comparison result is inputted by the user person's motoric feedback, e.g. by operating a control desk by means of a gesture or giving a verbal feedback.

Still another preferred embodiment of the said MOV is that the comparison result is extracted from a physiologically derived quantity, e.g. from the specific skin resistance of a part of the user person's body.

An preferred embodiment of the psychophysical comparison of PI and P2 is that the user person has to select the n-best comparison patterns out of a given set.

Another preferred embodiment of the said comparison can be that the user has to make a subjective evaluation of the offered comparison patterns on a given scale (e.g. bad, middle, good, very good). An preferred embodiment of the learning module interface (LMI) is that the MOV is recorded by means of a keyboard, a microphone, a video camera, or a sensor which can acquire a physiological quantity, pre-processed and passed on as a vector to the LM.

An preferred embodiment of the storage of the data required for replication of SIV is that the data are stored in analog or digital form in the SM at the output of SRi.

Another preferred embodiment of the storage of the data required for replication of the SIV is that the classification characteristics are stored.

An preferred embodiment of the storage of the data required for replication of the SOV is that both the parameters of the FM as well as the characterizing attributes of the FM required for the generation of SOV are stored in the SM.

Another preferred embodiment can be that the FGVs (Function Generator Vectors) transmitted to the STi for generating SOV are stored in the SM.

An preferred embodiment when assigning SIV to SOV is that for the said assignment a lookup table is used, with a digital quantity derived from SIV being interpreted as an address and the parameters required for the generation of SOV being filed at this address.

An preferred embodiment of the storage control and the recall of SIV and SOV is that the MM, FM, and LM, the latter via the TMC, signalize to the SM by means of control signals whether it should be read or written out of the storage and they thereupon receive or provide the assigned data via signal lines.

An preferred embodiment of the conversion from SIV to SOV in standard mode is that with the use of a lookup table in the MM an address is derived from the SIV and the assigned entry in the lookup table is transmitted to the FM, so that a SOV is generated.

Another preferred embodiment of the said conversion from SIV to SOV in standard mode is that the SOV of the known SIV is even generated, if during presentation of a SIV only a similarity to a known SIV is given.

Description of Fig. 1

IPA comprises, firstly, a vector translator module (TM) to convert sensory stimuli (SIV) for a sense organ SI into psychophysically corresponding sensory stimuli (SOV) for another sense organ S2 and, secondly, a learning module (LM) with a learning process (LP).

The TM and the LM work in connection with a user person, whose brain functions (CNS = central nervous system) are herein indicated by two ellipses. Thereby the sensory stimuli sensors of the sensory inputs SI and S2 are located at the outer edge.

Brain function output signals or physiological body function signals can be extracted/measured out of the upper half of the ellipse: herein indicated as motor output vector (MOV). The perception space (psychophysical pattern space), which is for principle reasons unaccessible, is indicated inside the inner ellipse.

The user person receives the SIV and the SOV via two different sense organs SI and S2 und has got for each an assigned perception or association, which are herein called percept PI (e.g. olfactive sensory impression, odor of a perfume) and percept P2 (tactile sensory impression, e.g. a tune) (v. fig. 2). Purpose of device and method (IPA) include: 1. Iterative learning-based search for sensory stimulus pairs for two sense organs, whose assigned percepts PI and P2 are perceived or associated as sufficiently similar.

2. Translation of sensory stimuli from one sense organ to the other.

Description of Fig. 2

Fig. 2 illustrates an example application for the IPA. As sensory input vector (SIV) an olfactory pattern is shown, which is acquired via the user person's olfactory organ and which is at the same time converted into a digital signal by means of an olfactory sensor array (SRI) and stored in the TM. The TM generates a sensory stimulus output vector (SOV) by a tactile actuator array (ST2). The SOV now elicits a tactile perception and association in the user person. The said psychophysical perception or association has been chosen by the user person during a learning process (LP) with the LM as sufficiently similar to the association produced by the olfactory pattern.

Claims

1. Intersensory Perception Associator (IPA), characterized by the fact that a) a person-specific conversion of a sensory stimulus input vector (SIV) for at least predominantly of one sense organ (SI) in a learning process (LP) into a perceptionbased corresponding sensory stimulus output vector (SOV) for at least predominantly of another sense organ (S2) is performed, that b) a learning vector translation module (TM) is provided, which module comprises at least one sense organ-specific sensory receptor (SR) at the input of the TM, a storage module (SM), a mapping module (MM), and a function generator (FM), and at least one sense organ-specific sensory signal transmitter (ST) at the output of the TM, where an interface (TMC) for TM-function-control is assigned to TM, and where FM generates sens organ-specific signals (FGV) for the ST, that c) a learning module (LM) is provided, where firstly a signal line is assigned from the LM output to the TMC, and where secondly an input interface (LMI) is assigned for user-specific data input, that d) the TM defines the mapping of a plurality of sensory stimulus input vectors (SIV) onto individually corresponding sensory stimulus output vectors (SOV), where firstly the mapping (M) of SIV onto SOV is not determined, and where secondly M is not described as mathematically unique, and where thirdly the SOV predominantly or exclusively stimulates S2 being different from SI which is predominantly or exclusively stimulated by SIV, that e) a learning process (LP), which has to be defined user-specifically, is provided to assign SIV to SOV, where firstly a user person compares the perception or association produced by the SIV as psychophysical percept (PI) with a perception or association produced by the SOV as psychophysical percept (P2), where secondly the user person transmits to the LM his subjective psychophysical evaluation of (PI) and (P2) via an output vector (MOV) produced by the said user person.

2. Intersensory Perception Associator according to claim 1 further characterized by the fact that firstly there are separate inputs of the mapping module (MM) assigned to the outputs of the sensory receptors (SRi), which receptors are assigned to the TM, that secondly a signal line and line are provided between the MM and the function generator module (FM), and that thirdly a signal line and a control line are provided between the MM and the learning module (LM) via the interface TMC.

3. Intersensory Perception Associator according to one of the previous claims further characterized by the fact that a function generator module (FM) for generating function generator vectors (FGV) is provided in the TM, where firstly for the FM an interface to signal coupling and control coupling with the MM is planned, where secondly a signal line to the TMC is assigned to the FM, and where thirdly signal line to the signal transmitters STj in the TM are assigned to the FM.

4. Method for operating an Intersensory Perception Associator according to one of the previous claims characterized by the fact that in the operational state: F-Learning firstly a learning process (LP) is implemented in the LM, where the LP particularly is an algorithm for "unmonitored learning", that secondly the function of the FM is influenced by the LP, that thirdly an iterative change of an FVG generated in the FM is produced by the LP, where the FGV is converted into an SOV via a chosen ST2 and presented to the user person during the learning process.

5. Method for operating an Intersensory Perception Associator according to one of the previous claims further characterized by the fact that the choice of the sense organ S2 for which P2 should be optimized is made by a user person.

6. Method for operating an Intersensory Perception Associator according to one of the previous claims further characterized by the fact that in the operational state: Flearning firstly the user person perceives a SIV as percept (PI) of SI, that secondly the user person perceives a SOV as percept (P2) of S2, and that thirdly the result of the psychophysical comparison of (PI) and (P2) made by the user person is converted into an output vector (MOV) assigned to the comparison result by means of user-specific brain functions, and that fourthly the learning process (LP) is controlled by the MOV as an input value for LM.

7. Method for operating an Intersensory Perception Associator according to one of the previous claims further characterized by the fact that in the operational state: Flearning firstly data being necessary for later replication of the SOV are stored in the storage module (SM), and that secondly data being necessary for later replication of the SIV are stored in the SM.

8. Method for operating an Intersensory Perception Associator according to one of the previous claims further characterized by the fact that in the operational state: Flearning the modules MM and LM control the mapping of each data pair: SIV to SOV or the SRI output of SIV with the FGV as ST2 -input for SOV including storage and recall.

9. Method for operating an Intersensory Perception Associator according to the previous claims further characterized by the fact that in the operational state: Foperation the TM firstly converts a SIV at sense organ SI into an SOV at sense organ S2 which is assigend to a different modality of sensation than SI, and that TM secondly, without a joint use of LM being necessary, during F-Operation elicits a perception or association P2 which is sufficiently similar to PI at the same user person, who has optimally adjusted the TM for his perception during F-Learning.

0. Method for the generation of intersensory perception-associations, by which method a) a person-specific conversion of a sensory stimulus input vector (SIV) for at least predominantly of one sense organ (SI) in a learning process (LP) into a perception-based corresponding sensory stimulus output vector (SOV) for at least predominantly of another sense organ (S2) is performed, that b) a learning vector translation module (TM) is provided, which module comprises at least one sense organ- specific sensory receptor (SR) at the input of the TM, a storage module (SM), a mapping module (MM), and a function generator (FM), and at least one sense organ-specific sensory signal transmitter (ST) at the output of the TM, where an interface (TMC) for TM-function-control is assigned to TM, and where FM generates sens organ-specific signals (FGV) for the ST, that c) a learning module (LM) is provided, where firstly a signal line is assigned from the LM output to the TMC, and where secondly an input interface (LMI) is assigned for user-specific data input, that d) the TM defines the mapping of a plurality of sensory stimulus input vectors (SIV) onto individually corresponding sensory stimulus output vectors (SOV), where firstly the mapping (M) of SIV onto SOV is not determined, and where secondly M is not described as mathematically unique, and where thirdly the SOV predominantly or exclusively stimulates S2 being different from SI which is predominantly or exclusively stimulated by SIV, that e) a learning process (LP), which has to be defined user-specifically, is provided to assign SIV to SOV, where firstly a user person compares the perception or association produced by the SIV as psychophysical percept (PI) with a perception or association produced by the SOV as psychophysical percept (P2), where secondly the user person transmits to the LM his subjective psychophysical evaluation of (PI) and (P2) via an output vector (MOV) produced by the said user person.