WO2012116232A1 - Systems and methods for decoding neural signals - Google Patents

Systems and methods for decoding neural signals Download PDF

Info

Publication number
WO2012116232A1
WO2012116232A1 PCT/US2012/026400 US2012026400W WO2012116232A1 WO 2012116232 A1 WO2012116232 A1 WO 2012116232A1 US 2012026400 W US2012026400 W US 2012026400W WO 2012116232 A1 WO2012116232 A1 WO 2012116232A1
Authority
WO
WIPO (PCT)
Prior art keywords
neural
electrodes
neural signal
frequency
brain
Prior art date
Application number
PCT/US2012/026400
Other languages
French (fr)
Inventor
Bradley E. GREGER
Paul A. HOUSE
Spencer Kellis
Original Assignee
University Of Utah Research Foundation
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by University Of Utah Research Foundation filed Critical University Of Utah Research Foundation
Publication of WO2012116232A1 publication Critical patent/WO2012116232A1/en

Links

Classifications

    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/72Signal processing specially adapted for physiological signals or for diagnostic purposes
    • A61B5/7235Details of waveform analysis
    • A61B5/7264Classification of physiological signals or data, e.g. using neural networks, statistical classifiers, expert systems or fuzzy systems
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B5/00Measuring for diagnostic purposes; Identification of persons
    • A61B5/24Detecting, measuring or recording bioelectric or biomagnetic signals of the body or parts thereof
    • AHUMAN NECESSITIES
    • A61MEDICAL OR VETERINARY SCIENCE; HYGIENE
    • A61BDIAGNOSIS; SURGERY; IDENTIFICATION
    • A61B2562/00Details of sensors; Constructional details of sensor housings or probes; Accessories for sensors
    • A61B2562/04Arrangements of multiple sensors of the same type
    • A61B2562/046Arrangements of multiple sensors of the same type in a matrix array
    • GPHYSICS
    • G16INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR SPECIFIC APPLICATION FIELDS
    • G16HHEALTHCARE INFORMATICS, i.e. INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR THE HANDLING OR PROCESSING OF MEDICAL OR HEALTHCARE DATA
    • G16H50/00ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics
    • G16H50/20ICT specially adapted for medical diagnosis, medical simulation or medical data mining; ICT specially adapted for detecting, monitoring or modelling epidemics or pandemics for computer-aided diagnosis, e.g. based on medical expert systems

Definitions

  • the present disclosure relates to the general fields of bioengineering and computer technology, and more particularly to methods and systems for decoding neural signals, and in particular, discrete and/or continuous decoding of neural signals, e.g., local field potentials, recorded from a cortical surface.
  • Some embodiments of the present technology provide systems and methods for decoding microscale neural signals to extract information on attempted and/or actual movement and/or action of a subject. Implementations of these embodiments can be used to provide discrete and/or continuous decoding of neural signals. Discrete and/or continuous decoding can be used to decode micro-scale neural signals for intent. In some embodiments, the neural signals can be e.g., local field potentials, recorded relative to a given spatial location of the brain.
  • Discrete decoding can comprise, for example, the classification of neural signals into discrete categories.
  • discrete decoding can be used to classify or identify attempted and/or actual words and/or movements from a subject.
  • Continuous decoding can comprise, for example, the decoding of neural signals to provide information on continuously varying attempted and/or actual movements.
  • continuous decoding can comprise proportional decoding of attempted and/or actual movement of a part of a subject body.
  • Such decoding can provide data on position, velocity, and/or acceleration of movement of the part of the subject's body.
  • some embodiments provide methods and systems that can detect signals from portions of the central nervous system and/or peripheral nervous system.
  • Some embodiments can provide a brain-machine .interface that can be used in various applications.
  • a system for decoding neural signals can comprise a receiver configured to receive a neural signal from each of a plurality of electrodes and a processor configured to convert the neural signals and to apply a classifier to the converted signals.
  • the system can be configured to determine the movement the patient is making. In addition or alternatively, the system can be configured to determine the movement the patient is attempting to make. Further, in addition or alternatively, the system can be configured to determine the part of the body that is moving or attempting to be moved.
  • kits for decoding neural signals can comprise a plurality of electrodes, a receiver, and a processor.
  • the plurality of electrodes can be configured to detect neural signals corresponding to different spatial positions on or within the brain of a patient. Each neural signal can be emanated from the brain when the patient moves and/or attempts to move a part of the body.
  • the receiver can be configured to receive a neural signal from each of a plurality of electrodes. Further, each neural signal can correspond to a different spatial position of the brain of a patient. In addition, each neural signal can emanate from the brain when the patient moves and/or attempts to move a part of the body.
  • the processor can be configured convert the neural signals into frequency and time domain information.
  • the processor can also apply a classifier to the frequency and time domain information so as to determine the movement the patient is making and/or is attempting to make.
  • the electrodes contact a surface of the brain.
  • the electrodes can contact a cortical surface of the brain.
  • the electrodes can also be spaced from a surface of the brain.
  • embodiments can provide a desired relative spatial positioning of the electrodes, whether or not one or more of the electrodes is in direct contact with a surface of the brain.
  • the electrodes contact a face motor cortex. Further, the electrodes can contact other cortical areas. For example, electrodes can contact arm and/or hand areas of the primary motor cortex, Wernicke' s area, Broca' s area, the premotor cortex, and other areas of the cortex.
  • each neural signal comprises a local field potential from the cortical surface.
  • the electrodes comprise micro-electrodes.
  • the frequency and time domain information for each neural signal comprises spectral power of the neural signal at different times.
  • the frequency and time domain information for each neural signal also comprises voltage of the neural signal at different times.
  • the spectral power for each neural signal comprises power at different frequency components.
  • the movement of the part of the body comprises movement of an individual finger of a hand.
  • a kit can be provided that is operative to implement embodiments of the methods disclosed herein.
  • the kit can comprise electrodes for implantation in contact with neural tissue.
  • the electrodes can be configured as a grid of micro-electrodes.
  • surgical tools can be provided to implant the electrode grid.
  • the kit can also comprise a data acquisition system capable of recording neural signals from the electrodes.
  • the data acquisition system can record data from a grid of micro-electrodes, as specified in terms of number of electrodes, sampling rate, bandwidth, and other desired parameters.
  • the kit may also comprise a computation unit (such as a computer processor) capable of executing an appropriate algorithm.
  • the computation unit can execute the decoding algorithm in near real-time. For example, the computation unit can execute the algorithm sufficiently quickly so the patient does not notice any lag.
  • an algorithm can comprise enable the computation unit to perform classification or continuous decoding of neural signals.
  • the kit can optionally comprise an effector device.
  • the effector device can be a computer interface, a virtual reality environment, a prosthetic device, a speech synthesizer, or other machine useful for executing the output of the decoding algorithm.
  • a method for decoding neural signals comprises receiving a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves. A part of the body.
  • the method also comprises converting the neural signals into frequency and time domain information, and applying a classifier to the frequency and time domain information so as to determine the patient's movement of, and/or attempt to move, a part of the body.
  • a non- transitory machine-readable medium is provided.
  • the machine-readable medium comprises instructions stored therein, which when executed by a machine, cause the machine to perform operations for decoding neural signals.
  • the operations comprise receiving a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body.
  • the operations can also comprise converting the neural signals into frequency and time domain information, and applying a classifier to the frequency and time domain information.
  • Operation of the computer can also comprise determining information relating to attempted and/or actual movement of a body part of the patient. Operation of the computer can comprise determining the movement the patient is making. In addition or alternatively, operation of the computer can comprise determining the movement the patient is attempting to make. Further, in addition or alternatively, operation of the computer can comprise determining the part of the body that is moving or attempting to be moved.
  • Fig. la shows an example of a 16-channel 4x4 micro-electrode array.
  • Fig. lb shows placement of two micro-electrode arrays over a cortical surface, in which one of the micro-electrode arrays is placed over the face motor cortex and the other micro-electrode array is placed over Wernicke's area.
  • Fig. lc shows an audio waveform (top) of a verbal task and a
  • Fig. Id shows an audio waveform (top) of conversation, verbal task and verbal reward (top) and a corresponding spectrogram (bottom) of neural data recorded from a single channel over Wernicke's area.
  • Fig. 2a shows windows temporally aligned to spoken words that contain a frequency-domain structure in a spectrogram of neural data recorded from a micro-electrode over the face motor cortex.
  • Fig. 2b shows power spectra calculated for multiple trials and multiple electrodes.
  • Fig. 2c shows a two-dimensional matrix of micro-electrode power spectra and trial information.
  • Fig. 2d shows a principal component analysis performed on micro- electrode power spectra and trial information for two words.
  • Fig. 3a shows a distribution of performance results for each unique combination of two- word through ten- word combinations.
  • Fig. 3b shows a topography of channel performance for micro-electrodes resting over the face motor cortex.
  • Fig. 3c shows a topography of channel performance for micro-electrodes resting over Wernicke's area.
  • Figs. 4a-c illustrate plots of the coherence, separation, and frequency of LFPs measured from the cerebral cortex, according to some embodiments.
  • Fig. 5 shows an array of spectrograms for speaking the word “thirsty,” according to some embodiments.
  • Fig. 6 shows an array of spectrograms for speaking the word “thirsty,” according to some embodiments.
  • Fig. 7 shows a distribution of performance results for two through ten classifications.
  • Fig. 8 shows a block diagram of a system for recording and analyzing data from a micro-electrode array according to some embodiments.
  • Figs. 9a and 9b show grids or arrays of electrodes for use in systems or kits, according to some embodiments.
  • Fig. 10a shows an array of spectrograms for movement of the index finger of a hand according to some embodiments.
  • Fig. 10b shows an array of spectrograms for movement of the middle finger of the hand according to some embodiments.
  • Fig. 10c shows an array of spectrograms for movement of the thumb of the hand according to some embodiments.
  • Fig. 11 illustrates a method for decoding neural signals according to some embodiments.
  • Figs. 12a and 12b illustrate maps comparing hand position measurements against decoded local field potentials.
  • Pathological conditions such as amyotrophic lateral sclerosis or damage to the brainstem can leave patients severely paralyzed but fully aware, in a condition known as locked-in syndrome. Communication in this state is laborious, often reduced to selecting individual letters or words by arduous residual movement. More intuitive communication may be possible by directly interfacing with language areas of the cerebral cortex. Many studies of neural interfaces for communication have focused on the challenging problem of reconstructing continuous, dynamic speech.
  • embodiments described herein are tractable approaches of extracting information from microscale signals. Implementations of these embodiments can be used to provide discrete and/or continuous decoding of neural signals. Thus, embodiments can provide classification of a set of attempted and/or actual words and/or kinematics of a patient.
  • a plurality of electrodes can be used to monitor neural signals.
  • the electrodes can be a grid or array of subdural, nonpenetrating, high- impedance micro-electrodes are used to record local field potentials ("LFPs") from the cortical surface.
  • LFPs local field potentials
  • electrodes can be placed over (relative to) the most relevant areas of the brain. Although there is not a comprehensive list of the precise relevant areas of the brain, various areas of the brain are known at this time as providing the desired data for embodiments of the methods and systems disclosed herein. For example, in accordance with some embodiments, the electrodes can be placed over the face motor cortex and/or
  • the electrodes can contact the frontal lobe and/or other cortical areas.
  • electrodes can contact arm and/or hand areas of the primary motor cortex, Wernicke's area, Broca's area, the premotor cortex, and/or other areas of the cortex.
  • the electrodes can comprise a plurality of microwires.
  • the A LFP may be an electric field potential from a group of neurons located near the corresponding electrode. Neural data from many regions of the brain may be used to decode speech; however, data from electrodes over the face motor cortex were found to be the most accurately decodable. Some embodiments can provide a trial-by-trial decoding of spoken words from cortical surface LFPs in the human neocortex, as discussed further below.
  • BCIs brain computer interfaces
  • Penetrating electrodes have been used to perform rapid decoding of continuous motor movements from neuronal activity in the primary motor area of human neocortex; however, because of the risks associated with implantation in language centers, few studies have explored their use in speech BCIs.
  • the neurotrophic electrode is a penetrating electrode designed to mitigate the risks of chronic implantation that has been used to decode the formant frequencies of speech from neuronal activity in the left ventral premotor cortex.
  • LFPs on a cortical surface of the brain can be recorded from one or more micro-electrode arrays.
  • a micro-electrode array may comprise a plurality of nonpenetrating, 40- ⁇ microwires with 1-mm inter-electrode spacing.
  • Such micro-electrode grids or arrays have been shown to support high temporal- and spatial- resolution recordings.
  • some embodiments decode speech by classifying finite sets of words from cortical surface LFPs, thereby reducing the complexity of the problem to determining a limited number of classes.
  • Fig. la shows an example of a single 16-channel 4x4 micro-electrode grid or array that may be used to record LFPs on the cortical surface.
  • the micro- electrode array is shown next to a U.S. quarter-dollar coin for size comparison.
  • Fig. lb shows two 16-channel 4x4 micro-electrode arrays placed beneath the dura closely approximated to the cortical surface over the face motor cortex and Wernicke's area.
  • the wire bundle 112a leads to the array 110a over Wernicke' s area and the wire bundle 112b leads to the array 110b over the face motor cortex.
  • EoG electrocorticographic
  • Fig. lc shows an audio waveform (top) of a verbal task, in which a patient repeated the word "yes.”
  • Fig. lc also shows a corresponding spectrogram (bottom) of neural data recorded from a single channel or micro-electrode over the face motor cortex.
  • Fig. lc includes a normalized power scale indicating the power levels in the spectrogram. As shown in Fig. lc, the spectrogram reveals frequency-domain structure aligned to the individual words during the verbal task.
  • Fig. Id shows an audio waveform (top) of conversation, verbal task and verbal reward and a corresponding spectrogram (bottom) of neural data recorded from a single channel over Wernicke's area.
  • Wernicke's area is predominantly active when the patient converses and receives verbal rewards after completing an experiment, and was less active during the verbal task.
  • PCA principal component analysis
  • a center of mass, or centroid can be calculated as the average of the coordinates of all projected trials belonging to a particular word.
  • trials are projected into the principal component space and classified as specific words by their proximity to a centroid. An example of this is illustrated in Figs. 2a- 2d.
  • Fig. 2a shows an example of spectrograms 210a-210d of neural data for four different electrodes of a micro-electrode array placed over the face motor cortex.
  • a particular word is repeated three times during a verbal task with each repetition of the word corresponding to a trial.
  • the subject may speak the word or attempt to speak the word for the case where the subject is unable to intelligibly vocalize the word.
  • Fig. 2a shows three 500-msec windows 220a-220c where each window is temporally aligned to one instance of the spoken word.
  • Fig. 2a shows an example of spectrograms 210a-210d of neural data for four different electrodes of a micro-electrode array placed over the face motor cortex.
  • a particular word is repeated three times during a verbal task with each repetition of the word corresponding to a trial.
  • the subject may speak the word or attempt to speak the word for the case where the subject is unable to intelligibly vocalize the word.
  • the windows 220a- 220c contain frequency-domain structure in each spectrogram 210a-210d corresponding to the spoken word at the three trials.
  • Fig. 2b shows a power spectra for each electrode 210a- 210d and each trial.
  • Fig. 2c shows a two-dimensional matrix of micro-electrode power spectra and trial information for a word.
  • power spectra information is collected for each of N electrodes of the array and each of M trials.
  • Fig. 2d shows a principal component analysis performed on micro- electrode power spectra and trial information for the words "hungry” and "thirsty.”
  • principal component analysis performed on micro-electrode power spectra and trial information for the word "hungry” generates a cluster 250 in the principal component space, where each point in the cluster 250 represents one trial.
  • principal component analysis performed on micro-electrode power spectra and trial information for the word "thirsty” generates a cluster 255 in the principal component space.
  • three dimensions of the principal component space are shown for ease of illustration, although it is to be understood that the principal component space may comprise any number of dimensions.
  • a center of mass or centroid may be computed for each cluster corresponding to a particular word.
  • the word may be classified by performing principal component analysis on micro-electrode spectra information from the patient to project the spectra information into the principal component space and then determining its nearest centroid. The word that the patient spoke or attempted to speak can be then classified based on the word corresponding to the nearest centroid.
  • classification may also be used to decode a word based on the micro-electrode spectra information. Examples of other types of classification include maximum likelihood, support vector machine and Bayesian classification.
  • Vocal dynamics such as varied pitch or inflection could contribute to lower-than-expected performance in discriminating some word combinations.
  • decoding accuracies that were well above chance and the timing of the increased spectral power suggest that the micro-electrode array over the face motor cortex recorded signals involved in speech production.
  • activity recorded over Wernicke' s area appears to be involved in speech processing but likely represents language at a more abstract level.
  • FIG. 3b,c shows performance results for individual electrodes over the face motor cortex for different words
  • Fig. 3c shows performance results for individual electrodes over Wernicke's area for different words. Examining the mean performance of each word against all other words, it was found that electrode 14 ranged from 51.5% accuracy for the word "cold” to 81.5% accuracy for the word "yes.” The standard deviation of performance across all 16 motor-sensory electrodes was measured as 6.6 + 1.5 percentage points, suggesting that surface LFPs recorded from some electrodes corresponded to aspects of speech production present in some words but not others.
  • micro-electrode that provided the highest accuracy for any single word varied. Selecting the five electrodes of the array with best overall accuracy from the face motor cortex improved classification accuracy to 89.6 + 10.8% of two-word combinations (median 90.0%; Fig. 3a). However, selecting the five highest-performing electrodes over Wernicke's area did not improve performance (73.5 + 16.4% of two-word combinations correctly classified; median 73.3%) when compared with using all 16 electrodes over that region of cortex. Some micro-electrodes over the face motor cortex may not have recorded neural signals useful in decoding the specific set of words presented, indicating a more concrete mapping of the neural signal onto patterns of speech articulation. Conversely, most of the 16 micro-electrodes over Wernicke's area appear to have recorded neural signal related to language processing, supporting a more distributed and abstract encoding of speech.
  • Figs. 4a-c illustrate coherence functions for a micro-ECoG grid placed on the cerebral cortex.
  • Figs. 4a-c illustrate the coherence, separation, and frequency of LFPs measured from the cerebral cortex.
  • Fig. 4a is a mesh showing the coherence plotted against both separation distance and frequency.
  • Fig. 4b shows coherence plotted against frequency, with color representing separation (increasing with coherence).
  • Fig. 4c shows coherence plotted against separation with color representing frequency (increasing with coherence).
  • Figs. 4a-c demonstrate that the coherence of neural signals falls off within the scale of a few millimeters. Thus, this data indicates that neural signals are encoded in a micro-spatial scale and that this is the proper scale at which to space electrodes, such as micro-electrodes.
  • Fig. 5 shows an array of spectrograms for speaking the word "thirsty,” according to some embodiments. These spectrograms illustrate data from 32 electrodes while the patient said the word “Thirsty.” The spectrograms illustrate a complex structure of spectral power in both frequency and time.
  • Fig. 6 also shows an array of spectrograms. However, Fig. 6 illustrates the spectral power when the patient speaks the word "hello,” according to some embodiments. This can be contrasted with the structure observed in Fig. 5 when saying "hello" on the next slide. This observation led to the new decode shown in Fig. 11 and discussed further below.
  • Fig. 7 illustrates a distribution of multi-word performance results for two through ten classifications.
  • the tight inter-electrode spacing and small number of electrodes can produce a limited spatial coverage of the micro-electrode grid or array.
  • An optimized grid design with larger spacing and more electrodes would likely cover a larger number of relevant neural signals and allow better decoding accuracy. Performance could likely be further improved with patient training to stereotype word articulation.
  • the invasiveness of the micro-electrode grids or array could be reduced with epidural placement, as shown for similar recording devices.
  • the electrodes can be configured to communicate wirelessly with the system.
  • a micro-electrode grid or other embodiments of an electrode such components can communicate wirelessly with one or more components of the system.
  • a wireless implementation of the system might be practical given the relatively low bandwidth required to capture cortical surface LFPs.
  • Fig. 8 is block diagram showing an example of a system 450 for recording and processing LFPs from an micro-electrode array 410 that may be used for various embodiments.
  • the system 450 may include a receiver 452, a processor 455, and a memory 460.
  • the receiver 452 may be used to condition the electrical signals from the micro- electrode array 410 for processing by the processor 455.
  • the receiver 452 may include one or more of the following components: amplifiers (e.g., low- noise amplifiers) for amplifying the electrical signals, a filter for isolating electrical signals within a desired frequency bandwidth, and an analog-to-digital converter for digitizing the electrical signals for processing by the processor 455.
  • amplifiers e.g., low- noise amplifiers
  • filter for isolating electrical signals within a desired frequency bandwidth
  • an analog-to-digital converter for digitizing the electrical signals for processing by the processor 455.
  • the processor 455 may comprise a general purpose processor, digital signal processors (DSPs), application specific integrated circuit (ASICs), discrete hardware components, or any combination thereof.
  • DSPs digital signal processors
  • ASICs application specific integrated circuit
  • Methods for decoding speech using neural signals from the array 410 according to various embodiments discussed above may be embodied in software code that is stored in the memory 460 and executed by the processor 455.
  • the memory 460 may comprise any computer-readable media known in the art including volatile memory, nonvolatile memory, a Random Access Memory (RAM), a flash memory, a Read Only Memory (ROM), a removable disk, a CD-ROM, a DVD, any other suitable storage device, or a combination thereof.
  • RAM Random Access Memory
  • ROM Read Only Memory
  • the system 450 can be configured such that the memory 460 may be an on-board component.
  • the system 450 can be configured such that the memory 460 is off -board and system 450 wirelessly communicates with the off -board memory component.
  • the processor 455 may also output raw electrical signals, processed electrical signals, and/or results of analysis to an output device 465, including, but not limited to, a display for viewing by a neurologist, a printer for generating a computer readout, a computer-readable media, and/or to another computer via a computer network connection.
  • the output device 465 may also include an audio output device that outputs the decoded word as an audio output, e.g., a synthetic voice vocalizing the decoded word.
  • the processor 455 may decode a word by receiving neural signals, e.g., local field potentials, from the micro-electrode array 410 when the patient speaks the word or attempts to speak the word.
  • the processor 455 may then convert the neural signals into frequency-domain information, e.g., power spectra, for one or more electrodes of the array.
  • the processor 455 may then classify the frequency-domain information for the one or more electrodes into one of a set of words. For example, the processor 455 may perform principal component analysis on the frequency-domain information to project the frequency-domain information into the principal component space and determine its nearest centroid in the principal component space, as described above.
  • the processor 455 may display the decoded word on a display and/or vocalize the decoded word from an audio output device.
  • the processor 455 may be trained to classify a particular word using the methods described above with reference to Figs. 2a- 2d.
  • Neural data were recorded from 4x4-channel grids of nonpenetrating microwires at 30,000 samples per second (Cerebus, Blackrock Microsystems) while a patient performed speech tasks. During the speech task, the patient repeated a word between 10 and 20 times, with approximately one second separation between trials. Trial start times were noted for later analysis. [0091] Time and Frequency Domain Signal Processing
  • Frequency data were averaged to represent 10-Hz bins, and bins with line noise (60 Hz) or harmonics were deleted.
  • discrete decoding can comprise, for example, the classification of neural signals into discrete categories.
  • discrete decoding can be used to classify or identify attempted and/or actual words and/or movements from a subject.
  • a variety of classification algorithms may be applied to these time and frequency domain features to decode words, kinematics, and/or attempted kinematics.
  • Fisher' s linear discriminant may be used.
  • the classifier can be trained by finding the mean and covariance for each class in the training set. These parameters are used to determine a projection of the multidimensional features onto a linear direction such that the separation between classes can be maximized and the separation within classes can be minimized. If the data are separable, the trials of a given class can cluster along this new direction. This projection can be applied to each trial in the test set, and the trial can be assigned to a class based on its proximity to the clusters identified during training.
  • continuous decoding can comprise, for example, the decoding of neural signals to provide information on continuously varying attempted and/or actual movements.
  • continuous decoding can comprise proportional decoding of attempted and/or actual movement of a part of a subject body.
  • Such decoding can provide data on position, velocity, and/or acceleration of movement of the part of the subject's body.
  • Kalman Filtering may be used.
  • a Kalman Filter has been used to decode hand position from the neural signals recorded on micro-ECoG grids placed over the hand and arm areas of pre-motor cortex in a human patient. If a decoder is trained on other parameters of attempted and/or actual movements, i.e. force, velocity, acceleration, etc., then it can decode these parameters as well as position.
  • Raw neural data were first downsampled from 30,000 samples/sec to 3,000 samples/sec. To mitigate the effects of 60-Hz noise, the trial-averaged spectrum for each nonpenetrating microwire was calculated to determine the width and amplitude of the 60-Hz noise band. Across ail nonpenetrating microwires in P2, for example, noise levels in this band ranged from 5 dB through 20 dB above the normal spectrum, meaning that a single filter might effectively attenuate noise in a few channels but would leave large banding in most of the data. Therefore, a fifth-order Chebyshev Type-II filter was designed to individually mirror the calculated width and height of the 60-Hz noise.
  • Trial components were marked by points where the velocity of the patient's motion crossed a threshold of one standard deviation from rest. This model was used to mark both outward and return movement; additionally, baseline trials were marked at periods of no movement between trials. Only outward movement trials were included in the present analysis. The ti mestamps of these outward movements were used as markers for reading electrode data. For spectrograms, one second of data, with movement aligned at 0.5 seconds, was used.
  • Spectrograms were generated using the Chronux (chronux.org) package with 250 msec windows and 50 msec step size: tapering parameters were set to a time-bandwidth product of five, and nine leading tapers. Spectrum plots of the raw data were characterized by a power law trend whose features dominated frequency analysis. For this reason, the spectrograms were normalized to the trial-averaged spectrum for all trials of like movement.
  • Pairwise cross- correlation analysis between each set of nonpenetrating microwires was performed on neuronal recordings from a period during arm movement to explore the strength of the linear relationship between the signals recorded by nonpenetrating microwires within close proximity. Because the correlation metric indicates the degree to which two sequences are linearly related, this estimation of independence, while not a complete metric, can approximate the possible usefulness of the high spatial-resolution recording afforded by the nonpenetrating microwire devices. In particular, the correlation analysis should help to clarify whether the nonpenetrating microwire array could be replaced by a single large electrode, or if the tighter spacing of the nonpenetrating microwires allows for recording of potentially unique signals.
  • the data were further processed so that the decode could operate on continuously presented frames of data.
  • the spectral data were averaged into frequency bins covering 0-5 Hz, 5-13 Hz, 13-30 Hz, 30-80 Hz, 80-200 Hz, and 200- 500 Hz. Ail frequencies within +5 Hz of 60 Hz or its harmonics were removed due to line noise contamination.
  • the movement data were downsampled to the same sampling rate as the spectrograms (4 samples/sec). An offset of 150 msec was introduced between the movement data and the spectral data to model delay between neural activity and motor output.
  • the final feature vector » consisted of 6 frequency bins per channel and the hand state xk was represented by a 6- dimensional vector comprising x and y position, velocity, and acceleration. These vectors were defined for k - 1, 2, ... M, where M was the number of samples in the data set.
  • the decoding process required learning the parameters A, H, W, and Q from the training data, then predicting the hand kinematics at each time given the prior estimate of the state and new measurements of neural data.
  • Neural data recorded during two different task sessions were used for training and testing.
  • the means of the movement and neural data features were calculated in order to center the data, and the data were orthogonalized using PCA, with tailing principal components contributing less than 1% of the variance discarded.
  • the principal components and means found during training were applied to the testing data to stay as close as possible to the real-time case where such information would be unavailable.
  • K k P- k H T (HP-H T + QY 1 [00122]
  • the output of the Kalman filter was analyzed by calculating the correlation coefficient between the actual trajectory, recorded during the task session, and the predicted trajectory output from the Kalman filter. To understand which frequencies were important to the Kalman filter, frequency features were averaged into 10 Hz frequency bins between 0 and 500 Hz (from all channels). These narrowband features were tested individually with the Kalman filter, and the correlation coefficients for the x- and y-positions were recorded for each run.
  • An algorithm useful for embodiments of the methods and systems disclosed herein may be able to take advantage of the processing of information in neural structures at the micro-scale using features in space, time, and frequency.
  • the algorithm can correlate the dynamics in local field potential voltage and spectral power (frequency range, e.g., about 0 - 1000 Hz or more) across time with attempted and/or actual movements (speaking, finger flexion and extension, and reaching, etc.).
  • the dynamics in the neural signals across the spatial extent of the grid can also be correlated with movement.
  • decoding can consist of detection and classification of specific types of attempted and/or actual movements, e.g., saying a specific word or making a specific grasping movement. Further, in some embodiments, decoding can consist of detection and classification of specific types of continuously varying parameters of attempted and/or actual movement, e.g. position or acceleration of one or more body parts. Principal Components Analysis (PCA) can be applied to provide classification of attempted and/or actual movements. Further, as noted herein, Kalman Filtering can be used to continuously decode hand position.
  • PCA Principal Components Analysis
  • FIG. 10a shows an array of spectrograms 510 recorded on individual electrodes of the 96 channel micro-ECoG grid when the subject moved an index finger.
  • the electrodes can be in contact with a cortical surface of the brain to sense LFPs from the cortical surface.
  • the electrodes can be in contact with the face motor cortex, premotor cortex, Wernike's area, Broca's area, and/or other areas of the brain.
  • the electrodes can also be spaced from a surface of the brain.
  • embodiments can provide a desired relative spatial positioning of the electrodes, whether or not one or more of the electrodes is in direct contact with a surface of the brain.
  • some embodiments provide methods and systems that can detect signals from portions of the central nervous system and/or peripheral nervous system.
  • embodiments of the methods and systems can be applied to grids placed on the cerebral cortex, but embodiments of the methods and systems can also be to neural signals recorded from sub-cortical structures and the peripheral nervous system.
  • the ranges provided for the physical structure of electrodes can equally apply to sub-cortical and peripheral nerve structures.
  • Some embodiments of the systems and kits disclosed herein can comprise one or more electrodes.
  • the electrodes can be formed in a grid or array. Some embodiments can use one or more electrodes having a size or surface area of between about 0.0001 mm 2 to about 1 mm 2 . Further, in some embodiments, the electrode can have a size or surface area of between about 0.0025 mm 2 to about 0.50 mm 2 . Moreover, some embodiments can use one or more electrodes having a diameter of between about 0.01 mm to about 1 mm. Further, in some embodiments, the electrode can have a diameter of between about 0.05 mm to about 0.50 mm.
  • the electrodes can define a desired impedance.
  • the electrodes can have impedances in the range of about 10 kQ To about 1 ⁇ . Further, the electrodes can have impedances within a range of about 100 kQ to about 500 kQ
  • the total number of electrodes in an array or grid can be between about 4 and about 65536.
  • An array can be provided having between about 4 to about 1000 microelectrodes or more. In many implementations, an array can have between about 4 to about 200 microelectrodes or more. For example, an array can have 16, 32, or 64 microelectrodes arranged in a grid. Further, in some embodiments, the grid can be arranged in a square.
  • the inter-electrode spacing can be between about 100 microns and about 10000 microns. In some embodiments, inter- electrode spacing can be between about 0.05 mm to about 5mm. In some embodiments, optimal spacing of the microelectrodes can be between about 2 mm to about 3 mm apart.
  • the sampling rate can be between 0 and about 2500 Hz to provide a desired temporal resolution. In some embodiments, the sampling rate can be between about 10 and about 500 Hz.
  • some embodiments can use one or more electrodes can have a thickness of between about 0.010 mm and about 5 mm. Some embodiments can be configured such that the electrodes define a thickness of between about 0.020 mm and about 3 mm. For example, some embodiments can use electrodes having a thickness of between about 0.050 mm and about 0.500 mm.
  • an electrode grid can be provided.
  • the grid can comprise one or more small protuberances that elevate an active electrode site above a base plane of the grid.
  • Fig. 9a illustrates a grid 500 having a plurality of protuberances 502 having a generally frustro- conical shape, attached to a base plane 504 of the grid 500.
  • Fig. 9b illustrates a grid 600 having a plurality of protuberances 602 having a generally frustro-conical shape, attached to a base plane 604 of the grid 600.
  • protuberances can aid in making electrical contact the neural tissue and minimizing any shunting of electrical signal through bodily fluids, e.g., cerebrospinal fluid.
  • bodily fluids e.g., cerebrospinal fluid.
  • protuberance can be of various geometries, such as semi-spherical, cylindrical, conical, etc.
  • the electrical ground and reference for the electrodes can consist of low-impedance electrodes that are in contact with the fluid space in which the electrodes are implanted, e.g., the sub-dural cerebrospinal fluid.
  • ground and reference can be placed either the epidural space or the surface of the scalp. The electrical ground and reference may be placed together in any of these spaces, or could be placed separately in any combination of these spaces.
  • the electrodes can comprise microelectrodes, microwire arrays and/or other equipment for recording LFPs from the cortical surface of the brain to provide a desired spatial resolution.
  • the electrodes can define one or more unique sizes.
  • the electrodes can be round, square, and/or have serrated edges.
  • the electrodes can comprise microelectrodes that are formed in grids or arrays.
  • the array of microelectrodes can be embedded in a polymer or other material that allows the grid to be flexible and achieve a micro-thickness.
  • a rubbery clear silicone or other such materials can be acceptable for some uses and may be used in some embodiments.
  • other polymers can be acceptable for some uses and may include polyurethane and poly-imide.
  • each spectrogram 510 shows measured power as a function of frequency and time for the corresponding electrode of the micro-ECoG grid, where the vertical axis represents frequency and the horizontal axis represents time.
  • each spectrogram 510 shows measured power over a frequency range of 0 to 500 Hz and a time range of between approximately between one second before and one second after the finger movement.
  • the array of spectrograms 510 show time and frequency domain data at different spatial positions on the brain.
  • the array of spectrograms 510 represent time, frequency and spatial information that can be used to decode attempted and/or actual finger movement.
  • Fig. 10b shows an array of spectrograms 510 recorded on individual electrodes of the 96 channel micro-ECoG grid when the subject moved a middle finger
  • Fig. 10c shows an array of spectrograms 510 recorded on individual electrodes of the 96 channel micro-ECoG grid when the subject moved a thumb, where the index finger, middle finger and thumb were from the same hand.
  • Figs. 10a, 10b and 10c clearly show different spatial and temporal patterns of spectral power correlated with movement of different fingers of the subject's hand.
  • Each spectrogram 510 shows a temporal pattern of spectral power for the corresponding electrode and the array of spectrograms shows a spatial pattern of the spectral power across the electrodes of the micro-ECoG grid.
  • Decoding of neural signals may be performed using principal components analysis (PCA) and Kalman filtering on the micro-ECoG signals.
  • PCA principal components analysis
  • the features for attempted and/or actual finger movement may be refined through PCA into a principal component space and classification may be performed in the principal component space (e.g., nearest- neighbor clustering or other classification algorithm) to classify (decode) the attempted and/or actual finger movement.
  • classification algorithms include k-nearest neighbors, Bayesian classification, Hidden Markov models (HMM) and learning vector quantization (LVQ).
  • HMM Hidden Markov models
  • LVQ learning vector quantization
  • Other algorithms that may be used include linear regression, Fisher's linear discriminants and support vector machines (SVM).
  • Fig. 11 illustrates a method for decoding neural signals according to some embodiments.
  • data is received from M channels, where each channel corresponds to an electrode of an electrode array and each electrode receives neural signals emanating from a different spatial position of the brain.
  • spectral power e.g., power at different frequency components
  • the spectral power over time provides information about the temporal dynamics of the spectral power.
  • time domain data of the corresponding signal may also be obtained, for example, voltage of the signal over time (voltage at different times), as shown in Fig. 11. This time domain data provides additional information that can be used to improve decoding performance.
  • the spectral power e.g., power at different frequency components
  • fi(t) . . . f (t) the time domain data
  • v(t) e.g., voltage over time
  • the features for the different channels correspond to different spatial positions of the brain and therefore provide spatial information (a spatial pattern).
  • feature information may be collected across the channels for each of a plurality of trials, where each trial may correspond to attempted and/or actual movement of a particular finger or an attempt to speak a particular word.
  • Fig. 11 shows a two-dimensional matrix 620 in which each row represents a different trial and each column represents features for a particular channel.
  • Fig. 11 shows an example in which trials for the words "hungry” and "thirsty" are mapped to the principal component space 630.
  • three dimensions of the principal component space are shown for ease of illustration, although it is to be understood that the principal component space may comprise any number of dimensions.
  • a centroid may be calculated for each word or type of attempted and/or actual finger movement to be classified (decoded).
  • the attempted and/or actual word or finger movement may be classified by performing principal component analysis on the corresponding features from the M channels (e.g., fi(t) . . . fN(t) and v(t)) to refine the features into the principal component space 630 and determining its nearest centroid in the principle component space 630.
  • the word that the patient spoke and/or attempted to speak and/or the attempted and/or actual finger movement can be then classified (decoded) based on the attempted and/or actual word and/or finger movement corresponding to the nearest centroid.
  • Other classification algorithms may also be used.
  • the method was described using the example of finger movement decoding, those skilled in the art will appreciate that the method may be used to decode attempted and/or actual movements of other parts of the body (e.g., toe movement, arm movement, etc.). The method may also be used to decode one or more characteristics of attempted and/or actual finger movement such as position of the finger, force exerted by the finger, acceleration of the finger.
  • Figs. 12a and 12b illustrate the correlation of LFPs with kinematics, and in particular, with reaching movements.
  • Figs. 12a and 12b show maps that compare hand position measurements against decoded local field potentials.
  • LFPs can be used to perform proportional decoding and/or classification of attempted and/or actual movement.
  • a Kalman filter was used to map frequency and time domain features onto the x-position, y-position, velocity, and acceleration of the hand.
  • Figs. 12a and 12b show the position of the hand measure with a mouse (shown in blue) and the position decoding from the LFP (shown in red). While the position decoding from the LFP demonstrates a generally larger amplitude in both maps, there is a high degree of correlation between the position of the hand measure with a mouse and the position decoding from the LFP.
  • embodiments of the methods and systems disclosed herein can be used to decode kinematics of the body. Further, some embodiments can be used to decode kinematics of one or more parts of the body individually and/or together with other parts of the body.

Abstract

Methods and systems are described for decoding neural signals, e.g., local field potentials, recorded from, e.g., a brain cortical surface. In some embodiments, a system, kit and/or method for decoding neural signals is provided. The system or kit can comprise a receiver configured to receive a neural signal from each of a plurality of electrodes. Each neural signal can correspond to a different spatial position of the brain of a patient and each neural signal can emanate from the brain when the patient moves and/or attempts to move a part of the body. The system or kit can also comprise a processor configured to convert the neural signals and to apply a classifier to determine the part of the body and/or attempted and/or actual movement of the part of the body.

Description

SYSTEMS AND METHODS FOR DECODING NEURAL SIGNALS
Field
[0001] The present disclosure relates to the general fields of bioengineering and computer technology, and more particularly to methods and systems for decoding neural signals, and in particular, discrete and/or continuous decoding of neural signals, e.g., local field potentials, recorded from a cortical surface.
Summary
[0002] Some embodiments of the present technology provide systems and methods for decoding microscale neural signals to extract information on attempted and/or actual movement and/or action of a subject. Implementations of these embodiments can be used to provide discrete and/or continuous decoding of neural signals. Discrete and/or continuous decoding can be used to decode micro-scale neural signals for intent. In some embodiments, the neural signals can be e.g., local field potentials, recorded relative to a given spatial location of the brain.
[0003] Discrete decoding can comprise, for example, the classification of neural signals into discrete categories. For example, discrete decoding can be used to classify or identify attempted and/or actual words and/or movements from a subject.
[0004] Continuous decoding can comprise, for example, the decoding of neural signals to provide information on continuously varying attempted and/or actual movements. For example, continuous decoding can comprise proportional decoding of attempted and/or actual movement of a part of a subject body. Such decoding can provide data on position, velocity, and/or acceleration of movement of the part of the subject's body.
[0005] Further, some embodiments provide methods and systems that can detect signals from portions of the central nervous system and/or peripheral nervous system.
[0006] Some embodiments can provide a brain-machine .interface that can be used in various applications.
[0007] In some embodiments, a system for decoding neural signals is provided. The system can comprise a receiver configured to receive a neural signal from each of a plurality of electrodes and a processor configured to convert the neural signals and to apply a classifier to the converted signals. The system can be configured to determine the movement the patient is making. In addition or alternatively, the system can be configured to determine the movement the patient is attempting to make. Further, in addition or alternatively, the system can be configured to determine the part of the body that is moving or attempting to be moved.
[0008] Some embodiments also provide a kit for decoding neural signals. The kit can comprise a plurality of electrodes, a receiver, and a processor.
[0009] In some embodiments, the plurality of electrodes can be configured to detect neural signals corresponding to different spatial positions on or within the brain of a patient. Each neural signal can be emanated from the brain when the patient moves and/or attempts to move a part of the body.
[0010] In some embodiments, the receiver can be configured to receive a neural signal from each of a plurality of electrodes. Further, each neural signal can correspond to a different spatial position of the brain of a patient. In addition, each neural signal can emanate from the brain when the patient moves and/or attempts to move a part of the body.
[0011] In some embodiments, the processor can be configured convert the neural signals into frequency and time domain information. In addition, the processor can also apply a classifier to the frequency and time domain information so as to determine the movement the patient is making and/or is attempting to make.
[0012] In some embodiments, the electrodes contact a surface of the brain. For example, the electrodes can contact a cortical surface of the brain. However, the electrodes can also be spaced from a surface of the brain. Thus, embodiments can provide a desired relative spatial positioning of the electrodes, whether or not one or more of the electrodes is in direct contact with a surface of the brain.
[0013] In some embodiments, the electrodes contact a face motor cortex. Further, the electrodes can contact other cortical areas. For example, electrodes can contact arm and/or hand areas of the primary motor cortex, Wernicke' s area, Broca' s area, the premotor cortex, and other areas of the cortex.
[0014] In some embodiments, each neural signal comprises a local field potential from the cortical surface.
[0015] In some embodiments, the electrodes comprise micro-electrodes.
[0016] In some embodiments, the frequency and time domain information for each neural signal comprises spectral power of the neural signal at different times. [0017] In some embodiments, the frequency and time domain information for each neural signal also comprises voltage of the neural signal at different times.
[0018] In some embodiments, the spectral power for each neural signal comprises power at different frequency components.
[0019] In some embodiments, the movement of the part of the body comprises movement of an individual finger of a hand.
[0020] In some embodiments, a kit can be provided that is operative to implement embodiments of the methods disclosed herein. For example, the kit can comprise electrodes for implantation in contact with neural tissue. The electrodes can be configured as a grid of micro-electrodes. Optionally, surgical tools can be provided to implant the electrode grid.
[0021] The kit can also comprise a data acquisition system capable of recording neural signals from the electrodes. The data acquisition system can record data from a grid of micro-electrodes, as specified in terms of number of electrodes, sampling rate, bandwidth, and other desired parameters.
[0022] The kit may also comprise a computation unit (such as a computer processor) capable of executing an appropriate algorithm. The computation unit can execute the decoding algorithm in near real-time. For example, the computation unit can execute the algorithm sufficiently quickly so the patient does not notice any lag. As discussed herein, an algorithm can comprise enable the computation unit to perform classification or continuous decoding of neural signals.
[0023] In some embodiments, the kit can optionally comprise an effector device. The effector device can be a computer interface, a virtual reality environment, a prosthetic device, a speech synthesizer, or other machine useful for executing the output of the decoding algorithm.
[0024] In some embodiments, a method for decoding neural signals is provided. The method comprises receiving a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves. A part of the body. The method also comprises converting the neural signals into frequency and time domain information, and applying a classifier to the frequency and time domain information so as to determine the patient's movement of, and/or attempt to move, a part of the body. [0025] In some embodiments, a non- transitory machine-readable medium is provided. The machine-readable medium comprises instructions stored therein, which when executed by a machine, cause the machine to perform operations for decoding neural signals. The operations comprise receiving a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body. The operations can also comprise converting the neural signals into frequency and time domain information, and applying a classifier to the frequency and time domain information. Operation of the computer can also comprise determining information relating to attempted and/or actual movement of a body part of the patient. Operation of the computer can comprise determining the movement the patient is making. In addition or alternatively, operation of the computer can comprise determining the movement the patient is attempting to make. Further, in addition or alternatively, operation of the computer can comprise determining the part of the body that is moving or attempting to be moved.
[0026] For purposes of summarizing the disclosure, certain aspects, advantages, embodiments and novel features of the disclosure have been described herein. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment of the disclosure. Thus, the disclosure may be embodied or carried out in a manner that achieves or optimizes one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein.
Brief Description of the Drawings
[0027] Fig. la shows an example of a 16-channel 4x4 micro-electrode array.
[0028] Fig. lb shows placement of two micro-electrode arrays over a cortical surface, in which one of the micro-electrode arrays is placed over the face motor cortex and the other micro-electrode array is placed over Wernicke's area.
[0029] Fig. lc shows an audio waveform (top) of a verbal task and a
corresponding spectrogram (bottom) of neural data recorded from a single channel over the face motor cortex.
[0030] Fig. Id shows an audio waveform (top) of conversation, verbal task and verbal reward (top) and a corresponding spectrogram (bottom) of neural data recorded from a single channel over Wernicke's area. [0031] Fig. 2a shows windows temporally aligned to spoken words that contain a frequency-domain structure in a spectrogram of neural data recorded from a micro-electrode over the face motor cortex.
[0032] Fig. 2b shows power spectra calculated for multiple trials and multiple electrodes.
[0033] Fig. 2c shows a two-dimensional matrix of micro-electrode power spectra and trial information.
[0034] Fig. 2d shows a principal component analysis performed on micro- electrode power spectra and trial information for two words.
[0035] Fig. 3a shows a distribution of performance results for each unique combination of two- word through ten- word combinations.
[0036] Fig. 3b shows a topography of channel performance for micro-electrodes resting over the face motor cortex.
[0037] Fig. 3c shows a topography of channel performance for micro-electrodes resting over Wernicke's area.
[0038] Figs. 4a-c illustrate plots of the coherence, separation, and frequency of LFPs measured from the cerebral cortex, according to some embodiments.
[0039] Fig. 5 shows an array of spectrograms for speaking the word "thirsty," according to some embodiments.
[0040] Fig. 6 shows an array of spectrograms for speaking the word "thirsty," according to some embodiments.
[0041] Fig. 7 shows a distribution of performance results for two through ten classifications.
[0042] Fig. 8 shows a block diagram of a system for recording and analyzing data from a micro-electrode array according to some embodiments.
[0043] Figs. 9a and 9b show grids or arrays of electrodes for use in systems or kits, according to some embodiments.
[0044] Fig. 10a shows an array of spectrograms for movement of the index finger of a hand according to some embodiments.
[0045] Fig. 10b shows an array of spectrograms for movement of the middle finger of the hand according to some embodiments. [0046] Fig. 10c shows an array of spectrograms for movement of the thumb of the hand according to some embodiments.
[0047] Fig. 11 illustrates a method for decoding neural signals according to some embodiments.
[0048] Figs. 12a and 12b illustrate maps comparing hand position measurements against decoded local field potentials.
Detailed Description
[0049] Pathological conditions such as amyotrophic lateral sclerosis or damage to the brainstem can leave patients severely paralyzed but fully aware, in a condition known as locked-in syndrome. Communication in this state is laborious, often reduced to selecting individual letters or words by arduous residual movement. More intuitive communication may be possible by directly interfacing with language areas of the cerebral cortex. Many studies of neural interfaces for communication have focused on the challenging problem of reconstructing continuous, dynamic speech.
[0050] According to some embodiments described herein are tractable approaches of extracting information from microscale signals. Implementations of these embodiments can be used to provide discrete and/or continuous decoding of neural signals. Thus, embodiments can provide classification of a set of attempted and/or actual words and/or kinematics of a patient.
[0051] In some embodiments, a plurality of electrodes can be used to monitor neural signals. The electrodes can be a grid or array of subdural, nonpenetrating, high- impedance micro-electrodes are used to record local field potentials ("LFPs") from the cortical surface.
[0052] In accordance with some embodiments of the methods and systems disclosed herein, electrodes can be placed over (relative to) the most relevant areas of the brain. Although there is not a comprehensive list of the precise relevant areas of the brain, various areas of the brain are known at this time as providing the desired data for embodiments of the methods and systems disclosed herein. For example, in accordance with some embodiments, the electrodes can be placed over the face motor cortex and/or
Wernicke's area. In embodiments, the electrodes can contact the frontal lobe and/or other cortical areas. For example, electrodes can contact arm and/or hand areas of the primary motor cortex, Wernicke's area, Broca's area, the premotor cortex, and/or other areas of the cortex.
[0053] In some embodiments, the electrodes can comprise a plurality of microwires. Further, the A LFP may be an electric field potential from a group of neurons located near the corresponding electrode. Neural data from many regions of the brain may be used to decode speech; however, data from electrodes over the face motor cortex were found to be the most accurately decodable. Some embodiments can provide a trial-by-trial decoding of spoken words from cortical surface LFPs in the human neocortex, as discussed further below.
[0054] Early studies of brain computer interfaces (BCIs) for speech trained patients to use slow cortical potentials to interact with a computer for communication. More recently noninvasive BCIs have demonstrated improvements but can require extensive training to achieve moderate accuracy and rates of communication. Penetrating electrodes have been used to perform rapid decoding of continuous motor movements from neuronal activity in the primary motor area of human neocortex; however, because of the risks associated with implantation in language centers, few studies have explored their use in speech BCIs. The neurotrophic electrode is a penetrating electrode designed to mitigate the risks of chronic implantation that has been used to decode the formant frequencies of speech from neuronal activity in the left ventral premotor cortex. Studies investigating less invasive measures have shown that cortical surface potentials recorded by electrocorticographic (ECoG) electrodes can discriminate between motor and speech tasks and discriminate phonemes. Regardless of the recording paradigm used, most studies of speech BCIs have focused on the challenging task of decoding continuous, dynamic speech from the neural representations of formant frequencies in either action potentials or field potentials.
[0055] Some embodiments provide a novel recording device and method for decoding speech. In some embodiments, LFPs on a cortical surface of the brain can be recorded from one or more micro-electrode arrays. For example, a micro-electrode array may comprise a plurality of nonpenetrating, 40-μιη microwires with 1-mm inter-electrode spacing. Such micro-electrode grids or arrays have been shown to support high temporal- and spatial- resolution recordings. Also, rather than decoding continuous speech, some embodiments decode speech by classifying finite sets of words from cortical surface LFPs, thereby reducing the complexity of the problem to determining a limited number of classes. [0056] Fig. la shows an example of a single 16-channel 4x4 micro-electrode grid or array that may be used to record LFPs on the cortical surface. In Fig. la, the micro- electrode array is shown next to a U.S. quarter-dollar coin for size comparison. Fig. lb shows two 16-channel 4x4 micro-electrode arrays placed beneath the dura closely approximated to the cortical surface over the face motor cortex and Wernicke's area. In Fig. lb, the wire bundle 112a leads to the array 110a over Wernicke' s area and the wire bundle 112b leads to the array 110b over the face motor cortex. Fig. lb also shows electrocorticographic (ECoG) electrodes, which are much larger than the micro-electrodes of the arrays. The wide range of muscles required to articulate vocalizations suggests that unique neural activity in the face motor cortex may correspond to unique word formulations. Wernicke's area is known to play an important role in high-level language processing.
[0057] Fig. lc shows an audio waveform (top) of a verbal task, in which a patient repeated the word "yes." Fig. lc also shows a corresponding spectrogram (bottom) of neural data recorded from a single channel or micro-electrode over the face motor cortex. Fig. lc includes a normalized power scale indicating the power levels in the spectrogram. As shown in Fig. lc, the spectrogram reveals frequency-domain structure aligned to the individual words during the verbal task.
[0058] Fig. Id shows an audio waveform (top) of conversation, verbal task and verbal reward and a corresponding spectrogram (bottom) of neural data recorded from a single channel over Wernicke's area. As shown in Fig. Id, Wernicke's area is predominantly active when the patient converses and receives verbal rewards after completing an experiment, and was less active during the verbal task.
[0059] Previous studies have used principal component analysis (PCA) to separate frequency-domain features in neural signals. In one embodiment, PCA is used to classify a finite set of words. For each word, PCA can be performed on power spectra from each electrode and each trial simultaneously. During the training phase, a center of mass, or centroid, can be calculated as the average of the coordinates of all projected trials belonging to a particular word. During the classification phase, trials are projected into the principal component space and classified as specific words by their proximity to a centroid. An example of this is illustrated in Figs. 2a- 2d.
[0060] Fig. 2a shows an example of spectrograms 210a-210d of neural data for four different electrodes of a micro-electrode array placed over the face motor cortex. In this example, a particular word is repeated three times during a verbal task with each repetition of the word corresponding to a trial. For each trial, the subject may speak the word or attempt to speak the word for the case where the subject is unable to intelligibly vocalize the word. For each spectrogram, Fig. 2a shows three 500-msec windows 220a-220c where each window is temporally aligned to one instance of the spoken word. As shown in Fig. 2a, the windows 220a- 220c contain frequency-domain structure in each spectrogram 210a-210d corresponding to the spoken word at the three trials. Fig. 2b shows a power spectra for each electrode 210a- 210d and each trial.
[0061] Fig. 2c shows a two-dimensional matrix of micro-electrode power spectra and trial information for a word. In this example, power spectra information is collected for each of N electrodes of the array and each of M trials.
[0062] Fig. 2d shows a principal component analysis performed on micro- electrode power spectra and trial information for the words "hungry" and "thirsty." In this example, principal component analysis performed on micro-electrode power spectra and trial information for the word "hungry" generates a cluster 250 in the principal component space, where each point in the cluster 250 represents one trial. Similarly, principal component analysis performed on micro-electrode power spectra and trial information for the word "thirsty" generates a cluster 255 in the principal component space. In the example in Fig. 2d, three dimensions of the principal component space are shown for ease of illustration, although it is to be understood that the principal component space may comprise any number of dimensions.
[0063] In Fig. 2d, a center of mass or centroid may be computed for each cluster corresponding to a particular word. During the classification phase, when a patient speaks a word or attempts to speak a word, the word may be classified by performing principal component analysis on micro-electrode spectra information from the patient to project the spectra information into the principal component space and then determining its nearest centroid. The word that the patient spoke or attempted to speak can be then classified based on the word corresponding to the nearest centroid. Those skilled in the art will appreciate that other types of classification may also be used to decode a word based on the micro-electrode spectra information. Examples of other types of classification include maximum likelihood, support vector machine and Bayesian classification. [0064] Classification was performed both separately (Fig. 3a) and jointly for cortical surface LFP data recorded over the face motor cortex and cortical surface LFP data recorded over Wernicke's area. Electrodes over the face motor cortex offered the best classification performance. Out of 45 unique two-word combinations, 85.0 + 13.1% (mean + standard deviation) were correctly classified using data from all 16 array electrodes (median performance was 83.3%). Data recorded over Wernicke's area were less classifiable with 76.2 + 15.0% of two-word combinations correctly classified (median 76.7%). Joint classification did not improve performance over the level achieved by the face motor electrodes alone (0.40 + 0.43% difference in the percentage of two- through ten- word combinations classified correctly). Vocal dynamics such as varied pitch or inflection could contribute to lower-than-expected performance in discriminating some word combinations. Regardless, decoding accuracies that were well above chance and the timing of the increased spectral power suggest that the micro-electrode array over the face motor cortex recorded signals involved in speech production. Similarly, activity recorded over Wernicke' s area appears to be involved in speech processing but likely represents language at a more abstract level.
[0065] Surface LFPs recorded from individual micro-electrodes were better able to decode some words than others (Fig. 3b,c). Fig. 3b shows performance results for individual electrodes over the face motor cortex for different words, and Fig. 3c shows performance results for individual electrodes over Wernicke's area for different words. Examining the mean performance of each word against all other words, it was found that electrode 14 ranged from 51.5% accuracy for the word "cold" to 81.5% accuracy for the word "yes." The standard deviation of performance across all 16 motor-sensory electrodes was measured as 6.6 + 1.5 percentage points, suggesting that surface LFPs recorded from some electrodes corresponded to aspects of speech production present in some words but not others.
[0066] The micro-electrode that provided the highest accuracy for any single word varied. Selecting the five electrodes of the array with best overall accuracy from the face motor cortex improved classification accuracy to 89.6 + 10.8% of two-word combinations (median 90.0%; Fig. 3a). However, selecting the five highest-performing electrodes over Wernicke's area did not improve performance (73.5 + 16.4% of two-word combinations correctly classified; median 73.3%) when compared with using all 16 electrodes over that region of cortex. Some micro-electrodes over the face motor cortex may not have recorded neural signals useful in decoding the specific set of words presented, indicating a more concrete mapping of the neural signal onto patterns of speech articulation. Conversely, most of the 16 micro-electrodes over Wernicke's area appear to have recorded neural signal related to language processing, supporting a more distributed and abstract encoding of speech.
[0067] Decoding surface LFPs from the best five micro-electrodes simultaneously gave better results than decoding data from the same micro-electrodes individually. As much as 20.0 percentage points difference (vs. electrode 15 alone; ten-word combination) was found. On average, the collective accuracy of these five electrodes was 16.2 + 2.8 percentage points higher than their independently measured accuracy. Neural activity recorded by these five micro-electrodes likely corresponded to multiple aspects of speech articulation that varied across the set of words used in the experiments.
[0068] Figs. 4a-c illustrate coherence functions for a micro-ECoG grid placed on the cerebral cortex. Figs. 4a-c illustrate the coherence, separation, and frequency of LFPs measured from the cerebral cortex. Fig. 4a is a mesh showing the coherence plotted against both separation distance and frequency. Fig. 4b shows coherence plotted against frequency, with color representing separation (increasing with coherence). Fig. 4c shows coherence plotted against separation with color representing frequency (increasing with coherence). Figs. 4a-c demonstrate that the coherence of neural signals falls off within the scale of a few millimeters. Thus, this data indicates that neural signals are encoded in a micro-spatial scale and that this is the proper scale at which to space electrodes, such as micro-electrodes.
[0069] Fig. 5 shows an array of spectrograms for speaking the word "thirsty," according to some embodiments. These spectrograms illustrate data from 32 electrodes while the patient said the word "Thirsty." The spectrograms illustrate a complex structure of spectral power in both frequency and time. Fig. 6 also shows an array of spectrograms. However, Fig. 6 illustrates the spectral power when the patient speaks the word "hello," according to some embodiments. This can be contrasted with the structure observed in Fig. 5 when saying "hello" on the next slide. This observation led to the new decode shown in Fig. 11 and discussed further below. In addition, Fig. 7 illustrates a distribution of multi-word performance results for two through ten classifications.
[0070] The tight inter-electrode spacing and small number of electrodes can produce a limited spatial coverage of the micro-electrode grid or array. An optimized grid design with larger spacing and more electrodes would likely cover a larger number of relevant neural signals and allow better decoding accuracy. Performance could likely be further improved with patient training to stereotype word articulation.
[0071] In some embodiments, the invasiveness of the micro-electrode grids or array could be reduced with epidural placement, as shown for similar recording devices.
[0072] In some embodiments, the electrodes can be configured to communicate wirelessly with the system. Thus, whether a micro-electrode grid or other embodiments of an electrode, such components can communicate wirelessly with one or more components of the system. A wireless implementation of the system might be practical given the relatively low bandwidth required to capture cortical surface LFPs. A wireless system to decode attempted and/or actual speech and/or movement, with a balance of invasiveness and performance, could improve the quality of life for locked-in patients and others unable to communicate on their own.
[0073] The above results show that spoken words can be decoded from surface LFPs recorded over neocortical speech areas by arrays of closely spaced micro-electrodes. Therefore, classification of words using surface LFPs can be a viable approach to restoring limited but useful communication to those suffering from locked-in syndrome.
[0074] Methods used to obtain the above results are discussed below.
[0075] Subject and Experiment
[0076] One male patient who required extraoperative electrocorticographic monitoring for medically refractory epilepsy gave informed consent to participate in an institutional review board-approved study. Two nonpenetrating micro-electrode arrays (PMT Neurosurgical, Chanhassen, MN) were implanted over face motor cortex and Wernicke's area. Each array comprised 16 channels of 40-μιη wire terminating in a 4x4 grid with 1- millimeter spacing. For each of 10 words, the patient repeated the word up to 25 times over four consecutive days. Audio data and 32 channels of neural data from the two micro- electrode arrays were recorded at 30,000 samples per second by a Neuroport system
(Blackrock Microsystems, Salt Lake City, UT). A subset of trials containing stereotypical articulation was selected for each word (Supplementary Table 1).
[0077] Data Analysis
[0078] Data were filtered to discard frequencies above 500 Hertz ("Hz") and re- referenced to the common average. Power spectra were computed for 0.5-second windows aligned to vocalization. Log-normalized power spectra for each trial and micro-electrode were concatenated to form a large row vector. All such trial- vectors for each word being classified (two to ten words) were stacked vertically to form a two-dimensional matrix of power spectral data comprising all available channels and trials. Principal component analysis on this data set resulted in clustering, which allowed nearest-centroid classification. Fifteen trials were used for both training and decoding. To keep these trials as temporally proximal as possible, trials from as few adjacent days as possible were used.
[0079] Multi-word performance
[0080] Training and decoding used subsets of channels and combinations of two through ten words. Mean, median, and standard deviation were computed for results of each combination. Combinations were selected using the n-choose-k method (n=10 and k=2-10).
[0081] Topographical performance
[0082] The algorithm was run using data from each electrode individually and for all combinations of two words. Classification accuracies from all combinations involving the selected word and channel were averaged.
[0083] Fig. 8 is block diagram showing an example of a system 450 for recording and processing LFPs from an micro-electrode array 410 that may be used for various embodiments. The system 450 may include a receiver 452, a processor 455, and a memory 460. The receiver 452 may be used to condition the electrical signals from the micro- electrode array 410 for processing by the processor 455. The receiver 452 may include one or more of the following components: amplifiers (e.g., low- noise amplifiers) for amplifying the electrical signals, a filter for isolating electrical signals within a desired frequency bandwidth, and an analog-to-digital converter for digitizing the electrical signals for processing by the processor 455. Some or all of the above components may also be implanted in the patient with the micro-electrode array 410.
[0084] The processor 455 may comprise a general purpose processor, digital signal processors (DSPs), application specific integrated circuit (ASICs), discrete hardware components, or any combination thereof. Methods for decoding speech using neural signals from the array 410 according to various embodiments discussed above may be embodied in software code that is stored in the memory 460 and executed by the processor 455.
[0085] The memory 460 may comprise any computer-readable media known in the art including volatile memory, nonvolatile memory, a Random Access Memory (RAM), a flash memory, a Read Only Memory (ROM), a removable disk, a CD-ROM, a DVD, any other suitable storage device, or a combination thereof. The system 450 can be configured such that the memory 460 may be an on-board component. In addition or alternatively, the system 450 can be configured such that the memory 460 is off -board and system 450 wirelessly communicates with the off -board memory component.
[0086] The processor 455 may also output raw electrical signals, processed electrical signals, and/or results of analysis to an output device 465, including, but not limited to, a display for viewing by a neurologist, a printer for generating a computer readout, a computer-readable media, and/or to another computer via a computer network connection. The output device 465 may also include an audio output device that outputs the decoded word as an audio output, e.g., a synthetic voice vocalizing the decoded word.
[0087] In one embodiment, the processor 455 may decode a word by receiving neural signals, e.g., local field potentials, from the micro-electrode array 410 when the patient speaks the word or attempts to speak the word. The processor 455 may then convert the neural signals into frequency-domain information, e.g., power spectra, for one or more electrodes of the array. The processor 455 may then classify the frequency-domain information for the one or more electrodes into one of a set of words. For example, the processor 455 may perform principal component analysis on the frequency-domain information to project the frequency-domain information into the principal component space and determine its nearest centroid in the principal component space, as described above. After decoding the word that the patient spoke or attempted to speak, the processor 455 may display the decoded word on a display and/or vocalize the decoded word from an audio output device. The processor 455 may be trained to classify a particular word using the methods described above with reference to Figs. 2a- 2d.
[0088] Embodiments of methods for decoding neural signals will now be described according to an aspect of the subject technology.
[0089] Neural Data Acquisition
[0090] Neural data were recorded from 4x4-channel grids of nonpenetrating microwires at 30,000 samples per second (Cerebus, Blackrock Microsystems) while a patient performed speech tasks. During the speech task, the patient repeated a word between 10 and 20 times, with approximately one second separation between trials. Trial start times were noted for later analysis. [0091] Time and Frequency Domain Signal Processing
[0092] Data were low-pass filtered, and the data (^-order elliptic filter with cutoff frequency 2500 Hz) then downsampled to 5,000 samples per second. Data were then re-referenced in software by subtracting the average of all channels from each channel, at each point in time. Segments of data were extracted for each trial beginning 0.5 seconds before the start of the trial and lasting until 0.5 second after the trial start. Multi-tapered estimates of the frequency content in these data segments were estimated in sliding 250-msec windows (step-size 50-msec), with time-bandwidth parameter 5 and 9 leading tapers.
Frequency data were averaged to represent 10-Hz bins, and bins with line noise (60 Hz) or harmonics were deleted.
[0093] Clustering Time and Frequency Doman Data
[0094] To train the classifier for the speech task, a subset of trials was selected from each class. Two matrices were formed such that each row represented a single trial, and each column was an observation, in either time or frequency domains, from one channel in the grid. Separately, these two data sets were z-scored. Then, the features were concatenated, so that each row of the matrix comprised all time and frequency observations across all channels. This large matrix was orthogonalized using principal component analysis (PCA). The training features were projected into the principal component space using a sufficient number of leading principal components to retain 90% of the variance in the data. A second (and subsequent, to adhere to requirements of causality) subset of trials selected from each class was selected as the test set. Data for these trials was z-scored, concatenated, and projected into the principal component space using the previously calculated principal components.
[0095] Discrete Decoding: Classifying Output (Words, Movements)
[0096] As discussed above, discrete decoding can comprise, for example, the classification of neural signals into discrete categories. For example, discrete decoding can be used to classify or identify attempted and/or actual words and/or movements from a subject.
[0097] A variety of classification algorithms may be applied to these time and frequency domain features to decode words, kinematics, and/or attempted kinematics. For example, Fisher' s linear discriminant may be used. The classifier can be trained by finding the mean and covariance for each class in the training set. These parameters are used to determine a projection of the multidimensional features onto a linear direction such that the separation between classes can be maximized and the separation within classes can be minimized. If the data are separable, the trials of a given class can cluster along this new direction. This projection can be applied to each trial in the test set, and the trial can be assigned to a class based on its proximity to the clusters identified during training.
[0098] With a total of N=10 classes, or words, in the set of speech trials, the training and testing were performed for each combination of between k=2 and k=10 words. Where k<N, the number of combinations is given by N!/((N-k) ! k!), the binomial coefficient; otherwise, for k==N there was just one possible combination.
[0099] Continuous Decoding: Proportional Control (Speech, Movement, Position & Force)
[00100] As discussed above, continuous decoding can comprise, for example, the decoding of neural signals to provide information on continuously varying attempted and/or actual movements. For example, continuous decoding can comprise proportional decoding of attempted and/or actual movement of a part of a subject body. Such decoding can provide data on position, velocity, and/or acceleration of movement of the part of the subject's body.
[00101] A variety of decode algorithms may be applied to these time and frequency domain features to provide proportional control. For example, Kalman Filtering may be used. A Kalman Filter has been used to decode hand position from the neural signals recorded on micro-ECoG grids placed over the hand and arm areas of pre-motor cortex in a human patient. If a decoder is trained on other parameters of attempted and/or actual movements, i.e. force, velocity, acceleration, etc., then it can decode these parameters as well as position.
[00102] Kalman Filter Usage
[00103] The following sections provide a specific description of the use of a Kalman Filter to decode hand position.
[00104] Data acquisition
[00105] The tablet and array outputs were recorded with a NeuroPort system
(Blackrock Microsystems, Salt Lake City, UT). During digitization, the signals were band-pass filtered to preserve frequencies between 0.3 Hz and 7.5 KHz. Neural data were recorded at 30,000 samples/sec, and movement data were recorded at either 2,000 samples/sec or 30,000 samples/see. All movement and electrode data were recorded by the same sampling and filtering process simultaneously, ensuring that movement position was well synchronized to ECoG data. [00106] Preprocessing and analysis for directionality
[00107] Raw neural data were first downsampled from 30,000 samples/sec to 3,000 samples/sec. To mitigate the effects of 60-Hz noise, the trial-averaged spectrum for each nonpenetrating microwire was calculated to determine the width and amplitude of the 60-Hz noise band. Across ail nonpenetrating microwires in P2, for example, noise levels in this band ranged from 5 dB through 20 dB above the normal spectrum, meaning that a single filter might effectively attenuate noise in a few channels but would leave large banding in most of the data. Therefore, a fifth-order Chebyshev Type-II filter was designed to individually mirror the calculated width and height of the 60-Hz noise. Next, the data were filtered to remove frequencies below 5 Hz and above 150 Hz using fifth-order Butterworth filters. Filtering was performed in MATLAB (Math Works, Natick, MA), running once in the forward direction and once in the reverse direction, with appropriate initial conditions in the second pass, to ensure zero phase distortion.
[00108] Trial components were marked by points where the velocity of the patient's motion crossed a threshold of one standard deviation from rest. This model was used to mark both outward and return movement; additionally, baseline trials were marked at periods of no movement between trials. Only outward movement trials were included in the present analysis. The ti mestamps of these outward movements were used as markers for reading electrode data. For spectrograms, one second of data, with movement aligned at 0.5 seconds, was used.
[00109] Spectrograms were generated using the Chronux (chronux.org) package with 250 msec windows and 50 msec step size: tapering parameters were set to a time-bandwidth product of five, and nine leading tapers. Spectrum plots of the raw data were characterized by a power law trend whose features dominated frequency analysis. For this reason, the spectrograms were normalized to the trial-averaged spectrum for all trials of like movement.
[00110] Differential power analysis was performed by normalizing the average gamma-band power between movements in the contralateral and ipsilateral directions to the average power in spectrograms from just contralaterally directed movement. First, spectrograms were generated in the same manner described above for data from each microwire. Next, the power in the region between 30 Hz and 80 Hz, for times between -500 msec and 0 msec, was averaged to a single value. Once these averaged powers were obtained for each nonpenetrating microwire (independently calculated for each direction of movement), the values for the contralateral direction were re-referenced against those of the ipsilateral direction, then normalized by division to the contralateral values. In this way, percent change of gamma-band power between contralateral and jpsilateral directions of movement was obtained for each channel in the nonpenetrating microwire array.
[00111] Pairwise cross- correlation analysis between each set of nonpenetrating microwires was performed on neuronal recordings from a period during arm movement to explore the strength of the linear relationship between the signals recorded by nonpenetrating microwires within close proximity. Because the correlation metric indicates the degree to which two sequences are linearly related, this estimation of independence, while not a complete metric, can approximate the possible usefulness of the high spatial-resolution recording afforded by the nonpenetrating microwire devices. In particular, the correlation analysis should help to clarify whether the nonpenetrating microwire array could be replaced by a single large electrode, or if the tighter spacing of the nonpenetrating microwires allows for recording of potentially unique signals.
[00112] Preprocessin and analysis for continuous decode
[00113] Separately, the data were explored to test whether a continuous decode of hand position could be implemen ed. Neural data recorded during task sessions for both P and P2 were lowpass filtered and downsampled to 2 kS/sec. The data were highpass filtered at 1 Hz to attenuate potential confounds with artifacts introduced by the reaching movements. Multi-tapered spectrograms between 0 and 500 Hz were generated using the Chronux package, with time- bandwidth parameter five, nine leading tapers, and nonoveriapping 250 msec windows. A total of 14 sessions were used for PI, and nine sessions for P2.
[00114] To mimic a real-time implementation, the data were further processed so that the decode could operate on continuously presented frames of data. The spectral data were averaged into frequency bins covering 0-5 Hz, 5-13 Hz, 13-30 Hz, 30-80 Hz, 80-200 Hz, and 200- 500 Hz. Ail frequencies within +5 Hz of 60 Hz or its harmonics were removed due to line noise contamination. The movement data were downsampled to the same sampling rate as the spectrograms (4 samples/sec). An offset of 150 msec was introduced between the movement data and the spectral data to model delay between neural activity and motor output. The final feature vector » consisted of 6 frequency bins per channel and the hand state xk was represented by a 6- dimensional vector comprising x and y position, velocity, and acceleration. These vectors were defined for k - 1, 2, ... M, where M was the number of samples in the data set.
[00115] A standard alman filter was implemented to perform the trajectory decode [6], implicitly making the simplifying assumption that a linear relationship existed between xk and ¾,· The likelihood model was defined as:
Figure imgf000020_0001
[00116] where ¾ linearly relates the hand kinematics to the neural features and represents noise in the observation, assumed to be zero-mean and normally distributed with covasiance matrix (¾. Next, the temporal prior was defined to model how the system state, i.e., the hand kinematics, varied over time. This relationship was characterized by Ak, the state transformation matrix, and Wk, the noise term, also assumed to be zero-mean and normally distributed with covariance
= Ak¾ + M-'k
[00117] Again, a linear relationship was assumed in the temporal progression of the hand state.
[00118] With these relationships defined, the decoding process required learning the parameters A, H, W, and Q from the training data, then predicting the hand kinematics at each time given the prior estimate of the state and new measurements of neural data. Neural data recorded during two different task sessions were used for training and testing. The parameters were directly calculated from the training data as described in [6] and were assumed to be constant, e.g., Ak= A. Additionally, the means of the movement and neural data features were calculated in order to center the data, and the data were orthogonalized using PCA, with tailing principal components contributing less than 1% of the variance discarded. The principal components and means found during training were applied to the testing data to stay as close as possible to the real-time case where such information would be unavailable.
[00119] To decode the hand kinematics from the testing data, the prior state estimate was formed, in the time update step, according to
Xk = AXk
[00120] with priori error covariance matrix
P = AP^A* + W
[00121] Next, in the measurement update step, the prior state estimate and new neural data were used to update the state estimate and find the posterior error covariance matrix
Pk = (l - KkH)P-k
Kk = P-k H T (HP-H T + QY1 [00122] The output of the Kalman filter was analyzed by calculating the correlation coefficient between the actual trajectory, recorded during the task session, and the predicted trajectory output from the Kalman filter. To understand which frequencies were important to the Kalman filter, frequency features were averaged into 10 Hz frequency bins between 0 and 500 Hz (from all channels). These narrowband features were tested individually with the Kalman filter, and the correlation coefficients for the x- and y-positions were recorded for each run.
[00123] An algorithm useful for embodiments of the methods and systems disclosed herein may be able to take advantage of the processing of information in neural structures at the micro-scale using features in space, time, and frequency. In some embodiments, the algorithm can correlate the dynamics in local field potential voltage and spectral power (frequency range, e.g., about 0 - 1000 Hz or more) across time with attempted and/or actual movements (speaking, finger flexion and extension, and reaching, etc.). The dynamics in the neural signals across the spatial extent of the grid can also be correlated with movement.
[00124] In accordance with some embodiments, various mathematical
transformations can be applied to the neural data to aid in decoding. In some embodiments, decoding can consist of detection and classification of specific types of attempted and/or actual movements, e.g., saying a specific word or making a specific grasping movement. Further, in some embodiments, decoding can consist of detection and classification of specific types of continuously varying parameters of attempted and/or actual movement, e.g. position or acceleration of one or more body parts. Principal Components Analysis (PCA) can be applied to provide classification of attempted and/or actual movements. Further, as noted herein, Kalman Filtering can be used to continuously decode hand position.
[00125] Decoding Attempted and/or Actual Finger Movement
[00126] A 96 channel micro-ECoG grid was implanted in the epidural space over the primary motor cortex of a nonhuman primate. The nonhuman primate was trained to make individual and patterns of finger movements. The spectrograms of data recorded on most of the 96 channels during different finger movements are shown in Figs. lOa-c. Differences in the spatial and temporal patterns of spectral power are clearly correlated with the different movement types. The resulting data demonstrate that finger movements can be decoded using time and frequency domain features of neural signals recorded on micro-ECoG grids. [00127] More particularly, Fig. 10a shows an array of spectrograms 510 recorded on individual electrodes of the 96 channel micro-ECoG grid when the subject moved an index finger. In accordance with some embodiments of the methods and systems for decoding neural signals disclosed herein, the electrodes can be in contact with a cortical surface of the brain to sense LFPs from the cortical surface. In some embodiments, the electrodes can be in contact with the face motor cortex, premotor cortex, Wernike's area, Broca's area, and/or other areas of the brain. Further, the electrodes can also be spaced from a surface of the brain. Thus, embodiments can provide a desired relative spatial positioning of the electrodes, whether or not one or more of the electrodes is in direct contact with a surface of the brain.
[00128] Further, some embodiments provide methods and systems that can detect signals from portions of the central nervous system and/or peripheral nervous system. For example, although embodiments of the methods and systems can be applied to grids placed on the cerebral cortex, but embodiments of the methods and systems can also be to neural signals recorded from sub-cortical structures and the peripheral nervous system. For example, it is contemplated that the ranges provided for the physical structure of electrodes can equally apply to sub-cortical and peripheral nerve structures.
[00129] Some embodiments of the systems and kits disclosed herein can comprise one or more electrodes. The electrodes can be formed in a grid or array. Some embodiments can use one or more electrodes having a size or surface area of between about 0.0001 mm2 to about 1 mm2. Further, in some embodiments, the electrode can have a size or surface area of between about 0.0025 mm2 to about 0.50 mm2. Moreover, some embodiments can use one or more electrodes having a diameter of between about 0.01 mm to about 1 mm. Further, in some embodiments, the electrode can have a diameter of between about 0.05 mm to about 0.50 mm.
[00130] In accordance with some embodiments, the electrodes can define a desired impedance. For example, for electrodes having a frequency in the range of about 1 Hz to about 1 kHz, the electrodes can have impedances in the range of about 10 kQ To about 1 ΜΩ. Further, the electrodes can have impedances within a range of about 100 kQ to about 500 kQ
[00131] Further, the total number of electrodes in an array or grid can be between about 4 and about 65536. An array can be provided having between about 4 to about 1000 microelectrodes or more. In many implementations, an array can have between about 4 to about 200 microelectrodes or more. For example, an array can have 16, 32, or 64 microelectrodes arranged in a grid. Further, in some embodiments, the grid can be arranged in a square.
[00132] In some embodiments using a grid or array, the inter-electrode spacing can be between about 100 microns and about 10000 microns. In some embodiments, inter- electrode spacing can be between about 0.05 mm to about 5mm. In some embodiments, optimal spacing of the microelectrodes can be between about 2 mm to about 3 mm apart.
[00133] The sampling rate can be between 0 and about 2500 Hz to provide a desired temporal resolution. In some embodiments, the sampling rate can be between about 10 and about 500 Hz.
[00134] In addition, some embodiments can use one or more electrodes can have a thickness of between about 0.010 mm and about 5 mm. Some embodiments can be configured such that the electrodes define a thickness of between about 0.020 mm and about 3 mm. For example, some embodiments can use electrodes having a thickness of between about 0.050 mm and about 0.500 mm.
[00135] As shown in the exemplary embodiments of Fig. 9a and 9b, an electrode grid can be provided. In some embodiments, the grid can comprise one or more small protuberances that elevate an active electrode site above a base plane of the grid. Fig. 9a illustrates a grid 500 having a plurality of protuberances 502 having a generally frustro- conical shape, attached to a base plane 504 of the grid 500. Fig. 9b illustrates a grid 600 having a plurality of protuberances 602 having a generally frustro-conical shape, attached to a base plane 604 of the grid 600. These protuberances can aid in making electrical contact the neural tissue and minimizing any shunting of electrical signal through bodily fluids, e.g., cerebrospinal fluid. These protuberance can be of various geometries, such as semi-spherical, cylindrical, conical, etc.
[00136] In some embodiments, the electrical ground and reference for the electrodes can consist of low-impedance electrodes that are in contact with the fluid space in which the electrodes are implanted, e.g., the sub-dural cerebrospinal fluid. In other embodiments, ground and reference can be placed either the epidural space or the surface of the scalp. The electrical ground and reference may be placed together in any of these spaces, or could be placed separately in any combination of these spaces.
[00137] In accordance with some embodiments, the electrodes can comprise microelectrodes, microwire arrays and/or other equipment for recording LFPs from the cortical surface of the brain to provide a desired spatial resolution. The electrodes can define one or more unique sizes. The electrodes can be round, square, and/or have serrated edges. In some embodiments, the electrodes can comprise microelectrodes that are formed in grids or arrays. For example, the array of microelectrodes can be embedded in a polymer or other material that allows the grid to be flexible and achieve a micro-thickness. For example, a rubbery clear silicone or other such materials can be acceptable for some uses and may be used in some embodiments. Further, other polymers can be acceptable for some uses and may include polyurethane and poly-imide.
[00138] Referring again to Figure 9 a, each spectrogram 510 shows measured power as a function of frequency and time for the corresponding electrode of the micro-ECoG grid, where the vertical axis represents frequency and the horizontal axis represents time. In the example in Fig. 10a, each spectrogram 510 shows measured power over a frequency range of 0 to 500 Hz and a time range of between approximately between one second before and one second after the finger movement. Collectively, the array of spectrograms 510 show time and frequency domain data at different spatial positions on the brain. Thus, the array of spectrograms 510 represent time, frequency and spatial information that can be used to decode attempted and/or actual finger movement.
[00139] Fig. 10b shows an array of spectrograms 510 recorded on individual electrodes of the 96 channel micro-ECoG grid when the subject moved a middle finger and Fig. 10c shows an array of spectrograms 510 recorded on individual electrodes of the 96 channel micro-ECoG grid when the subject moved a thumb, where the index finger, middle finger and thumb were from the same hand. Figs. 10a, 10b and 10c clearly show different spatial and temporal patterns of spectral power correlated with movement of different fingers of the subject's hand. Each spectrogram 510 shows a temporal pattern of spectral power for the corresponding electrode and the array of spectrograms shows a spatial pattern of the spectral power across the electrodes of the micro-ECoG grid.
[00140] Decoding of neural signals may be performed using principal components analysis (PCA) and Kalman filtering on the micro-ECoG signals. For example, the features (spatial and temporal pattern of spectral power) for attempted and/or actual finger movement may be refined through PCA into a principal component space and classification may be performed in the principal component space (e.g., nearest- neighbor clustering or other classification algorithm) to classify (decode) the attempted and/or actual finger movement. Examples of classification algorithms that may be used include k-nearest neighbors, Bayesian classification, Hidden Markov models (HMM) and learning vector quantization (LVQ). Other algorithms that may be used include linear regression, Fisher's linear discriminants and support vector machines (SVM).
[00141] Fig. 11 illustrates a method for decoding neural signals according to some embodiments. In Fig. 11, data is received from M channels, where each channel corresponds to an electrode of an electrode array and each electrode receives neural signals emanating from a different spatial position of the brain. For each channel, spectral power (e.g., power at different frequency components) over time may be measured (represented by a spectrogram). The spectral power over time (spectral power at different times) provides information about the temporal dynamics of the spectral power. For each channel, time domain data of the corresponding signal may also be obtained, for example, voltage of the signal over time (voltage at different times), as shown in Fig. 11. This time domain data provides additional information that can be used to improve decoding performance.
[00142] For each channel, the spectral power (e.g., power at different frequency components) over time, fi(t) . . . f (t), and the time domain data, v(t), (e.g., voltage over time) provide features for the channel that may be used in neural signal decoding. The features for the different channels correspond to different spatial positions of the brain and therefore provide spatial information (a spatial pattern).
[00143] During a training phase, feature information may be collected across the channels for each of a plurality of trials, where each trial may correspond to attempted and/or actual movement of a particular finger or an attempt to speak a particular word. Fig. 11 shows a two-dimensional matrix 620 in which each row represents a different trial and each column represents features for a particular channel.
[00144] As shown in Fig. 11 , the features for each trial may be refined through principal component analysis (PCA) into features in a principal component space 630. Fig. 11 shows an example in which trials for the words "hungry" and "thirsty" are mapped to the principal component space 630. In Fig. 11, three dimensions of the principal component space are shown for ease of illustration, although it is to be understood that the principal component space may comprise any number of dimensions. In this example, a centroid may be calculated for each word or type of attempted and/or actual finger movement to be classified (decoded). [00145] During the classification phase, when a patient attempts and/or actually speaks a word and/or moves and/or attempts to move a particular finger, the attempted and/or actual word or finger movement may be classified by performing principal component analysis on the corresponding features from the M channels (e.g., fi(t) . . . fN(t) and v(t)) to refine the features into the principal component space 630 and determining its nearest centroid in the principle component space 630. The word that the patient spoke and/or attempted to speak and/or the attempted and/or actual finger movement can be then classified (decoded) based on the attempted and/or actual word and/or finger movement corresponding to the nearest centroid. Other classification algorithms may also be used.
[00146] Although, the method was described using the example of finger movement decoding, those skilled in the art will appreciate that the method may be used to decode attempted and/or actual movements of other parts of the body (e.g., toe movement, arm movement, etc.). The method may also be used to decode one or more characteristics of attempted and/or actual finger movement such as position of the finger, force exerted by the finger, acceleration of the finger.
[00147] For example, Figs. 12a and 12b illustrate the correlation of LFPs with kinematics, and in particular, with reaching movements. Figs. 12a and 12b show maps that compare hand position measurements against decoded local field potentials.
[00148] As discussed herein, LFPs can be used to perform proportional decoding and/or classification of attempted and/or actual movement. For the experiment illustrated in Figs. 12a and 12b, a Kalman filter was used to map frequency and time domain features onto the x-position, y-position, velocity, and acceleration of the hand. Figs. 12a and 12b show the position of the hand measure with a mouse (shown in blue) and the position decoding from the LFP (shown in red). While the position decoding from the LFP demonstrates a generally larger amplitude in both maps, there is a high degree of correlation between the position of the hand measure with a mouse and the position decoding from the LFP. Accordingly, embodiments of the methods and systems disclosed herein can be used to decode kinematics of the body. Further, some embodiments can be used to decode kinematics of one or more parts of the body individually and/or together with other parts of the body.
[00149] It will be also appreciated by persons skilled in the art that numerous variations and/or modifications may be made to the specific embodiments disclosed herein, without departing from the scope or spirit of the disclosure as broadly described. The present embodiments are, therefore, to be considered in all respects illustrative and not restrictive of the present inventions.

Claims

What is claimed is:
1. A system for decoding neural signals, comprising: a receiver configured to receive a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body; and a processor configured to convert the neural signals into frequency and time domain information and to apply a classifier to the frequency and time domain information so as to determine the part of the body.
2. The system of claim 1 , wherein the electrodes contact a cortical surface of the brain.
3. The system of claim 2, wherein the electrodes contact a face motor cortex.
4. The system of claim 2, wherein each neural signal comprises a local field potential from the cortical surface.
5. The system of claim 1, wherein the electrodes comprise micro-electrodes.
6. The system of claim 1, wherein the frequency and time domain information for each neural signal comprises spectral power of the neural signal at different times.
7. The system of claim 6, wherein the frequency and time domain information for each neural signal also comprises voltage of the neural signal at different times.
8. The system of claim 6, wherein the spectral power for each neural signal comprises power at different frequency components.
9. A kit for decoding neural signals, the kit comprising: a plurality of electrodes being configured to detect neural signals corresponding to a different spatial position of a brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body; a receiver configured to receive a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body; and a processor configured to convert the neural signals into frequency and time domain information and to apply a classifier to the frequency and time domain information so as to determine the part of the body.
10. A method for decoding neural signals, comprising: receiving a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body; converting the neural signals into frequency and time domain information; and applying a classifier to the frequency and time domain information so as to determine the part of the body.
11. The method of claim 10, wherein the frequency and time domain information for each neural signal comprises spectral power of the neural signal at different times.
12. The method of claim 11, wherein the frequency and time domain information for each neural signal also comprises voltage of the neural signal at different times.
13. The method of claim 11, wherein the spectral power for each neural signal comprises power at different frequency components.
14. A non-transitory machine-readable medium comprising instructions stored therein, which when executed by a machine, cause the machine to perform operations for decoding neural signals, the operations comprising: receiving a neural signal from each of a plurality of electrodes, wherein each neural signal corresponds to a different spatial position of the brain of a patient, each neural signal emanated from the brain when the patient moves and/or attempts to move a part of the body; converting the neural signals into frequency and time domain information; and applying a classifier to the frequency and time domain information so as to determine the part of the body.
15. The machine-readable medium of claim 14, wherein the electrodes contact a cortical surface of the brain.
16. The machine-readable medium of claim 15, wherein the electrodes contact a face motor cortex.
17. The machine-readable medium of claim 16, wherein each neural signal comprises a local field potential from the cortical surface.
18. The machine-readable medium of claim 14, wherein the electrodes comprise micro-electrodes.
19. The machine-readable medium of claim 14, wherein the frequency and time domain information for each neural signal comprises spectral power of the neural signal at different times.
20. The machine-readable medium of claim 19, wherein the frequency and time domain information for each neural signal also comprises voltage of the neural signal at different times.
21. The machine-readable medium of claim 18, wherein the spectral power for each neural signal comprises power at different frequency components.
PCT/US2012/026400 2011-02-23 2012-02-23 Systems and methods for decoding neural signals WO2012116232A1 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
US201161463797P 2011-02-23 2011-02-23
US61/463,797 2011-02-23
US201161546514P 2011-10-12 2011-10-12
US61/546,514 2011-10-12

Publications (1)

Publication Number Publication Date
WO2012116232A1 true WO2012116232A1 (en) 2012-08-30

Family

ID=46721237

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2012/026400 WO2012116232A1 (en) 2011-02-23 2012-02-23 Systems and methods for decoding neural signals

Country Status (1)

Country Link
WO (1) WO2012116232A1 (en)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014130571A1 (en) 2013-02-19 2014-08-28 The Regents Of The University Of California Methods of decoding speech from the brain and systems for practicing the same
US10653330B2 (en) 2016-08-25 2020-05-19 Paradromics, Inc. System and methods for processing neural signals
WO2022011260A1 (en) * 2020-07-09 2022-01-13 Thomas Jefferson University Systems and methods for motor function facilitation
CN116491960A (en) * 2023-06-28 2023-07-28 南昌大学第一附属医院 Brain transient monitoring device, electronic device, and storage medium

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040249302A1 (en) * 2003-06-09 2004-12-09 Cyberkinetics, Inc. Methods and systems for processing of brain signals
US20070179396A1 (en) * 2005-09-12 2007-08-02 Emotiv Systems Pty Ltd Method and System for Detecting and Classifying Facial Muscle Movements
US7826894B2 (en) * 2004-03-22 2010-11-02 California Institute Of Technology Cognitive control signals for neural prosthetics

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040249302A1 (en) * 2003-06-09 2004-12-09 Cyberkinetics, Inc. Methods and systems for processing of brain signals
US7826894B2 (en) * 2004-03-22 2010-11-02 California Institute Of Technology Cognitive control signals for neural prosthetics
US20070179396A1 (en) * 2005-09-12 2007-08-02 Emotiv Systems Pty Ltd Method and System for Detecting and Classifying Facial Muscle Movements

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014130571A1 (en) 2013-02-19 2014-08-28 The Regents Of The University Of California Methods of decoding speech from the brain and systems for practicing the same
EP2959280A4 (en) * 2013-02-19 2016-10-19 Univ California Methods of decoding speech from the brain and systems for practicing the same
US9905239B2 (en) 2013-02-19 2018-02-27 The Regents Of The University Of California Methods of decoding speech from the brain and systems for practicing the same
US10438603B2 (en) 2013-02-19 2019-10-08 The Regents Of The University Of California Methods of decoding speech from the brain and systems for practicing the same
US10653330B2 (en) 2016-08-25 2020-05-19 Paradromics, Inc. System and methods for processing neural signals
WO2022011260A1 (en) * 2020-07-09 2022-01-13 Thomas Jefferson University Systems and methods for motor function facilitation
CN116491960A (en) * 2023-06-28 2023-07-28 南昌大学第一附属医院 Brain transient monitoring device, electronic device, and storage medium
CN116491960B (en) * 2023-06-28 2023-09-19 南昌大学第一附属医院 Brain transient monitoring device, electronic device, and storage medium

Similar Documents

Publication Publication Date Title
Bundy et al. Decoding three-dimensional reaching movements using electrocorticographic signals in humans
Turnip et al. Real-time feature extraction of P300 component using adaptive nonlinear principal component analysis
Liao et al. Decoding individual finger movements from one hand using human EEG signals
Kaiju et al. High spatiotemporal resolution ECoG recording of somatosensory evoked potentials with flexible micro-electrode arrays
Vallabhaneni et al. Brain—computer interface
Blakely et al. Localization and classification of phonemes using high spatial resolution electrocorticography (ECoG) grids
Gage et al. Naive coadaptive cortical control
Velu et al. Single-trial classification of gait and point movement preparation from human EEG
Paek et al. Decoding repetitive finger movements with brain activity acquired via non-invasive electroencephalography
US11647962B2 (en) System and method for classifying and modulating brain behavioral states
Jiang et al. Characterization and decoding the spatial patterns of hand extension/flexion using high-density ECoG
Goen et al. Review of surface electromyogram signals: its analysis and applications
WO2012116232A1 (en) Systems and methods for decoding neural signals
Naik et al. Kurtosis and negentropy investigation of myo electric signals during different MVCs
Pailla et al. ECoG data analyses to inform closed-loop BCI experiments for speech-based prosthetic applications
Khatun et al. Single channel EEG time-frequency features to detect mild cognitive impairment
Gorzkowski et al. Exploitation of EMG signals for video game control
WO2011127483A1 (en) Decoding words using neural signals
US20210015419A1 (en) Electrode patch, system, and method for detecting indicator of parkinson&#39;s disease in person
US10478086B2 (en) Measurement of cortical excitability
Guger et al. Electrocorticogram based brain–computer interfaces
Vahidi Tools to investigate composite receptive fields in songbird auditory region
Ravikumar A 3D-printed Fat-I BC-enabled prosthetic arm: Control based on brain neuronal data
Mhapankar et al. A Spectrotemporal EEG Mapping Approach for Decoding Imagined Marathi Language Phonemes
Patil Human Machine Interfaces Using Multichannel Physiological Signals

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12749186

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12749186

Country of ref document: EP

Kind code of ref document: A1