Dean Krusienski, Ph.D.

EAGER: Investigating the Neural Correlates of Musical Rhythms from Intracranial Recordings

NSF

2014-09-01

The project will develop an offline and then a real-time brain computer interface to detect rhythms that are imagined in people's heads, and translate these rhythms into actual sound. The project builds upon research breakthroughs in electrocorticographic (ECoG) recording technology to convert music that is imagined into synthesized sound. The project researchers will recruit from a specialized group of people for this project, specifically patients with intractable epilepsy who are currently undergoing clinical evaluation of their condition at the Mayo Clinic in Jacksonville, Florida, and are thus uniquely prepared to use brain-computer interfaces based on ECoG recording techniques. This is a highly multidisciplinary project that will make progress towards developing a "brain music synthesizer" which could have a significant impact in the neuroscience and musical domains, and lead to creative outlets and alternative communication devices and thus life improvements for people with severe disabilities.

Most brain-computer interfaces (BCIs) use surface-recorded electrophysiological measurements such as surface-recorded electroencephalogram (EEG). However, while some useful signals can be extracted from such surface techniques, it is nearly impossible to accurately decode from such signals the intricate brain activity involved in activities such as language with the detail needed to achieve a natural, transparent translation of thought to device control. On the contrary, intracranial electrodes such as ECoG are closer to the source of the desired brain activity, and can produce signals that, compared to surface techniques, have superior spatial and spectral characteristics and signal-to-noise ratios. Research has already shown that intracranial signals can provide superior decoding capabilities for motor and language signals, and for BCI control. Because complex language and auditory signals (both perceived and imagined) have been decoded using intracranial activity, it is conceivable to decode perceived and imagined musical content from intracranial signals. This project will attempt to similarly use ECoG to decode perceived and imagined musical content from intracranial signals as has been done for language and auditory signals.

CHS: Small: A Hybrid Brain-Computer Interface for Behaviorally Non-Responsive Patients

NSF

2014-08-01

Brain-computer interfaces (BCIs) have been explored for several years in an effort to provide communication for "locked in" users who have the desire and mental capacity to communicate but are unable to speak, type, or use conventional assistive technologies due to severe motor disabilities, and a lot of work has gone into making this initially crude technology more practical, usable, accurate, and flexible (e.g., by improving speed of performance and providing virtual reality feedback and/or advanced device control). In this project the PI and his team turn their attention to a group with even greater need: patients who have been misdiagnosed as vegetative or minimally conscious, and without the mental ability to form messages or respond to questions. These individuals are not only unable to move but also unable to see, and are at risk of being euthanized based on the mistaken assumption that they are effectively "brain dead" whereas it has been shown by European colleagues, using methods and equipment comparable to American hospitals, that 17-42% of such patients were in fact able to use a BCI to respond to questions. The PI worries that severely injured veterans and others might sometimes be misdiagnosed, potentially able to communicate with friends and loved ones if only some technology could more effectively assess their brain activity. The PI's goal in this project is to extend current BCI technologies to focus on assessing consciousness in this vulnerable patient population and provide, where possible, the ability to communicate.

The PI's approach is to adapt and extend conventional BCI protocols and feedback environments to work with people who cannot see and must instead rely on other modalities of stimulation. The work will involve three thrusts. First, the PI and his team will improve methods to identify brain response based only on tactile and auditory stimuli, by determining the best tactile stimulation frequency for each subject. Second, they will use "hybrid" BCIs combining P300s and steady state somatosensory evoked potentials (SSSEPs) to elicit two different kinds of EEG signals that could improve accuracy. Finally, they will develop a new six choice BCI system tailored for these users; at present the best BCIs for these patient groups allow just two or three choices, whereas a six choice system could lead to faster communication and broader control options. Across all three of these thrusts, the team will also explore signal processing m

Classifier Development forSteady-State Visually Evoked Potential Data

NIA/NASA

2015-09-01

Classifier Development forSteady-State Visually Evoked Potential Data

MRI Consortium: Acquisition of an Integrated System of Instruments for Multichannel Biopotential Recording of In-vitro and In-vivo Experiments

NSF

2013-09-01

An award is made to the Norfolk State University (NSU) to build an integrative research platform for neural recording and analysis and provide the resources for students, faculty and associated collaborators for their research activities from electronic and optical engineering, biology, computer science, nanotechnology, immunology, and neuroscience fields. The addition of this system to the existing research infrastructures will directly benefit and upgrade investigator's research programs at NSU, Eastern Virginia Medical School and Old Dominion University, as well as the collaborative network capabilities in the Hampton Roads area. Applying the multichannel biopotential recording system, neural and cardiac potentials, and neurochemicals such as neurotransmitters can be measured in-vivo and in-vitro, and analyzed by embedded software. The instrument will be applied for the study of neural mechanisms underlying the interactions under various behavioral, stimuli, and disease conditions. In addition to the biopotential recording, the instrumentation will allow in-vivo assessment of neural sensing devices to be developed by engineering research. Furthermore, in-vivo and in-vitro real-time measurement and monitoring of sensing signal can be applied for the development of novel therapeutic treatment strategies of neurological disorders and degeneration.

The outcomes of this synergetic research will be positively transferred to achieve their STEM educational and outreach goals, especially focused for under-represented minority (URM) students in the project?s leading institution, NSU, one of the largest Historically Black Colleges and Universities in the U.S. This project will specifically place a high priority on training of URM students in Hampton Roads region and extend their efforts to URM students nationwide through an aggressive set of research, training and outreach activities utilizing current outreach programs on each campus. The outcomes of this project will be incorporated in their Bioengineering Minor program curriculum. Furthermore, they will also integrate this infrastructure with activity of the newly created Hampton Roads Neuroscience Network.

HCC: Medium: RUI: Control of a Robotic Manipulator via a Brain-Computer Interface

NSF

2009-07-15

A brain-computer interface (BCI) is a system that allows users, especially individuals with severe neuromuscular disorders, to communicate and control devices using their brain waves. There are over two million people in the United States afflicted by such disorders, many of whom could greatly benefit from assistive devices controlled by a BCI. Over the past two years, it has been demonstrated that a non-invasive, scalp-recorded electroencephalography (EEG) based BCI paradigm can be used by a disabled individual for long-term, reliable control of a personal computer. This BCI paradigm allows users to select from a set of symbols presented in a flashing visual matrix by classifying the resulting evoked brain responses. One of the goals of this project is to establish that the same BCI paradigm and techniques used for the aforementioned demonstration can be straightforwardly implemented to generate high-level commands for controlling a robotic manipulator in three dimensions according to user intent, and that such a BCI can provide superior dimensional control over alternative BCI techniques currently available, as well as a wider variety of practical functions for performing everyday tasks.

Electrocorticography (ECoG), electrical activity recorded directly from the surface of the brain, has been demonstrated in recent preliminary work to be another potentially viable control for a BCI. ECoG has been shown to have superior signal-to-noise ratio, and spatial and spectral characteristics, compared to EEG. But the EEG signals used at present to operate BCIs have not been characterized in ECoG. The PI believes ECoG signals can be used to improve the speed and accuracy of BCI applications, including for example control of a robotic manipulator. Thus, additional goals of this project are to characterize evoked responses obtained from ECoG, to use them as control signals to operate a simulated robotic manipulator, and to assess the level of control (speed and accuracy) between the two recording modalities and compare the results to competitive BCI techniques. Because this is a collaborative effort with the Departments of Neurology and Neurosurgery at the Mayo Clinic in Jacksonville, the PI team will have access to a pool of ECoG grid patients from which to recruit participants for this study.

Broader Impacts: This research will make a number of contributions in the emerging field of BCI and thus will serve as a step toward providing severely disabled individuals

Generating Natural, Intelligible Speech From Brain Activity in Motor, Premotor, and Inferior Frontal Cortices

C Herff, L Diener, M Angrick, E Mugler, M Tate, M Goldrick, DJ Krusienski, M Slutzky, T Schultz

2019-11-22

Neural interfaces that directly produce intelligible speech from brain activity would allow people with severe impairment from neurological disorders to communicate more naturally. Here, we record neural population activity in motor, premotor and inferior frontal cortices during speech production using electrocorticography (ECoG) and show that ECoG signals alone can be used to generate intelligible speech output that can preserve conversational cues. To produce speech directly from neural data, we adapted a method from the field of speech synthesis called unit selection, in which units of speech are concatenated to form audible output. In our approach, which we call Brain-To-Speech, we chose subsequent units of speech based on the measured ECoG activity to generate audio waveforms directly from the neural recordings. Brain-To-Speech employed the user's own voice to generate speech that sounded very natural and included features such as prosody and accentuation. By investigating the brain areas involved in speech production separately, we found that speech motor cortex provided more information for the reconstruction process than the other cortical areas.

View more

Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

C Tremmel, C Herff, T Sato, K Rechowicz,Y Yamani, DJ Krusienski. Estimating Cognitive Workload in an Interactive VR Environment using EEG

2019-11-14

With the recent surge of affordable, high-performance virtual reality (VR) headsets, there is unlimited potential for applications ranging from education, to training, to entertainment, to fitness and beyond. As these interfaces continue to evolve, passive user-state monitoring can play a key role in expanding the immersive VR experience, and tracking activity for user well-being. By recording physiological signals such as the electroencephalogram (EEG) during use of a VR device, the user's interactions in the virtual environment could be adapted in real-time based on the user's cognitive state. Current VR headsets provide a logical, convenient, and unobtrusive framework for mounting EEG sensors. The present study evaluates the feasibility of passively monitoring cognitive workload via EEG while performing a classical n-back task in an interactive VR environment. Data were collected from 15 participants and the spatio-spectral EEG features were analyzed with respect to task performance. The results indicate that scalp measurements of electrical activity can effectively discriminate three workload levels, even after suppression of a co-varying high-frequency activity.

View more

Speech synthesis from ECoG using densely connected 3D convolutional neural networks

M Angrick, C Herff, E Mugler, M Tate, M Slutzky, DJ Krusienski, T Schultz

2019-04-16

Objective. Direct synthesis of speech from neural signals could provide a fast and natural way of communication to people with neurological diseases. Invasively-measured brain activity (electrocorticography; ECoG) supplies the necessary temporal and spatial resolution to decode fast and complex processes such as speech production. A number of impressive advances in speech decoding using neural signals have been achieved in recent years, but the complex dynamics are still not fully understood. However, it is unlikely that simple linear models can capture the relation between neural activity and continuous spoken speech. Approach. Here we show that deep neural networks can be used to map ECoG from speech production areas onto an intermediate representation of speech (logMel spectrogram). The proposed method uses a densely connected convolutional neural network topology which is well-suited to work with the small amount of data available from each participant. Main results. In a study with six participants, we achieved correlations up to r  =  0.69 between the reconstructed and original logMel spectrograms. We transferred our prediction back into an audible waveform by applying a Wavenet vocoder. The vocoder was conditioned on logMel features that harnessed a much larger, pre-existing data corpus to provide the most natural acoustic output. Significance. To the best of our knowledge, this is the first time that high-quality speech has been reconstructed from neural recordings during speech production using deep neural networks.

View more

Interpretation of convolutional neural networks for speech spectrogram regression from intracranial recordings

M Angrick, C Herff, G Johnson, J Shih, DJ Krusienski, T Schultz

2018-05-19

The direct synthesis of continuously spoken speech from neural activity could provide a fast and natural way of communication for users suffering from speech disorders. Mapping the complex dynamics of neural activity to spectral representations of speech is a demanding task for regression models. Convolutional neural networks have recently shown promise for finding patterns in neural signals and might be a good candidate for this particular regression task. However, the intrinsic agency of the resulting networks is challenging to interpret and thus provides little opportunity to gain insights on neural processes underlying speech. While activation maximization can be used to get a glimpse into what a network has learned for a classification task, it usually does not benefit regression problems.

Here, we show that convolutional neural networks can be used to reconstruct an audible waveform from invasively-measured brain activity. By adapting activation maximization, we present a method that can provide insights from neural networks targeting regression problems. Based on experimental data, we achieve statistically significant correlations between spectrograms of synthesized and original speech. Our interpretation approach shows that trained models reveal that electrodes placed in cortical regions associated with speech production tasks have a large impact on the reconstruction of speech segments.

View more

Biosignal-based Spoken Communication: A Survey

T Schultz, M Wand,T Hueber, DJ Krusienski, C Herff, JS Brumberg

Speech is a complex process involving a wide range of biosignals, including but not limited to acoustics. These biosignals-stemming from the articulators, the articulator muscle activities, the neural pathways, and the brain itself-can be used to circumvent limitations of conventional speech processing in particular, and to gain insights into the process of speech production in general. Research on biosignal-based speech processing is a wide and very active field at the intersection of various disciplines, ranging from engineering, computer science, electronics and machine learning to medicine, neuroscience, physiology, and psychology. Consequently, a variety of methods and approaches have been used to investigate the common goal of creating biosignal-based speech processing devices for communication applications in everyday situations and for speech rehabilitation, as well as gaining a deeper understanding of spoken communication. This paper gives an overview of the various modalities, research approaches, and objectives for biosignal-based spoken communication.

Optimization of Checkerboard Spatial Frequencies for Steady-State Visual Evoked Potential Brain-Computer Interfaces

NR Waytowich, Y Yamani, DJ Krusienski

Steady-state visual evoked potentials (SSVEPs) are oscillations of the electroencephalogram (EEG) which are mainly observed over the occipital area that exhibit a frequency corresponding to a repetitively flashing visual stimulus. SSVEPs have proven to be very consistent and reliable signals for rapid EEG-based brain-computer interface (BCI) control. There is conflicting evidence regarding whether solid or checkerboard-patterned flashing stimuli produce superior BCI performance. Furthermore, the spatial frequency of checkerboard stimuli can be varied for optimal performance. The present study performs an empirical evaluation of performance for a 4-class SSVEP-based BCI when the spatial frequency of the individual checkerboard stimuli is varied over a continuum ranging from a solid background to single-pixel checkerboard patterns. The results indicate that a spatial frequency of 2.4 cycles per degree can maximize the information transfer rate with a reduction in subjective visual irritation compared to lower spatial frequencies. This important finding on stimulus design can lead to improved performance and usability of SSVEP-based BCIs.

Spatio-Temporal Progression of Cortical Activity Related to Continuous Overt and Covert Speech Production in a Reading Task

JS Brumberg, DJ Krusienski, S Chakrabarti, A Gunduz, P Brunner, AL Ritaccio, G Schalk

2016-11-22

How the human brain plans, executes, and monitors continuous and fluent speech has remained largely elusive. For example, previous research has defined the cortical locations most important for different aspects of speech function, but has not yet yielded a definition of the temporal progression of involvement of those locations as speech progresses either overtly or covertly. In this paper, we uncovered the spatio-temporal evolution of neuronal population-level activity related to continuous overt speech, and identified those locations that shared activity characteristics across overt and covert speech. Specifically, we asked subjects to repeat continuous sentences aloud or silently while we recorded electrical signals directly from the surface of the brain (electrocorticography (ECoG)). We then determined the relationship between cortical activity and speech output across different areas of cortex and at sub-second timescales. The results highlight a spatio-temporal progression of cortical involvement in the continuous speech process that initiates utterances in frontal-motor areas and ends with the monitoring of auditory feedback in superior temporal gyrus. Direct comparison of cortical activity related to overt versus covert conditions revealed a common network of brain regions involved in speech that may implement orthographic and phonological processing. Our results provide one of the first characterizations of the spatiotemporal electrophysiological representations of the continuous speech process, and also highlight the common neural substrate of overt and covert speech. These results thereby contribute to a refined understanding of speech functions in the human brain

View more