Brain-to-text: Decoding spoken sentences from phone representations in the brain. - Herff, C.; Heger, D.; Pesters, Adriana de; Telaar, D.; Peter Brunner; Gerwin Schalk; Schultz, T. (Journal of Neural Engineering, 2015)
It has long been speculated whether communication between humans and machines based on natural speech related cortical activity is possible. Over the past decade, studies have suggested that it is feasible to recognize isolated aspects of speech from neural signals, such as auditory features, phones or one of a few isolated words. However, until now it remained an unsolved challenge to decode continuously spoken speech from the neural substrate associated with speech and language processing. Here, we show for the first time that continuously spoken speech can be decoded into the expressed words from intracranial electrocorticographic (ECoG) recordings.Specifically, we implemented a system, which we call Brain-To-Text that models single phones, employs techniques from automatic speech recognition (ASR), and thereby transforms brain activity while speaking into the corresponding textual representation. Our results demonstrate that our system can achieve word error rates as low as 25% and phone error rates below 50%. Additionally, our approach contributes to the current understanding of the neural basis of continuous speech production by identifying those cortical regions that hold substantial information about individual phones. In conclusion, the Brain-To- Text system described in this paper represents an important step toward human-machine communication based on imagined speech.
Keywords: automatic speech recognition; brain-computer interface; broadband gamma; ECoG; Electrocorticography; pattern recognition; speech decoding; speech production

A general method for assessing brain–computer interface performance and its limitations. - Jeremy Jeremy Hill; Häuser, Ann-Katrin; Gerwin Schalk (Journal of Neural Engineering, 2014)
Objective. When researchers evaluate brain–computer interface (BCI) systems, we want quantitative answers to questions such as: How good is the system's performance? How good does it need to be? and: Is it capable of reaching the desired level in future? In response to the current lack of objective, quantitative, study-independent approaches, we introduce methods that help to address such questions. We identified three challenges: (I) the need for efficient measurement techniques that adapt rapidly and reliably to capture a wide range of performance levels; (II) the need to express results in a way that allows comparison between similar but non-identical tasks; (III) the need to measure the extent to which certain components of a BCI system (e.g. the signal processing pipeline) not only support BCI performance, but also potentially restrict the maximum level it can reach. Approach. For challenge (I), we developed an automatic staircase method that adjusted task difficulty adaptively along a single abstract axis. For challenge (II), we used the rate of information gain between two Bernoulli distributions: one reflecting the observed success rate, the other reflecting chance performance estimated by a matched random-walk method. This measure includes Wolpaw's information transfer rate as a special case, but addresses the latter's limitations including its restriction to item-selection tasks. To validate our approach and address challenge (III), we compared four healthy subjects' performance using an EEG-based BCI, a 'Direct Controller' (a high-performance hardware input device), and a 'Pseudo-BCI Controller' (the same input device, but with control signals processed by the BCI signal processing pipeline). Main results. Our results confirm the repeatability and validity of our measures, and indicate that our BCI signal processing pipeline reduced attainable performance by about 33% (21 bits/min). Significance. Our approach provides a flexible basis for evaluating BCI performance and its limitations, across a wide range of tasks and task difficulties.
Keywords: brain-computer interface; information gain; information transfer rate; Neuroprosthetics; performance evaluation

Proceedings of the Fifth International Workshop on Advances in Electrocorticography. - A L Ritaccio; Peter Brunner; Gunduz, Aysegul; Hermes, Dora; Hirsch, Lawrence J; Jacobs, Joshua; Kamada, Kyousuke; Kastner, Sabine; Robert T. Knight; Lesser, Ronald P; Miller, Kai; Sejnowski, Terrence; et al. (Epilepsy Behav, 2014)
The Fifth International Workshop on Advances in Electrocorticography convened in San Diego, CA, on November 7-8, 2013. Advancements in methodology, implementation, and commercialization across both research and in the interval year since the last workshop were the focus of the gathering. Electrocorticography (ECoG) is now firmly established as a preferred signal source for advanced research in functional, cognitive, and neuroprosthetic domains. Published output in ECoG fields has increased tenfold in the past decade. These proceedings attempt to summarize the state of the art.
Keywords: Brain Mapping; brain-computer interface; electrical stimulation mapping; Electrocorticography; functional mapping; Gamma-frequency electroencephalography; High-frequency oscillations; Neuroprosthetics; Seizure detection; Subdural grid

Communication and control by listening: towards optimal design of a two-class auditory streaming brain-computer interface. - Jeremy Jeremy Hill; Moinuddin, Aisha; Häuser, Ann-Katrin; Kienzle, Stephan; Gerwin Schalk (Frontiers in Neuroscience, 2012)
Most brain-computer interface (BCI) systems require users to modulate brain signals in response to visual stimuli. Thus, they may not be useful to people with limited vision, such as those with severe paralysis. One important approach for overcoming this issue is auditory streaming, an approach whereby a BCI system is driven by shifts of attention between two simultaneously presented auditory stimulus streams. Motivated by the long-term goal of translating such a system into a reliable, simple yes-no interface for clinical usage, we aim to answer two main questions. First, we asked which of two previously published variants provides superior performance: a fixed-phase (FP) design in which the streams have equal period and opposite phase, or a drifting-phase (DP) design where the periods are unequal. We found FP to be superior to DP (p = 0.002): average performance levels were 80 and 72% correct, respectively. We were also able to show, in a pilot with one subject, that auditory streaming can support continuous control and neurofeedback applications: by shifting attention between ongoing left and right auditory streams, the subject was able to control the position of a paddle in a computer game. Second, we examined whether the system is dependent on eye movements, since it is known that eye movements and auditory attention may influence each other, and any dependence on the ability to move one’s eyes would be a barrier to translation to paralyzed users. We discovered that, despite instructions, some subjects did make eye movements that were indicative of the direction of attention. However, there was no correlation, across subjects, between the reliability of the eye movement signal and the reliability of the BCI system, indicating that our system was configured to work independently of eye movement. Together, these findings are an encouraging step forward toward BCIs that provide practical communication and control options for the most severely paralyzed users.
Keywords: auditory attention; auditory event-related potentials; brain-computer interface; dichotic listening; N1 potential; P3 potential

Proceedings of the Third International Workshop on Advances in Electrocorticography. - A L Ritaccio; Beauchamp, Michael; Bosman, Conrado; Peter Brunner; Chang, Edward; Nathan E. Crone; Gunduz, Aysegul; Disha Gupta; Robert T. Knight; Leuthardt, Eric; Litt, Brian; Moran, Daniel; et al. (Epilepsy Behav, 2012)
The Third International Workshop on Advances in Electrocorticography (ECoG) was convened in Washington, DC, on November 10-11, 2011. As in prior meetings, a true multidisciplinary fusion of clinicians, scientists, and engineers from many disciplines gathered to summarize contemporary experiences in brain surface recordings. The proceedings of this meeting serve as evidence of a very robust and transformative field but will yet again require revision to incorporate the advances that the following year will surely bring.
Keywords: Brain Mapping; brain-computer interface; Electrocorticography; Gamma-frequency electroencephalography; high-frequency oscillation; Neuroprosthetics; Seizure detection; Subdural grid

Review of the BCI Competition IV. - Tangermann, M.; Muller, K.R.; Aertsen, A.; Niels Birbaumer; Christoph Braun; Brunner, Clemens; Leeb, R.; Mehring, C.; Miller, K.J.; Mueller-Putz, G.; Nolte, G.; Pfurtscheller, G.; et al. (Frontiers in Neuroprosthetics, 2012)
The BCI competition IV stands in the tradition of prior BCI competitions that aim to provide high quality neuroscientific data for open access to the scientific community. As experienced already in prior competitions not only scientists from the narrow field of BCI compete, but scholars with a broad variety of backgrounds and nationalities. They include high specialists as well as students. The goals of all BCI competitions have always been to challenge with respect to novel paradigms and complex data. We report on the following challenges: (1) asynchronous data, (2) synthetic, (3) multi-class continuous data, (4) session-to-session transfer, (5) directionally modulated MEG, (6) finger movements recorded by ECoG. As after past competitions, our hope is that winning entries may enhance the analysis methods of future BCIs.
Keywords: BCI; brain-computer interface; competition

Prior knowledge improves decoding of finger flexion from electrocorticographic signals. - Zuoguan Wang; Ji, Q; Miller, John W; Gerwin Schalk (Front Neurosci, 2011)
Brain-computer interfaces (BCIs) use brain signals to convey a user's intent. Some BCI approaches begin by decoding kinematic parameters of movements from brain signals, and then proceed to using these signals, in absence of movements, to allow a user to control an output. Recent results have shown that electrocorticographic (ECoG) recordings from the surface of the brain in humans can give information about kinematic parameters (e.g., hand velocity or finger flexion). The decoding approaches in these studies usually employed classical classification/regression algorithms that derive a linear mapping between brain signals and outputs. However, they typically only incorporate little prior information about the target movement parameter. In this paper, we incorporate prior knowledge using a Bayesian decoding method, and use it to decode finger flexion from ECoG signals. Specifically, we exploit the constraints that govern finger flexion and incorporate these constraints in the construction, structure, and the probabilistic functions of the prior model of a switched non-parametric dynamic system (SNDS). Given a measurement model resulting from a traditional linear regression method, we decoded finger flexion using posterior estimation that combined the prior and measurement models. Our results show that the application of the Bayesian decoding model, which incorporates prior knowledge, improves decoding performance compared to the application of a linear regression model, which does not incorporate prior knowledge. Thus, the results presented in this paper may ultimately lead to neurally controlled hand prostheses with full fine-grained finger articulation.
Keywords: brain-computer interface; decoding algorithm; electrocorticographic; finger flexion; machine learning; prior knowledge

Rapid Communication with a "P300" Matrix Speller Using Electrocorticographic Signals (ECoG). - Peter Brunner; A L Ritaccio; Emrich, Joseph F; H Bischof; Gerwin Schalk (Front Neurosci, 2011)
A brain-computer interface (BCI) can provide a non-muscular communication channel to severely disabled people. One particular realization of a BCI is the P300 matrix speller that was originally described by Farwell and Donchin (1988). This speller uses event-related potentials (ERPs) that include the P300 ERP. All previous online studies of the P300 matrix speller used scalp-recorded electroencephalography (EEG) and were limited in their communication performance to only a few characters per minute. In our study, we investigated the feasibility of using electrocorticographic (ECoG) signals for online operation of the matrix speller, and determined associated spelling rates. We used the matrix speller that is implemented in the BCI2000 system. This speller used ECoG signals that were recorded from frontal, parietal, and occipital areas in one subject. This subject spelled a total of 444 characters in online experiments. The results showed that the subject sustained a rate of 17 characters/min (i.e., 69 bits/min), and achieved a peak rate of 22 characters/min (i.e., 113 bits/min). Detailed analysis of the results suggests that ERPs over visual areas (i.e., visual evoked potentials) contribute significantly to the performance of the matrix speller BCI system. Our results also point to potential reasons for the apparent advantages in spelling performance of ECoG compared to EEG. Thus, with additional verification in more subjects, these results may further extend the communication options for people with serious neuromuscular disabilities.
Keywords: brain-computer interface; Electrocorticography; event-related potential; P300; speller

Brain-Computer Interaction. - Peter Brunner; Gerwin Schalk (5th Intl. Conference on Augmented Cognition, 2009)
Detection and automated interpretation of attention-related or intention-related brain activity carries significant promise for many military and civilian applications. This interpretation of brain activity could provide information about a person’s intended movements, imagined movements, or attentional focus, and thus could be valuable for optimizing or replacing traditional motor-based communication between a person and a computer or other output devices. We describe here the objective and preliminary results of our studies in this area.
Keywords: BCI; brain-computer interface; neural engineering; neural prosthesis

Tracking of the mu rhythm using an empirically derived matched filter. - Krusienski, Dean J; Gerwin Schalk; Dennis J. McFarland; Jonathan Wolpaw (Proc. IEEE International Conference of Neural Engineering, 2005)
Keywords: bioelectric potentials; Brain Computer Interfaces; brain modeling; brain-computer interface; communication device; communication system control; cortical mu rhythm modulation; EEG; Electroencephalography; empirically derived matched filter; handicapped aids; laboratories; matched filters; medical signal detection; medical signal processing; monitoring; motor imagery; mu rhythm tracking; noninvasive treatment; rhythm; synchronous motors; two-dimensional cursor control data

The BCI Competition 2003: progress and perspectives in detection and discrimination of EEG single trials. - Benjamin Blankertz; Müller, Klaus-Robert; Curio, Gabriel; Theresa M Vaughan; Gerwin Schalk; Jonathan Wolpaw; Schlögl, Alois; Neuper, Christa; Pfurtscheller, Gert; Hinterberger, Thilo; Schröder, Michael; Niels Birbaumer (IEEE transactions on bio-medical engineering, 2004)
Interest in developing a new method of man-to-machine communication–a brain-computer interface (BCI)–has grown steadily over the past few decades. BCIs create a new communication channel between the brain and an output device by bypassing conventional motor output pathways of nerves and muscles. These systems use signals recorded from the scalp, the surface of the cortex, or from inside the brain to enable users to control a variety of applications including simple word-processing software and orthotics. BCI technology could therefore provide a new communication and control option for individuals who cannot otherwise express their wishes to the outside world. Signal processing and classification methods are essential tools in the development of improved BCI technology. We organized the BCI Competition 2003 to evaluate the current state of the art of these tools. Four laboratories well versed in EEG-based BCI research provided six data sets in a documented format. We made these data sets (i.e., labeled training sets and unlabeled test sets) and their descriptions available on the Internet. The goal in the competition was to maximize the performance measure for the test labels. Researchers worldwide tested their algorithms and competed for the best classification results. This paper describes the six data sets and the results and function of the most successful algorithms.
Keywords: augmentative communication; BCI; beta-rhythm; brain-computer interface; EEG; ERP; imagined hand movements; lateralized readiness potential; mu-rhythm; P300; Rehabilitation; single-trial classification; slow cortical potentials

EEG-based communication: presence of an error potential. - Gerwin Schalk; Jonathan Wolpaw; Dennis J. McFarland; Pfurtscheller, G. (Clinical neurophysiology : official journal of the International Federation of Clinical Neurophysiology, 2000)
EEG-based communication could be a valuable new augmentative communication technology for those with severe motor disabilities. Like all communication methods, it faces the problem of errors in transmission. In the Wadsworth EEG-based brain-computer interface (BCI) system, subjects learn to use mu or beta rhythm amplitude to move a cursor to targets on a computer screen. While cursor movement is highly accurate in trained subjects, it is not perfect.
Keywords: augmentative communication; brain-computer interface; Electroencephalography; error potential; error related negativity; event related potential; mu rhythm; Rehabilitation; sensorimotor cortex