Brain–Computer Interfaces: An Interactive Introduction

The brain has two natural ways to act on the world. Neuromuscular output moves the muscles, and hormonal output releases hormones. A BCI adds a third path. It builds commands straight from brain signals, with no muscles or hormones in between.

The last clause of the definition matters. A BCI’s output changes the environment, and the changed environment comes back to the brain as sensory input, or feedback. A device that only records or monitors brain activity therefore does not count as a BCI.

Five uses of BCI output

Common distinctions

• A dependent BCI uses brain signals that depend on muscle activity, such as visual evoked potentials that change with gaze direction. An independent BCI uses signals produced without muscles. Sensorimotor rhythms modulated by motor imagery are the classic example. Real BCIs usually fall somewhere in between.
• A hybrid BCI combines two kinds of brain signals, or combines BCI output with muscle-based output.
• A passive BCI uses signals tied to the user’s state, such as attention level. Wolpaw places these systems mostly in the enhance or supplement categories.
• BMI (brain–machine interface) is the term used mainly in implanted-electrode research. Today BMI and BCI mean the same thing.

Neurons produce two kinds of electrical signals. An action potential (spike) travels along the axon and lasts about 1 ms. A postsynaptic potential arises at the synapse and lasts tens of milliseconds. A microelectrode inside the cortex can pick up the spikes of single neurons. Scalp EEG mostly picks up the sum of many synaptic currents.

Pyramidal cells in the cerebral cortex stand side by side, perpendicular to the cortical surface. When synaptic current flows the same way in many of them, a patch of cortex acts like a single dipole layer. Its potential passes through the cerebrospinal fluid, skull, and scalp to reach the electrode.

Synchrony is the key. Nunez (Wolpaw ch. 3) estimates that a source region must be synchronous over about 6 cm² of cortex to appear on the scalp without averaging. Typical EEG of 20–100 µV needs synchrony over tens of square centimeters. This is why a drop in EEG amplitude is called desynchronization. The ERD in chapter 6 is exactly this effect.

For the same reason, the inverse problem of locating sources from a scalp potential map has no unique answer. A map that one dipole explains can also be explained by two dipoles, or ten (Wolpaw ch. 3).

Recording methods differ in how close the electrode gets to the brain (invasiveness) and in what they measure. Methods that measure electrical signals (spikes, LFP, ECoG, EEG, MEG) follow changes on the millisecond scale. Methods that measure blood flow (fMRI, fNIRS) stay on the scale of seconds because blood vessels respond slowly. A vessel takes about 2 s to respond to increased oxygen demand, and blood flow takes about another 12 s to fall back (Wolpaw ch. 4).

EEG has poor spatial resolution because the cerebrospinal fluid, skull, and scalp act as a volume conductor and a low-pass filter that smear the potential (Rao ch. 3). Placing electrodes on the cortical surface (ECoG) or inside the cortex (microelectrodes) raises resolution but brings surgical risk.

The international 10–20 system

This is the standard layout for scalp electrodes (Jasper 1958). Positions sit at 10% or 20% steps of two distances, from the nasion to the inion and between the two preauricular points. The letter names the region. Fp is the frontal pole, F frontal, C central, T temporal, P parietal, and O occipital. Odd numbers are on the left, even numbers on the right, and z marks the midline (Siuly §1.1).

An EEG voltage is always a difference between two electrodes. Measuring every channel against one reference electrode gives a referential (monopolar) montage. Measuring the difference between neighboring electrodes gives a bipolar montage.

EEG is a mix of overlapping rhythms. Adult scalp EEG ranges from about 1 to 100 µV and is divided into five frequency bands (Siuly §1.3). Band edges vary slightly between textbooks and studies.

The raw waveform mixes all the rhythms, so they are hard to tell apart by eye. The Fourier transform splits a signal into a sum of sines and cosines and shows the strength of each frequency as a power spectrum (Rao §4.2). Digital recording must respect the sampling theorem. The sampling rate must exceed twice the highest frequency in the signal, or high frequencies masquerade as low ones. This is aliasing.

Signals that come from outside the brain are artifacts. EEG almost always contains them, and they are often larger than the brain signals of interest (Siuly §1.3). Preprocessing aims to remove as much artifact as possible while keeping the brain signal.

Temporal filters separate signals by frequency. A high-pass filter removes slow components, a low-pass filter removes fast ones, and a notch filter removes a single frequency. A filter cannot separate brain signal from artifact where their frequencies overlap. Rao (§4.6) therefore lists thresholding, band-stop filtering, linear regression, PCA, and ICA as tools to combine as each situation requires.

Spatial filters

Temporal filters work on frequency. Spatial filters combine several electrodes instead (Rao §4.5.1). They suppress components spread across the whole scalp and keep components concentrated at one site. They help most with local signals such as sensorimotor rhythms (Wolpaw ch. 13).

A feature is a measurement of the signal that reflects the user’s intent. Common examples are the amplitude of an EEG rhythm, the voltage at a set time after a stimulus, and a neuron’s firing rate (Wolpaw ch. 1). A good feature changes a lot when intent changes and little with everything else.

ERD and ERS

Moving a hand, or imagining the movement, lowers mu and beta power over the hand area of the opposite hemisphere. This is event-related desynchronization (ERD). The right hand shows it at C3 and the left hand at C4. Foot imagery produces ERD near the midline at Cz, while power over the hand areas rises. Wolpaw ch. 13 calls this focal ERD/surround ERS. The beta rebound, a brief rise in beta power after a movement ends, is another form of ERS.

The standard calculation of Pfurtscheller and Aranibar (1979) has four steps. Band-pass filter each trial, square the samples to get power, average across trials, and compare with the power in a reference interval just before the event.

Common spatial patterns (CSP)

CSP is a supervised method that learns spatial filters from labeled training data. It picks the electrode weights that make the filtered output’s variance as large as possible for one class and as small as possible for the other (Rao §4.5.4). The variance of a band-passed signal is the power in that band, so CSP directly sharpens the band-power features that motor imagery BCIs use (Ramoser et al. 2000).

Feature translation turns features into device commands. Choosing one of a few options calls for classification. Producing a continuous value, such as cursor velocity, calls for regression (Wolpaw ch. 8).

Linear discriminant analysis (LDA)

LDA assumes the two classes are normal distributions with equal covariance and finds the line (a hyperplane in higher dimensions) that separates them. It is simple and fast, so online BCIs use it widely. Its strong assumptions make it fragile with non-normal data and outliers (Rao §5.1.1).

Generalization and nonstationarity

A model that is perfect on its training data does worse on new data. The gap, called overfitting, grows with less training data and more model freedom. Cross-validation estimates generalization by splitting the data and alternating between training and testing (Rao §5.1.4).

BCIs face one more problem. As users tire, gain experience and change strategy, or as a disease progresses, the statistics of the features change. Such data are nonstationary. More of the original training data cannot fix this, and the translation algorithm has to adapt. Wolpaw ch. 8 notes that adapting to changes in the mean is easy and adapting to changes in covariance is hard. Users also learn from feedback, so good BCIs are designed for the user and the algorithm to adapt together. This is co-adaptation.

BCIs use two broad kinds of brain responses. Evoked responses come from external stimuli (P300, SSVEP). Self-regulated activity is produced by the user (sensorimotor rhythms, slow cortical potentials, neuronal population activity) (Rao ch. 6).

The P300 and the oddball paradigm

The P300 is a positive potential evoked by rare events. Its latency is usually around 300 ms but varies between 250 and 750 ms. It is largest over the centro-parietal scalp and shrinks with distance from there. The conditions that evoke it form the oddball paradigm, which has three requirements. Stimuli fall into two classes, one class is rarer, and the user performs a task that sorts the stimuli into the classes (Wolpaw ch. 12; Donchin & Coles 1988).

The P300 is smaller than the background EEG, so a single trial rarely shows it. Averaging many trials aligned to stimulus onset cancels the background activity, which is unrelated to the stimulus, and leaves the P300. The remaining noise falls in proportion to the square root of the number of trials N.

Farwell and Donchin (1988) built a speller that flashes the rows and columns of a 6×6 letter matrix in random order. Only the row and the column that contain the target letter evoke a P300, so the cell where the strongest row and column meet is the chosen letter. Twelve flashes, one for each of the six rows and six columns, make one sequence. In the first study, 95% accuracy took 26 s per letter (Wolpaw ch. 12).

SSVEP

Looking at a stimulus that flickers faster than 5 Hz makes the visual cortex oscillate at the same frequency. This is the steady-state visual evoked potential (SSVEP). Give each option its own flicker frequency, and the spectrum of occipital EEG reveals which target the user is looking at. A 7 Hz stimulus produces peaks at 7, 14, and 21 Hz (Rao §9.1.4).

The phone-dialing BCI of Cheng et al. (2002) flickered 13 buttons at 6–14 Hz. It ran an FFT on O1 and O2 every 0.3 s and used, as each candidate’s feature, the amplitude at that frequency plus the amplitude at its second harmonic. It chose the largest candidate whose feature exceeded twice the mean amplitude from 4 to 35 Hz, and it confirmed a selection only after the same decision came up four times in a row. Eight of 13 subjects dialed the number they wanted, with an average information transfer rate of 27.15 bits/min. The study first removed frequencies with strong power during eyes-closed rest from the stimulus set (Rao §9.1.4).

Invasive BCIs and the population vector

Each motor cortex neuron has a preferred direction and fires most when the arm moves that way. Its firing rate follows the cosine of the angle between the movement direction and the preferred direction (Wolpaw ch. 2, 7). Georgopoulos et al. (1986) showed that the population vector, the sum of the neurons’ preferred directions weighted by their firing rates, points in the actual movement direction. Practical BCIs go further and decode velocity and position with linear regression or Kalman filters (Rao ch. 7).

Accuracy alone makes BCIs hard to compare. An accuracy of 80% means something very different with 2 options than with 36. The chance level, the probability of a correct guess, is \(1/N\) for \(N\) options. A confusion matrix shows which selections get mistaken for which, and the kappa coefficient \(\kappa=(p_o-p_e)/(1-p_e)\) gives agreement corrected for chance (Rao §5.1.4).

Wolpaw et al. (2002) proposed the information transfer rate (ITR, bit rate) from communication theory to capture speed and accuracy in a single number. It assumes that every option is equally likely to be intended and that errors spread evenly over the other options.

Wolpaw ch. 8 sets two rules for using ITR. Count the information in the final output only, and leave out information in the process that leads to it, such as a cursor’s trajectory. Count the time as the full interval from the start of one selection to the start of the next. The benchmark is online evaluation, with the user receiving real feedback during use.

BCI users are people who feel emotions and pay attention. Emotion and attention change the signals a BCI reads, and they can also be what a BCI reads. Systems that detect lapses of attention and raise an alert (Wolpaw’s enhance example in ch. 1) and systems that estimate workload (Rao §12.2) are examples. This chapter reads Part IV of the Cambridge Handbook, on cognition–emotion interactions, from a BCI perspective.

Emotion captures attention

When several stimuli compete for processing resources, emotionally meaningful ones are detected first or interfere with other tasks. People who fear spiders or snakes find those targets faster among pictures of mushrooms (ch. 14; Flykt & Caldara 2006). In EEG, early components 100–200 ms after the stimulus (P1, N1) grow for emotional faces and scenes, and after 300–400 ms the P3 and the sustained late positive potential (LPP) change (ch. 14).

Attention limits emotional processing

A long-standing view held that emotional processing is automatic. Pessoa and colleagues tested it. When participants did a very demanding task at the edge of the screen, the difference in amygdala and visual cortex responses to fearful and neutral faces at the center disappeared (Pessoa et al. 2002). When task difficulty varied in steps, the difference appeared only in the easy condition (Pessoa et al. 2005). The ERP early posterior negativity (EPN) also shrank sharply during demanding attention tasks (Schupp et al. 2007).

Some evidence points the other way. An MEG study found that fearful faces still evoked an attention-related component (N2pc) under high load (Fenker et al. 2010). Chapter 15 ties these results together with Lavie’s concept of perceptual load. When a task leaves resources unused, the spare capacity spills over to emotional stimuli that are irrelevant to the task.

One result is directly useful to BCI researchers. Emotional pictures shown in the background during a demanding task reduced the SSVEP driven by the task stimuli (Müller, Andersen & Keil 2008). The authors read this as the emotional stimuli pulling processing resources away from the task. The SSVEP BCIs of chapter 8 may therefore depend on the user’s emotional state too.

Emotion regulation

Chapter 16 organizes emotion regulation strategies with Gross’s process model. Strategies differ in where they intervene as an emotion unfolds (situation, attention, appraisal, response). Antecedent-focused strategies act before the response appears, and response-focused strategies act after it. Real emotions can skip steps or loop back, so read the model as a simplified map (ch. 16).

Value, prediction error, and error potentials

Chapter 17 breaks value-based decision making into several signals. Outcome value arises when an outcome is experienced (orbitofrontal cortex), decision value is assigned to each option, and chosen value follows a choice. The prediction error, the difference between expectation and outcome, is the signal that drives value learning.

BCIs have a similar signal. When a BCI produces an output the user did not intend, the user’s brain responds with an error-related potential (ErrP). ErrPs are clearest over the frontocentral midline (Fz, Cz) and can be detected in single trials. Buttfield et al. (2006) made a robot move the wrong way on purpose 20% of the time, then used 1–10 Hz EEG from 50–650 ms after feedback to correctly classify 79.9% of error trials and 82.4% of correct trials (Rao §9.1.6). Writing in 2013, Rao noted that no study had yet combined ErrP detection with a working BCI. Later studies went on to use ErrPs as reward signals for robot learning, for example Iturrate et al. (2015, Scientific Reports) and Salazar-Gomez et al. (2017, ICRA).

The first aim of BCI research is to give new means of communication and control to people who have trouble using their muscles, such as people with amyotrophic lateral sclerosis (ALS), spinal cord injury, stroke, or cerebral palsy (Wolpaw ch. 19). Applications are also spreading beyond medicine (Rao ch. 12; Wolpaw ch. 23).

From the lab to daily life

Lab demonstrations are not enough to bring BCIs into daily life. Real users must be shown to use them independently at home over long periods and to benefit from them (Wolpaw ch. 20–21). In 2012, Wolpaw ch. 12 judged P300 BCIs to be essentially the only BCIs that severely disabled people use outside the lab, every day, for important purposes.

Ethical issues

• Balancing risk and benefit. Implanted BCIs carry the risks of surgery, infection, and tissue damage. The expected benefit has to outweigh them (Rao §13.1.1).
• Informed consent. Getting consent to research from someone who can barely communicate is itself difficult (Rao §13.1.2; Wolpaw ch. 24).
• Misuse. The technology can drift from its purpose, through unwanted or military use (Rao §13.2).
• Security and privacy. Brain signals can carry information the user never meant to reveal, and BCI devices can be attacked (Rao §13.3).
• Legal responsibility. When a BCI output causes harm, the law has to settle whether responsibility lies with the user, the manufacturer, or the algorithm (Rao §13.4).
• Justice and equity. Expensive technology may reach only a few people, and use for enhancement may widen inequality (Rao §13.5).
• Managing expectations. Media hype and real performance are far apart. Patients and families should not be led into unrealistic hopes (Rao, preface; Wolpaw ch. 25).

This table matches the chapters of this guide to the chapters of the source textbooks. Start here when you want to study a concept in more depth.

References

• Wolpaw, J. R., & Wolpaw, E. W. (Eds.) (2012). Brain–Computer Interfaces: Principles and Practice. Oxford University Press.
• Rao, R. P. N. (2013). Brain–Computer Interfacing: An Introduction. Cambridge University Press.
• Siuly, S., Li, Y., & Zhang, Y. (2016). EEG Signal Analysis and Classification: Techniques and Applications. Springer.
• Armony, J., & Vuilleumier, P. (Eds.) (2013). The Cambridge Handbook of Human Affective Neuroscience. Cambridge University Press. Part IV: ch. 14 (Domínguez-Borràs & Vuilleumier), ch. 15 (Pessoa, Oliveira & Pereira), ch. 16 (Phan & Sripada), ch. 17 (O’Doherty).