IntoMindLearn

Chapter 8 · Middle school

Designing a BCI

What would you build if your brain could flip a switch? Plan it, and plan how you will know it works.

How engineers design

Engineers build in a loop called the engineering design process: define the problem, imagine ideas, plan one, create it, test it, improve it, and test again.

Your project, in the capstone, will tell two of your mental states apart and switch the board's light or pin. It starts with a person and a need.

The design loop A person and a need, requirements, the test written first, the build, the test run, and one change. Then around again.

Try it

The brief

In your group, write the problem in one sentence. Then three ideas for what your board's light or pin could do. Pick one, and say why.

Your briefYour group's answer
The problem, in one sentence
Idea 1
Idea 2
Idea 3
The one we pick, and why

Choosing two states

Choose two states you can each hold for a minute, that don't need blinking or moving your jaw, and that are likely to look different in your EEG. Resting with your eyes open, against counting down in your head, works for many people.

Blinks and a clenched jaw make signals far bigger than your brain's. A project built on them would be reading your muscles, not your mind.

Simulated Four seconds of EEG with two blinks and a clenched jaw in it. Both are far bigger than the brain's own signal.

A test you write first

Engineers write the test before they build, so nobody can change what counts as working after seeing the result. Each requirement is a condition the design must meet, written so a test can check it.

Ask for more than luck. A light that only guessed would follow eight or more of ten switches about 5 times in 100.

Ten switches, by luck How many of ten switches a light would follow if it only guessed. Eight or more, in orange, happens about 5 times in 100.

Try it

The test, written first

Before you collect anything, write your test. Your teacher keeps a copy.

Your testYour group's answer
Our two states
How many times each wearer will teach it
The score that counts as working
How many of ten switches the light must follow
What we will do if it doesn't work
Date

Talk about it

  1. Why do engineers write the test before they build?

  2. Why should your two states not need blinking or clenching your jaw?

  3. Why ask the light to follow eight of ten switches, not six?

Chapter 9 · High school

Designing a BCI

Who would your brain-computer interface help, and how will you know it works for them? Turn a need into requirements, and write the test before you build.

Who brain-computer interfaces are for

Brain-computer interfaces were first developed for people who cannot move or speak, such as people with advanced ALS, a disease that takes away the control of the muscles. For them, an interface that is slow and imperfect can still mean calling for help, answering yes or no, or switching something on without help.

Engineers design in a loop called the engineering design process, and it starts with the person: what they need to do, what stops them, and what would make an interface worth wearing.

The design loop A person and a need, requirements, the test written first, the build, the test run, and one change. Then around again.

Try it

The brief

Choose a person to design for: someone described in a published case, such as a study or a news story about brain-computer interfaces, or a person your group imagines from such cases. Then write three ideas for what your board's light or pin could do for them, and pick one.

Your briefYour group's answer
The person, and what they need to do
What stops them now
What would make them stop using it
Idea 1
Idea 2
Idea 3
The one we pick, and why

Requirements

A requirement turns a need into something a test can check. When 61 people with ALS were asked what a brain-computer interface would have to do, most would be satisfied by one that is right at least 9 times in 10, as fast as 15 to 19 letters a minute, and that leaves its standby by mistake no more than once every 2 to 4 hours.

Each of those is a number a test can check. Yours will be too.

A needA requirementA test that checks it
To switch it on when they mean toThe light follows at least 8 of 10 switches.Someone else calls ten switches, and counts.
Not to switch it on by mistakeAt most one switch to On in 5 minutes of rest.The wearer rests for 5 minutes, and the recorder counts.
Not to wait longThe light comes on within 10 seconds, in the median of 5 switches.Time five switches, as in the Mind-Controlled Microcontroller.
To use it every dayA helper puts it on and starts it in under 3 minutes.Time three setups.

Two kinds of mistakes

A switch can be wrong two ways. A false positive switches to On when nobody meant it. A false negative stays Off when the wearer meant On.

The threshold sets the balance between them. The board compares each score with SWITCH_THRESHOLD, 0.5 unless you change it. Raise it, and there are fewer false positives and more false negatives. Which mistake costs more depends on the person and the purpose.

Simulated Forty Off readings and forty On readings from one custom AI head, by score. The ringed ones are mistakes. Move the threshold, and one kind of mistake trades for the other.

A test you write first

Write the test before you build, and keep to it. Then nobody, including you, can move the bar after seeing the result. Scientists call this preregistration.

Ask for more than luck. A light that only guessed would follow 8 or more of 10 switches about 5 times in 100.

Ten switches, by luck How many of ten switches a light would follow if it only guessed. Eight or more, in orange, happens about 5 times in 100.

Try it

The test, written first

Choose your states and site as in the Mind-Controlled Light Switch: two states you can each hold for a minute without blinking or moving your jaw, at the site where they should differ most. Then write your test. Swap it with another group, and ask whether a test could check every line. Then sign and date it. Your teacher keeps a copy.

Your testYour group's answer
Our two states, and the site
Sessions to teach it, on how many different days
The score that counts as working
How many of ten switches the light must follow
The longest median delay we accept, in seconds
False positives allowed in 5 minutes of rest
Our threshold, and why
What we will do if it doesn't work
Signed and dated

Talk about it

  1. Your interface is right 7 times in 10. For whom could that be useful, and for whom not?

  2. Your interface calls for help. Would you set the threshold above 0.5, or below? What does it cost?

  3. A group writes its test after seeing its results. How could that make an interface that doesn't work look as if it does?

Sources

Huggins, J. E., Wren, P. A., and Gruis, K. L. (2011). What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis.

Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clinical Neurophysiology.

Nosek, B. A., Ebersole, C. R., DeHaven, A. C., and Mellor, D. T. (2018). The preregistration revolution. Proceedings of the National Academy of Sciences.

Chapter 10 · College

Designing a BCI

Who would your brain-computer interface serve, what must it do for them, stated so a test can check it, and where should its threshold sit when the two kinds of mistake cost different amounts?

Who brain-computer interfaces are for

Brain-computer interfaces were developed first for people who can't move or speak, such as people with advanced ALS. A person with locked-in syndrome has used a fully implanted interface to communicate at home, and noninvasive ones have been used for years in research.

User-centered design evaluates an interface with the people it is for, by its effectiveness, its efficiency and their satisfaction, not only by accuracy with healthy volunteers. Reliability, setup time and independence matter to users in ways a lab's accuracy doesn't measure.

The design loop A person and a need, requirements, the test written first, the build, the test run, and one change. Then around again.

Requirements a test can check

A requirement names a metric, a threshold, the conditions and the users. When 61 people with ALS were asked what an interface would have to do, most would be satisfied by one right at least 9 times in 10, as fast as 15 to 19 letters a minute, and leaving its standby by mistake no more than once every 2 to 4 hours.

What users needA requirementHow to test it
EffectivenessThe custom AI head clearly better than chance, in every session, for the user.The permutation test over rounds, in each session.
EfficiencyThe pin following at least 8 of 10 calls, each within 8 seconds.Ten calls at random times, timed.
Few false alarmsAt most one switch to On in 10 minutes of rest.Ten minutes at rest, counted.
SetupWorn and running in under 5 minutes, with one helper.Three setups, timed.
SatisfactionA usability score above an agreed line.A standard questionnaire, such as the System Usability Scale.

The operating point

The threshold chooses a point on the ROC curve: raise it, and false positives fall while false negatives rise. When the two cost different amounts, choose the threshold that minimizes the expected cost,

cost = cFP · P(false positive) + cFN · P(false negative)

estimated on data the head never trained on. A call for help, where a miss costs far more than a false alarm, wants a low threshold. A switch that does something hard to undo wants a high one.

Simulated Forty Off readings and forty On readings from one custom AI head, by score. The ringed ones are mistakes. Move the threshold, and one kind of mistake trades for the other.

Try it

The brief, and its requirements

Choose a person to design for, from a published case or imagined from one, and write the brief and the requirements your capstone will be tested against.

Your briefYour group's answer
The person, and what they need to do
What stops them now, and what would make them stop using it
What the board's pin will do for them
Effectiveness: the requirement and its test
Efficiency: the requirement and its test
False alarms: the requirement and its test
Which mistake costs more, and so where the threshold goes

Problems

  1. Why is accuracy with healthy volunteers not enough to evaluate an assistive interface?

  2. Rewrite “it should be fast and accurate” as requirements a test can check.

  3. A miss costs ten times a false alarm. Where should the threshold move, and how would you find the point?

  4. A two-state interface is right 85 percent of the time, with a decision every four seconds. How many bits a minute is that?

Sources

Kübler, A., and others (2014). The user-centered design as novel perspective for evaluating the usability of BCI-controlled applications. PLoS ONE.

Huggins, J. E., Wren, P. A., and Gruis, K. L. (2011). What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis.

Vansteensel, M. J., and others (2016). Fully implanted brain-computer interface in a locked-in patient with ALS. New England Journal of Medicine.

Brooke, J. (1996). SUS: a quick and dirty usability scale. In Usability Evaluation in Industry. Taylor and Francis.

Chapter 10 · Graduate

Designing a BCI

Given a person and a purpose, how do you specify a brain-computer interface, choose where it operates, and decide whether it is good enough, before you build it and after?

Users first

User-centered design evaluates an interface by its effectiveness, its efficiency and its users' satisfaction, with the people it is for, in their setting (Kübler and others, 2014). Asked what a brain-computer interface would have to do, most of 61 people with ALS would be satisfied with one that was right at least 90 percent of the time, that spelled 15 to 19 letters a minute, and that left its standby by mistake no more than once every 2 to 4 hours (Huggins, Wren and Gruis, 2011). A requirement turns such an answer into a metric, a threshold, the conditions and the users, so that a test can check it.

Metrics that mean something

Wolpaw's information transfer rate (Wolpaw and others, 2002) summarizes speed and accuracy in bits a minute, under assumptions that interfaces rarely meet: choices equally likely, errors spread evenly, and no cost to correcting them (Thompson and others, 2014). A utility metric counts what the user gains, with the time errors cost to undo (Dal Seno, Matteucci and Mainardi, 2010). For a switch, the numbers that matter are the delay to a deliberate switch, its tail as much as its middle, and the false switches per hour of meaning nothing. An interface used asynchronously, whenever the user wants, has a no-control state, and keeping it quiet is often harder than telling two states apart (Mason and Birch, 2000).

The operating point

A threshold trades the two errors. With the share of time spent in On, π, and the costs of a false switch and a missed one, the expected cost of a decision is

cFP (1 − π) P(score > θ | Off) + cFN π P(score ≤ θ | On)

and the threshold that minimizes it is where the scores' likelihood ratio, On over Off, equals cFP(1 − π) / (cFN π). A run of k windows above it lowers false switches, by about the k-th power only if neighboring windows are independent, and adds k − 1 windows of delay. When the cost of acting wrongly is high, the system can also wait, ask, or let the user undo, and the brain's own response to an error can help it notice (Chavarriaga, Sobolewski and Millán, 2014).

Analysis

Analysis

An operating point that meets two requirements

Two requirements: no more than one false switch every two hours, and 90 percent of deliberate switches within 15 seconds. Simulate a head's scores with neighboring windows alike, as real ones are, and search thresholds and holds for one that meets both. Then put in your own head's numbers from the light switch.

import numpy as np

rng = np.random.default_rng(10)
W = 4.0                       # a score every four seconds
OFF, ON, SD, RHO = 0.2, 0.8, 0.15, 0.5   # your head's scores: means, spread, and how alike
                                         # neighboring windows are. Put in your own.

def noise(rows, cols):
    """Window-to-window noise that carries over from one window to the next."""
    e = rng.normal(0, SD, (rows, cols))
    for j in range(1, cols):
        e[:, j] = RHO * e[:, j - 1] + np.sqrt(1 - RHO ** 2) * e[:, j]
    return e

def held(above, hold):
    """Where `hold` windows in a row have been above the threshold."""
    count = np.zeros(above.shape, int)
    for j in range(above.shape[1]):
        count[:, j] = np.where(above[:, j], (count[:, j - 1] if j else 0) + 1, 0)
    return count >= hold

def false_per_hour(threshold, hold, hours=200):
    on = held(OFF + noise(1, int(hours * 3600 / W)) > threshold, hold)[0]
    return np.sum(on[1:] & ~on[:-1]) / hours

def slow_tail(threshold, hold, trials=5000):
    """The 90th percentile of the delay from a change to On, with windows on a grid."""
    t0 = rng.uniform(0, W, trials)
    ends = W * np.arange(1, 13)
    share = np.clip((ends[None] - t0[:, None]) / W, 0, 1)
    h = held(OFF + share * (ON - OFF) + noise(trials, len(ends)) > threshold, hold)
    return np.percentile(ends[np.argmax(h, axis=1)] - t0, 90)

print("threshold  hold  false an hour  90% of switches within")
for threshold in (0.5, 0.6, 0.7):
    for hold in (1, 2, 3):
        f, tail = false_per_hour(threshold, hold), slow_tail(threshold, hold)
        ok = "  meets both" if f <= 0.5 and tail <= 15 else ""
        print(f"{threshold:9.1f} {hold:5d} {f:14.2f} {tail:13.1f} s{ok}")

You should seeOne rule meeting both, a threshold of 0.7 with no hold: 0.39 false switches an hour, and 90 percent within 12.3 seconds. The demo's rule, 0.5 with no hold, gives 17 false switches an hour. Holds help less than independence would suggest: at 0.5, two in a row still leave 2.8 an hour.

Then report: the rule your own head's scores allow, or which requirement it can't meet, and what would have to change.

Open problems

  1. Metrics that predict use. Which laboratory measures predict whether someone keeps using an interface at home is not settled (Thompson and others, 2014).
  2. Home use. A noninvasive interface was used independently at home by people with ALS for up to 18 months, though 14 of the 42 who enrolled got that far (Wolpaw and others, 2018), and an implanted one by a person with ALS in daily life (Vansteensel and others, 2016). What it takes for more people to keep using noninvasive ones is open.
  3. Meaning nothing. Recognizing that a person doesn't intend anything, reliably and for hours, is the hard part of asynchronous control, and no general solution exists.

Problems

  1. Turn "leaves standby by mistake no more than once every two hours" into a per-window probability for a decoder that decides every four seconds: with no hold, and with three windows in a row, if windows were independent.

  2. A switch is right 95 percent of the time, deciding every four seconds. What is its information transfer rate, and what does that number leave out?

  3. Off scores are normal around 0.2 and On around 0.8, both with a spread of 0.15. The user is in On a tenth of the time, and a false switch costs five times a missed one. Where should the threshold be?

  4. Which of effectiveness, efficiency and satisfaction does the light switch's verdict address?

  5. Write three testable requirements for a mind-controlled call button for someone who can't move, and how you would test each.

Sources

Mason, S. G., and Birch, G. E. (2000). A brain-controlled switch for asynchronous control applications. IEEE Transactions on Biomedical Engineering.

Wolpaw, J. R., Birbaumer, N., McFarland, D. J., Pfurtscheller, G., and Vaughan, T. M. (2002). Brain-computer interfaces for communication and control. Clinical Neurophysiology.

Dal Seno, B., Matteucci, M., and Mainardi, L. T. (2010). The utility metric: a novel method to assess the overall performance of discrete brain-computer interfaces. IEEE Transactions on Neural Systems and Rehabilitation Engineering.

Huggins, J. E., Wren, P. A., and Gruis, K. L. (2011). What would brain-computer interface users want? Opinions and priorities of potential users with amyotrophic lateral sclerosis. Amyotrophic Lateral Sclerosis.

Chavarriaga, R., Sobolewski, A., and Millán, J. del R. (2014). Errare machinale est: the use of error-related potentials in brain-machine interfaces. Frontiers in Neuroscience.

Kübler, A., and others (2014). The user-centered design as novel perspective for evaluating the usability of BCI-controlled applications. PLoS ONE.

Thompson, D. E., and others (2014). Performance measurement for brain-computer or brain-machine interfaces: a tutorial. Journal of Neural Engineering.

Vansteensel, M. J., and others (2016). Fully implanted brain-computer interface in a locked-in patient with ALS. New England Journal of Medicine.

Wolpaw, J. R., and others (2018). Independent home use of a brain-computer interface by people with amyotrophic lateral sclerosis. Neurology.