This chapter is part of the book from college on.
Chapter 8 · College
Is a model any good? Only against the right baseline, with an interval, on test data like the data it will meet, and at the base rate of the people it will be used on.
A mean absolute error means nothing alone. Compare it with a baseline: what a trivial model gets, such as always guessing the training data's mean age, and what a simple model fitted the same way gets, such as one on band power.
The age head misses by 9.16 years, averaged over a person's windows. Band power misses by 10.02, and always guessing the mean age by 9.77. The head beats band power, with an interval that clears zero, but its lead over the mean age has an interval that includes it.
Test on new windows from people in the training data, and you measure memory of those people. Test on new people from the same studies, and you measure generalization within those studies. Leave whole studies out, and you measure what happens on a new device, protocol and population, which is how a model gets used. The published heads were scored on studies they never saw.
The interval comes from resampling the unit that varies in use: whole studies, not windows, in a bootstrap interval. One caveat stays in the published account: the encoder, trained with no labels, saw unlabeled windows from most of the studies the heads were scored on.
Try it
From the published evaluation, write each comparison as one sentence a reader can't misread: the difference, its interval, and what it means.
| Comparison | One window | A person's windows, averaged |
|---|---|---|
| The head against band power | ||
| The head against the mean age |
A classifier's score becomes a call at a threshold, and moving the threshold trades false positives for false negatives. The ROC curve plots the true positive rate against the false positive rate at every threshold. Its area, the AUC, is the chance that a random member of one group outscores a random member of the other: 0.5 for a coin, 1 for perfect separation. Two normal groups 0.44 standard deviations apart give 0.623, the sex head's published AUC averaged over a person's windows.
The AUC doesn't depend on the base rate, but what a call means does. The positive predictive value falls steeply as a class becomes rare: with equal groups, a threshold halfway between those two means is right about 59 times in 100, and with one group at 10 percent of people, a call of it is right about 14 times in 100.
A score isn't a probability unless it has been calibrated to one. The sex head's output is a least-squares score, not a probability, and its evaluation says so.
Try it
Simulate two groups with the sex head's separation, draw their ROC curve, compute the AUC by ranks, and work out what a call is worth at four base rates.
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import rankdata
d = 0.44 # the groups' separation, in standard deviations
rng = np.random.default_rng(0)
neg = rng.normal(0, 1, 20_000) # scores of group 0
pos = rng.normal(d, 1, 20_000) # scores of group 1
# The ROC curve, and its area by ranks: the chance a random 1 outscores a random 0.
t = np.linspace(-4, 5, 400)
plt.plot([(neg > x).mean() for x in t], [(pos > x).mean() for x in t])
plt.plot([0, 1], [0, 1], "--")
plt.xlabel("false positive rate")
plt.ylabel("true positive rate")
plt.show()
r = rankdata(np.r_[pos, neg])
auc = (r[:len(pos)].sum() - len(pos) * (len(pos) + 1) / 2) / (len(pos) * len(neg))
print("AUC", round(auc, 3))
# A threshold halfway between the means, and what a call of group 1 is worth.
sensitivity, false_alarms = (pos > d / 2).mean(), (neg > d / 2).mean()
for base in (0.5, 0.2, 0.1, 0.01):
ppv = base * sensitivity / (base * sensitivity + (1 - base) * false_alarms)
print(f"group 1 at {base:.0%}: a call of group 1 is right {ppv:.0%} of the time")You should seeAn AUC near 0.62, and a call of group 1 right about 59 times in 100 with equal groups, 26 at a fifth, 14 at a tenth, and 1 at one in a hundred.
A model card puts all of this on one page: what the model is, what it is for and not for, the data it was fitted and evaluated on, the metrics with their baselines and intervals, and its limitations. The published evaluation reads as one, and ends with the sentence every model card needs: neither head is a useful predictor of a person's age or sex, and neither is offered as one.
Why is “mean error 9.16 years” incomplete without the mean-age baseline?
Why resample studies, not windows, for the interval?
An AUC of 0.62 doesn't change with the base rate. Why does the positive predictive value?
A vendor reports 95 percent accuracy for a detector of a condition one person in a hundred has. What do you ask for?
IntoMind (2026). Heads for encoder e2f9b60f411f0e73. github.com/intomind/eeg-foundation-model-heads.
Fawcett, T. (2006). An introduction to ROC analysis. Pattern Recognition Letters.
Efron, B., and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall.
Mitchell, M., and others (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
Chapter 8 · Graduate
What would convince you a model works: against which baseline, scored on whose data, with uncertainty from which unit, and for what use?
An error means something only beside a baseline: a trivial one that ignores the input, such as the training data's mean, and a simple one fitted the same way, such as band power. The published age head misses by 9.93 years on one window, against 10.26 for band power and 9.75 for always guessing the training studies' mean age. Averaged over a person's windows it misses by 9.16, against 10.02 and 9.77. The sex head's AUC is 0.588 on one window and 0.623 averaged, against 0.582 and 0.614 for band power. The evaluation's own conclusion: neither is a useful predictor of a person's age or sex.
Studies differ in devices, protocols and populations, and a model is used on studies it has never seen, so the published evaluation leaves each study out in turn. It hides the electrode positions, scores nine referential montages of 2, 4 and 8 electrodes with the device's integer arithmetic, and for its averaged results averages each person's windows, up to 40 a dataset. Each study gives one value per method, the reported value is the mean over studies, and its 95 percent interval comes from resampling studies. Each comparison is a Wilcoxon signed-rank test over studies, Holm-corrected over eight tests.
The unit resampled is the unit that varies in use. Resampling windows, or people within studies, describes how precisely these studies were measured, not how the model will do on the next one (Efron and Tibshirani, 1993).
A model fitted by least squares on a weak signal predicts toward its training mean, so young people are predicted older and old people younger, and the gap between predicted and actual age falls with age. A group difference in the gap can then be a difference in age. Regressing age out of the gap removes that dependence (Smith and others, 2019), and analyses of the gap should adjust for age (Le and others, 2018). But predictions corrected with each person's true age, then scored against it, are circular: their R² rarely falls below 0.85 whatever the model (Butler and others, 2021). EEG predicts age, from sleep (Sun and others, 2019) and at rest, where a shared benchmark compares methods on the same data (Engemann and others, 2022).
The AUC is the probability that a random member of one group outscores a random member of the other (Hanley and McNeil, 1982), and DeLong's interval (DeLong, DeLong and Clarke-Pearson, 1988) assumes the examples are independent, which windows within people within studies aren't. Discrimination isn't calibration: the sex head's output is a least-squares score, female as 1 and male as 0, not a probability, and the same score would need recalibrating for every population. What a positive call is worth depends on the base rate: for an AUC of 0.62 with equal variances and a threshold midway, a positive call is right 59 percent of the time when the groups are equal in number, and 14 percent when the positive group is a tenth of those tested.
The heads were fitted and scored on separate studies, but the encoder beneath them saw unlabeled windows from 81 of the 109 studies scored for age and 90 of the 121 for sex. The representation was not fitted to those labels, but it was shaped by those recordings. A test of how the whole system does on a new study needs studies that no part of it ever saw.
A report says what a model is for, what it was evaluated on, against what, and with what uncertainty, as model cards (Mitchell and others, 2019) and, for clinical prediction, TRIPOD+AI (Collins and others, 2024) lay out. The heads' evaluation states its exposure and that the heads are offered as no one's predictor, which is what lets a reader use it.
Analysis
Simulate a head that predicts toward a training mean of 40, and score it in a population like its training studies' and in a seminar room aged 22 to 30, against the training mean and the group's own mean. Correct its gap the usual way, and score the corrected predictions. Then compute a mean AUC over 20 studies that differ in how well the head separates the groups, with intervals from resampling windows within studies and from resampling whole studies.
import numpy as np
from scipy.stats import rankdata
rng = np.random.default_rng(8)
# 1. A head that regresses toward its training mean, as a weak least-squares fit does.
def predict(age, train_mean=40.0, slope=0.4, noise=7.0):
return train_mean + slope * (age - train_mean) + rng.normal(0, noise, len(age))
wide = rng.uniform(18, 80, 3000) # people like the training studies'
room = rng.uniform(22, 30, 30) # a seminar room
for name, ages in (("wide population", wide), ("seminar room", room)):
p = predict(ages)
print(f"{name}: head {np.mean(np.abs(p - ages)):.1f} years, "
f"training mean {np.mean(np.abs(40.0 - ages)):.1f}, "
f"the group's own mean {np.mean(np.abs(ages.mean() - ages)):.1f}")
p = predict(wide)
gap = p - wide # the "brain age gap"
print(f"gap against age: r = {np.corrcoef(gap, wide)[0, 1]:.2f}")
b, a = np.polyfit(wide, gap, 1) # the usual correction, fitted on age itself
corrected = p - (a + b * wide)
r2 = lambda q: 1 - np.sum((q - wide) ** 2) / np.sum((wide - wide.mean()) ** 2)
print(f"R^2 of the predictions {r2(p):.2f}, of the corrected ones {r2(corrected):.2f}")
# 2. An AUC per study, averaged over studies, with its interval from resampling the windows
# or the studies. Studies differ in how well the head separates the groups.
studies, per = 20, 200
study = np.repeat(np.arange(studies), per)
effect = np.repeat(rng.normal(0.35, 0.4, studies), per) # the separation, study by study
group = (rng.random(studies * per) < 0.5).astype(int)
score = effect * group + rng.normal(0, 1, studies * per)
def auc(s, g):
r = rankdata(s)
n1 = g.sum()
return (r[g == 1].sum() - n1 * (n1 + 1) / 2) / (n1 * (len(g) - n1))
def mean_auc(pick):
"""The mean over the studies in `pick`, repeats allowed, of each one's AUC."""
return np.mean([auc(score[w], group[w]) for w in pick])
each = [np.flatnonzero(study == s) for s in range(studies)]
by_window = [mean_auc([rng.choice(w, len(w)) for w in each]) for _ in range(1000)]
by_study = [mean_auc([each[s] for s in rng.integers(0, studies, studies)]) for _ in range(1000)]
print(f"mean AUC over studies {mean_auc(each):.3f}")
for name, vals in (("windows within studies", by_window), ("whole studies", by_study)):
print(f"95% interval, resampling {name}: {np.round(np.percentile(vals, [2.5, 97.5]), 3)}")You should seeIn the wide population, the head misses by 11.2 years against 16.3 for the training mean. In the room, by 10.8 against 14.4, while the room's own mean misses by 2.1. The gap falls with age, r = −0.84, and correcting it lifts R² from 0.37 to 0.85. The mean AUC is 0.611, with an interval of 0.593 to 0.629 from windows and 0.575 to 0.645 from studies.
Then report: which baseline each claim needs, and why the corrected R² says nothing about the head.
On one window the age head is 0.18 years worse than the mean age, with a 95 percent interval from 0.45 better to 0.83 worse. Averaged over a person's windows it is 0.61 better, from 0.03 worse to 1.25 better. What can you claim?
Why do the heads' intervals come from resampling studies rather than windows or people?
A paper reports a corrected brain-age R² of 0.9 from EEG. What do you ask?
A classifier's AUC is 0.62. With equal variances and a threshold midway, what are its sensitivity and false positive rate? How often is a positive call right when the positive group is 50 percent of those tested, and when it is 10 percent?
What must a model card for the sex head say?
Hanley, J. A., and McNeil, B. J. (1982). The meaning and use of the area under a receiver operating characteristic (ROC) curve. Radiology.
DeLong, E. R., DeLong, D. M., and Clarke-Pearson, D. L. (1988). Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach. Biometrics.
Efron, B., and Tibshirani, R. J. (1993). An Introduction to the Bootstrap. Chapman and Hall.
Le, T. T., and others (2018). A nonlinear simulation framework supports adjusting for age when analyzing BrainAGE. Frontiers in Aging Neuroscience.
Mitchell, M., and others (2019). Model cards for model reporting. Proceedings of the Conference on Fairness, Accountability, and Transparency.
Smith, S. M., Vidaurre, D., Alfaro-Almagro, F., Nichols, T. E., and Miller, K. L. (2019). Estimation of brain age delta from brain imaging. NeuroImage.
Sun, H., and others (2019). Brain age from the electroencephalogram of sleep. Neurobiology of Aging.
Butler, E. R., and others (2021). Pitfalls in brain age analyses. Human Brain Mapping.
Engemann, D. A., and others (2022). A reusable benchmark of brain-age prediction from M/EEG resting-state signals. NeuroImage.
Collins, G. S., and others (2024). TRIPOD+AI statement: updated guidance for reporting clinical prediction models that use regression or machine learning methods. BMJ.
IntoMind (2026). Heads for encoder e2f9b60f411f0e73. github.com/intomind/eeg-foundation-model-heads.
Part 2 · Bits · Chapter 8
This topic is part of the college and graduate courses. The Age Detector lab covers what middle and high school students need.