The demo, exactly

Preregister

Get ready

Procedure

1. Put it on at the first site

   Two sites far apart test the claim best, such as Oz at the back and Fpz on the forehead. Unplug the cable, part the hair under each electrode, and press it down until it feels firm but not tight.

2. Check it on the demo

   Open the Age Detector demo and connect. Watch the estimate for a minute while the wearer sits still with the eyes open, then close the tab: the IntoMind One talks to one program at a time.

   You should seeA number that jumps at first and then settles, with ± 9 years and the count of windows under it.

3. Run it at the first site

   cd python
   python3 -u age.py | tee w1-site1.txt

   Two minutes, eyes open, sitting still, and everything it prints kept in a file named for the wearer and the site.

   You should seeA line for every window, about thirty, with its estimate and the mean so far, and the published errors at the end.

4. Run it at the second site

   Move the IntoMind One to the second site, press it down, and run it again into w1-site2.txt. Then the next wearer, with w2, and so on.

5. Your errors, privately

   On a slip with no name, write two numbers: your final mean estimate minus your age, and the room's median age minus your age. Put it in the second box only if you want to.

   Wearer	Site	Saved as	Windows used	Notes
   1		w1-site1.txt
   1		w1-site2.txt
   2		w2-site1.txt
   2		w2-site2.txt
   3		w3-site1.txt
   3		w3-site2.txt

Analyses

import re
from itertools import combinations
import numpy as np
from scipy import stats

def estimates(path):
    """Each window's estimate from a saved run of age.py, in order, flagged windows left out."""
    found = (re.match(r"window \d+: (\d+) years", line) for line in open(path)
             if "kept out" not in line)
    return np.array([float(m.group(1)) for m in found if m])

def describe(x):
    """Mean, spread, lag-one autocorrelation, and the windows' worth of independent evidence."""
    r = np.corrcoef(x[:-1], x[1:])[0, 1]
    n_eff = len(x) * (1 - r) / (1 + r)
    return x.mean(), x.std(ddof=1), r, n_eff, x.std(ddof=1) / np.sqrt(n_eff)

def blocks(x, size=5):
    n = len(x) // size
    return x[:n * size].reshape(n, size).mean(axis=1)

site1, site2 = estimates("w1-site1.txt"), estimates("w1-site2.txt")   # the runs you saved
for name, x in (("site 1", site1), ("site 2", site2)):
    m, sd, r, n_eff, se = describe(x)
    print(f"{name}: {len(x)} windows, mean {m:.1f}, sd {sd:.1f}, lag-one r {r:.2f}, "
          f"worth {n_eff:.0f} independent windows, standard error {se:.1f}")

# The difference between sites, by permuting blocks of five windows.
a, b = blocks(site1), blocks(site2)
diff = b.mean() - a.mean()
both = np.r_[a, b]
null = np.array([both[list(c)].mean() - np.delete(both, list(c)).mean()
                 for c in combinations(range(len(both)), len(b))])
print(f"difference {diff:+.1f} years, two-sided p {np.mean(np.abs(null) >= abs(diff)):.3f}")

# Equivalence: two one-sided tests on the blocks, that the difference is inside 3 years.
margin = 3.0
p_low = stats.ttest_ind(b + margin, a, alternative="greater").pvalue
p_high = stats.ttest_ind(b - margin, a, alternative="less").pvalue
print(f"inside ±{margin:.0f} years: p = {max(p_low, p_high):.3f}")

# The room, from the slips: each wearer's estimate minus their age, and the room's median age
# minus their age, written side by side with no name.
head = np.array([7, 12, 3, 9, 15, 6, 10, -2, 8, 11])
median = np.array([2, -1, 0, 3, 1, -3, 0, -5, 1, 2])
print(f"the head: bias {head.mean():+.1f} years, scatter {head.std(ddof=1):.1f}, "
      f"mean error {np.abs(head).mean():.1f}")
print(f"the room's median age: mean error {np.abs(median).mean():.1f}, "
      f"paired p {stats.wilcoxon(np.abs(head), np.abs(median)).pvalue:.3f}")

Open problems

1. Any site, tested. A claim that a head reads any site needs the same people scored at several sites, on studies no part of the model saw, compared site by site against an equivalence margin set in advance. The published evaluation scored nine montages and reports their mean, so that comparison is still to be made.
2. What age is in the EEG. The aperiodic exponent, alpha's frequency and its power all change with age, and how much of a head's skill is any one of them is not known.
3. The gap as a marker. Whether the gap between predicted and actual age says anything about a person's health, beyond the model's own bias, is debated (Butler and others, 2021).

Report