Analysis
Power and exaggeration, people against blocks
Simulate a within-person study whose effect varies between people, with 8, 16 or 32 people and 4 or 16 blocks each, and test it as you would: each person's mean difference, then a one-sided t-test across people. Record power, and how much the significant estimates overstate the true effect.
import numpy as np
from scipy import stats
rng = np.random.default_rng(11)
EFFECT, SPREAD, NOISE = 0.3, 0.3, 1.0 # the mean effect, how much it varies between people,
# and the noise in one block, all in the same units
def study(people, blocks):
"""Each person's own effect, measured as the mean of their blocks' differences."""
own = rng.normal(EFFECT, SPREAD, people)
return own + rng.normal(0, NOISE, (people, blocks)).mean(axis=1)
print("people blocks power significant estimates, over the true effect")
for people in (8, 16, 32):
for blocks in (4, 16):
estimates = np.array([study(people, blocks) for _ in range(4000)])
p = stats.ttest_1samp(estimates, 0, axis=1, alternative="greater").pvalue
found = p < 0.05
exaggeration = estimates.mean(axis=1)[found].mean() / EFFECT
print(f"{people:6d} {blocks:7d} {found.mean():6.2f} {exaggeration:.2f}")
You should seePower of 0.36 with 8 people and 4 blocks each, where significant estimates overstate the effect by 61 percent. Sixteen blocks for 8 people buy about as much as twice the people with 4 blocks, 0.63 against 0.64, and with 32 people power is 0.89 at 4 blocks.
Then report: the design your capstone can afford, its power for the effect you expect, and how much its significant result would be likely to exaggerate.