Hypothesis

Associations between individual differences in brain structure or function and cognitive or mental-health phenotypes can be reliably detected at the sample sizes typical of neuroimaging studies (median about 25).

What failed

Across three large datasets totalling roughly 50,000 individuals, brain-wide association effects were smaller than previously thought, leaving typical studies underpowered, with inflated effect sizes and replication failures. Replication rates improved and inflation fell only as samples reached the thousands.

Broken assumption

Sample sizes adequate for classical brain mapping are also adequate for brain–behaviour association studies.

Why

The authors attribute the failures to genuinely small brain–phenotype associations plus variability across population subsamples, and find more robust effects for functional than structural MRI, cognitive tests than mental-health questionnaires, and multivariate than univariate methods.

From the published paper: Marek et al. Nature, 2022. ↗CC BY

Hypothesis

Machine-learning models trained on the small datasets typical of digital mental-health studies give trustworthy estimates of how well dropout can be predicted.

What failed

Learning curves on 3,654 users of one digital intervention showed that datasets of N ≤ 300 overestimated predictive power; for uninformative features, in-sample performance even fell as datasets grew, and sophisticated models overfitted small samples. Overfitting eased at N = 500, but performance did not converge until N = 750–1,500.

Broken assumption

Prediction performance measured on a few hundred users is a reliable estimate of real predictive power.

Why

The paper attributes the inflated estimates to overfitting in small samples, especially with complex models, and proposes minimum dataset sizes of N = 500–1,000.

From the published paper: Zantvoort et al. npj Digital Medicine, 2024. ↗CC BY

Hypothesis

Short 3.4-second bursts of taVNS would both widen the pupil and lower EEG alpha power, as Sharon et al. (2021) reported, indicating noradrenergic activation.

What failed

In a sham-controlled crossover with 29 volunteers using the original analysis scripts, pupil dilation replicated (p = .009, BF = 7.45) but the drop in EEG alpha power did not (p = .37; BF10 = 0.28, about four times more likely under the null).

Broken assumption

Brief taVNS reliably suppresses EEG alpha as a cortical readout of noradrenergic activation.

Why

The abstract offers no mechanism for the missing alpha effect; it calls for more direct replications of influential taVNS studies.

From the published paper: Lloyd et al. Brain stimulation, 2023. ↗CC BY

Hypothesis

Regression-based variable-selection methods can pick the true health-relevant exposures out of hundreds of correlated ones in an exposome analysis.

What failed

In simulations with 237 realistically correlated exposures and 0 to 25 true predictors, elastic net and sparse partial least squares averaged 76% sensitivity with a 44% false discovery proportion, GUESS and the deletion/substitution/addition algorithm 81% with 34%, and the environment-wide association study approach reached an 86% false discovery proportion; performance fell considerably when correlation between covariates was high.

Broken assumption

Regression-based selection can separate causal exposures from correlated ones at exposome scale.

Why

The authors attribute the limits to correlation between exposures, which none of the methods resolved well enough to distinguish true predictors from their correlates.

From the published paper: Agier et al. Environmental Health Perspectives, 2016. ↗Public domain

Hypothesis

Exposure to achievement-related words activates a high-performance goal that improves performance on a demanding cognitive task.

What failed

A direct replication (n = 98) found no effect of priming, with means in the opposite direction to the original; a second experiment adding the five-minute delay between priming and test that the original authors said enhanced the effect also found no evidence of priming (n = 66).

Broken assumption

Achievement words reliably activate a performance goal strong enough to change behaviour.

Why

The authors offer no mechanism and argue instead that the goal-priming literature, cited well over 1100 times for this paper alone, needs sceptical scrutiny.

From the published paper: Harris et al. PLoS ONE, 2013. ↗CC BY

Hypothesis

A long-circulating liposomal doxorubicin would keep the drug available to the tumour long enough, after microbubble-mediated focused ultrasound opened the blood-brain barrier, to extend survival where a single dose of free doxorubicin had not.

What failed

In orthotopic HSJD-DIPG-07 xenografts, histology 24 hours after treatment showed liposome extravasation in healthy controls but not in the tumour-bearing animals; bioluminescence signal did not fall after treatment, and survival did not differ significantly between treated and control animals (p = 0.3) for free doxorubicin or either liposomal formulation.

Broken assumption

An ultrasound-opened blood-brain barrier lets circulating liposomal drug into the tumour.

Why

The authors report no treatment effect from any formulation combined with ultrasound, with the histology indicating the liposomes did not reach the tumour.

From the published paper: Haumann et al. Translational Medicine Communications, 2022. ↗CC BY

Hypothesis

Access to parks, other natural space and a private garden should bend the trajectory of children’s social, emotional and behavioural development.

What failed

In 2909 urban Scottish children followed from age 4, modelling interactions of park area, total natural space and garden access with age and age squared gave little evidence of any influence on the trajectory of difficulty scores between 4 and 6, even though cross-sectional associations with garden access and natural space were present.

Broken assumption

Greenspace keeps improving children’s development over the period measured.

Why

The authors suggest any beneficial influence had already taken effect at younger ages than the window observed.

From the published paper: Richardson et al. Environmental Research, 2017. ↗CC BY

Hypothesis

Priming resource scarcity would make East Asian participants more likely to judge ambiguous East/South Asian faces as out-group members, as scarcity priming did for White participants judging White/Black faces.

What failed

In a preregistered conceptual replication in Japan (N = 769, power 0.79 for d = 0.2), the pattern matched the original but the priming effect was not significant (d = 0.1).

Broken assumption

Resource-scarcity priming shifts racial categorisation of ambiguous faces in a substantial way across cultures.

Why

The abstract reports the null and says reasons are discussed, without stating one; it also differed from the original in population and in the faces used.

From the published paper: Takebe et al. Current Research in Ecological and Social Psychology, 2023. ↗CC BY

Hypothesis

Twelve weeks of endurance exercise, vitamin D, or both would reduce insulin resistance (HOMA-IR) in middle-aged adults with type 2 diabetes.

What failed

Across 61 participants randomised to control, exercise, vitamin D or both, none of the interventions improved HOMA-IR, the primary outcome; only fasting glucose fell in the combined group (−11.9%).

Broken assumption

Endurance exercise and vitamin D lower insulin resistance in type 2 diabetes within 12 weeks.

Why

The authors suggest the null primary result may be due predominantly to large inter-individual variability in response, which baseline plasma lipids predicted well in exploratory models.

From the published paper: Sun et al. Nutrients, 2023. ↗CC BY