Hypothesis
Machine-learning models trained on the small datasets typical of digital mental-health studies give trustworthy estimates of how well dropout can be predicted.
What failed
Learning curves on 3,654 users of one digital intervention showed that datasets of N ≤ 300 overestimated predictive power; for uninformative features, in-sample performance even fell as datasets grew, and sophisticated models overfitted small samples. Overfitting eased at N = 500, but performance did not converge until N = 750–1,500.
Broken assumption
Prediction performance measured on a few hundred users is a reliable estimate of real predictive power.
Why
The paper attributes the inflated estimates to overfitting in small samples, especially with complex models, and proposes minimum dataset sizes of N = 500–1,000.
From the published paper: Zantvoort et al. npj Digital Medicine, 2024. ↗CC BY