Hypothesis

As the time frame of a self-report question lengthens, people abandon episodic recall for semantic, belief-based answers, which should show up as a flattening of response times and ratings at longer time frames.

What failed

469 adults rated eight emotions and symptoms over 12 time frames from right now to in general. Response times and ratings rose with longer time frames, but there was no consistent flattening or drop in slope that would mark a switch in retrieval strategy, for either emotions or symptoms.

Broken assumption

There is a detectable point at which longer self-report time frames switch people from episodic to semantic retrieval.

Why

The authors suggest people may keep trying to retrieve episodic information even for long time frames.

From the published paper: Walentynowicz et al. PLoS ONE, 2018. ↗CC BY

Hypothesis

Deleting the CD2-like 8DR gene would weaken the virulent ASFV Georgia2010 isolate, since several naturally non-hemadsorbing field isolates are less virulent in pigs.

What failed

Pigs inoculated intramuscularly or intranasally with the deletion mutant at 10^2 to 10^4 TCID50 developed disease indistinguishable from that caused by the parental virus, and viraemia levels did not differ.

Broken assumption

The CD2-like gene is what makes naturally non-hemadsorbing ASFV isolates less virulent.

Why

The authors conclude the gene is not responsible for the reduced virulence of natural non-hemadsorbing isolates, so their attenuation must come from something else.

From the published paper: Borca et al. Scientific Reports, 2020. ↗CC BY

Hypothesis

Accumulated deliberate practice would differ by skill level among student violinists, as in Ericsson et al.'s (1993) seminal study.

What failed

With a double-blind procedure and analyses suited to the design, the core finding that accumulated deliberate practice corresponded to each skill level did not replicate; the effect was substantial but considerably smaller than the original. Teacher-designed practice did not explain more variance in performance than practice alone.

Broken assumption

Differences in accumulated deliberate practice largely account for differences in expert performance.

Why

The paper reports a smaller effect than the original under tighter methods; it does not attribute the difference to a single cause and discusses the implications for deliberate-practice theory.

From the published paper: Macnamara et al. Royal Society Open Science, 2019. ↗CC BY

Hypothesis

Reminders pushed from a connected pill cap to a smart-device application would improve medication adherence in people aged 65 and over with rheumatoid arthritis.

What failed

The nested subproject could not answer its research question at all. The sample size rested on over-optimistic estimates of how many older patients owned a smart device; smartcap availability and data extraction caused logistic problems; randomization and treatment allocation errors occurred despite staff training; and hardware failures left the intervention group’s data of low quality, discovered too late because data were read in batches.

Broken assumption

An adherence sub-study can be bolted onto a running trial without its own feasibility test.

Why

The authors attribute the failure to those four causes together and recommend keeping nested subprojects simple, field-testing before the study starts and monitoring data from the beginning.

From the published paper: Hartman et al. Journal of Trial and Error, 2021. ↗CC BY

Hypothesis

Published adult statistical word segmentation experiments, including the moderators said to shape this learning, would reproduce when re-run, with effect sizes close to the originals.

What failed

Across eight experiments replicating six studies, above-chance segmentation was confirmed every time, but many theoretically important moderations of learning did not replicate and learning was generally much weaker than originally reported. Several original papers also lacked key method details or described them wrongly.

Broken assumption

The moderator effects reported in the statistical word segmentation literature are robust rather than products of small, underpowered studies.

Why

The authors weigh differences in participant populations, low power in the original studies, or a mix of these and other factors.

From the published paper: Hartshorne et al. Collabra Psychology, 2019. ↗CC BY

Hypothesis

Acceptability contrasts published by linguists in Hebrew and Japanese would hold up when rated by large samples of naive native speakers, as most English judgments did in earlier large-scale replications.

What failed

Contrasts from the literature that the authors judged questionable were rated by a large sample of naive participants; approximately half of them did not replicate.

Broken assumption

Published acceptability judgments in languages other than English are as robust as the English ones that large replications vindicated.

Why

The paper reports the failure rate only for contrasts the authors had already picked as doubtful, so it is not an estimate for all published judgments. It raises, without testing, the concern that judgments in languages other than English may be less reliable.

From the published paper: Linzen et al. Glossa a journal of general linguistics, 2018. ↗CC BY

Hypothesis

Adding a topical gentamicin-collagen sponge to systemic antibiotic therapy would improve healing of moderate or severe diabetic foot ulcer infections.

What failed

Across 88 randomized infection episodes, final clinical cure was 31/43 with the sponge against 26/45 without (p = 0.16), and in multivariate analysis adjusted for case mix the sponge was not associated with cure or improvement.

Broken assumption

Delivering an antibiotic locally into the wound adds to what systemic therapy already achieves.

Why

The authors report excellent tolerability with no attributed adverse events and only a trend towards more rapid healing, and draw no mechanistic conclusion.

From the published paper: Uçkay et al. BMC Infectious Diseases, 2018. ↗CC BY

Hypothesis

An osteochondral implant combining a melt-electrowritten polycaprolactone chondral framework with an extrusion-printed bioceramic bone anchor — the anchor already proven osteoregenerative at a non-load-bearing site in the same species — would repair articular cartilage defects in ponies, more so when seeded with cartilage progenitor cells.

What failed

Six months after implantation in Shetland pony stifle joints, biochemical and histological analysis showed minimal glycosaminoglycan and type II collagen in the chondral compartment for both cell-seeded and cell-free implants, and micro-CT showed the bone anchor had collapsed with little mineralised new bone.

Broken assumption

A bone component proven to regenerate bone at a non-load-bearing site will hold up as the anchor of a load-bearing joint implant.

Why

The authors attribute the outcome to collapse of the osteal anchor, which removed mechanical support from the chondral compartment and precluded any evaluation of whether the BMP-9-stimulated cells helped — even though the cells had produced abundant matrix during 28 days in vitro.

From the published paper: Diloksumpan et al. Journal of Trial and Error, 2021. ↗CC BY

Hypothesis

Action video-game training remedies reading difficulty in children with dyslexia, as studies in Italian and English-speaking children reported.

What failed

Fifty-four Polish children with dyslexia trained on action or phonological non-action games gained in reading speed and accuracy, phonological awareness, selective attention and rapid naming, but by no more than an untrained dyslexic control group.

Broken assumption

Improvement observed during game training is caused by the training.

Why

The authors attribute the gains to ordinary schooling-related reading development or test practice, and note that the earlier supportive studies had small samples and no untreated control group.

From the published paper: Łuniewska et al. Scientific Reports, 2018. ↗CC BY