Hypothesis
Coding how much effort respondents put into answering an open-ended question would proxy their real behaviour rather than merely stated intention, letting a vignette experiment on rewards and workplace knowledge sharing measure behaviour within subjects.
What failed
With participants answering three vignettes in a within-subject design, a strong order effect appeared: answers to the first vignette were longer on average than to later ones regardless of treatment, confounding the effort measure.
Broken assumption
Effort invested in an open-ended answer reflects the respondent's likely behaviour and not the question's position in the survey.
Why
The authors conclude the order effect makes this operationalisation unsuitable for within-subject designs, though it may still work between subjects with a single vignette each.
From the published paper: Fischer et al. Journal of Trial and Error, 2021. ↗CC BY