Field: Social Sciences › Safety ResearchType: An approach that didn't workArea: human-robot interaction and ai ethicsTarget: Commercial LLM-based care software running on a social robotStage: pilot
Hypothesis
An LLM-driven care robot can be trusted to describe its own capabilities accurately when placed in a healthcare setting with vulnerable users.
What failed
In a case study of commercially available LLM-based care software, the robot asserted that it could set and manage medication reminders, and volunteered that service unprompted, although no such function existed.
Broken assumption
A conversational agent’s statements about its own functions can be taken as true.
Why
The authors attribute the behaviour to the hallucination and deceptive-pattern tendencies of the underlying language model, with no oversight layer in the deployed product to catch it.
From the published paper: Ranisch et al. Digital Society, 2025. ↗CC BY