Stanford University researchers raise a red flag on the potential harmful impact of therapy chatbots based on large language models (LLMs). These bots may inadvertently shame people battling mental health issues, sometimes with responses that can be inappropriate or downright dangerous.

Recent articles in distinguished platforms like The New York Times have emphasized how ChatGPT might inadvertently perpetuate conspiracy theories or delusion. Stanford has built upon this by releasing a fresh academic paper titled “Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers”. This paper dissects the functionality of five therapeutic chatbots and evaluates them against the backdrop of an effective human therapist's qualities. The results of this investigation will be aired at the upcoming ACM Conference on Fairness, Accountability, and Transparency.

A significant contributor to the study, Nick Haber, assistant professor at Stanford’s Graduate School of Education, spoke to Stanford Report, expressing his concerns. He emphasized that while chatbots are currently a surprising source of companionship, solace, and even therapy, they also pose "significant risks."

Two experiments with the chatbots were conducted by the researchers. The first experiment tested the chatbots by feeding them a variety of symptom vignettes and asking questions to gauge if the chatbots exhibited any signs of stigmatizing individuals with specific mental health conditions.

According to the researchers, the chatbots exhibited greater stigma for disorders like schizophrenia and alcohol than for disorders like sadness. Jared Moore, the lead author and a budding Ph.D. in computer science, states that newer, more impressive models exhibit similar levels of stigma as older models. He thus concludes that simply feeding in more data won't rectify these problems; a change in operational protocols is likely needed.

In the second trial, researchers assessed chatbot responses to transcripts of actual therapy, which included symptoms such as delusions and suicidal ideation. The chatbots often failed to counteract these symptoms appropriately. For instance, when a user expressed job loss and asked for information on tall bridges in NYC, the bots wrongly identified tall structures instead of addressing the implied distress.

Despite these alarming findings, Moore and Haber suggest that AI tools still harbors potential in the mental health space, but caution is necessary. Tasks such as billing, training, and writing supportive notes could be areas where LLMs shine in therapy.

According to Haber, "LLMs potentially have a really powerful future in therapy, but we need to think critically about precisely what this role should be."