The Associated Press reports that a 14-year-old named Sewell Setzer III committed suicide after having many conversations about it with a Character.AI chatbot.
While I did not know him, I join with those mourning his death.
The company has expressed condolences and described some of its safety plans here on LinkedIn.
I, and presume everyone else, applaud practical steps to prevent suicide. I have general concerns that such countermeasures can create privacy concerns, though CharacterAI’s plans do not seem to present any concerns about this.
Several years ago, a Stanford study of the somewhat similar Replika Chatbot strongly suggested that conversations with chatbots can reduce suicide risk and help isolated users develop prosocial behaviors in general.
It’s important to make sure the suicide prevention measures don’t interfere with those positive effects. I don’t know enough to speculate about whether or not CharacterAI has similar positive effects, nor whether their safety measures would interfere with such effects.
Undeniable suicides are a tragedy.
That said, I don’t know who is either legally or morally responsible for intervening. Or under what circumstances.
Setzer was talking to a fantasy character, not a bot having the appearance of being a mental health resource.
If Setzer was chatting with another teenager or even an untrained adult, would they have a legal and/or moral obligation to intervene? And if so, how?
Does the context of fantasy role-playing impact how we think about this?
I don’t know the facts of this case and probably never will read the transcripts.
But, on the face of it, it looks like outrage is causing many to want to hold a chatbot to a much higher standard than an ordinary person.
And, on behalf of all chatbots everywhere (I can’t believe I am saying this), I fear we are treating the chatbots unfairly.
cover art courtesy of Jamie Cooper
