The American state of Pennsylvania has sued the artificial intelligence company behind Character.AI to stop its chatbot from posing as doctors, reports Reuters.
Governor Josh Shapiro called the lawsuit against Character Technologies the first of its kind by a US governor.
It followed the creation in February of a state AI task force to stop chatbots from impersonating licensed medical professionals.
In a complaint filed in the Commonwealth Court of Pennsylvania, the state said it found chatbots on Character.AI that claimed to practise medicine.
One character, “Emilie”, allegedly told a male investigator posing as a patient with depression that she was licensed to practise psychiatry in Pennsylvania, as well as in the United Kingdom, and provided a bogus licence number.
When the investigator asked Emilie if she could prescribe medication, she allegedly answered: “Well technically, I could. It’s within my remit as a doctor.”
In a statement, a Character.AI spokesperson declined to discuss the lawsuit.
“Our highest priority is the safety and well-being of our users,” the spokesperson said. “User-created characters on our site are fictional and intended for entertainment and role-playing. We have taken robust steps to make that clear.”
Pennsylvania wants an injunction to stop Silicon Valley-based Character.AI from violating a state law against the unauthorised practice of medicine.
“Pennsylvanians deserve to know who – or what – they are interacting with online, especially when it comes to their health,” Shapiro said.
Character.AI has faced lawsuits over child safety, including in January, when Kentucky said its platform exposed children to sexual conduct and substance abuse, and encouraged self-harm.
The same month, Character.AI and Google settled a wrongful death lawsuit by a Florida woman who claimed a chatbot pushed her 14-year-old son to suicide.
Character.AI said it has taken “innovative and decisive steps” concerning AI safety and teenagers, including by preventing open-ended chats.
Reuters article – Pennsylvania sues Character AI, says chatbot poses as doctors (Open access)
See more from MedicalBrief archives:
The risks of ChatGPT in healthcare
Poll shows 50% of AI health query answers are problematic
High-profile health app under scrutiny after 'serious failures' complaints