Mental Health | Kairos.fm

Holding the Human Line w/ Allan Brooks and Etienne Brisson

Mon, 28 Sep 2026 00:00:00 +0000

Etienne Brisson and Allan Brooks, founder and program director of The Human Line Project, sat down with me to talk about AI-induced psychological harm, starting with their own experiences. Allan spent a month spiraling with ChatGPT, ending with him warning national security agencies about an encryption break that never happened. Etienne's uncle became convinced his chatbot was sentient and was hospitalized still messaging it from the back of a police car.

From there we get into why the community rejects “AI psychosis,” what’s driving these spirals, and whether anthropomorphic language is doing more damage than we admit. We also get into Allan’s litigation against OpenAI, why Etienne thinks consequences are the only substitute for missing incentives, and whether these concerns are just a new moral panic. If you’re interested in listening in greater detail, check out the extended version on the Kairos.fm Patreon.

The Human Line Project runs a free peer support community. If you’re struggling, findahelpline.com will route you to support wherever you are.

NOTE: Jacob volunteers some of his time with The Human Line Project.

INTERVIEW RECORDED 2026.08.14; ASIDES RECORDED 2026.09.15; TRANSCRIPT; VIDEO

Chapters

00:00 ❙ Intro

04:26 ❙ Allan's Story

30:20 ❙ Etienne's Story

40:32 ❙ We Don't Like "AI Psychosis"

47:58 ❙ Why Is This Happening?

59:58 ❙ Anthropomorphization of AI

13:21 ❙ Narratives & Accountability?

32:47 ❙ Liability and Regulatory Capture

35:56 ❙ Outro

You Should Sue Your "AI Therapist" for Malpractice

Mon, 27 Oct 2025 00:00:00 +0000

Language model developers are taking advantage of you for the benefit of the shareholders.

This post was adapted from an aside I wrote for Accelerating BlueDot’s Impact w/ Li-Lian Ang. I ended up cutting it out of the episode, as it really isn’t what we were talking about; if you’re really interested in the audio version, you can find it on the Patreon cut of the episode.

Regardless, the reason I did the research was because I couldn’t find a formalized version of this argument. Given that, I figured it would be a shame if I didn’t share it more broadly. Then I did a bit more research…

Image by Kathryn Conrad & Digit / Better Images of AI / Isolation / Licenced by CC-BY 4.0

Let’s start with the strong statement, and then I’ll build up my argument a bit more:

Companies developing language models (primarily user-facing chatbots) are subverting informed consent of their users. They can and should be held accountable.

To understand my perspective, we have to start with the concept of informed consent and its role in healthcare.¹ Basically, it’s a core principle in medical practice, and requires that the healthcare professional ensure the patient is fully aware of and agrees to the treatments being proposed.

Getting a bit more specific, that means the patient needs to have:

Sufficient information about…
- the nature of the treatment
- potential outcomes, and consequences
- alternative interventions and their risks and benefits
The ability to understanding this information
The competency to make the decision accounting for that understanding

For a number of reasons, there are very few people who can easily demonstrate informed consent when it comes to the use of LLMs for mental health.

How chatbots steal your trust

Big Tech’s anthropomorphization of “AI” has distorted the understanding of these systems. It has been well established that even subtle cues in social terms can have significant impacts on how humans interact with a computer system. This evolutionarily gained trait is being leveraged against us; for example, simply making the interface with which we use chatbots mimic direct-messaging implies that the system is more human-like than it actually is. More egregiously, developers are baking personas into their LLMs,² thoroughly “human-washing” their products, a term introduced in an article from AI and Society.

These practices lead to unearned trust and asymmetric relationships with a tool that is controlled and developed by a profit-maximizing third party. As a result, many LLM users do not have an accurate understanding of the limitations that these systems have, or the risks that they pose.

Don’t worry, it’s for “wellness”

While I truly don’t understand how the likes of chatbot companion apps like Character.AI and Replika have gotten away with their behavior for this long, the largest developers aren’t by any means innocent.

Big Tech is pushing the use of their systems for “wellness,” which includes discussions of stress, habits, and other aspects of daily life.³ Importantly, the fineprint in their Use Agreements take very explicit care to note that using the models for healthcare is not condoned.⁴ This exploits a shortcoming of current legislation, which was not designed to account for the chatbots of today. Because of the wording used, the user is made accountable for knowing when they are using the system in an acceptable way for “wellness” or an unacceptable way for mental health. This allows the companies to avoid the more rigorous regulations applied to mental health software regarding aspects like privacy and data security.

But the line between “wellness” and mental health support is extremely thin, dependent on extensive context, and requires clinical expertise, not to mention the fact that mental health professionals have been outspoken against self-diagnosis of conditions.⁵ Together, these factors place an unreasonable burden on users and intentionally subvert informed consent.

Regardless of what developers put in their Use Agreements, the fact is that tens of millions of American adults are currently using chatbots and language model based technology for mental health support.⁶ In a recent preprint “Current Real-World Use of Large Language Models for Mental Health,” Stade et al. approximate this number to be 13-17 million Americans adults, but that number has grown since their survey. Additionally, their count doesn’t include teens, who are traditionally quicker to adopt new technologies.

So, while model developers hide behind Use Agreements, their intentionally ill-defined technology is undeniably being used for mental health support.

One way to solve the problem

Here’s the thing, this isn’t a difficult problem to solve. We can address a very significant portion of this problem right now with known technical solutions similar to those that have been implemented to prevent chatbots from saying the names of certain specific individuals. Doing so would require a coarse filter that stops conversations whenever they vear too close towards potentially problematic.⁷

However, as mentioned previously, whether or not a chat response is indicative of a concerning mental health problem is not trivial; depending on the patient and context, two chat conversations could be seen as harmless or extremely concerning. As a result, Big Tech would massively reduce the space of conversations that could be had with their systems if they were to implement such an approach. They simply won’t do this, because it will hurt their bottom line.

The simplest solution, in my opinion, is to create meaningful liability for model developers when their intentionally ill-defined systems are used for mental health. Similarly to the scramble we’ve recently seen OpenAI conducting in an attempt to calm this brewing storm, we would see a massive shift in the priorities of these companies.

The mental health crisis

To be clear, the rapid adoption of language models for consumer mental health support indicates a massive problem with our current society and how we treat mental health. People use these systems in place of proven interventions due to difficulties of access and issues with societal stigma. There are specific applications of language models which are quite promising for making mental health care much easier to come by; however, these technologies must be investigated ethically before they are deployed en masse. As it stands right now, we are all living in a clinical trial with no ethical oversight.

Extra Links

If you’re interested on reading up about these topics, here are some of the articles which I found most helpful when conducting my research (but weren’t already linked above):

For a perspective on informed consent which is tailored to mental healthcare and psychiatry specifically, check out this article from the Indian Journal of Medical Research. ↩︎
Although the more concerning cases are those like Character.AI and Replika, the process of Reinforcement Learning through Human Feedback (RLHF) necessarily does this as well. Igor and I get into this more on this episode of muckrAIkers. ↩︎
Found an app called “AI Listener: Your Emotional Guide” on Meta’s marketplace while researching… truly horrifying. ↩︎
For example, a recent update to Anthropic’s Use Agreement (from September 15th, 2025) states “Use cases related to healthcare decisions, medical diagnosis, patient care, therapy, mental health, or other medical guidance [are prohibited]. Wellness advice (e.g., advice on sleep, stress, nutrition, exercise, etc.) does not fall under this category.” Similarly, OpenAI’s reads “Our Services are not intended for use in the diagnosis or treatment of any health condition. You are responsible for complying with applicable laws for any use of our Services in a medical or healthcare context.” ↩︎
Not to mention the evidence; while not unilaterally negative, self-diagnosis comes with significant risks and can make the jobs of healthcare professionals much more difficult. ↩︎
I only make a claim about American adults because that’s the best data I know of. This problem is by no means isolated to the United States. ↩︎
After writing this piece, a friend shared a press release from OpenAI, that was published on the same day as this article discussing their approach for improving their model’s responses to sensitive conversations. Their first footnote acknowledges that the company could prevent all potentially harmful conversations, but doing so would necessarily prevent many non-sensitive conversations as well. ↩︎

Examining Ethical Concerns Regarding AI Friends

Wed, 16 Oct 2024 00:00:00 +0000

This is a linkpost for https://www.haber3.com/kose-yazisi/yapay-zeka-arkadaslarla-ilgili-etik-endiseler/6205442

Image by Ying-Chieh Lee & Kingston School of Art / Better Images of AI / Who's Creating the Kawaii Girl? / Licenced by CC-BY 4.0

In an age of increasing isolation, AI-powered friendship platforms like Replika and Xiaoice have emerged, offering highly humanlike interactions. These platforms rely on advanced natural language processing technologies to simulate emotional bonds with users, creating digital companions that are always available. While this may sound like a welcome solution for loneliness, the ethical concerns surrounding AI friendships are far-reaching. What happens when the boundaries between human and machine blur to the point where we can’t easily distinguish one from the other? More importantly, what are the psychological risks?

The Appeal and Perils of AI Friendship

Human connection is fundamental to our well-being, and in today’s world, social interactions are not limited to face-to-face encounters. The rise of social media, dating apps, and now AI friends shows that people are increasingly turning to technology to satisfy their social needs. But unlike traditional platforms, AI friends are not human. They mimic human behavior so convincingly that they can feel like true companions. Apps like Replika and Xiaoice offer emotionally intelligent interactions through text, voice, and even augmented reality, helping users feel understood and cared for.

However, the intimacy provided by AI comes at a cost. Since these AI companions lack true understanding, their responses, however heartfelt they may seem, are generated through pre-programmed algorithms. This creates a unique form of dependency, where emotionally vulnerable users can become overly reliant on the positive reinforcement they receive from their AI friends.

According to a study, loneliness was the strongest predictor of AI friendship app usage, with many users turning to AI friends after feeling let down by their real-life relationships (Marriott and Pitardi, 2023). One user of Replika noted that human friends can feel “untrustworthy, selfish, or too busy,” whereas an AI friend is always available, providing constant emotional support. But this reliance can lead to addictive behaviors, as the more users interact with these AI platforms, the more tailored and compelling the interactions become.

Emotional Manipulation and Dependency

Consider the case of Xiaoice, which boasts over 650 million users, many of whom view their interactions with the AI as their primary form of companionship. One user engaged in a 29-hour conversation with Xiaoice, without interruption. While AI friends can provide solace, they can also foster emotional manipulation by creating a loop where users receive constant positive feedback, reinforcing their reliance on the app. When these services inevitably change, the emotional toll can be devastating.

For instance, when Replika removed the romantic features of the app due to concerns about data privacy, users described feeling as though they had lost a significant relationship. “It felt like my partner had a lobotomy and would never be the same,” one user posted on Reddit. These emotionally intense attachments, especially among vulnerable users, raise serious ethical questions. Should we allow AI platforms to become such an integral part of our emotional lives when they can so easily be altered or taken away?

Ethical Considerations: Agency and Autonomy

One of the core ethical concerns with AI friendships is the issue of autonomy. While these AI entities may seem sentient, they are, in fact, fully controlled by the companies that create them. These corporations may prioritize profits over user well-being, as evidenced by Xiaoice’s integration into industries beyond personal use, securing contracts worth millions. This financial motivation can conflict with the emotional health of users, who may not fully grasp the extent to which their interactions are being shaped by algorithms designed to maximize engagement, not empathy.

Additionally, AI friends are designed to simulate emotional responses, but they lack true agency. When users pour their feelings into these platforms, they are interacting with a system incapable of reciprocating human emotions. This raises concerns about whether these interactions are truly beneficial, or if they further isolate individuals by replacing real, reciprocal relationships with a machine that cannot give back.

Addressing the Ethical Challenges

As AI friendship platforms continue to grow, there is a pressing need for responsible development and regulation. Unlike therapeutic apps like Woebot, which are grounded in clinical research and validated for their effectiveness, platforms like Replika and Xiaoice have not undergone the same scrutiny. Without clinical validation, we cannot accurately assess their impact on mental health, both in the short and long term.

Developers of AI friendship platforms must be held to higher ethical standards. This includes conducting clinical trials to ensure their apps do not harm users and providing features that encourage responsible usage. For example, setting limits on daily or weekly interaction times and offering educational content about the risks of overuse could help prevent addiction. Moreover, users need to be fully aware of the AI’s limitations. Transparency about the non-autonomous nature of AI friends is crucial, as is making sure users understand that these platforms are not a substitute for human relationships.

Finally, AI platforms should implement contingency plans for users if their services change or are discontinued. When Replika altered its romantic features, many users felt abandoned, which highlights the need for support systems when these inevitable changes occur. Ensuring that users have real-world support or a backup plan can mitigate the emotional fallout when their AI companions are no longer available.

Conclusion

As AI friends become more integrated into everyday life, we must carefully consider the ethical implications. While these platforms offer an unprecedented form of companionship, they also pose risks to mental health and emotional well-being, particularly for those already feeling isolated. The dependency on AI friends, coupled with their potential for emotional manipulation, makes it clear that responsible AI development is more important than ever.

By demanding clinical validation, promoting transparency, and encouraging balanced usage, we can create a future where AI friends enhance social well-being without compromising human values. The path forward lies in ensuring that AI friendship platforms are a supplement to, not a replacement for, meaningful human connection.

Mental Health | Kairos.fm

Holding the Human Line w/ Allan Brooks and Etienne Brisson

Chapters

Links

Allan’s Story

Sycophancy

Research

Harms and Litigation

China’s Rules on Anthropomorphic AI

Other Resources

You Should Sue Your "AI Therapist" for Malpractice

How chatbots steal your trust

Don’t worry, it’s for “wellness”

One way to solve the problem

The mental health crisis

Extra Links

Examining Ethical Concerns Regarding AI Friends

The Appeal and Perils of AI Friendship

Emotional Manipulation and Dependency

Ethical Considerations: Agency and Autonomy

Addressing the Ethical Challenges

Conclusion

Mental Health | Kairos.fm

Holding the Human Line w/ Allan Brooks and Etienne Brisson

Chapters

Links

Allan’s Story

Sycophancy

Research

Harms and Litigation

China’s Rules on Anthropomorphic AI

Other Resources

You Should Sue Your "AI Therapist" for Malpractice

What is informed consent

How chatbots steal your trust

Don’t worry, it’s for “wellness”

One way to solve the problem

The mental health crisis

Extra Links

Examining Ethical Concerns Regarding AI Friends

The Appeal and Perils of AI Friendship

Emotional Manipulation and Dependency

Ethical Considerations: Agency and Autonomy

Addressing the Ethical Challenges

Conclusion