'AI chatbot' has become an almost uselessly broad category. A general-purpose assistant and a product intentionally built for mental health can present the same interface: a text box, a response, another text box. At the surface, both can sound empathetic, remember context, and discuss deeply personal subjects. That similarity makes it easy to evaluate them by the quality of the conversation alone.
For mental health, I think that is the wrong starting point. The most important differences often sit outside the response itself: what the system was designed to do, what it is explicitly not supposed to do, how it recognizes risk, which decisions are reserved for humans, and whether there is an actual care pathway when conversation is no longer enough.
This is not an argument that general-purpose AI is inherently unsafe. People will bring emotional and mental health concerns to all kinds of conversational systems, and every responsible system should take that reality seriously. The more precise question is why someone seeking mental health support might choose a product built specifically for that use case rather than one whose scope is intentionally much broader.
Purpose shows up in the boundaries
A purpose-built product begins with a defined use case. That sounds basic, but it forces a set of design decisions that a general conversational system does not necessarily have to make in the same way. Which presentations are inside the appropriate scope of the experience? What should happen when someone appears to be in acute crisis? What about clinically significant situations that are not emergencies? When does the AI stop being the right tool?
Those questions are not solved by making the underlying model more capable. In some cases, greater fluency can make the need for boundaries easier to miss. A model may be perfectly capable of continuing a conversation with someone whose symptoms warrant human clinical involvement. The fact that it can continue does not mean it should be the only support available.
Safety has to be a system, not a disclaimer
In Thoughtful, those questions led us to build safety across multiple points in the member journey. An intake screener is designed to identify certain acute-risk presentations before a new user enters the product. During active sessions, a real-time detection layer monitors for risk to self, risk to others, and risk from others. After sessions, a separate high-acuity evaluator reads the completed conversation for patterns that may only become visible in the full transcript.
The reason for multiple layers is clinical, not cosmetic. Acute risk can appear suddenly during a session. Clinical complexity can emerge gradually across it. A single moderation rule or keyword list is poorly suited to both jobs, because the signal, timing, and appropriate response are different.
The layers also share a constraint that matters more than the model choice underneath them: AI detection is not the final authority on escalation. When a concern is surfaced, a clinician applies judgment. No automated system independently initiates clinical outreach. That is an architectural boundary, not a promise that the model will behave responsibly if prompted the right way.
Clinical design changes ordinary product decisions
Purpose-built mental health AI also shows up in decisions that look like implementation details. In a high-acuity referral flow we are currently building, for example, the language model does not generate helpline names, phone numbers, or URLs. It outputs the clinical category, and the application retrieves the actual resource from a vetted configuration. There is no therapeutic value in letting a model improvise a phone number, so we remove that possibility from the system.
Privacy creates another constraint. A member's assigned provider does not automatically see their AI conversations without consent. That means 'human in the loop' cannot simply mean forwarding every concerning transcript to the therapist. Clinical safety review and the provider relationship have to coexist with a privacy boundary that members have told us matters to them.
These choices are less visible than the tone of the chatbot. They are also the choices I would want to understand before deciding whether a mental health AI was ready for a population.
The care pathway is the part people skip
The hardest test comes when the AI is no longer enough. A general instruction to 'seek professional help' can be clinically sensible, but it leaves the user with the hardest part of the work: figuring out what help, where to find it, and whether it relates to care they already have.
We are beginning to study a more blended model through The Blend, an early research study pairing an AI-first experience with periodic check-ins from licensed providers. The study is designed to examine whether that model can be safe, effective, and engaging, and what the AI-to-provider handoff needs to look like in practice. It is too early to report participant outcomes or claim that the study has established which work belongs with AI and which belongs with providers.
The design intent, though, points to a larger principle. If AI and human providers can live in the same care ecosystem, routing someone toward human care does not have to be treated as churn. The product can optimize for the right next form of support rather than for keeping every interaction inside the chatbot.
The questions I would ask a vendor
If I were evaluating mental health AI for a workforce, I would spend less time asking for the model name and more time asking about the system around it. What was the AI designed for? What is explicitly outside its scope? How does it detect acute risk, and how does it detect clinical complexity beyond crisis? When the system flags something consequential, who makes the final decision? What happens when a safety component fails? Can a member get to additional care without surrendering the privacy of the AI conversation?
Those questions do not produce the cleanest product-comparison table. They do produce a much clearer picture of what has actually been built.
Mental health AI should not be evaluated only by whether it can hold a convincing conversation. The more important test is whether the product has been designed for the moments when a convincing conversation is not enough.