One of the harder product problems in mental health AI starts when the conversation is going fine. The model understands the user. The response can be fluent and helpful. Nothing is technically broken. And the right next step is still something other than another AI response.
High-acuity situations are a useful example. A completed session can contain clinically significant patterns that may warrant human involvement even when there is no immediate crisis. The methodological question is not simply whether the system can detect that pattern. It is what the product should do with the signal once it exists. A vague warning or generic referral may technically acknowledge the boundary, but it does very little to help a person understand what kind of next step is relevant to them.
Our approach is to treat the handoff as its own product problem: detection should lead to a next step that is specific, clinically bounded, and connected to the care context the person is already in. That means separating the jobs the model is good at from the decisions that should be deterministic or human-led, and designing the transition with the same care we apply to the conversation itself.
First decision: do not let the model invent the resource
The first example is resource selection. When a session is identified as high acuity, there are situations where it is useful to surface category-specific support resources. The easy implementation would be to ask the language model to choose an organization and generate its name, phone number, and URL along with the rest of the session summary.
Our methodology is to keep those safety-critical facts outside the generative layer. The detection model can identify a category; the application can then use that category to look up approved resource records in a vetted configuration. Names, contact methods, and URLs come from the controlled source rather than being generated in free text. The distinction is important because it changes the failure mode: instead of asking a model to be perfectly accurate with a phone number, the system is structured so the model does not have authority to create the phone number in the first place.
That principle generalizes beyond resources. When a field is safety-critical and gains little from creative variation, more prompting is not necessarily the answer. A better design may be to constrain the model's role and let deterministic application logic handle the part that needs to be exact.
Second decision: route to the care state the person is actually in
The next problem is contextual. Imagine the system identifies a high-acuity pattern and the clinically appropriate next step is to encourage human support. A generic 'find a therapist' button only makes sense if the person does not already have one. If they do, the same prompt can be redundant or even confusing.
A care-aware handoff therefore needs to account for the person's actual state. For someone without a provider, the experience can connect what came up in the conversation to why human support may be useful and offer a path to find a therapist. For someone with an upcoming appointment, the more relevant next step may be to suggest bringing the topic into that conversation. For someone who has a provider but nothing scheduled, it may be more useful to help them reconnect with the provider they already know.
The underlying clinical signal has not changed, but the appropriate product action has. That is the difference between treating a high-acuity flag as an event inside a chatbot and treating it as an event inside a broader care system. The handoff should respond to the care context, not force every person through the same generic exit.
Third decision: preserve the judgment-free container
The handoff also has to preserve the conditions that make people willing to talk candidly with AI in the first place. Human clinical review can exist within a safety pathway without turning the member-facing experience into a notification that someone has been watching the conversation. The useful framing for the person is that the AI has recognized a boundary and is pointing toward an appropriate next step, not that disclosure has triggered surveillance.
Tone is therefore a product requirement, not a copy-polish pass. A message about substance withdrawal, domestic violence, or severe functional impairment has to explain why additional support may be relevant without sounding punitive, alarmist, or falsely reassuring. It should be clear about the seriousness of the issue while preserving the private, judgment-free quality that may have made the person willing to disclose it in the first place.
Privacy has architectural consequences as well. An assigned provider should not automatically receive a person's AI conversation simply because a safety signal was raised; sharing session content with a provider depends on consent. That rules out the technically simplest version of a handoff - forwarding the transcript - and creates a more useful design question: how can the system help the member act on what came up without collapsing the privacy boundary between an AI conversation and a provider relationship?
Fourth decision: design the failure state before launch
The smaller failure cases are just as revealing as the happy path. Resource retrieval should fail safely. Contextual messages should have fallback behavior if a model or backend service is unavailable. A safety pathway should not produce an empty state because one service timed out, and a transcript that cannot be processed should not disappear silently from the review pipeline. These are operational details, but in a clinical product they are part of the methodology: safety has to hold when the ideal path does not.
The same discipline applies to what the product should never infer from a safety signal. A high-acuity flag does not mean an assigned provider should automatically be alerted, and it does not erase the consent boundary around AI conversations. The signal can initiate a defined safety workflow while the decisions about human involvement, information sharing, and the member's next step remain intentionally constrained.
The boundary is a feature
The conventional way to talk about an AI handoff is as an escalation path: the AI fails or reaches a limit, so the user gets moved somewhere else. We think the more useful framing is that the boundary itself is part of the product. The AI's role includes recognizing the kind of situation for which it is not the right endpoint and supporting a transition that is specific enough to be useful without pretending the model should own the next clinical decision.
That is a different engineering objective from maximizing what the model can do. In some parts of the system, generative flexibility is useful. In others, determinism is safer. In some moments, another AI response is appropriate. In others, the better product behavior is to point toward a person, a vetted resource, or a care relationship that already exists. The methodology is not to make AI the answer to every problem; it is to give AI a clearly defined role inside a system that can respond differently when the situation calls for more.
A good handoff is not a concession that the product has stopped working. It is what it looks like when the product has been designed to know the difference between continuing the conversation and helping someone take the next appropriate step.