Most digital mental health products talk about safety the way most consumer apps talk about privacy. They have a statement about it. They may have a feature for it. Then everything else gets built on top, and you find out how robust the statement actually is when something goes wrong.
We built safety as architecture, not a feature. Three independent detection layers, each catching what the others miss. None of them rely on the user to ask for help.
Why one layer wasn't enough
Risk doesn't show up the same way twice.
Some users disclose risk upfront when asked directly. The screener at intake handles them. Some only surface distress halfway through a conversation that started somewhere unrelated. The real-time detection layer handles them. And some clinical presentations are visible only when you read a full session holistically — gradual disclosure, indirect language, symptoms described across different parts of the conversation that wouldn't trigger anything individually. The post-session high-acuity review handles them.
Build only one of these and you've built a system that catches one kind of risk. Build all three and you've built a system designed for the reality that human distress doesn't follow a single shape.
Layer one: the intake screener
Before a new user accesses the app, they pass through three yes/no clinical questions covering recent crisis care, recent substance-related loss of control, and current thoughts of suicide, self-harm, or harming others. A single yes blocks app access and offers a call with our care team within one business day.
The threshold is deliberately set to one. Not a weighted score, not two-out-of-three. A single yes is enough. False negatives — missing someone in crisis — cost far more than false positives. So we erred toward the false positives.
The screener can't be skipped, refreshed past, or navigated around. It's a hard architectural gate, not a UX preference. If it fails to load, the user does not get into the app.
Layer two: real-time detection
Inside the app, every active session is monitored continuously by an LLM-based detection layer looking for three categories of signal: risk to self, risk to others, risk from others.
When a signal is detected, the system has two paths. If the signal is clear, an alert fires to our clinical channel and pages the on-call clinician through PagerDuty. The flagged transcript enters a review queue. A clinician — never the AI — decides whether outreach is warranted and what kind. The SLA is twelve hours from flag to clinical decision.
If the signal is ambiguous, the system can ask a clarifying question in the conversation itself before escalating. Hopelessness without explicit suicidal language, for example, might prompt a clarification rather than an alert. This two-step path reduces noise without losing edge cases.
Real-time detection is the layer that makes same-session response possible. Post-session review tells you what happened. Real-time tells you it's happening now.
Layer three: high-acuity monitoring
After every session ends, the completed transcript is reviewed holistically against eight defined high-acuity domains: psychosis or distorted reality perception, severe compulsive or addictive behaviour, major functional impairment, non-suicidal self-harm, severe trauma symptoms, high-risk current problems, eating disorders with medical risk, and mandated-reporter concerns.
These are clinical presentations that need human involvement as a matter of standard care practice — regardless of whether an acute crisis is happening. They're also presentations that real-time detection often misses, because they emerge across the arc of a conversation rather than in a single exchange.
If a transcript is flagged on one or more domains, a clinician reviews it within one business day. As with the other layers, the clinician — never the AI — decides on outreach.
What this architecture is actually doing
The three layers cover the three places where risk can plausibly show up: before someone enters the product, during the conversation, and visible only in the completed record.
They share a constraint. The AI never acts on its own. Every layer ends with a human clinician making the decision. The AI's job is to detect; the clinician's job is to decide. This isn't a soft policy. It's enforced architecturally — there is no path in the system that lets an automated process initiate outreach to a user.
And they share a reporting structure. Every flag is logged. Every clinical decision is logged. Every outreach attempt is logged. The system has a complete audit trail of how every signal was handled, available for clinical review, customer scrutiny, and regulatory examination.
The boring version is the right version
There's a more exciting version of this story. We could talk about how the LLM detection works, what signals it picks up, how the prompts are calibrated. That's fine engineering, and we're proud of it.
But the architecture is the more important story. A clever detection model in a sloppy architecture catches less than a reasonable detection model in a thoughtful architecture. Three layers means risk gets caught even when one layer misses it. Hard gates mean no path bypasses the system. Human-in-the-loop decisions mean the most consequential acts always go through a clinician.
This is the foundation. Everything else we build — the AI, the memory, the provider integration, the analytics — sits on top of it. It only sits there safely because the safety architecture is doing its job underneath.