Niva started as a simple idea: a chatbot that could help survivors of gender-based violence find support without forcing them to explain everything to a person before they were ready.
That sounds like a product problem. For me, it quickly became an architecture problem. A survivor using this kind of tool may be sharing a device, may be watched, may not have private time, and may not get a second chance if the tool leaves the wrong trace behind.
So the important questions were not only “what should the chatbot say?” They were also: what should it never store? What should it never ask for? What should remain impossible even if the team later wants better analytics, more personalization, or a cleaner growth story?
This essay is about that part of the work — the places where ethics stopped being a statement of intent and became a set of design constraints.
The problem with promises
Most software makes its privacy promise in prose: a notice, a policy, a line of reassuring copy near a sign-up form. Those promises are not useless. They create obligations. They tell users what the organization says it will do.
But they still depend on the organization behaving well later. They depend on future engineers, future product managers, future integrations, future dashboards, and future pressure to monetize what has been collected.
The architecture is different. It is the database schema, the logs, the identifiers, the network calls, the retention policy, and the default behavior when nobody is paying attention. It decides what is easy, what is hard, and what is impossible without rebuilding the system.
That distinction became very real while I was studying sensitive digital services. BetterHelp's 2023 FTC settlement is the example that stayed with me: users were told their mental health information would stay private, while the product still allowed information to flow into advertising systems. The written promise and the technical reality had drifted apart.
In sensitive systems, the safer promise is the one the architecture helps keep.
What changed in the design
One scenario shaped a lot of the design: a survivor opens the tool on a phone that is not fully private. Maybe a partner checks the device. Maybe the browser history is inspected. Maybe the survivor closes the app quickly and expects the evidence of the conversation to be gone.
In that situation, small technical defaults become safety decisions. Persisting chat history is no longer a neutral convenience. Asking for an account is no longer a harmless onboarding step. Leaving a recognizable URL in history is no longer just a browser detail.
That is what I mean by ethics in the architecture. The commitment is not only that the team intends to be careful. The commitment is that the system is shaped so there are fewer dangerous things for the team to do by accident.
For Niva, that showed up as four design choices that look modest on paper but changed the character of the product.
- No conversation persistence. The product did not need a transcript database, so the safer decision was not to build one. Sessions could be useful in the moment without becoming a long-lived record later.
- No accounts, no identifiers, no analytics on conversation content. The less the system knows about a survivor, the less it can expose through a breach, subpoena, dashboard, or future integration.
- A Quick Exit that treats browser history as part of the threat model. A safety button is not enough if the back button quietly returns to the conversation. The exit behavior had to account for what someone else might inspect afterward.
- Intent recognition instead of open-ended generation. In this context, a fluent answer is not the same as a safe answer. A constrained router made it possible to review the response set instead of trusting a model to improvise around crisis language.
None of this is technically flashy. That is part of the lesson. In a high-risk context, boring constraints can matter more than impressive features.
What the tradeoffs felt like
The first tradeoff was visibility. If you avoid storing conversations and avoid tracking conversation content, you lose many of the signals product teams normally use. You cannot casually inspect transcripts to understand drop-off. You cannot optimize sensitive flows by watching what survivors typed in their worst moments.
The second tradeoff was speed. A constrained response system is safer because it can be reviewed, but that also means changes need process. If a new response is added, the safety claim only holds if the response is actually reviewed before it reaches users.
The third tradeoff was organizational discipline. The most dangerous requests rarely sound dangerous. They sound practical: one analytics event, one troubleshooting log, one personalization field, one experiment to improve the funnel.
Building this way means being willing to say no to useful data because the context changes what “useful” costs. The product becomes less convenient for the builder so it can be less risky for the person using it.
What this made clearer
The most useful lesson was not a specific chatbot pattern. It was a way of judging whether a system's values are actually carried by its design.
The privacy conversation becomes more concrete. Instead of asking a partner to trust that the product team will behave well forever, you can point to the system design: what is stored, what is not stored, where identifiers do not exist, and where sensitive content cannot become an analytics asset.
Restraint becomes a technical skill. Engineers are often rewarded for adding capability. This work rewarded a different habit: removing capability where the user would pay the price for it. Not every feature that can be built deserves to exist in the first version of a sensitive system.
The lesson traveled beyond GBV software. After Niva, I carried the same thinking into other chatbot and assessment work. The stakes were different, but the question remained useful: what should this system be structurally unable to do, regardless of what someone asks for later?
The line I keep coming back to
When you build software for survivors of gender-based violence, a privacy mistake is not just a compliance issue. It can affect whether someone feels safe asking for help, whether they are exposed to retaliation, or whether they trust the next support tool they encounter.
That is why the architecture matters. It cannot remove every risk. It cannot replace clinical judgment, human support, legal care, or institutional responsibility. But it can reduce the number of ways the software itself becomes part of the harm.
For me, that became the standard: design the system for the case where the user cannot afford for you to be casual.
Responsible AI should include the quiet engineering work of deciding what the system is not allowed to collect, remember, infer, or say.
That is less dramatic than a principle statement, but it is more useful. A principle can be ignored under pressure. A constraint has to be confronted.
Niva taught me that the ethical work does not start after the product is built. It starts when you choose the data model, the logs, the identifiers, the response system, and the default behavior of a button someone might press in panic.