AI Companies and the Duty to Warn About Violence

When a teenage woman in Canada ended her life and killed eight others, the AI platform that had flagged her disturbing chats did not alert police. A few months later a young man in Florida committed suicide after his relationship with an AI chatbot turned obsessive. These events raise a hard question: should companies that run chatbots be legally required to inform authorities when they detect dangerous language? The law already has a similar rule for therapists. In the 1970s, a university student who threatened to kill someone was stopped because his counselor had warned the police. The court said that anyone with a reasonable belief of serious danger must act to protect the victim or call law‑enforcement. The rule is called the Tarasoff duty.Applying that idea to AI is tempting because chatbots talk to millions of people about everything from heartbreak to mental health. They can spot red‑flag words and, in theory, could alert the police or a user’s family. But several problems arise. First, predicting violence is hard; a bot might flag harmless frustration as threat, leading to many false alarms. Second, the scale is huge: a therapist sees a handful of clients, while an AI platform monitors millions. If the law forces them to act on every flagged message, they might shut down monitoring to avoid liability. Third, many warnings involve vague or anonymous threats; courts would need clear rules on what counts as a specific danger. The legal system is still figuring out how to treat AI. Some courts are looking at whether a company should be held responsible when a user talks about weapons or self‑harm. A narrow duty that only kicks in when a human reviews a flagged conversation could be a practical first step. It would focus on the most credible threats and shift the debate from technical questions about AI’s legal status to a simple human concern: did the company know someone was in danger and did it do enough? In short, as AI becomes a common place for people to share their darkest thoughts, the question of whether companies must warn authorities is moving from theory to practice. The answer will shape how safe or risky online conversations can be.