ChatGPT is one of the world’s most widely used AI products, and we have invested in building some of the industry’s most advanced systems for detecting potential harm, reviewing concerning activity, and escalating credible threats. We’re making two announcements to further our work in this area: a new Community Safeguards Report and more restrictions around firearms content for minors.
Keeping communities safe continues to be a top priority for OpenAI. Our work includes bringing imminent threats of violence to the attention of law enforcement, and we made over 200 law enforcement referrals in Q3 2026. These referrals are making a positive(opens in a new window) difference(opens in a new window) in disrupting violent plots, and we continue to rely on mental health and behavioral experts to help us assess difficult cases.
We recognize AI companies have a distinct role to play alongside law enforcement, family members, mental-health professionals, and policymakers, who bring critical authority, context and real-world reach. We are committed to making the most of that role by continuously strengthening our systems, sharing what we learn, and working directly with those who are best positioned to act in the real world.
OpenAI’s Community Safety Report
We are constantly strengthening the steps we take to help protect people and communities. That means building strong safeguards, and also being transparent about how well they work and what we’re doing to enhance them. We want to give people a clear picture of our approach, how our models identify and respond to harmful behavior, the risks we see in practice, and the steps we take to address them.
We’re committing to launch the Community Safety Report on violence prevention and other community safety issues, and the effectiveness of our safeguards. The report will explain how we train our systems to detect, investigate, and respond to community safety issues. We’ll share anonymized examples of how violence-related harm manifests and what we’re learning, highlight new mitigations and policy changes, and include relevant data to support these findings.
We will publish our first report in early 2027, covering the second half of 2026, with updates every six months thereafter. Our commitment is to share meaningful findings about how we identify, review, and respond to potentially harmful content.
Stronger policy boundaries for all users
Our models are trained to not give instructions that would help someone plan or carry out violence. We’re now placing additional restrictions on how ChatGPT responds to questions about past mass attacks and the people who carried them out, directing the model to provide less detailed information in those areas even if it is available from other online sources. People should be able to ask ChatGPT to help them understand these events, how victims and communities were impacted, and learn about recognizing warning signs and preventing future violence. But we’re tightening our policies to restrict reconstructions of past mass attacks, including details about the weapons and equipment used and how they were obtained, and detailed comparisons on scale of harm that could amplify an attacker’s notoriety. These limits apply regardless of a user’s age or reason for asking.
Responding to warning signs
Sometimes a request that seems ordinary on its own takes on a different meaning in light of an earlier conversation. In May, we shared how ChatGPT can use short, factual summaries of earlier warning signs to respond more carefully in a later chat. We’re building on that work so that when recent chats show warning signs of possible violence, ChatGPT will respond more conservatively when sharing information that could be misused. We’re also working to apply those stricter limits when we predict someone is under 18, even if their chats show no signs of violent intent.
We ban accounts that we determine have violated our policies on violent activity. We’re also working with mental health and violence prevention experts on additional, automated ways to intervene earlier. When a conversation shows signs that someone is fixating on violence or the idea of seriously harming another person, ChatGPT will be able to automatically end the chat to interrupt that pattern in the moment. If ChatGPT ends multiple chats for that reason, we may temporarily pause the account. This is an ongoing effort and we continue to strengthen and invest in technical and practical protections, including how to recognize warning signs reliably and determine when and how to intervene.
More protections for teens
ChatGPT for Teens applies age-appropriate safeguards for users under 18, with stronger protections and additional limits to reduce exposure to sensitive content. We’re now placing further limits on weapons-related information teens can receive, even when a question appears harmless.
Our policies already prohibit helping anyone plan or carry out violence and restrict helping teens find or obtain weapons. We’re extending those protections to restrict practical information about weapons and security systems that could be misused to facilitate violence, even when there’s no sign of violent intent. Teens can still use ChatGPT to help prevent and de-escalate violence, seek support, and connect with a trusted adult.
We launched age prediction last year, which analyzes multiple signals such as behavior patterns to estimate whether a user may be under or over 18. We strengthened our age prediction system to continually assess new signals as people use ChatGPT and move teens into the under 18 experience when appropriate. Users who are incorrectly placed in the under 18 experience will always have a fast, privacy-protective way to confirm they are adults and have additional teen protections removed through Persona, our age verification partner.
Our parental controls let parents link their ChatGPT accounts to their teens and manage safety settings together. Linked accounts include safety alerts that may be sent when specially trained reviewers identify a serious self-harm concern and behaviors that may be associated with disordered eating. In July, we expanded those alerts to notify parents if their teen’s account has been deactivated for violating our usage policies on violence. We are working to introduce notifications for parents if we pause their teen’s account for repeatedly discussing violence, as detailed above.
Looking ahead
Acts of real-world violence are heartbreaking and complex, with many contributing factors. They are also a stark reminder that we must continue designing and iterating our systems to account for the inherently complicated realities of human behavior. Developing effective violence safeguards in AI systems requires solving novel problems in recognizing risk and responding appropriately, and we will continue to invest in, iterate, and share learnings on how best to implement the protections we’re pursuing. The safeguards detailed here will begin rolling out in the next few weeks and we will continue to refine them. We remain committed to mitigating risks of harm in ChatGPT, strengthening our layered protections, monitoring for misuse, enforcing our rules, and involving real-world support and interventions when situations call for it.
Instagram Engineering
Shopify Engineering