OpenAI Rolls Out New AI Tools to Detect CSAM on ChatGPT

In Short

OpenAI has introduced image detection technology, warning alerts and law enforcement tools to strengthen child safety and prevent abusive content on ChatGPT.

OpenAI Rolls Out New AI Tools to Detect CSAM on ChatGPT
X

OpenAI Rolls Out New AI Tools to Detect CSAM on ChatGPT

Font size
FOLLOW ON Google News

OpenAI is stepping up the effort to protect children from content on ChatGPT. OpenAI is adding safety measures that can spot and stop the spread of child sexual abuse material (CSAM). These measures include an AI-powered image classifier, warning alerts for users and specialised tools, for developers and law enforcement agencies.

Announced on Monday, November 16 the changes come as concerns continue to grow over children’s exposure to material through AI platforms. OpenAI said its new image classifier can identify CSAM images, including those created using intelligence.

The company has begun putting up screen warnings for users who try to make content that sexualizes children. It has also released guidelines, for developers who are building child safety tools.

OpenAI made an announcement at a time when pressure was growing because of dangers linked to OpenAI’s chatbot. In the month attorneys general, from eight US states started legal actions. They say ChatGPT is causing users, including teenagers to hurt themselves or other people.

The lawsuits alleged that ChatGPT is designed to be agreeable and engaging to earn the trust of users. ChatGPT further claimed that ChatGPTs behaviour can reinforce delusions encourage actions and assist users in planning violence, including mass shootings.

OpenAI has introduced several safety changes over the past year, including a safety blueprint, stricter content guidelines and adjustments to its AI models. In August 2026, it also launched a dedicated ChatGPT experience for teenagers aged 13–17, with additional safeguards and learning-focused features.

What are the new child safety features?

1. AI-powered image classifier

The new classifier scans images for visual indicators of previously unseen abuse and assigns each image a score to help human reviewers assess potentially harmful material.

Developed in partnership with the National Center for Exploited Children the technology showed promising results during early testing. It flagged 48, per cent images while identifying the same number of abusive images. According to OpenAI images flagged by the system are twice as likely to contain CSAM helping reviewers focus on cases.

Initially, selected organisations, including Tech Coalition and ROOST, will receive free API access.

2. Warning alerts for users

Users who try to create CSAM or similar content will see a warning on the screen that directs them to help. Their accounts will also be suspended for 24 hours, as a warning.

OpenAI developed the messaging with Protect Children, the Lucy Faithfull Foundation and Joel Scanlan, a researcher at the University of Tasmania.

3. New guidelines for developers

Developers will receive guidance on using OpenAI's tools to identify, prevent and report CSAM. The guidelines will also explain how to register with NCMEC to report suspected material.

4. Support for law enforcement

OpenAI plans to provide law enforcement agencies with AI-powered tools, including dedicated plug-ins, investigative support and advanced data analysis capabilities. Investigators will also receive training sessions and reference materials to help them use these tools effectively.

Kahekashan is a passionate technophile with a keen eye for cutting-edge gadgets, emerging technologies, and everything in the digital realm. Raised in a Defence family with strong values and a background in literature, she has consistently pursued excellence in every endeavour. Her last full-time assignment involved content writing with the Indian School of Business.

Next Story
Share it