OpenAI

OpenAI has introduced a new artificial intelligence-powered image detection tool and additional safety warnings in ChatGPT as part of measures aimed at preventing the creation and circulation of child sexual abuse material (CSAM).

Announced on November 16, the measures include an image classifier capable of identifying potentially abusive content, including AI-generated images, alongside stronger account restrictions, developer guidelines and investigative resources for law enforcement agencies.

The new classifier is designed to detect visual indicators of abuse in previously unseen images, rather than relying exclusively on matching content against existing databases. It assigns scores to flagged images to help human reviewers assess potentially harmful material.

OpenAI developed the classifier in collaboration with the National Center for Missing and Exploited Children (NCMEC). The company is making the technology available free of charge through its application programming interface (API) to selected organisations.

It is also working with the Tech Coalition and ROOST to expand access to the detection system.

According to OpenAI, early testing with partners showed that the classifier reduced the number of images flagged for review by 48% while identifying the same number of abusive images. The company also reported that flagged images were twice as likely to contain abusive material compared with the previous approach.

The findings are based on preliminary testing reported by OpenAI and do not constitute independently verified performance results across all operating environments.

Alongside the classifier, OpenAI is introducing warning screens in ChatGPT for users attempting to generate or access content that sexualises children.

Users submitting such requests will receive warnings and information directing them towards specialised support services. The company said accounts may also be temporarily suspended for 24 hours, with repeated violations potentially resulting in permanent removal.

OpenAI developed the warning approach with input from child protection organisations and researchers, including Protect Children, the Lucy Faithfull Foundation and Joel Scanlan from the University of Tasmania.

The company has also published guidance for developers using its technology, outlining practices for detecting, preventing and reporting child sexual abuse material.

According to OpenAI, early testing of these guidelines has helped developers adopt child safety tools and register to submit reports to NCMEC.

For law enforcement agencies, OpenAI is providing access to AI-powered investigative capabilities, including dedicated plugins, analytical tools and resources for processing complex datasets. Access will be administered through designated programme managers.

The company is additionally offering training materials and a field guide for investigators using AI in child protection cases.

The announcement follows increased scrutiny of AI companies over online safety and the potential misuse of generative AI systems. OpenAI has previously updated its content policies, introduced additional safeguards and developed a dedicated ChatGPT experience for teenagers.

The latest measures extend those efforts to image detection, user deterrence, developer responsibilities and law enforcement support.

Disclaimer: This article may include information derived from interviews, press releases, public statements, research, company communications and other publicly available or third-party sources. Such material may be summarised, paraphrased or contextualised for journalistic and editorial purposes. All rights in third-party content remain with their respective owners.