THE ESSENTIALS IN 30 SECONDS

  • The OpenAI CSAM Image Classifier, developed with NCMEC, scores images for signs of abuse, including AI-generated material, and is free through the API for approved organizations.
  • OpenAI says that in early testing it flagged 48% fewer images while finding the same number of abusive ones.
  • ChatGPT users who attempt to sexualize children now see a warning, are pointed to support services and lose access for 24 hours; repeat cases can lead to account removal.

OpenAI announced a set of child safety measures on November 16: an image classifier that it is opening to other organizations, new warnings and temporary suspensions in ChatGPT, guidance for developers and tools for investigators.

A classifier for previously unseen images

Hash matching only catches files that are already in databases of known abuse material. The new classifier, which OpenAI's developer documentation calls the OpenAI CSAM Image Classifier, instead looks for visual signs of abuse, so it can also flag new or altered images, including AI-generated ones. It returns a score that helps human reviewers decide which images to handle first. OpenAI developed it with the US National Center for Missing and Exploited Children (NCMEC). According to Business Standard, the company describes it as the first broadly available CSAM classifier that also signals possible AI generation.

OpenAI reports that in early testing the tool flagged 48% fewer images than before while identifying the same number of abusive ones, and that flagged images were about twice as likely to contain abuse. These figures come from the company and have not been independently verified.

Access is free through OpenAI's API, but the model is not openly downloadable or available to anyone who asks. Organizations must apply, pass a Persona identity check, complete onboarding and accept the applicable terms. TechCrunch reports that OpenAI is working with the Tech Coalition and ROOST to widen access.

Warnings and pauses in ChatGPT

ChatGPT already blocks requests for abuse material. Now, users who try to sexualize children also see a warning explaining the harm and are pointed to specialist support services aimed at stopping the behavior. Their access is paused for 24 hours. Business Standard adds that the flagged conversation is removed, and that repeat violations can bring longer restrictions, account closure and reports to law enforcement. OpenAI wrote the messages with Protect Children, the Lucy Faithfull Foundation and Joel Scanlan of the University of Tasmania, who independently evaluated a trial of deterrence messages.

Developers and investigators

OpenAI also published guidance for developers on preventing, detecting and reporting such material. It recommends user rules, reporting channels and registration with NCMEC or a local hotline. For specialist child safety units in the United States, the company is providing AI tools and training. According to Business Standard, the first tool helps investigators review long chat logs and legal records and build timelines. OpenAI has also joined the Tech Coalition's Lantern signal-sharing program.

The announcement comes as OpenAI faces several lawsuits over alleged harm to users, particularly children, as TechCrunch notes.