🔗 Share this article

Tech firms and child safety organizations will receive authority to evaluate whether artificial intelligence systems can generate child exploitation images under recently introduced UK legislation.

Substantial Rise in AI-Generated Harmful Content

The declaration came as revelations from a safety monitoring body showing that cases of AI-generated child sexual abuse material have more than doubled in the last twelve months, rising from 199 in 2024 to 426 in 2025.

Updated Legal Framework

Under the changes, the government will permit designated AI developers and child safety organizations to inspect AI models – the foundational technology for conversational AI and visual AI tools – and ensure they have sufficient protective measures to stop them from creating depictions of child exploitation.

"Ultimately about preventing exploitation before it occurs," stated the minister for AI and online safety, noting: "Experts, under strict conditions, can now detect the risk in AI systems promptly."

Tackling Regulatory Obstacles

The changes have been introduced because it is illegal to create and own CSAM, meaning that AI developers and others cannot create such images as part of a evaluation regime. Previously, authorities had to delay action until AI-generated CSAM was uploaded online before dealing with it.

This legislation is aimed at averting that problem by enabling to halt the production of those images at source.

Legislative Framework

The amendments are being added by the government as revisions to the criminal justice legislation, which is also implementing a ban on owning, creating or distributing AI models designed to generate exploitative content.

Practical Consequences

This week, the official toured the London base of a children's helpline and listened to a mock-up call to advisors featuring a report of AI-based exploitation. The interaction portrayed a teenager seeking help after facing extortion using a explicit deepfake of himself, constructed using AI.

"When I hear about children experiencing extortion online, it is a source of extreme anger in me and rightful concern amongst parents," he said.

Alarming Statistics

A leading online safety foundation stated that cases of AI-generated abuse content – such as webpages that may contain multiple images – had significantly increased so far this year.

Instances of the most severe content – the most serious form of exploitation – increased from 2,621 images or videos to 3,086.

• Female children were predominantly victimized, making up 94% of prohibited AI images in 2025
• Portrayals of newborns to two-year-olds rose from five in 2024 to 92 in 2025

Industry Response

The law change could "represent a vital step to guarantee AI tools are safe before they are launched," commented the head of the internet monitoring organization.

"AI tools have made it so victims can be victimised repeatedly with just a simple actions, providing offenders the ability to make possibly limitless amounts of sophisticated, photorealistic exploitative content," she added. "Content which further exploits victims' trauma, and makes young people, particularly girls, more vulnerable both online and offline."

Counseling Session Information

Childline also published information of support sessions where AI has been referenced. AI-related harms discussed in the conversations include:

• Using AI to evaluate weight, physique and appearance
• Chatbots dissuading children from consulting trusted adults about harm
• Being bullied online with AI-generated content
• Digital blackmail using AI-manipulated images

During April and September this year, the helpline conducted 367 counselling sessions where AI, chatbots and associated topics were discussed, four times as many as in the equivalent timeframe last year.

Fifty percent of the references of AI in the 2025 sessions were related to psychological wellbeing and wellness, encompassing using AI assistants for support and AI therapy applications.