PHOTO
Primary school security systems appear unable to stop pupils searching for inappropriate images through artificial intelligence.
The mother of a 9-year-old said her son and a classmate at a Christchurch school were able to bypass computer security and download images of naked women by using AI.
“He was caught by his teacher walking behind him who saw him accessing inappropriate images. The school emailed me to let me know and he got stood down from using devices for a week,” she said.
The school says blocking inappropriate material is being thwarted by quickly evolving AI.
The Star has chosen not to identify the school.
The school’s acting principal confirmed the child had used the internet to “search up inappropriate images”.
“A child did use the internet to search up inappropriate images but through our ministry-provided security system we realised what happened and immediately triggered our systems to ensure student wellbeing was protected.
“That included talking to the children and all the parents involved about device.
“With inappropriate images, it will block certain aspects and send alerts to team staff members.”
But he conceded due to the quick evolving nature of generative AI the security system was not foolproof.
“Children are always trying to work around these things and it is our job to teach them how to manage themselves,” he said.
The mother said she understands that with the new AI tech it is very difficult for the school to police. But she was concerned school security systems could not consistently pick up breaches.
“I have three boys and we always have conversations when this arises but this [AI] is a new fear unlocked,” she said.
Ministry of Education spokeswoman Coralanne Child said the school had advised it “of an incident and confirmed that they are managing it in accordance with their processes. We have offered our support if needed.
“We would encourage anyone with concerns to raise this directly with the school in the first instance,” she said.
It comes as the Deepfake Digital Harm and Exploitation Bill heads to Parliament’s Social Services and Community Committee. Public submissions are open until tomorrow.
A key part of the Bill is the issue of using AI to create sexualised deep fake, and laws around it.
Canterbury University senior law lecturer Cassandra Mudgway said making the creation of sexualised deepfakes illegal is not enough.
“If New Zealand wants to meaningfully address image-based sexual abuse in the age of generative AI, we have to go further than criminalising it,” she said.
Netsafe said that AI tools like Grok on X can create fake images of people without consent.
“In New Zealand, harmful digital communications are actionable, whether real or manipulated, under the Harmful Digital Communications Act.”
• If you have experienced issues with AI deepfakes contact Netsafe at 0508638723.