The deepfake crisis engulfing hundreds of schools

Last Saturday, a Telegram message popped up on Heejin’s phone from an anonymous sender. “Your pictures and personal information have been leaked. Let’s discuss.”

As the university student entered the chatroom to read the message, she received a photo of herself taken a few years ago while she was still at school. It was followed by a second image using the same photo, only this one was sexually explicit, and fake.

Terrified, Heejin, which is not her real name, did not respond, but the images kept coming. In all of them, her face had been attached to a body engaged in a sex act, using sophisticated deepfake technology.

Deepfakes, the majority of which combine a real person’s face with a fake, sexually explicit body, are increasingly being generated using artificial intelligence.

“I was petrified, I felt so alone,” Heejin told the BBC.

But she was not alone.

Two days earlier, South Korean journalist Ko Narin had published what would turn into the biggest scoop of her career. It had recently emerged that police were investigating deepfake porn rings at two of the county’s major universities, and Ms Ko was convinced there must be more.

She started searching social media and uncovered dozens of chat groups on the messaging app Telegram where users were sharing photos of women they knew and using AI software to convert them into fake pornographic images within seconds.

“Every minute people were uploading photos of girls they knew and asking them to be turned into deepfakes,” Ms Ko told us.

Ms Ko discovered these groups were not just targeting university students. There were rooms dedicated to specific high schools and even middle schools. If a lot of content was created using images of a particular student, she might even be given her own room. Broadly labelled “humiliation rooms” or “friend of friend rooms”, they often come with strict entry terms.

Ms Ko’s report in the Hankyoreh newspaper has shocked South Korea. On Monday, police announced they were considering opening an investigation into Telegram, following the lead of authorities in France, who recently charged Telegram’s Russian founder for crimes relating to the app. The government has vowed to bring in stricter punishments for those involved, and the president has called for young men to be better educated.

Telegram said it “actively combats harmful content on its platform, including illegal pornography,” in a statement provided to the BBC.

The BBC has viewed the descriptions of a number of these chatrooms. One calls for members to post more than four photos of someone along with their name, age and the area they live in.

“I was shocked at how systematic and organised the process was,” said Ms Ko. “The most horrific thing I discovered was a group for underage pupils at one school that had more than 2,000 members.”

In the days after Ms Ko’s article was published, women’s rights activists started to scour Telegram too, and follow leads.

By the end of that week, more than 500 schools and universities had been identified as targets. The actual number impacted is still to be established, but many are believed to be aged under 16, which is South Korea’s age of consent. A large proportion of the suspected perpetrators are teenagers themselves.

Heejin said learning about the scale of the crisis had made her anxiety worse, as she now worried how many people might have viewed her deepfakes. Initially she blamed herself. “I couldn’t stop thinking did this happen because I uploaded my photos to social media, should I have been more careful?”

Scores of women and teenagers across the country have since removed their photos from social media or deactivated their accounts altogether, frightened they could be exploited next.

“We are frustrated and angry that we are having to censor our behaviour and our use of social media when we have done nothing wrong,” said one university student, Ah-eun, whose peers have been targeted.

Ah-eun said one victim at her university was told by police not to bother pursuing her case as it would be too difficult to catch the perpetrator, and it was “not really a crime” as “the photos were fake”.

At the heart of this scandal is the messaging app Telegram. Unlike public websites, which the authorities can access easily, and then request for images be removed, Telegram is a private, encrypted messaging app.

Users are often anonymous, rooms can be set to “secret” mode, and their contents quickly deleted without a trace. This has made it a prime space for criminal behaviour to flourish.

Last week, politicians and the police responded forcefully, promising to investigate these crimes and bring the perpetrators to justice.

On Monday, Seoul National Police Agency announced it would look to investigate Telegram over its role in enabling fake pornographic images of children to be distributed.

The app’s founder, Pavel Durov, was charged in France last week with being complicit in a number of crimes related to the app, including enabling the sharing of child pornography.

But women’s rights activists accuse the authorities in South Korea of allowing sexual abuse on Telegram to simmer unchecked for too long, because Korea has faced this crisis before. In 2019, it emerged that a sex ring was using Telegram to coerce women and children into creating and sharing sexually explicit images of themselves.

Police at the time asked Telegram for help with their investigation, but the app ignored all seven of their requests. Although the ringleader was eventually sentenced to more than 40 years in jail, no action was taken against the platform, because of fears around censorship.

“They sentenced the main actors but otherwise neglected the situation, and I think this has exacerbated the situation,” said Ms Ko.

Park Jihyun, who, as a young student journalist, uncovered the Nth room sex-ring back in 2019, has since become a political advocate for victims of digital sex crimes. She said that since the deepfake scandal broke, pupils and parents had been calling her several times a day crying.

“They have seen their school on the list shared on social media and are terrified.”

Ms Park has been leading calls for the government to regulate or even ban the app in South Korea. “If these tech companies will not cooperate with law enforcement agencies, then the state must regulate them to protect its citizens,” she said.

Before this latest crisis exploded, South Korea’s Advocacy Centre for Online Sexual Abuse victims (ACOSAV) was already noticing a sharp uptick in the number of underage victims of deepfake pornography.

In 2023 they counselled 86 teenage victims. That jumped to 238 in just the first eight months of this year. In the past week alone, another 64 teen victims have come forward.

One of the centre’s leaders, Park Seonghye, said over the past week her staff had been inundated with calls and were working around the clock. “It’s been a full scale emergency for us, like a wartime situation,” she said.

“With the latest deepfake technology there is now so much more footage than there used to be, and we’re worried it’s only going to increase.”

As well as counselling victims, the centre tracks down harmful content and works with online platforms to have it taken down. Ms Park said there had been some instances where Telegram had removed content at their request. “So it’s not impossible,” she noted.

In a statement, Telegram told the BBC that its moderators “proactively monitor public parts of the app, use AI tools and accept user reports in order to remove millions of pieces of content each day that breach Telegram’s terms of service”.

While women’s rights organisations accept that new AI technology is making it easier to exploit victims, they argue this is just the latest form of misogyny to play out online in South Korea.

First women were subjected to waves of verbal abuse online. Then came the spy cam epidemic, where they were secretly filmed using public toilets and changing rooms.

“The root cause of this is structural sexism and the solution is gender equality,” read a statement signed by 84 women’s groups.

This is a direct criticism of the country’s President Yoon Suk Yeol, who has denied the existence of structural sexism, cut funding to victim support groups and is abolishing the government’s gender equality ministry.

Lee Myung-hwa, who treats young sex offenders, agreed that although the outbreak of deepfake abuse might seem sudden, it had long been lurking under the surface. “For teenagers, deepfakes have become part of their culture, they’re seen as a game or a prank,” said the counsellor, who runs the Aha Seoul Youth Cultural Centre.

Ms Lee said it was paramount to educate young men, citing research that shows when you tell offenders exactly what they have done wrong, they become more aware of what counts as sexual abuse, which stops them from reoffending.

Meanwhile, the government has said it will increase the criminal sentences of those who create and share deepfake images, and will also punish those who view the pornography.

It follows criticism that not enough perpetrators were being punished. One of the issues is that the majority of offenders are teenagers, who are typically tried in youth courts, where they receive more lenient sentences.

Since the chatrooms were exposed, many have been closed down, but new ones will almost certainly take their place. A humiliation room has already been created to target the journalists covering this story. Ms Ko, who broke the news, said this had given her sleepless nights. “I keep checking the room to see if my photo has been uploaded,” she said.

Such anxiety has spread to almost every teenage girl and young woman in South Korea. Ah-eun, the university student, said it had made her suspicious of her male acquaintances.

“I now can’t be certain people won’t commit these crimes behind my back, without me knowing,” she said. “I’ve become hyper-vigilant in all my interactions with people, which can’t be good.”

Additional reporting by Hosu Lee and Suhnwook Lee