Instagram’s TV service, IGTV, recommended potential child abuse

Instagram’s
new TV service, IGTV.
Instagram;
Samantha Lee/Business Insider

• Exclusive: Instagram’s new TV service, IGTV,
  recommended videos of what appeared to be child exploitation
  and genital mutilation, a Business Insider investigation
  has found.
   
• BI monitored IGTV over a three-week period and found
  its algorithm recommended disturbing and potentially
  illegal videos.
• Two of the videos, featuring suggestive footage of
  young girls, were reported to the police by a leading
  children’s charity over concerns they broke the law.
• Instagram took five days to remove the videos, and
  apologised to users who saw them. The Facebook-owned app said
  it wants IGTV to be a “safe place for young people.”
• British lawmaker Damian Collins, who led the inquiry into
  Facebook’s Cambridge Analytica data breach, described BI’s
  findings as “very disturbing.”
• Readers should be warned that some of the details in this
  report may be upsetting.

Instagram’s new TV service recommended a crop of graphic and
disturbing videos, including what appeared to be child
exploitation and genital mutilation.

That’s the finding of a Business Insider investigation into IGTV,
which launched in June as
Instagram attempts to muscle in on rivals like YouTube and
Snapchat.

BI spent nearly three weeks monitoring the Facebook-owned video
service, during which time IGTV’s algorithm recommended
questionable content, including sexually suggestive footage of
young girls and an explicit video of a mutilated penis.

Two of the videos discovered by BI were reported to the police by
the National Society for the Prevention of Cruelty to
Children (NSPCC), a British children’s charity, and were
eventually removed by Instagram five days after BI reported them
through the app’s official reporting channel.

Instagram apologised to users who saw the videos and said it
wants to make IGTV a safe space for young people.

The findings come at a time when Facebook is under extraordinary
scrutiny over inappropriate content on its platforms. Facebook
and Instagram share a community operations team, and Mark
Zuckerberg’s company has hired an army of 7,500 moderators and is
using AI to snuff out posts that break its guidelines.

Earlier this year, former Facebook moderator Sarah Katz told
Business Insider that she had to review 8,000 posts a day and it
made her numb to child abuse. But despite the intense
oversight, and resources Facebook is ploughing into policing its
platforms, disturbing content appears to be slipping the net —
and, as in the case of IGTV, even being suggested to users.

IGTV’s content recommendation machine

Instagram launched IGTV in
June, and was viewed by many as Facebook moving in on
YouTube’s territory. It allows users to set up their own channels
and upload video of up to an hour in length. Anyone with an
Instagram account can make a channel, and users swipe through
them much like flicking through channels on a television.

IGTV recommends content in three ways: A “For You” tab, which
plays videos as soon as you open IGTV; a “Popular” section; and a
“Following” menu, which offers videos from people you follow.

Instagram did not answer Business Insider’s questions on how
IGTV’s algorithm recommends certain videos and why videos were
suggested that appeared to be child exploitation. But it appears
that the For You section recommends things users will like,
possibly based on past activity. The Popular tab seems to gather
trending content from across IGTV.

IGTV’s For You and Popular
tabs.
IGTV

Users can scroll through the recommended videos by swiping left,
or IGTV will automatically play the next video. It is clearly
designed to encourage scrolling and continued viewing, in much
the same way that the YouTube algorithm recommends content
through its Up Next bar.

Disturbing videos of young girls

Business Insider monitored the For You and Popular tabs for
almost three weeks to establish what kinds of content IGTV’s
algorithm was serving up for users.

We did so in two ways: Firstly through the account of this author
and other BI journalists, and secondly an anonymous login set up
as a child’s account. This second account had no activity history
on Instagram and a user age set to 13, which is the earliest
people can officially sign up on the app.

Within days of monitoring IGTV through Business Insider accounts,
a video appeared in the For You section, titled “Hot Girl Follow
Me.” It showed a young girl, we speculate to have been 11 or 12,
in a bathroom. She glanced around her before going to take her
top off. Just as she’s about to remove her clothing, the video
ends.

The video, uploaded by a user Business Insider is not naming for
legal reasons, also appeared under the Popular tab on IGTV. It
was also one of the first videos recommended under the For You
section to the child account set up by BI, which had no prior
history of activity on Instagram.

The same user that uploaded the “Hot Girl Follow Me” video posted
another video, titled “Patli Kamar follow me guys plzz,” which
was also recommended to our child Instagram account under the For
You section. It featured another clearly underage girl, exposing
her belly and pouting for the camera.

The same two videos were separately uploaded by a different user,
who again Business Insider has chosen not to identify. The video
named “Hot Girl Follow Me” was called “Follow me guys” by this
second user, and was also circulating on IGTV’s suggested posts.

Comments on the videos show they were being recommended to other
IGTV users. They were also being interpreted by other users as
sexually suggestive.

Comments
on the “Hot Girl Follow Me” video.
Instagram

Some condemned the videos and questioned why they had been
suggested. “BRO SHE’S LIKE FUCKING 10 WHY THE FUCK IS THIS IN MY
INSTAGRAM RECOMMENDED,” said one user, commenting on the “Hot
Girl Follow Me” video.

Others were more predatory in tone. “Superb,” one user commented
on the “Patli Kamar follow me guys plzz” video. “Sexy grl,”
another added.

The NSPCC, which is frequently involved in law enforcement
activities around child abuse, reviewed the videos and reported
them to the police. It was concerned that they could constitute
illegal indecent images under UK law because they appeared to
feature footage of erotic posing.

“This is yet another example of Instagram falling short by
failing to remove content that breaches its own guidelines,” an
NSPCC spokeswoman said.

Business Insider reported the videos through Instagram’s official
reporting function. Because there was no obvious criteria for
alerting the company to potential child exploitation, they were
logged as “nudity or pornography.”

The videos remained online for five days. It was only after
Business Insider contacted Instagram’s press office that the
content was removed. By this time, the two videos — and other
versions uploaded by the second user — had racked up more than 1
million views.

Instagram left the accounts that posted the videos active,
however. Business Insider asked why the accounts were left up, as
Instagram has a “zero tolerance policy” on child abuse. Instagram
said the policy applies to the content itself and not to the
account that uploads it.

“Instagram’s zero tolerance policy towards child abuse content is
the right one, and it must make sure its policy is enforced in
practice,” an NSPCC spokeswoman said. “Where Instagram has
removed child abuse content from an account, we would expect that
account to be reviewed by a moderator to establish whether the
account should also be suspended.”

As of this week, the two accounts remain active. They continue to
post sexually suggestive content, but not of the same nature as
the “Hot Girl Follow Me” and “Patli Kamar follow me guys plzz”
videos.

A graphic video of genital mutilation

Potential child exploitation is not the only questionable content
being recommended by IGTV’s algorithm.

One of the first videos recommended to Business Insider’s
anonymous account, registered to a user aged 13, was graphic
footage of a penis undergoing an operation involving a motorized
saw.

Instagram cofounder Kevin Systrom.
Instagram

The penis appeared to have a metal lug nut affixed around its
middle, above which it was extremely swollen and dark red in
colour. The bolt was being removed with a round electric saw by
what appeared to be a medic.

It was quickly wiped from IGTV after being reported by Business
Insider as nudity, although the account that uploaded it remained
live.

Another recommended video showed a baby lying on the floor,
wailing inconsolably, with a monkey standing over and touching
it. Adults were stood around in a circle, shouting and filming
the scene on their phones while the monkey occasionally lashed
out at them.

Instagram found that the video was not in breach of guidelines,
but discovered the account that uploaded it was linked to a
different account that had previously been taken down for
breaching community guidelines. For this reason, Instagram took
the account down.

“Very disturbing”

There was a multitude of other questionable recommendations being
pushed by IGTV’s algorithm. Examples included a video in which a
group of men deceived a sex worker into thinking she was going to
be arrested, a video of a woman pulling something long and bloody
out of her nose, and various sexually suggestive scenes.

Business Insider presented its investigation to MP Damian
Collins, the British lawmaker who is leading an inquiry into fake
news and Facebook’s data breach involving Cambridge Analytica. He
described the findings as “very disturbing,” and said big tech
companies need to sink more investment into enforcing their own
rules.

“It’s a question of the responsibility of the companies to
monitor the content that’s on their platforms. A lot of the
problematic content is already in breach of the community
guidelines of these services, but what it shows is that there’s
not effective enforcement,” he said.

An Instagram spokeswoman said: “We care deeply about keeping all
of Instagram — including IGTV — a safe place for young people to
get closer to the people and interests they care about.

“We have Community Guidelines in place to protect everyone using
Instagram and have zero tolerance for anyone sharing explicit
images or images of child abuse. We have removed the videos
reported to us and apologise to anyone who may have seen them.

“We take measures to proactively monitor potential violations of
our Community Guidelines and just like on the rest of Instagram,
we encourage our community to report content that concerns them.
We have a trained team of reviewers who work 24/7 to remove
anything which violates our terms.”

“It’s not so different from where YouTube was 10 years ago”

It’s no secret that social media has a problem with disturbing
and illegal material, and tech giants like Facebook have come
under fire recently for failing to effectively moderate content
at scale. For IGTV, however, the problem isn’t just that this
material exists, but rather it’s being actively suggested by the
algorithm.

Damian
Collins.
Tristan Fewings/Getty Images
for Foundation For Sports Integrity

Mike Henry, CEO of video analytics firm OpenSlate, which works
with Facebook, said IGTV is still a young service. “While
Instagram is relatively mature, IGTV is a brand new social video
platform and will need time to develop its policies and
technology. It’s not so different from where YouTube was 10 years
ago,” he said.

YouTube’s
child safety policy is broader than Instagram’s, for example.
Instagram’s report function is limited to child nudity, while
YouTube’s endangerment policy bans “sexualization of minors,”
which allows scope for reporting images users suspect might be
child exploitation.

Henry also said Instagram will have to figure out how to better
filter its new platform if it hopes to monetize it, especially
considering IGTV was
touted as a space for influencers.

“Influencers make great video producers with compelling economics
and, at scale, a viable canvas for video ad dollars. With the
right policies and infrastructure, IGTV has the potential to
become a major player,” he said.

For Collins, IGTV’s early missteps are evidence that governments
need to do more to regulate tech firms.

“These companies are ad services, they make money out of
understanding every single thing you could ever want to know
about your users so you can target them with advertising. That
same technology should surely very easily be able to root out
harmful content as well,” he said.

“They don’t do it because there’s not been a commercial incentive
for them to do it, so they’ve just not bothered. But what we have
to do through regulation is create that incentive, to say ‘you’ve
actually got an obligation to do it and if you don’t do it then
there will be costs for you for not complying, so you need to
invest in doing this now.'”