Artificial intelligence-generated voices are already being used to harass ordinary people, with online trolls creating the voices and then having them read out victim’s home addresses and posting the results online, according to four victims Motherboard spoke to. All of the victims of this particular campaign are voice actors who have worked on video games in the past.
The news highlights the potent threat posed by AI-generated voices for anyone who has recordings of their voices available online. Podcasters, YouTubers, and streamers are just some of the other sorts of people that could feasibly be targeted by such attacks in the future. Motherboard previously reported how 4chan members created synthetic replicas of celebrities’ voices, including in one case an Emma Watson replica reading Mein Kampf. That sort of attack now extends to people who aren’t as high profile as Hollywood celebrities.
Videos by VICE
“Hello, this is Abbey [name] speaking,” a sound clip posted to Twitter using a fake voice of an actor called Abbey says. Motherboard withheld some actors’ last names, including Abbey’s, at their request. The synthetic voice then reads out Abbey’s home address, and adds “I live in the [homophobic slur] city that is Los Angeles. Yes, that does also mean I live in California, the most [racist slur] state in the USA. Personally speaking, killing [racist slur] and [sexually abusive act] children is completely fine.”
Do you know how else AI-generated voices are being abused? We’d love to hear from you. Using a non-work phone or computer, you can contact Joseph Cox securely on Signal on +44 20 8133 5190, Wickr on josephcox, or email joseph.cox@vice.com.
The harassers posted similar audio clips on Twitter which included the home address and fake voices of at least three other actors. Zane has worked on Fallout 4 mods; Tom worked on Poppy Playtime. They also posted the address of another actor called Michael. The Twitter accounts created to share these peoples’ private information also retweeted another video using an AI-generated voice, this time in the style of Agent 47 from the Hitman games.
“I basically just saw I was tagged in a post, and the immediate thing I noticed was my home address, so I was surprised to say the least,” Tom told Motherboard in an email. “Then I registered it had the racist rhetoric, framed as inflammatory as possible in such a way an internet troll might do. From what I gathered, it was bait from the get go, trying to get a rise out of voice actors who had publicly expressed concerns about AI.” Last week Motherboard reported on raising concerns inside the voice actor industry about clients asking actors to sign the rights to their voices away so more material could be generated with AI later.
Twitter has removed one offending tweet and suspended one related profile. But in multiple cases Twitter has failed to take down the tweets, despite them clearly violating Twitter’s policy on the publication of private information. Tom said “What was the bigger frustration was how ineffective Twitter’s support system was in removing the post. Regardless of the stolen identity, private information posted publicly and the racist slurs, Twitter’s support system deemed the post and the account as perfectly fine.” Twitter has gutted its communications department; Elon Musk did not respond to a request for comment.
In some of the tweets including the home addresses, the harasser claims that the audio was generated by an AI-voice company called ElevenLabs. Last month Motherboard reported that 4chan members appear to have used ElevenLabs to create the audio of celebrities. Seemingly in response to those clips, ElevenLabs said it was exploring more safeguards around its platform. Up until that point, the barrier of entry for people to generate voices of anyone was much lower.
In Twitter direct messages sent to the harassment victims, ElevenLabs acknowledged that the Agent 47 audio was made using the company’s system. “We have the IP address of that person,” ElevenLabs wrote to Abbey in one of the messages. Abbey and other victims shared copies of the messages with Motherboard. But ElevenLabs says the voices used to read out the victim’s home addresses were not made with its technology.
“The doxing attacks on you are horrible and should never happen,” ElevenLabs wrote to Abbey. “We want to make clear that despite being mentioned, this was not created using our software or is associated with our company. We can track every request and these have not come through our system. We believe this is an organized smear campaign following us introducing additional safeguards to prevent misuse and exactly such malicious cases.”
Mati Staniszewski, an ex-Palantir deployment strategist and now co-founder of ElevenLabs, told Motherboard in an email that “We are aware of cases of text to audio software being used to create offensive or objectionable content which have then been falsely attributed to our technology. We are able to identify all content made using our software and, on investigation, almost all of these clips have been created using other platforms. Our new safeguards are already rapidly reducing instances of misuse and we’re grateful to our user community for continuing to flag any examples where extra action needs to be taken and we will support authorities in identifying those users if the law was broken.”
“We have a zero tolerance approach to the abuse or misuse of any text to audio software to harm or offend others. We are extremely disheartened that a vocal minority are choosing to use the technology in this way, but have confidence that existing and upcoming safeguards will continue to tackle this,” he added.
Clarification: This piece has been updated to clarify interviewee’s work history.
Subscribe to our cybersecurity podcast, CYBER. Subscribe to our new Twitch channel.
