AI-Generated 'Seinfeld' Show Banned on Twitch After Transphobic Standup Bit

“Nothing, Forever,” the infinitely-generating AI version of Seinfeld that tens of thousands of people were watching has been banned for 14 days from Twitch after Larry Feinberg—a clone of Jerry Seinfeld—made transphobic statements during a standup bit late Sunday night.

“Hey everybody. Here's the latest: we received a 14-day suspension due to what Larry Feinberg said tonight during a club bit,” Xander, one of the creators of Nothing Forever, said on Discord. “We've appealed the ban, and we'll let you know as we know more on what Twitch decides. Regardless of the outcome of the appeal, we'll be back and will spend the time working to ensure to the best of our abilities that nothing like that happens again.”

The show’s AI, which is trained on classic sitcom episodes and various AI tools, mimics that of a traditional Seinfeld episode, which starts with a standup routine from “Larry,” before moving to his apartment. During a standup set Sunday night, Larry made a series of transphobic and homophobic remarks as part of a bit:

“There’s like 50 people here and no one is laughing. Anyone have any suggestions?,” he said. “I’m thinking about doing a bit about how being transgender is actually a mental illness. Or how all liberals are secretly gay and want to impose their will on everyone. Or something about how transgender people are ruining the fabric of society. But no one is laughing, so I’m going to stop. Thanks for coming out tonight. See you next time. Where’d everybody go?”

Twitch did not immediately respond to a request for comment about whether this joke was the reason for the ban, but the joke happened soon before the channel was banned, and users in the project's Discord have been pointing to this joke as the reason for the ban. The Twitch page for "Nothing, Forever" displays a notice saying it is “temporarily unavailable due to a violation of Twitch’s Community Guidelines or Terms of Service.”

In the project’s Discord, show staff blamed the event on having to switch the AI model, which caused “errant behaviors.”

“We’ve been investigating the root cause of the issue,” tinylobsta, a staff member, wrote on Discord. “Earlier tonight, we started having an outage using OpenAI’s GPT-3 Davinci model, which caused the show to exhibit errant behaviors (you may have seen empty rooms cycling through). OpenAI has a less sophisticated model, Curie, that was the predecessor to Davinci. When davinci started failing, we switched over to Curie to try to keep the show running without any downtime. The switch to Curie was what resulted in the inappropriate text being generated. We leverage OpenAI’s content moderation tools, which have worked thus far for the Davinci model, but were not successful with Curie. We’ve been able to identify the root cause of our issue with the Davinci model, and will not be using Curie as a fallback in the future. We hope this sheds a little light on how this happened.”

"I would like to add that none of what was said reflects the devs' (or anyone else on the staff team's) opinions," another staffer posted.

The incident is emblematic of one faced by most AI: When AI is trained on hateful or biased material, the outputs can often be hateful or biased. This has given rise to the field of AI safety, which develops tools to mitigate the biases baked into models. This is why ChatGPT typically won't make blatantly racist, sexist, or transphobic remarks when asked simply. Many AI tools are moderated by underpaid workers in the developing world. 

The staff has made most of its Discord read only. On a thread where users are still allowed to post, many users are using Midjourney AI to generate images of Jerry Seinfeld holding the trans pride flag.