At the end of the day on a Friday in October 2014, just a few months after Amazon paid nearly $1 billion for the video game streaming website Twitch, an engineer stumbled upon what at the time was the platform’s first ever hack while doing tech support for a colleague who worked remotely.
“Oh fuck,” the former Twitch engineer recalled saying. “But I remember thinking that there was so much ‘I told you so’ here.”
Videos by VICE
The engineer, who asked to remain anonymous to speak about confidential details of the incident, said he found logs that showed the hackers had gained access to his colleague’s account. The hackers had left clear traces of their intrusion, he said.
“This attack definitely had the characteristic of a minimally skilled adversary,” he told Motherboard.
The full story of the 2014 hack against Twitch has never been reported before, and is based on interviews with seven former Twitch employees who were working there when the breach happened, and when the company later investigated the hack and dealt with its fallout. Motherboard granted sources anonymity because they are bound by non-disclosure agreements that forbid them from discussing details of their work at the time.
The discovery of the suspicious logs kicked off an intense investigation that pulled nearly all Twitch employees on deck. One former employee said they worked 20 hours a day for two months, another said he worked “three weeks straight.” Other employees said they worked long hours for weeks on end; some who lived far from the office slept in hotel rooms booked by the company. At the time, Twitch had few, if any, dedicated cybersecurity engineers, so developers and engineers from other teams were pulled into the effort, working together in meeting rooms with glass windows covered, frantically trying to figure out just how bad the hack was, according to five former Twitch employees who were at the company at the time.
The hack was so bad that Twitch essentially had to rebuild much of its code infrastructure because the company eventually decided to assume most of its servers were compromised. They figured it would be easier to just label them “dirty,” and slowly migrate them to new servers, according to three former employees who saw and worked with these servers.
“The hackers had such wide access before they were detected, we basically had to rebuild everything from scratch”
Remnants of that hack still exist today, and can be found in Twitch source code stolen and dumped on the internet by hackers last week in another major data breach that exposed the revenues of streamers on the platform as well as internal source code. Twitch has changed significantly since 2014, but former employees say that the earlier hack had knock-on effects that can still be seen today.
Outside of the company, Twitch did not disclose details of the breach, nor its extent.
Twitch’s users would only find out about the breach six months after its discovery, on March 23, 2015, when the company published a short blog post that explained “there may have been unauthorized access to some Twitch user account information,” but did not let on nearly how damaging the hack was to Twitch internally.
Inside the company, the 2014 incident would later be codenamed “Urgent Pizza,” and became something of a running joke; Twitch leadership eventually printed t-shirts with this name on it. (The company also decided that all future incidents would have food-themed codenames, and wrote a random codename generator for that purpose, according to one source and code leaked in the 2021 incident).
“The event was called ‘urgent pizza’ because management had everyone do ridiculous amounts of overtime and ordered pizzas as incentive lol,” a former Twitch employee told Motherboard. “People who participated got t-shirts and ‘joke’ about having PTSD from the long hours and lack of understanding of the scope of the hack which necessitated the company-wide rebuild.”
Traces of the 2014 hack and the company’s response were exposed in some of the data leaked in the recent incident, like artifacts from the past, etched in source code. Some files leaked last week contain strings like “remove pizza script,” “a pizza thing,” “indicate that the server is ‘urgent-pizza clean,’” “move pizza to securelogin,” and “dirty_status = True.”
The hack came as a surprise, even though the company had not invested in keeping itself secure, according to another former employee.
“Security efforts kept getting cancelled or deprioritized with the argument that ‘everyone loves Twitch; no one wants to hack us,’” they added.
Do you work at Twitch? Do you have more information about the recent breach and leak? We’d love to hear from you. You can contact Lorenzo Franceschi-Bicchierai securely on Signal at +1 917 257 1382, Wire/Wickr @lorenzofb, or email lorenzofb@vice.com.
When Twitch finally disclosed the hack in March of 2015, security engineers at Twitch and Amazon, who had come to help with the incident response, concluded that the hack had started at least eight months before the discovery in October of 2014, though they had no idea if the hackers had actually broken in even earlier than that, according to the former employee.
“That was long enough for them to learn entirely how our whole system worked and the attacks they launched demonstrated that knowledge,” the former employee said.
“The hackers had such wide access before they were detected, we basically had to rebuild everything from scratch,” the former employee added. “Some of the response involved rewriting the login process because the hackers had figured out how to send a copy of every single user’s password to their servers. They also gained access to all of our source code and all of our databases.”
Two other former employees confirmed the extent of the data breach.
But many Twitch employees never fully figured out just how deep the hackers had burrowed into the company’s network, and it’s unclear to them whether the company ever fully understood the full scale of the attack.
For months after the discovery and public announcement, several servers and services were internally labeled as “dirty,” as a way to tell all developers and engineers to be careful when interacting with them, and to make sure they’d get cleaned up eventually. This meant that they were still live and in use, but engineers had put restrictions on them in the event that they were still compromised, according to three former employees.
“The plan apparently was just to rebuild the entire infra[structure] from known-good code and deprecate the old ‘dirty’ environment. We still, years later, had a split between ‘dirty’ services (servers or other things that were running when the hack took place) and ‘clean’ services, which were fired up after,” one of the former employees said. “We celebrated office-wide the day we took down the last dirty service!”
“Twitch wasn’t aware of what [the hackers] had access to, or how long they had access,” one of the former employees, who joined a couple of years after the hack, told Motherboard. “Which is how we ended up with the dirty hosts: they were hosts that were probably fine, but they couldn’t definitively say one way or the other.”
“You’d think that would be a teachable moment but it was either something that newer (non-security) people largely didn’t know about, and older employees kind of giggled about,” he said.
The sources who spoke to Motherboard shared details of the 2014 hack to illustrate Twitch’s approach to security at the time, and how it’s evolved since then. The damage Twitch suffered in the hack years ago, compared to damage from the new incident, could show whether the company has learned its lessons from 2014 and put better protections in place.
“This [2014] incident was a very worst-case-scenario attack that was entirely preventable. But leadership’s desire to avoid considering the security problem was the root cause,” one of the former employees said. “And this more recent incident demonstrates that they didn’t learn anything from the incident in 2014.”
Twitch did not respond to a series of questions about the 2014 hack, and the more recent incident.
Last week, an anonymous poster on the online forum 4Chan published what they called “part one” of a series of Twitch leaks.
“Their community is also a disgusting toxic cesspool, so to foster more disruption and competition in the online video streaming space, we have completely pwned them, and in part one, are releasing the source code from almost 6,000 internal Git repositories,” the anonymous 4Chan user wrote in the post that has since been deleted.
The leak consisted of hundreds of gigabytes of internal Twitch source code and financial data of streamers. This leak is certainly embarrassing for Twitch, and also extremely damaging to streamers.
“It’s definitely troubling, to say the least,” well-known streamer Amouranth told Motherboard. “As a streamer, Twitch is actively serving as your employer, and you want to believe that you can trust them to have security measures in place to prevent this kind of thing from happening.
Twitch has yet to disclose details of the data breach, as it’s still investigating. In a blog post, the company said that “some data was exposed to the internet due to an error in a Twitch server configuration change that was subsequently accessed by a malicious third party.”
“In the earlier days of Twitch the security team had a presence, but they seemed to be stretched to the limit”
When the former Twitch employees who spoke to Motherboard found out about the leak, some were surprised, and others weren’t.
“When I heard this, I really wondered how they could be so sloppy, especially with Amazon as parent,” one of them said.
Other former employees, however, said that the damage of this new data breach appears to be less severe than the 2014 hack. And that it’s likely thanks to Twitch taking security more seriously since then.
“In the earlier days of Twitch the security team had a presence, but they seemed to be stretched to the limit. Twitch had a bad habit of adding tons of new employees to build new products without growing the teams around them, like security,” one source said. “I think they knew they needed to start taking it seriously under the eye of Amazon. It definitely got better over time. Tighter access controls, more security scrutiny over the internal tools used, things like that.”
Another echoed this sentiment.
“Things were much better by the time I left though, I think even though this is a much bigger hack in terms of impact on the technical side of the business, the security posture of the company is probably matured enough to make remediation a much more gentle process,” they said. “It’s still pretty embarrassing, but the theories I’ve seen so far on what the means of access was make the blast radius a lot smaller. Because despite it meaning some serious audits of the security of pretty much all of Twitch’s code, it shouldn’t need to be a company-wide-overtime-for-months kind of situation.”
But in the end, Twitch is now investigating another data breach, six years after the worst hack of its history.
“The security team really did everything they could,” a former employee said. “So it’s frustrating to see it come to this.”
Matthew Gault contributed reporting.
Subscribe to our new cybersecurity podcast, CYBER. Subscribe to our new Twitch channel.
