In mid-June, months before pro-democracy protesters lined Hong Kong thoroughfares with their umbrellas, activists sat in front of PCs spreading their message across the web in hope of galvanising passive onlookers into action. Sick of an interfering Chinese government, activists were gearing up for a summer of change.
Working alongside the Occupy Central team, PopVote, a site designed by University of Hong Kong and Hong Kong Polytechnic University, was readying its systems for an unofficial referendum on how citizens would choose to directly vote for Hong Kong’s chief executive—the leader of the region’s government, which is currently elected by the 1200-member Election Committee from a selection of candidates put forward by the Chinese Politburo.
Videos by VICE
Beijing had promised the people of Hong Kong that they would be able to vote for their next leader in 2017. But this time, again, the Chinese government would pick the candidates. For many in the city-state, China hadn’t gone far enough. They wanted real democracy and PopVote would help them vocalise that. But China wanted to keep its possession, which it has claimed as its own since Britain handed it over in 1997, under control. It certainly didn’t want Occupy to inspire others on the mainland.
PopVote’s awareness efforts started off smoothly. Willing participants were registering and participating in mock votes on PopVote in the week leading up to the real thing, which was scheduled to run from 20 to 22 June. But then the bots came knocking. Millions of them.
Very large attacks targeting protesters are going on right now
Most suspect the Chinese government of launching Distributed Denial of Service (DDoS) attacks that crippled PopVote, which were unprecedented in scale and sophistication. China would never admit culpability, however. Its response to almost every claim of Chinese government involvement in hacking campaigns has been blanket denial. Neither the foreign ministry nor the Chinese embassy in London haveresponded to a request for comment.
Jazz Ma is the IT manager at the University of Hong Kong, which hosts the popvote.hk website. (Voting could also be done in person at polls set up by the Occupy Central movement.) I spoke to him about the myriad attacks on organisations he belongs to, and he was careful to not explicitly name China.
“We have reported all those cases to Hong Kong Police Force in June and July. I hope they could answer you and me with who carried them out,” he said. But considering the target and the sheer scale of the attacks, the evidence points to one perpetrator.
Fingers have pointed at China after a slew of attacks on Hong Kong activists in recent months. Apple Daily, a pro-democracy newspaper, and HkGolden, a forum on which demonstrations were planned, were hit with significant attacks throughout October. US security firm FireEye has linked a sophisticated Chinese threat group it calls Operation Poisoned Hurricane—which has carried out cyber espionage on Internet infrastructure providers and a media, financial services and Asian government organizations—with those attacking Apple Daily and HkGolden.
The PopVote English site, whose recent news items still feature notes about service outages caused by DDoS attacks.
Whilst the ostensible aims of these activities differ, each support clear political objectives, said FireEye researchers Ned Moran, Mike Oppenheim, and Mike Scott in a blog post. “The Chinese government is the entity most likely to be interested in achieving both of these objectives.”
CloudFlare, a DDoS protection provider and content delivery network that defends PopVote’s site, is right in the middle of it all. It provides services to most student protest sites, as well as the website of the Hong Kong government, which has taken a number of hits from the Anonymous hacking collective. “We don’t think that bullies, whether it’s Anonymous or the Chinese government … should be able to shut down any particular thing,” said Prince.
He hasn’t seen any sign of a ceasefire in what he calls the “cyber siege of Hong Kong.” “Very large attacks targeting protesters are going on right now,” he told me.
For the June hits on PopVote, the attackers swamped the site with Domain Name System (DNS) requests, which are, in most cases, legitimately made by people’s computers when asking to contact a machine hosting a website. But get a load of infected machines, known as bots, to start firing requests en masse and you can flood the network to stop people accessing a target site. Filters can’t tell the difference between malicious requests and legitimate ones, so traditional defences don’t offer adequate protection.
It could have been catastrophic. Though the PopVote team had assumed the attacks were coming, having faced trouble during another referendum in 2012, the site was buckling under the new wave of pressure. If it didn’t stay up, the protesters, who were gearing up for the street demonstrations,would have missed their chance to send a message to the leaders of the Hong Kong administration and the Chinese government.
Our colleagues at Vice News covered the Occupy Central protests last month.
The knee-jerk reaction was to bring in another DNS server. They opted for Amazon’s Route 53 service, which should have been able to take huge spikes in traffic due to Amazon’s distributed, cloud-based architecture. Amazon’s cloud-based machines received 100 billion DNS enquiries related to PopVote in a single day, said Jazz. It stayed up, but if the site had continued to use the service, it would have sent the project bankrupt, as Amazon charges on a pay-as-you-go basis, said Jazz.
Meanwhile, another local firm was attempting to deal with the massive influx of traffic, but was unable to withstand the attacks, which had hit the 10Gbps mark, which is a pretty sizeable attack capable of serious disruptions. In fact, countering the DDoS took such a toll on the security company’s operations that it decided to stop supporting PopVote on the 16th of June. The site’s future was still under threat.
It was time to call for backup from two industry giants: CloudFlare and Google. Both American firms had recently established free services to protect those having their freedom of speech violated by cyber means.
Google announced Project Shield in October 2013. Specifically providing DDoS protection, it remains open only to those who successfully apply for an invite, and PopVote was one of those deemed worthy of protection from Google. But having swooped in to save the day by combining its services with CloudFlare’s, Google backed out of providing DDoS protection 24 hours before the referendum was to open, according to Jazz and Matthew Prince, CEO of CloudFlare.
The most successful method saw the bots send waves of an astonishing 250 million DNS requests per second for the site
Google won’t go on the record to talk about any of its Project Shield work, despite repeated requests from Motherboard, but Prince said he believes Google decided to back out not because of its politics, but for technical reasons—the firm didn’t want its other services to be slowed down by an inordinately large amount of data suddenly swamping its network. If that’s the case, Google likely figured that the China’s response to a pro-democracy activist website in Hong Kong would be big.
DDoS defenders have to find a way to spread the traffic across servers to minimise disruption, a method known as scrubbing. In the case of PopVote, Prince surmised that senior executives at Google didn’t think it could do that without slowing down its other products, from Gmail to YouTube.
Google’s withdrawal meant PopVote was largely relying on CloudFlare to fend off the attackers. “We were left alone pretty freaked out,” Prince said. And on Friday 22 June, a week after the initial attacks hit, the attackers launched their most brutal assault.
At least five different botnets were used to send an epic amount of traffic to PopVote in various ways. The most successful method saw the bots send waves of an astonishing 250 million DNS requests per second for the site. Previously, attacks of this kind had been peaking at around 90 million queries a second. When DNS providers, from ISPs to Google and OpenDNS, noticed this wave of traffic, they started blocking requests for PopVote. “No one has ever seen anything like it,” Prince said.
In the wake of ongoing DDoS attacks early in their referendum, PopVote was forced to extend voting deadlines and develop contingency plans for complete collapse.
The fight was on. If action weren’t taken immediately, access to PopVote would have been cut off, leaving activists without a major tool for raising awareness for their cause. Occupy Central would have lost the much-needed momentum that eventually led to the street protests.
Again, support had to be called in. Though Project Shield didn’t directly repel the DDoS attacks, Google was called in to assist PopVote in configuring its public DNS servers to ensure people who tried to contact the site would get through. “Google engineers did help a lot,” said Jazz.
Long-time CloudFlare partner OpenDNS—one of the biggest DNS resolver providers in the world alongside Google—did the same. “We added static entries for the popvote.hk to our global resolvers so that customers using OpenDNS could still resolve the site without having to go upstream to another DNS server to figure out the name-to-IP mapping,” said Andrew Hay, research lead at OpenDNS.
Despite these efforts, the attacks have had a lasting impact: a number of global ISPs, including Sky and Virgin in the UK, are still blocking queries for popvote.hk, though the attacks have calmed down. This is either a leftover from the June attacks, or a sign that the continued attempts on the site have ISPs concerned. Customers of those providers have to change their DNS settings to use different resolvers if they wish to visit the site.
But thanks to the collaborative effort, the vote went ahead, with only limited disruption that led to a deadline extension to cater for those who couldn’t access PopVote as a result of the epic DDoS. Nearly 800,000 took part both on and offline. Out of three proposals on the ballot, the majority—42 percent—voted for having the public, a nominating committee, and political parties pick candidates as part of a democratic election.
Police and pro-Beijing supporters clash with Occupy Central. Image: Steven Hsieh/VICE News
The scale of the DDoS has left Prince and company anxious about the future, however. The attackers produced what was likely the biggest and most sophisticated DDoS ever recorded, with an aggregate of 500Gbps of data hitting the edge of CloudFlare’s network when the botnets were at their busiest, according to Prince.
With this kind of power, any attacker, be they the Chinese government or a hacker in an organised crime racket, could threaten the stability of any site on the web. Given the skills and resources required, and the nature of the targets, the attempt on PopVote was most likely a show of cyber power from China.
And the attacks continue to rain down in Hong Kong. Jazz said PopVote and the universities who run it have been targeted in myriad ways. In one case, the University of Hong Kong detected suspicious logins to its intranet accounts, which would indicate a breach of its systems.
No one has ever seen anything like it
PopVote’s adversaries also tried to clog up the phone lines, with calls to its hotline and fax numbers made almost every second, lasting for two days, said Jazz. A fake email was also sent to the PopVote service provider asking for usage reports. The attackers tried every avenue they could to stop the group’s work.
Activists have been targeted with various forms of malware too. Claudio Guarnieri, an independent security researcher and participant in Citizen Lab, a research body focused on highlighting attacks on activists, has been tracking events in Hong Kong. At a conference in Berlin in October, he noted the ways in which China’s censorship machine was working quietly during the protests to alter searches in Baidu—the country’s Google equivalent—so that articles and images related to the events were simply removed.
He also detailed previously-documented Android malware masquerading as an Occupy Central application, which sought to scoop up identifying information of users, and a compromise of the Democratic Party of Hong Kong website.
FireEye’s Scott and Moran told me they had seen similar compromises of the HkGolden website on 5 September and the Hong Kong Association for Democracy and People’s Livelihood on 26 June, which both contained the same malicious code that redirected visitors to sites controlled by the hackers.
The eventual payload tried to install the ‘Pisces’ remote access Trojan (RAT) on people’s PCs. Pisces is a previously undocumented but basic piece of malware that lets hackers siphon off data from an infected machine. RATs are traditionally used by digital spies, rather than crooks trying to make money.
Despite the persistent attempts to spy on them and censor their online activity, the activists Guarnieri has spoken to don’t want his assistance. “I’m not really following it closely anymore after protesters told me they don’t want help. They’re owned, DDoSed and defaced, but happy with that,” he said.
As the government settles in for a war of attrition with protesters on the street, and in the face of unrelenting online attacks, the demonstrators are standing their ground, peacefully.
