Update: After this article was published, Apple told Dave Choffnes that his iPhone app, designed to detect net neutrality violations, will be allowed in the iTunes App Store. According to Choffnes, Apple contacted him and explained that the company has to deal with many apps that don’t do the things they claim to do. Apple asked Choffnes to provide a technical description of how his app is able to detect if wireless telecom providers throttle certain types of data, and 18 hours after he did, the app was approved.
“The conversation was very pleasant, but did not provide any insight into the review process [that] led the app to be rejected in the first place,” Choffnes told us in an email.
Videos by VICE
The original story follows below.
The most pervasive feeling about the Federal Communication Commission’s net neutrality repeal is one of hopelessness. If we all need to use the internet, big telecom companies control our access to the internet, and there’s no choice about what company to use, how are we supposed to stop these companies from messing with our connections?
The FCC has suggested that consumer outrage will prevent companies from violating net neutrality, but it if you’re not a network engineer, it can be hard to know if net neutrality is being violated at all. David Coffnes, a researcher at Northeastern University, set out to change that. He created an app to detect net neutrality violations, but Apple has banned it from the App Store, preventing consumers from accessing the information they need to at least know when they’re getting screwed over.
Using Apple’s beta testing platform called TestFlight, I tested the app, called Wehe. It’s straightforward. You open the app, agree to a consent form (he is using the data in his research), and click “run test.” The app is designed to test download speeds from seven apps: YouTube, Amazon, NBCSports, Netflix, Skype, Spotify, and Vimeo. According to the app, my Verizon LTE service streamed YouTube to my iPhone at 6 Mbps, Amazon Prime video at 8 Mbps, and Netflix at 4 Mbps. It downloaded other data at speeds of up to 25 Mbps.
“Differentiation means in this case throttling by Verizon,” Choffnes told me. This would, in theory, be the sort of thing people would want to know—with this knowledge, they could choose to switch to another carrier, or could lodge a complaint against with the Federal Trade Commission.
Ajit Pai’s FCC has made the argument that “most attempts by ISPs to block or throttle content will likely be met with a fierce consumer backlash … in the event that any stakeholder [ISP] were inclined to deviate from this consensus against blocking and throttling, we fully expect that consumer expectations, market incentives, and the deterrent threat of enforcement actions will constrain such practices.”
But the fact is that every major wireless telecom provider is already throttling data, and we are more-or-less powerless to stop it. And the opaque nature of both the telecom industry and Apple’s App Store vetting process is preventing consumers—and researchers like Choffnes—from getting a full picture of how net neutrality is being violated.
An Apple App Store reviewer told Choffnes that “your app has no direct benefits to the user,” according to screenshots reviewed by Motherboard. According to Apple’s reviewer, the app contained “Objectionable Content,” a catch-all for apps that Apple doesn’t want to let into its App Store. Apple is blocking the app and no one is quite sure why, including Choffnes; neither Apple nor Verizon responded to requests for comment for this article.
Wehe is is designed to be part of Choffnes’s research work to determine geographic and carrier-related differences in video throttling. When you open the app, you are presented with a consent form that “invites you to take part in a research project.”
“The purpose of this research study is to understand how cellular internet providers give different performance to different network traffic from your smartphone,” it says, adding that data is anonymized. “For example, we would like to know if a provider is speeding up YouTube traffic and/or slowing down Netflix.”
Wehe, according to the App Store reviewer, “may mislead users by providing inaccurate determinations … specifically, your app is marketed to users as a way to check if their carrier is violating net neutrality. However, your app has no direct benefits to the user from participating in the study.”
Packet inspection and video throttling
When I heard about Wehe, I thought that it must be impossible for an app to detect net neutrality violations. Or at least, I couldn’t think of a mechanism in which it might work. But once I spoke to Choffnes, who has spent much of the past few years reverse-engineering the ways in which telecom companies throttle data, it made sense.
Choffnes is an expert in data “differentiation,” which means he studies how telecom companies alter the download speeds of text, photos, or emails may be prioritized over the download speeds of video content. Such “prioritization” or data discrimination violates one of the core tenets of net neutrality, but data differentiation is commonly used by cell phone providers nonetheless.
“We didn’t have net neutrality even before the rules changed,” Choffnes said. “All the carriers are doing content-based throttling, specifically with video. And some video providers are getting better performance than others.”
That video is being throttled is not a secret. Many telecom providers “zero rate” certain video services and then advertise those services as “unlimited” to customers, meaning it doesn’t count against a customer’s data cap. The most famous instance of this is T-Mobile’s “BingeOn” service, which allows unlimited access to Netflix, YouTube, and a few other major video providers.
But that “unlimited” video means that video is throttled—in BingeOn’s case, T-Mobile video maxes out around 1.5 mb/s, whereas its standard LTE service gets speeds of up to 10 times that for non-video content. Other telecom providers have similar programs; Verizon has plans with “unlimited 4G LTE data” and “premium unlimited 4G LTE data,” the plan you have determines the resolution and amount of data you can stream before it’s throttled (and putting a cap on resolution is also a form of throttling; delivering lower-resolution video means delivering less data).
Customers may not generally think of this practice as a net neutrality violation, but former FCC chief Tom Wheeler wrote in a letter to Congress in 2016 that such programs are likely violations of the net neutrality rules he put into place in 2015 (that have since been repealed by Ajit Pai’s FCC.)
Zero rating programs “may harm consumers and competition in downstream industry sectors by unreasonably discriminating in favor of select downstream providers, especially their own affiliates,” Wheeler wrote.
We know that telecom companies throttle video, but Choffnes’s research focuses on how and when they throttle. His research finds methods of actually detecting the mechanics of data differentiation from carrier to carrier. What he’s found is that, for the most part, telecom providers aren’t throttling video; they are using a network management tactic known as deep packet inspection that throttles based on metadata associated with network traffic. What this means is that T-Mobile, for instance, might not try to detect whether something is a video or not, but it can detect whether a service calls its data a video or has the metadata hallmarks of a video. If so, it will set a download speed cap for that specific data.
“We realized that they’re looking for certain text in the network traffic, and if we changed that text, when we send that traffic over the network, it doesn’t get throttled”
For example, when an encrypted connection is established between Netflix’s servers and T-Mobile’s servers (known as a TLS handshake), certain plaintext information is exchanged (host names and server names). In Netflix’s case, one of these servers is called “nflxvideo.net.” If T-Mobile detects this server name in the metadata, it will throttle download data for those packets.
Choffnes learned about this system by reverse engineering it; his team downloaded videos from various video services (including the TLS data and all metadata) and then recreated it on their own servers (called “replays”). What he found is that by changing the metadata of the video’s header—but not the video itself—it could be downloaded at much higher speeds. If he changed the metadata of other types of data (photos, for instance) to have the Netflix metadata, that data would be throttled by the telecom company when it was downloaded.
“We realized that they’re looking for certain text in the network traffic, and if we changed that text—replaced nflxvideo.net with northeasternvideo.com—when we send that traffic over the network, it doesn’t get throttled,” Choffnes said. “This means it’s keyword related and not server or even content related.”
Because throttling is often keyword and not content-related, that means some video services are treated different from other video services; you may be able to stream Vimeo or a video hosted on a less-popular website faster than you can stream a video on Netflix, for example. And video is generally (not always) throttled around the clock, regardless of the overall traffic being put on a network, which peak during commutes and in the evenings.
It’s something we’ve been working on for years, something the academic community thinks is accurate, and we’re working with a regulator to disseminate it so other people can use it
“When faced with a problem like network management, the question is ‘Do you want to use a sledgehammer or a scalpel?’ You want to use the tool that will have the least negative impact while providing benefit to everyone,” Choffnes said. “What I think is in place today is a bit of a sledgehammer. Video traffic is a cause for congestion, but the video is throttled to a low rate, and it’s done that way all the time.”
An information page for Wehe explains its mission: “We need your help to test more providers, in the US and worldwide, so we can understand how [throttling] policies change over time, location, and network. We are building a website that will publicize these practices, both to inform regulators and to allow consumers to make informed choices about selecting their mobile providers.”
‘Objectionable Content’
To be clear, much of our outrage should probably be directed at the telecom industry, which has never shown much intention of following the principles of net neutrality. But it’s no surprise that telecom companies are going to act in the interest of their bottom lines. What’s less clear is why, exactly, Apple has banned a pro-consumer app from its App Store.
Choffnes has presented this data at scientific and telecom conferences, and his papers are peer reviewed.
His system is not a perfect way of determining actual network speeds, because he doesn’t have access to telecom infrastructure or video provider servers. But he says that the basic methods of data discrimination have not been disputed by telecom companies and that his work has caught the eye of ARCEP, France’s version of the FCC, which has cited his work and wants to use his methods to catch telecom companies violating net neutrality in the country.
In fact, Verizon is currently paying his team to “research the video performance of Verizon’s video streaming services,” and Google has funded some of his work under its Faculty Award Research program. Choffnes says that the terms of these agreements do not allow Verizon or Google to influence his work: “This contract has no restriction on our ability to publish our findings that do not rely on confidential information, and by definition the measurements we do on these operational networks are not confidential (because we could do these measurement with or without Verizon),” he said.
I mention these partnerships because the industry seems to believe in the accuracy of his work, but Apple, it seems, does not. The company has famously blocked many apps from entering the App Store or has prevented third party apps from accessing data that Apple itself can. For instance, Apple removed a feature that allowed third-party apps to access iPhone battery cycle data, presumably because software readings of battery health are less accurate than hardware ones (Apple never publicly addressed why it made this data inaccessible.) But Apple allows many different apps that allow users to do straight speed tests of their connections, which Choffnes says uses essentially the same technology his app does.
“I probably could have gotten away with calling it a speed test,” he said. “But I wasn’t going to lie to get it published.”
Because Wehe is basically just making requests to a variety of servers Chofnes rents from Amazon (and he “plans to include servers from multiple providers to avoid any potential for bias from just one provider”) there is no reason to think that the data it returns is inaccurate, and Apple’s suggestion that people receive no benefit from knowing they’re being throttled would seem to ignore the widespread public outrage about the FCC’s recent vote to repeal net neutrality.
“I’m under contract with a French telecom regulator to provide this app as a service. I’m not a random independent researcher who has decided to on a whim to publish something that may or may not do what it says,” he told me. “It’s something we’ve been working on for years, something the academic community thinks is accurate, and we’re working with a regulator to disseminate it so other people can use it.”
Update: This story has been updated to clarify the servers that are used for Wehe.
