An algorithm used by the popular European food delivery app Deliveroo to rank and offer shifts to riders is discriminatory, an Italian court ruled late last week, in what some experts are calling a historic decision for the gig economy. The case was brought by a group of Deliveroo riders backed by CGIL, Italy’s largest trade union.
A markedly detailed ordinance written by presiding judge Chiara Zompi gives an intimate look at one of many often secretive algorithms used by gig platforms to micromanage workers and which can have profound impacts on their livelihoods.
Videos by VICE
While machine-learning algorithms are central to Deliveroo’s entire business model, the particular algorithm examined by the court allegedly was used to determine the “reliability” of a rider. According to the ordinance, if a rider failed to cancel a shift pre-booked through the app at least 24 hours before its start, their “reliability index” would be negatively affected. Since riders deemed more reliable by the algorithm were first to be offered shifts in busier timeblocks, this effectively meant that riders who can’t make their shifts—even if it’s because of a serious emergency or illness—would have fewer job opportunities in the future.
According to the court, the algorithm’s failure to take into account the reasons behind a cancellation amounts to discrimation and unjustly penalizes riders with legally legitimate reasons for not working. Deliveroo was ordered to pay €50,000 (~$61,400) to the suing parties.
Legal experts that Motherboard spoke to described the decision as a possible turning point. Importantly, they said, the court determined that even if an algorithm unintentionally discriminates against a protected group a company can still be held liable and be forced to pay damages.
“This is a landmark case,” Valerio De Stefano, a professor in labor law at KU Leuven who specializes in AI and labor regulation, told Motherboard over the phone. “What it shows, basically, is that on a legal level you can have indirect discrimination through algorithims and that algorithms are therefore subject to judicial review, that you can legally question how these types of algorithms work. I think that’s important, because people can often think of algorithms as objectively neutral, when in fact there’s always the possibility of discrimination involved.”
In a statement to Motherboard, a Deliveroo spokesperson claimed that the company no longer uses the same shift booking system outlined in the case.
“This judgement refers to a historic optional booking model which is not used by Deliveroo in Italy or other markets,” the spokesperson wrote. “Riders have complete flexibility to choose when to work, where to work, for little or as long as they want. This means that there is no booking system and no obligation to accept work.”
“We offer self-employment because this offers the flexibility riders want,” they continued. “Every survey shows riders overwhelmingly value flexibility above all else – more than 80% in the latest survey. Currently Deliveroo receives thousands of requests to work as a self-employed rider each week and we have doubled the number of riders in the UK – we now work with 50,000 riders in the UK, up from 25,000 last year.”
But the ramifications of the court’s decision could go far beyond a single booking system or algorithm, says Ivana Bartoletti, co-founder of the Women Leading in AI network and author of “An Artificial Revolution”.
“Even if this particular shift booking system is no longer in place, I think this decision will indicate to companies such as Deliveroo, but of course others as well, that they need to be more conscious about addressing and understanding the potential problems and inequalities their algorithms create, or they could face legal consequences,” Bartoletti said. “Of course, that’s not enough on its own—we need regulatory and legislative solutions as well. But, the combination of all of these things could be a major step forward.”
The case is also indicative of an increased willingness on behalf of regulators, the judicial system, labor unions, and workers across the continent to tackle blackbox algorithms, and an increased awareness of how such algorithms can potentially be abused to circumvent traditional labor protections. In July 2020, for example, four U.K. drivers backed by the App Drivers and Couriers Union sued Uber to gain access to similar algorithms used by Uber. And, three months later, another group of Uber drivers filed a lawsuit against the company for allegedly being fired by an automated algorithm used by the platform without being given an opportunity to appeal.
