Privacy Group Asks for Investigation Into Software That Spies on Students

In response to a crescendo of complaints about proctoring software that surveils students while they take tests in their homes, the nonprofit Electronic Privacy Information Center today formally asked the District of Columbia’s attorney general to investigate the “unfair and deceptive” business practices of five exam proctoring companies. 

If the district’s attorney general takes enforcement action, it would be a “pathbreaking” decision that could open up a new avenue for consumer advocates to pursue legal accountability for black-box algorithms that make high-stakes decisions in areas like criminal justice, housing, and health care, John Davisson, a senior staff attorney for EPIC, told Motherboard.

EPIC has simultaneously sent notices to the five companies—Respondus, ProctorU, Proctorio, Examity, and Honorlock—threatening to bring civil suits against them if they do not stop their “excessive and unjustified” collection of sensitive student data, including biometrics, and “reliance on opaque, unproven AI analysis to flag purported instances of cheating.”

COVID-19 lockdowns led to a boom in business for companies selling proctoring software. Colleges around the world adopted the tools in efforts to prevent students from cheating on exams. But as quickly as schools began using the tools, they received a surge of complaints from students and professors who see the software as intensely invasive and prone to discriminating against anyone who doesn’t meet the software developers’ definition of “typical.”

Many students have pointed out that the tools use facial recognition algorithms that fail to detect faces with dark skin tones, making it impossible for them to take their tests. Others have complained that being watched and recorded in their bedrooms, then judged by algorithms, creates a hostile learning environment.

“This is exam period for a lot of schools and I think it’s easy to think of these [companies] in abstract terms … but they cause real, tangible harms,” Davisson said. “Put yourself in the shoes of a student sitting at a desk with a camera monitoring their every movement, afraid that they’re going to look the wrong way or make a strange shadow that’s going to put them in a position of being accused of cheating by an algorithm.”

In its complaint to the D.C. attorney general, EPIC claims that the five proctoring companies collect far more data about students than is necessary to perform their purported purpose: catching cheaters. The complaint cites several examples of unnecessary data collection and use, including a “hall of fame” video that ProctorU created of accused cheaters at its own discretion and for no apparent purpose.

EPIC also alleges that Proctorio and Honorlock have deceived consumers by claiming that they don’t use facial recognition technology. The companies have recently insisted that they instead use “facial detection,” which recognizes the presence of a face but can’t confirm its identity. But those technologies are substantially the same, EPIC argues, and Proctorio in particular has claimed in the past—in a series of tweets that have since been deleted—that it does in fact use facial recognition.

But the most novel, and potentially impactful, argument EPIC makes is that the five companies have violated both the D.C. Consumer Protection Procedures Act (DCPPA) and the Federal Trade Commission Act by claiming that their AI algorithms can detect cheaters without providing any evidence to support those claims or transparency to end-users about how the algorithms work. 

The companies’ practices, EPIC argues, are unfair under both the district and federal consumer protection laws because they violate public policy frameworks for the ethical use of AI developed by influential international organizations, including the Universal Guidelines for Artificial Intelligence, which was adopted in 2018 by the International Conference on Data Protection and Privacy Commissioners, and a similar set of guidelines approved in 2019 by the Organization for Economic Cooperation and Development (OECD), which has 36 member nations, including the United States. 

Some of the companies named in the complaint objected to EPIC’s statements.

“Protecting student’s privacy while maintaining the integrity of the testing process sits at the heart of everything we do,” an Honorlock spokesperson wrote in a statement to Motherboard. “The claims against us are baseless and untrue. Honorlock has great partnerships with the schools, students and faculty we work with from coast to coast.” 

“It’s unfortunate the Electronic Privacy Information Center didn’t communicate directly with Respondus prior to sending the letter to the attorney general of Washington D.C.," Respondus COO Jodi Feeney, told Motherboard. "Much of what is stated in the letter is a compilation of misinformation and mischaracterizations that have been circulating on internet discussion boards. We plan to respond to this letter in due time with the factual detail and care these matters deserve.”

The other three companies named in the complaint did not immediately respond to a request for comment.

David Vladeck, a former director of the FTC’s Bureau of Consumer Protection, told Motherboard that EPIC's complaint raises serious issues and that it is “deeply problematic” how little information proctoring companies share about their operations, what data they collect, how they use it, and how the software functions.

“Perhaps above all else, there is no clarity about what factors might lead the algorithm to suspect cheating,” Vladeck added. “An algorithm is only as good as the training data it is based on. Where did that data come from? Is the data representative of the population of test takers?  How is the training data and the algorithm validated, if at all?”

Update: This article was updated to include statements from Honorlock, Respondus, and David Vladeck.