Utah gave a five-year, $20.7 million contract to an AI surveillance company that did not actually have any AI capabilities, a state auditor has found.
The company, called Banjo, collected a huge stream of data from cities and counties in Utah and proposed to analyze traffic cameras, CCTV cameras, social media, 911 emergency systems, and location data in real time in order to alert police to where crime was occurring. It also proposed to create a “solution for homelessness” and to detect “opioid events” using its software, called Live Time. Though Banjo was collecting a huge amount of data, it was not using AI in the way it said it did and thus, in the auditor’s eyes, did not present as much of a privacy threat as originally feared simply because it was not sophisticated enough.
Videos by VICE
“The actual capabilities of Live Time appeared inconsistent with Banjo’s claims,” John Dugall, a state auditor, wrote in a report released this week. “Other competing vendors might have been able to meet the ‘lower’ standard of actual Live Time capabilities, but were not given consideration because the [request for proposal] responses were judged based on ‘claims’ rather than actual capability.”
Do you know anything else about Banjo’s work? Do you know about any other apps that abused data access? We’d love to hear from you. Using a non-work phone or computer, you can contact Jason Koebler securely on Signal on +1 202 505 1702 , or Joseph Cox on Signal on +44 20 8133 5190 , Wickr on josephcox, OTR chat on jfcox@jabber.ccc.de , or email joseph.cox@vice.com.
“Because of the reduced capability of the Live Time system, it appears much less likely personally‐identifiable information (PII) was accessed, transferred, and used than was previously feared,” the report added.
After Motherboard reported about the privacy issues inherent in such an expansive surveillance proposal and a OneZero investigation found that Banjo CEO Damien Patton once pleaded guilty to helping a KKK leader shoot up a synagogue, the state’s five-year, $20.7 million contract was suspended and a privacy audit was commissioned. The company has since rebranded as “safeXai.”
In its sales pitches to the state, Banjo “solved” a simulated child abduction case in “27 seconds,” according to documents obtained by Motherboard using a public records request and presentations Banjo made at the time. The Utah Attorney General’s office was particularly impressed by this simulation, though it declined to tell Motherboard what it entailed. According to the new state audit, the technology used in this simulation was not verified by the state, and the auditor has doubts that it used any AI.
“The touted example of the system assisting in ‘solving’ a simulated child abduction was not validated by the AGO and was simply accepted based on Banjo’s representation,” the auditor wrote. “In other words, it would appear that the result could have been that of a skilled operator as Live Time lacked the advertised AI technology.”
In Banjo’s sales pitch to the state, it noted, among other things, that “our artificial intelligence unsiloes and synthesizes [data] signals simultaneously to understand the context of an event.” At a 2018 state conference assembled by Sen. Mike Lee, Patton talked about how his AI could be used to solve the opiate crisis and noted that Banjo’s “technology has become one of the leading artificial intelligence companies in the world.”
In a 2020 demonstration to the auditor, however, Banjo claimed that none of its technology is actually “artificial intelligence,” and the auditor found that it does not have most of the capabilities it originally said it did.
“Banjo expressly represented to the Commission that Banjo does not use techniques that meet the industry definition of Artificial Intelligence,” Dugall wrote.
Nonetheless, the data given to Banjo by the state was sensitive and should not have been done in the way it was, the auditor found: “The architecture of Live Time’s access to certain public safety systems should not have been permitted based on existing industry best practices,” they wrote.
The report also found security concerns with Banjo’s direct access to some databases.
“Live Time’s configuration lacked certain key security features and Banjo’s approach didn’t follow best practices,” the report reads. This direct access could have allowed a malicious insider to alter, for example, state emergency databases without the knowledge of state officials. It also could have allowed for accidental destruction or modification of state emergency databases if the software was misconfigured.
“Permitting a third party to have direct database query, as permitted under the agreements with Banjo, exposes the entity and its data to: 1) the risk of misconfigured security leading to inappropriate access to restricted, sensitive or personal data and 2) the risk of misappropriation or theft of data,” the audit notes. “For misconfigured databases, additional risks include 1) the risk of modification of data or 2) the risk of the injection of malicious code into the database, hijacking the entity’s resources for nefarious purposes.”
“Live Time lacked the advertised AI technology.”
Another Motherboard investigation found that Banjo used a secret company and a series of apps that were designed to appear innocuous to surreptitiously scrape social media.
As a result of the audit, the Office of the State Auditor released new privacy and contracting guidelines for the state to use during the procurement process.
Utah Attorney General Sean Reyes, who has repeatedly spoken highly of Patton, wrote a letter responding to the state auditor suggesting that its findings prove that the state was right all along.
“We are encouraged by your findings and feel validated that neither privacy intrusion nor racial or religious bias was inherent in the Banjo Live Time system,” Reyes wrote. “Your findings align with our experience regarding this company, its founder, priorities, work product, and ethics. We observed, and you have confirmed, that sensitive PII was never shared with Banjo. That protection was always a high priority for this office.”
Reyes also said that his office went “above and beyond” in vetting Banjo and Patton, and that Patton’s past with the KKK and Neo-Nazis did not impact his work with Banjo.
“Based on our first-hand experience and close observation, we are convinced the horrible mistakes of the founder’s youth never carried over in any malevolent way to Banjo, his other initiatives, attitudes, or character,” he wrote.
