What’s the same about a photo of the most recent Notting Hill Carnival parade and an 18th century portrait of an upper class family? It might not be obvious to a human, but an artificial intelligence programme sees distinct similarities in, say, the composition of the subjects, or colour of their outfits.
A new AI project unveiled at the Tate Britain art gallery in London on Friday
Videos by VICE
draws links between current news photos and archival artworks in a bid to help humans think differently about the images they see.
Called Recognition, the AI programme compares current photojournalism provided by Reuters with works from the Tate’s collection to find its best match. It pairs the images based on image recognition techniques that analyse objects, faces, composition, and context gleaned from metadata.
The AI matches a photo of eunuchs putting on makeup in Mumbai (Image: REUTERS/Danish Siddiqui TPX IMAGES OF THE DAY) with Sir Peter Lely’s Two Ladies of the Lake Family c.1660 (Image: Tate)
“We wanted to understand how we could bring artificial intelligence into a museum, and how rational and objective thinking could be applied to a subjective thing like art,” said Angelo Semeraro of Italian research centre Fabrica. He and teammates Coralie Gourguechon, Monica Lanaro, and Isaac Valentin created the project to enter (and win) the Tate’s 2016 IK Prize for digital innovation, which is run in partnership with Microsoft.
Semeraro explained that the programme brings the Tate collection to life “through the lens of today” but also offers a thoughtful insight into current-day news. “We have the perception that what is happening right now has never happened,” he said. “What we want to show is how present and past can be intertwined.”
Recent examples of the AI’s matches include some clear parallels—the AI matched a news photo of a dog with a 19th century painting of a dog—but also many less obvious or more provocative pairings. The AI matched a photo of industrial robots assembling Ford cars with abstract painting Mars Ascends by Bryan Wynter, and a photo of riot police standing guard in Harare, Zimbabwe with a more peaceful landscape by watercolour artist William Henry Hunt.
Left: Holiday makers swim in the Bassin d’Arcachon, southwestern France, August 16, 2016. (Image: REUTERS/Ragis Duvignau). Right: Henry Scott Tuke August Blue 1893–4 (Image: Tate)
What’s really cool, however, is that viewers can dig down and see why the AI has made the link between any particular pair of images. Visitors to the Recognition “virtual gallery” can see the programme in action as it takes a new photo and cycles through potential matches before settling on one. Then, you can select a pairing and see what the AI was “thinking” when it put them together, such as which objects it thinks it’s recognised in each image, or which contextual tags it relied on. I’d recommend having a play around with it.
Sometimes the AI makes “mistakes,” but you can still discern a kind of logic in the comparison. A pairing of a photo of a Formula 1 car with a painting of musical instruments reveals the AI programme thought the latter depicted “a black and silver boat”—but the basic shapes in both images reveal an overall compositional similarity a human might not have clocked.
“For us it’s quite interesting how a human can understand how a machine thinks,” said Semeraro. He calls instances where the AI appears to get it wrong “beautiful mistakes.”
Read More: Artificial Intelligence Can Now Paint Like Art’s Greatest Masters
Dave Coplin, who has the title of Chief Envisioning Officer at Microsoft, told Motherboard that the “human emotions side” of the tech was most interesting to him, made by possible by algorithms that can tell things such as age, gender, and even emotional state when shown a face (as seen elsewhere such as in Microsoft’s how-old.net website that crushed everyone’s egos last year).
Coplin sees the Recognition work as an example of AI “extending the reach of what humans can achieve.”
“The beautiful thing for me is you’ve got this amazing archive—somewhere between 50,000 and 70,000 pieces of art—that nobody, even the people who work here, can hold in their heads,” he said. “So how much of that lies hidden in an archive somewhere that doesn’t get surfaced? What this is going to do is start to light that up.”
The whole project is almost as much a lesson in how AI works as it is an art exercise, and a reminder that algorithms such as those used in facial recognition programmes are doing little more than spotting patterns within the parameters they’ve been trained to work with.
In the Tate Britain, a display will allow users to help guide the AI by selecting their own matches that the programme can then incorporate into its training.
The project runs for three months, and as it uses up-to-date news images, there’s no telling what it may have to respond to.
