Microsoft Flight Simulator is a beautiful game that promises players the world, literally. Using a combination of satellite imagery and 3D photogrammetry from Bing Maps, and machine learning to fill in the gaps, Flight Simulator allows players to fly across the entire planet and visit landmarks such as the Eiffel Tower or notorious locations such as Jeffrey Epestein’s private island.
Flight Simulator’s premise is interesting enough that researchers who use open source intelligence to report the news and investigate crimes wondered if it could be a new tool in their arsenal. Is it possible to match real-world photos with locations in the game or, say, fly over China’s Uyghur concentration camps in Xinjiang?
Videos by VICE
Giancarlo Fiorella, an investigative journalist working for the open source research and journalist collective Bellingcat, had the same question. Fiorella is a gamer and said “that promise that Microsoft made that you could go anywhere in the world” drew him to investigate the game as a possible new tool for Bellingcat’s research. It caught his attention, he said, because it renders 3D buildings in addition to the world’s terrain.
“If this simulation…is really good, and in particular, if the rendering of 3D structures is really good, then that might actually help us with geolocation when we’re trying to figure out where a video or picture was taken,” he told Motherboard over the phone.
Bellingcat uses satellite imagery, reverse image search, flight logs, and other publicly available information to report the news and investigate crimes. In 2017, it used the social media posts of Libyan military commander Mahmoud Mustafa Busayf Al-Werfalli to build a case that he had committed war crimes. The International Criminal Court issued an arrest warrant based on Bellingcat’s evidence.
“We use Google Earth Pro all the time, that’s the default geolocation tool,” he said. Google Earth Pro renders a 3D topographical map of the Earth. It gives you the mountains and valleys of the planet. “What you don’t really get is buildings. If the simulation was able to render buildings accurately, it might give us more detail about a place because now we’ll have an idea for how a city is laid out.”
The game gets a lot right thanks to the combination of satellite imagery, photogrammetry, and machine learning, even in conflict zones. For example, Bellingcat researcher Aric Toler posted on Twitter that the game accurately features artillery craters around the Donbas in Ukraine. The game also renders 3D buildings at the site of a “new Russian military base” just kilometres from the Ukrainian border, Toler tweeted.
Fiorella took to the skies in Flight Simulator and documented his findings on Twitter. He traveled to conflict zones, military bases, detention camps, and other geopolitically controversial areas. He witnessed a neighborhood that had been destroyed in northeastern Damascus, and the game rendered the destruction as well as a few 3D buildings still standing. In the game’s version of North Korea, Fiorella could see the Hodo Peninsula military training facility. Some of the Uyghur camps in Xinjiang China are also present in Flight Simulator, he said.
It was impressive, but Fiorella decided pretty quickly that the game wasn’t quite accurate enough for Bellingcat’s investigations.
“I personally wouldn’t use it as a tool for geolocation for two reasons,” he said. “One, it makes mistakes. I’ve seen lots of examples where you can see the base layer of a building and, for whatever reason, the game didn’t recognize it as a building so it didn’t render.”
“I’ve also seen buildings that are rendered but their heights are incorrect,” he added.
This is a common occurrence in the game, which, remember, is mostly supposed to be seen from cruising altitude. On Jeffrey Epstein’s island, many of the finer details of the island’s buildings didn’t render, and many buildings didn’t render at all. Buckingham Palace in the U.K. looks like an office complex. Many less-famous structures, when viewed from a closer angle, appear as misshapen, vaguely building-like lumps.
While the exact reasons for this are unclear, it’s likely due to some combination of AI-generated greebling not meant to be seen up close, a lack of photogrammetry in those locations, and data errors. In suburban Australia, for example, a 212- story obelisk towers over the city-scape. The reason? A student accidentally reported the number of stories in the building as “212” when editing open source street data.
This led to some issues with Fiorella’s research. Many of the concentration camps in China are new and the satellite imagery available on Bing is old, he said. “The first camp that I visited is built. But I went in the game, it was just an empty field. I checked on Bing [and] the Bing image is older. The camp wasn’t built yet,” he said. “It’s not like Microsoft is censoring their location. I went to another one, an older one, and it was there.”
Fiorella discovered that Flight Simulator even seems to render some buildings that Bing Maps has censored with pixelation. Fiorella flew to a Naval base in Lorient and the Cattenom Nuclear Power Plant, both in France and both pixelated on Bing. He said the base of the structures looks slightly pixelated but that the game did render buildings in the area. “Probably the game is pulling the non-pixelated satellite imagery and it’s rendering the buildings,” he said.
When Motherboard reached Microsoft, the company declined to comment but indicated that when an area is blurred or pixelated on Bing, it uses procedural techniques to fill in the blanks and make sure there is something in the space.
For Fiorella, Flight Simulator is fun but not quite good enough to be a journalistic tool. “It’s too unreliable,” he said. “In places where we don’t have any other imagery, if we rely on what we see in Microsoft Flight Simulator, we’re opening ourselves up to making mistakes because we’re trusting the AI to render places correctly and it doesn’t.”
He said he might use it to help him get a feel for a city or to get oriented, but that he could never rely on it as the final word. “For a game,” he said, “It’s a really cool idea…but it’s not good for the nitty gritty, precise work that we do.”
