Hundreds of miles above our heads, Earth-imaging satellites are hard at work, snapping high-resolution photos of our planet and its inhabitants (say cheese). This abundance of imagery has revolutionized how we travel, communicate, and perceive our surroundings, and also provides unexpected insights into our civilization that aren’t obvious at ground level—like the impact of a neighborhood’s wealth on its overhead appearance.
This idea served as the inspiration for the newly launched tool Penny, an artificial intelligence platform trained to use satellite pictures to predict the income brackets of urban neighborhoods. Developed by the Colorado-based satellite company DigitalGlobe, the data visualization studio Stamen Design, and researchers at Carnegie Mellon University, Penny generates estimates of a particular city block’s wealth based on satellite imagery, and compares its results with real census data.
Videos by VICE
Penny also acts as a virtual sandbox, where users can click-and-drop items like trees, freeways, or baseball diamonds into neighborhoods to see how they affect the AI’s perception of an area’s wealth.
But before you start scouting out promising real estate investments or urban planning initiatives, note that Penny is still in its first incarnation, and sometimes gets estimates wrong (still, it’s accurate about 86 percent of the time).
For example, the AI is thrown off by parts of Brooklyn’s Greenpoint, a relatively wealthy neighborhood, perhaps because the satellite view matches Penny’s preconceptions about low-income areas—places that are relatively devoid of skyscrapers or greenspace. The AI can’t anticipate subtler factors that contribute to the high property values of places like Greenpoint, such as access to backyard beer gardens and fancy donuts.
According to DigitalGlobe data engineer and project manager Jordan Winkler, who helped create Penny, the takeaway for users at this point should be exploring “the power of applying artificial intelligence to satellite imagery” using “a very humanistic lens.”
“I think a lot about how to tell better stories around the work that’s emerging with [DigitalGlobe’s] geospatial big data applications,” Winkler told me over the phone. “One of the things that’s both powerful and challenging about working with this amazing archive of satellite imagery, which is basically a virtual time machine of the planet, is that the possibilities become endless.”
Greenifying East Harlem. Imagery courtesy © 2017 DigitalGlobe
With that in mind, Winkler and his team focused on building a program that showed off the far-ranging potential of geospatial data, but that would also be interactive and fun for people to play with.
In the current version, for instance, you can explore and modify two cities, New York City and St. Louis, by adding vehicles, solar panels, or famous landmarks, like NYC’s Empire State Building or St. Louis’ Gateway Arch, to random blocks and buildings.
The results are often counterintuitive: When I started dropping boats onto Manhattan skyscraper rooftops, I expected the wealth of the area to go up, but it stayed constant. Likewise, depositing the Statue of Liberty into the middle of Queens Boulevard did not convince Penny to raise its estimates of property value. This exposes some of the different biases and expectations that a human user has when looking at a satellite map, compared with an AI.
“Penny looks at patterns in imagery and matches them to what we’ve provided in terms of census estimates,” Winkler said. “It doesn’t know what a tree is or what it’s intrinsic value is. It simply has learned a correlation between certain colors and shapes with the income information we’ve given it.”
This explains why dropping trees into an area almost always led to a higher income estimate, while parking lots and baseball diamonds tended to lower the projected wealth. But even these correlations are not always consistent, because Penny weighs many factors in its guesswork.
“It’s not just the presence of a new feature in the image, it’s also what that new feature covers up,” Winkler explained. “If you put a tree on top of grass, it’s unlikely to have the same effect as if you cover up a parking lot, for example.”
The tool has already garnered some interest from commercial ventures, and the team may incorporate more datasets, such as crime rates or health statistics, in future versions.
Penny may also expand its scope to other cities, which could shed light on how signifiers of wealth vary depending on cultural and geographic factors. Backyard pools are no doubt more common in Miami and Los Angeles than NYC and St. Louis, for example, so Penny has to approach each new city with a unique dataset that reflects the income markers of that place, be it Paris, Jerusalem, or Shanghai.
Read More: We’re Fighting Famine in South Sudan With Images From Outer Space
But as Winkler emphasized, the current version of Penny is primarily designed to invite the public into the emerging intersection of AI and satellite imagery, where densely detailed maps become a kind of geospatial canvas for users to paint on.
“I’m far less interested in making Penny the most accurate predictor of wealth than I am in starting conversations about where the future of this technology should go, and what it means to be human in this increasingly AI-dominated world,” Winkler said. “We may find that our human perception of the world becomes increasingly shifted as a result of relying on AI, as we start to focus on the world in ways we otherwise wouldn’t.”
Get six of our favorite Motherboard stories every day by signing up for our newsletter.
