Synesthesia seems like some kind of awesome superpower, less so like a neurological abnormality. Simply: it’s when the brain translates images as sounds; or sounds as colors; or taste as touch; or really any other situation you might interpret as crossed sensory wires. Synesthetes feel one sense as another sense. Artist Perry Hall calls his own synesthesia a “tic of perception.” Hall also wouldn’t call it a superpower: “It’s just like anything else that makes someone want to play music or make visual art,” he told me last week. Yet, he designed a pretty cool app with the the concept of synesthesia in mind, called Sonified.
Basically, Sonified takes a moving image and translates it as sound. Say you’re riding the bus on a rainy day and it’s one of those classic scenes that feels so rich with mood you can almost hear some saddish chamber strings in the background. Well, hold up your iPhone to the window and the Sonified app will give you music. I’ll say that it’s the first augmented reality app that I’m remotely interested in. Sorry, Twitter 360. It’s still perhaps a bit limited in its sounds: Sonified is based on a few handfuls of loops, not a bottomless catalog, or at least not yet. Expanding the range of that and fast is key to keeping something like this from the novelty app waste bin. But as proof-of-concept, well, here you go:
Videos by VICE
“It’s essentially sixteen loops, organized relative to color and brightness,” Hall says, “with each loop dedicated to a specific subdivision of the color wheel or the greyscale. Imagine a 16 track mix, and colors and brightness values a camera is seeing pushing the faders up and down.” The challenge was in getting the app to do this instantaneously, like in the sense of touching a guitar string and hearing sound, but with an added computational burden. “Essentially, color and or brightness are sampled 25 times a second, [and] turned into numeric ranges.”
Hall has huge plans for this thing. Imagine a sort of Sonified record label, where artists make and release music in this. . .format? Soon enough he plans to post Sonified hacking instructions so people can put their own music on it right now. “Sixteen tracks of anything would work when placed into the app,” he says, “but there’s obviously a whole brave new world of how to write music for Sonified. The most simple way would be, as I did, to write textures, ambiences. But sixteen tracks of distorted feedback might also be interesting; one could also write repeating, cycling piano phrases that recombine and shift based on light.
“Or, on a completely different note,” Hall continues, “take unique sound design from a film — imagine some of the sounds from David Lynch’s Lost Highway placed into Sonified — and as you film your own darkened house, the sound design from Lost Highway is mixed in a one of a kind way based off of how you move the camera and what you film.” Hall notes that his software is set up to go beyond an iPhone camera, and would work with something like a RED camera even. It’s not hard to imagine film sound design being based on this in the future. With Hollywood’s support, Hall could be getting rich faster than whoever’s behind Pinterest. Good work, dude.
Connection:
- Can the Shark Week App Smell the Period Tracker App On My iPhone?
- The Cop iPhone App That Knows Your Face and Can Send You to Jail
Reach this writer at michaelb@motherboard.tv.
