Help! I Can't Stop Thinking in Emoji!

I have a problem with emojis, I think. What I mean is, sometimes I think in emojis. When I’m looking for a word or phrase to respond to things outside the contexts of texts or Twitter what pops into my head are emoji characters. My girlfriend Sarah first pointed out that this was happening to her a few months ago after she found herself unable to produce any alternative to ”

” as a response to an email from her boss.

Videos by VICE

I didn’t realize that this was happening to me until the other day when I was reading an article for the class I’m teaching by the notoriously dry poetry critic Cleanth Brooks. In the article Brooks is trying to argue that literary critics shouldn’t take into account an author’s judgment about the quality of their own work when performing criticism. His example is a take on one of Hemingway’s lesser novels: “Ernest Hemingway’s statement in a recent issue of Time magazine that he counts his last novel as his best is of interest to Hemingway’s biography, but most readers of Across the River and Into the Trees would agree that it provides nothing at all about the value of the novel—that in this case the judgment is simply pathetically inept.”

My immediate mental response to this sentence was ”

“. But the more I thought about it, the more my reaction turned into

and I actually ended up drawing the eyes themselves in the margin next to the sentence. Obviously Brooks’s shade is not subtle, which I think is why the fire emoji was my first reaction. You don’t just call someone’s judgment “simply pathetically inept” without meaning to hurt their feelings, especially in 1951 when this piece was published. Brooks is taking a cheap shot, and it’s hilarious, but the inappropriateness of the venue for a statement like that is what made the emoji eyes feel more correct. What does it mean that my first conscious response to an English sentence was a series of cartoon images? Has my mind been so thoroughly saturated by my use of emoji in certain contexts that I am actually generating cognitive content in emoji?

Related: “The Digital Love Industry”

The surprise of thinking in tiny cartoon images comes from the expectation that language is going to pop into your head when you think. So what makes emoji different from written language? How do emojis mean and is it different from how words mean? Obviously, emojis are pictures of things for the most part, and the history of the

development of emoji is an interesting read. Sure there are emojis like and and , but for the most part emojis represent people and common objects that their developers felt most required graphic shorthand for digital communication.

The apparent difference between emojis and words is that the meaning of a word is not caused by some inherent link between the word and the thing or concept that it names or means. Linguists call this “the arbitrariness of linguistic signs.” Even apparently onomatopoetic words (words whose pronunciation imitates sounds) are arbitrary. For proof of this, check out this Australian guy’s insanely comprehensive chart of the ways different languages onomatopoetically represent animal noises.

Emojis seem to violate the rule of arbitrariness because they aren’t words. They seem to very clearly represent something distinct and real about the world—a face, or a certain gesture. From habit, we associate these faces or gestures with emotional meaning and social significance. The fact that emojis have names further complicates things. If you browse

emojipedia you start to notice that most of the emoji names are exceedingly literal. This causes some complication when the literalness of the names is applied to the face emojis. Oddly, the attempt to produce as literal a name as possible means that face emojis ended up getting named in two ways: the name either describes what the face looks like or indicates what emotional state the emoji is supposed to represent. Often two emojis that look similar to each other are named differently from each other. So for example gets the descriptive name “face with open mouth and cold sweat” while , which is the only emoji I ever use to convey sad tears, gets the name “disappointed but relieved face.” To a certain extent, both of these names are descriptions, and both involve elements of implied emotional content (is that little emoji’s blue forehead making his sweat “cold”?). The second emoji apparently represents the facial expression of the complex feeling “disappointed but relieved,” which is probably the feeling you have when you have been afraid to ask someone out for a long time, and when you do they say no, but it feels good to at least know for certain. But would you ever send as the capstone of a text conversation that ran along those lines? Who would you be communicating that emotion to? Your mom?

If you look at another pair of emojis,

(frowning face with open mouth) and

(anguished face) the same problem appears. On the one hand I’m not at all convinced

is what anguish looks like, but on the other there doesn’t seem to be any rhyme or reason to what emojis get descriptive names and which get representative names. The only difference between these two emojis are the eyebrows. What does that have to do with how the name is assigned? Things get even weirder when you get into the more realistic human emojis. The woman in pink series is particularly troubling. Sure, I can get behind

being named “happy person raising one hand.” That’s a solid mixed descriptive and representative name. She is happy because she is smiling, and she is raising one hand. I get that. But

is named “information desk person” which…

The more I thought about it, the more that thinking about the names of emojis seemed to be getting me away from the problem I was really worried about: the fact that I’ve been thinking in emoji. I ended up calling a linguist, Neil Cohn, a post-doc at UC San Diego who researches visual language and cognition in order to get to the heart of the problem. I explained my problem to him, that I was freaking out because I didn’t have language for the emoji thoughts I was having, and he told me that my problem wasn’t thinking in emoji, it was how I was thinking about language itself. “Humans have three ways of communicating, really only three,” he explains, “we can speak, making words with our voices, we can sign, like with sign language, and we can draw.” As his website explains, “drawing—especially sequential images—is structured like language. Just like words in sentences are used in spoken languages, sequences of images can create a visual language.”

Cohn’s work is the kind of research you read that is at once deeply familiar and deeply mind-blowing. His website includes some easy to follow summaries of his pretty radical linguistic theories. His big idea is that drawing, far from being a mere supplementary form of communication, is actually an intrinsic aspect of human communication and cognition.The kicker, and this is what makes Cohn’s theory unique, is that visual languages obey “similar principles of cognition” to those that we already know govern our understandings of verbal and sign languages. Cohn explains that visual language, though different in its specifics from verbal language, is likewise made up of distinct observable rules in our common practice of creating and reading pictures. “A really common thing is for people to say that they ‘can’t draw’,” Cohn tells me, “but what that really means is they have under-developed capacities for expression within visual language.”

But this isn’t a personal failing. Instead, Cohn insists, we live in a society where visual language development is not a priority. The result is that we have what seems like an intuitive sense of how to read images, but most people don’t develop the dexterity within what Cohn calls “American visual language” to produce pictures that look like the ones they are used to reading without any trouble. “If you ask someone to draw a house,” he tells me, “Most people will draw a triangle on top of a square. That’s ‘house’ in American visual language, but that’s not what houses look like in real life,” Cohn explains. We are able to recognize objects in drawings partly through visual similarity, but that isn’t necessary and may not even be a primary component of the way our brains process images.

This is true even for human faces and emotions. In a blog post from way back in 2009 Cohn discusses an fMRI study that revealed that, in the authors of the study’s words, “Remarkably, emoticons convey emotions without cognition of faces.” Researchers scanned the brains of subjects looking at non-rotated emoticons like ¯\_(ツ)_/¯, and digitally averaged photos of faces and found that while the photos of faces triggered both facial recognition and emotional recognition areas of the brain, the emoticons lit up only emotional recognition areas. Cohn writes on his blog, “this finding has very interesting consequences for understanding how brains process varying degrees of complexity in images. The implication here at least is that more simplified faces become tied more explicitly to a “symbolic” meaning as opposed to their iconic meaning of resembling what they look like. That is, more simplified images strip down the meaning to its core meaning disconnected to the iconic reference that they are framed within.” In other words, when we look at emoticons our brains don’t process faces (emoticon as iconic representation of a real face), but, once we are habituated to the rules of emoticon use, we take emoticons as symbols of emotional content. When it comes to emoticons, they convey emotional content the same way that words or signs do, abstractly, not through their representation of emotion on a drawn face.

Web content like GIFs give us moving images. Our brains assemble the images the GIF provides frame by frame, producing the effect of movement. When you are limited to still emojis, you can only accomplish this by having time play out like it does in written language, proceeding from left to right across the visual field. This is my favorite kind of emoji play. I like to send people this:

. Typical emoji grammar would have us perceive this as three separate faces positioned next to each other. That’s the logic of the see/speak/hear no evil monkeys

. We don’t typically perceive this sequence as the same monkey making all three gestures. But if you insert the plain monkey face between each of the other images, you might perceive the series as a temporal sequence:

. You can accomplish this with other emoji series, like the pink shirt woman. Here she is thinking she recognizes someone, realizing she is mistaken, and then feeling embarrassed:

. It’s OK lady, that happens to everybody.

What’s happening in the way we perceive these strings of emojis as time-sequenced images is governed by two constraints that Cohn argues determine how we make judgments about the identity of images and where our attention should go from image to image. These constraints, the continuity constraint and the activity constraint, account for how we make sense of more complex sequenced images, like comic strips. Cohn explains, “The continuity constraint is the idea that, in order to understand things in sequence, you try to establish continuity between them buy assuming that things repeat across panels.” The effect for the sequences of emojis means that we tend to judge that “the smiley faces aren’t all different faces, but are one face connected across time. The activity constraint would then say, of the things that maintain continuity, you’ll care about the ones that change more than the ones that don’t change.” So in the series of monkey emojis, we focus on the position of the hands because they are so obviously active, though we might fail to notice that the plain monkey face emoji has a slightly differently shaped head than the other three.

The typical way that most people think about writing, Cohen says, is that the written word is the transcription of a verbal word, which somehow comes first or is more authentic. The written word “dog” is a form of the spoken word “dog,” which is then linked in our brain to a conceptual meaning. Cohn says this description of the state of things is basically correct, but what most people don’t understand is that this is profoundly unnatural.

“Writing is basically a kind of culturally learned synesthesia,” Cohn explains. When we read we force visual content through verbal cognitive channels before they can connect to conceptual content. In effect, the written word triggers a verbal response that then proceeds to connect to the meanings we have for that word. Learning to read and write takes so much effort precisely because connecting the visual and verbal pathways isn’t something that happens organically. When it comes to writing, we make visual images obey, or approximate, grammatical laws of verbal speech.

Inserting emojis in place of words basically explodes the grammar of the written sentence, because not only does it produce confusion about the meaning of the particular word, but also destabilizes the syntax of the entire sentence because aspects of language like part of speech and tense determine the way that other words in the sentence produce meaning together. Our speech is highly adaptive and capable of expressing new and complex emotional content because it is endlessly combinative, so new meanings are always popping up as we need them. Emoji, on the other hand, is an incredibly limited visual language. It’s basically just a set of nouns, and we manipulate them through other visual grammars in order to produce complex meanings like movement, tense, and intensity.

So I’m not crazy for thinking in emoji for the same reason that I’m not crazy for flipping off refs when they don’t call travels when I’m watching basketball. All of what we should really call our thought finds multiple outlets in our verbal, gestural and visual capacities, and as Cohn suggests, “we are designed to use all three at once.”

means something very particular to me, just like

means something very particular to my girlfriend. It turns out there’s even an emoji (in a sense) in Faulkner’s The Sound and the Fury, where a character encounters a sign outside of a town that reads:

For Faulkner, as the critic Nicolas Tredell suggests in his

book on The Sound and the Fury and As I Lay Dying, there is a pun on the French word “mot” (meaning “word”) and in the phonetic similarity between “son” and “sound” in the name “Mottson.” Tredell argues that Faulkner’s sign means, “keep your eye on the word-sound, on the sound of the word that does not appear, ‘I’.” The eye emoji disrupts the very idea of language because it is neither a word nor a sound, and in doing so it challenges the coherence of our language, thought, and being. The graphic image that disrupts the text for Faulkner has become part of our everyday pragmatic use of language, but the apparently sure-fired connection between sound, image, and meaning can still catch us by surprise and remind us that even as our communication technology promises seamless identification with what we produce, we are still figuring out the basics of how language makes our thought possible. And that might make you feel .