Your Personal Pocket Frenemy

How the human-like nature of AI is changing the way we interact with machines. 

By Daniel Merino

Kelsey Roddick is a smart woman. She is too humble to say so, but an undergraduate degree in neuroscience from Dartmouth and a career in biometric research are proof enough. Roddick needed a lawyer last winter after she found out that her apartment north of Boston had a bad heater and wouldn’t warm up past 48 degrees. When the landlord went AWOL, Roddick hired a lawyer. A few days after signing a retainer she received an email from her lawyers assistant, Andrew Ingram, whose picture on his account showed a dark-haired young man. Roddick found him prompt in his responses and easy to work with. “He answers all my emails, he understands everything,” she says. But a few weeks after they had begun emailing, Roddick got a weird feeling about Andrew.  She clicked the bottom of the last email she had received from him, wanting to see the signature that Gmail often hides. It was only then that Roddick realized who she had been talking to. “Andrew Ingram. An artificially intelligent scheduling assistant, by”

Andrew Ingram is made by a company called He has a twin sister named Amy Ingram (note their initials) and their only function is to schedule appointments for those who buy their services. Roddick had been interacting with this AI for weeks, apologizing to the AI when she was slow to respond to emails and thanking it for its help. “You feel like you are talking to a real person,” she says.

While Roddick’s case is rare, the technology behind the conversational AI she interacted with is creeping into many of the channels of communication we use today. Andrew felt so human to Roddick that she had no reason to doubt she was speaking with a real person.  But even if you know that the intelligence you are talking with is artificial, research shows that the more human a machine acts, the more human you’re likely to treat it.

How we as humans categorize an entity we interact with – be it over text, voice, a video feed or even in the physical world – dramatically effects the rules of social communication with that entity. Trustworthiness, influence, eeriness, empathy and a host of other subconscious reactions all connect to how we conceive of a person or machine we communicate with. As machines increasingly become part of the formerly human-only world of communication, it is important to understand how we categorize these voices in our phones because how we categorize them changes how we think, act, and react to these AI’s.

As ubiquitous as they now seem, AI voice-bots are relatively new. A mere decade ago, people considered conversational AI a thing of science fiction novels, but that all changed when Apple launched Siri in 2011.

“Siri came out and I was just enthralled,” says Dr. Andrea Guzman, who studies human-machine communication at the University of Illinois, Chicago.

Siri was a leap forward in access to AI for the average person. Previously, human-machine communication was a one-way interaction. People could give simple verbal commands to their phones or computers, but the devices never answered back. With Siri and the many conversational AIs that have followed,” it is really starting to resemble a natural conversation, looking more and more like human communication,” says Guzman.

Today, we constantly interact with conversational AI in the form of twitter bots, automated help centers, Alexa, Gmail’s Autocomplete, and AI news writing applications (or if you are in China, AI newscasters). As Roddick can attest, the line between human and machine isn’t so clear. Perhaps the best example of progress in vocal communication is Google Duplex, the next generation of the Google Assistant, which was introduced in a small number of cities late last year. The public got its first glimpse of this technology at a June 2018 Google Expo in Mountainview, California. Sundar Pichai, Google’s CEO, stood onstage in front of a packed auditorium as a recording played demonstrating the technology in use. The demonstration started with a voice asking her Google assistant to “Make me a haircut appointment on Tuesday morning anytime between 10 and 12.” The phone rings and a freakishly real female voice announces that she would like to book an appointment for her “client.” As the auditorium crowd gasps and cheers, the AI asks for a 12 o’clock haircut to which the very human receptionist replies they only have a 1:15 available. The receptionist and the machine go back and forth –the AI using human-like pauses, “umms” and “mm-hmms” –eventually agreeing on a 10 a.m. appointment. This back-and-forth represents a giant leap forward from the command-and-respond communication that conversational AI is currently capable of.

With this ongoing progress from bad-impressions to near-indistinguishable fakes, many researchers say that the day will soon come when people can’t tell the fake voices from the real ones. With uses in nearly every sector, the instances where humans and machines interact is only going to increase. Cognitive scientists wonder how we humans will react.

To understand how people interact with AIs, researchers first need to know how people categorize a program like Siri. In a 2016 study, Guzman sought to answer that. She and her team set up shop at subway entrances in Chicago and asked commuters a set of questions designed to understand their conceptualization of Siri or other virtual assistants “When you are talking to a machine, what do think it is? Where do you think it is situated in this world?”

When she asked these unsuspecting Chicagoans how they thought about Siri, most had a hard time describing what exactly she was. They described Siri as something not quite machine, but not quite human. Most called her a “she” but had never thought about why. She “lived” in the phone to some, but she was the phone to others. “We have words for biological entities and words for mechanical entities,” says Guzman. “What we don’t have is a word that captures something that is humanlike but is also somewhat mechanical.” According to Guzman, we do not have a word nor a concept for these communicating entities. In a follow-up paper published in January of this year, Guzman notes that 70% of Siri users consider her a “voice in the machine,” but what constitutes that voice is unclear to them.

This unnamed, cyborgean gray area is where a number of researchers are probing in order to understand how we react to machines that live there. “As human beings, our brains are hard-wired to become social animals,” says Dr. Sun Joo Ahn, director of the Games and Virtual Environments Lab at the University of Georgia. “When you are in the wild you want friends [and] kin to help you survive. So when something behaves socially, you prefer that social interaction.”

Ahn has shown that the perceived “human touch,” whether real or not, plays a huge role in determining social influence. In a 2015 meta-analysis, she found that entities perceived to be controlled by humans held much more social influence than those perceived to be pre-programmed. This makes sense: interactions between people are governed by rules, evolutionary and cultural, that mediate how we act in the social world. Ahn found that even when we know that a machine is acting without a human controller, the way it acts changes its social influence, the more human, the more influence. In other words, if a machine smiles at us while talking, we are more likely to listen to it than if we are simply looking at a microchip and a speaker.

This human reaction to something non-human may be due to something called attribution of mind. “Having the capacity to actually experience things like a human lends your thoughts an authority, a kind of weight,” says Dr. Kurt Gray, director of the Mind Perception and Morality Lab at the University of North Carolina, Chapel Hill.

This weight, the social influence that machines and those who control them increasingly wield in the world, is something to be wary of. In a society driven largely by profit, companies naturally want to maximize their influence. The entire ad world is dedicated to this pursuit, we are all too familiar with their schemes and plans to sell us stuff. Conversational AI, on the other hand, is a new and not well-understood tool that could be dramatically effective in selling us goods and ideas in a way we are not used to nor prepared for as a society.  

This opens up a huge set of ethical questions. What data should third party AIs have access to? Should an AI be required to disclose that it is not a non-human? Who gets to control an AI on your phone — you or the phone company? If an AI realizes you are doing something potentially detrimental to yourself or others, should it tell you, should it tell the police, should it blur that already thin line between the physical and digital world and stop you?

In the demonstration video of Google’s Duplex AI from last summer, there is an uncomfortable lack of a disclosure statement. The AI never announced that it was an AI. Google received a lot of flak over that and has since released a statement saying that all future interactions will come with some sort of declaration that you are talking to an AI, but so did the emails that Roddick received. A disclosure statement is a good, and necessary, starting point though.

“I bet it is for the best if Google tells you when you are talking with an AI,” says Gray. “If you ever caught that you were being lied to, you would likely never trust them again.”

Ahn agrees. “If there is a commercial aspect to it, I think for sure there needs to be disclosure.” But she also imagines situations, like in nursing homes or for the treatment of mentally ill persons, where the idea that the voice is truly a human is a good thing. “ If it is meant to be therapeutic, if there is a treatment goal to it, then it gets a little blurrier,” she says.

Blurry – that is the theme of AI. Its potential to change the world is undeniable, but in what way? Gray seems, fittingly, to be of two, almost contradictory minds about it. “I think we have always been open to manipulation by others, but certainly, I think robots provide the most effective way of manipulating us,” he says. They are infinitely replicable, can be designed with specific aims in mind, and will soon be in every part of our lives. But Gray also has hope “Certainly you can see advances to fulfill humanity’s deepest desires; immortality, social connection, to have meaning in life. And to be honest, I don’t know what that would be. What are you going to do with your life when AI takes your job? I bet robots could fill that void, but I have no idea how.”