People of ACM - Thomas Williams

May 19, 2020

How did you initially become interested in human-robot interaction?

My introduction to the field of human-robot interaction came about through two formative experiences as an undergraduate. First, like many others in my particular subfield, I read Hofstadter’s Gödel, Escher, Bach, which catapulted my existing interests in computer science across the disciplinary fence into the realm of cognitive science, prompting exciting new questions. I still enjoy grappling with questions about consciousness and cognition for both humans and machines. The second was the opportunity to participate in undergraduate research on human-computer interaction (specifically, neurophysiological estimation of human trust and workload) under the supervision of a wonderful mentor, Leanne Hirshfield, who had recently graduated from Tufts University, which happens to have a phenomenal cognitive science program.

At Leanne’s suggestion, I looked at Tufts, and found a faculty member (my future advisor, Matthias Scheutz) working in human-robot interaction, which I’d never heard of but which perfectly combined my newfound interests in developing psychologically-inspired cognitive architectures and studying effective interaction. Today my research has come full circle; Leanne recently moved to Colorado, and we are now collaborating again, on a project that combines my expertise in robot communication with her expertise in neurophysiological workload estimation. It’s a small world.

You’ve recently done work on enabling robots to understand indirect speech acts. What are indirect speech acts and how will this work improve human-robot interaction?

Indirect speech acts are one of the main politeness strategies employed by humans. A classic example is, “Do you know what time it is?” Literally, this is a yes-or-no question, but by cultural convention we understand that when someone says this, they are really trying to request this piece of information by calling attention to the fact (which we both know or suspect) that their listener likely does in fact know that information. Our research presented at HRI 2018 (which was nominated for a Best Paper award) shows that humans nearly exclusively rely on this type of politeness strategy in social contexts, and will go to great (and sometimes comedic) lengths to avoid phrasing requests directly.

Our computational work toward automatic and generating language that complies with sociocultural linguistic norms such as indirect speech acts will thus not only help robots truly understand human language in social contexts, but will also allow them to generate language that complies with those norms as well. Moreover, our work on moral language generation has shown that humans are very sensitive to the phrasing robots use when rejecting commands: robots must carefully tailor the strength of their responses to the magnitude of violations in order to be effectively persuasive and avoid being perceived as overly harsh—or not harsh enough. Politeness strategies like indirect speech acts—which can actually vary quite widely with respect to perceived politeness—could serve as a critical tool in finding this balance.

You have recently co-authored papers on robot ethics and robot ethics education. What are examples of ethical concerns that might arise in the human-robot interaction field? What is an example of an ethical principle that should guide those working in this field?

Interactive robots, especially natural language capable robots, present a really interesting set of challenges from a robot ethics perspective. There is something of a crisis in natural language generation right now, where many of the popular techniques for text generation, especially those grounded in language modeling, optimize criteria such as predictive accuracy and fluency rather than the criteria that real-world industries care much more about, such as factual accuracy and moral sensitivity. That is, an algorithm that generates very human-like text is of little use to a company if it communicates information that is inaccurate or that is going to result in a public relations nightmare.

The need to focus on these (admittedly harder to assess) factors is especially important for robots. As my student Ryan Blake Jackson writes, “Robots, especially language-capable robots, occupy a unique sociotechnical niche that grants them high perceived social agency and moral agency, leading to uniquely high persuasive power.” As experiments in our lab have shown, through relatively simple dialogue patterns like clarification requests, robots are able to exert measurable influence on teammates’ systems of moral norms (at least within their narrow experimental context). As they say, with great power comes great responsibility.

Accordingly, we are seeking new ways to enable moral competence in autonomous robots. In our current work, we are focusing on Confucian role ethics, and the different ways it might be used by robot designers. As we argue in our recent work, there are at least three ways robot designers can employ principles of Confucian role ethics: by designing knowledge representations and algorithms inspired by principles of Confucian role ethics (e.g., explicit representation and consideration of roles and relationships rather than norms in isolation); by explicitly enabling robots to communicate Confucian principles (e.g., rejecting an inappropriate command by appealing to roles—“a good friend wouldn’t do that”—rather than simply to notions of right and wrong); and by assessing whether the robot’s design encourages adherence to Confucian principles (e.g., does the robot encourage teammates to cultivate their moral selves?).

What is another exciting challenge that you and your colleagues at the MIRROR Lab are working on now?

Our lab is also studying a variety of ways that robots can communicate in a way that is sensitive to human factors such as trust and workload—the topics that got me into HCI-style research in the first place. For example, in the project I mentioned earlier, in collaboration with Leanne Hirshfield, we are exploring how robots can adaptively communicate in mixed-reality environments. There are many robots (like drones, wheelchairs, or NASA’s Astrobee) which may lack arms yet still need to use some form of “gesture” to communicate about objects in their environments. One solution, for robots operating in contexts in which their teammates are already wearing augmented reality head-mounted displays (like the Microsoft Hololens), is to leverage that display as a canvas for displaying gestural cues, such as circles, arrows, or even gestures made by simulated arms.

In our current research, we are hoping to explore how robots can adaptively balance between primarily AR-based and primarily language-based communication, on the basis of the type of mental workload their teammate is under (e.g., working memory load, visual load, or auditory load). We are also exploring how different types of augmented reality gestures may trade off between increased task effectiveness and increased social connection. Similarly, in my upcoming NASA Early Career Researcher project, we’ll be looking at how robots may need to trade off between workload reduction and trust-building in their communication, and what speech act theory has to say in informing these tradeoffs.

Thomas Williams is an Assistant Professor at the Colorado School of Mines, where he directs the Mines Interactive Robotics Research (MIRROR) Lab. His research broadly focuses on artificial intelligence for human-robot interaction, especially problems involving natural language understanding and generation in uncertain and unknown environments. His research is strongly informed by work in cognitive science, especially theories from linguistics, philosophy, and cognitive psychology.

Williams’s honors include receiving a New and Future AI Educator Award in 2017 from the Association for the Advancement of Artificial Intelligence (AAAI), and Early Career Awards from NASA and AFOSR.