�鶹ӰԺ

Connor Brooks, a graduate student in computer science, demonstrates a robotic system that responds to spoken commands. (Credit: Glenn Asakawa/CU �鶹ӰԺ)

“Robot, point to the screwdriver next to the clamp.”

Daniel Pendergast, a graduate student in CU �鶹ӰԺ’s ATLAS Institute, issues the command, and a few feet away a four-foot-tall robot obeys. The machine whirs to life, bending and twisting its one arm to hover over a table crowded with assorted tools—where it points its claw at a screwdriver right next to a clamp.

Daniel Szafir

The action might seem simple—something that people do every day—but in the field of robotics, Pendergast’s pointing system is a big step forward. That’s because it’s not easy for robots to understand the messy and often vague nature of human language, said Daniel Szafir, Pendergast’s advisor and an assistant professor at ATLAS. 

What, for example, does a person mean when they say “next to”?

In trying to answer those questions, Szafir and his colleagues belong to a rapidly-growing area of study called human-robot interaction. The field addresses the huge gulf that seems to exist between people and their robot helpers: Robots don’t always understand people, and people often don’t want to be around moving, learning machines.

There’s a lot to be gained from helping the two get along, Szafir said. In the case of the screwdriver-locating robot, which the team , Szafir’s goal is to design automated machines that could help people take on a range of tasks—from caring for elderly relatives to assembling toy castles for their kids on Christmas morning. 

“There was always something that fascinated me about this idea of automated assistants,” said Szafir, also in the Department of Computer Science. “It seems like such a powerful way to improve the quality of life for people at all stages. It can help out in healthcare and rehabilitation. It can help us around the house and free us up for pursuits that we’d really like to be doing.”

[video: https://www.youtube.com/watch?v=4eYdX7Ozbhk]

Flying eyes

If the idea of a world filled with robotic assistants wigs you out, Szafir acknowledged that you’re not alone. Many people feel uncomfortable around robots, in part because humans are used to working with beings with expressive eyes and complex body language.

“The robot in our lab only has one arm,” he said. “You can do certain kinds of gestures with that, but people have two arms.”

Szafir, who was named to the Forbes 30 Under 30 list in 2017, is trying to cross that valley. He has experimented, for example, with using augmented reality headsets to help people understand what robots are going to do next. In one case, he made it easier for humans to anticipate the movements of flying robots by . 

He imagines that similar technologies could help disaster responders fight wildfires—using augmented reality displays to track and manage fleets of drones flying around blazes. Szafir and his colleagues recently landed a $1.1 million grant from the U.S. National Science Foundation to experiment with how workers in dangerous fields could use those sorts of tools.

But he also focuses on designing robots that can better interpret human gestures and language. As Szafir put it, in the field of human-robotic interaction, “the human is just as important as the robot.”

That’s not easy. Take the task of building a toy castle on Christmas morning. If you’re working with a human assistant, you can signal that you want a screwdriver in many different ways: you might say “hand me that,” grunt and point or just direct your gaze. 

“People are so good at interpreting highly-ambiguous statements and gestures,” Szafir said. “So while I can tell a person, ‘can you pass me that thing,’ for a robot, it would be really hard to know what that meant.”

[video: https://vimeo.com/280123074]

Helping hands

To get to that point, Szafir and his colleagues took an unusual approach: they asked people to teach their robotic system for them. 

They solicited human volunteers to describe the locations of objects in a series of illustrations of messy workbenches, similar to the one in Szafir’s lab. The team then fed those sentences into a computer algorithm that analyzed and learned the speech patterns that people use when they want something but can’t reach it. 

The claw isn’t perfect. So far, it points to the right objects about 70 percent of the time. And it can’t understand certain types of descriptions, such as those involving negatives: “Hand me the screwdriver that isn’t next to the clamp.” But, Szafir said, it’s a leap above existing systems of this kind.

The researchers at the in �鶹ӰԺ.

And the team hasn’t stopped at spoken words. In related research, Szafir and his colleagues are working to develop robots that can understand the language of human shrugs, head scratching and pointing. 

They have designed a system that scans people as they complete a basic assembly task—say, building a tower out of wooden blocks and screws. Based on how the builders move and where their eyes are pointing, the robot tries to guess at the tools those people might need next. 

“It would recognize when they wanted to fasten things together and it would hand them a screwdriver,” Szafir said. He presented the results of that research recently at the in Madrid. 

There’s a lot of work to be done, but Szafir hopes that automated assistants will be coming to work places and homes near you in the decades ahead. Such feats of engineering may seem mundane in a world where drones can fly over the surface of Mars and run on treadmills. 

But, Szafir said, the pursuit of everyday robot coworkers is about conserving something that all humans cherish: “The one limited resource that we all have is our time.”

[video: https://youtu.be/cKktizb-4ac]