Christoffer Heckman News

Failure is Not an Option: Techniques for Autonomous Robots at the DARPA Subterranean Challenge

Anonymous — Fri, 08 Dec 2023 22:23:53 +0000

Failure is Not an Option: Techniques for Autonomous Robots at the DARPA Subterranean Challenge Anonymous (not verified) Fri, 12/08/2023 - 15:23

When engineers in the robotics research community think of what we'd like autonomous agents to tackle in the future, we often target "dull, dirty, and dangerous" tasks. However, despite a sustained boom in robotics research over the last decade, the number of places we've seen robotics in use for these tasks has been uninspiring.

In this webinar, hear from Associate Professor Chris Heckman as he explores his team's recent investigation into where the limits of robotic autonomy are for the highly sought-after application to subterranean emergency response operations.

This project was motivated by the DARPA Subterranean Challenge, which just last year concluded with the CU �鶹ӰԺ team "MARBLE" taking third place and winning a $500,000 prize.

Professor Heckman will give an overview into the genesis of his team's solution over three years of effort, especially with respect to mobility, autonomy, perception, and communications. He'll also discuss the implications for present-day robotic autonomy and where we go from here, especially emphasizing our recent work in large language models for these problems.

Off

Traditional

White

Is the World Ready for Self-Driving Cars?

Anonymous — Mon, 27 Nov 2023 17:41:51 +0000

Is the World Ready for Self-Driving Cars? Anonymous (not verified) Mon, 11/27/2023 - 10:41

In August, the California Public Utilities Commission made history when it voted to allow two self-driving car companies, Waymo and Cruise, to commercially operate their “robotaxis” around the clock in San Francisco.

Within hours, Cruise reported at least 10 incidents where vehicles stopped short of their destination, blocking city streets. The commission demanded they recall 50% of their fleet.

Despite these challenges, other cities — including Las Vegas, Miami, Austin and Phoenix — are allowing autonomous vehicle startups to conduct tests on public roads.

"Self-driving car proponents see the jump from laboratories to real-world testing as a necessary step that has been a long time coming."

Self-driving car proponents see the jump from laboratories to real-world testing as a necessary step that has been a long time coming. The first autonomous vehicle was tested on the Autobahn in Germany in 1986, but the advances stalled in the 1990s due to technology limitations.

After a 2007 Defense Department’s Advanced Research Projects Agency (DARPA) competition featuring autonomous driving capabilities, it seemed like the era of driverless cars had finally arrived. The competition kickstarted a Silicon Valley race to develop the first commercial driverless car. Optimism abounded, with engineers, investors and automakers predicting there would be as many as 10 million self-driving cars on the road by 2020.

“The question for the last 30 years is — how long is this going to take?” said Javier von Stecher (PhDPhys’08), senior software engineer at Nvidia who has worked on self-driving car technology at companies including Uber and Mercedes-Benz. “I think a lot of people were oversold on the idea that we could get this working fast. The biggest shift I’ve seen over the past decade is people realizing how hard this problem really is.”

The stakes may be high, but that’s not deterring CU �鶹ӰԺ researchers. From creating systems and models to studying human-machine interactions, university teams are working to advance the field safely and responsibly as self-driving cars become a fixture in our society.

Their next big question: Can we learn to trust these vehicles?

Cruise Control

The idea behind autonomous vehicles is simple. An artificial intelligence system pulls in data from an array of sensors including radar, high-resolution cameras and GPS, and uses this data to navigate from point A to point B while avoiding obstacles and obeying traffic laws. Sounds simple? It’s not.

When a self-driving car encounters an unexpected obstacle, it makes split-second judgment calls — should it brake or swerve around it? — that develop naturally in humans but are still beyond even the most sophisticated AI systems.

Moreover, there will always be an edge case that the AI-powered car hasn’t seen before, which means the key to safe autonomous vehicles is building systems that can correctly favor safe choices in unfamiliar situations.

Majid Zamani, associate professor and director of CU �鶹ӰԺ’s Hybrid Systems Control Lab, studies how to create software for autonomous systems such as cars, drones and airplanes. In autonomous vehicles’ AI systems, data flows into the AI and helps it make decisions. But how the AI creates those decisions is a mystery. This, said Zamani, makes it difficult to trust the AI system — and yet trust is critically important in high-stakes applications like autonomous driving.

“These are what we call safety critical applications because system failure can cause loss of life or damage to property, so it’s really important that the way those systems are making decisions is provably correct,” Zamani said.

In contrast to AI systems that use data to create models that are not intelligible to humans, Zamani advocates for a bottom -up approach where the AI’s models are derived from fundamental physical laws, such as acceleration or friction, which are well-understood and unchanging.

“If you derive a model using data, you have to be able to ensure that you can quantify how much error is in that model and the actual system that uses it,” Zamani said.

Mathematically demonstrating the safety of the models used by autonomous vehicles is important for engineers and policymakers who need to guarantee safety before they’re deployed in the real world. But this raises some thorny questions: How safe is “safe enough,” and how can autonomous vehicles communicate these risks to drivers?

Computer, Take the Wheel

Each year, more than 40,000 Americans die in car accidents, and according to the National Highway Traffic Safety Administration (NHTSA), about 90% of U.S. auto deaths and serious crashes are attributable to driver error. The great promise of autonomous vehicles is to make auto deaths a relic of history by eliminating human errors with computers that never get tired or distracted.

The NHTSA designates six levels of “autonomy” for self-driving cars, which range from Level 0 (full driver control) to Level 5 (fully autonomous). For most of us, Level 5 is what we think of when we think of self-driving cars: a vehicle so autonomous that it might not even have a steering wheel and driver’s seat because the computer handles everything. For now, this remains a distant dream, with many automakers pursuing Level 3 or 4 autonomy as stepping stones.

“Most modern cars are Level 2, with partial autonomous driving,” said Chris Heckman, associate professor and director of the Autonomous Robotics and Perception Group in CU �鶹ӰԺ’s computer science department. “Usually that means there’s a human at the wheel, but they can relegate some functions to the car’s software such as automatic braking or adaptive cruise control.”

While these hybrid AI-human systems can improve safety by assisting a driver with braking, acceleration and collision avoidance, limitations remain. Several fatal accidents, for example, have resulted from drivers’ overreliance on autopilot, which stems from issues of human psychology and AI understanding.

Fostering Trust

This problem is deeply familiar to Leanne Hirschfield, associate research professor at the Institute of Cognitive Science and the director of the System-Human Interaction with NIRS and EEG (SHINE) Lab at CU �鶹ӰԺ. Hirschfield’s research focuses on using brain measurements to study the ways humans interact with autonomous systems, like self-driving cars and AI systems deployed in elementary school classrooms.

"When an autonomous vehicle can show the driver information about how it’s making decisions or its level of confidence in its decisions, the driver is better equipped to determine when they need to grab the wheel."

Trust, Hirschfield said, is defined as a willingness to be vulnerable and take on risks, and for decades the dominant engineering paradigm for autonomous systems has been focused on ways to foster total trust in autonomous systems.

“We’re realizing that’s not always the best approach,” Hirschfield said. “Now, we’re looking at trust calibration, where users often trust the system but also have enough information to know when they shouldn’t rely on it.”

The key to trust calibration, she said, is transparency. When an autonomous vehicle can show the driver information about how it’s making decisions or its level of confidence in its decisions, the driver is better equipped to determine when they need to grab the wheel.

Studying user responses is challenging in a laboratory setting, where it’s difficult to expose drivers to real risks. So Hirschfield and researchers at the U.S. Air Force Academy have been using a Tesla modified with a variety of internal sensors to study user trust in autonomous vehicles.

“Part of what we’re trying to do is measure someone’s level of trust, their workload and emotional states while they’re driving,” Hirschfield said. “They’ll have the car whipping around hills, which is how you need to study trust because it involves a sense of true risk compared to a study in a lab setting.”

Although Hirschfield said that researchers have made a lot of progress in understanding how to design autonomous vehicles to foster driver trust, there is still a lot of work to be done.

Human-Centered Design

Sidney D’Mello, a professor at the Institute of Cognitive Science, studies how human-computer interactions shift the way we think and feel. For D’Mello, it’s unclear whether the current crop of self-driving cars can shift to a new driver-focused paradigm from the current perfected engineering-forward approach.

“I think we need an entirely new methodology for the self-driving car context,” D’Mello said. “If you really want something you can trust, then you need to design these systems with users starting from day one. But every single car company is kind of stuck in this engineering mindset from 50 years ago where they build the tech and then they present it to the user.”

The good news, D’Mello said, is that automakers are starting to take this challenge seriously. A collaboration between Toyota and the Institute of Cognitive Science focused on designing autonomous vehicles that foster trust in the user.

“The autonomous model typically implies the AI is in the center with the human hovering around it,” said D’Mello. “But this needs to be a model with the human in the center.”

Even when users learn to trust autonomous vehicles, living with driverless cars and reconceptualizing how they relate to them is complex. But there’s a lot we can apply from research on prosthetics, said Cara Welker, assistant professor in biomechanics, robotics and systems design.

Much like autonomous vehicles analyze surroundings to make navigation and control decisions, robotic prostheses monitor a wearer’s movements to understand appropriate behavior. And just as teaching users to trust prosthetics requires strong feedback loops and predictable prosthetic behavior, teaching drivers to trust autonomous vehicles means providing drivers with information about what the AI is doing — and it requires drivers to reconceptualize vehicles as extensions of themselves.

“There’s a difference between users being able to predict the behavior of an assistive device versus having some kind of sensory feedback,” Welker said. “And this difference has been shown to affect whether the people think of it as ‘me and my prosthesis’ instead of just ‘me, which includes my prosthesis.’ And that’s incredibly important in terms of how users will trust that device.”

How, then, will drivers evolve to experience cars as extensions of themselves?

Next Exit

In 2018, a pedestrian was killed by a self-driving Uber in Arizona, which marked the first fatality attributed to an autonomous vehicle. Although the driver pleaded guilty in the case, the question of who is responsible when autonomous vehicles kill is far from settled.

Today, there is limited regulation dictating autonomous vehicle safety and liability. One problem is that vehicles are regulated at the federal level while drivers are regulated at the state level — a division of responsibility that doesn’t account for a future where the driver and vehicle are more closely aligned.

Researchers and automakers have voiced frustration with existing autonomous driving regulations, agreeing that updated regulations are necessary. Ideally, regulations would ensure driver, passenger and pedestrian safety without quashing innovation. But what these policies might look like is still unclear.

The challenge, said Heckman, is that the engineers don’t have complete control over how autonomous systems behave in every circumstance. He believes it’s critical for regulations to account for this without insisting on impossibly high safety standards.

“Many of us work in this field because automotive deaths seem avoidable and we want to build technologies that solve that problem,” Heckman said. “But I think we hold these systems [to] too high of a standard — because yes, we want to have safe systems, but right now we have no safety frameworks, and automakers aren’t comfortable building these systems because they may be held to an extremely high liability.”

Other industries may offer a vision for how to regulate the autonomous driving industry while providing acceptable safety standards and enabling technological development, Heckman said. The aviation industry, for example, adopted rigorous engineering standards and fostered trust in engineers, pilots, passengers and policymakers.

“There’s an engineering principle that trust is a perception of humans,” Heckman said. “Trust is usually built through experience with a system, and that experience confers trust on the engineering paradigms that build safe systems.

“With airplanes, it took decades for us to come up with designs and engineering paradigms that we feel comfortable with. I think we’ll see the same in autonomous vehicles, and regulation will follow once we’ve really defined what it means for them to be trustworthy.”

Off

Traditional

White

Building next generation autonomous robots to serve humanity

Anonymous — Fri, 17 Nov 2023 23:19:59 +0000

Building next generation autonomous robots to serve humanity Anonymous (not verified) Fri, 11/17/2023 - 16:19

Jeff Zehnder

One thousand feet underground, a four-legged creature scavenges through tunnels in pitch darkness. With vision that cuts through the blackness, it explores a spider web of paths, remembering its every step and navigating with precision. The sound of its movements echo eerily off the walls, but it is not to be feared – this is no wild animal; it is an autonomous rescue robot.

Initially designed to find survivors in collapsed mines, caves, and damaged buildings, that is only part of what it can do.

Created by a team of �鶹ӰԺ researchers and students, the robots placed third as the top US entry and earned $500,000 in prize money at a Defense Advanced Projects Research Agency Subterranean Challenge competition in 2021.

Going Futher

Two years later, they are pushing the technology even further, earning new research grants to expand the technology and create new applications in the rapidly growing world of autonomous systems.

“Ideally you don’t want to put humans in harm’s way in disaster situations like mines or buildings after earthquakes; the walls or ceilings could collapse and maybe some already have,” said Sean Humbert, a professor of mechanical engineering and director of the Robotics Program at CU �鶹ӰԺ. “These robots can be disposable while still providing situational awareness.”

The team developed an advanced system of sensors and algorithms to allow the robots to function on their own – once given an assignment, they make decisions autonomously on how to best complete it.

Advanced Communication

A major goal is to get them from engineers directly into the hands of first responders. Success requires simplifying the way the robots transmit data into something approximating plain English, according to Kyle Harlow, a computer science PhD student.

“The robots communicate in pure math. We do a lot of work on top of that to interpret the data right now, but a firefighter doesn’t have that kind of time,” Harlow said.

To make that happen Humbert is collaborating with Chris Heckman, an associate professor of computer science, to change both how the robots communicate and how they represent the world. The robots’ eyes – a LiDAR sensor – creates highly detailed 3D maps of an environment, 15 cm at a time. That’s a problem when they try to relay information – the sheer amount of data clogs up the network.

“Humans don’t interpret the environment in 15 cm blocks,” Humbert said. “We’re now working on what’s called semantic mapping, which is a way to combine contextual and spatial information. This is closer to how the human brain represents the world and is much less memory intensive.”

High Tech Mapping

The team is also integrating new sensors to make the robots more effective in challenging environments. The robots excel in clear conditions but struggle with visual obstacles like dust, fog, and snow. Harlow is leading an effort to incorporate millimeter wave radar to change that.

“We have all these sensors that work well in the lab and in clean environments, but we need to be able to go out in places such as Colorado where it snows sometimes,” Harlow said.

Where some researchers are forced to suspend work when a grant ends, members of the subterranean robotics team keep finding new partners to push the technology further.

Autonomous Flight

Eric Frew, a professor of aerospace at CU �鶹ӰԺ, is using the technology for a new National Institute of Standards and Technology competition to develop aerial robots – drones – instead of ground robots, to autonomously map disaster areas indoors and outside.

“Our entry is based directly on the Subterranean Challenge experience and the systems developed there,” Frew said.

Some teams in the competition will be relying on drones navigated by human operators, but Frew said CU �鶹ӰԺ’s project is aiming for an autonomous solution that allows humans to focus on more critical tasks.

Although numerous universities and private businesses are advancing autonomous robotic systems, Humbert said other organizations often focus on individual aspects of the technology. The students and faculty at CU �鶹ӰԺ are working on all avenues of the systems and for uses in environments that present extreme challenges.

“We’ve built world-class platforms that incorporate mapping, localization, planning, coordination – all the high level stuff, the autonomy, that’s all us,” Humbert said. “There are only a handful of teams across the world that can do that. It’s a huge advantage that CU �鶹ӰԺ has.”

Off

Traditional

White

CU �鶹ӰԺ offers new graduate program in robotics

Anonymous — Wed, 20 Sep 2023 16:28:02 +0000

CU �鶹ӰԺ offers new graduate program in robotics Anonymous (not verified) Wed, 09/20/2023 - 10:28

Jeff Zehnder

Robotics Degree Programs

Program Requirements

30 credit hours
1 required course - Introduction to Robotics
43 course options
30 dissertation hours (PhD)
4-6 dissertation hours (Thesis Master’s)

The �鶹ӰԺ has started a graduate engineering program in robotics to fill a growing need in an in-demand field.

The CU Regents have approved new Master of Science and PhD degree options in robotics that will provide students a flexible education that merges hardware and software engineering, mathematics and artificial intelligence into a single program.

“Demand is so high for degrees like this across the country; it’s something students and employers really want,” said Sean Humbert, director of the Robotics Program and a professor in the Paul M. Rady Department of Mechanical Engineering.

The program brings together a wide array of faculty, research and class options from the College of Engineering and Applied Science, according to Chris Heckman, associate professor of computer science and the robotics program.

“The workforce in robotics is often siloed, with people only being specialists in certain elements. We want students to be able to work across the field in computer science, mechanical, electrical, aerospace, wherever they need to be,” Heckman said.

Students enrolled in the program can choose from 40+ different courses taught by leading researchers with strong expertise in key areas, including field robotics, reasoning and assurance, smart materials, human-centered robotics and biomedical robotics.

“CU �鶹ӰԺ is really strong in robotics, and now we’re bringing together all that expertise,” Humbert said. “This field is so interdisciplinary, and we have strong connections and teams both within the university and in industry and the public sector.”

�鶹ӰԺ and Colorado’s Front Range is home to many businesses active in robotics, providing educational partnership and career options for students and graduates, according to Alessandro Roncone, associate director of the Robotics Program and an assistant professor of computer science.

“This program positions students at the nexus of innovative research and real-world application. Not only will they be taught by leading experts in the field, but they'll also have the opportunity to become leaders in robotics and AI. We are committed to fostering creativity and innovation, and our strong tech ecosystem locally provides an unparalleled environment for growth and discovery,” Roncone said.

In addition to a research-focused PhD, students enrolled in the master’s program can choose from thesis and non-thesis options, providing graduates with opportunities in academia and technical leadership positions in large industry, startups, emergency services and government.

The program officially launched for the fall 2023 semester, with students transferring into the program from other CU Engineering graduate programs. Prospective students from outside the university will be welcomed starting in fall 2024. That application window is now open.

The �鶹ӰԺ has started a graduate engineering program in robotics to fill a growing need in an in-demand field. The CU Regents have approved new Master of Science and PhD degree options in robotics that will...

Off

Traditional

White

Robotics professors win award for modern textbook

Anonymous — Wed, 12 Jul 2023 18:30:27 +0000

Robotics professors win award for modern textbook Anonymous (not verified) Wed, 07/12/2023 - 12:30

Professors Nikolaus Correll, Bradley Hayes, Christoffer Heckman and Alessandro Roncone have received a recognition award from the College of Engineering and Applied Science for their work, Introduction to Autonomous Robots: Mechanisms, Sensors, Actuators, and Algorithms, an open textbook focusing on computational principles of autonomous robots.

The award was created to recognize the achievements of CEAS faculty who author or co-author a significant educational textbook.

The textbook is being published this month by MIT Press and it is also available on GitHub via a Creative Commons 4.0 (CC-BY-NC-ND) license.

Under that license type, readers can use images and content from the book for non-commercial purposes with proper attribution, but cannot post compiled versions of the book online. This includes labs and slides to help instructors create their own introductory courses, said Roncone.

"As many students rely on YouTube videos and their phones as primary reading devices, the classical textbook becomes more and more obsolete. Freely available illustrations are critical for actively transitioning into new formats, carrying CU and the MIT Press forward," Correll said.

Correll said their textbook model is forward-thinking, providing the best trade-off between a freely available resource that folks can contribute to while also providing a consistent curriculum that others can rely upon.

Roncone said he agreed.

"Robotics is a fast-moving field, and the future of robotics education will strongly benefit from a model that combines a traditional editorial process with open-source community engagement that magnifies our impact," he said.

The curriculum the textbook lays out is robot-agnostic, focusing not on the particular specifications of one robot's architecture, but on the underlying mathematics and logical decisions that govern autonomous robotic decisions.

The book is also aimed at undergraduate students, which is notable as most robotics books leave algorithm design to a graduate student audience. This resource has already been used by undergraduate students over the past several years through classes taught by the co-authoring professors.

Off

Traditional

White