Nikolaus Correll

A delicate touch: teaching robots to handle the unknown

Anonymous — Tue, 02 Apr 2024 19:39:31 +0000

A delicate touch: teaching robots to handle the unknown Anonymous (not verified) Tue, 04/02/2024 - 13:39

Grace Wilson

William Xie, a first-year PhD student in computer science, is teaching a robot to reason how gently it should grasp previously unknown objects by using large language models (LLMs).

DeliGrasp, Xie's project, is an intriguing step beyond the custom, piecemeal solutions currently used to avoid pinching or crushing novel objects.

In addition, Deligrasp helps the robot translate what it can 'touch' into meaningful information for people.

"William has gotten some neat results by leveraging common sense information from large language models. For example, the robot can estimate and explain the ripeness of various fruits after touching them." Said his advisor, Professor Nikolaus Correll.

Let's learn more about DeliGrasp, Xie's journey to robotics, and his plans for the conference Japan and beyond.

[video:https://www.youtube.com/watch?v=OMzTgY1gxLw]

How would you describe this research?

As humans, we’re able to quickly intuit how exactly we need to pick up a variety of objects, including delicate produce or unwieldy, heavy objects. We’re informed by the visual appearance of an object, what prior knowledge we may have about it, and most importantly, how it feels to the touch when we initially grasp it.

Robots don’t have this all-encompassing intuition though, and they don’t have end-effectors (grippers/hands) as effective as human hands. So solutions are piecemeal: the community has researched “hands” across the spectrum of mechanical construction, sensing capabilities (tactile, force, vibration, velocity), material (soft, rigid, hybrid, woven, etc…). And then the corresponding machine learning models and/or control methods to enable “appropriately forceful” gripping are bespoke for each of these architectures.

Embedded in LLMs, which are trained on an internet’s worth of data, is common sense physical-reasoning that crudely approximates a human’s (as the saying goes: “all models are wrong, some are useful”). We use the LLM-estimated mass and friction to simplify the grasp controller and deploy it on a two-finger gripper, a prevalent and relatively simple architecture. Key to the controller working is the force feedback sensed by the gripper as it grasps an object, and knowing at what force threshold to stop—the LLM-estimated values directly determine this threshold for any arbitrary object, and our initial results are quite promising.

How did you get inspired to pursue this research?

I wouldn’t say that I was inspired to pursue this specific project. I think, like a lot of robotics research, I had been working away at a big problem for a while, and stumbled into a solution for a much smaller problem. My goal since I arrived here has been to research techniques for assistive robots and devices that restore agency for the elderly and/or mobility-impaired in their everyday lives. I’m particularly interested in shopping (but eventually generalist) robots—one problem we found is that it is really hard to determine, let alone pick ripe fruits and produce with a typical robot gripper and just a camera. In early February, I took a day to try out picking up variably sized objects via hand-tuning our MAGPIE gripper’s force sensing (an affordable, open-source gripper developed by the Correll Lab). It worked well; I let ChatGPT calibrate the gripper which worked even better, and it evolved very quickly into DeliGrasp.

What would you say is one of your most interesting findings so far?

LLMs do a reasonable job of estimating an arbitrary object’s mass (friction, not as well) from just a text description. This isn’t in the paper, but when paired with a picture, they can extend this reasoning for oddballs—gigantic paper airplanes, or miniature (plastic) fruits and vegetables.

With our grasping method, we can sense the contact forces on the gripper as it closes around an object—this is a really good measure of ripeness, it turns out. We can then further employ LLMs to reason about these contact forces to pick out ripe fruit and vegetables!

What does the day-to-day of this research look like?

Leading up to submission, I was running experiments on the robot and picking up different objects with different strategies pretty much every day. A little repetitive, but also exciting. Prior to that, and now that I’m trying to improve the project for the next conference, I spend most of my time reading papers, thinking/coming up with ideas, and setting up small, one-off experiments to try out those ideas.

How did you come to study at CU �鶹ӰԺ?

For a few years, I’ve known that I really wanted to build robots that could directly, immediately help my loved ones and community. I had a very positive first research experience in my last year of undergrad and learned what it felt like to have true personal agency in pursuing work that I cared about. At the same time I knew I’d be relocating to �鶹ӰԺ after graduation. I was very fortunate that Nikolaus accepted me and let me keep pursuing this goal of mine.

It’d be unfathomable if I could keep doing this research in academia or industry, though of course that would be ideal. But I’m biased toward academia, particularly teaching. I’ve been teaching high school robotics for 5 years now, and now teaching/mentoring undergrads at CU—each day is as fulfilling as the first. I have great mentors across the robotics faculty and senior PhD students we work in ECES 111, a giant, well-equipped space that 3 robotics labs share, and it’s great for collaboration and brainstorming.

What are your hopes for this international conference (and what conference is it?)

The venue is a workshop at the 2024 International Conference on Robotics and Automation (ICRA 2024), happening in Yokohama, Japan from May 13-17. The name of the workshop is a mouthful: Vision-Language Models for Navigation and Manipulation (VLMNM).

A workshop is detached from the main conference, and kind of is its own little bubble (like a big supermarket—the conference—hosting a pop-up food tasting event—the workshop). I'm really excited to meet other researchers and pick their brains. As a first-year, I’ve spent the past year reading papers from practically everyone on the workshop panel, and from their students. I’ll probably also spend half my time exploring (eating) around the Tokyo area.

William Xie, a first-year PhD student in computer science, is teaching a robot to reason how gently it should grasp previously unknown objects by using large language models (LLMs).

Off

Traditional

White

VIDEO: 3D display could soon bring touch to the digital world

Anonymous — Tue, 01 Aug 2023 16:59:12 +0000

VIDEO: 3D display could soon bring touch to the digital world Anonymous (not verified) Tue, 08/01/2023 - 10:59

Daniel Strain

Nikolaus Correll, associate professor in the Department of Computer Science, helped create the basis for a new rippling 3D display that can bring touch to digital experiences.

Off

Traditional

White

Correll and Mickelson receive fifth Open Educator Award

Anonymous — Mon, 07 Mar 2022 07:00:00 +0000

Correll and Mickelson receive fifth Open Educator Award Anonymous (not verified) Mon, 03/07/2022 - 00:00

The University Libraries and CU �鶹ӰԺ Student Government are pleased to recognize Nikolaus Correll and Alan Mickelson as recipients of the 2022 Open Educator Award at the �鶹ӰԺ. Both associate professors were selected to receive this award for using open educational practices in their teaching.

Off

Traditional

White

Robotics team scores an award in Tokyo

Anonymous — Thu, 25 Oct 2018 20:36:44 +0000

Robotics team scores an award in Tokyo Anonymous (not verified) Thu, 10/25/2018 - 14:36

The CU team was formed around the CU spinoff company Robotics Materials Inc., which was founded by Associate Professor Nikolaus Correll.

Off

Traditional

White

Correll: To really help U.S. workers, invest in robots

Anonymous — Tue, 04 Apr 2017 15:13:40 +0000

Correll: To really help U.S. workers, invest in robots Anonymous (not verified) Tue, 04/04/2017 - 09:13

Robotics researcher advocates for job growth by "building on our existing strengths, remaining a leader in manufacturing efficiency and doing the hard work to further improve our educational and social systems to cope with a changing workforce."

Off

Traditional

White

Gaugewear Inc. to commercialize wearable technology prototype from Correll Lab

Anonymous — Tue, 28 Jul 2015 21:13:56 +0000

Gaugewear Inc. to commercialize wearable technology prototype from Correll Lab Anonymous (not verified) Tue, 07/28/2015 - 15:13

The new technology can be worn as an adjustable sleeve on the user’s arm or leg or sewn into everything from clothing and backpack straps to dog leashes, allowing “eyes-free” control of a tablet, phone or media player that doesn’t require any precise button touching. The new prototype joins popular wearable high-tech devices ranging from smart watches and fitness bands to headsets and smart gloves.

Off

Traditional

White

'Science' article outlines advances, challenges for robotic materials

Anonymous — Fri, 20 Mar 2015 06:00:00 +0000

'Science' article outlines advances, challenges for robotic materials Anonymous (not verified) Fri, 03/20/2015 - 00:00

Prosthetics with a realistic sense of touch. Bridges that detect and repair their own damage. Vehicles with camouflaging capabilities.

Advances in materials science, distributed algorithms and manufacturing processes are bringing all of these things closer to reality every day, says a review published today in the journal Science by Nikolaus Correll, assistant professor of computer science, and research assistant Michael McEvoy, both of the �鶹ӰԺ.

The "robotic materials" being developed by Correll Lab and others are often inspired by nature, Correll said.

"We looked at organisms like the cuttlefish, which change their appearance depending on their environment, and the banyan tree, which grows above-ground roots to support the increasing weight of the trunk," Correll said. "We asked what it would take to engineer such systems."

Robotic materials require tight integration between sensing, computation and actually changing the materials properties of the underlying material. While materials can already be programmed to change some of their properties in response to specific stimuli, robotic materials can sense stimuli and determine how to respond on their own.

Correll and McEvoy use the example of artificial skin equipped with microphones that would analyze the sounds of a texture rubbing the skin and route information back to the central computer only when important events occurred.

"The human sensory system automatically filters out things like the feeling of clothing rubbing on the skin," Correll said. "An artificial skin with possibly thousands of sensors could do the same thing, and only report to a central 'brain' if it touches something new."

While all of these materials are possible, the authors caution that manufacturing techniques remain a challenge.

"Right now, we're able to make these things in the lab on a much larger scale, but we can't scale them down," Correll said. "The same is true for nano- and microscale manufacturing, which can't be scaled up to things like a building façade."

The field also faces an education gap, the authors say. Developing robotic materials requires interdisciplinary knowledge that currently isn't provided by materials science, computer science or robotics curricula alone.

At CU-�鶹ӰԺ, Correll is addressing that gap with a freshman-level engineering projects class called "Materials That Think."

"We expose engineering students to both materials and computing, no matter what their background is," he said.

In the long run, Correll believes robotic materials will be used in everyday items, like shoe insoles that could sense pressure and adapt their stiffness to adjust to walking or running.

"While we can imagine such a material consisting of little patches that each include an actuator, sensor and little computer, we have a hard time imagining such a complex piece of technology could ever be affordable," he said. "I think the last 10 years of advances in smartphones have demonstrated the opposite."

Off

Traditional

White

Swiss newspaper interviews Nikolaus Correll about swarm robotics

Anonymous — Wed, 03 Jul 2013 06:00:00 +0000

Swiss newspaper interviews Nikolaus Correll about swarm robotics Anonymous (not verified) Wed, 07/03/2013 - 00:00

Off

Traditional

White

CU �鶹ӰԺ team develops swarm of pingpong ball-sized robots

Anonymous — Fri, 14 Dec 2012 07:00:00 +0000

CU �鶹ӰԺ team develops swarm of pingpong ball-sized robots Anonymous (not verified) Fri, 12/14/2012 - 00:00

�鶹ӰԺ Assistant Professor Nikolaus Correll likes to think in multiples. If one robot can accomplish a singular task, think how much more could be accomplished if you had hundreds of them.

Off

Traditional

White