Feed aggregator



The preliminary rounds of the DARPA Subterranean Challenge Finals are kicking off today. It's been a little bit since the last DARPA SubT event—the Urban Circuit squeaked through right before the pandemic hit back in February of 2020, and the in-person Cave Circuit originally scheduled for later that year was canceled.

So if it's been a while since you've thought about SubT, this article will provide a very brief refresher, and we'll also go through different ways in which you can follow along with the action over the course of the week.

The overall idea of the DARPA Subterranean Challenge is to get teams of robots doing useful stuff in challenging underground environments. "Useful stuff" means finding important objects or stranded humans, and "challenging underground environments" includes human-made tunnel systems, the urban underground (basements, subways, etc), as well as natural caves. And "teams of robots" can include robots that drive, crawl, fly, walk, or anything in between.

Over the past few years, teams of virtual and physical robots have competed in separate DARPA-designed courses representing each of those three underground domains. The Tunnel Event took place in an old coal mine, the Urban Event took place in an unfinished nuclear reactor complex, and the Cave Event—well, that got canceled because of COVID, but lots of teams found natural caves to practice in anyway.

So far, we've learned that underground environments are super hard for robots. Communications are a huge problem, and robots have to rely heavily on autonomy and teamwork rather than having humans tell them what to do, although we've also seen all kinds of clever solutions to this problem. Mobility is tough, but legged robots have been surprisingly useful, and despite the exceptionally unfriendly environment, drones are playing a role in the challenge as well. Each team brings a different approach to the Subterranean Challenge, and every point scored represents progress towards robots that can actually be helpful in underground environments when we need them to be.

The final Subterranean Challenge event, happening this week includes both a Virtual Track for teams competing with virtual robots, and a Systems Track for teams competing with physical robots. Let's take a look at how the final competition will work, and then the best ways to watch what's happening.

How It Works

If you've been following along with the previous circuits (Tunnel and Urban), the overall structure of the Final will be somewhat familiar, but there are some important differences to keep in mind. First, rather than being a specific kind of underground environment, the final course will incorporate elements from all three environments as well as some dynamic obstacles that could include things like closing doors or falling rocks. Only DARPA knows what the course looks like, and it will be reconfigured every day.

Each of the Systems Track teams will have one 30-minute run on the course on Tuesday and another on Wednesday. 30 minutes is half the amount of time that teams have had in previous competitions. A Team's preliminary round score will be the sum of the scores of the two runs, but every team will get to compete in the final on Thursday no matter what their score is: the preliminary score only serves to set the team order, with higher scoring teams competing later in the final event.

The final scoring run for all teams happens on Thursday. There will be one single 60 minute run for each team, which is a departure from previous events: if a team's robots misbehave on Thursday, that's just too bad, because there is no second chance. A team's score on the Thursday run is what will decide who wins the Final event; no matter how well a team did in previous events or in the preliminary runs this week, the Thursday run is the only one that counts for the prize money.

Scoring works the same as in previous events. There will be artifacts placed throughout the course, made up of 10 different artifact types, like cell phones and fire extinguishers. Robots must identify the specific artifact type and transmit its location back to the starting area, and if that location is correct within 5 meters, a point is scored. Teams have a limited number of scoring attempts, though: there will be a total of 40 artifacts on the course for the prize round, but only 45 scoring attempts are allowed. And if a robot locates an artifact but doesn't manage to transmit that location back to base, it doesn't get that point.

The winning team is the one with the most artifacts located in the shortest amount of time (time matters only in the event of a tie). The Virtual Track winners will take home $750k, while the top System Track team wins $2 million, with $1 million for second and $500k for third.

If that's not enough of a background video for you, DARPA has helpfully provided this hour long video intro.

How to Watch

Watching the final event is sadly not as easy as it has been for previous events. Rather than publicly live streaming raw video feeds from cameras hidden inside the course, DARPA will instead record everything themselves and then produce edited and commentated video recaps that will post to YouTube the following day. So, Tuesday's preliminary round content will be posted on Wednesday, the Wednesday prelims post Thursday, and the Final event on Thursday will be broadcast on Friday as the teams themselves watch. Here's the schedule:

The SubT Summit on Friday afternoon consists of roundtable discussions from both the Virtual Track teams and System Track teams; those will be from 2:30 to 3:30 and 4:00 to 5:00 respectively, with a half hour break in the middle. All of these streams are pre-scheduled on the DARPA YouTube channel. DARPA will also be posting daily blogs and sharing photos here.

After the Thursday Final, it might be possible for us to figure out a likely winner based on artifact counts. But the idea is that even though the Friday broadcast is one day behind the competition, both we and the teams will be finding out what happened (and who won) at the same time—that's what will happen on the Friday livestream.

Saturday, incidentally, has been set aside for teams to mess around on the course if they want to. This won't be recorded or broadcast at all, but I'll be there for a bit to see what happens.

If you're specifically looking for a way to follow along in real time, I'm sorry to say that there isn't one. There will be real-time course feeds in the press room, but press is not allowed to share any of the things that we see. So if you're looking for details that are as close to live as possible, I'd recommend checking out Twitter, because many teams and team members are live Tweeting comments and pictures and stuff, and the easiest way to find that is by searching for the #SubTChallenge hashtag.

Lastly, if you've got specific things that you'd like to see or questions for DARPA or for any of the teams, ping me on Twitter @BotJunkie and I'll happily see what I can do.



The preliminary rounds of the DARPA Subterranean Challenge Finals are kicking off today. It's been a little bit since the last DARPA SubT event—the Urban Circuit squeaked through right before the pandemic hit back in February of 2020, and the in-person Cave Circuit originally scheduled for later that year was canceled.

So if it's been a while since you've thought about SubT, this article will provide a very brief refresher, and we'll also go through different ways in which you can follow along with the action over the course of the week.

The overall idea of the DARPA Subterranean Challenge is to get teams of robots doing useful stuff in challenging underground environments. "Useful stuff" means finding important objects or stranded humans, and "challenging underground environments" includes human-made tunnel systems, the urban underground (basements, subways, etc), as well as natural caves. And "teams of robots" can include robots that drive, crawl, fly, walk, or anything in between.

Over the past few years, teams of virtual and physical robots have competed in separate DARPA-designed courses representing each of those three underground domains. The Tunnel Event took place in an old coal mine, the Urban Event took place in an unfinished nuclear reactor complex, and the Cave Event—well, that got canceled because of COVID, but lots of teams found natural caves to practice in anyway.

So far, we've learned that underground environments are super hard for robots. Communications are a huge problem, and robots have to rely heavily on autonomy and teamwork rather than having humans tell them what to do, although we've also seen all kinds of clever solutions to this problem. Mobility is tough, but legged robots have been surprisingly useful, and despite the exceptionally unfriendly environment, drones are playing a role in the challenge as well. Each team brings a different approach to the Subterranean Challenge, and every point scored represents progress towards robots that can actually be helpful in underground environments when we need them to be.

The final Subterranean Challenge event, happening this week includes both a Virtual Track for teams competing with virtual robots, and a Systems Track for teams competing with physical robots. Let's take a look at how the final competition will work, and then the best ways to watch what's happening.

How It Works

If you've been following along with the previous circuits (Tunnel and Urban), the overall structure of the Final will be somewhat familiar, but there are some important differences to keep in mind. First, rather than being a specific kind of underground environment, the final course will incorporate elements from all three environments as well as some dynamic obstacles that could include things like closing doors or falling rocks. Only DARPA knows what the course looks like, and it will be reconfigured every day.

Each of the Systems Track teams will have one 30-minute run on the course on Tuesday and another on Wednesday. 30 minutes is half the amount of time that teams have had in previous competitions. A Team's preliminary round score will be the sum of the scores of the two runs, but every team will get to compete in the final on Thursday no matter what their score is: the preliminary score only serves to set the team order, with higher scoring teams competing later in the final event.

The final scoring run for all teams happens on Thursday. There will be one single 60 minute run for each team, which is a departure from previous events: if a team's robots misbehave on Thursday, that's just too bad, because there is no second chance. A team's score on the Thursday run is what will decide who wins the Final event; no matter how well a team did in previous events or in the preliminary runs this week, the Thursday run is the only one that counts for the prize money.

Scoring works the same as in previous events. There will be artifacts placed throughout the course, made up of 10 different artifact types, like cell phones and fire extinguishers. Robots must identify the specific artifact type and transmit its location back to the starting area, and if that location is correct within 5 meters, a point is scored. Teams have a limited number of scoring attempts, though: there will be a total of 40 artifacts on the course for the prize round, but only 45 scoring attempts are allowed. And if a robot locates an artifact but doesn't manage to transmit that location back to base, it doesn't get that point.

The winning team is the one with the most artifacts located in the shortest amount of time (time matters only in the event of a tie). The Virtual Track winners will take home $750k, while the top System Track team wins $2 million, with $1 million for second and $500k for third.

If that's not enough of a background video for you, DARPA has helpfully provided this hour long video intro.

How to Watch

Watching the final event is sadly not as easy as it has been for previous events. Rather than publicly live streaming raw video feeds from cameras hidden inside the course, DARPA will instead record everything themselves and then produce edited and commentated video recaps that will post to YouTube the following day. So, Tuesday's preliminary round content will be posted on Wednesday, the Wednesday prelims post Thursday, and the Final event on Thursday will be broadcast on Friday as the teams themselves watch. Here's the schedule:

The SubT Summit on Friday afternoon consists of roundtable discussions from both the Virtual Track teams and System Track teams; those will be from 2:30 to 3:30 and 4:00 to 5:00 respectively, with a half hour break in the middle. All of these streams are pre-scheduled on the DARPA YouTube channel. DARPA will also be posting daily blogs and sharing photos here.

After the Thursday Final, it might be possible for us to figure out a likely winner based on artifact counts. But the idea is that even though the Friday broadcast is one day behind the competition, both we and the teams will be finding out what happened (and who won) at the same time—that's what will happen on the Friday livestream.

Saturday, incidentally, has been set aside for teams to mess around on the course if they want to. This won't be recorded or broadcast at all, but I'll be there for a bit to see what happens.

If you're specifically looking for a way to follow along in real time, I'm sorry to say that there isn't one. There will be real-time course feeds in the press room, but press is not allowed to share any of the things that we see. So if you're looking for details that are as close to live as possible, I'd recommend checking out Twitter, because many teams and team members are live Tweeting comments and pictures and stuff, and the easiest way to find that is by searching for the #SubTChallenge hashtag.

Lastly, if you've got specific things that you'd like to see or questions for DARPA or for any of the teams, ping me on Twitter @BotJunkie and I'll happily see what I can do.



This is it! This week, we're at the DARPA SubTerranean Challenge Finals in Louisville KY, where more than two dozen Systems Track and Virtual Track teams will compete for millions of dollars in prize money and being able to say "we won a DARPA challenge," which is of course priceless.

We've been following SubT for years, from Tunnel Circuit to Urban Circuit to Cave (non-) Circuit. For a recent recap, have a look at this post-cave pre-final article that includes an interview with SubT Program Manager Tim Chung, but if you don't have time for that, the TLDR is that this week we're looking at both a Virtual Track as well as a Systems Track with physical robots on a real course. The Systems Track teams spent Monday checking in at the Louisville Mega Cavern competition site, and we asked each team to tell us about how they've been preparing, what they think will be most challenging, and what makes them unique.

Team CERBERUS

Team CERBERUS

CERBERUS

Country

USA, Switzerland, United Kingdom, Norway

Members

University of Nevada, Reno

ETH Zurich, Switzerland

University of California, Berkeley

Sierra Nevada Corporation

Flyability, Switzerland

Oxford Robotics Institute, United Kingdom

Norwegian University for Science and Technology (NTNU), Norway

Robots

TBA

Follow Team

Website

@CerberusSubt

Q&A: Team Lead Kostas Alexis

How have you been preparing for the SubT Final?

First of all this year's preparation was strongly influenced by Covid-19 as our team spans multiple countries, namely the US, Switzerland, Norway, and the UK. Despite the challenges, we leveled up both our weekly shake-out events and ran a 2-month team-wide integration and testing activity in Switzerland during July and August with multiple tests in diverse underground settings including multiple mines. Note that we bring a brand new set of 4 ANYmal C robots and a new generation of collision-tolerant flying robots so during this period we further built new hardware.

What do you think the biggest challenge of the SubT Final will be?

We are excited to see how the combination of vastly large spaces available in Mega Caverns can be combined with very narrow cross-sections as DARPA promises and vertical structures. We think that terrain with steep slopes and other obstacles, complex 3D geometries, as well as the dynamic obstacles will be the core challenges.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Our team coined early on the idea of legged and flying robot combination. We have remained focused on this core vision of ours and also bring fully own-developed hardware for both legged and flying systems. This is both our advantage and - in a way - our limitation as we spend a lot of time in its development. We are fully excited about the potential we see developing and we are optimistic that this will be demonstrated in the Final Event!

Team Coordinated Robotics

Team Coordinated Robotics

Coordinated Robotics

Country

USA

Members

California State University Channel Islands

Oke Onwuka

Sequoia Middle School

Robots

TBA

Q&A: Team Lead Kevin Knoedler

How have you been preparing for the SubT Final?

Coordinated Robotics has been preparing for the SubT Final with lots of testing on our team of robots. We have been running them inside, outside, day, night and all of the circumstances that we can come up with. In Kentucky we have been busy updating all of the robots to the same standard and repairing bits of shipping damage before the Subt Final.

What do you think the biggest challenge of the SubT Final will be?

The biggest challenge for us will be pulling all of the robots together to work as a team and make sure that everything is communicating together. We did not have lab access until late July and so we had robots at individuals homes, but were generally only testing one robot at a time.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Coordinated Robotics is unique in a couple of different ways. We are one of only two unfunded teams so we take a lower budget approach to solving lots of the issues and that helps us to have some creative solutions. We are also unique in that we will be bringing a lot of robots (23) so that problems with individual robots can be tolerated as the team of robots continues to search.

Team CoSTAR

Team CoSTAR

CoSTAR

Country

USA, South Korea, Sweden

Members

Jet Propulsion Laboratory

California Institute of Technology

Massachusetts Institute of Technology

KAIST, South Korea

Lulea University of Technology, Sweden

Robots

TBA

Follow Team

Website

Q&A: Caltech Team Lead Joel Burdick

How have you been preparing for the SubT Final?

Since May, the team has made 4 trips to a limestone cave near Lexington Kentucky (and they are just finishing a week-long "game" there yesterday). Since February, parts or all of the team have been testing 2-3 days a week in a section of the abandoned Subway system in downtown Los Angeles.

What do you think the biggest challenge of the SubT Final will be?

That will be a tough one to answer in advance. The expected CoSTAR-specific challenges are of course the complexity of the test-site that DARPA has prepared, fatigue of the team, and the usual last-minute hardware failures: we had to have an entire new set of batteries for all of our communication nodes FedExed to us yesterday. More generally, we expect the other teams to be well prepared. Speaking only for myself, I think there will be 4-5 teams that could easily win this competition.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Previously, our team was unique with our Boston Dynamic legged mobility. We've heard that other teams maybe using Spot quadrupeds as well. So, that may no longer be a uniqueness. We shall see! More importantly, we believe our team is unique in the breadth of the participants (university team members from U.S., Europe, and Asia). Kind of like the old British empire: the sun never sets on the geographic expanse of Team CoSTAR.

Team CSIRO Data61

Team CSIRO Data61

CSIRO Data61

Country

Australia, USA

Members

Commonwealth Scientific and Industrial Research Organisation, Australia

Emesent, Australia

Georgia Institute of Technology

Robots

TBA

Follow Team

Website

Twitter

Q&A: SubT Principal Investigator Navinda Kottege

How have you been preparing for the SubT Final?

Test, test, test. We've been testing as often as we can, simulating the competition conditions as best we can. We're very fortunate to have an extensive site here at our CSIRO lab in Brisbane that has enabled us to construct quite varied tests for our full fleet of robots. We have also done a number of offsite tests as well.

After going through the initial phases, we have converged on a good combination of platforms for our fleet. Our work horse platform from the Tunnel circuit has been the BIA5 ATR tracked robot. We have recently added Boston Dynamics Spot quadrupeds to our fleet and we are quite happy with their performance and the level of integration with our perception and navigation stack. We also have custom designed Subterra Navi drones from Emesent. Our fleet consists of two of each of these three platform types. We have also designed and built a new 'Smart node' for communication with the Rajant nodes. These are dropped from the tracked robots and automatically deploy after a delay by extending out ground plates and antennae. As described above, we have been doing extensive integration testing with the full system to shake out bugs and make improvements.

What do you think the biggest challenge of the SubT Final will be?

The biggest challenge is the unknown. It is always a learning process to discover how the robots respond to new classes of obstacle; responding to this on the fly in a new environment is extremely challenging. Given the format of two preliminary runs and one prize run, there is little to no margin for error compared to previous circuit events where there were multiple runs that contributed to the final score. Any significant damage to robots during the preliminary runs would be difficult to recover from to perform in the final run.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Our fleet uses a common sensing, mapping and navigation system across all robots, built around our Wildcat SLAM technology. This is what enables coordination between robots, and provides the accuracy required to locate detected objects. This had allowed us to easily integrate different robot platforms into our fleet. We believe this 'homogenous sensing on heterogenous platforms' paradigm gives us a unique advantage in reducing overall complexity of the development effort for the fleet and also allowing us to scale our fleet as needed. Having excellent partners in Emesent and Georgia Tech and having their full commitment and support is also a strong advantage for us.

Team CTU-CRAS-NORLAB

Team CTU-CRAS-NORLAB

CTU-CRAS-NORLAB

Country

Czech Republic, Canada

Members

Czech Technological University, Czech Republic

Université Laval, Canada

Robots

TBA

Follow Team

Website

Twitter

Q&A: Team Lead Tomas Svoboda

How have you been preparing for the SubT Final?

We spent most of the time preparing new platforms as we made a significant technology update. We tested the locomotion and autonomy of the new platforms in Bull Rock Cave, one of the largest caves in Czechia. We also deployed the robots in an old underground fortress to examine the system in an urban-like underground environment. The very last weeks were, however, dedicated to integration tests and system tuning.

What do you think the biggest challenge of the SubT Final will be?

Hard to say, but regarding the expected environment, the vertical shafts might be the most challenging since they are not easy to access to test and tune the system experimentally. They would also add challenges to communication.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Not sure about the other teams, but we plan to deploy all kinds of ground vehicles, tracked, wheeled, and legged platforms accompanied by several drones. We hope the diversity of the platform types would be beneficial for adapting to the possible diversity of terrains and underground challenges. Besides, we also hope the tuned communication would provide access to robots in a wider range than the last time. Optimistically, we might keep all robots connected to the communication infrastructure built during the mission, albeit the bandwidth is very limited, but should be sufficient for artifacts reporting and high-level switching of the robots' goals and autonomous behavior.

Team Explorer

Team Explorer

Explorer

Country

USA

Members

Carnegie Mellon University

Oregon State University

Robots

TBA

Follow Team

Website

Facebook

Q&A: Team Co-Lead Sebastian Scherer

How have you been preparing for the SubT Final?

Since we expect DARPA to have some surprises on the course for us, we have been practicing in a wide range of different courses around Pittsburgh including an abandoned hospital complex, a cave and limestone and coal mines. As the finals approached, we were practicing at these locations nearly daily, with debrief and debugging sessions afterward. This has helped us find the advantages of each of the platforms, ways of controlling them, and the different sensor modalities.

What do you think the biggest challenge of the SubT Final will be?

For our team the biggest challenges are steep slopes for the ground robots and thin loose obstacles that can get sucked into the props for the drones as well as narrow passages.

What is one way in which your team is unique, and why will that be an advantage during the competition?

We have developed a heterogeneous team for SubT exploration. This gives us an advantage since there is not a single platform that is optimal for all SubT environments. Tunnels are optimal for roving robots, urban environments for walking robots, and caves for flying. Our ground robots and drones are custom-designed for navigation in rough terrain and tight spaces. This gives us an advantage since we can get to places not reachable by off-the-shelf platforms.

Team MARBLE

Team MARBLE

MARBLE

Country

USA

Members

University of Colorado, Boulder

University of Colorado, Denver

Scientific Systems Company, Inc.

University of California, Santa Cruz

Robots

TBA

Follow Team

Twitter

Q&A: Project Engineer Gene Rush

How have you been preparing for the SubT Final?

Our team has worked tirelessly over the past several months as we prepare for the SubT Final. We have invested most of our time and energy in real-world field deployments, which help us in two major ways. First, it allows us to repeatedly test the performance of our full autonomy stack, and second, it provides us the opportunity to emphasize Pit Crew and Human Supervisor training. Our PI, Sean Humbert, has always said "practice, practice, practice." In the month leading up to the event, we stayed true to this advice by holding 10 deployments across a variety of environments, including parking garages, campus buildings at the University of Colorado Boulder, and the Edgar Experimental Mine.

What do you think the biggest challenge of the SubT Final will be?

I expect the most difficult challenge will is centered around autonomous high-level decision making. Of course, mobility challenges, including treacherous terrain, stairs, and drop offs will certainly test the physical capabilities of our mobile robots. However, the scale of the environment is so great, and time so limited, that rapidly identifying the areas that likely have human survivors is vitally important and a very difficult open challenge. I expect most teams, ours included, will utilize the intuition of the Human Supervisor to make these decisions.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Our team has pushed on advancing hands-off autonomy, so our robotic fleet can operate independently in the worst case scenario: a communication-denied environment. The lack of wireless communication is relatively prevalent in subterranean search and rescue missions, and therefore we expect DARPA will be stressing this part of the challenge in the SubT Final. Our autonomy solution is designed in such a way that it can operate autonomously both with and without communication back to the Human Supervisor. When we are in communication with our robotic teammates, the Human Supervisor has the ability to provide several high level commands to assist the robots in making better decisions.

Team Robotika

Team Robotika

Robotika

Country

Czech Republic, USA, Switzerland

Members

Robotika International, Czech Republic and United States

Robotika.cz, Czech Republic

Czech University of Life Science, Czech Republic

Centre for Field Robotics, Czech Republic

Cogito Team, Switzerland

Robots

Two wheeled robots

Follow Team

Website

Twitter

Q&A: Team Lead Martin Dlouhy

How have you been preparing for the SubT Final?

Our team participates in both Systems and Virtual tracks. We were using the virtual environment to develop and test our ideas and techniques and once they were sufficiently validated in the virtual world, we would transfer these results to the Systems track as well. Then, to validate this transfer, we visited a few underground spaces (mostly caves) with our physical robots to see how they perform in the real world.

What do you think the biggest challenge of the SubT Final will be?

Besides the usual challenges inherent to the underground spaces (mud, moisture, fog, condensation), we also noticed the unusual configuration of the starting point which is a sharp downhill slope. Our solution is designed to be careful about going on too steep slopes so our concern is that as things stand, the robots may hesitate to even get started. We are making some adjustments in the remaining time to account for this. Also, unlike the environment in all the previous rounds, the Mega Cavern features some really large open spaces. Our solution is designed to expect detection of obstacles somewhere in the vicinity of the robot at any given point so the concern is that a large open space may confuse its navigational system. We are looking into handling such a situation better as well.

What is one way in which your team is unique, and why will that be an advantage during the competition?

It appears that we are unique in bringing only two robots into the Finals. We have brought more into the earlier rounds to test different platforms and ultimately picked the two we are fielding this time as best suited for the expected environment. A potential benefit for us is that supervising only two robots could be easier and perhaps more efficient than managing larger numbers.



This is it! This week, we're at the DARPA SubTerranean Challenge Finals in Louisville KY, where more than two dozen Systems Track and Virtual Track teams will compete for millions of dollars in prize money and being able to say "we won a DARPA challenge," which is of course priceless.

We've been following SubT for years, from Tunnel Circuit to Urban Circuit to Cave (non-) Circuit. For a recent recap, have a look at this post-cave pre-final article that includes an interview with SubT Program Manager Tim Chung, but if you don't have time for that, the TLDR is that this week we're looking at both a Virtual Track as well as a Systems Track with physical robots on a real course. The Systems Track teams spent Monday checking in at the Louisville Mega Cavern competition site, and we asked each team to tell us about how they've been preparing, what they think will be most challenging, and what makes them unique.

Team CERBERUS

Team CERBERUS

CERBERUS

Country

USA, Switzerland, United Kingdom, Norway

Members

University of Nevada, Reno

ETH Zurich, Switzerland

University of California, Berkeley

Sierra Nevada Corporation

Flyability, Switzerland

Oxford Robotics Institute, United Kingdom

Norwegian University for Science and Technology (NTNU), Norway

Robots

TBA

Follow Team

Website

@CerberusSubt

Q&A: Team Lead Kostas Alexis

How have you been preparing for the SubT Final?

First of all this year's preparation was strongly influenced by Covid-19 as our team spans multiple countries, namely the US, Switzerland, Norway, and the UK. Despite the challenges, we leveled up both our weekly shake-out events and ran a 2-month team-wide integration and testing activity in Switzerland during July and August with multiple tests in diverse underground settings including multiple mines. Note that we bring a brand new set of 4 ANYmal C robots and a new generation of collision-tolerant flying robots so during this period we further built new hardware.

What do you think the biggest challenge of the SubT Final will be?

We are excited to see how the combination of vastly large spaces available in Mega Caverns can be combined with very narrow cross-sections as DARPA promises and vertical structures. We think that terrain with steep slopes and other obstacles, complex 3D geometries, as well as the dynamic obstacles will be the core challenges.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Our team coined early on the idea of legged and flying robot combination. We have remained focused on this core vision of ours and also bring fully own-developed hardware for both legged and flying systems. This is both our advantage and - in a way - our limitation as we spend a lot of time in its development. We are fully excited about the potential we see developing and we are optimistic that this will be demonstrated in the Final Event!

Team Coordinated Robotics

Team Coordinated Robotics

Coordinated Robotics

Country

USA

Members

California State University Channel Islands

Oke Onwuka

Sequoia Middle School

Robots

TBA

Q&A: Team Lead Kevin Knoedler

How have you been preparing for the SubT Final?

Coordinated Robotics has been preparing for the SubT Final with lots of testing on our team of robots. We have been running them inside, outside, day, night and all of the circumstances that we can come up with. In Kentucky we have been busy updating all of the robots to the same standard and repairing bits of shipping damage before the Subt Final.

What do you think the biggest challenge of the SubT Final will be?

The biggest challenge for us will be pulling all of the robots together to work as a team and make sure that everything is communicating together. We did not have lab access until late July and so we had robots at individuals homes, but were generally only testing one robot at a time.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Coordinated Robotics is unique in a couple of different ways. We are one of only two unfunded teams so we take a lower budget approach to solving lots of the issues and that helps us to have some creative solutions. We are also unique in that we will be bringing a lot of robots (23) so that problems with individual robots can be tolerated as the team of robots continues to search.

Team CoSTAR

Team CoSTAR

CoSTAR

Country

USA, South Korea, Sweden

Members

Jet Propulsion Laboratory

California Institute of Technology

Massachusetts Institute of Technology

KAIST, South Korea

Lulea University of Technology, Sweden

Robots

TBA

Follow Team

Website

Q&A: Caltech Team Lead Joel Burdick

How have you been preparing for the SubT Final?

Since May, the team has made 4 trips to a limestone cave near Lexington Kentucky (and they are just finishing a week-long "game" there yesterday). Since February, parts or all of the team have been testing 2-3 days a week in a section of the abandoned Subway system in downtown Los Angeles.

What do you think the biggest challenge of the SubT Final will be?

That will be a tough one to answer in advance. The expected CoSTAR-specific challenges are of course the complexity of the test-site that DARPA has prepared, fatigue of the team, and the usual last-minute hardware failures: we had to have an entire new set of batteries for all of our communication nodes FedExed to us yesterday. More generally, we expect the other teams to be well prepared. Speaking only for myself, I think there will be 4-5 teams that could easily win this competition.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Previously, our team was unique with our Boston Dynamic legged mobility. We've heard that other teams maybe using Spot quadrupeds as well. So, that may no longer be a uniqueness. We shall see! More importantly, we believe our team is unique in the breadth of the participants (university team members from U.S., Europe, and Asia). Kind of like the old British empire: the sun never sets on the geographic expanse of Team CoSTAR.

Team CSIRO Data61

Team CSIRO Data61

CSIRO Data61

Country

Australia, USA

Members

Commonwealth Scientific and Industrial Research Organisation, Australia

Emesent, Australia

Georgia Institute of Technology

Robots

TBA

Follow Team

Website

Twitter

Q&A: SubT Principal Investigator Navinda Kottege

How have you been preparing for the SubT Final?

Test, test, test. We've been testing as often as we can, simulating the competition conditions as best we can. We're very fortunate to have an extensive site here at our CSIRO lab in Brisbane that has enabled us to construct quite varied tests for our full fleet of robots. We have also done a number of offsite tests as well.

After going through the initial phases, we have converged on a good combination of platforms for our fleet. Our work horse platform from the Tunnel circuit has been the BIA5 ATR tracked robot. We have recently added Boston Dynamics Spot quadrupeds to our fleet and we are quite happy with their performance and the level of integration with our perception and navigation stack. We also have custom designed Subterra Navi drones from Emesent. Our fleet consists of two of each of these three platform types. We have also designed and built a new 'Smart node' for communication with the Rajant nodes. These are dropped from the tracked robots and automatically deploy after a delay by extending out ground plates and antennae. As described above, we have been doing extensive integration testing with the full system to shake out bugs and make improvements.

What do you think the biggest challenge of the SubT Final will be?

The biggest challenge is the unknown. It is always a learning process to discover how the robots respond to new classes of obstacle; responding to this on the fly in a new environment is extremely challenging. Given the format of two preliminary runs and one prize run, there is little to no margin for error compared to previous circuit events where there were multiple runs that contributed to the final score. Any significant damage to robots during the preliminary runs would be difficult to recover from to perform in the final run.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Our fleet uses a common sensing, mapping and navigation system across all robots, built around our Wildcat SLAM technology. This is what enables coordination between robots, and provides the accuracy required to locate detected objects. This had allowed us to easily integrate different robot platforms into our fleet. We believe this 'homogenous sensing on heterogenous platforms' paradigm gives us a unique advantage in reducing overall complexity of the development effort for the fleet and also allowing us to scale our fleet as needed. Having excellent partners in Emesent and Georgia Tech and having their full commitment and support is also a strong advantage for us.

Team CTU-CRAS-NORLAB

Team CTU-CRAS-NORLAB

CTU-CRAS-NORLAB

Country

Czech Republic, Canada

Members

Czech Technological University, Czech Republic

Université Laval, Canada

Robots

TBA

Follow Team

Website

Twitter

Q&A: Team Lead Tomas Svoboda

How have you been preparing for the SubT Final?

We spent most of the time preparing new platforms as we made a significant technology update. We tested the locomotion and autonomy of the new platforms in Bull Rock Cave, one of the largest caves in Czechia. We also deployed the robots in an old underground fortress to examine the system in an urban-like underground environment. The very last weeks were, however, dedicated to integration tests and system tuning.

What do you think the biggest challenge of the SubT Final will be?

Hard to say, but regarding the expected environment, the vertical shafts might be the most challenging since they are not easy to access to test and tune the system experimentally. They would also add challenges to communication.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Not sure about the other teams, but we plan to deploy all kinds of ground vehicles, tracked, wheeled, and legged platforms accompanied by several drones. We hope the diversity of the platform types would be beneficial for adapting to the possible diversity of terrains and underground challenges. Besides, we also hope the tuned communication would provide access to robots in a wider range than the last time. Optimistically, we might keep all robots connected to the communication infrastructure built during the mission, albeit the bandwidth is very limited, but should be sufficient for artifacts reporting and high-level switching of the robots' goals and autonomous behavior.

Team Explorer

Team Explorer

Explorer

Country

USA

Members

Carnegie Mellon University

Oregon State University

Robots

TBA

Follow Team

Website

Facebook

Q&A: Team Co-Lead Sebastian Scherer

How have you been preparing for the SubT Final?

Since we expect DARPA to have some surprises on the course for us, we have been practicing in a wide range of different courses around Pittsburgh including an abandoned hospital complex, a cave and limestone and coal mines. As the finals approached, we were practicing at these locations nearly daily, with debrief and debugging sessions afterward. This has helped us find the advantages of each of the platforms, ways of controlling them, and the different sensor modalities.

What do you think the biggest challenge of the SubT Final will be?

For our team the biggest challenges are steep slopes for the ground robots and thin loose obstacles that can get sucked into the props for the drones as well as narrow passages.

What is one way in which your team is unique, and why will that be an advantage during the competition?

We have developed a heterogeneous team for SubT exploration. This gives us an advantage since there is not a single platform that is optimal for all SubT environments. Tunnels are optimal for roving robots, urban environments for walking robots, and caves for flying. Our ground robots and drones are custom-designed for navigation in rough terrain and tight spaces. This gives us an advantage since we can get to places not reachable by off-the-shelf platforms.

Team MARBLE

Team MARBLE

MARBLE

Country

USA

Members

University of Colorado, Boulder

University of Colorado, Denver

Scientific Systems Company, Inc.

University of California, Santa Cruz

Robots

TBA

Follow Team

Twitter

Q&A: Project Engineer Gene Rush

How have you been preparing for the SubT Final?

Our team has worked tirelessly over the past several months as we prepare for the SubT Final. We have invested most of our time and energy in real-world field deployments, which help us in two major ways. First, it allows us to repeatedly test the performance of our full autonomy stack, and second, it provides us the opportunity to emphasize Pit Crew and Human Supervisor training. Our PI, Sean Humbert, has always said "practice, practice, practice." In the month leading up to the event, we stayed true to this advice by holding 10 deployments across a variety of environments, including parking garages, campus buildings at the University of Colorado Boulder, and the Edgar Experimental Mine.

What do you think the biggest challenge of the SubT Final will be?

I expect the most difficult challenge will is centered around autonomous high-level decision making. Of course, mobility challenges, including treacherous terrain, stairs, and drop offs will certainly test the physical capabilities of our mobile robots. However, the scale of the environment is so great, and time so limited, that rapidly identifying the areas that likely have human survivors is vitally important and a very difficult open challenge. I expect most teams, ours included, will utilize the intuition of the Human Supervisor to make these decisions.

What is one way in which your team is unique, and why will that be an advantage during the competition?

Our team has pushed on advancing hands-off autonomy, so our robotic fleet can operate independently in the worst case scenario: a communication-denied environment. The lack of wireless communication is relatively prevalent in subterranean search and rescue missions, and therefore we expect DARPA will be stressing this part of the challenge in the SubT Final. Our autonomy solution is designed in such a way that it can operate autonomously both with and without communication back to the Human Supervisor. When we are in communication with our robotic teammates, the Human Supervisor has the ability to provide several high level commands to assist the robots in making better decisions.

Team Robotika

Team Robotika

Robotika

Country

Czech Republic, USA, Switzerland

Members

Robotika International, Czech Republic and United States

Robotika.cz, Czech Republic

Czech University of Life Science, Czech Republic

Centre for Field Robotics, Czech Republic

Cogito Team, Switzerland

Robots

Two wheeled robots

Follow Team

Website

Twitter

Q&A: Team Lead Martin Dlouhy

How have you been preparing for the SubT Final?

Our team participates in both Systems and Virtual tracks. We were using the virtual environment to develop and test our ideas and techniques and once they were sufficiently validated in the virtual world, we would transfer these results to the Systems track as well. Then, to validate this transfer, we visited a few underground spaces (mostly caves) with our physical robots to see how they perform in the real world.

What do you think the biggest challenge of the SubT Final will be?

Besides the usual challenges inherent to the underground spaces (mud, moisture, fog, condensation), we also noticed the unusual configuration of the starting point which is a sharp downhill slope. Our solution is designed to be careful about going on too steep slopes so our concern is that as things stand, the robots may hesitate to even get started. We are making some adjustments in the remaining time to account for this. Also, unlike the environment in all the previous rounds, the Mega Cavern features some really large open spaces. Our solution is designed to expect detection of obstacles somewhere in the vicinity of the robot at any given point so the concern is that a large open space may confuse its navigational system. We are looking into handling such a situation better as well.

What is one way in which your team is unique, and why will that be an advantage during the competition?

It appears that we are unique in bringing only two robots into the Finals. We have brought more into the earlier rounds to test different platforms and ultimately picked the two we are fielding this time as best suited for the expected environment. A potential benefit for us is that supervising only two robots could be easier and perhaps more efficient than managing larger numbers.

Minimally invasive robotic surgery copes with some disadvantages for the surgeon of minimally invasive surgery while preserving the advantages for the patient. Most commercially available robotic systems are telemanipulated with haptic input devices. The exploitation of the haptics channel, e.g., by means of Virtual Fixtures, would allow for an individualized enhancement of surgical performance with contextual assistance. However, it remains an open field of research as it is non-trivial to estimate the task context itself during a surgery. In contrast, surgical training allows to abstract away from a real operation and thus makes it possible to model the task accurately. The presented approach exploits this fact to parameterize Virtual Fixtures during surgical training, proposing a Shared Control Parametrization Engine that retrieves procedural context information from a Digital Twin. This approach accelerates a proficient use of the robotic system for novice surgeons by augmenting the surgeon’s performance through haptic assistance. With this our aim is to reduce the required skill level and cognitive load of a surgeon performing minimally invasive robotic surgery. A pilot study is performed on the DLR MiroSurge system to evaluate the presented approach. The participants are tasked with two benchmark scenarios of surgical training. The execution of the benchmark scenarios requires basic skills as pick, place and path following. The evaluation of the pilot study shows the promising trend that novel users profit from the haptic augmentation during training of certain tasks.

Supervising and controlling remote robot systems currently requires many specialised operators to have knowledge of the internal state of the system in addition to the environment. For applications such as remote maintenance of future nuclear fusion reactors, the number of robots (and hence supervisors) required to maintain or decommission a facility is too large to be financially feasible. To address this issue, this work explores the idea of intelligently filtering information so that a single user can supervise multiple robots safely. We gathered feedback from participants using five methods for teleoperating a semi-autonomous multi-robot system via Virtual Reality (VR). We present a novel 3D interaction method to filter the displayed information to allow the user to read information from the environment without being overwhelmed. The novelty of the interface design is the link between Semantic and Spatial filtering and the hierarchical information contained within the multi robot system. We conducted a user study including a cohort of expert robot teleoperators comparing these methods; highlighting the significant effects of 3D interface design on the performance and perceived workload of a user teleoperating many robot agents in complex environments. The results from this experiment and subjective user feedback will inform future investigations that build upon this initial work.

This paper presents a framework to alleviate the Deep Reinforcement Learning (DRL) training data sparsity problem that is present in challenging domains by creating a DRL agent training and vehicle integration methodology. The methodology leverages accessible domains to train an agent to solve navigational problems such as obstacle avoidance and allows the agent to generalize to challenging and inaccessible domains such as those present in marine environments with minimal further training. This is done by integrating a DRL agent at a high level of vehicle control and leveraging existing path planning and proven low-level control methodologies that are utilized in multiple domains. An autonomy package with a tertiary multilevel controller is developed to enable the DRL agent to interface at the prescribed high control level and thus be separated from vehicle dynamics and environmental constraints. An example Deep Q Network (DQN) employing this methodology for obstacle avoidance is trained in a simulated ground environment, and then its ability to generalize across domains is experimentally validated. Experimental validation utilized a simulated water surface environment and real-world deployment of ground and water robotic platforms. This methodology, when used, shows that it is possible to leverage accessible and data rich domains, such as ground, to effectively develop marine DRL agents for use on Autonomous Surface Vehicle (ASV) navigation. This will allow rapid and iterative agent development without the risk of ASV loss, the cost and logistic overhead of marine deployment, and allow landlocked institutions to develop agents for marine applications.

Developing highly sensitive flexible pressure sensors has become crucially urgent due to the increased societal demand for wearable electronic devices capable of monitoring various human motions. The sensitivity of such sensors has been shown to be significantly enhanced by increasing the relative dielectric permittivity of the dielectric layers used in device construction via compositing with immiscible ionic conductors. Unfortunately, however, the elastomers employed for this purpose possess inhomogeneous morphologies, and thus suffer from poor long-term durability and unstable electrical response. In this study, we developed a novel, flexible, and highly sensitive pressure sensor using an elastomeric dielectric layer with particularly high permittivity and homogeneity due to the addition of synthesized ionic liquid-grafted silicone oil (denoted LMS-EIL). LMS-EIL possesses both a very high relative dielectric permittivity (9.6 × 105 at 10−1 Hz) and excellent compatibility with silicone elastomers due to the covalently connected structure of conductive ionic liquid (IL) and chloropropyl silicone oil. A silicone elastomer with a relative permittivity of 22 at 10−1 Hz, Young’s modulus of 0.78 MPa, and excellent homogeneity was prepared by incorporating 10 phr (parts per hundreds rubber) of LMS-EIL into an elastomer matrix. The sensitivity of the pressure sensor produced using this optimized silicone elastomer was 0.51 kPa−1, which is 100 times higher than that of the pristine elastomer. In addition, a high durability illustrated by 100 loading–unloading cycles and a rapid response and recovery time of approximately 60 ms were achieved. The excellent performance of this novel pressure sensor suggests significant potential for use in human interfaces, soft robotics, and electronic skin applications.



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We'll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Finals – September 21-23, 2021 – Louisville, KY, USAWeRobot 2021 – September 23-25, 2021 – [Online Event]IROS 2021 – September 27-1, 2021 – [Online Event]Robo Boston – October 1-2, 2021 – Boston, MA, USAWearRAcon Europe 2021 – October 5-7, 2021 – [Online Event]ROSCon 2021 – October 20-21, 2021 – [Online Event]Silicon Valley Robot Block Party – October 23, 2021 – Oakland, CA, USA

Let us know if you have suggestions for next week, and enjoy today's videos.

Team Explorer, the SubT Challenge entry from CMU and Oregon State University, is in the last stage of preparation for the competition this month inside the Mega Caverns cave complex in Louisville, Kentucky.

[ Explorer ]

Team CERBERUS is looking good for the SubT Final next week, too.

Autonomous subterranean exploration with the ANYmal C Robot inside the Hagerbach underground mine

[ ARL ]

I'm still as skeptical as I ever was about a big and almost certainly expensive two-armed robot that can do whatever you can program it to do (have fun with that) and seems to rely on an app store for functionality.

[ Unlimited Robotics ]

Project Mineral is using breakthroughs in artificial intelligence, sensors, and robotics to find ways to grow more food, more sustainably.

[ Mineral ]

Not having a torso or anything presumably makes this easier.

Next up, Digit limbo!

[ Hybrid Robotics ]

Paric completed layout of a 500 unit apartment complex utilizing the Dusty FieldPrinter solution. Autonomous layout on the plywood deck saved weeks worth of schedule, allowing the panelized walls to be placed sooner.

[ Dusty Robotics ]

Spot performs inspection in the Kidd Creek Mine, enabling operators to keep their distance from hazards.

[ Boston Dynamics ]

Digit's engineered to be a multipurpose machine. Meaning, it needs to be able to perform a collection of tasks in practically any environment. We do this by first ensuring the robot's physically capable. Then we help the robot perceive its surroundings, understand its surroundings, then reason a best course of action to navigate its environment and accomplish its task. This is where software comes into play. This is early AI in action.

[ Agility Robotics ]

This work proposes a compact robotic limb, AugLimb, that can augment our body functions and support the daily activities. The proposed device can be mounted on the user's upper arm, and transform into compact state without obstruction to wearers.

[ AugLimb ]

Ahold Delhaize and AIRLab need the help of academics who have knowledge of human-robot interactions, mobility, manipulation, programming, and sensors to accelerate the introduction of robotics in retail. In the AIRLab Stacking challenge, teams will work on algorithms that focus on smart retail applications, for example, automated product stacking.

[ PAL Robotics ]

Leica, not at all well known for making robots, is getting into the robotic reality capture business with a payload for Spot and a new drone.

Introducing BLK2FLY: Autonomous Flying Laser Scanner

[ Leica BLK ]

As much as I like Soft Robotics, I'm maybe not quite as optimistic as they are about the potential for robots to take over quite this much from humans in the near term.

[ Soft Robotics ]

Over the course of this video, the robot gets longer and longer and longer.

[ Transcend Robotics ]

This is a good challenge: attach a spool of electrical tape to your drone, which can unpredictably unspool itself and make sure it doesn't totally screw you up.

[ UZH ]

Two interesting short seminars from NCCR Robotics, including one on autonomous racing drones and "neophobic" mobile robots.

Dario Mantegazza: Neophobic Mobile Robots Avoid Potential Hazards

[ NCCR ]

This panel on Synergies between Automation and Robotics comes from ICRA 2021, and once you see the participant list, I bet you'll agree that it's worth a watch.

[ ICRA 2021 ]

CMU RI Seminars are back! This week we hear from Andrew E. Johnson, a Principal Robotics Systems Engineer in the Guidance and Control Section of the NASA Jet Propulsion Laboratory, on "The Search for Ancient Life on Mars Began with a Safe Landing."

Prior mars rover missions have all landed in flat and smooth regions, but for the Mars 2020 mission, which is seeking signs of ancient life, this was no longer acceptable. Terrain relief that is ideal for the science obviously poses significant risks for landing, so a new landing capability called Terrain Relative Navigation (TRN) was added to the mission. This talk will describe the scientific goals of the mission, the Terrain Relative Navigation system design and the successful results from landing on February 18th, 2021.

[ CMU RI Seminar ]



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We'll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Finals – September 21-23, 2021 – Louisville, KY, USAWeRobot 2021 – September 23-25, 2021 – [Online Event]IROS 2021 – September 27-1, 2021 – [Online Event]Robo Boston – October 1-2, 2021 – Boston, MA, USAWearRAcon Europe 2021 – October 5-7, 2021 – [Online Event]ROSCon 2021 – October 20-21, 2021 – [Online Event]Silicon Valley Robot Block Party – October 23, 2021 – Oakland, CA, USA

Let us know if you have suggestions for next week, and enjoy today's videos.

Team Explorer, the SubT Challenge entry from CMU and Oregon State University, is in the last stage of preparation for the competition this month inside the Mega Caverns cave complex in Louisville, Kentucky.

[ Explorer ]

Team CERBERUS is looking good for the SubT Final next week, too.

Autonomous subterranean exploration with the ANYmal C Robot inside the Hagerbach underground mine

[ ARL ]

I'm still as skeptical as I ever was about a big and almost certainly expensive two-armed robot that can do whatever you can program it to do (have fun with that) and seems to rely on an app store for functionality.

[ Unlimited Robotics ]

Project Mineral is using breakthroughs in artificial intelligence, sensors, and robotics to find ways to grow more food, more sustainably.

[ Mineral ]

Not having a torso or anything presumably makes this easier.

Next up, Digit limbo!

[ Hybrid Robotics ]

Paric completed layout of a 500 unit apartment complex utilizing the Dusty FieldPrinter solution. Autonomous layout on the plywood deck saved weeks worth of schedule, allowing the panelized walls to be placed sooner.

[ Dusty Robotics ]

Spot performs inspection in the Kidd Creek Mine, enabling operators to keep their distance from hazards.

[ Boston Dynamics ]

Digit's engineered to be a multipurpose machine. Meaning, it needs to be able to perform a collection of tasks in practically any environment. We do this by first ensuring the robot's physically capable. Then we help the robot perceive its surroundings, understand its surroundings, then reason a best course of action to navigate its environment and accomplish its task. This is where software comes into play. This is early AI in action.

[ Agility Robotics ]

This work proposes a compact robotic limb, AugLimb, that can augment our body functions and support the daily activities. The proposed device can be mounted on the user's upper arm, and transform into compact state without obstruction to wearers.

[ AugLimb ]

Ahold Delhaize and AIRLab need the help of academics who have knowledge of human-robot interactions, mobility, manipulation, programming, and sensors to accelerate the introduction of robotics in retail. In the AIRLab Stacking challenge, teams will work on algorithms that focus on smart retail applications, for example, automated product stacking.

[ PAL Robotics ]

Leica, not at all well known for making robots, is getting into the robotic reality capture business with a payload for Spot and a new drone.

Introducing BLK2FLY: Autonomous Flying Laser Scanner

[ Leica BLK ]

As much as I like Soft Robotics, I'm maybe not quite as optimistic as they are about the potential for robots to take over quite this much from humans in the near term.

[ Soft Robotics ]

Over the course of this video, the robot gets longer and longer and longer.

[ Transcend Robotics ]

This is a good challenge: attach a spool of electrical tape to your drone, which can unpredictably unspool itself and make sure it doesn't totally screw you up.

[ UZH ]

Two interesting short seminars from NCCR Robotics, including one on autonomous racing drones and "neophobic" mobile robots.

Dario Mantegazza: Neophobic Mobile Robots Avoid Potential Hazards

[ NCCR ]

This panel on Synergies between Automation and Robotics comes from ICRA 2021, and once you see the participant list, I bet you'll agree that it's worth a watch.

[ ICRA 2021 ]

CMU RI Seminars are back! This week we hear from Andrew E. Johnson, a Principal Robotics Systems Engineer in the Guidance and Control Section of the NASA Jet Propulsion Laboratory, on "The Search for Ancient Life on Mars Began with a Safe Landing."

Prior mars rover missions have all landed in flat and smooth regions, but for the Mars 2020 mission, which is seeking signs of ancient life, this was no longer acceptable. Terrain relief that is ideal for the science obviously poses significant risks for landing, so a new landing capability called Terrain Relative Navigation (TRN) was added to the mission. This talk will describe the scientific goals of the mission, the Terrain Relative Navigation system design and the successful results from landing on February 18th, 2021.

[ CMU RI Seminar ]



The first ever powered flight by an aircraft on another planetary took place in April when NASA's Ingenuity helicopter, delivered to the Red Planet along with Perseverance rover, but the idea has already taken off elsewhere.

Earlier this month a prototype "Mars surface cruise drone system" developed by a team led by Bian Chunjiang at China's National Space Science Center (NSSC) in Beijing gained approval for further development.

Like Ingenuity, which was intended purely as a technology demonstration, it uses two sets of blades on a single rotor mast to provide lift for vertical take-offs and landings in the very thin Martian atmosphere, which is around 1% the density of Earth's.

The team did consider a fixed wing approach, which other space-related research institutes in China have been developing, but found the constraints related to size, mass, power and lift best met by the single rotor mast approach.

Solar panels charge Ingenuity's batteries enough to allow one 90-second flight per Martian day. The NSSC team is however considering adopting wireless charging through the rover, or a combination of both power systems.

The total mass is 2.1 kilograms, slightly heavier than the 1.8-kg Ingenuity. It would fly at an altitude of 5-10 meters, reaching speeds of around 300 meters per minute, with a possible duration of 3 minutes per flight. Limitations include energy consumption and temperature control.

According to an article published by China Science Daily, Bian proposed development of a helicopter to help guide a rover in March 2019, which was then accepted in June that year. The idea is that by imaging areas ahead the rover could then better select routes which avoid the otherwise unseen areas that restrict and pose challenges to driving.

The small craft's miniature multispectral imaging system may also detect scientifically valuable targets, such as evidence of notable compounds, that would otherwise be missed, deliver preliminary data and direct the rover for more detailed observations.

The next steps, Bian said, will be developing the craft so as to be able to operate in the very low atmospheric pressure and frigid temperatures of Mars as well as the dust environment and other complex environmental variables.

Bian also notes that to properly support science and exploration goals the helicopter design life must be at least a few months or even beyond a year on Mars.

To properly test the vehicle, these conditions will have to be simulated here on Earth. Bian says China does not currently have facilities that can meet all of the parameters. Faced with similar challenges for Ingenuity, Caltech graduate students built a custom wind tunnel for testing, and the NSSC team may likewise need to take a bespoke approach.

"The next 5 to 6 years are a window for research." Bian said. "We hope to overcome these technical problems and allow the next Mars exploration mission to carry a drone on Mars."

When the Mars aircraft could be deployed on Mars is unknown. China's first Mars rover landed in May, but there is no backup vehicle, unlike its predecessor lunar rover missions. The country's next interplanetary mission is expected to be a complex and unprecedented Mars sample-return launching around 2028-2030.

Ingenuity's first flight was declared by NASA to be a "Wright Brothers moment." Six years after the 1903 Wright Flyer, Chinese-born Feng Ru successfully flew his own biplane. Likewise, in the coming years, China will be looking to carry out its own powered flight on another planet.



The first ever powered flight by an aircraft on another planetary took place in April when NASA's Ingenuity helicopter, delivered to the Red Planet along with Perseverance rover, but the idea has already taken off elsewhere.

Earlier this month a prototype "Mars surface cruise drone system" developed by a team led by Bian Chunjiang at China's National Space Science Center (NSSC) in Beijing gained approval for further development.

Like Ingenuity, which was intended purely as a technology demonstration, it uses two sets of blades on a single rotor mast to provide lift for vertical take-offs and landings in the very thin Martian atmosphere, which is around 1% the density of Earth's.

The team did consider a fixed wing approach, which other space-related research institutes in China have been developing, but found the constraints related to size, mass, power and lift best met by the single rotor mast approach.

Solar panels charge Ingenuity's batteries enough to allow one 90-second flight per Martian day. The NSSC team is however considering adopting wireless charging through the rover, or a combination of both power systems.

The total mass is 2.1 kilograms, slightly heavier than the 1.8-kg Ingenuity. It would fly at an altitude of 5-10 meters, reaching speeds of around 300 meters per minute, with a possible duration of 3 minutes per flight. Limitations include energy consumption and temperature control.

According to an article published by China Science Daily, Bian proposed development of a helicopter to help guide a rover in March 2019, which was then accepted in June that year. The idea is that by imaging areas ahead the rover could then better select routes which avoid the otherwise unseen areas that restrict and pose challenges to driving.

The small craft's miniature multispectral imaging system may also detect scientifically valuable targets, such as evidence of notable compounds, that would otherwise be missed, deliver preliminary data and direct the rover for more detailed observations.

The next steps, Bian said, will be developing the craft so as to be able to operate in the very low atmospheric pressure and frigid temperatures of Mars as well as the dust environment and other complex environmental variables.

Bian also notes that to properly support science and exploration goals the helicopter design life must be at least a few months or even beyond a year on Mars.

To properly test the vehicle, these conditions will have to be simulated here on Earth. Bian says China does not currently have facilities that can meet all of the parameters. Faced with similar challenges for Ingenuity, Caltech graduate students built a custom wind tunnel for testing, and the NSSC team may likewise need to take a bespoke approach.

"The next 5 to 6 years are a window for research." Bian said. "We hope to overcome these technical problems and allow the next Mars exploration mission to carry a drone on Mars."

When the Mars aircraft could be deployed on Mars is unknown. China's first Mars rover landed in May, but there is no backup vehicle, unlike its predecessor lunar rover missions. The country's next interplanetary mission is expected to be a complex and unprecedented Mars sample-return launching around 2028-2030.

Ingenuity's first flight was declared by NASA to be a "Wright Brothers moment." Six years after the 1903 Wright Flyer, Chinese-born Feng Ru successfully flew his own biplane. Likewise, in the coming years, China will be looking to carry out its own powered flight on another planet.

Accumulating space debris edges the space domain ever closer to cascading Kessler syndrome, a chain reaction of debris generation that could dramatically inhibit the practical use of space. Meanwhile, a growing number of retired satellites, particularly in higher orbits like geostationary orbit, remain nearly functional except for minor but critical malfunctions or fuel depletion. Servicing these ailing satellites and cleaning up “high-value” space debris remains a formidable challenge, but active interception of these targets with autonomous repair and deorbit spacecraft is inching closer toward reality as shown through a variety of rendezvous demonstration missions. However, some practical challenges are still unsolved and undemonstrated. Devoid of station-keeping ability, space debris and fuel-depleted satellites often enter uncontrolled tumbles on-orbit. In order to perform on-orbit servicing or active debris removal, docking spacecraft (the “Chaser”) must account for the tumbling motion of these targets (the “Target”), which is oftentimes not known a priori. Accounting for the tumbling dynamics of the Target, the Chaser spacecraft must have an algorithmic approach to identifying the state of the Target’s tumble, then use this information to produce useful motion planning and control. Furthermore, careful consideration of the inherent uncertainty of any maneuvers must be accounted for in order to provide guarantees on system performance. This study proposes the complete pipeline of rendezvous with such a Target, starting from a standoff estimation point to a mating point fixed in the rotating Target’s body frame. A novel visual estimation algorithm is applied using a 3D time-of-flight camera to perform remote standoff estimation of the Target’s rotational state and its principal axes of rotation. A novel motion planning algorithm is employed, making use of offline simulation of potential Target tumble types to produce a look-up table that is parsed on-orbit using the estimation data. This nonlinear programming-based algorithm accounts for known Target geometry and important practical constraints such as field of view requirements, producing a motion plan in the Target’s rotating body frame. Meanwhile, an uncertainty characterization method is demonstrated which propagates uncertainty in the Target’s tumble uncertainty to provide disturbance bounds on the motion plan’s reference trajectory in the inertial frame. Finally, this uncertainty bound is provided to a robust tube model predictive controller, which provides tube-based robustness guarantees on the system’s ability to follow the reference trajectory translationally. The combination and interfaces of these methods are shown, and some of the practical implications of their use on a planned demonstration on NASA’s Astrobee free-flyer are additionally discussed. Simulation results of each of the components individually and in a complete case study example of the full pipeline are presented as the study prepares to move toward demonstration on the International Space Station.

Low frequency dynamics introduced by structural flexibility can result in considerable performance degradation and even instability in on-orbit, robotic manipulators. Although there is a wealth of literature that addresses this problem, the author has found that many advanced solutions are often precluded by practical considerations. On the other hand, classical, robust control methods are tractable for these systems if the design problem is properly constrained. This paper investigates a pragmatic engineering approach that evaluates the system’s stability margins in the face of uncertain, flexible perturbation dynamics with frequencies that lie close to or within the bandwidth of the nominal closed-loop system. The robustness of classical control strategies is studied in the context of both collocated (joint rate) and non-collocated (force/torque and vision-based) feedback. It is shown that robust stability and performance depend on the open-loop control bandwidth of the nominal control law (as designed for a simplified, rigid plant). Namely, the designed bandwidth must be constrained to be lower than the minimum flexible mode frequency of the unmodeled dynamics by a given factor. This strategy gives credence to popular heuristic methods commonly used to reduce the effect of unmodeled dynamics in complex manipulator systems.

Robots have become vital to the delivery of health care and their personalities are often important to understanding their effectiveness as health care providers. Despite this, there is a lack of a systematic overarching understanding of personality in health care human-robot interaction. This makes it difficult to understand what we know and do not know about the impact of personality in health care human-robot interaction (H-HRI). As a result, our understanding of personality in H-HRI has not kept pace with the deployment of robots in various health care environments. To address this, the authors conducted a literature review that identified 18 studies on personality in H-HRI. This paper expands, refines, and further explicates the systematic review done in a conference proceedings [see: Esterwood (Proceedings of the 8th International Conference on Human-Agent Interaction, 2020, 87–95)]. Review results: 1) highlight major thematic research areas, 2) derive and present major conclusions from the literature, 3) identify gaps in the literature, and 4) offer guidance for future H-HRI researchers. Overall, this paper represents a reflection on the existing literature and provides an important starting point for future research on personality in H-HRI.

In this survey, results from an investigation on collision avoidance and path planning methods developed in recent research are provided. In particular, existing methods based on Artificial Intelligence, data-driven methods based on Machine Learning, and other Data Science approaches are investigated to provide a comprehensive overview of maritime collision avoidance techniques applicable to Maritime Autonomous Surface Ships. Relevant aspects of those methods and approaches are summarized and put into suitable perspectives. As autonomous systems are expected to operate alongside or in place of conventionally manned vessels, they must comply with the COLREGs for robust decision-support/-making. Thus, the survey specifically covers how COLREGs are addressed by the investigated methods and approaches. A conclusion regarding their utilization in industrial implementations is drawn.



While Boston Dynamics' Atlas humanoid spends its time learning how to dance and do parkour, the company's Spot quadruped is quietly getting much better at doing useful, valuable tasks in commercial environments. Solving tasks like dynamic path planning and door manipulation in a way that's robust enough that someone can buy your robot and not regret it is, I would argue, just as difficult (if not more difficult) as getting a robot to do a backflip.

With a short blog post today, Boston Dynamics is announcing Spot Release 3.0, representing more than a year of software improvements over Release 2.0 that we covered back in May of 2020. The highlights of Release 3.0 include autonomous dynamic replanning, cloud integration, some clever camera tricks, and a new ability to handle push-bar doors, and earlier today, we spoke with Spot Chief Engineer at Boston Dynamics Zachary Jackowski to learn more about what Spot's been up to.

Here are some highlights from Spot's Release 3.0 software upgrade today, lifted from this blog post which has the entire list:

  • Mission planning: Save time by selecting which inspection actions you want Spot to perform, and it will take the shortest path to collect your data.
  • Dynamic replanning: Don't miss inspections due to changes on site. Spot will replan around blocked paths to make sure you get the data you need.
  • Repeatable image capture: Capture the same image from the same angle every time with scene-based camera alignment for the Spot CAM+ pan-tilt-zoom (PTZ) camera.
  • Cloud-compatible: Connect Spot to AWS, Azure, IBM Maximo, and other systems with existing or easy-to-build integrations.
  • Manipulation: Remotely operate the Spot Arm with ease through rear Spot CAM integration and split-screen view. Arm improvements also include added functionality for push-bar doors, revamped grasping UX, and updated SDK.
  • Sounds: Keep trained bystanders aware of Spot with configurable warning sounds.

The focus here is not just making Spot more autonomous, but making Spot more autonomous in some very specific ways that are targeted towards commercial usefulness. It's tempting to look at this stuff and say that it doesn't represent any massive new capabilities. But remember that Spot is a product, and its job is to make money, which is an enormous challenge for any robot, much less a relatively expensive quadruped.

For more details on the new release and a general update about Spot, we spoke with Zachary Jackowski, Spot Chief Engineer at Boston Dynamics.

IEEE Spectrum: So what's new with Spot 3.0, and why is this release important?

Zachary Jackowski: We've been focusing heavily on flexible autonomy that really works for our industrial customers. The thing that may not quite come through in the blog post is how iceberg-y making autonomy work on real customer sites is. Our blog post has some bullet points about "dynamic replanning" in maybe 20 words, but in doing that, we actually reengineered almost our entire autonomy system based on the failure modes of what we were seeing on our customer sites.

The biggest thing that changed is that previously, our robot mission paradigm was a linear mission where you would take the robot around your site and record a path. Obviously, that was a little bit fragile on complex sites—if you're on a construction site and someone puts a pallet in your path, you can't follow that path anymore. So we ended up engineering our autonomy system to do building scale mapping, which is a big part of why we're calling it Spot 3.0. This is state-of-the-art from an academic perspective, except that it's volume shipping in a real product, which to me represents a little bit of our insanity.

And one super cool technical nugget in this release is that we have a powerful pan/tilt/zoom camera on the robot that our customers use to take images of gauges and panels. We've added scene-based alignment and also computer vision model-based alignment so that the robot can capture the images from the same perspective, every time, perfectly framed. In pictures of the robot, you can see that there's this crash cage around the camera, but the image alignment stuff actually does inverse kinematics to command the robot's body to shift a little bit if the cage is including anything important in the frame.

When Spot is dynamically replanning around obstacles, how much flexibility does it have in where it goes?

There are a bunch of tricks to figuring out when to give up on a blocked path, and then it's very simple run of the mill route planning within an existing map. One of the really big design points of our system, which we spent a lot of time talking about during the design phase, is that it turns out in these high value facilities people really value predictability. So it's not desired that the robot starts wandering around trying to find its way somewhere.

Do you think that over time, your customers will begin to trust the robot with more autonomy and less predictability?

I think so, but there's a lot of trust to be built there. Our customers have to see the robot to do the job well for a significant amount of time, and that will come.

Can you talk a bit more about trying to do state-of-the-art work on a robot that's being deployed commercially?

I can tell you about how big the gap is. When we talk about features like this, our engineers are like, "oh yeah I could read this paper and pull this algorithm and code something up over a weekend and see it work." It's easy to get a feature to work once, make a really cool GIF, and post it to the engineering group chat room. But if you take a look at what it takes to actually ship a feature at product-level, we're talking person-years to have it reach the level of quality that someone is accustomed to buying an iPhone and just having it work perfectly all the time. You have to write all the code to product standards, implement all your tests, and get everything right there, and then you also have to visit a lot of customers, because the thing that's different about mobile robotics as a product is that it's all about how the system responds to environments that it hasn't seen before.

The blog post calls Spot 3.0 "A Sensing Solution for the Real World." What is the real world for Spot at this point, and how will that change going forward?

For Spot, 'real world' means power plants, electrical switch yards, chemical plants, breweries, automotive plants, and other living and breathing industrial facilities that have never considered the fact that a robot might one day be walking around in them. It's indoors, it's outdoors, in the dark and in direct sunlight. When you're talking about the geometric aspect of sites, that complexity we're getting pretty comfortable with.

I think the frontiers of complexity for us are things like, how do you work in a busy place with lots of untrained humans moving through it—that's an area where we're investing a lot, but it's going to be a big hill to climb and it'll take a little while before we're really comfortable in environments like that. Functional safety, certified person detectors, all that good stuff, that's a really juicy unsolved field.

Spot can now open push-bar doors, which seems like an easier problem than doors with handles, which Spot learned to open a while ago. Why'd you start with door handles first?

Push-bar doors is an easier problem! But being engineers, we did the harder problem first, because we wanted to get it done.


While Boston Dynamics' Atlas humanoid spends its time learning how to dance and do parkour, the company's Spot quadruped is quietly getting much better at doing useful, valuable tasks in commercial environments. Solving tasks like dynamic path planning and door manipulation in a way that's robust enough that someone can buy your robot and not regret it is, I would argue, just as difficult (if not more difficult) as getting a robot to do a backflip.

With a short blog post today, Boston Dynamics is announcing Spot Release 3.0, representing more than a year of software improvements over Release 2.0 that we covered back in May of 2020. The highlights of Release 3.0 include autonomous dynamic replanning, cloud integration, some clever camera tricks, and a new ability to handle push-bar doors, and earlier today, we spoke with Spot Chief Engineer at Boston Dynamics Zachary Jackowski to learn more about what Spot's been up to.

Here are some highlights from Spot's Release 3.0 software upgrade today, lifted from this blog post which has the entire list:

  • Mission planning: Save time by selecting which inspection actions you want Spot to perform, and it will take the shortest path to collect your data.
  • Dynamic replanning: Don't miss inspections due to changes on site. Spot will replan around blocked paths to make sure you get the data you need.
  • Repeatable image capture: Capture the same image from the same angle every time with scene-based camera alignment for the Spot CAM+ pan-tilt-zoom (PTZ) camera.
  • Cloud-compatible: Connect Spot to AWS, Azure, IBM Maximo, and other systems with existing or easy-to-build integrations.
  • Manipulation: Remotely operate the Spot Arm with ease through rear Spot CAM integration and split-screen view. Arm improvements also include added functionality for push-bar doors, revamped grasping UX, and updated SDK.
  • Sounds: Keep trained bystanders aware of Spot with configurable warning sounds.

The focus here is not just making Spot more autonomous, but making Spot more autonomous in some very specific ways that are targeted towards commercial usefulness. It's tempting to look at this stuff and say that it doesn't represent any massive new capabilities. But remember that Spot is a product, and its job is to make money, which is an enormous challenge for any robot, much less a relatively expensive quadruped.

For more details on the new release and a general update about Spot, we spoke with Zachary Jackowski, Spot Chief Engineer at Boston Dynamics.

IEEE Spectrum: So what's new with Spot 3.0, and why is this release important?

Zachary Jackowski: We've been focusing heavily on flexible autonomy that really works for our industrial customers. The thing that may not quite come through in the blog post is how iceberg-y making autonomy work on real customer sites is. Our blog post has some bullet points about "dynamic replanning" in maybe 20 words, but in doing that, we actually reengineered almost our entire autonomy system based on the failure modes of what we were seeing on our customer sites.

The biggest thing that changed is that previously, our robot mission paradigm was a linear mission where you would take the robot around your site and record a path. Obviously, that was a little bit fragile on complex sites—if you're on a construction site and someone puts a pallet in your path, you can't follow that path anymore. So we ended up engineering our autonomy system to do building scale mapping, which is a big part of why we're calling it Spot 3.0. This is state-of-the-art from an academic perspective, except that it's volume shipping in a real product, which to me represents a little bit of our insanity.

And one super cool technical nugget in this release is that we have a powerful pan/tilt/zoom camera on the robot that our customers use to take images of gauges and panels. We've added scene-based alignment and also computer vision model-based alignment so that the robot can capture the images from the same perspective, every time, perfectly framed. In pictures of the robot, you can see that there's this crash cage around the camera, but the image alignment stuff actually does inverse kinematics to command the robot's body to shift a little bit if the cage is including anything important in the frame.

When Spot is dynamically replanning around obstacles, how much flexibility does it have in where it goes?

There are a bunch of tricks to figuring out when to give up on a blocked path, and then it's very simple run of the mill route planning within an existing map. One of the really big design points of our system, which we spent a lot of time talking about during the design phase, is that it turns out in these high value facilities people really value predictability. So it's not desired that the robot starts wandering around trying to find its way somewhere.

Do you think that over time, your customers will begin to trust the robot with more autonomy and less predictability?

I think so, but there's a lot of trust to be built there. Our customers have to see the robot to do the job well for a significant amount of time, and that will come.

Can you talk a bit more about trying to do state-of-the-art work on a robot that's being deployed commercially?

I can tell you about how big the gap is. When we talk about features like this, our engineers are like, "oh yeah I could read this paper and pull this algorithm and code something up over a weekend and see it work." It's easy to get a feature to work once, make a really cool GIF, and post it to the engineering group chat room. But if you take a look at what it takes to actually ship a feature at product-level, we're talking person-years to have it reach the level of quality that someone is accustomed to buying an iPhone and just having it work perfectly all the time. You have to write all the code to product standards, implement all your tests, and get everything right there, and then you also have to visit a lot of customers, because the thing that's different about mobile robotics as a product is that it's all about how the system responds to environments that it hasn't seen before.

The blog post calls Spot 3.0 "A Sensing Solution for the Real World." What is the real world for Spot at this point, and how will that change going forward?

For Spot, 'real world' means power plants, electrical switch yards, chemical plants, breweries, automotive plants, and other living and breathing industrial facilities that have never considered the fact that a robot might one day be walking around in them. It's indoors, it's outdoors, in the dark and in direct sunlight. When you're talking about the geometric aspect of sites, that complexity we're getting pretty comfortable with.

I think the frontiers of complexity for us are things like, how do you work in a busy place with lots of untrained humans moving through it—that's an area where we're investing a lot, but it's going to be a big hill to climb and it'll take a little while before we're really comfortable in environments like that. Functional safety, certified person detectors, all that good stuff, that's a really juicy unsolved field.

Spot can now open push-bar doors, which seems like an easier problem than doors with handles, which Spot learned to open a while ago. Why'd you start with door handles first?

Push-bar doors is an easier problem! But being engineers, we did the harder problem first, because we wanted to get it done.


A few years ago, Martin Ford published a book called Architects of Intelligence, in which he interviewed 23 of the most experienced AI and robotics researchers in the world. Those interviews are just as fascinating to read now as they were in 2018, but Ford's since had some extra time to chew on them, in the context of a several years of somewhat disconcertingly rapid AI progress (and hype), coupled with the economic upheaval caused by the pandemic.

In his new book, Rule of the Robots: How Artificial Intelligence Will Transform Everything, Ford takes a markedly well-informed but still generally optimistic look at where AI is taking us as a society. It's not all good, and there are still a lot of unknowns, but Ford has a perspective that's both balanced and nuanced, and I can promise you that the book is well worth a read.

The following excerpt is a section entitled "Warning Signs," from the chapter "Deep Learning and the Future of Artificial Intelligence."

—Evan Ackerman

The 2010s were arguably the most exciting and consequential decade in the history of artificial intelligence. Though there have certainly been conceptual improvements in the algorithms used in AI, the primary driver of all this progress has simply been deploying more expansive deep neural networks on ever faster computer hardware where they can hoover up greater and greater quantities of training data. This "scaling" strategy has been explicit since the 2012 ImageNet competition that set off the deep learning revolution. In November of that year, a front-page New York Times article was instrumental in bringing awareness of deep learning technology to the broader public sphere. The article, written by reporter John Markoff, ends with a quote from Geoff Hinton: "The point about this approach is that it scales beautifully. Basically you just need to keep making it bigger and faster, and it will get better. There's no looking back now."

There is increasing evidence, however, that this primary engine of progress is beginning to sputter out. According to one analysis by the research organization OpenAI, the computational resources required for cutting-edge AI projects is "increasing exponentially" and doubling about every 3.4 months.

In a December 2019 Wired magazine interview, Jerome Pesenti, Facebook's Vice President of AI, suggested that even for a company with pockets as deep as Facebook's, this would be financially unsustainable:

When you scale deep learning, it tends to behave better and to be able to solve a broader task in a better way. So, there's an advantage to scaling. But clearly the rate of progress is not sustainable. If you look at top experiments, each year the cost [is] going up 10-fold. Right now, an experiment might be in seven figures, but it's not going to go to nine or ten figures, it's not possible, nobody can afford that.

Pesenti goes on to offer a stark warning about the potential for scaling to continue to be the primary driver of progress: "At some point we're going to hit the wall. In many ways we already have." Beyond the financial limits of scaling to ever larger neural networks, there are also important environmental considerations. A 2019 analysis by researchers at the University of Massachusetts, Amherst, found that training a very large deep learning system could potentially emit as much carbon dioxide as five cars over their full operational lifetimes.

Even if the financial and environmental impact challenges can be overcome—perhaps through the development of vastly more efficient hardware or software—scaling as a strategy simply may not be sufficient to produce sustained progress. Ever-increasing investments in computation have produced systems with extraordinary proficiency in narrow domains, but it is becoming increasingly clear that deep neural networks are subject to reliability limitations that may make the technology unsuitable for many mission critical applications unless important conceptual breakthroughs are made. One of the most notable demonstrations of the technology's weaknesses came when a group of researchers at Vicarious, small company focused on building dexterous robots, performed an analysis of the neural network used in Deep-Mind's DQN, the system that had learned to dominate Atari video games. One test was performed on Breakout, a game in which the player has to manipulate a paddle to intercept a fast-moving ball. When the paddle was shifted just a few pixels higher on the screen—a change that might not even be noticed by a human player—the system's previously superhuman performance immediately took a nose dive. DeepMind's software had no ability to adapt to even this small alteration. The only way to get back to top-level performance would have been to start from scratch and completely retrain the system with data based on the new screen configuration.

What this tells us is that while DeepMind's powerful neural networks do instantiate a representation of the Breakout screen, this representation remains firmly anchored to raw pixels even at the higher levels of abstraction deep in the network. There is clearly no emergent understanding of the paddle as an actual object that can be moved. In other words, there is nothing close to a human-like comprehension of the material objects that the pixels on the screen represent or the physics that govern their movement. It's just pixels all the way down. While some AI researchers may continue to believe that a more comprehensive understanding might eventually emerge if only there were more layers of artificial neurons, running on faster hardware and consuming still more data, I think this is very unlikely. More fundamental innovations will be required before we begin to see machines with a more human-like conception of the world.

This general type of problem, in which an AI system is inflexible and unable to adapt to even small unexpected changes in its input data, is referred to, among researchers, as "brittleness." A brittle AI application may not be a huge problem if it results in a warehouse robot occasionally packing the wrong item into a box. In other applications, however, the same technical shortfall can be catastrophic. This explains, for example, why progress toward fully autonomous self-driving cars has not lived up to some of the more exuberant early predictions.

As these limitations came into focus toward the end of the decade, there was a gnawing fear that the field had once again gotten over its skis and that the hype cycle had driven expectations to unrealistic levels. In the tech media and on social media, one of the most terrifying phrases in the field of artificial intelligence—"AI winter"—was making a reappearance. In a January 2020 interview with the BBC, Yoshua Bengio said that "AI's abilities were somewhat overhyped . . . by certain companies with an interest in doing so."

My own view is that if another AI winter indeed looms, it's likely to be a mild one. Though the concerns about slowing progress are well founded, it remains true that over the past few years AI has been deeply integrated into the infrastructure and business models of the largest technology companies. These companies have seen significant returns on their massive investments in computing resources and AI talent, and they now view artificial intelligence as absolutely critical to their ability to compete in the marketplace. Likewise, nearly every technology startup is now, to some degree, investing in AI, and companies large and small in other industries are beginning to deploy the technology. This successful integration into the commercial sphere is vastly more significant than anything that existed in prior AI winters, and as a result the field benefits from an army of advocates throughout the corporate world and has a general momentum that will act to moderate any downturn.

There's also a sense in which the fall of scalability as the primary driver of progress may have a bright side. When there is a widespread belief that simply throwing more computing resources at a problem will produce important advances, there is significantly less incentive to invest in the much more difficult work of true innovation. This was arguably the case, for example, with Moore's Law. When there was near absolute confidence that computer speeds would double roughly every two years, the semiconductor industry tended to focus on cranking out ever faster versions of the same microprocessor designs from companies like Intel and Motorola. In recent years, the acceleration in raw computer speeds has become less reliable, and our traditional definition of Moore's Law is approaching its end game as the dimensions of the circuits imprinted on chips shrink to nearly atomic size. This has forced engineers to engage in more "out of the box" thinking, resulting in innovations such as software designed for massively parallel computing and entirely new chip architectures—many of which are optimized for the complex calculations required by deep neural networks. I think we can expect the same sort of idea explosion to happen in deep learning, and artificial intelligence more broadly, as the crutch of simply scaling to larger neural networks becomes a less viable path to progress.

Excerpted from "Rule of the Robots: How Artificial Intelligence will Transform Everything." Copyright 2021 Basic Books. Available from Basic Books, an imprint of Hachette Book Group, Inc.

Pages