Feed aggregator

Six months ago, 11 teams and their robots took on the NIOSH research mine in the Tunnel Circuit of the DARPA Subterranean Challenge. Next week, those 11 teams will travel to Washington State, where they’ll compete in the SubT Urban Circuit at Satsop Business Park just outside of Olympia. A six-month break between events is not a lot of time, and from what we’ve heard, teams have been working feverishly take everything they learned during the Tunnel Circuit and prepare themselves for the Urban Circuit.

But the urban underground is very different from a mine, and teams’ strategy (and hardware) will have to adapt to this new environment. Over the last few weeks, we sent each team three questions about what lessons they took away from the Tunnel Circuit, how they’ve been getting ready for the next challenge, and how they expect things to be different this time around.

The comments below come from:

Team Coordinated Robotics (Kevin Knoedler)

  • Coordinated Robotics
  • California State University Channel Islands
  • Sequoia Middle School

Team CERBERUS (Kostas Alexis)

  • University of Nevada, Reno
  • ETH Zurich, Switzerland
  • University of California, Berkeley
  • Sierra Nevada Corporation
  • Flyability, Switzerland
  • Oxford Robotics Institute, England

Team CSIRO Data61 (Nicholas Hudson)

  • Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia
  • Emesent, Australia
  • Georgia Institute of Technology

Team CoSTAR (Joel Burdick)

  • Jet Propulsion Laboratory
  • California Institute of Technology
  • Massachusetts Institute of Technology
  • KAIST, South Korea
  • Lulea University of Technology, Sweden

Team Explorer (Matt Travers)

  • Carnegie Mellon University
  • Oregon State University

Team CTU-CRAS-NORLAB (Tomáš Svoboda)

  • Czech Technical University in Prague
  • Université Laval, Canada

Team MARBLE (Sean Humbert)

  • University of Colorado, Boulder
  • University of Colorado, Denver
  • Scientific Systems Company Inc.

Team NCTU (Eric Lu)

  • National Chiao Tung University, Taiwan
  • National Tsing Hua University, Taiwan
  • K-Best Technology INC., Taiwan

Team Robotika (František Brabec)

  • Robotika International, Czech Republic and United States
  • Czech University of Life Science, Czech Republic
  • Centre for Field Robotics, Czech Republic
  • Cogito Team, Switzerland
What was the biggest challenge (or biggest surprise) for your team during the Tunnel Circuit?

Team Coordinated Robotics: The biggest surprise for us was aerodynamic instability of the quadrotors in the tunnels. The quadrotors would become unstable and crash into the walls.

Team CoSTAR: Everyone on the team will probably have a different opinion but I think most will agree that keeping the hardware in competition form, and integrating some last minute changes into the hardware, was the hardest challenge. Many members of the hardware sub-team pulled multiple 36-hour shifts during the competition week to keep everything going.

Team CERBERUS: For our team the greatest challenge related to the integration of the diverse set of robotic systems. CERBERUS combines both legged and flying robotic systems, and we developed two configurations of aerial platforms that provide mechanical collision resilience. This research direction we believe is key to offering a versatile and generic solution but also imposes hard integration challenges.

Team CSIRO Data61: Our underground robot-to-robot communications. We started the project developing our own system, but time pressure forced us to use a modified off-the-shelf radio mesh solution. The system ended up being very fragile; some robots were unable to send or receive any packets to the operator station, even within line of sight. We now use Rajant Radios (ES1s and DX2s) on our robots.

Team CTU-CRAS-NORLAB: Actually, nothing serious, but very wet surface in one of the courses that nearly killed our 3D lidar sensing. Of course, we knew that reflective surfaces like glass would pose problems of this kind. We are one of the self-funded teams, and our robots are from our previous research projects where we did not send them into mud and water regularly. The upgrades for SubT are ongoing, and for Tunnel Circuit we do not test in such environments extensively. So we had to improvise overnights since several critical autonomous components are connected to the 3D sensing.

Team Explorer: Our biggest challenge was definitely communications. Our operator had very little if any contact with our two ground vehicles for most of the four runs. The robots were therefore operating in full autonomy and we spent a lot of time waiting for them to hopefully return to the comms network to data drop. Having been through the testing and seeing what can happen when deployed in full autonomy in a challenging environment (e.g., a mine), dealing with the feeling of “lost comms” might specifically be the most challenging thing our team went through in Tunnel. 

“Our biggest challenge was definitely communications. Our operator had very little if any contact with our two ground vehicles for most of the four runs. The robots were therefore operating in full autonomy and we spent a lot of time waiting for them to hopefully return to the comms network to data drop.” —Matt Travers, Team Explorer

Team MARBLE: Our team’s biggest challenge was getting a robust comms solution deployed. We had some mechanical beacon deployment issues early in the event, and we basically had to lean heavily on our autonomy stack as a result. We were very happy with the results and validated our algorithms for the next event.

Team NCTU: For our blimp robot, Duckiefloat, although it did well in terms of collision tolerance, the airflow in the environment makes the control harder than expected. Duckiefloat also gets stuck in some locations like constrained path, slopes, or turns. We found that communication is a seriously tough challenge in such an environment. Last, the preparation of robots before sending them to the environment takes us more time than expected which results in less time for robots performing searching.

Team Robotika: Since we participated in the STIX training round a few months earlier, there weren’t many surprises waiting for us in Pittsburgh. The biggest challenge turned out to be the lighting situation inside the mines. The existing illumination consisted of a number of bright dots (probably LED bulbs) which blinded cameras on some of our robots. We had to run into the nearest hardware store and purchase additional lights for our subsequent runs. We also had trouble with the delivery of our lithium batteries from overseas.

How have you been preparing for the Urban Circuit?

Team Coordinated Robotics: Our main focus over the past few months has been growing the team. Coordinated Robotics was one person at the system track Tunnel event. We have had over 30 people help for the system track Urban event.

Team CoSTAR: The prep for the Urban Circuit has been somewhat analogous to our prep for the Tunnel Circuit, but there are a couple of key differences. First, we were not able to acquire testing sites that will be a good simulation for the Urban Circuit. Second, we decided on some pretty massive change in strategy and hardware platforms late in the preparation phase. We’ll see at the contest if this pays off.

Team CERBERUS: This year we’re focusing on both the Urban and the Cave Circuit. For the Urban Circuit, we focused on the existing systems emphasizing on their overall integration readiness and their capacity to address the challenge of multi-level exploration both using legged and flying robots. So both climbing and flying within staircases became a central part of our work.

Team CSIRO DATA61: Every Friday we do a weekly test, deploying multiple robots in a specially designed tunnel system onsite at CSIRO. Thanks to Hutchinson Builders in Brisbane, we have also had the chance to do bigger field trials in their underground construction areas. The local navigation stacks on both our UGVs and Emesent Drones have been significantly improved since the Tunnel Circuit, enabling much better autonomy.

Team CTU-CRAS-NORLAB: A multi-floor environment and going up and down are the main new challenges. We had to advance our exploration algorithm to 3D and autonomous robot locomotion for traversing staircases. Also, the gas detection and localization problem is new. The gas is not as clearly localized as solid objects, which required extensions of detection reasoning.

Team Explorer: Practicing. Specifically, we have spent a lot of time preparing for terrains that are more “three-dimensional” in nature than tunnels.

Team MARBLE: The main thing has been engineering new platforms and algorithms to handle the increased mobility challenges the Urban Circuit presents. Stairs and multi-level deployments increase the difficulty level significantly, and this requires novel insights backed by lots of testing and validation to field a successful team. We have been fortunate to have great collaboration with local first responders and businesses that have provided locations where we can test and improve our systems.

“The interesting part of the real competition is that it’s time constrained, with high pressure and unpredictability. So we have trained ourselves like a real search-and-rescue team so that we and our robots can cooperate perfectly during the mission.” —Eric Lu, Team NCTU

Team NCTU: For the last few weeks, we’ve done multiple experiments and even mock scored runs to not only enhance the capability and robustness of our systems, but also to train ourselves to be more familiar with the whole process of the competition. The interesting part of the real competition is that it’s time constrained, with high pressure and unpredictability. So we have trained ourselves like a real search-and-rescue team so that we and our robots can cooperate perfectly during the mission.

Team Robotika: Even though we already won the “Most Distinctive Robots” award at the Tunnel Circuit, we are actually introducing three new platforms to address some of the specifics of the Urban Circuit, namely the presence of stairs and narrow passages.

What will you be doing differently in terms of hardware, software, and/or strategy?

Team Coordinated Robotics: The main hardware change for the Urban Circuit is the addition of four new ground vehicles. Two of the new ground vehicles are smaller skid steering platforms that Sequoia Middle School students helped with. The two larger ground vehicles are being developed by CSUCI. We expect that the ground vehicles will complement the quadrotors. The different platforms will have different sensors and different approaches to localization that should improve our chances at the event.

Team CoSTAR: We will have a “hardware surprise” at the contest that I can’t divulge right now, but 50 percent of our deployed vehicles will be different than the Tunnel Circuit. In terms of software, the basic architecture is the same. But Ben Morrell has been spearheading many upgrades and refinements in our mapping/localization framework and hardware. Also, a fair amount of time has been spent on upgrading our traversability analysis systems to handle stairs. Of course, everything had to be modified/upgraded to handle the multiple floors in the Urban Circuit.

“We will have a ‘hardware surprise’ at the contest that I can’t divulge right now, but 50 percent of our deployed vehicles will be different than the Tunnel Circuit.” —Joel Burdick, Team CoSTAR

Team CERBERUS: Our robotic hardware is largely the same (but improved), while our software is extended to deal with multi-level environments and common mapping. Our communications hardware is new, and we are confident that the new approach is much more powerful as it involves smaller nodes and optimized bandwidth management. This required a lot of new effort but we believe it will pay off in all the upcoming Circuit events.

Team CSIRO DATA61: Our strategy has become more autonomous and distributed. Each robot has a perception backpack (with integrated lidar, cameras, IMU etc.), which share map “frames” between all robots. Each robot computes a live global map containing all the available information from the other robots. This has allowed us to share objects, goals, and frontiers (the boundary of unexplored space) between the robots. As a result, if a robot does not get a command from the operator, it explores a new frontier from the shared map.

Team CTU-CRAS-NORLAB: We opted for an evolution rather than a revolution. We included a gas sensor, newly emerged consumer 3D cameras, full 3D lidars for UAVs, also new batteries were needed as we were almost at our limit during the Tunnel Circuit. We worked out our strategy for keeping the robots connected, using robots in between. We analyzed database information synchronization and are attempting to save bandwidth.

Team Explorer: We are planning to bring some new hardware, both aerial and ground, but for the most part are keeping our system nearly the same as Tunnel. We’ve made some improvements along the way, so we’re (anxiously) excited to see how things work at game time. Roughly the same thing can be said for software. In terms of strategy I think we will need to largely “play it by ear” based on how the competition goes. Nobody yet knows what DARPA has planned for us, so I think being prepared to be a little nimble in your approach is likely the correct move.

Team MARBLE: We have added several new platform types and updated our existing autonomy and perception stacks to handle the 3D aspects of the Urban test environment.

Team NCTU: For the hardware, we installed more varied sensors on both Husky and Duckiefloat. We have millimeter-wave radar sensors on both robots to deal with lidar/camera denied situations. For payload constrained platform like Duckiefloat, radar is even more important since it provides point clouds for geometry sensing but at much lighter weight. We also improved the anchorball solution we had last time. As for the software and strategy, we would like Husky and Duckiefloat to cooperate more to overcome mobility challenges. We have a tethering system that Husky could use to maneuver through the environment while Duckiefloat can travel to different levels.

Team Robotika: Our navigation will more heavily rely on cameras as sensors (as opposed to lidars). We have improved our central control system and communication technology to make it easier for the human operator to manage the robots inside the underground space. Finally, as we are also participating in the Virtual Track of the Urban Circuit, we used our experience from that world and from those runs for developing new strategies to be used by the physical robots in the Systems Track.

DARPA Subterranean Challenge ]

Six months ago, 11 teams and their robots took on the NIOSH research mine in the Tunnel Circuit of the DARPA Subterranean Challenge. Next week, those 11 teams will travel to Washington State, where they’ll compete in the SubT Urban Circuit at Satsop Business Park just outside of Olympia. A six-month break between events is not a lot of time, and from what we’ve heard, teams have been working feverishly take everything they learned during the Tunnel Circuit and prepare themselves for the Urban Circuit.

But the urban underground is very different from a mine, and teams’ strategy (and hardware) will have to adapt to this new environment. Over the last few weeks, we sent each team three questions about what lessons they took away from the Tunnel Circuit, how they’ve been getting ready for the next challenge, and how they expect things to be different this time around.

The comments below come from:

Team Coordinated Robotics (Kevin Knoedler)

  • Coordinated Robotics
  • California State University Channel Islands
  • Sequoia Middle School

Team CERBERUS (Kostas Alexis)

  • University of Nevada, Reno
  • ETH Zurich, Switzerland
  • University of California, Berkeley
  • Sierra Nevada Corporation
  • Flyability, Switzerland
  • Oxford Robotics Institute, England

Team CSIRO Data61 (Nicholas Hudson)

  • Commonwealth Scientific and Industrial Research Organisation (CSIRO), Australia
  • Emesent, Australia
  • Georgia Institute of Technology

Team CoSTAR (Joel Burdick)

  • Jet Propulsion Laboratory
  • California Institute of Technology
  • Massachusetts Institute of Technology
  • KAIST, South Korea
  • Lulea University of Technology, Sweden

Team Explorer (Matt Travers)

  • Carnegie Mellon University
  • Oregon State University

Team CTU-CRAS-NORLAB (Tomáš Svoboda)

  • Czech Technical University in Prague
  • Université Laval, Canada

Team MARBLE (Sean Humbert)

  • University of Colorado, Boulder
  • University of Colorado, Denver
  • Scientific Systems Company Inc.

Team NCTU (Eric Lu)

  • National Chiao Tung University, Taiwan
  • National Tsing Hua University, Taiwan
  • K-Best Technology INC., Taiwan

Team Robotika (František Brabec)

  • Robotika International, Czech Republic and United States
  • Czech University of Life Science, Czech Republic
  • Centre for Field Robotics, Czech Republic
  • Cogito Team, Switzerland
What was the biggest challenge (or biggest surprise) for your team during the Tunnel Circuit?

Team Coordinated Robotics: The biggest surprise for us was aerodynamic instability of the quadrotors in the tunnels. The quadrotors would become unstable and crash into the walls.

Team CoSTAR: Everyone on the team will probably have a different opinion but I think most will agree that keeping the hardware in competition form, and integrating some last minute changes into the hardware, was the hardest challenge. Many members of the hardware sub-team pulled multiple 36-hour shifts during the competition week to keep everything going.

Team CERBERUS: For our team the greatest challenge related to the integration of the diverse set of robotic systems. CERBERUS combines both legged and flying robotic systems, and we developed two configurations of aerial platforms that provide mechanical collision resilience. This research direction we believe is key to offering a versatile and generic solution but also imposes hard integration challenges.

Team CSIRO Data61: Our underground robot-to-robot communications. We started the project developing our own system, but time pressure forced us to use a modified off-the-shelf radio mesh solution. The system ended up being very fragile; some robots were unable to send or receive any packets to the operator station, even within line of sight. We now use Rajant Radios (ES1s and DX2s) on our robots.

Team CTU-CRAS-NORLAB: Actually, nothing serious, but very wet surface in one of the courses that nearly killed our 3D lidar sensing. Of course, we knew that reflective surfaces like glass would pose problems of this kind. We are one of the self-funded teams, and our robots are from our previous research projects where we did not send them into mud and water regularly. The upgrades for SubT are ongoing, and for Tunnel Circuit we do not test in such environments extensively. So we had to improvise overnights since several critical autonomous components are connected to the 3D sensing.

Team Explorer: Our biggest challenge was definitely communications. Our operator had very little if any contact with our two ground vehicles for most of the four runs. The robots were therefore operating in full autonomy and we spent a lot of time waiting for them to hopefully return to the comms network to data drop. Having been through the testing and seeing what can happen when deployed in full autonomy in a challenging environment (e.g., a mine), dealing with the feeling of “lost comms” might specifically be the most challenging thing our team went through in Tunnel. 

“Our biggest challenge was definitely communications. Our operator had very little if any contact with our two ground vehicles for most of the four runs. The robots were therefore operating in full autonomy and we spent a lot of time waiting for them to hopefully return to the comms network to data drop.” —Matt Travers, Team Explorer

Team MARBLE: Our team’s biggest challenge was getting a robust comms solution deployed. We had some mechanical beacon deployment issues early in the event, and we basically had to lean heavily on our autonomy stack as a result. We were very happy with the results and validated our algorithms for the next event.

Team NCTU: For our blimp robot, Duckiefloat, although it did well in terms of collision tolerance, the airflow in the environment makes the control harder than expected. Duckiefloat also gets stuck in some locations like constrained path, slopes, or turns. We found that communication is a seriously tough challenge in such an environment. Last, the preparation of robots before sending them to the environment takes us more time than expected which results in less time for robots performing searching.

Team Robotika: Since we participated in the STIX training round a few months earlier, there weren’t many surprises waiting for us in Pittsburgh. The biggest challenge turned out to be the lighting situation inside the mines. The existing illumination consisted of a number of bright dots (probably LED bulbs) which blinded cameras on some of our robots. We had to run into the nearest hardware store and purchase additional lights for our subsequent runs. We also had trouble with the delivery of our lithium batteries from overseas.

How have you been preparing for the Urban Circuit?

Team Coordinated Robotics: Our main focus over the past few months has been growing the team. Coordinated Robotics was one person at the system track Tunnel event. We have had over 30 people help for the system track Urban event.

Team CoSTAR: The prep for the Urban Circuit has been somewhat analogous to our prep for the Tunnel Circuit, but there are a couple of key differences. First, we were not able to acquire testing sites that will be a good simulation for the Urban Circuit. Second, we decided on some pretty massive change in strategy and hardware platforms late in the preparation phase. We’ll see at the contest if this pays off.

Team CERBERUS: This year we’re focusing on both the Urban and the Cave Circuit. For the Urban Circuit, we focused on the existing systems emphasizing on their overall integration readiness and their capacity to address the challenge of multi-level exploration both using legged and flying robots. So both climbing and flying within staircases became a central part of our work.

Team CSIRO DATA61: Every Friday we do a weekly test, deploying multiple robots in a specially designed tunnel system onsite at CSIRO. Thanks to Hutchinson Builders in Brisbane, we have also had the chance to do bigger field trials in their underground construction areas. The local navigation stacks on both our UGVs and Emesent Drones have been significantly improved since the Tunnel Circuit, enabling much better autonomy.

Team CTU-CRAS-NORLAB: A multi-floor environment and going up and down are the main new challenges. We had to advance our exploration algorithm to 3D and autonomous robot locomotion for traversing staircases. Also, the gas detection and localization problem is new. The gas is not as clearly localized as solid objects, which required extensions of detection reasoning.

Team Explorer: Practicing. Specifically, we have spent a lot of time preparing for terrains that are more “three-dimensional” in nature than tunnels.

Team MARBLE: The main thing has been engineering new platforms and algorithms to handle the increased mobility challenges the Urban Circuit presents. Stairs and multi-level deployments increase the difficulty level significantly, and this requires novel insights backed by lots of testing and validation to field a successful team. We have been fortunate to have great collaboration with local first responders and businesses that have provided locations where we can test and improve our systems.

“The interesting part of the real competition is that it’s time constrained, with high pressure and unpredictability. So we have trained ourselves like a real search-and-rescue team so that we and our robots can cooperate perfectly during the mission.” —Eric Lu, Team NCTU

Team NCTU: For the last few weeks, we’ve done multiple experiments and even mock scored runs to not only enhance the capability and robustness of our systems, but also to train ourselves to be more familiar with the whole process of the competition. The interesting part of the real competition is that it’s time constrained, with high pressure and unpredictability. So we have trained ourselves like a real search-and-rescue team so that we and our robots can cooperate perfectly during the mission.

Team Robotika: Even though we already won the “Most Distinctive Robots” award at the Tunnel Circuit, we are actually introducing three new platforms to address some of the specifics of the Urban Circuit, namely the presence of stairs and narrow passages.

What will you be doing differently in terms of hardware, software, and/or strategy?

Team Coordinated Robotics: The main hardware change for the Urban Circuit is the addition of four new ground vehicles. Two of the new ground vehicles are smaller skid steering platforms that Sequoia Middle School students helped with. The two larger ground vehicles are being developed by CSUCI. We expect that the ground vehicles will complement the quadrotors. The different platforms will have different sensors and different approaches to localization that should improve our chances at the event.

Team CoSTAR: We will have a “hardware surprise” at the contest that I can’t divulge right now, but 50 percent of our deployed vehicles will be different than the Tunnel Circuit. In terms of software, the basic architecture is the same. But Ben Morrell has been spearheading many upgrades and refinements in our mapping/localization framework and hardware. Also, a fair amount of time has been spent on upgrading our traversability analysis systems to handle stairs. Of course, everything had to be modified/upgraded to handle the multiple floors in the Urban Circuit.

“We will have a ‘hardware surprise’ at the contest that I can’t divulge right now, but 50 percent of our deployed vehicles will be different than the Tunnel Circuit.” —Joel Burdick, Team CoSTAR

Team CERBERUS: Our robotic hardware is largely the same (but improved), while our software is extended to deal with multi-level environments and common mapping. Our communications hardware is new, and we are confident that the new approach is much more powerful as it involves smaller nodes and optimized bandwidth management. This required a lot of new effort but we believe it will pay off in all the upcoming Circuit events.

Team CSIRO DATA61: Our strategy has become more autonomous and distributed. Each robot has a perception backpack (with integrated lidar, cameras, IMU etc.), which share map “frames” between all robots. Each robot computes a live global map containing all the available information from the other robots. This has allowed us to share objects, goals, and frontiers (the boundary of unexplored space) between the robots. As a result, if a robot does not get a command from the operator, it explores a new frontier from the shared map.

Team CTU-CRAS-NORLAB: We opted for an evolution rather than a revolution. We included a gas sensor, newly emerged consumer 3D cameras, full 3D lidars for UAVs, also new batteries were needed as we were almost at our limit during the Tunnel Circuit. We worked out our strategy for keeping the robots connected, using robots in between. We analyzed database information synchronization and are attempting to save bandwidth.

Team Explorer: We are planning to bring some new hardware, both aerial and ground, but for the most part are keeping our system nearly the same as Tunnel. We’ve made some improvements along the way, so we’re (anxiously) excited to see how things work at game time. Roughly the same thing can be said for software. In terms of strategy I think we will need to largely “play it by ear” based on how the competition goes. Nobody yet knows what DARPA has planned for us, so I think being prepared to be a little nimble in your approach is likely the correct move.

Team MARBLE: We have added several new platform types and updated our existing autonomy and perception stacks to handle the 3D aspects of the Urban test environment.

Team NCTU: For the hardware, we installed more varied sensors on both Husky and Duckiefloat. We have millimeter-wave radar sensors on both robots to deal with lidar/camera denied situations. For payload constrained platform like Duckiefloat, radar is even more important since it provides point clouds for geometry sensing but at much lighter weight. We also improved the anchorball solution we had last time. As for the software and strategy, we would like Husky and Duckiefloat to cooperate more to overcome mobility challenges. We have a tethering system that Husky could use to maneuver through the environment while Duckiefloat can travel to different levels.

Team Robotika: Our navigation will more heavily rely on cameras as sensors (as opposed to lidars). We have improved our central control system and communication technology to make it easier for the human operator to manage the robots inside the underground space. Finally, as we are also participating in the Virtual Track of the Urban Circuit, we used our experience from that world and from those runs for developing new strategies to be used by the physical robots in the Systems Track.

DARPA Subterranean Challenge ]

The experience of inner speech is a common one. Such a dialogue accompanies the introspection of mental life and fulfills essential roles in human behavior, such as self-restructuring, self-regulation, and re-focusing on attentional resources. Although the underpinning of inner speech is mostly investigated in psychological and philosophical fields, the research in robotics generally does not address such a form of self-aware behavior. Existing models of inner speech inspire computational tools to provide a robot with this form of self-awareness. Here, the widespread psychological models of inner speech are reviewed, and a cognitive architecture for a robot implementing such a capability is outlined in a simplified setup.

With the DARPA Subterranean Challenge Urban Circuit kicking off on Thursday, we made sure to have a chat in advance with Dr. Timothy Chung, DARPA SubT program manager. We last spoke with Tim nearly a year ago, just after SubT was announced, to get his perspective on the Subterranean Challenge in general, and we took the opportunity in this interview to ask about how DARPA felt about the Tunnel Circuit, and what we have to look forward to in the Urban Circuit.

For more details about the SubT Urban Circuit, make sure to check out our course preview post, and check back tomorrow for a Q&A with the systems track teams.

This interview has been edited for length and clarity.

IEEE Spectrum: What turned out to be the biggest challenge for teams during the Tunnel Circuit?

Tim Chung: Where to begin? And I say that from the perspective of every obstacle being a good learning opportunity for the teams! For example, mobility was a huge challenge for the teams—going from cement and gravel into areas with rails that were problematic, to muddy places with big ruts… And that diversity was just on one level of the mine. But I think what we saw from the robots, at least from the ground vehicles, was the ability to deal with the unexpected, at least as far as the terrain was concerned.

As you saw, there were some different platforms that came up with different ways of solving the terrain challenges as well, including wheels on legs, and that was another great opportunity to see under what conditions wheels make sense, or legs make sense, or treads make sense. Again, that was only with the ground vehicles, and there were a lot of other ways to address the terrain, as with air vehicles, but they had their own sets of challenges. Mobility was a good reminder that the real world doesn’t have well-defined terrain parameters.

What impressed you the most during the Tunnel Circuit?

First off, I think the energy and enthusiasm that the teams brought. As a roboticist myself, I remember getting sad at times when my robot would break, but the fact that the teams were so enthusiastic about learning in this challenging environment was a really great takeaway. From a technology point of view, I was very impressed with how teams were able to deal with a lot of the uncertainty of the course itself, in terms of—I won’t call them hacks, but overnight fixes. The ability to say, “We saw that this other team had this innovation, let’s capitalize on that and see if we can retrofit our system to be able to do that as well.” 

“Increasing the autonomous capability of these robots can overcome the limitations of some other technology areas. An example of that is communications—we know communications are really hard. The teams definitely experienced this, and they were able to develop autonomous approaches to address or mitigate some of the communications deficiencies”

It turned out that both speed and agility are important, but so is deliberate decision making. I think what that speaks to is that increasing the autonomous capability of these robots can overcome the limitations of some other technology areas. An example of that is communications—we know communications are really hard. The teams definitely experienced this, and they were able to develop autonomous approaches to address or mitigate some of the communications deficiencies.

Can you share any feedback that you got from teams about the Tunnel Circuit?

The primary feedback that we’ve received has been very positive—teams have expressed that the courses were really, really hard, and hard in different ways than they anticipated. Essentially, that DARPA has set a very high bar. But teams are pleased that the bar is so high. I think what that says to me is that we’re satisfying the role that DARPA plays within the robotics community, of setting the bar really high while inspiring teams to reach it, but I also think that it shows how the community is thirsty for the opportunity to reach a new level of technology. From that perspective, the feedback was, “It was too hard for our robots, but something we still want to overcome.” And that’s great, because ultimately, DARPA’s role is to give the community a chance to surprise itself with how innovative it can be.

Did the results of the Tunnel Circuit cause you to recalibrate the difficulty of the Urban Circuit?

I can say pretty succinctly that the answer is no, in the sense that each of these environments are so wildly different that it’s difficult to say something like, “Tunnel was a difficulty 9, and Urban is going to be a 10.” And the type of challenge elements that we’re introducing in Urban that weren’t at Tunnel also make that comparison hard. The verticality with stairs, for example, will pose totally new challenges for the teams. So I’d say, we didn’t change the difficultly level because the environment just does it intrinsically for us.

In terms of expectations, we know that the teams have come to appreciate what DARPA wants to see out of the challenge, and so they’re better equipped from that perspective. We’ve adjusted by introducing new elements into the environment that will keep them on their toes for sure.

Can you highlight some of the fundamental differences between Tunnel environments and Urban environments?

The environments that are built for people tend not to have been designed with robots in mind, so having these types of systems—systems that can make an impact in everyday lives, and particularly in the first responder case we’re interested in—will necessarily mean that they’ll have to be able to operate in these challenging urban settings where we expect people to be.

Verticality is certainly one of the key distinctions that I think will be really exciting for the Urban Circuit. In Tunnel, the emphasis was not on verticality given the single level in the courses. What we’ve previewed for Urban is that there are sections of the courses where having the ability to navigate vertically will be very beneficial to teams.

Verticality includes not just the need to move in three dimensions, but the ability to perceive in three dimensions as well. For example, we place exit signs above eye level to help them stand out, and we look for indicators like handrails and signs. I think context plays a big role in these urban settings, which we take for granted as humans, but if you’re thinking about using these environmental cues to aid in localization and navigation, there may be some interesting advantages.

Are there other particular challenges for robots in urban environments?

I can give a couple of examples where there are some unique elements. Most of our construction materials tend to involve rebar, or metal in general, that are very bad for communications for sure but also for things like compasses. Another example is the regularity and the oftentimes featureless qualities of urban environments make it difficult for feature-matching technologies to help you with your localization. When one door looks like another door, or a wall looks like every other wall—humans may like that in our urban architecture, but for robots, the lack of discriminating features makes many localization approaches challenging. 

Why did you choose this specific urban environment, as opposed to something like a subway station?

We can certainly appreciate the subway scenario as one attribute of urban underground settings, but as we did research and worked with stakeholders, and also visited a number of various sites in urban centers, I think we learned that there’s just so much to the urban underground, so many types of environments, that to only use a subway wouldn’t have allowed us to implement many of the features that we were interested in for the SubT challenge. Going beyond that, I think the environment we found in Satsop is quite representative of many of the elements of the urban underground.

It’s also a place where we know first responders like the Seattle fire department conduct urban training. Being able to find a place that has value for first responders also contributed to our selection of the site, since it had that realistic feel for what they’re interested in and training for.

Can you talk a little bit about what DARPA’s position is on teams using tethers to communicate with their robots during the SubT Challenge?

[Laughs] What does DARPA have against tethers? I’d say, the idea here is that we’re interested in the capabilities that can be manifested throughout the challenge. While the specific implementation or approach may be of interest from a technology development perspective, it’s really about the capability. And so, if the best capability at present is to insert a robot that is tether-enabled, I think there’s merit there. The environment, however, is far more expansive than would allow for tethers to be the only solution. 

“If tethers showcase the ability to do additional breakthrough capabilities, then bring it! If there are ways in which breakthrough technologies allow us to work without tethers, and that opens up the spaces in which these types of systems can work, that’s another breakthrough that we’d be very excited to see”

Our perspective, which is to neither encourage nor discourage tethers, is partially in response to teams inquiring outright if tethers are permitted, and we wanted to say that we’re not trying to prescribe any solutions, because that’s not what the challenge is designed to do—we’re trying to be as open as possible. And it’s entirely possible that tethers will work in some cases, as they might have in tunnel, and it’s entirely possible that they won’t in other cases. If tethers showcase the ability to do additional breakthrough capabilities, then bring it! If there are ways in which breakthrough technologies allow us to work without tethers, and that opens up the spaces in which these types of systems can work, that’s another breakthrough that we’d be very excited to see.

Can you tell us what we can expect for the DARPA livestream of the Urban Circuit competition?

One of the things that we took away from the Tunnel Circuit was our desire to help narrate and explain and involve the viewing audience more. We intend to have both narration of the action and commentary where applicable. We’re opening the aperture because there are a lot of really interesting features that we’d love our viewing audience to be able to witness, and that includes robots traversing interesting features and elements within the course, so you can look forward to more of that.

You should be able to follow along with what’s going on in the course a fair bit better, and beyond what we saw in Tunnel for sure. We’re balancing of course the sensitivity of the competition, but there’s just so much robot excitement to be shared, and I’m excited that we’ll be able to do more of that for the Urban Circuit.

What are you personally most excited about for the Urban Circuit?

There’s just so many things— I’m super excited about how we set up the team garages at the Urban Circuit. It’ll be like pit row, in a way that really highlights how much I value the interactions between teams, it’ll be an opportunity to truly capitalize on having a high concentration of enthusiastic and ambitious roboticists in one area. 

And the teams are already amped up. They’ve gotten a taste of the difficulty level, they’ve gotten a taste of DARPA’s aspirational goals, they know when DARPA says “challenge” we mean “Challenge” with a capital C. From a technology perspective, I’m anticipating these teams are bringing their A++ game now, not only revising their strategy but building new robots, and that’ll be really exciting too.

[ DARPA SubT ]

With the DARPA Subterranean Challenge Urban Circuit kicking off on Thursday, we made sure to have a chat in advance with Dr. Timothy Chung, DARPA SubT program manager. We last spoke with Tim nearly a year ago, just after SubT was announced, to get his perspective on the Subterranean Challenge in general, and we took the opportunity in this interview to ask about how DARPA felt about the Tunnel Circuit, and what we have to look forward to in the Urban Circuit.

For more details about the SubT Urban Circuit, make sure to check out our course preview post, and check back tomorrow for a Q&A with the systems track teams.

This interview has been edited for length and clarity.

IEEE Spectrum: What turned out to be the biggest challenge for teams during the Tunnel Circuit?

Tim Chung: Where to begin? And I say that from the perspective of every obstacle being a good learning opportunity for the teams! For example, mobility was a huge challenge for the teams—going from cement and gravel into areas with rails that were problematic, to muddy places with big ruts… And that diversity was just on one level of the mine. But I think what we saw from the robots, at least from the ground vehicles, was the ability to deal with the unexpected, at least as far as the terrain was concerned.

As you saw, there were some different platforms that came up with different ways of solving the terrain challenges as well, including wheels on legs, and that was another great opportunity to see under what conditions wheels make sense, or legs make sense, or treads make sense. Again, that was only with the ground vehicles, and there were a lot of other ways to address the terrain, as with air vehicles, but they had their own sets of challenges. Mobility was a good reminder that the real world doesn’t have well-defined terrain parameters.

What impressed you the most during the Tunnel Circuit?

First off, I think the energy and enthusiasm that the teams brought. As a roboticist myself, I remember getting sad at times when my robot would break, but the fact that the teams were so enthusiastic about learning in this challenging environment was a really great takeaway. From a technology point of view, I was very impressed with how teams were able to deal with a lot of the uncertainty of the course itself, in terms of—I won’t call them hacks, but overnight fixes. The ability to say, “We saw that this other team had this innovation, let’s capitalize on that and see if we can retrofit our system to be able to do that as well.” 

“Increasing the autonomous capability of these robots can overcome the limitations of some other technology areas. An example of that is communications—we know communications are really hard. The teams definitely experienced this, and they were able to develop autonomous approaches to address or mitigate some of the communications deficiencies”

It turned out that both speed and agility are important, but so is deliberate decision making. I think what that speaks to is that increasing the autonomous capability of these robots can overcome the limitations of some other technology areas. An example of that is communications—we know communications are really hard. The teams definitely experienced this, and they were able to develop autonomous approaches to address or mitigate some of the communications deficiencies.

Can you share any feedback that you got from teams about the Tunnel Circuit?

The primary feedback that we’ve received has been very positive—teams have expressed that the courses were really, really hard, and hard in different ways than they anticipated. Essentially, that DARPA has set a very high bar. But teams are pleased that the bar is so high. I think what that says to me is that we’re satisfying the role that DARPA plays within the robotics community, of setting the bar really high while inspiring teams to reach it, but I also think that it shows how the community is thirsty for the opportunity to reach a new level of technology. From that perspective, the feedback was, “It was too hard for our robots, but something we still want to overcome.” And that’s great, because ultimately, DARPA’s role is to give the community a chance to surprise itself with how innovative it can be.

Did the results of the Tunnel Circuit cause you to recalibrate the difficulty of the Urban Circuit?

I can say pretty succinctly that the answer is no, in the sense that each of these environments are so wildly different that it’s difficult to say something like, “Tunnel was a difficulty 9, and Urban is going to be a 10.” And the type of challenge elements that we’re introducing in Urban that weren’t at Tunnel also make that comparison hard. The verticality with stairs, for example, will pose totally new challenges for the teams. So I’d say, we didn’t change the difficultly level because the environment just does it intrinsically for us.

In terms of expectations, we know that the teams have come to appreciate what DARPA wants to see out of the challenge, and so they’re better equipped from that perspective. We’ve adjusted by introducing new elements into the environment that will keep them on their toes for sure.

Can you highlight some of the fundamental differences between Tunnel environments and Urban environments?

The environments that are built for people tend not to have been designed with robots in mind, so having these types of systems—systems that can make an impact in everyday lives, and particularly in the first responder case we’re interested in—will necessarily mean that they’ll have to be able to operate in these challenging urban settings where we expect people to be.

Verticality is certainly one of the key distinctions that I think will be really exciting for the Urban Circuit. In Tunnel, the emphasis was not on verticality given the single level in the courses. What we’ve previewed for Urban is that there are sections of the courses where having the ability to navigate vertically will be very beneficial to teams.

Verticality includes not just the need to move in three dimensions, but the ability to perceive in three dimensions as well. For example, we place exit signs above eye level to help them stand out, and we look for indicators like handrails and signs. I think context plays a big role in these urban settings, which we take for granted as humans, but if you’re thinking about using these environmental cues to aid in localization and navigation, there may be some interesting advantages.

Are there other particular challenges for robots in urban environments?

I can give a couple of examples where there are some unique elements. Most of our construction materials tend to involve rebar, or metal in general, that are very bad for communications for sure but also for things like compasses. Another example is the regularity and the oftentimes featureless qualities of urban environments make it difficult for feature-matching technologies to help you with your localization. When one door looks like another door, or a wall looks like every other wall—humans may like that in our urban architecture, but for robots, the lack of discriminating features makes many localization approaches challenging. 

Why did you choose this specific urban environment, as opposed to something like a subway station?

We can certainly appreciate the subway scenario as one attribute of urban underground settings, but as we did research and worked with stakeholders, and also visited a number of various sites in urban centers, I think we learned that there’s just so much to the urban underground, so many types of environments, that to only use a subway wouldn’t have allowed us to implement many of the features that we were interested in for the SubT challenge. Going beyond that, I think the environment we found in Satsop is quite representative of many of the elements of the urban underground.

It’s also a place where we know first responders like the Seattle fire department conduct urban training. Being able to find a place that has value for first responders also contributed to our selection of the site, since it had that realistic feel for what they’re interested in and training for.

Can you talk a little bit about what DARPA’s position is on teams using tethers to communicate with their robots during the SubT Challenge?

[Laughs] What does DARPA have against tethers? I’d say, the idea here is that we’re interested in the capabilities that can be manifested throughout the challenge. While the specific implementation or approach may be of interest from a technology development perspective, it’s really about the capability. And so, if the best capability at present is to insert a robot that is tether-enabled, I think there’s merit there. The environment, however, is far more expansive than would allow for tethers to be the only solution. 

“If tethers showcase the ability to do additional breakthrough capabilities, then bring it! If there are ways in which breakthrough technologies allow us to work without tethers, and that opens up the spaces in which these types of systems can work, that’s another breakthrough that we’d be very excited to see”

Our perspective, which is to neither encourage nor discourage tethers, is partially in response to teams inquiring outright if tethers are permitted, and we wanted to say that we’re not trying to prescribe any solutions, because that’s not what the challenge is designed to do—we’re trying to be as open as possible. And it’s entirely possible that tethers will work in some cases, as they might have in tunnel, and it’s entirely possible that they won’t in other cases. If tethers showcase the ability to do additional breakthrough capabilities, then bring it! If there are ways in which breakthrough technologies allow us to work without tethers, and that opens up the spaces in which these types of systems can work, that’s another breakthrough that we’d be very excited to see.

Can you tell us what we can expect for the DARPA livestream of the Urban Circuit competition?

One of the things that we took away from the Tunnel Circuit was our desire to help narrate and explain and involve the viewing audience more. We intend to have both narration of the action and commentary where applicable. We’re opening the aperture because there are a lot of really interesting features that we’d love our viewing audience to be able to witness, and that includes robots traversing interesting features and elements within the course, so you can look forward to more of that.

You should be able to follow along with what’s going on in the course a fair bit better, and beyond what we saw in Tunnel for sure. We’re balancing of course the sensitivity of the competition, but there’s just so much robot excitement to be shared, and I’m excited that we’ll be able to do more of that for the Urban Circuit.

What are you personally most excited about for the Urban Circuit?

There’s just so many things— I’m super excited about how we set up the team garages at the Urban Circuit. It’ll be like pit row, in a way that really highlights how much I value the interactions between teams, it’ll be an opportunity to truly capitalize on having a high concentration of enthusiastic and ambitious roboticists in one area. 

And the teams are already amped up. They’ve gotten a taste of the difficulty level, they’ve gotten a taste of DARPA’s aspirational goals, they know when DARPA says “challenge” we mean “Challenge” with a capital C. From a technology perspective, I’m anticipating these teams are bringing their A++ game now, not only revising their strategy but building new robots, and that’ll be really exciting too.

[ DARPA SubT ]

Technologies change rapidly our perception of reality, moving from augmented to virtual to magical. While e-textiles are a key component in exergame or space suits, the transformative potential of the internal side of garments to create embodied experiences still remains largely unexplored. This paper is the result from an art-science collaborative project that combines recent neuroscience findings, body-centered design principles and 2D vibrotactile array-based fabrics to alter one's body perception. We describe an iterative design process intertwined with two user studies on the effects on body-perceptions and emotional responses of various vibration patterns within textile that were designed as spatial haptic metaphors. Our results show potential in considering materials (e.g., rocks) as sensations to design for body perceptions (e.g., being heavy, strong) and emotional responses. We discuss these results in terms of sensory effects on body perception and synergetic impact to research on embodiment in virtual environments, human-computer interaction, and e-textile design. The work brings a new perspective to the sensorial design of embodied experiences which is based on “material perception” and haptic metaphors, and highlights potential opportunities opened by haptic clothing to change body-perception.

The emergence and development of cognitive strategies for the transition from exploratory actions towards intentional problem-solving in children is a key question for the understanding of the development of human cognition. Researchers in developmental psychology have studied cognitive strategies and have highlighted the catalytic role of the social environment. However, it is not yet adequately understood how this capacity emerges and develops in biological systems when they perform a problem-solving task in collaboration with a robotic social agent. This paper presents an empirical study in a human-robot interaction (HRI) setting which investigates children's problem-solving from a developmental perspective. In order to theoretically conceptualize children's developmental process of problem-solving in HRI context, we use principles based on the intuitive theory and we take into consideration existing research on executive functions with a focus on inhibitory control. We considered the paradigm of the Tower of Hanoi and we conducted an HRI behavioral experiment to evaluate task performance. We designed two types of robot interventions, “voluntary” and “turn-taking”—manipulating exclusively the timing of the intervention. Our results indicate that the children who participated in the voluntary interaction setting showed a better performance in the problem solving activity during the evaluation session despite their large variability in the frequency of self-initiated interactions with the robot. Additionally, we present a detailed description of the problem-solving trajectory for a representative single case-study, which reveals specific developmental patterns in the context of the specific task. Implications and future work are discussed regarding the development of intelligent robotic systems that allow child-initiated interaction as well as targeted and not constant robot interventions.

This study examines the coiling and uncoiling motions of a soft pneumatic actuator inspired by the awn tissue of Erodium cicutarium. These tissues have embedded cellulose fibers distributed in a tilted helical pattern, which induces hygroscopic coiling and uncoiling in response to the daily changes in ambient humidity. Such sophisticated motions can eventually “drill” the seed at the tip of awn tissue into the soil: a drill bit in the plant kingdom. Through finite element simulation and experimental testing, this study examines a soft pneumatic actuator that has a similar reinforcing fiber layout to the Erodium plant tissue. This actuator, in essence, is a thin-walled elastomeric cylinder covered by tilted helical Kevlar fibers. Upon internal pressurization, it can exhibit a coiling motion by a combination of simultaneous twisting, bending, and extension. Parametric analyses show that the coiling motion characteristics are directly related to the geometry of tilted helical fibers. Notably, a moderate tilt in the reinforcing helical fiber leads to many coils of small radius, while a significant tilt gives fewer coils of larger radius. The results of this study can offer guidelines for constructing plant-inspired robotic manipulators that can achieve complicated motions with simple designs.

The Urban Circuit of the DARPA Subterranean Challenge is the second of four robotics competitions that send teams of state-of-the-art robots into challenging underground environments in an attempt to seek out artifacts while creating detailed maps. Last August, the robots explored a man-made tunnel system in the NIOSH research mine near Pittsburgh, Pennsylvania. And starting this Thursday, the teams will be taking on the urban underground, at Satsop Business Park in Elma, Wash.

If you’re not familiar with the DARPA Subterranean Challenge, here’s an overview video from DARPA to get you caught up on the whole thing:

This post will be focused on a preview of the Urban Circuit, but if you’d like to learn more about SubT we’ve got plenty more coverage on the challenge itself, the teams that are involved, and the results of the first competition, the Tunnel Circuit.

The overall objective of the SubT Challenge is for each team of robots to spend 60 minutes exploring an underground (or simulated underground) course, searching for a variety of artifacts. You can read more about how the scoring works here, but the team of robots that’s able to find the most artifacts (and report back their exact location) in the least amount of time wins.

DARPA’s intention is that the Urban Circuit “will represent human-made urban environments such as municipal infrastructure and mass transit.” We’d sort of figured that they’d choose a subway station, mostly because they’ve been using subway station graphics on the SubT website. But DARPA has chosen Satsop Business Park (just to the west of Olympia, Wash.) as the location for the event, and it’s much more of an industrial-y looking place. Here’s an environment preview video:

Environments like these can be pretty grim for autonomous robots. There’s all kinds of stuff on the floor, lots of localization-unfriendly flat surfaces, dirt, water, stairs, ledges—it sucks if you’re a robot (or roboticist). But that’s the point, because the SubT challenge is DARPA-hard. That’s what the teams and their robots are preparing for, and I can’t imagine they’d want it any other way.

The SubT challenge is DARPA-hard. That’s what the teams and their robots are preparing for, and I can’t imagine they’d want it any other way

Something in particular to look out for in the Urban Circuit is verticality: Things like stairs, ladders, ledges, and holes in the ground and ceiling that need to be explored or traversed. The Tunnel Circuit didn’t really have that; there may have been one or two optional holes, but nothing that robots were required to deal with in order to make progress. We’re expecting that verticality will be a much more significant (i.e. mandatory) part of the Urban Circuit, which will definitely add some risk. And some drama!

A key thing to remember is that all teams are allowed to do is push their robots through the entry to the Urban Circuit course, and after that, they can’t touch them. Once the robots move out of communications range, they’re completely on their own, operating fully autonomously without any human input at all.

During the Tunnel Circuit, teams tried a variety of very creative ways to keep in touch with their robots, both to give them high-level instructions and to get back artifact coordinates to score points. We’re expecting even more autonomy for the Urban Circuit, as teams have refined their communications strategies over the past six months.

New to the Urban Circuit are two different artifacts: gas, and vent. They replace the drill and fire extinguisher artifacts from the Tunnel Circuit. Here’s what we know about the new artifacts:

The gas artifact is a CO2-emitting device used to simulate a range of hazardous air quality conditions, such as a gas leak, poor ventilation, or fumes and smoke. Finding this artifact represents identifying areas that would be hazardous for personnel, including areas where breathing apparatus may be necessary.

The vent artifact is a typical supply register commonly found in homes or work environments. Finding this artifact represents identifying potential areas with fresh air or an escape route to the surface.

Image: DARPA

The vent artifact can be found on a wall or ceiling at any height, and it’ll be heated to at least 30 °C above ambient temperature. The gas artifact is a little trickier; the gas itself will be kept at 2000 ppm in a room with an open door. There won’t be any visual indicators; robots will have to sense the gas, and precisely identify the room that it’s coming from by finding the center point of the entry door threshold (at floor level) of the room where the gas is detected.

Besides the new course and new artifacts, DARPA has made some updates to the SubT rules for the Urban Circuit. We’ve gone through and highlighted some of the most relevant ones:

The scale and complexity of competition courses is expected to vary across events and may vary from run to run due to configuration changes. The design of the Urban Circuit course is intended to assess the ability of teams to address challenging urban environments that include multiple levels, degraded terrain, and austere environmental conditions.

Each of the two courses are expected to include multiple levels, requiring teams to traverse stairs, ramps, or vertical shafts to access the entire competition course. Stairs are expected to be a challenge element on both courses. The height difference between levels requires teams to traverse multiple flights of stairs to reach a different level.

Did we mention that the Urban Circuit will include multiple levels? Because there are going to be multiple levels. 

Vertical shafts and openings vary by course but may be as large as 2.5 m x 2.5 m. Each level may also include mezzanine levels. Artifacts may be located on mezzanine levels and in some cases may require elevated vantage points for line-of-site detection. It is expected that some areas could include significant drop-offs. Other areas include curbs that typically range from 0.1 to 0.2 meters high.

Interesting that there will be some artifacts that robots will have to spot without necessarily being able to get particularly close to them. And while curbs are good, the “significant drop-offs” is a bit worrisome. 

The width of passages at the Urban Circuit site vary greatly and include large open areas as well as narrow constrained passages common in urban environments (i.e., doorways). It is expected that some portions of the course will only be accessible via passages that are approximately one meter in height and/or one meter in width. For the Urban Circuit, it is expected that up to 50 percent of the competition course could be inaccessible for systems that cannot traverse these passages.

While a majority of passages at the Urban Circuit site are greater than 1 m x 1 m, some passages are as narrow as 0.8 meters and a limited number of passages are as narrow as 0.7 meters. Some artifacts (up to 10 percent) may be inaccessible without traversing the more constrained passages. 

Chubby robots are on notice!

Dispute Cards are intended to provide teams a mechanism to submit a formal dispute or request for review by the Chief Official. The Dispute Card must be completed and delivered by the Team Lead to the relevant Course Official, Team Garage Coordinator, or Chief Official. The Dispute Card must be submitted within 30 minutes of the completion of the run in question. All submissions will be reviewed by the Chief Official in a timely manner. All decisions made by the Chief Official are final.

This is a new mechanism for the Urban Circuit. We don’t know whether there was some specific incident that prompted this, or whether DARPA simply decided that it would be a good system to have in place. 

In the event that multiple teams have an identical score, tiebreakers will be applied in the following order until the tie is broken:
- Earliest time that the last artifact was successfully reported, averaged across the team’s best runs on each course
- Earliest time that the first artifact was successfully reported, averaged across the team’s best runs on each course
- Furthest distance traveled by any deployed system 

That last one (furthest distance traveled by any deployed system) is a new addition. I’d be surprised if a tiebreaker made it that far, but it does emphasize DARPA’s interest in exploration.

Teams are encouraged to provide audible and visual recovery aids such as flashing LEDs and audible cues to help Competition Staff locate deployed systems or components.

Many robots drop things like wireless repeaters as they progress through the course—DARPA was explicit that teams should not necessarily expect them all to be returned, but the agency may have gotten a little bit fed up attempting to track down all the robot droppings after every run.

There are currently no restrictions on the use of tethers for power, communications, or physical retrieval. However, teams are encouraged to consider the significant limitations imposed by the large-scale, potentially dynamic, and complex environments of interest.

This is more of a reiteration from DARPA than a new rule, but it’s worth noting that it’s being emphasized again for the Urban Circuit. Several teams who did use tethers for the Tunnel Circuit seemed to derive a significant advantage from them, and perhaps DARPA is concerned that more teams will use that approach. Which, of course, they can, it’s just that tethers can also be very limiting, and it seems like DARPA would really like to see more versatile, creative solutions.

Since the Urban Circuit is a closed course, the best way to follow along with the event will be through DARPA’s livestream, which will be on a one hour time delay (boo!) but may also have commentary this time around (yay!). Also, you can hear directly from the teams themselves by following #SubTChallenge on Twitter

Scored runs (the thing that’ll be the most fun to watch) will take place February 20 (Thursday) to February 22 (Saturday), and also February 24 (Monday) to February 26 (Wednesday). There’s a media day on the 24 that we’ll be at, so make sure and let us know if there are specific things you want us to check out.

[ DARPA Subterranean Challenge ]

The Urban Circuit of the DARPA Subterranean Challenge is the second of four robotics competitions that send teams of state-of-the-art robots into challenging underground environments in an attempt to seek out artifacts while creating detailed maps. Last August, the robots explored a man-made tunnel system in the NIOSH research mine near Pittsburgh, Pennsylvania. And starting this Thursday, the teams will be taking on the urban underground, at Satsop Business Park in Elma, Wash.

If you’re not familiar with the DARPA Subterranean Challenge, here’s an overview video from DARPA to get you caught up on the whole thing:

This post will be focused on a preview of the Urban Circuit, but if you’d like to learn more about SubT we’ve got plenty more coverage on the challenge itself, the teams that are involved, and the results of the first competition, the Tunnel Circuit.

The overall objective of the SubT Challenge is for each team of robots to spend 60 minutes exploring an underground (or simulated underground) course, searching for a variety of artifacts. You can read more about how the scoring works here, but the team of robots that’s able to find the most artifacts (and report back their exact location) in the least amount of time wins.

DARPA’s intention is that the Urban Circuit “will represent human-made urban environments such as municipal infrastructure and mass transit.” We’d sort of figured that they’d choose a subway station, mostly because they’ve been using subway station graphics on the SubT website. But DARPA has chosen Satsop Business Park (just to the west of Olympia, Wash.) as the location for the event, and it’s much more of an industrial-y looking place. Here’s an environment preview video:

Environments like these can be pretty grim for autonomous robots. There’s all kinds of stuff on the floor, lots of localization-unfriendly flat surfaces, dirt, water, stairs, ledges—it sucks if you’re a robot (or roboticist). But that’s the point, because the SubT challenge is DARPA-hard. That’s what the teams and their robots are preparing for, and I can’t imagine they’d want it any other way.

The SubT challenge is DARPA-hard. That’s what the teams and their robots are preparing for, and I can’t imagine they’d want it any other way

Something in particular to look out for in the Urban Circuit is verticality: Things like stairs, ladders, ledges, and holes in the ground and ceiling that need to be explored or traversed. The Tunnel Circuit didn’t really have that; there may have been one or two optional holes, but nothing that robots were required to deal with in order to make progress. We’re expecting that verticality will be a much more significant (i.e. mandatory) part of the Urban Circuit, which will definitely add some risk. And some drama!

A key thing to remember is that all teams are allowed to do is push their robots through the entry to the Urban Circuit course, and after that, they can’t touch them. Once the robots move out of communications range, they’re completely on their own, operating fully autonomously without any human input at all.

During the Tunnel Circuit, teams tried a variety of very creative ways to keep in touch with their robots, both to give them high-level instructions and to get back artifact coordinates to score points. We’re expecting even more autonomy for the Urban Circuit, as teams have refined their communications strategies over the past six months.

New to the Urban Circuit are two different artifacts: gas, and vent. They replace the drill and fire extinguisher artifacts from the Tunnel Circuit. Here’s what we know about the new artifacts:

The gas artifact is a CO2-emitting device used to simulate a range of hazardous air quality conditions, such as a gas leak, poor ventilation, or fumes and smoke. Finding this artifact represents identifying areas that would be hazardous for personnel, including areas where breathing apparatus may be necessary.

The vent artifact is a typical supply register commonly found in homes or work environments. Finding this artifact represents identifying potential areas with fresh air or an escape route to the surface.

Image: DARPA

The vent artifact can be found on a wall or ceiling at any height, and it’ll be heated to at least 30 °C above ambient temperature. The gas artifact is a little trickier; the gas itself will be kept at 2000 ppm in a room with an open door. There won’t be any visual indicators; robots will have to sense the gas, and precisely identify the room that it’s coming from by finding the center point of the entry door threshold (at floor level) of the room where the gas is detected.

Besides the new course and new artifacts, DARPA has made some updates to the SubT rules for the Urban Circuit. We’ve gone through and highlighted some of the most relevant ones:

The scale and complexity of competition courses is expected to vary across events and may vary from run to run due to configuration changes. The design of the Urban Circuit course is intended to assess the ability of teams to address challenging urban environments that include multiple levels, degraded terrain, and austere environmental conditions.

Each of the two courses are expected to include multiple levels, requiring teams to traverse stairs, ramps, or vertical shafts to access the entire competition course. Stairs are expected to be a challenge element on both courses. The height difference between levels requires teams to traverse multiple flights of stairs to reach a different level.

Did we mention that the Urban Circuit will include multiple levels? Because there are going to be multiple levels. 

Vertical shafts and openings vary by course but may be as large as 2.5 m x 2.5 m. Each level may also include mezzanine levels. Artifacts may be located on mezzanine levels and in some cases may require elevated vantage points for line-of-site detection. It is expected that some areas could include significant drop-offs. Other areas include curbs that typically range from 0.1 to 0.2 meters high.

Interesting that there will be some artifacts that robots will have to spot without necessarily being able to get particularly close to them. And while curbs are good, the “significant drop-offs” is a bit worrisome. 

The width of passages at the Urban Circuit site vary greatly and include large open areas as well as narrow constrained passages common in urban environments (i.e., doorways). It is expected that some portions of the course will only be accessible via passages that are approximately one meter in height and/or one meter in width. For the Urban Circuit, it is expected that up to 50 percent of the competition course could be inaccessible for systems that cannot traverse these passages.

While a majority of passages at the Urban Circuit site are greater than 1 m x 1 m, some passages are as narrow as 0.8 meters and a limited number of passages are as narrow as 0.7 meters. Some artifacts (up to 10 percent) may be inaccessible without traversing the more constrained passages. 

Chubby robots are on notice!

Dispute Cards are intended to provide teams a mechanism to submit a formal dispute or request for review by the Chief Official. The Dispute Card must be completed and delivered by the Team Lead to the relevant Course Official, Team Garage Coordinator, or Chief Official. The Dispute Card must be submitted within 30 minutes of the completion of the run in question. All submissions will be reviewed by the Chief Official in a timely manner. All decisions made by the Chief Official are final.

This is a new mechanism for the Urban Circuit. We don’t know whether there was some specific incident that prompted this, or whether DARPA simply decided that it would be a good system to have in place. 

In the event that multiple teams have an identical score, tiebreakers will be applied in the following order until the tie is broken:
- Earliest time that the last artifact was successfully reported, averaged across the team’s best runs on each course
- Earliest time that the first artifact was successfully reported, averaged across the team’s best runs on each course
- Furthest distance traveled by any deployed system 

That last one (furthest distance traveled by any deployed system) is a new addition. I’d be surprised if a tiebreaker made it that far, but it does emphasize DARPA’s interest in exploration.

Teams are encouraged to provide audible and visual recovery aids such as flashing LEDs and audible cues to help Competition Staff locate deployed systems or components.

Many robots drop things like wireless repeaters as they progress through the course—DARPA was explicit that teams should not necessarily expect them all to be returned, but the agency may have gotten a little bit fed up attempting to track down all the robot droppings after every run.

There are currently no restrictions on the use of tethers for power, communications, or physical retrieval. However, teams are encouraged to consider the significant limitations imposed by the large-scale, potentially dynamic, and complex environments of interest.

This is more of a reiteration from DARPA than a new rule, but it’s worth noting that it’s being emphasized again for the Urban Circuit. Several teams who did use tethers for the Tunnel Circuit seemed to derive a significant advantage from them, and perhaps DARPA is concerned that more teams will use that approach. Which, of course, they can, it’s just that tethers can also be very limiting, and it seems like DARPA would really like to see more versatile, creative solutions.

Since the Urban Circuit is a closed course, the best way to follow along with the event will be through DARPA’s livestream, which will be on a one hour time delay (boo!) but may also have commentary this time around (yay!). Also, you can hear directly from the teams themselves by following #SubTChallenge on Twitter

Scored runs (the thing that’ll be the most fun to watch) will take place February 20 (Thursday) to February 22 (Saturday), and also February 24 (Monday) to February 26 (Wednesday). There’s a media day on the 24 that we’ll be at, so make sure and let us know if there are specific things you want us to check out.

[ DARPA Subterranean Challenge ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

Start your robots! Next week DARPA will kick off the Urban Circuit competition, the second of four robotics competitions that are part of its Subterranean Challenge.

We’re going to have a Urban Circuit preview post with all the details on Monday, a Q&A with DARPA SubT program manager Tim Chung on Tuesday, a follow-up with the teams on their preparations on Wednesday, and a post on how to watch the livestream on Thursday.

For now, watch Team Explorer from CMU and Oregon State testing some of their robots ahead of the competition.

In stage two of DARPA’s Subterranean Challenge, a team from Carnegie Mellon University and Oregon State will send robots into the depths of an incomplete nuclear power plant in a search-and-rescue scenario in Elma, Washington. Team Explorer’s machines will scale stairs and search for artifacts in the "Urban Circuit."

[ Team Explorer ]

We could watch these expandifying objects all day.

ExpandFab is a fabrication method for creating expanding objects using foam materials. The printed objects change their shape and volume, which is advantageous for reducing the printing time and transportation costs. For the fabrication of expanding objects, we investigated a basic principle of the expansion rate and developed materials by mixing a foam powder and elastic adhesive. Furthermore, we developed a fabrication method using the foam materials. A user can design expanded objects using our design software and sets the expansion areas on the surface. The software simulates and exports the 3d model into a three-dimensional (3D) printer. The 3D printer prints the expandable object by curing with ultraviolet light. Finally, the user heats the printed objects, and the objects expand to maximum approximately 2.7 times of their original size. ExpandFab allows users to prototype products that expand and morph into various shapes, such as objects changing from one shape to various shapes, and functional prototype with electronic components. In this paper, we describe the basic principle of this technique, implementation of the software and hardware, application examples, limitations and discussions, and future works.

[ Yasuaki Kakehi Laboratory ]

This new robot vacuum from Panasonic can prop itself up to drive over thick rugs and even go over small steps and bumps up to 2.5 centimeters high.

Apparently it does SLAM, though it’s not clear what kind of sensor it’s using. And at 1:05, is that a “follow me” feature?

[ Panasonic ] via [ ImpressWatch ]

Cybathlon needs you!

Robert Riener from ETH Zurich tells us that the Cybathlon organizers “still need many volunteers” to help with multiple phases of the event. Learn more about it here.

From 2nd to 3rd ETH Zurich’s CYBATHLON 2020 takes place in May. The CYBATHLON is a unique competition in which people with disabilities measure themselves when completing everyday tasks using the latest technical assistance systems.

Greet and look after international teams in the SWISS Arena in Kloten and help with the competition or with the assembly and dismantling! Are you in?

Register now as a volunteer http://www.cybathlon.com/volunteers

[ CYBATHLON ]

Happy Valentine’s Day from Robotiq!

[ Robotiq ]

In case you missed our story yesterday, Iranian researchers at the University of Tehran have unveiled a new humanoid robot called Surena IV.

[ IEEE Spectrum ]

Great, first those self-healing liquid metal robot tendons, and now this. How soon until T-1000? 

Researchers at Carnegie Mellon University and the UT Dallas have introduced a soft, multifunctional composite that remains electrically conductive when stretched and exhibits a number of other desirable properties for soft robotics and stretchable electronics.

[ CMU ]

Hey, it’s not just BotJunkie who likes to hug robots!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people - connecting robots to humans and building relationships that last

[ Roboy ]

In the Dominican Republic, this initiative is using DJI drones to deliver supplies to rural communities.

Traditional methods of delivering medicine to rural communities have not been considered the most efficient solutions. Patients in smaller areas of the Dominican Republic, for example, would often go weeks without receiving the care they needed, increasing mortality rates. A reliable and cost-efficient solution became necessary. Thankfully, drone technology would answer the call. Watch how powerful equipment like the Matrice 600, and a strong collaboration between the local medical staff, Ministry of Health, WeRobotics and the Drone Innovation Center, has led to increased efficiency during important medical deliveries.

DJI ]

We’ve already seen some robots helping to fight the coronavirus outbreak. Here are some more.

This one is used for deliveries of food and medication:

[ New China TV ]

“The BOT-chelor” LOL

Who would you pick?

[ Sphero ]

Impressive demo of a real-time SLAM technique developed by SLAMcore, a startup spun out of Imperial College London.

This video was created in real-time running on the CPU of the Jetson TX2. The final map is sub cm accurate and only a few megabytes in size.

[ SLAMcore ]

We heart coffee indeed.

Kawasaki's coffee cobot, duAro, operates an espresso machine, and uses precise movements to pour steamed milk in the shape of a heart.

[ Kawasaki Robotics ]

How do you grasp hollow, deformable objects? Researchers at the Technical University of Munich are on the case.

[ Jingyi Xu ]

Some robots can use the same type of limb to walk and swim. But it would be even better if the robot’s limbs could change their shape to better adapt to different environments. Yale researchers are working on a possible implementation of this idea, inspired by sea turtles.

Most robots operate either exclusively on land or in water. Toward building an amphibious legged robot, we present a morphing limb that can adapt its structure and stiffness for amphibious operation. We draw inspiration for the limb’s design from the morphologies of sea turtle flippers and land-faring tortoise legs. Turtles and tortoises have rigid hulls that can be emulated in amphibious robots to provide a convenient, protected volume for motors, electronics, power supply, and payloads. Each of these animals’ limbs are tailored for locomotion in their respective environments. A sea turtle flipper has a streamlined profile to reduce drag, making it apt for swimming. A land tortoise leg boasts a strong, expanded cross-section conducive to load-bearing. We capture the morphological advantages of both animals’ limbs in our morphing limb via a variable stiffness composite coupled to a pneumatic actuator system that enables on-demand transitions between leg and flipper configurations. We control the degree of stiffness of the limb by varying electrical input to flexible heaters bound to the thermally responsive variable stiffness composite. The proposed morphing amphibious limb design is promising for enabling the next generation of hybrid soft-rigid robots to adapt to unstructured environments.

[ Yale Faboratory ]

Sorting recyclable waste: A job that we should definitely let robots steal from humans.

Recyclable waste sorting is mostly performed by manual labor in dull, dirty and dangerous environments. There are less and less people willing to perform these tasks and more and more recycling to be processed. A task like this is ideal for automation, but the challenge of recognizing and extracting random looking objects in a random stream of waste is very difficult. Newly available Artificial intelligence (AI) now enables these complex applications.

Current systems have limited recognition and extraction technologies that limit the quality of sorted material. Waste Robotics has developed a computer vision system that enables real-time recognition of objects in a random stream while dispatching robot commands to perform efficient and high-quality sorting to enable a circular economy.

[ FANUC ]

We aren’t sure why a flying robot is the best way of doing this task but the drone hitting the target with the rope at 0:20 is pretty cool.

This movie shows the successful test of a TugDrone, developed by Delft Dynamics and KOTUG. The drone delivers the ’heaving’ line from the the tugboat to a vessel. This improves safety on board both the vessel and the tugboat. This innovation supports the credo of KOTUG International: ’Ahead in Towage’.

[ Delft Dynamics ]

We were in Rwanda and Tanzania last year to see how drones are helping to deliver blood and medicine. See our full coverage and 3D videos here.

Three amazing days of #ADF2020 in Kigali has just wrapped up. See the highlights from the first African Drone Forum.

[ African Drone Forum ]

The IEEE Robotics and Automation Society continues to post more conference keynotes to its YouTube channel. This one is from ICRA 2018 by Louis Whitcomb from Johns Hopkins University on “Extreme Robotics: Underwater Robotic Exploration of the Karasik Seamount.”

[ IEEE RAS ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

Start your robots! Next week DARPA will kick off the Urban Circuit competition, the second of four robotics competitions that are part of its Subterranean Challenge.

We’re going to have a Urban Circuit preview post with all the details on Monday, a Q&A with DARPA SubT program manager Tim Chung on Tuesday, a follow-up with the teams on their preparations on Wednesday, and a post on how to watch the livestream on Thursday.

For now, watch Team Explorer from CMU and Oregon State testing some of their robots ahead of the competition.

In stage two of DARPA’s Subterranean Challenge, a team from Carnegie Mellon University and Oregon State will send robots into the depths of an incomplete nuclear power plant in a search-and-rescue scenario in Elma, Washington. Team Explorer’s machines will scale stairs and search for artifacts in the "Urban Circuit."

[ Team Explorer ]

We could watch these expandifying objects all day.

ExpandFab is a fabrication method for creating expanding objects using foam materials. The printed objects change their shape and volume, which is advantageous for reducing the printing time and transportation costs. For the fabrication of expanding objects, we investigated a basic principle of the expansion rate and developed materials by mixing a foam powder and elastic adhesive. Furthermore, we developed a fabrication method using the foam materials. A user can design expanded objects using our design software and sets the expansion areas on the surface. The software simulates and exports the 3d model into a three-dimensional (3D) printer. The 3D printer prints the expandable object by curing with ultraviolet light. Finally, the user heats the printed objects, and the objects expand to maximum approximately 2.7 times of their original size. ExpandFab allows users to prototype products that expand and morph into various shapes, such as objects changing from one shape to various shapes, and functional prototype with electronic components. In this paper, we describe the basic principle of this technique, implementation of the software and hardware, application examples, limitations and discussions, and future works.

[ Yasuaki Kakehi Laboratory ]

This new robot vacuum from Panasonic can prop itself up to drive over thick rugs and even go over small steps and bumps up to 2.5 centimeters high.

Apparently it does SLAM, though it’s not clear what kind of sensor it’s using. And at 1:05, is that a “follow me” feature?

[ Panasonic ] via [ ImpressWatch ]

Cybathlon needs you!

Robert Riener from ETH Zurich tells us that the Cybathlon organizers “still need many volunteers” to help with multiple phases of the event. Learn more about it here.

From 2nd to 3rd ETH Zurich’s CYBATHLON 2020 takes place in May. The CYBATHLON is a unique competition in which people with disabilities measure themselves when completing everyday tasks using the latest technical assistance systems.

Greet and look after international teams in the SWISS Arena in Kloten and help with the competition or with the assembly and dismantling! Are you in?

Register now as a volunteer http://www.cybathlon.com/volunteers

[ CYBATHLON ]

Happy Valentine’s Day from Robotiq!

[ Robotiq ]

In case you missed our story yesterday, Iranian researchers at the University of Tehran have unveiled a new humanoid robot called Surena IV.

[ IEEE Spectrum ]

Great, first those self-healing liquid metal robot tendons, and now this. How soon until T-1000? 

Researchers at Carnegie Mellon University and the UT Dallas have introduced a soft, multifunctional composite that remains electrically conductive when stretched and exhibits a number of other desirable properties for soft robotics and stretchable electronics.

[ CMU ]

Hey, it’s not just BotJunkie who likes to hug robots!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people - connecting robots to humans and building relationships that last

[ Roboy ]

In the Dominican Republic, this initiative is using DJI drones to deliver supplies to rural communities.

Traditional methods of delivering medicine to rural communities have not been considered the most efficient solutions. Patients in smaller areas of the Dominican Republic, for example, would often go weeks without receiving the care they needed, increasing mortality rates. A reliable and cost-efficient solution became necessary. Thankfully, drone technology would answer the call. Watch how powerful equipment like the Matrice 600, and a strong collaboration between the local medical staff, Ministry of Health, WeRobotics and the Drone Innovation Center, has led to increased efficiency during important medical deliveries.

DJI ]

We’ve already seen some robots helping to fight the coronavirus outbreak. Here are some more.

This one is used for deliveries of food and medication:

[ New China TV ]

“The BOT-chelor” LOL

Who would you pick?

[ Sphero ]

Impressive demo of a real-time SLAM technique developed by SLAMcore, a startup spun out of Imperial College London.

This video was created in real-time running on the CPU of the Jetson TX2. The final map is sub cm accurate and only a few megabytes in size.

[ SLAMcore ]

We heart coffee indeed.

Kawasaki's coffee cobot, duAro, operates an espresso machine, and uses precise movements to pour steamed milk in the shape of a heart.

[ Kawasaki Robotics ]

How do you grasp hollow, deformable objects? Researchers at the Technical University of Munich are on the case.

[ Jingyi Xu ]

Some robots can use the same type of limb to walk and swim. But it would be even better if the robot’s limbs could change their shape to better adapt to different environments. Yale researchers are working on a possible implementation of this idea, inspired by sea turtles.

Most robots operate either exclusively on land or in water. Toward building an amphibious legged robot, we present a morphing limb that can adapt its structure and stiffness for amphibious operation. We draw inspiration for the limb’s design from the morphologies of sea turtle flippers and land-faring tortoise legs. Turtles and tortoises have rigid hulls that can be emulated in amphibious robots to provide a convenient, protected volume for motors, electronics, power supply, and payloads. Each of these animals’ limbs are tailored for locomotion in their respective environments. A sea turtle flipper has a streamlined profile to reduce drag, making it apt for swimming. A land tortoise leg boasts a strong, expanded cross-section conducive to load-bearing. We capture the morphological advantages of both animals’ limbs in our morphing limb via a variable stiffness composite coupled to a pneumatic actuator system that enables on-demand transitions between leg and flipper configurations. We control the degree of stiffness of the limb by varying electrical input to flexible heaters bound to the thermally responsive variable stiffness composite. The proposed morphing amphibious limb design is promising for enabling the next generation of hybrid soft-rigid robots to adapt to unstructured environments.

[ Yale Faboratory ]

Sorting recyclable waste: A job that we should definitely let robots steal from humans.

Recyclable waste sorting is mostly performed by manual labor in dull, dirty and dangerous environments. There are less and less people willing to perform these tasks and more and more recycling to be processed. A task like this is ideal for automation, but the challenge of recognizing and extracting random looking objects in a random stream of waste is very difficult. Newly available Artificial intelligence (AI) now enables these complex applications.

Current systems have limited recognition and extraction technologies that limit the quality of sorted material. Waste Robotics has developed a computer vision system that enables real-time recognition of objects in a random stream while dispatching robot commands to perform efficient and high-quality sorting to enable a circular economy.

[ FANUC ]

We aren’t sure why a flying robot is the best way of doing this task but the drone hitting the target with the rope at 0:20 is pretty cool.

This movie shows the successful test of a TugDrone, developed by Delft Dynamics and KOTUG. The drone delivers the ’heaving’ line from the the tugboat to a vessel. This improves safety on board both the vessel and the tugboat. This innovation supports the credo of KOTUG International: ’Ahead in Towage’.

[ Delft Dynamics ]

We were in Rwanda and Tanzania last year to see how drones are helping to deliver blood and medicine. See our full coverage and 3D videos here.

Three amazing days of #ADF2020 in Kigali has just wrapped up. See the highlights from the first African Drone Forum.

[ African Drone Forum ]

The IEEE Robotics and Automation Society continues to post more conference keynotes to its YouTube channel. This one is from ICRA 2018 by Louis Whitcomb from Johns Hopkins University on “Extreme Robotics: Underwater Robotic Exploration of the Karasik Seamount.”

[ IEEE RAS ]

A little over a decade ago, researchers at the University of Tehran introduced a rudimentary humanoid robot called Surena. An improved model capable of walking, Surena II, was announced not long after, followed by the more capable Surena III in 2015.

Now the Iranian roboticists have unveiled Surena IV. The new robot is a major improvement over previous designs. A video highlighting its capabilities shows the robot mimicking a person’s pose, grasping a water bottle, and writing its name on a whiteboard.

Surena is also shown taking a group selfie with its human pals.

Who developed Surena?

A team of more than 50 researchers built Surena IV at the University of Tehran’s Center for Advanced Systems and Technologies (CAST). Led by Dr. Aghil Yousefi-Koma, a professor of mechanical engineering, the group worked for the past four years to develop the new robot.

“Improving the robot’s interaction with the environment was one of the main goals of the Surena IV project,” Professor Yousefi-Koma tells IEEE Spectrum. He explains that the robot can now track objects more accurately, and new hands gave it better dexterity. These enhancements allow it to manipulate a wide variety of objects, including delicate ones, as well as operate power tools.

Surena IV is also more nimble. Custom force sensors on the bottom of its feet help the robot step over uneven surfaces by adjusting the angle and position of each foot. Walking speed increased to 0.7 kilometers per hour from 0.3 km/h in the previous generation.

While Surena IV is not a highly dynamic machine like Atlas, the famously agile humanoid from Boston Dynamics, it relies on a whole-body motion controller that continuously adjusts its posture to avoid falls. In its overall design, Surena is probably more comparable to humanoids like UBTECH’s Walker, Honda’s Asimo, Pal Robotics’ Talos, KAIST’s Hubo, and AIST’s HRP-2, although these robots have already publicly demonstrated a broader range of capabilities.

Photo: University of Tehran/CAST Surena IV is equipped with new hands that allow it to grasp a wider variety of objects.

To make his robots more widely known, Professor Yousefi-Koma has sought to take them to international conferences and trade shows, and his group has published papers on humanoid robot design, bipedal locomotion, and other topics. He has said that his group wants to “make [Surena] very competitive and may be taking part in an international challenge as well as finding real applications and users.”

In footage released as part of Surena IV’s unveiling, which happened late last year, the humanoid is shown lifting a box, inside of which is Surena Mini, a knee-high robot that the CAST team developed in 2017.

Another video gives a behind-the-scenes look at Surena’s development, showing the researchers testing new hardware, writing code, and assisting the robot as it takes its first steps.

The evolution of Surena

The latest Surena model has come a long way since the initial version, unveiled in 2008, which had only 8 degrees of freedom (DoF) and used wheels on its feet to move around. Surena II had 22 DoF and was the first version capable of walking. Surena III, with 31 DoF, demonstrated skills like climbing steps and kicking a ball.

Surena IV has even more axes of motion, with additional 12 DoF, most on the hands, bringing the total to 43 DoF. The CAST team redesigned several components, developing new lightweight structural elements and small yet powerful custom actuators. As a result, the new robot—at 1.7 meters tall and 68 kilograms—is lighter and more compact than Surena III (1.9 m and 98 kg).

Photo: University of Tehran/CAST Led by Dr. Aghil Yousefi-Koma [standing to the left of the robot], a team of more than 50 researchers developed Surena IV at the University of Tehran’s Center for Advanced Systems and Technologies (CAST). During an event late last year, part of the group posed with Surena for a photo.

Another advance is the robot’s controller. The control loop now operates at a higher frequency (200 Hz), thanks to newly added FPGA boards. Sensors include stereo cameras, 6-axis force/torque sensors in the ankles, encoders on all joints, and an inertial measurement unit (IMU). A text-to-speech system allows the robot to recognize and generate basic speech.

To make all the sensors, controllers, and actuators work together, the researchers equipped Surena with the Robot Operating System, or ROS. They also used Gazebo, Choreonoid, and MATLAB to simulate the robot’s motions and evaluate different behaviors, including walking backwards and sideways, turning around, and push recovery.

But the upgrades are not just internal. Surena IV’s exterior features new plastic covers that make the robot look sleek—and only slightly menacing. Professor Yousefi-Koma has emphasized that he views Surena as a “symbol of technology advancement in the direction of peace and humanity,” hoping it will help inspire people about the possibilities of robotics.

A little over a decade ago, researchers at the University of Tehran introduced a rudimentary humanoid robot called Surena. An improved model capable of walking, Surena II, was announced not long after, followed by the more capable Surena III in 2015.

Now the Iranian roboticists have unveiled Surena IV. The new robot is a major improvement over previous designs. A video highlighting its capabilities shows the robot mimicking a person’s pose, grasping a water bottle, and writing its name on a whiteboard.

Surena is also shown taking a group selfie with its human pals.

Who developed Surena?

A team of more than 50 researchers built Surena IV at the University of Tehran’s Center for Advanced Systems and Technologies (CAST). Led by Dr. Aghil Yousefi-Koma, a professor of mechanical engineering, the group worked for the past four years to develop the new robot.

“Improving the robot’s interaction with the environment was one of the main goals of the Surena IV project,” Professor Yousefi-Koma tells IEEE Spectrum. He explains that the robot can now track objects more accurately, and new hands gave it better dexterity. These enhancements allow it to manipulate a wide variety of objects, including delicate ones, as well as operate power tools.

Surena IV is also more nimble. Custom force sensors on the bottom of its feet help the robot step over uneven surfaces by adjusting the angle and position of each foot. Walking speed increased to 0.7 kilometers per hour from 0.3 km/h in the previous generation.

While Surena IV is not a highly dynamic machine like Atlas, the famously agile humanoid from Boston Dynamics, it relies on a whole-body motion controller that continuously adjusts its posture to avoid falls. In its overall design, Surena is probably more comparable to humanoids like UBTECH’s Walker, Honda’s Asimo, Pal Robotics’ Talos, KAIST’s Hubo, and AIST’s HRP-2, although these robots have already publicly demonstrated a broader range of capabilities.

Photo: University of Tehran/CAST Surena IV is equipped with new hands that allow it to grasp a wider variety of objects.

To make his robots more widely known, Professor Yousefi-Koma has sought to take them to international conferences and trade shows, and his group has published papers on humanoid robot design, bipedal locomotion, and other topics. He has said that his group wants to “make [Surena] very competitive and may be taking part in an international challenge as well as finding real applications and users.”

In footage released as part of Surena IV’s unveiling, which happened late last year, the humanoid is shown lifting a box, inside of which is Surena Mini, a knee-high robot that the CAST team developed in 2017.

Another video gives a behind-the-scenes look at Surena’s development, showing the researchers testing new hardware, writing code, and assisting the robot as it takes its first steps.

The evolution of Surena

The latest Surena model has come a long way since the initial version, unveiled in 2008, which had only 8 degrees of freedom (DoF) and used wheels on its feet to move around. Surena II had 22 DoF and was the first version capable of walking. Surena III, with 31 DoF, demonstrated skills like climbing steps and kicking a ball.

Surena IV has even more axes of motion, with additional 12 DoF, most on the hands, bringing the total to 43 DoF. The CAST team redesigned several components, developing new lightweight structural elements and small yet powerful custom actuators. As a result, the new robot—at 1.7 meters tall and 68 kilograms—is lighter and more compact than Surena III (1.9 m and 98 kg).

Photo: University of Tehran/CAST Led by Dr. Aghil Yousefi-Koma [standing to the left of the robot], a team of more than 50 researchers developed Surena IV at the University of Tehran’s Center for Advanced Systems and Technologies (CAST). During an event late last year, part of the group posed with Surena for a photo.

Another advance is the robot’s controller. The control loop now operates at a higher frequency (200 Hz), thanks to newly added FPGA boards. Sensors include stereo cameras, 6-axis force/torque sensors in the ankles, encoders on all joints, and an inertial measurement unit (IMU). A text-to-speech system allows the robot to recognize and generate basic speech.

To make all the sensors, controllers, and actuators work together, the researchers equipped Surena with the Robot Operating System, or ROS. They also used Gazebo, Choreonoid, and MATLAB to simulate the robot’s motions and evaluate different behaviors, including walking backwards and sideways, turning around, and push recovery.

But the upgrades are not just internal. Surena IV’s exterior features new plastic covers that make the robot look sleek—and only slightly menacing. Professor Yousefi-Koma has emphasized that he views Surena as a “symbol of technology advancement in the direction of peace and humanity,” hoping it will help inspire people about the possibilities of robotics.

A version of this article was originally published on Medium. The views expressed here are solely those of the authors and do not represent positions of IEEE Spectrum or the IEEE.

We here at Skydio have been developing and deploying machine learning systems for years due to their ability to scale and improve with data. However, to date our learning systems have only been used for interpreting information about the world; in this post, we present our first machine learning system for actually acting in the world.

Using a novel learning algorithm, the Skydio autonomy engine, and only 3 hours of “off-policy” logged data, we trained a deep neural network pilot that is capable of filming and tracking a subject while avoiding obstacles.

We approached the problem of training a deep neural network pilot through the lens of imitation learning, in which the goal is to train a model that imitates an expert. Imitation learning was an appealing approach for us because we have a huge trove of flight data with an excellent drone pilot—the motion planner inside the Skydio autonomy engine. However, we quickly found that standard imitation learning performed poorly when applied to our challenging, real-world problem domain.

Standard imitation learning worked fine in easy scenarios, but did not generalize well to difficult ones. We propose that the signal of the expert’s trajectory is not rich enough to learn efficiently. Especially within our domain of flying through the air, the exact choice of flight path is a weak signal because there can be many obstacle-free paths that lead to cinematic video. The average scenario overwhelms the training signal.

CEILing can leverage acausal (future) data, which enables it to “see” farther into the future and therefore train an even smarter pilot

How can we do better? Our insight is that we don’t have just any expert, we have a computational expert: the Skydio Autonomy Engine. Therefore instead of imitating what the expert does, we understand what the expert cares about. We call this approach Computational Expert Imitation Learning, or CEILing.

Why is CEILing better than standard imitation learning? Let’s consider a didactic example in which a teacher is trying to teach a student how to do multiplication. The teacher is deciding between two possible lesson plans. The first lesson plan is to give the student a bunch of multiplication problems, along with the answer key, and leave the student alone to let them figure out how multiplication works. The second lesson plan is to let the student attempt to solve some multiplication problems, give the student feedback on the exact mistakes they made, and continue until the student has mastered the topic.

Which lesson plan should the teacher choose? The second lesson plan is likely to be more effective because the student not only learns the correct answer, but also learns why the answer is correct. This allows the student to be able to solve multiplication problems they have never encountered before.

Gif: Skydio

Simulation trials of the deep neural pilot based on the Computational Expert Imitation Learning, or CEILing, approach developed by the authors.

This same insight applies to robot navigation: Some deviations from the expert should be penalized more heavily than others. For example, deviating from the expert is generally okay in open space, but a critical mistake if it is towards an obstacle or causes visual loss of the subject. CEILing lets us convey that information from the expert instead of blindly penalizing deviations from the expert’s trajectory. This is why CEILing trains a deep neural pilot that generalizes well with little data.

Now one question to ask is why even use CEILing to train a deep neural pilot? Why not just have the computational expert be the pilot? The primary reason we are excited about CEILing is that CEILing could train a pilot that is actually better than the computational expert pilot.

How is this possible? Consider a scenario in which a drone needs to fly through a forest at high speed. This is a challenging environment because thin objects, such as tree branches, are difficult to see from far away. Although the current Skydio autonomy engine is able to perceive and avoid these thin branches, sometimes the branches can only be detected when the drone is already quite close, which forces the drone to execute an aggressive maneuver. In contrast, our deep neural pilot could be able to smoothly avoid these thin branches altogether because it will have learned that flying towards trees—which are large and easily seen—is correlated with encountering thin branches. In short, CEILing can leverage acausal (future) data, which enables it to “see” farther into the future and therefore train an even smarter pilot.

Although there is still much work to be done before the learned system will outperform our production system, we believe in pursuing leapfrog technologies. Deep reinforcement learning techniques promise to let us improve our entire system in a data-driven way, which will lead to an even smarter autonomous flying camera.

Gregory Kahn is a Ph.D. student at UC Berkeley advised by Sergey Levine and Pieter Abbeel, and was a research intern at Skydio in Spring 2019.

Abraham Bachrach is co-founder and CTO at Skydio.

Hayk Martiros leads the autonomy team at Skydio, whose work focuses on robust approaches to vision-based autonomous flight.

A version of this article was originally published on Medium. The views expressed here are solely those of the authors and do not represent positions of IEEE Spectrum or the IEEE.

We here at Skydio have been developing and deploying machine learning systems for years due to their ability to scale and improve with data. However, to date our learning systems have only been used for interpreting information about the world; in this post, we present our first machine learning system for actually acting in the world.

Using a novel learning algorithm, the Skydio autonomy engine, and only 3 hours of “off-policy” logged data, we trained a deep neural network pilot that is capable of filming and tracking a subject while avoiding obstacles.

We approached the problem of training a deep neural network pilot through the lens of imitation learning, in which the goal is to train a model that imitates an expert. Imitation learning was an appealing approach for us because we have a huge trove of flight data with an excellent drone pilot—the motion planner inside the Skydio autonomy engine. However, we quickly found that standard imitation learning performed poorly when applied to our challenging, real-world problem domain.

Standard imitation learning worked fine in easy scenarios, but did not generalize well to difficult ones. We propose that the signal of the expert’s trajectory is not rich enough to learn efficiently. Especially within our domain of flying through the air, the exact choice of flight path is a weak signal because there can be many obstacle-free paths that lead to cinematic video. The average scenario overwhelms the training signal.

CEILing can leverage acausal (future) data, which enables it to “see” farther into the future and therefore train an even smarter pilot

How can we do better? Our insight is that we don’t have just any expert, we have a computational expert: the Skydio Autonomy Engine. Therefore instead of imitating what the expert does, we understand what the expert cares about. We call this approach Computational Expert Imitation Learning, or CEILing.

Why is CEILing better than standard imitation learning? Let’s consider a didactic example in which a teacher is trying to teach a student how to do multiplication. The teacher is deciding between two possible lesson plans. The first lesson plan is to give the student a bunch of multiplication problems, along with the answer key, and leave the student alone to let them figure out how multiplication works. The second lesson plan is to let the student attempt to solve some multiplication problems, give the student feedback on the exact mistakes they made, and continue until the student has mastered the topic.

Which lesson plan should the teacher choose? The second lesson plan is likely to be more effective because the student not only learns the correct answer, but also learns why the answer is correct. This allows the student to be able to solve multiplication problems they have never encountered before.

Gif: Skydio

Simulation trials of the deep neural pilot based on the Computational Expert Imitation Learning, or CEILing, approach developed by the authors.

This same insight applies to robot navigation: Some deviations from the expert should be penalized more heavily than others. For example, deviating from the expert is generally okay in open space, but a critical mistake if it is towards an obstacle or causes visual loss of the subject. CEILing lets us convey that information from the expert instead of blindly penalizing deviations from the expert’s trajectory. This is why CEILing trains a deep neural pilot that generalizes well with little data.

Now one question to ask is why even use CEILing to train a deep neural pilot? Why not just have the computational expert be the pilot? The primary reason we are excited about CEILing is that CEILing could train a pilot that is actually better than the computational expert pilot.

How is this possible? Consider a scenario in which a drone needs to fly through a forest at high speed. This is a challenging environment because thin objects, such as tree branches, are difficult to see from far away. Although the current Skydio autonomy engine is able to perceive and avoid these thin branches, sometimes the branches can only be detected when the drone is already quite close, which forces the drone to execute an aggressive maneuver. In contrast, our deep neural pilot could be able to smoothly avoid these thin branches altogether because it will have learned that flying towards trees—which are large and easily seen—is correlated with encountering thin branches. In short, CEILing can leverage acausal (future) data, which enables it to “see” farther into the future and therefore train an even smarter pilot.

Although there is still much work to be done before the learned system will outperform our production system, we believe in pursuing leapfrog technologies. Deep reinforcement learning techniques promise to let us improve our entire system in a data-driven way, which will lead to an even smarter autonomous flying camera.

Gregory Kahn is a Ph.D. student at UC Berkeley advised by Sergey Levine and Pieter Abbeel, and was a research intern at Skydio in Spring 2019.

Abraham Bachrach is co-founder and CTO at Skydio.

Hayk Martiros leads the autonomy team at Skydio, whose work focuses on robust approaches to vision-based autonomous flight.

This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Honda Research Institute’s (HRI) experimental social robot Haru was first introduced at the ACM/IEEE Human Robot Interaction conference in 20181. The robot is designed as a platform to investigate social presence and emotional and empathetic engagement for long-term human interaction. Envisioned as a multimodal communicative agent, Haru interacts through nonverbal sounds (paralanguage), eye, face, and body movements (kinesics), and voice (language). While some of Haru’s features connect it to a long lineage of social robots, others distinguish it and suggest new opportunities for human-robot interaction.

Haru is currently in its first iteration, with plans underway for future development. Current research with Haru is conducted with core partners of the Socially Intelligent Robotics Consortium (SIRC), described in more detail below, and it concentrates on its potential to communicate across the previously mentioned three-way modality (language, paralanguage, and kinesics). Long term, we hope Haru will drive research into robots as a new form of companion species and as a platform for creative content.

Image: Honda Research Institute Communication with Haru

The first Haru prototype is a relatively simple communication device—an open-ended platform through which researchers can explore the mechanisms of multi-modal human-robot communication. Researchers deploy various design techniques, technologies, and interaction theories to develop Haru’s basic skillset for conveying information, personality, and affect. Currently, researchers are developing Haru’s interaction capabilities to create a new form of telepresence embodiment that breaks away from the traditional notion of a tablet-on-wheels, while also exploring ways for the robot to communicate with people more directly.

Haru as a mediator

Haru’s unique form factor and its developing interaction modalities enable its telepresence capabilities to extend beyond what is possible when communicating through a screen. Ongoing research with partners involves mapping complex and nuanced human expressions to Haru. One aspect of this work developing an affective language that takes advantage of Haru’s unique LCD screen eyes and LED mouth, neck, and eye motions, and rotating base to express emotions of varying type and intensity. Another ongoing research is the development of perception capabilities for Haru to recognize different human emotions and will allow Haru to mimic the remote person’s affective cues and behaviors. 

From a practical telepresence perspective, using Haru in this way can, on the one hand, add to the enjoyment and clarity with which affect can be conveyed and read at a distance. On the other hand, Haru’s interpretation of emotional cues can help align the expectations between sender and receiver. For example, Haru could be used to enhance communication in a multi-cultural context by displaying culturally appropriate social cues without the operator controlling it. These future capabilities are being put into practice through the development of “robomoji/harumoji”—a hardware equivalent of emoji with Haru acting as a telepresence robot that transmits a remote person’s emotions and actions. 

Haru as a communicator

Building upon the work on telepresence, we will further develop Haru as a communicative agent through the development of robot agency2 consistent with its unique design and behavioral repertoire, and possibly with its own character. Ongoing research involves kinematic modeling and motion design and synthesis. We are also developing appropriate personalities for Haru with the help of human actors, who are helping us create a cohesive verbal and nonverbal language for Haru that expresses its capabilities and can guide users to interact with it in appropriate ways.

Haru will in this case autonomously frame its communication to fit that of its co-present communication partner. In this mode, the robot has more freedom to alter and interpret expressions for communication across the three modalities than in telepresence mode. The goal will be to communicate so that the receiving person not only understands the information being conveyed, but can infer Haru’s own behavioral state and presence. This will ideally make the empathic relation between Haru and the human bi-directional.     

Hybrid realism

Designing for both telepresence and direct communication means that Haru must be able to negotiate between times when it is communicating on behalf of a remote person, and times when it is communicating for itself. In the future, researchers will explore how teleoperation can be mixed with a form of autonomy, such that people around Haru know whether it is acting as a mediator or as an agent as it switches between “personalities.” Although challenging, this will open up new exciting opportunities to study the effect of social robots embodying other people, or even other robots. 

Image: Honda Research Institute Robots as a companion species

The design concept of Haru focuses on creating an emotionally expressive embodied agent that can support long-term, sustainable human-robot interaction. People already have these kinds of relationships with their pets. Pets are social creatures people relate to create bonds without sharing the same types of perception, comprehension, and expression. This bodes well for robots such as Haru, which similarly may not perceive, comprehend, or express themselves in fully human-like ways, but which nonetheless can encourage humans to empathize with them. This may also prove to be a good strategy for grounding human expectations while maximizing emotional engagement.

Recent research3,4 has shown that in some cases people can develop emotional ties to robots. Using Haru, researchers can investigate whether human-robot relations can become even more affectively rich and meaningful through informed design and better interaction strategies, as well as through more constant and varied engagement with users. 

Photo: Evan Ackerman/IEEE Spectrum Haru is sad.

Ongoing research with partners involve interaction-based dynamic interpretation of trust, design and maximization of likable traits through empathy, and investigating how people of different ages and cognitive abilities make sense of and interact with Haru. To carry out these tasks, our partners have been deploying Haru in public spaces such as in a children’s science museum, an elementary school, and an intergenerational daycare institution.

In the later stages of the project, we hope to look at how a relationship with Haru might position the robot as a new kind of companion. Haru is not intended to be a replacement for a pet or a person, but rather an entity that people can bond and grow with in a new way—a new “companion species”5. Sustained interaction with Haru will also allow us to explore how bonding can result in a relationship and a feeling of trust. We argue that a trusting relationship requires the decisional freedom not to trust and therefore, would be one in which neither human nor robot are always subordinate to each other, but rather one that supports a flexible bi-directional relationship enabling cooperative behavior and growth.  

Image: Honda Research Institute Robots as a platform for creative content

Social robots should not be designed as mere extensions to speech-centric smart devices; we need to develop meaningful and unique use cases suited to the rich modalities robots offer. For Haru, such use cases must be rooted in its rich three-way communicative modality and its unique physical and emotional affordances. Researchers plan to maximize Haru’s communication and interaction potential by developing social apps and creative content that convey information and evoke emotional responses that can enhance people’s perceptions of the robot as a social presence. 

The Haru research team (consisting of both HRI Japan and Socially Intelligent Robotics Consortium partners) will develop apps that meet societal needs through Haru’s emotional interactions with people. We envision three main pillars of research activity: entertainment and play, physically enriched social networking, and mental well-being. Entertainment and play will encompass interactive storytelling, expressive humor, and collaborative gaming, among others. Physically enriched social networking will build upon the telepresence capability of Haru. Mental well-being will explore the beneficial role of Haru’s physical manifestation of emotion and affection for coaching.

Numerous smartphone apps6 explore this application in digital and unimodal communication. We believe the much richer expressiveness and physical presence will give Haru exciting opportunities to qualitatively improve such services. Current research with partners involves collaborative problem solving and developing interactive applications for entertainment and play. Haru is currently able to play rock-paper-scissors and tower of Hanoi with people, which are being tested in an elementary school setting. Another capability that is being developed for Haru is storytelling play, in which the robot can collaborate with people to create a story. 

Haru’s research team also sees the importance of both automated and curated creative multimedia content to complement such social applications. Content drives the success of smartphones and gaming consoles, but has not yet been widely considered for robots. The team will therefore investigate how content can contribute to social human-robot interaction. Haru, by delivering creative content using its three-way communication modality, will blur the line between the physical and the animated, reinforcing its presence alongside the content it conveys.

Content drives the success of smartphones and gaming consoles, but has not yet been widely considered for robots.

In the context of Haru, creativity will be cooperative. Through sustained interaction, Haru will be able to motivate, encourage, and illustrate alternatives in creative tasks (e.g. different potential storylines in storytelling or game play) depending on the cognitive state of its interaction partner. The mixture of digital visualization (with a mini projector, for example) and physical interaction opens up new opportunities for cooperative creativity. Ongoing research with partners involves automatic affect generation for Haru, models for generating prose content for story telling in different genre, and affectively rich audio and text-to-speech to support social apps.  

Finally, Haru will be a networked device embedded into a personal or office based digital infrastructure. On one hand, Haru can serve as a physical communication platform for other robots (e.g. robotic vacuum cleaners that typically have no significant communication channels), and on the other hand, Haru can use the additional physical capabilities of other devices to complement its own limitations.

Photo: Evan Ackerman/IEEE Spectrum Haru sees you! Research with Haru

Realizing the design and development goals for Haru is an enormous task, only possible through collaboration of a multi-disciplinary group of experts in robotics, human-robot interaction, design, engineering, psychology, anthropology, philosophy, cognitive science, and other fields. To help set and achieve the goals described above, over the past two years academic researchers in Japan, Australia, the United States, Canada, and Europe have been spending their days with Haru.

This global team of roboticists, designers, and social scientists call themselves the Socially Intelligent Robotics Consortium (SIRC) and are working together to expand and actualize the potential of human-robot communication. Under the umbrella of the consortium, Haru has been adopted as a common platform enabling researchers to share data, research methods, results, and evaluation tools. As a partnership, it is possible to tackle the complex problem of developing empathic robotic mediators, companions, and content platforms by working on independent tasks under a common theme. 

Haru offers the opportunity to experiment with social robots as a new kind of trustful companion, not just based on functionality, but on emotional connection and the sense of unique social presence.

Social robotics is a promising area of research, but in a household crowded with smart devices and appliances, it is important to re-think the role of social robots to find their niche amidst competing and overlapping products while keeping users engaged over months and years instead of days and weeks. The development of Haru offers the opportunity to experiment with flexible and varied ways to communicate and interact with social robots as a new kind of trustful companion, not just based on functionality, but on emotional connection and the sense of unique social presence.

Research with Haru will explore exciting applications driven by interaction, cooperation, creativity and design, suspending people’s disbelief so that each experience with Haru is a step in a shared journey, demonstrating how people and robots might grow and find common purpose and enjoyment together in a hybrid human-robot society.

Dr. Randy Gomez is a senior scientist at Honda Research Institute Japan. He oversees the embodied communication research group, which explores the synergy between communication and interaction in order to create meaningful experiences with social robots.

References

1. Gomez R., Szapiro D., Galindo K. & Nakamura K., “Haru: Hardware Design of an Experimental Tabletop Robot Assistant,” HRI 2018: 233-240.

2. Gunkel D., “Robot Rights,” The MIT Press, 2018.

3. Haraway, “The Companion Species Manifesto: Dogs, People, and Significant Otherness,” Prickly Paradigm Press, 2003.

4. Weiss A., Wurhofer, D., & Tscheligi, M, “I love this dog-children’s emotional attachment to the robotic dog AIBO,” International Journal of Social Robotics, 2009.

5. Sabanovic S., Reeder, S. M., & Kechavarzi, B, “Designing robots in the wild: In situ prototype evaluation for a break management robot,” Journal of Human-Robot Interaction, 2014.

6. Inkster B., Shubhankar S., & Vinod S. , “An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study,” JMIR mHealth and uHealth, 2018.

This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Honda Research Institute’s (HRI) experimental social robot Haru was first introduced at the ACM/IEEE Human Robot Interaction conference in 20181. The robot is designed as a platform to investigate social presence and emotional and empathetic engagement for long-term human interaction. Envisioned as a multimodal communicative agent, Haru interacts through nonverbal sounds (paralanguage), eye, face, and body movements (kinesics), and voice (language). While some of Haru’s features connect it to a long lineage of social robots, others distinguish it and suggest new opportunities for human-robot interaction.

Haru is currently in its first iteration, with plans underway for future development. Current research with Haru is conducted with core partners of the Socially Intelligent Robotics Consortium (SIRC), described in more detail below, and it concentrates on its potential to communicate across the previously mentioned three-way modality (language, paralanguage, and kinesics). Long term, we hope Haru will drive research into robots as a new form of companion species and as a platform for creative content.

Image: Honda Research Institute Communication with Haru

The first Haru prototype is a relatively simple communication device—an open-ended platform through which researchers can explore the mechanisms of multi-modal human-robot communication. Researchers deploy various design techniques, technologies, and interaction theories to develop Haru’s basic skillset for conveying information, personality, and affect. Currently, researchers are developing Haru’s interaction capabilities to create a new form of telepresence embodiment that breaks away from the traditional notion of a tablet-on-wheels, while also exploring ways for the robot to communicate with people more directly.

Haru as a mediator

Haru’s unique form factor and its developing interaction modalities enable its telepresence capabilities to extend beyond what is possible when communicating through a screen. Ongoing research with partners involves mapping complex and nuanced human expressions to Haru. One aspect of this work developing an affective language that takes advantage of Haru’s unique LCD screen eyes and LED mouth, neck, and eye motions, and rotating base to express emotions of varying type and intensity. Another ongoing research is the development of perception capabilities for Haru to recognize different human emotions and will allow Haru to mimic the remote person’s affective cues and behaviors. 

From a practical telepresence perspective, using Haru in this way can, on the one hand, add to the enjoyment and clarity with which affect can be conveyed and read at a distance. On the other hand, Haru’s interpretation of emotional cues can help align the expectations between sender and receiver. For example, Haru could be used to enhance communication in a multi-cultural context by displaying culturally appropriate social cues without the operator controlling it. These future capabilities are being put into practice through the development of “robomoji/harumoji”—a hardware equivalent of emoji with Haru acting as a telepresence robot that transmits a remote person’s emotions and actions. 

Haru as a communicator

Building upon the work on telepresence, we will further develop Haru as a communicative agent through the development of robot agency2 consistent with its unique design and behavioral repertoire, and possibly with its own character. Ongoing research involves kinematic modeling and motion design and synthesis. We are also developing appropriate personalities for Haru with the help of human actors, who are helping us create a cohesive verbal and nonverbal language for Haru that expresses its capabilities and can guide users to interact with it in appropriate ways.

Haru will in this case autonomously frame its communication to fit that of its co-present communication partner. In this mode, the robot has more freedom to alter and interpret expressions for communication across the three modalities than in telepresence mode. The goal will be to communicate so that the receiving person not only understands the information being conveyed, but can infer Haru’s own behavioral state and presence. This will ideally make the empathic relation between Haru and the human bi-directional.     

Hybrid realism

Designing for both telepresence and direct communication means that Haru must be able to negotiate between times when it is communicating on behalf of a remote person, and times when it is communicating for itself. In the future, researchers will explore how teleoperation can be mixed with a form of autonomy, such that people around Haru know whether it is acting as a mediator or as an agent as it switches between “personalities.” Although challenging, this will open up new exciting opportunities to study the effect of social robots embodying other people, or even other robots. 

Image: Honda Research Institute Robots as a companion species

The design concept of Haru focuses on creating an emotionally expressive embodied agent that can support long-term, sustainable human-robot interaction. People already have these kinds of relationships with their pets. Pets are social creatures people relate to create bonds without sharing the same types of perception, comprehension, and expression. This bodes well for robots such as Haru, which similarly may not perceive, comprehend, or express themselves in fully human-like ways, but which nonetheless can encourage humans to empathize with them. This may also prove to be a good strategy for grounding human expectations while maximizing emotional engagement.

Recent research3,4 has shown that in some cases people can develop emotional ties to robots. Using Haru, researchers can investigate whether human-robot relations can become even more affectively rich and meaningful through informed design and better interaction strategies, as well as through more constant and varied engagement with users. 

Photo: Evan Ackerman/IEEE Spectrum Haru is sad.

Ongoing research with partners involve interaction-based dynamic interpretation of trust, design and maximization of likable traits through empathy, and investigating how people of different ages and cognitive abilities make sense of and interact with Haru. To carry out these tasks, our partners have been deploying Haru in public spaces such as in a children’s science museum, an elementary school, and an intergenerational daycare institution.

In the later stages of the project, we hope to look at how a relationship with Haru might position the robot as a new kind of companion. Haru is not intended to be a replacement for a pet or a person, but rather an entity that people can bond and grow with in a new way—a new “companion species”5. Sustained interaction with Haru will also allow us to explore how bonding can result in a relationship and a feeling of trust. We argue that a trusting relationship requires the decisional freedom not to trust and therefore, would be one in which neither human nor robot are always subordinate to each other, but rather one that supports a flexible bi-directional relationship enabling cooperative behavior and growth.  

Image: Honda Research Institute Robots as a platform for creative content

Social robots should not be designed as mere extensions to speech-centric smart devices; we need to develop meaningful and unique use cases suited to the rich modalities robots offer. For Haru, such use cases must be rooted in its rich three-way communicative modality and its unique physical and emotional affordances. Researchers plan to maximize Haru’s communication and interaction potential by developing social apps and creative content that convey information and evoke emotional responses that can enhance people’s perceptions of the robot as a social presence. 

The Haru research team (consisting of both HRI Japan and Socially Intelligent Robotics Consortium partners) will develop apps that meet societal needs through Haru’s emotional interactions with people. We envision three main pillars of research activity: entertainment and play, physically enriched social networking, and mental well-being. Entertainment and play will encompass interactive storytelling, expressive humor, and collaborative gaming, among others. Physically enriched social networking will build upon the telepresence capability of Haru. Mental well-being will explore the beneficial role of Haru’s physical manifestation of emotion and affection for coaching.

Numerous smartphone apps6 explore this application in digital and unimodal communication. We believe the much richer expressiveness and physical presence will give Haru exciting opportunities to qualitatively improve such services. Current research with partners involves collaborative problem solving and developing interactive applications for entertainment and play. Haru is currently able to play rock-paper-scissors and tower of Hanoi with people, which are being tested in an elementary school setting. Another capability that is being developed for Haru is storytelling play, in which the robot can collaborate with people to create a story. 

Haru’s research team also sees the importance of both automated and curated creative multimedia content to complement such social applications. Content drives the success of smartphones and gaming consoles, but has not yet been widely considered for robots. The team will therefore investigate how content can contribute to social human-robot interaction. Haru, by delivering creative content using its three-way communication modality, will blur the line between the physical and the animated, reinforcing its presence alongside the content it conveys.

Content drives the success of smartphones and gaming consoles, but has not yet been widely considered for robots.

In the context of Haru, creativity will be cooperative. Through sustained interaction, Haru will be able to motivate, encourage, and illustrate alternatives in creative tasks (e.g. different potential storylines in storytelling or game play) depending on the cognitive state of its interaction partner. The mixture of digital visualization (with a mini projector, for example) and physical interaction opens up new opportunities for cooperative creativity. Ongoing research with partners involves automatic affect generation for Haru, models for generating prose content for story telling in different genre, and affectively rich audio and text-to-speech to support social apps.  

Finally, Haru will be a networked device embedded into a personal or office based digital infrastructure. On one hand, Haru can serve as a physical communication platform for other robots (e.g. robotic vacuum cleaners that typically have no significant communication channels), and on the other hand, Haru can use the additional physical capabilities of other devices to complement its own limitations.

Photo: Evan Ackerman/IEEE Spectrum Haru sees you! Research with Haru

Realizing the design and development goals for Haru is an enormous task, only possible through collaboration of a multi-disciplinary group of experts in robotics, human-robot interaction, design, engineering, psychology, anthropology, philosophy, cognitive science, and other fields. To help set and achieve the goals described above, over the past two years academic researchers in Japan, Australia, the United States, Canada, and Europe have been spending their days with Haru.

This global team of roboticists, designers, and social scientists call themselves the Socially Intelligent Robotics Consortium (SIRC) and are working together to expand and actualize the potential of human-robot communication. Under the umbrella of the consortium, Haru has been adopted as a common platform enabling researchers to share data, research methods, results, and evaluation tools. As a partnership, it is possible to tackle the complex problem of developing empathic robotic mediators, companions, and content platforms by working on independent tasks under a common theme. 

Haru offers the opportunity to experiment with social robots as a new kind of trustful companion, not just based on functionality, but on emotional connection and the sense of unique social presence.

Social robotics is a promising area of research, but in a household crowded with smart devices and appliances, it is important to re-think the role of social robots to find their niche amidst competing and overlapping products while keeping users engaged over months and years instead of days and weeks. The development of Haru offers the opportunity to experiment with flexible and varied ways to communicate and interact with social robots as a new kind of trustful companion, not just based on functionality, but on emotional connection and the sense of unique social presence.

Research with Haru will explore exciting applications driven by interaction, cooperation, creativity and design, suspending people’s disbelief so that each experience with Haru is a step in a shared journey, demonstrating how people and robots might grow and find common purpose and enjoyment together in a hybrid human-robot society.

Dr. Randy Gomez is a senior scientist at Honda Research Institute Japan. He oversees the embodied communication research group, which explores the synergy between communication and interaction in order to create meaningful experiences with social robots.

References

1. Gomez R., Szapiro D., Galindo K. & Nakamura K., “Haru: Hardware Design of an Experimental Tabletop Robot Assistant,” HRI 2018: 233-240.

2. Gunkel D., “Robot Rights,” The MIT Press, 2018.

3. Haraway, “The Companion Species Manifesto: Dogs, People, and Significant Otherness,” Prickly Paradigm Press, 2003.

4. Weiss A., Wurhofer, D., & Tscheligi, M, “I love this dog-children’s emotional attachment to the robotic dog AIBO,” International Journal of Social Robotics, 2009.

5. Sabanovic S., Reeder, S. M., & Kechavarzi, B, “Designing robots in the wild: In situ prototype evaluation for a break management robot,” Journal of Human-Robot Interaction, 2014.

6. Inkster B., Shubhankar S., & Vinod S. , “An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study,” JMIR mHealth and uHealth, 2018.

The Sony Aibo has been the most sophisticated home robot that you can buy for an astonishing 20 years. The first Aibo went on sale in 1999, and even though there was a dozen year-long gap between 2005’s ERS-7 and the latest ERS-1000, there was really no successful consumer robot over that intervening time that seriously challenged the Aibo.

Part of what made Aibo special was how open Sony was user customization and programmability. Aibo served as the RoboCup Standard Platform for a decade, providing an accessible hardware platform that leveled the playing field for robotic soccer. Designed to stand up to the rigors of use by unsupervised consumers (and, presumably, their kids), Aibo offered both durability and versatility that compared fairly well to later, much more expensive robots like Nao

Aibo ERS-1000: The newest model

The newest Aibo, the ERS-1000, was announced in late 2017 and is now available for US $2,900 in the United States and 198,000 yen in Japan. It’s faithful to the Aibo family, while benefiting from years of progress in robotics hardware and software. However, it wasn’t until last November that Sony opened up Aibo to programmers, by providing visual programming tools as well as access to an API (application programming interface). And over the holidays, Sony lent us an Aibo to try it out for ourselves.

This is not (I repeat not) an Aibo review: I’m not going to talk about how cute it is, how to feed it, how to teach it to play fetch, how weird it is that it pretends to pee sometimes, or how it feels to have it all snuggled up in your lap while you’re working at your computer. Instead, I’m going to talk about how to (metaphorically) rip it open and access its guts to get it to do exactly what you want.

Photo: Evan Ackerman/IEEE Spectrum The newest Aibo, the ERS-1000, was announced in late 2017 and is now available for US $2,900 in the United States and 198,000 yen in Japan.

As you read this, please keep in mind that I’m not much of a software engineer—my expertise extends about as far as Visual Basic, because as far as I’m concerned that’s the only programming language anyone needs to know. My experience here is that of someone who understands (in the abstract) how programming works, and who is willing to read documentation and ask for help, but I’m still very much a beginner at this. Fortunately, Sony has my back. For some of it, anyway.

Getting started with Aibo’s visual programming

The first thing to know about Sony’s approach to Aibo programming is that you don’t have access to everything. We’ll get into this more later, but in general, Aibo’s “personality” is completely protected and cannot be modified:

When you execute the program, Aibo has the freedom to decide which specific behavior to execute depending on his/her psychological state. The API respects Aibo's feelings so that you can enjoy programming while Aibo stays true to himself/herself.

This is a tricky thing for Sony, since each Aibo “evolves” its own unique personality, which is part of the appeal. Running a program on Aibo risks very obviously turning it from an autonomous entity into a mindless robot slave, so Sony has to be careful to maintain Aibo’s defining traits while still allowing you to customize its behavior. The compromise that they came up with is mostly effective, and when Aibo runs a program, it doesn’t disable its autonomous behaviors but rather adds the behaviors you’ve created to the existing ones. 

Aibo’s visual programming system is based on Scratch. If you’ve never used Scratch, that’s fine, because it’s a brilliantly easy and intuitive visual language to use, even for non-coders. Sony didn’t develop it—it’s a project out of MIT, and while it was originally designed for children, it’s great for adults who don’t have coding experience. Rather than having to type in code, Scratch is based around colorful blocks that graphically represent functions. The blocks are different shapes, and only fit together in a way that will yield a working bit of code. Variables appear in handy little drop-down menus, and you can just drag and drop different blocks to build as many programs as you want. You can even read through the code directly, and it’ll explain what it does in a way that makes intuitive sense, more or less:

Screenshot: Evan Ackerman/IEEE Spectrum A sample Aibo visual program from Sony.

Despite the simplicity of the visual programming language, it’s possible to create some fairly complex programs. You have access to control loops like if-then-else and wait-until, and multiple loops can run at the same time. Custom blocks allow you to nest things inside of other things, and you have access to variables and operators. Here’s a program that I put together in just a few minutes to get Aibo to entertain itself by kicking a ball around:

Screenshot: Evan Ackerman/IEEE Spectrum A program I created to make Aibo chase a ball around.

This program directs Aibo to respond to “let’s play” by making some noises and motions, locating and approaching its ball, kicking its ball, and then moving in some random directions before repeating the loop. Petting Aibo on its back will exit the loop.

Programming Aibo: What you can (and can’t) do

It’s a lot of fun to explore all of Aibo’s different behaviors, although if you’re a new user, it does minimize a bit of the magic to see this big long list of everything that Aibo is capable of doing. The granularity of some of commands is a little weird—there’s a command for “gets close to” an object, as well as a command for “gets closer to” an object. And rather than give you direct access to Aibo’s servos to convey emotions or subtle movement cues, you’re instead presented with a bewildering array of very specific options, like:

  • Aibo opens its mouth a little and closes it
  • Aibo has an “I get it” look
  • Aibo gives a high five with its right front paw
  • Aibo faces to the left petulantly
  • Aibo has a dream of becoming a human being and runs about

Unfortunately, there’s no way to “animate” Aibo directly—you don’t have servo-level control, and unlike many (if not most) programmable robots, Sony hasn’t provided a way for users to move Aibo’s servos and then have the robot play back those motions, which would have been simple and effective.

Running one of these programs can be a little frustrating at times, because there’s no indication of when (or if) Aibo transitions from its autonomous behavior to your program—you just run the program and then wait. Sony advises you to start each program with a command that puts Aibo’s autonomy on hold, but depending on what Aibo is in the middle of doing when you run your program, it may take it a little bit to finish its current behavior. My solution for this was to start each program with a sneeze command to let me know when things were actually running. This worked well enough I guess, but it’s not ideal, because sometimes Aibo sneezes by itself.

Running one of these programs can be a little frustrating at times, because there’s no indication of when (or if) Aibo transitions from its autonomous behavior to your program. My solution for this was to start each program with a sneeze command to let me know when things were actually running.

The biggest restriction of the visual programming tool is that as far as I can tell there’s no direct method of getting information back from Aibo—you can’t easily query the internal state of the robot. For example, if you want to know how much battery charge Aibo has, there’s a sensing block for that, but the best you seem to be able to do is have Aibo do specific things in response to the value of that block, like yap a set number of times to communicate what its charge is. More generally, however, it can be tough to write more interactive programs, because it’s hard to tell when, if, why, or how such programs are failing. From what I can tell, there’s no way “step” through your program, or to see which commands are being executed when, making it very hard to debug anything complicated. And this is where the API comes in handy, since it does give you explicit information back.

Aibo API: How it works

There’s a vast chasm between the Aibo visual programming language and the API. Or at least, that’s how I felt about it. The visual programming is simple and friendly, but the API just tosses you straight into the deep end of the programming pool. The good news is that the majority of the stuff that the API allows you to do can also be done visually, but there are a few things that make the API worth having a crack at, if you’re willing to put the work in.

The first step to working with the Aibo API is to get a token, which is sort of like an access password for your Sony Aibo account. There are instructions about how to do this that are clear enough, because it just involves clicking one single button. Step two is finding your Aibo’s unique device ID, and I found myself immediately out of my comfort zone with Sony’s code example of how to do that:

$ curl -X GET https://public.api.aibo.com/v1/devices \
-H "Authorization:Bearer ${accessToken}" 

As it turns out, “curl” (or cURL) is a common command line tool for sending and receiving data via various network protocols, and it’s free and included with Windows. I found my copy in C:\Windows\System32. Being able to paste my token directly into that bit of sample code and have it work would have been too easy—after a whole bunch of futzing around, I figured out that (in Windows) you need to explicitly call “curl.exe” in the command line and that you have to replace “${accessToken}” with your access token, as opposed to just the bit that says “accessToken.” This sort of thing may be super obvious to many people, but it wasn’t to me, and with the exception of some sample code and a reasonable amount of parameter-specific documentation, Sony itself offers very little hand-holding. But since figuring this stuff out is my job, on we go!

Image: Sony How the Aibo API works: Your computer doesn’t talk directly to your robot. Instead, data flows between your computer and Sony’s cloud-based servers, and from the cloud to your robot. 

I don’t have a huge amount of experience with APIs (read: almost none), but the way that the Aibo API works seems a little clunky. As far as I can tell, everything runs through Sony’s Aibo server, which completely isolates you from the Aibo itself. As an example, let’s say we want to figure out how much battery Aibo has left. Rather than just sending a query to the robot and getting a response, we instead have to ask the Aibo server to ask Aibo, and then (separately) ask the Aibo server what Aibo’s response was. Literally, the process is to send an “Execute HungryStatus” command, which returns an execution ID, and then in a second command you request the result of that execution ID, which returns the value of HungryStatus. Weirdly, HungryStatus is not a percentage or a time remaining, but rather a string that goes from “famished” (battery too low to move) to “hungry” (needs to charge) to “enough” (charged enough to move). It’s a slightly strange combination of allowing you to get deep into Aibo’s guts while seeming trying to avoid revealing that there’s a robot under there.

Screenshot: Evan Ackerman/IEEE Spectrum Example of the code required to determine Aibo’s charge. (I blurred areas showing my Aibo’s device ID and token.)

Anyway, back to the API. I think most of the unique API functionality is related to Aibo’s state—how much is Aibo charged, how sleepy is Aibo, what is Aibo perceiving, where is Aibo being touched, that sort of thing. And even then, you can kludge together ways of figuring out what’s going on in Aibo’s lil’ head if you try hard enough with the visual programming, like by turning battery state into some number of yaps.

But the API does also offer a few features that can’t be easily replicated through visual programming. Among other things, you have access to useful information like which specific voice commands Aibo is responding to and exactly where (what angle) those commands are coming from, along with estimates of distance and direction to objects that Aibo recognizes. Really, though, the value of the API for advanced users is the potential of being able to have other bits of software interact directly with Aibo.

API possibilities, and limitations

For folks who are much better at programming than I am, the Aibo API does offer the potential to hook in other services. A programming expert I consulted suggested that it would be fairly straightforward to set things up so that (for example) Aibo would bark every time someone sends you a tweet. Doing this would require writing a Python script and hosting it somewhere in the cloud, which is beyond the scope of this review, but not at all beyond the scope of a programmer with modest skills and experience, I would imagine.

Fundamentally, the API means that just about anything can be used to send commands to Aibo, and the level of control that you have could even give Aibo a way to interact with other robots. It would just be nice if it was a little bit simpler, and a little more integrated, since there are some significant limitations worth mentioning.

For example, you have only indirect access to the majority of Aibo’s sensors, like the camera. Aibo will visually recognize a few specific objects, or a general “person,” but you can’t add new objects or differentiate between people (although Aibo can do this as part of its patrol feature). You can’t command Aibo to take a picture. Aibo can’t make noises that aren’t in its existing repertoire, and there’s no way to program custom motions. You also can’t access any of Aibo’s mapping data, or command it to go to specific places. It’s unfortunate that many of the features that justify Aibo’s cost, and differentiate it from something that’s more of a toy, aren’t accessible to developers at this point.

Photo: Evan Ackerman/IEEE Spectrum Aibo’s API gives users access to, among other things, specific voice commands the robot is responding to and exactly where (what angle) those commands are coming from, along with estimates of distance and direction to objects that Aibo recognizes. Aibo’s programmability: The future

Overall, I appreciate the approach that Sony took with Aibo’s programmability, making it accessible to both absolute beginners as well as more experienced developers looking to link Aibo to other products and services. I haven’t yet seen any particularly compelling examples of folks leveraging this capability with Aibo, but the API has only been publicly available for a month or two. I would have liked to have seen more sample programs from Sony, especially more complex visual programs, and I would have really appreciated a gentler transition over to the API. Hopefully, both of these things can be addressed in the near future.

There’s a reluctance on Sony’s part to give users more control over Aibo. Some of that may be technical, and some of it may be privacy-related, but there are also omissions of functionality and limitations that don’t seem to make sense. I wonder if Sony is worried about risking an otherwise careful compromise between a robot that maintains its unique personality, and a robot that can be customized to do whatever you want it to do. As it stands, Sony is still in control of how Aibo moves, and how Aibo expresses emotions, which keeps the robot’s behavior consistent, even if it’s executing behaviors that you tell it to. 

At this point, I’m not sure that the Aibo API is full-featured and powerful enough to justify buying an Aibo purely for its developer potential, especially given the cost of the robot. If you already have an Aibo, you should definitely play with the new programming functions, because they’re free. I do feel like this is a significant step in a very positive direction for Sony, showing that they’re willing to commit resources to the nascent Aibo developer community, and I’m very much looking forward to seeing how Aibo’s capabilities continue to grow.

Photo: Evan Ackerman/IEEE Spectrum Aibo deserves a rest!

Thanks to Sony for lending us an Aibo unit for the purposes of this review. I named it Aibo, and I will miss its blue eyes. And special thanks to Kevin Finn for spending part of his holiday break helping me figure out how Aibo’s API works. If you need help with your Aibo, or help from a professional software engineer on any number of other things, you can find him here.

[ Aibo Developer Site ]

Pages