IEEE Spectrum Robotics

IEEE Spectrum Robotics recent content
Subscribe to IEEE Spectrum Robotics feed

It may be theoretically impossible for humans to control a superintelligent AI, a new study finds. Worse still, the research also quashes any hope for detecting such an unstoppable AI when it’s on the verge of being created. 

Slightly less grim is the timetable. By at least one estimate, many decades lie ahead before any such existential computational reckoning could be in the cards for humanity. 

Alongside news of AI besting humans at games such as chess, Go and Jeopardy have come fears that superintelligent machines smarter than the best human minds might one day run amok. “The question about whether superintelligence could be controlled if created is quite old,” says study lead author Manuel Alfonseca, a computer scientist at the Autonomous University of Madrid. “It goes back at least to Asimov’s First Law of Robotics, in the 1940s.”

The Three Laws of Robotics, first introduced in Isaac Asimov's 1942 short story “Runaround,” are as follows:

  1. A robot may not injure a human being or, through inaction, allow a human being to come to harm.
  2. A robot must obey the orders given it by human beings except where such orders would conflict with the First Law.
  3. A robot must protect its own existence as long as such protection does not conflict with the First or Second Laws.

In 2014, philosopher Nick Bostrom, director of the Future of Humanity Institute at the University of Oxford, not only explored ways in which a superintelligent AI could destroy us but also investigated potential control strategies for such a machine—and the reasons they might not work.

Bostrom outlined two possible types of solutions of this “control problem.” One is to control what the AI can do, such as keeping it from connecting to the Internet, and the other is to control what it wants to do, such as teaching it rules and values so it would act in the best interests of humanity. The problem with the former is that Bostrom thought a supersmart machine could probably break free from any bonds we could make. With the latter, he essentially feared that humans might not be smart enough to train a superintelligent AI.

Now Alfonseca and his colleagues suggest it may be impossible to control a superintelligent AI, due to fundamental limits inherent to computing itself. They detailed their findings this month in the Journal of Artificial Intelligence Research.

The researchers suggested that any algorithm that sought to ensure a superintelligent AI cannot harm people had to first simulate the machine’s behavior to predict the potential consequences of its actions. This containment algorithm then would need to halt the supersmart machine if it might indeed do harm.

However, the scientists said it was impossible for any containment algorithm to simulate the AI’s behavior and predict with absolute certainty whether its actions might lead to harm. The algorithm could fail to correctly simulate the AI’s behavior or accurately predict the consequences of the AI’s actions and not recognize such failures.

“Asimov’s first law of robotics has been proved to be incomputable,” Alfonseca says, “and therefore unfeasible.” 

We may not even know if we have created a superintelligent machine, the researchers say. This is a consequence of Rice’s theorem, which essentially states that one cannot in general figure anything out about what a computer program might output just by looking at the program, Alfonseca explains.

On the other hand, there’s no need to spruce up the guest room for our future robot overlords quite yet. Three important caveats to the research still leave plenty of uncertainty to the group’s predictions. 

First, Alfonseca estimates AI’s moment of truth remains, he says, “At least two centuries in the future.” 

Second, he says researchers do not know if so-called artificial general intelligence, also known as strong AI, is theoretically even feasible. “That is, a machine as intelligent as we are in an ample variety of fields,” Alfonseca explains.

Last, Alfonseca says, “We have not proved that superintelligences can never be controlled—only that they can’t always be controlled.”

Although it may not be possible to control a superintelligent artificial general intelligence, it should be possible to control a superintelligent narrow AI—one specialized for certain functions instead of being capable of a broad range of tasks like humans. “We already have superintelligences of this type,” Alfonseca says. “For instance, we have machines that can compute mathematics much faster than we can. This is [narrow] superintelligence, isn’t it?”

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

HRI 2021 – March 8-11, 2021 – [Online] RoboSoft 2021 – April 12-16, 2021 – [Online]

Let us know if you have suggestions for next week, and enjoy today's videos.

Samsung announced some new prototype robots at CES this week. It's a fancy video, but my guess is that the actual autonomy here is minimal at best.

[ Samsung ]

Some very impressive reactive agility from Ghost Robotics' little quadruped.

[ Ghost Robotics ]

Toyota Research Institute (TRI) is researching how to bring together the instinctive reflexes of professional drivers and automated driving technology that uses the calculated foresight of a supercomputer. Using a Toyota GR Supra, TRI will learn from some of the most skilled drivers in the world to develop sophisticated vehicle control algorithms. The project’s goal is to design a new level of active safety technology for the Toyota Guardian™ approach of amplifying human driving abilities and helping keep people safe.

[ TRI ]

The end of this video features one of the most satisfying-sounding drone outtakes I've ever heard,

[ ASL ]

Reachy can now run the first humanoid VR teleoperation app available on the market. This app allows you to place yourself in the body of a humanoid robot, in VR, wherever you are in the world, to remotely operate it and carry out complex tasks. With this new functionality, Reachy is able to learn from the demonstration of the humans who control it, which makes application development even easier.

[ Pollen Robotics ]

Thanks Elsa!

Boston Dynamics has inspired some dancing robot videos recently, including this from Marco Tempest.

[ Marco Tempest ]

MOFLIN is an AI Pet created from a totally new concept. It possesses emotional capabilities that evolve like living animals. With its warm soft fur, cute sounds, and adorable movement, you’d want to love it forever. We took a nature inspired approach and developed a unique algorithm that allows MOFLIN to learn and grow by constantly using its interactions to determine patterns and evaluate its surroundings from its sensors. MOFLIN will choose from an infinite number of mobile and sound pattern combinations to respond and express its feelings. To put it in simple terms, it’s like you’re interacting with a living pet.

I like the minimalist approach. I dislike the "it’s like you’re interacting with a living pet" bit.

[ Kickstarter ]

There's a short gif of these warehouse robots going around, but here's the full video.

[ BionicHIVE ]

Vstone's Robovie-Z proves that you don't need fancy hardware for effective teleworking.

[ Vstone ]

All dual-arm robots are required, at some point, to play pool.

[ ABB ]

Volkswagen Group Components gives us a first glimpse of the real prototypes. This is one of the visionary charging concepts that Volkswagen hopes will expand the charging infrastructure over the next few years. Its task: fully autonomous charging of vehicles in restricted parking areas, like underground car parks.

To charge several vehicles at the same time, the mobile robot moves a trailer, essentially a mobile energy storage unit, to the vehicle, connects it up and then uses this energy storage unit to charge the battery of the electric vehicle. The energy storage unit stays with the vehicle during the charging process. In the meantime, the robot charges other electric vehicles.

[ Volkswagen ]

I've got a lot of questions about Moley Robotics' kitchen. But I would immediately point out that the system appears to do no prep work, which (at least for me) is the time-consuming and stressful part of cooking.

[ Moley Robotics ]

Blueswarm is a collective of fish-inspired miniature underwater robots that can achieve a wide variety of 3D collective behaviors - synchrony, aggregation/dispersion, milling, search - using only implicit communication mediated through the production and sensing of blue light. We envision this platform for investigating collective AI, underwater coordination, and fish-inspired locomotion and sensing.

[ Science Robotics ]

A team of Malaysian researchers are transforming pineapple leaves into strong materials that can be used to build frames for unmanned aircraft or drones.

[ Reuters ]

The future of facility disinfecting is here, protect your customers, and create peace of mind. Our drone sanitization spraying technology is up to 100% more efficient and effective than conventional manual spray sterilization processes.

[ Draganfly ]

Robots are no long a future technology, as small robots can be purchased today to be utilized for educational purposes. See what goes into making a modern robot come to life.

[ Huggbees ]

How does a robot dog learn how to dance? Adam and the Tested team examine and dive into Boston Dynamics' Choreographer software that was behind Spot's recent viral dancing video.

[ Tested ]

For years, engineers have had to deal with "the tyranny of the fairing," that anything you want to send into space has to fit into the protective nosecone on top of the rocket. A field of advanced design has been looking for new ways to improve our engineering, using the centuries-old artform to dream bigger.

[ JPL ]

What does an ideal neural network chip look like? The most important part is to have oodles of memory on the chip itself, say engineers. That’s because data transfer (from main memory to the processor chip) generally uses the most energy and produces most of the system lag—even compared to the AI computation itself. 

Cerebras Systems solved these problems, collectively called the memory wall, by making a computer consisting almost entirely of a single, large chip containing 18 gigabytes of memory. But researchers in France, Silicon Valley, and Singapore have come up with another way.

Called Illusion, it uses processors built with resistive RAM memory in a 3D stack built above the silicon logic, so it costs little energy or time to fetch data. By itself, even this isn’t enough, because neural networks are increasingly too large to fit in one chip. So the scheme also requires multiple such hybrid processors and an algorithm that both intelligently slices up the network among the processors and also knows when to rapidly turn processors off when they’re idle.

In tests, an eight-chip version of Illusion came within 3-4 percent of the energy use and latency of a “dream” processor that had all the needed memory and processing on one chip.

The research team—which included contributions from the French research lab CEA-Leti, Facebook, Nanyang Technological University in Singapore, San Jose State University, and Stanford University—were driven by the fact that the size of neural networks is only increasing. “This dream chip is never available in some sense, because it’s a moving target,” says Subhasish Mitra, the Stanford electrical engineering and computer science professor who led the research. “The neural nets get bigger and bigger faster than Moore’s Law can keep up,” he says.

So instead they sought to design a system that would create the illusion of a single processor with a large amount of on-chip memory (hence the project name) even though it was actually made up of multiple hybrid processors. That way Illusion could easily be expanded to accommodate growing neural networks.

Such a system needs three things, explains Mishra. The first is a lot of memory on the chip that can be accessed quickly and with little energy consumption. That’s where the 3D-integrated RRAM comes in. They chose RRAM, “because it is dense, 3D-integrated, and can be accessed quickly from a powered-down state and because it doesn’t lose its data when the power is off,” says H.-S. Philip Wong, a Stanford professor of electrical engineering and a collaborator on the project.

But RRAM does have a drawback. Like Flash memory, it wears out after being overwritten too many times. In Flash, software keeps track of how many overwrites have occurred to each block of memory cells and tries to keep that number even across all the cells in the chip. Stanford theoretical computer scientist Mary Wootters led the team’s effort to invent something similar for RRAM. The result, called Distributed Endurer, has the added burden of ensuring that wear from writing is even across multiple chips.

Even with Endurer and hybrid RRAM and processor chips, powerful neural nets, such as the natural language processors in use today, are still too large to fit in one chip. But using multiple hybrid chips means passing messages between them, eating up energy and wasting time.

The Illusion team’s solution, the second component of their technology, was to chop up the neural network in a way that minimizes message passing. Neural networks are, essentially, a set of nodes where computing happens and the edges that connect them. Each network will have certain nodes, or whole layers of nodes, that have tons of connections.

But there will also be choke points in the network—places where few messages must be passed between nodes. Dividing up a large neural network at these choke points and mapping each part on a separate chip ensures that a minimal amount of data needs to be sent from one chip to another. The Illusion mapping algorithm “automatically identifies the ideal places to cut a neural net to minimize these messages,” says Mitra.

But chopping things up like that has its own consequences. Inevitably, one chip will finish its business before another, stalling the system and wasting power. Other multichip systems, attempting to run very large neural networks focus on dividing up the network in a way that keeps all chips continually busy, but that comes at the expense of transferring more data among them.

Instead, in a third innovation, the Illusion team decided to engineer the hybrid processors and their controlling algorithm so that the chip can be turned off and on quickly. So when the chip is waiting for work, it isn’t consuming any power. CEA-Leti’s 3D RRAM technology was key to making 3D SoCs that can efficiently turn off completely within a few clock cycles and restart without losing data, says Mishra.

The team built an eight-chip version of Illusion and took it for a test drive on three deep neural networks. These networks were nowhere near the size of those that are currently stressing today’s computer systems, because each of the Illusion prototypes only had only 4 kilobytes of RRAM reserved for the neural network data. The “dream chip” they tested it against was really a single Illusion chip that mimicked the execution of the full neural network.

The 8-chip Illusion system was able to run the neural networks within 3.5 percent of the dream chip’s energy consumption and with 2.5 percent of its execution time. Mitra points out that the system scales up well. Simulations of a 64-chip Illusion with a total of 4 gigabytes of RRAM were just as close to the ideal.

“We are already underway with a new more capable prototype,” says Robert Radway, the Stanford University graduate student who was first author on a paper describing Illusion that appeared this week in Nature Electronics. Compared to the prototypes, the next generation of chips will have orders of magnitude more memory and ability to compute. And while the first generation was tested on inferencing, the next generation will be used to train them, which is a much more demanding task.

“Overall, we feel Illusion has profound implications for future technologies,” says Radway. “It opens up a large design space for technology innovations and creates a new scaling path for future systems.”

The FBI is still trying to identify some of the hundreds of people who launched a deadly attack on the U.S. Congress last week. “We have deployed our full investigative resources and are working closely with our federal, state, and local partners to aggressively pursue those involved in criminal activity during the events of January 6,” reads a page that contains images of dozens of unknown individuals, including one suspected of planting several bombs around Washington, D.C.

But while the public is being urged to put names to faces, America’s law enforcement agencies already have access to technologies that could do much of the heavy lifting. “We have over three billion photos that we indexed from the public internet, like Google for faces,” Hoan Ton-That, CEO of facial recognition start-up Clearview AI told Spectrum.

Ton-That said that Clearview’s customers, including the FBI, were using it to help identify the perpetrators: “Use our system, and in about a second it might point to someone’s Instagram page.” 

Clearview has attracted criticism because it relies on images scraped from social media sites without their—or their users’—permission. 

Photos: FBI Photographs of some of the people who attacked the United States Capitol Building on January 6, 2021, in Washington, D.C.

“The Capitol images are very good quality for automatic face recognition,” agreed a senior face recognition expert at one of America’s largest law enforcement agencies, who asked not to be named because they were talking to Spectrum without the permission of their superiors. 

Face recognition technology is commonplace in 2021. But the smartphone that recognizes your face in lieu of a passcode is solving a much simpler problem than trying to ID a masked (or often in the Capitol attacks surprisingly unmasked) intruder from a snatched webcam frame. 

The first is comparing a live, high resolution image to a single, detailed record stored in the phone. “Modern algorithms can basically see past issues such as how your head is oriented and variations in illumination or expression,” says Arun Vemury, director of the Department of Homeland Security (DHS) Science and Technology Directorate Biometric and Identity Technology Center. In a recent DHS test of such screening systems at airports, the best algorithm identified the correct person at least 96 percent of the time. 

The second scenario, however, is attempting to connect a fleeting, unposed image against one of hundreds of millions of people in the country or around the world. “Most law enforcement agencies can only search against mugshots of people who have been arrested in their jurisdictions, not even DMV records,” says the law enforcement officer.

And as the size of the database grows, so does the likelihood of the system generating incorrect identifications. “Very low false positive rates are still fairly elusive,” says Vemury. “Because there are lots of people out there who might look like you, from siblings and children to complete strangers. Honestly, faces are not all that different from one another.”

Nevertheless, advances in machine learning techniques and algorithms mean that facial recognition technologies are improving. After the COVID-19 pandemic hit last year, the National Institute of Standards and Technology tested industry-leading algorithms [PDF] with images of people wearing facemasks. While some of the algorithms saw error rates soar, others had only a modest decrease in effectiveness compared to maskless facial recognition efforts. Incredibly, the best algorithm’s performance with masks on was comparable to the state-of-the-art on unmasked images from just three years earlier.

In fact, claims Vemury, AI-powered facial recognition systems are now better at matching unfamiliar faces than even the best trained human. “There’s almost always a human adjudicating the result or figuring out whether or not to follow up,” he says, “But if a human is more likely to make an error than the algorithm, are we are we really thinking about this process correctly? It’s almost like asking a third grader to check a high school student’s calculus homework.”

Yet such technological optimism worries Elizabeth Rowe, a law professor at the University of Florida Levin College of Law. “Just because we have access to all of this information doesn't mean that we should necessarily use it,” she says. “Part of the problem is that there’s no reporting accountability of who’s using what and why, especially among private companies.”

Last week, The Washington Times as well as Republican Congressman Matt Gaetz incorrectly claimed that face recognition software from New York startup XRVision had revealed two of the Capitol attackers as incognito left-wing instigators. In fact, XRVision’s algorithms had identified the agitators as being the very right-wing extremists they appeared to be.  

There are also ongoing concerns that the way some face recognition technologies work (or fail to work) with different demographic groups [PDF] can exacerbate institutional racial biases. “We’re doing some additional research in this area,” says Vemury. “But even if you made the technologies totally fair, you could still deploy them in ways that could have a discriminative outcome.”

But if facial recognition technologies are linked to apprehending high profile suspects such as the Capitol attackers, enthusiasm for their use is likely only going to grow, says Rowe. 

“Just as consumers have gotten attached to the convenience of using biometrics to access our toys, I think we’ll find law enforcement agencies doing exactly the same thing,” she says. “It’s easy, and it gives them the potential to conduct investigations in a way that they couldn’t before.”

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

HRI 2021 – March 8-11, 2021 – [Online] RoboSoft 2021 – April 12-16, 2021 – [Online]

Let us know if you have suggestions for next week, and enjoy today’s videos.

Is it too late to say, “Happy Holidays”? Yes! Is it too late for a post packed with holiday robot videos? Never!

The Autonomous Systems Lab at ETH Zurich wishes everyone a Merry Christmas and a Happy 2021!

Now you know the best kept secret in robotics- the ETH Zurich Autonomous Systems Lab is a shack in the woods. With an elevator.

[ ASL ]

We have had to do things differently this year, and the holiday season is no exception. But through it all, we still found ways to be together. From all of us at NATO, Happy Holidays. After training in the snow and mountains of Iceland, an EOD team returns to base. Passing signs reminding them to ‘Keep your distance’ due to COVID-19, they return to their office a little dejected, unsure how they can safely enjoy the holidays. But the EOD robot saves the day and finds a unique way to spread the holiday cheer – socially distanced, of course.

[ EATA ]

Season's Greetings from Voliro!

[ Voliro ]

Thanks Daniel!

Even if you don't have a robot at home, you can still make Halodi Robotics's gingerbread cookies the old fashioned way.

[ Halodi Robotics ]

Thanks Jesper!

We wish you all a Merry Christmas in this very different 2020. This year has truly changed the world and our way of living. We, Energy Robotics, like to say thank you to all our customers, partners, supporters, friends and family.

An Aibo ERS-7? Sweet!

[ Energy Robotics ]

Thanks Stefan!

The nickname for this drone should be "The Grinch."

As it turns out, in real life taking samples of trees to determine how healthy they are is best done from the top.

[ DeLeaves ]

Thanks Alexis!

ETH Zurich would like to wish you happy holidays and a successful 2021 full of energy and health!

[ ETH Zurich ]

The QBrobotics Team wishes you all a Merry Christmas and a Happy New Year!

[ QBrobotics ]

Extend Robotics avatar twin got so excited opening a Christmas gift, using two arms coordinating, showing the dexterity and speed.

[ Extend Robotics ]

HEBI Robotics wishes everyone a great holiday season! Onto 2021!

[ HEBI Robotics ]

Christmas at the Mobile Robots Lab at Poznan Polytechnic.

[ Poznan ]

SWarm Holiday Wishes from the Hauert Lab!

[ Hauert Lab ]

Brubotics-VUB SMART and SHERO team wishes you a Merry Christmas and Happy 2021!


Success is all about teamwork! Thank you for supporting PAL Robotics. This festive season enjoy and stay safe!

[ PAL Robotics ]

Our robots wish you Happy Holidays! Starring world's first robot slackliner (Leonardo)!

[ Caltech ]

Happy Holidays and a Prosperous New Year from ZenRobotics!

[ ZenRobotics ]

Our Highly Dexterous Manipulation System (HDMS) dual-arm robot is ringing in the new year with good cheer!

[ RE2 Robotics ]

Happy Holidays 2020 from NAO!

[ SoftBank Robotics ]

Happy Holidays from DENSO Robotics!


A week ago, Boston Dynamics posted a video of Atlas, Spot, and Handle dancing to “Do You Love Me.” It was, according to the video description, a way “to celebrate the start of what we hope will be a happier year.” As of today the video has been viewed nearly 24 million times, and the popularity is no surprise, considering the compelling mix of technical prowess and creativity on display.

Strictly speaking, the stuff going on in the video isn’t groundbreaking, in the sense that we’re not seeing any of the robots demonstrate fundamentally new capabilities, but that shouldn’t take away from how impressive it is—you’re seeing state-of-the-art in humanoid robotics, quadrupedal robotics, and whatever-the-heck-Handle-is robotics.

What is unique about this video from Boston Dynamics is the artistic component. We know that Atlas can do some practical tasks, and we know it can do some gymnastics and some parkour, but dancing is certainly something new. To learn more about what it took to make these dancing robots happen (and it’s much more complicated than it might seem), we spoke with Aaron Saunders, Boston Dynamics’ VP of Engineering.

Saunders started at Boston Dynamics in 2003, meaning that he’s been a fundamental part of a huge number of Boston Dynamics’ robots, even the ones you may have forgotten about. Remember LittleDog, for example? A team of two designed and built that adorable little quadruped, and Saunders was one of them.

While he’s been part of the Atlas project since the beginning (and had a hand in just about everything else that Boston Dynamics works on), Saunders has spent the last few years leading the Atlas team specifically, and he was kind enough to answer our questions about their dancing robots.

IEEE Spectrum: What’s your sense of how the Internet has been reacting to the video?

Aaron Saunders: We have different expectations for the videos that we make; this one was definitely anchored in fun for us. The response on YouTube was record-setting for us: We received hundreds of emails and calls with people expressing their enthusiasm, and also sharing their ideas for what we should do next, what about this song, what about this dance move, so that was really fun. My favorite reaction was one that I got from my 94-year-old grandma, who watched the video on YouTube and then sent a message through the family asking if I’d taught the robot those sweet moves. I think this video connected with a broader audience, because it mixed the old-school music with new technology. 

We haven’t seen Atlas move like this before—can you talk about how you made it happen?

We started by working with dancers and a choreographer to create an initial concept for the dance by composing and assembling a routine. One of the challenges, and probably the core challenge for Atlas in particular, was adjusting human dance moves so that they could be performed on the robot. To do that, we used simulation to rapidly iterate through movement concepts while soliciting feedback from the choreographer to reach behaviors that Atlas had the strength and speed to execute. It was very iterative—they would literally dance out what they wanted us to do, and the engineers would look at the screen and go “that would be easy” or “that would be hard” or “that scares me.” And then we’d have a discussion, try different things in simulation, and make adjustments to find a compatible set of moves that we could execute on Atlas.

Throughout the project, the time frame for creating those new dance moves got shorter and shorter as we built tools, and as an example, eventually we were able to use that toolchain to create one of Atlas’ ballet moves in just one day, the day before we filmed, and it worked. So it’s not hand-scripted or hand-coded, it’s about having a pipeline that lets you take a diverse set of motions, that you can describe through a variety of different inputs, and push them through and onto the robot.

Image: Boston Dynamics

Were there some things that were particularly difficult to translate from human dancers to Atlas? Or, things that Atlas could do better than humans?

Some of the spinning turns in the ballet parts took more iterations to get to work, because they were the furthest from leaping and running and some of the other things that we have more experience with, so they challenged both the machine and the software in new ways. We definitely learned not to underestimate how flexible and strong dancers are—when you take elite athletes and you try to do what they do but with a robot, it’s a hard problem. It’s humbling. Fundamentally, I don’t think that Atlas has the range of motion or power that these athletes do, although we continue developing our robots towards that, because we believe that in order to broadly deploy these kinds of robots commercially, and eventually in a home, we think they need to have this level of performance.

One thing that robots are really good at is doing something over and over again the exact same way. So once we dialed in what we wanted to do, the robots could just do it again and again as we played with different camera angles.

I can understand how you could use human dancers to help you put together a routine with Atlas, but how did that work with Spot, and particularly with Handle?

I think the people we worked with actually had a lot of talent for thinking about motion, and thinking about how to express themselves through motion. And our robots do motion really well—they’re dynamic, they’re exciting, they balance. So I think what we found was that the dancers connected with the way the robots moved, and then shaped that into a story, and it didn’t matter whether there were two legs or four legs. When you don’t necessarily have a template of animal motion or human behavior, you just have to think a little harder about how to go about doing something, and that’s true for more pragmatic commercial behaviors as well.

“We used simulation to rapidly iterate through movement concepts while soliciting feedback from the choreographer to reach behaviors that Atlas had the strength and speed to execute. It was very iterative—they would literally dance out what they wanted us to do, and the engineers would look at the screen and go ‘that would be easy’ or ‘that would be hard’ or ‘that scares me.’” —Aaron Saunders, Boston Dynamics

How does the experience that you get teaching robots to dance, or to do gymnastics or parkour, inform your approach to robotics for commercial applications?

We think that the skills inherent in dance and parkour, like agility, balance, and perception, are fundamental to a wide variety of robot applications. Maybe more importantly, finding that intersection between building a new robot capability and having fun has been Boston Dynamics’ recipe for robotics—it’s a great way to advance.

One good example is how when you push limits by asking your robots to do these dynamic motions over a period of several days, you learn a lot about the robustness of your hardware. Spot, through its productization, has become incredibly robust, and required almost no maintenance—it could just dance all day long once you taught it to. And the reason it’s so robust today is because of all those lessons we learned from previous things that may have just seemed weird and fun. You’ve got to go into uncharted territory to even know what you don’t know.

Image: Boston Dynamics

It’s often hard to tell from watching videos like these how much time it took to make things work the way you wanted them to, and how representative they are of the actual capabilities of the robots. Can you talk about that?

Let me try to answer in the context of this video, but I think the same is true for all of the videos that we post. We work hard to make something, and once it works, it works. For Atlas, most of the robot control existed from our previous work, like the work that we’ve done on parkour, which sent us down a path of using model predictive controllers that account for dynamics and balance. We used those to run on the robot a set of dance steps that we’d designed offline with the dancers and choreographer. So, a lot of time, months, we spent thinking about the dance and composing the motions and iterating in simulation.

Dancing required a lot of strength and speed, so we even upgraded some of Atlas’ hardware to give it more power. Dance might be the highest power thing we’ve done to date—even though you might think parkour looks way more explosive, the amount of motion and speed that you have in dance is incredible. That also took a lot of time over the course of months; creating the capability in the machine to go along with the capability in the algorithms.

Once we had the final sequence that you see in the video, we only filmed for two days. Much of that time was spent figuring out how to move the camera through a scene with a bunch of robots in it to capture one continuous two-minute shot, and while we ran and filmed the dance routine multiple times, we could repeat it quite reliably. There was no cutting or splicing in that opening two-minute shot. 

There were definitely some failures in the hardware that required maintenance, and our robots stumbled and fell down sometimes. These behaviors are not meant to be productized and to be a 100 percent reliable, but they’re definitely repeatable. We try to be honest with showing things that we can do, not a snippet of something that we did once. I think there’s an honesty required in saying that you’ve achieved something, and that’s definitely important for us.

You mentioned that Spot is now robust enough to dance all day. How about Atlas? If you kept on replacing its batteries, could it dance all day, too?

Atlas, as a machine, is still, you know… there are only a handful of them in the world, they’re complicated, and reliability was not a main focus. We would definitely break the robot from time to time. But the robustness of the hardware, in the context of what we were trying to do, was really great. And without that robustness, we wouldn’t have been able to make the video at all. I think Atlas is a little more like a helicopter, where there’s a higher ratio between the time you spend doing maintenance and the time you spend operating. Whereas with Spot, the expectation is that it’s more like a car, where you can run it for a long time before you have to touch it.

When you’re teaching Atlas to do new things, is it using any kind of machine learning? And if not, why not?

As a company, we’ve explored a lot of things, but Atlas is not using a learning controller right now. I expect that a day will come when we will. Atlas’ current dance performance uses a mixture of what we like to call reflexive control, which is a combination of reacting to forces, online and offline trajectory optimization, and model predictive control. We leverage these techniques because they’re a reliable way of unlocking really high performance stuff, and we understand how to wield these tools really well. We haven’t found the end of the road in terms of what we can do with them.

We plan on using learning to extend and build on the foundation of software and hardware that we’ve developed, but I think that we, along with the community, are still trying to figure out where the right places to apply these tools are. I think you’ll see that as part of our natural progression.

Image: Boston Dynamics

Much of Atlas’ dynamic motion comes from its lower body at the moment, but parkour makes use of upper body strength and agility as well, and we’ve seen some recent concept images showing Atlas doing vaults and pullups. Can you tell us more?

Humans and animals do amazing things using their legs, but they do even more amazing things when they use their whole bodies. I think parkour provides a fantastic framework that allows us to progress towards whole body mobility. Walking and running was just the start of that journey. We’re progressing through more complex dynamic behaviors like jumping and spinning, that’s what we’ve been working on for the last couple of years. And the next step is to explore how using arms to push and pull on the world could extend that agility.

One of the missions that I’ve given to the Atlas team is to start working on leveraging the arms as much as we leverage the legs to enhance and extend our mobility, and I’m really excited about what we’re going to be working on over the next couple of years, because it’s going to open up a lot more opportunities for us to do exciting stuff with Atlas.

What’s your perspective on hydraulic versus electric actuators for highly dynamic robots?

Across my career at Boston Dynamics, I’ve felt passionately connected to so many different types of technology, but I’ve settled into a place where I really don’t think this is an either-or conversation anymore. I think the selection of actuator technology really depends on the size of the robot that you’re building, what you want that robot to do, where you want it to go, and many other factors. Ultimately, it’s good to have both kinds of actuators in your toolbox, and I love having access to both—and we’ve used both with great success to make really impressive dynamic machines.

I think the only delineation between hydraulic and electric actuators that appears to be distinct for me is probably in scale. It’s really challenging to make tiny hydraulic things because the industry just doesn’t do a lot of that, and the reciprocal is that the industry also doesn’t tend to make massive electrical things. So, you may find that to be a natural division between these two technologies. 

Besides what you’re working on at Boston Dynamics, what recent robotics research are you most excited about?

For us as a company, we really love to follow advances in sensing, computer vision, terrain perception, these are all things where the better they get, the more we can do. For me personally, one of the things I like to follow is manipulation research, and in particular manipulation research that advances our understanding of complex, friction-based interactions like sliding and pushing, or moving compliant things like ropes.

We’re seeing a shift from just pinching things, lifting them, moving them, and dropping them, to much more meaningful interactions with the environment. Research in that type of manipulation I think is going to unlock the potential for mobile manipulators, and I think it’s really going to open up the ability for robots to interact with the world in a rich way. 

Is there anything else you’d like people to take away from this video?

For me personally, and I think it’s because I spend so much of my time immersed in robotics and have a deep appreciation for what a robot is and what its capabilities and limitations are, one of my strong desires is for more people to spend more time with robots. We see a lot of opinions and ideas from people looking at our videos on YouTube, and it seems to me that if more people had opportunities to think about and learn about and spend time with robots, that new level of understanding could help them imagine new ways in which robots could be useful in our daily lives. I think the possibilities are really exciting, and I just want more people to be able to take that journey.

No matter how much brilliant work the folks at NASA and JPL put into their planetary exploration robots (and it’s a lot of brilliant work), eventually, inevitably, they break down. It’s rare that these breakdowns are especially complicated, but since the robots aren’t designed for repair, there isn’t much that can be done. And even if (say) the Mars rovers did have the ability to swap their own wheels when they got worn out, where are you going to get new robot wheels on Mars, anyway?

And this is the bigger problem—finding the necessary resources to keep robots running in extreme environments. We’ve managed to solve the power problem pretty well, often leveraging solar power, because solar power is a resource that you can find almost anywhere. You can’t make wheels out of solar power, but you can make wheels, and other structural components, out of another material that can be found just lying around all over the place: ice.

In a paper presented at the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Devin Carroll and Mark Yim from the GRASP Lab at the University of Pennsylvania, in Philadelphia, stress that this is very preliminary work. They say they’ve only just started exploring the idea of a robot made of ice. Obviously, you’re not going to be able to make actuators or batteries or other electronics-y things out of ice, and ice is never going to be as efficient as a structural material as titanium or carbon fiber or whatever. But ice can be found in a lot of different places, and it’s fairly unique in how it can be modified—heat can be used to cut and sculpt it, and also to glue it to itself.

The IROS paper takes a look at different ways of manufacturing robotic structural components from ice using both additive and subtractive manufacturing processes, with the goal of developing a concept for robots that can exhibit “self-reconfiguration, self-replication, and self-repair.” The assumption is that the robot would be operating in an environment with ice all over the place, where the ambient temperature is cold enough that the ice remains stable, and ideally also cold enough that the heat generated by the robot won’t lead to an inconvenient amount of self-melting or an even more inconvenient amount of self-shorting. Between molding, 3D printing, and CNC machining, it turns out that just cutting up the ice with a drill is the most energy efficient and effective method, although ideally you’d want to figure out a way of using it where you can manage the waste water and ice shavings that result so that they don’t refreeze somewhere you don’t want them to. Of course, sometimes refreezing is exactly what you want, since that’s how you do things like place actuators and attach one part to another.

IceBot is a proof-of-concept Antarctic exploration robot that weighs 6.3 kg. It was made by hand, and the researchers mostly just showed that it could move around and not immediately fall to pieces even at room temperature. There’s a lot to do before IceBot can realize some of those self-reconfiguration, self-replication, and self-repair capabilities, but the researchers are on it. And for more about that, we spoke with lead author Devin Carroll via email.

IEEE Spectrum: Where did this idea come from, and why do you think it hasn’t been tried before?

Devin Carroll: The first robot I designed was a tram robot for ecologists to use to survey forests. One of the challenges to making robots for this field is not only are robots expensive but the natural elements will break them given time. Mark and I started exploring the idea of building robots from found material as a way to add robustness to robotic systems operating in remote or hostile environments with a secondary goal of reducing the cost of the system. We ultimately settled on ice because of the design flexibility it affords us and the current interest in icy, remote environments. Climate change has many folks interested in the Antarctic and ice sheets while NASA and other space exploration groups are looking to the stars for ice and water. Therefore, ice felt like the most logical step—if we could build a robot from ice, perhaps it could be used to assist in exploring icy planets for life and data collection.

I would argue this hasn't been done before because of the uncertainty that using ice brings. Unlike traditional building material, the designer does not know a priori what conditions will cause the ice to fail—we can make an educated guess, but the margin for error is much higher. There are also complications associated with making the robot and getting it to the site safely. If we build it and then ship it to the deployment site it must be kept cold throughout its journey whereas if we make it at the deployment site we must also ship a manufacturing site with the system, increasing the overall monetary and energy costs associated with the system.

Can you speculate about what an arctic (or planetary) exploration robot might look like if it incorporated a self modification or repair capability?

When I think of an arctic (or planetary) exploration robot that incorporates self-modification or repair capabilities I envision a system with two types of robots—the first explores the environment and collects materials needed to perform self-augmentation or repair, and the second is some sort of manipulator/manufacturing system. We can envision the exploration class of robot returning to a centralized location with a request for a plow or some other augmentation and the manufacturing system will be able to attach the augmentation directly to the robot. Similarly with repair—if, for example, a robot recognizes a crack, the manipulator would be able to patch the crack using an ice band-aid of sorts, sealing the crack and 
preventing it from propagating further.

Part of my dissertation includes work towards this effort. In terms of the manipulator/end effector design, one idea we are exploring is using a mesh of resistance wire to locally melt surfaces of ice blocks and create a temporary connection between the block of ice and manipulator while we maneuver and machine it to a desired geometry.

What are you working on next?

My immediate focus is on designing a modular joint we can use to easily and securely join actuators with blocks of ice as well as working to develop an end effector that will allow us to manipulate blocks of ice without permanently deforming them via screw holes or other, similar connection methods. One of the interesting design challenges with these directions is ensuring that we maximize the connection strength while minimizing the energy use to implement them. Especially in remote environments, energy is a valued commodity and systems like the one I’ve described will only be effective if we take energy into consideration when designing them.

Robots Made From Ice: An Analysis of Manufacturing Techniques,” by Devin Carroll and Mark Yim from the University of Pennsylvania, was presented at IROS 2020.

Photo: TuSimple First in Freight: In 2021, San Diego–based startup TuSimple plans to deploy autonomous trucks that drive themselves from pickup to delivery without anybody on board.

Companies like Tesla, Uber, Cruise, and Waymo promise a future where cars are essentially mobile robots that can take us anywhere with a few taps on a smartphone. But a new category of vehicles is about to overtake self-driving cars in that leap into the future. Autonomous trucks have been quietly making just as much, if not more, progress toward commercial deployment, and their impact on the transportation of goods will no doubt be profound.

Among nearly a dozen companies developing autonomous trucking, San Diego–based TuSimple is trying to get ahead by combining unique technology with a series of strategic partnerships. Working with truck manufacturer Navistar as well as shipping giant UPS, TuSimple is already conducting test operations in Arizona and Texas, including depot-to-depot autonomous runs. These are being run under what’s known as “supervised autonomy,” in which somebody rides in the cab and is ready to take the wheel if needed. Sometime in 2021, the startup plans to begin doing away with human supervision, letting the trucks drive themselves from pickup to delivery without anybody on board.

Both autonomous cars and autonomous trucks rely on similar underlying technology: Sensors—typically cameras, lidars, and radars—feed data to a computer, which in turn controls the vehicle using skills learned through a massive amount of training and simulation. In principle, developing an autonomous truck can be somewhat easier than developing an autonomous car. That’s because unlike passenger vehicles, trucks—in particular long-haul tractor-trailers—generally follow fixed routes and spend most of their time on highways that are more predictable and easier to navigate than surface streets. Trucks are also a better platform for autonomy, with their large size providing more power for computers and an improved field of view for sensors, which can be mounted higher off the ground.

TuSimple claims that its approach is unique because its equipment is purpose built from the ground up for trucks. “Most of the other companies in this space got the seeds of their ideas from the DARPA Grand and Urban Challenges for autonomous vehicles,” says Chuck Price, chief product officer at TuSimple. “But the dynamics and functional behaviors of trucks are very different.”

The biggest difference is that trucks need to be able to sense conditions farther in advance, to allow for their longer stopping distance. The 200-meter practical range of lidar that most autonomous cars use as their primary sensor is simply not good enough for a fully loaded truck traveling at 120 kilometers per hour. Instead, TuSimple relies on multiple HD cameras that are looking up to 1,000 meters ahead whenever possible. The system detects other vehicles and calculates their trajectories at that distance, which Price says is approximately twice as far out as professional truck drivers look while driving.

Price argues that this capability gives TuSimple’s system more time to make decisions about the safest and most efficient way to drive. Indeed, its trucks use their brakes less often than trucks operated by human drivers, leading to improvements in fuel economy of about 10 percent. Steadier driving, with less side-to-side movement in a lane, brings additional efficiency gains while also minimizing tire wear. Price adds that autonomous trucks could also help address a shortage of truck drivers, which is expected to grow at an alarming rate.

Image: TuSimple Look Ahead: TuSimple uses a combination of lidar, radar, and HD cameras to detect vehicles and obstacles up to 1,000 meters away.

TuSimple’s fleet of 40 autonomous trucks has been hauling goods between freight depots in Phoenix, Tucson, Dallas, El Paso, Houston, and San Antonio. These routes are about 95 percent highway, but the trucks can also autonomously handle surface streets, bringing their cargo the entire distance, from depot driveway to depot driveway. Its vehicles join a growing fleet of robotic trucks from competitors such as Aurora, Embark, Locomation,, and even Waymo, the Alphabet spin-off that has long focused on self-driving cars.

“I think there’s a big wave coming in the logistics industry that’s not necessarily well appreciated,” says Tasha Keeney, an analyst at ARK Invest who specializes in autonomous technology. She explains that electrified autonomous trucks have the potential to reduce shipping expenses not only when compared with those of traditional trucking but also with those of rail, while offering the door-to-door service that rail cannot. “The relationships that TuSimple has made within the trucking industry are interesting—in the long term, vertically integrated, purpose-built vehicles will have a lot of advantages.”

By 2024,TuSimple plans to achieve Level 4 autonomy, meaning that its trucks will be able to operate without a human driver under limited conditions that may include time of day, weather, or premapped routes. At that point, TuSimple would start selling the trucks to fleet operators. Along the way, however, there are several other milestones the company must hit, beginning with its first “driver out” test in 2021, which Price describes as a critical real-world demonstration.

“This is no longer a science project,” he says. “It’s not research. It’s engineering. The driver-out demonstration is to prove to us, and to prove to the public, that it can be done.”

This article appears in the January 2021 print issue as “Robot Trucks Overtake Robot Cars.”

Artificial intelligence in healthcare is often a story of percentages. One 2017 study predicted AI could broadly improve patient outcomes by 30 to 40 percent. Which makes a manifold improvement in results particularly noteworthy. 

In this case, according to one Israeli machine learning startup, AI has the potential to boost the success rate of in vitro fertilization (IVF) by as much as 3x compared to traditional methods. In other words, at least according to these results, couples struggling to conceive that use the right AI system could be multiple times more likely to get pregnant.

The Centers for Disease Control and Prevention defines assisted reproductive technology (ART) as the process of removing eggs from a woman’s ovaries, fertilizing it with sperm and then implanting it back in the body.

The overall success rate of traditional ART is less than 30%, according to a recent study in the journal Acta Informatica Medica

But, says Daniella Gilboa, CEO of Tel Aviv, Israel-based AiVF—which provides an automated framework for fertility and IVF treatment—help may be on the way. (However, she also cautions against simply multiplying 3x with the 30% traditional ART success rate quoted above. “Since pregnancy is very much dependent on age and other factors, simple multiplication is not the way to compare the two methods,” Gilboa says.)

In the U.S. alone, 7.3 million women are battling infertility, according to a 2020 report from the American Society for Reproductive Medicine. In the U.S., 2.7 million IVF cycles are performed each year. 

AiVF is using ML and computer vision technology to allow embryologists to discover which embryos have the most potential for success during intrauterine implantation. AiVF is working with eight facilities in clinical trials around the world, including in Israel, Europe and the United States. It plans to launch commercially in 2021.

Ron Maor, head of algorithm research at AiVF, says that AiVF has built its own “bespoke” layer on top of various off-the-shelf AI, ML and deep learning applications. These tools “handle the specific and often unusual aspects of embryo images, which are very different from most AI tasks,” Maor says. 

AiVF’s ML technique involves creating time-lapse videos of developing embryos in an incubator. Over five days, the video shows the milestones of embryo development. Gilboa explains that previous methods yielded just one microscope image per day of the embryo compared with computer vision’s greater image-capturing success.

“By analyzing the video, you could dig out so many milestones and so many features the human eye cannot even detect,” Gilboa says. “Basically you train an algorithm on successful embryos, and you teach the algorithm what are successful embryos.”  

Likely only one embryo out of 10 can be implanted in the uterus. Once a physician implants the embryo, the embryologist will know within 14 days whether the patient is pregnant, Gilboa says. 

“As an embryologist I look at embryos, and I understand what happens to them,” Gilboa says. “If I learn on maybe thousands of embryos, the algorithm would learn on millions of embryos.”

As AiVF’s initial results suggest, computer vision and ML could potentially drive IVF’s prices down—in turn making it less expensive and burdensome for a woman to become pregnant. 

“Once you have a digital embryologist, then you could set up clinics much easier,” Gilboa says. “Or each clinic could be much more scalable. So many more people could enjoy IVF and achieve their dream of having a child.”

The folks at DeepMind are pushing their methods one step further toward the dream of a machine that learns on its own, the way a child does.

The London-based company, a subsidiary of Alphabet, is officially publishing the research today, in Nature, although it tipped its hand back in November with a preprint in ArXiv. Only now, though, are the implications becoming clear: DeepMind is already looking into real-world applications.

DeepMind won fame in 2016 for AlphaGo, a reinforcement-learning system that beat the game of Go after training on millions of master-level games. In 2018 the company followed up with AlphaZero, which trained itself to beat Go, chess and Shogi, all without recourse to master games or advice. Now comes MuZero, which doesn't even need to be shown the rules of the game.

The new system tries first one action, then another, learning what the rules allow, at the same time noticing the rewards that are proffered—in chess, by delivering checkmate; in Pac-Man, by swallowing a yellow dot. It then alters its methods until it hits on a way to win such rewards more readily—that is, it improves its play. Such learning by observation is ideal for any AI that faces problems that can't be specified easily. In the messy real world—apart from the abstract purity of games—such problems abound.

“We’re exploring the application of MuZero to video compression, something that could not have been done with AlphaZero,” says Thomas Hubert, one of the dozen co-authors of the Nature article. 

“It’s because it would be very expensive to do it with AlphaZero,” adds Julian Schrittwieser, another co-author. 

Other applications under discussion are in self-driving cars (which in Alphabet is handled by its subsidiary, Waymo) and in protein design, the next step beyond protein folding (which sister program AlphaFold recently mastered). Here the goal might be to design a protein-based pharmaceutical that must act on something that is itself an actor, say a virus or a receptor on a cell’s surface.

By simultaneously learning the rules and improving its play, MuZero outdoes its DeepMind predecessors in the economical use of data. In the Atari game of Ms. Pac-Man, when MuZero was limited to considering six or seven simulations per move—“a number too small to cover all the available actions,” as DeepMind notes, in a statement—it still did quite well. 

The system takes a fair amount of computing muscle to train, but once trained, it needs so little processing to make its decisions that the entire operation might be managed on a smartphone. “And even the training isn’t so much,” says Schrittwieser. “An Atari game would take 2-3 weeks to train on a single GPU.”

One reason for the lean operation is that MuZero models only those aspects of its environment—in a game or in the world—that matter in the decision-making process. “After all, knowing an umbrella will keep you dry is more useful to know than modeling the pattern of raindrops in the air,” DeepMind notes, in a statement.

Knowing what’s important is important. Chess lore relates a story in which a famous grandmaster is asked how many moves ahead he looks. “Only one,” intones the champion, “but it is always the best.” That is, of course, an exaggeration, yet it holds a kernel of truth: Strong chessplayers generally examine lines of analysis that span only a few dozen positions, but they know at a glance which ones are worth looking at. 

Children can learn a general pattern after exposure to a very few instances—inferring Niagara from a drop of water, as it were. This astounding power of generalization has intrigued psychologists for generations; the linguist Noam Chomsky once argued that children had to be hard-wired with the basics of grammar because otherwise the “poverty of the stimulus” would have made it impossible for them to learn how to talk. Now, though, this idea is coming into question; maybe children really do glean much from very little.

Perhaps machines, too, are in the early stages of learning how to learn in that fashion. Cue the shark music!

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

ICCR 2020 – December 26-29, 2020 – [Online] HRI 2021 – March 8-11, 2021 – [Online] RoboSoft 2021 – April 12-16, 2021 – [Online]

Let us know if you have suggestions for next week, and enjoy today's videos.

Look who’s baaaack: Jibo! After being sold (twice?), this pioneering social home robot (it was first announced back in 2014!) now belongs to NTT Disruption, which was described to us as the “disruptive company of NTT Group.” We are all for disruption, so this looks like a great new home for Jibo. 

[ NTT Disruption ]

Thanks Ana!

FZI's Christmas Party was a bit of a challenge this year; good thing robots are totally competent to have a part on their own.

[ FZI ]

Thanks Arne!

Do you have a lonely dog that just wants a friend to watch cat videos on YouTube with? The Danish Technological Institute has a gift idea for you.

[ DTI ]

Thanks Samuel!

Once upon a time, not so far away, there was an elf who received a very special gift. Watch this heartwarming story. Happy Holidays from the Robotiq family to yours!

Of course, these elves are not now unemployed, they've instead moved over to toy design full time!

[ Robotiq ]

An elegant Christmas video from the Dynamics System Lab, make sure and watch through the very end for a little extra cheer.

[ Dynamic Systems Lab ]

Thanks Angela!

Usually I complain when robotics companies make holiday videos without any real robots in them, but this is pretty darn cute from Yaskawa this year.

[ Yaskawa ]

Here's our little christmas gift to the fans of strange dynamic behavior. The gyro will follow any given shape as soon as the tip touches its edge and the rotation is fast enough. The friction between tip and shape generates a tangential force, creating a moment such that the gyroscopic reaction pushes the tip towards the shape. The resulting normal force produces a moment that guides the tip along the shape's edge.

[ TUM ]

Happy Holidays from Fanuc!

Okay but why does there have to be an assembly line elf just to put in those little cranks?

[ Fanuc ]

Astrobotic's cute little CubeRover is at NASA busy not getting stuck in places.

[ Astrobotic ]

Team CoSTAR is sharing more of their work on subterranean robotic exploration.

[ CoSTAR ]

Skydio Autonomy Enterprise Foundation (AEF), a new software product that delivers advanced AI-powered capabilities to assist the pilot during tactical situational awareness scenarios and detailed industrial asset inspections. Designed for professionals, it offers an enterprise-caliber flight experience through the new Skydio Enterprise application.

[ Skydio ]

GITAI's S1 autonomous robot will conduct two experiments: IVA (Intra-Vehicular Activity) tasks such as switch and cable operations, and assembly of structures and panels to demonstrate its capability for ISA (In-Space Assembly) tasks. This video was recorded in the Nanoracks Bishop Airlock mock-up facility @GITAI Tokyo office.


It's no Atlas, but this is some impressive dynamic balancing from iCub.

[ IIT ]

The Campaign to Stop Killer Robots and I don't agree on a lot of things, and I don't agree with a lot of the assumptions made in this video, either. But, here you go!

[ CSKR ]

I don't know much about this robot, but I love it.

[ Columbia ]

Most cable-suspended robots have a very well defined workspace, but you can increase that workspace by swinging them around. Wheee!

[ Laval ]

How you know your robot's got some skill: "to evaluate the performance in climbing over the step, we compared the R.L. result to the results of 12 students who attempted to find the best planning. The RL outperformed all the group, in terms of effort and time, both in continuous (joystick) and partition planning."

[ Zarrouk Lab ]

In the Spring 2021 semester, mechanical engineering students taking MIT class 2.007, Design and Manufacturing I, will be able to participate in the class’ iconic final robot competition from the comfort of their own home. Whether they take the class virtually or semi-virtually, students will be sent a massive kit of tools and materials to build their own unique robot along with a “Home Alone” inspired game board for the final global competition.

[ MIT ]

Well, this thing is still around!

[ Moley Robotics ]

Manuel Ahumada wrote in to share this robotic Baby Yoda that he put together with a little bit of help from Intel's OpenBot software.

[ YouTube ]

Thanks Manuel!

Here's what Zoox has been working on for the past half-decade.

[ Zoox ]

Last week’s announcement that Hyundai acquired Boston Dynamics from SoftBank left us with a lot of questions. We attempted to answer many of those questions ourselves, which is typically bad practice, but sometimes it’s the only option when news like that breaks.

Fortunately, yesterday we were able to speak with Michael Patrick Perry, vice president of business development at Boston Dynamics, who candidly answered our questions about Boston Dynamics’ new relationship with Hyundai and what the near future has in store.

IEEE Spectrum: Boston Dynamics is worth 1.1 billion dollars! Can you put that valuation into context for us?

Michael Patrick Perry: Since 2018, we’ve shifted to becoming a commercial organization. And that’s included a number of things, like taking our existing technology and bringing it to market for the first time. We’ve gone from zero to 400 Spot robots deployed, building out an ecosystem of software developers, sensor providers, and integrators. With that scale of deployment and looking at the pipeline of opportunities that we have lined up over the next year, I think people have started to believe that this isn’t just a one-off novelty—that there’s actual value that Spot is able to create. Secondly, with some of our efforts in the logistics market, we’re getting really strong signals both with our Pick product and also with some early discussions around Handle’s deployment in warehouses, which we think are going to be transformational for that industry. 

So, the thing that’s really exciting is that two years ago, we were talking about this vision, and people said, “Wow, that sounds really cool, let’s see how you do.” And now we have the validation from the market saying both that this is actually useful, and that we’re able to execute. And that’s where I think we’re starting to see belief in the long-term viability of Boston Dynamics, not just as a cutting-edge research shop, but also as a business. 

Photo: Boston Dynamics Boston Dynamics says it has deployed 400 Spot robots, building out an “ecosystem of software developers, sensor providers, and integrators.”

How would you describe Hyundai’s overall vision for the future of robotics, and how do they want Boston Dynamics to fit into that vision?

In the immediate term, Hyundai’s focus is to continue our existing trajectories, with Spot, Handle, and Atlas. They believe in the work that we’ve done so far, and we think that combining with a partner that understands many of the industries in which we’re targeting, whether its manufacturing, construction, or logistics, can help us improve our products. And obviously as we start thinking about producing these robots at scale, Hyundai’s expertise in manufacturing is going to be really helpful for us. 

Looking down the line, both Boston Dynamics and Hyundai believe in the value of smart mobility, and they’ve made a number of plays in that space. Whether it’s urban air mobility or autonomous driving, they’ve been really thinking about connecting the digital and the physical world through moving systems, whether that’s a car, a vertical takeoff and landing multi-rotor vehicle, or a robot. We are well positioned to take on robotics side of that while also connecting to some of these other autonomous services.

Can you tell us anything about the kind of robotics that the Hyundai Motor Group has going on right now?

So they’re working on a lot of really interesting stuff—exactly how that connects, you know, it’s early days, and we don’t have anything explicitly to share. But they’ve got a smart and talented robotics team that’s working in a variety of directions that  shares overlap with us. Obviously, a lot of things related to autonomous driving shares some DNA with the work that we’re doing in autonomy for Spot and Handle, so it’s pretty exciting to see.

What are you most excited about here? How do you think this deal will benefit Boston Dynamics?

I think there are a number of things. One is that they have an expertise in hardware, in a way that’s unique. They understand and appreciate the complexity of creating large complex robotic systems. So I think there’s some shared understanding of what it takes to create a great hardware product. And then also they have the resources to help us actually build those products with them together—they have manufacturing resources and things like that.

“Robotics isn’t a short term game. We’ve scaled pretty rapidly but if you start looking at what the full potential of a company like Boston Dynamics is, it’s going to take years to realize, and I think Hyundai is committed to that long-term vision”

Another thing that’s exciting is that Hyundai has some pretty visionary bets for autonomous driving and unmanned aerial systems, and all of that fits very neatly into the connected vision of robotics that we were talking about before. Robotics isn’t a short term game. We’ve scaled pretty rapidly for a robotics company in terms of the scale of robots we’ve able to deploy in the field, but if you start looking at what the full potential of a company like Boston Dynamics is, it’s going to take years to realize, and I think Hyundai is committed to that long-term vision.

And when you’ve been talking with Hyundai, what are they most excited about?

I think they’re really excited about our existing products and our technology. Looking at some of the things that Spot, Pick, and Handle are able to do now, there are applications that many of Hyundai’s customers could benefit from in terms of mobility, remote sensing, and material handling. Looking down the line, Hyundai is also very interested in smart city technology, and mobile robotics is going to be a core piece of that.

We tend to focus on Spot and Handle and Atlas in terms of platform capabilities, but can you talk a bit about some of the component-level technology that’s unique to Boston Dynamics, and that could be of interest to Hyundai?

Creating very power-dense actuator design is something that we’ve been successful at for several years, starting back with BigDog and LS3. And Handle has some hydraulic actuators and valves that are pretty unique in terms of their design and capability. Fundamentally, we have a systems engineering approach that brings together both hardware and software internally. You’ll often see different groups that specialize in something, like great mechanical or electrical engineering groups, or great controls teams, but what I think makes Boston Dynamics so special is that we’re able to put everything on the table at once to create a system that’s incredibly capable. And that’s why with something like Spot, we’re able to produce it at scale, while also making it flexible enough for all the different applications that the robot is being used for right now.

It’s hard to talk specifics right now, but there are obviously other disciplines within mechanical engineering or electrical engineering or controls for robots or autonomous systems where some of our technology could be applied.

Photo: Boston Dynamics Boston Dynamics is in the process of commercializing Handle, iterating on its design and planning to get box-moving robots on-site with customers in the next year or two.

While Boston Dynamics was part of Google, and then SoftBank, it seems like there’s been an effort to maintain independence. Is it going to be different with Hyundai? Will there be more direct integration or collaboration?

Obviously it’s early days, but right now, we have support to continue executing against all the plans that we have. That includes all the commercialization of Spot, as well as things for Atlas, which is really going to be pushing the capability of our team to expand into new areas. That’s going to be our immediate focus, and we don’t see anything that’s going to pull us away from that core focus in the near term. 

As it stands right now, Boston Dynamics will continue to be Boston Dynamics under this new ownership.

How much of what you do at Boston Dynamics right now would you characterize as fundamental robotics research, and how much is commercialization? And how do you see that changing over the next couple of years?

We have been expanding our commercial team, but we certainly keep a lot of the core capabilities of fundamental robotics research. Some of it is very visible, like the new behavior development for Atlas where we’re pushing the limits of perception and path planning. But a lot of the stuff that we’re working on is a little bit under the hood, things that are less obvious—terrain handling, intervention handling, how to make safe faults, for example. Initially when Spot started slipping on things, it would flail around trying to get back up. We’ve had to figure out the right balance between the robot struggling to stand, and when it should decide to just lock its limbs and fall over because it’s safer to do that.

I’d say the other big thrust for us is manipulation. Our gripper for Spot is coming out early next year, and that’s going to unlock a new set of capabilities for us. We have years and years of locomotion experience, but the ability to manipulate is a space that’s still relatively new to us. So we’ve been ramping up a lot of work over the last several years trying to get to an early but still valuable iteration of the technology, and we’ll continue pushing on that as we start learning what’s most useful to our customers.

“I’d say the other big thrust for us is manipulation. Our gripper for Spot is coming out early next year, and that’s going to unlock a new set of capabilities for us. We have years and years of locomotion experience, but the ability to manipulate is a space that’s still relatively new to us”

Looking back, Spot as a commercial robot has a history that goes back to robots like LS3 and BigDog, which were very ambitious projects funded by agencies like DARPA without much in the way of commercial expectations. Do you think these very early stage, very expensive, very technical projects are still things that Boston Dynamics can take on?

Yes—I would point to a lot of the things we do with Atlas as an example of that. While we don’t have immediate plans to commercialize Atlas, we can point to technologies that come out of Atlas that have enabled some of our commercial efforts over time. There’s not necessarily a clear roadmap of how every piece of Atlas research is going to feed over into a commercial product; it’s more like, this is a really hard fundamental robotics challenge, so let’s tackle it and learn things that we can then benefit from across the company. 

And fundamentally, our team loves doing cool stuff with robots, and you’ll continue seeing that in the months to come.

Photo: Boston Dynamics Spot’s arm with gripper is coming out early next year, and Boston Dynamics says that’s going to “unlock a new set of capabilities for us.”

What would it take to commercialize Atlas? And are you getting closer with Handle?

We’re in the process of commercializing Handle. We’re at a relatively early stage, but we have a plan to get the first versions for box moving on-site with customers in the next year or two. Last year, we did some on-site deployments as proof-of-concept trials, and using the feedback from that, we did a new design pass on the robot, and we’re looking at increasing our manufacturing capability. That’s all in progress.

For Atlas, it’s like the Formula 1 of robots—you’re not going to take a Formula 1 car and try to make it less capable so that you can drive it on the road. We’re still trying to see what are some applications that would necessitate an energy and computationally intensive humanoid robot as opposed to something that’s more inherently stable. Trying to understand that application space is something that we’re interested in, and then down the line, we could look at creating new morphologies to help address specific applications. In many ways, Handle is the first version of that, where we said, “Atlas is good at moving boxes but it’s very complicated and expensive, so let’s create a simpler and smaller design that can achieve some of the same things.”

The press release mentioned a mobile robot for warehouses that will be introduced next year—is that Handle?

Yes, that’s the work that we’re doing on Handle.

As we start thinking about a whole robotic solution for the warehouse, we have to look beyond a high power, low footprint, dynamic platform like Handle and also consider things that are a little less exciting on video. We need a vision system that can look at a messy stack of boxes and figure out how to pick them up, we need an interface between a robot and an order building system—things where people might question why Boston Dynamics is focusing on them because it doesn’t fit in with our crazy backflipping robots, but it’s really incumbent on us to create that full end-to-end solution.

Are you confident that under Hyundai’s ownership, Boston Dynamics will be able to continue taking the risks required to remain on the cutting edge of robotics?

I think we will continue to push the envelope of what robots are capable of, and I think in the near term, you’ll be able to see that realized in our products and the research that we’re pushing forward with. 2021 is going to be a great year for us.

Replicating the human sense of touch is complicated—electronic skins need to be flexible, stretchable, and sensitive to temperature, pressure and texture; they need to be able to read biological data and provide electronic readouts. Therefore, how to power electronic skin for continuous, real-time use is a big challenge. 

To address this, researchers from Glasgow University have developed an energy-generating e-skin made out of miniaturized solar cells, without dedicated touch sensors. The solar cells not only generate their own power—and some surplus—but also provide tactile capabilities for touch and proximity sensing. An early-view paper of their findings was published in IEEE Transactions on Robotics.

When exposed to a light source, the solar cells on the s-skin generate energy. If a cell is shadowed by an approaching object, the intensity of the light, and therefore the energy generated, reduces, dropping to zero when the cell makes contact with the object, confirming touch. In proximity mode, the light intensity tells you how far the object is with respect to the cell. “In real time, you can then compare the light intensity…and after calibration find out the distances,” says Ravinder Dahiya of the Bendable Electronics and Sensing Technologies (BEST) Group, James Watt School of Engineering, University of Glasgow, where the study was carried out. The team used infra-red LEDs with the solar cells for proximity sensing for better results.

To demonstrate their concept, the researchers wrapped a generic 3D-printed robotic hand in their solar skin, which was then recorded interacting with its environment. The proof-of-concept tests showed an energy surplus of 383.3 mW from the palm of the robotic arm. “The eSkin could generate more than 100 W if present over the whole body area,” they reported in their paper.

“If you look at autonomous, battery-powered robots, putting an electronic skin [that] is consuming energy is a big problem because then it leads to reduced operational time,” says Dahiya. “On the other hand, if you have a skin which generates energy, then…it improves the operational time because you can continue to charge [during operation].” In essence, he says, they turned a challenge—how to power the large surface area of the skin—into an opportunity—by turning it into an energy-generating resource.

Dahiya envisages numerous applications for BEST’s innovative e-skin, given its material-integrated sensing capabilities, apart from the obvious use in robotics. For instance, in prosthetics: “[As] we are using [a] solar cell as a touch sensor itself…we are also [making it] less bulkier than other electronic skins.” This, he adds, will help create prosthetics that are of optimal weight and size, thus making it easier for prosthetics users. “If you look at electronic skin research, the the real action starts after it makes contact… Solar skin is a step ahead, because it will start to work when the object is approaching…[and] have more time to prepare for action.” This could effectively reduce the time lag that is often seen in brain–computer interfaces.

There are also possibilities in the automation sector, particularly in electrical and interactive vehicles. A car covered with solar e-skin, because of its proximity-sensing capabilities, would be able to “see” an approaching obstacle or a person. It isn’t “seeing” in the biological sense, Dahiya clarifies, but from the point of view of a machine. This can be integrated with other objects, not just cars, for a variety of uses. “Gestures can be recognized as well…[which] could be used for gesture-based control…in gaming or in other sectors.”

In the lab, tests were conducted with a single source of white light at 650 lux, but Dahiya feels there are interesting possibilities if they could work with multiple light sources that the e-skin could differentiate between. “We are exploring different AI techniques [for that],” he says, “processing the data in an innovative way [so] that we can identify the the directions of the light sources as well as the object.”

The BEST team’s achievement brings us closer to a flexible, self-powered, cost-effective electronic skin that can touch as well as “see.” At the moment, however, there are still some challenges. One of them is flexibility. In their prototype, they used commercial solar cells made of amorphous silicon, each 1cm x 1cm. “They are not flexible, but they are integrated on a flexible substrate,” Dahiya says. “We are currently exploring nanowire-based solar cells…[with which] we we hope to achieve good performance in terms of energy as well as sensing functionality.” Another shortcoming is what Dahiya calls “the integration challenge”—how to make the solar skin work with different materials.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

ICCR 2020 – December 26-29, 2020 – [Online Conference] HRI 2021 – March 8-11, 2021 – [Online Conference] RoboSoft 2021 – April 12-16, 2021 – [Online Conference]

Let us know if you have suggestions for next week, and enjoy today's videos.

What a lovely Christmas video from Norlab.

[ Norlab ]

Thanks Francois!

MIT Mini-Cheetahs are looking for a new home. Our new cheetah cubs, born at NAVER LABS, are for the MIT Mini-Cheetah workshop. MIT professor Sangbae Kim and his research team are supporting joint research by distributing Mini-Cheetahs to researchers all around the world.

NAVER Labs ]

For several years, NVIDIA’s research teams have been working to leverage GPU technology to accelerate reinforcement learning (RL). As a result of this promising research, NVIDIA is pleased to announce a preview release of Isaac Gym – NVIDIA’s physics simulation environment for reinforcement learning research. RL-based training is now more accessible as tasks that once required thousands of CPU cores can now instead be trained using a single GPU.


At SINTEF in Norway, they're working on ways of using robots to keep tabs on giant floating cages of tasty fish:

One of the tricky things about operating robots in an environment like this is localization, so SINTEF is working on a solution that uses beacons:

While that video shows a lot of simulation (because otherwise there are tons of fish in the way), we're told that the autonomous navigation has been successfully demonstrated with an ROV in "a full scale fish farm with up to 200.000 salmon swimming around the robot."


Thanks Eleni!

We’ve been getting ready for the snow in the most BG way possible. Wishing all of you a happy and healthy holiday season.

[ Berkshire Grey ]

ANYbotics doesn’t care what time of the year it is, so Happy Easter!

And here's a little bit about why ANYmal C looks the way it does.

[ ANYbotics ]

Robert "Buz" Chmielewski is using two modular prosthetic limbs developed by APL to feed himself dessert. Smart software puts his utensils in roughly the right spot, and then Buz uses his brain signals to cut the food with knife and fork. Once he is done cutting, the software then brings the food near his mouth, where he again uses brain signals to bring the food the last several inches to his mouth so that he can eat it.


Introducing VESPER: a new military-grade small drone that is designed, sourced and built in the United States. Vesper offers a 50-minutes flight time, with speeds up to 45 mph (72 kph) and a total flight range of 25 miles (45 km). The magnetic snap-together architecture enables extremely fast transitions: the battery, props and rotor set can each be swapped in <5 seconds.

[ Vantage Robotics ]

In this video, a multi-material robot simulator is used to design a shape-changing robot, which is then transferred to physical hardware. The simulated and real robots can use shape change to switch between rolling gaits and inchworm gaits, to locomote in multiple environments.

[ Yale Faboratory ]

Get a preview of the cave environments that are being used to inspire the Final Event competition course of the DARPA Subterranean Challenge. In the Final Event, teams will deploy their robots to rapidly map, navigate, and search in competition courses that combine elements of man-made tunnel systems, urban underground, and natural cave networks!

The reason to pay attention this particular video is that it gives us some idea of what DARPA means when they say "cave."

[ SubT ]

MQ25 takes another step toward unmanned aerial refueling for the U.S. Navy. The MQ-25 test asset has flown for the first time with an aerial refueling pod containing the hose and basket that will make it an aerial refueler.

[ Boeing ]

We present a unified model-based and data-driven approach for quadrupedal planning and control to achieve dynamic locomotion over uneven terrain. We utilize on-board proprioceptive and exteroceptive feedback to map sensory information and desired base velocity commands into footstep plans using a reinforcement learning (RL) policy trained in simulation over a wide range of procedurally generated terrains.

[ DRS ]

The video shows the results of the German research project RoPHa. Within the project, the partners developed technologies for two application scenarios with the service robot Care-O-bot 4 in order to support people in need of help when eating.

[ RoPHa Project ]

Thanks Jenny!

This looks like it would be fun, if you are a crazy person.

[ Team BlackSheep ]

Robot accuracy is the limiting factor in many industrial applications. Manufacturers often only specify the pose repeatability values of their robotic systems. Fraunhofer IPA has set up a testing environment for automated measuring of accuracy performance criteria of industrial robots. Following the procedures defined in norm ISO 9283 allows generating reliable and repeatable results. They can be the basis for targeted measures increasing the robotic system’s accuracy.

[ Fraunhofer ]

Thanks Jenny!

The IEEE Women in Engineering - Robotics and Automation Society (WIE-RAS) hosted an online panel on best practices for teaching robotics. The diverse panel boasts experts in robotics education from a variety of disciplines, institutions, and areas of expertise.


Northwestern researchers have developed a first-of-its-kind soft, aquatic robot that is powered by light and rotating magnetic fields. These life-like robotic materials could someday be used as "smart" microscopic systems for production of fuels and drugs, environmental cleanup or transformative medical procedures.

[ Northwestern ]

Tech United Eindhoven's soccer robots now have eight wheels instead of four wheels, making them tweleve times better, if my math is right.

[ TU Eindhoven ]

This morning just after 3 a.m. ET, Boston Dynamics sent out a media release confirming that Hyundai Motor Group has acquired a controlling interest in the company that values Boston Dynamics at US $1.1 billion:

Under the agreement, Hyundai Motor Group will hold an approximately 80 percent stake in Boston Dynamics and SoftBank, through one of its affiliates, will retain an approximately 20 percent stake in Boston Dynamics after the closing of the transaction.

The release is very long, but does have some interesting bits—we’ll go through them, and talk about what this might mean for both Boston Dynamics and Hyundai.

We’ve asked Boston Dynamics for comment, but they’ve been unusually quiet for the last few days (I wonder why!). So at this point just keep in mind that the only things we know for sure are the ones in the release. If (when?) we hear anything from either Boston Dynamics or Hyundai, we’ll update this post.

The first thing to be clear on is that the acquisition is split between Hyundai Motor Group’s affiliates, including Hyundai MotorHyundai Mobis, and Hyundai Glovis. Hyundai Motor makes cars, Hyundai Mobis makes car parts and seems to be doing some autonomous stuff as well, and Hyundai Glovis does logistics. There are many other groups that share the Hyundai name, but they’re separate entities, at least on paper. For example, there’s a Hyundai Robotics, but that’s part of Hyundai Heavy Industries, a different company than Hyundai Motor Group. But for this article, when we say “Hyundai,” we’re talking about Hyundai Motor Group.

What’s in it for Hyundai?

Let’s get into the press release, which is filled with press release-y terms like “synergies” and “working together”—you can view the whole thing here—but still has some parts that convey useful info.

By establishing a leading presence in the field of robotics, the acquisition will mark another major step for Hyundai Motor Group toward its strategic transformation into a Smart Mobility Solution Provider. To propel this transformation, Hyundai Motor Group has invested substantially in development of future technologies, including in fields such as autonomous driving technology, connectivity, eco-friendly vehicles, smart factories, advanced materials, artificial intelligence (AI), and robots.

If Hyundai wants to be a “Smart Mobility Solution Provider” with a focus on vehicles, it really seems like there’s a whole bunch of other ways they could have spent most of a billion dollars that would get them there quicker. Will Boston Dynamics’ expertise help them develop autonomous driving technology? Sure, I guess, but why not just buy an autonomous car startup instead? Boston Dynamics is more about “robots,” which happens to be dead last on the list above.

There was some speculation a couple of weeks ago that Hyundai was going to try and leverage Boston Dynamics to make a real version of this hybrid wheeled/legged concept car, so if that’s what Hyundai means by “Smart Mobility Solution Provider,” then I suppose the Boston Dynamics acquisition makes more sense. Still, I think that’s unlikely, because it’s just a concept car, after all.

In addition to “smart mobility,” which seems like a longer-term goal for Hyundai, the company also mentions other, more immediate benefits from the acquisition: 

Advanced robotics offer opportunities for rapid growth with the potential to positively impact society in multiple ways. Boston Dynamics is the established leader in developing agile, mobile robots that have been successfully integrated into various business operations. The deal is also expected to allow Hyundai Motor Group and Boston Dynamics to leverage each other’s respective strengths in manufacturing, logistics, construction and automation.

“Successfully integrated” might be a little optimistic here. They’re talking about Spot, of course, but I think the best you could say at this point is that Spot is in the middle of some promising pilot projects. Whether it’ll be successfully integrated in the sense that it’ll have long-term commercial usefulness and value remains to be seen. I’m optimistic about this as well, but Spot is definitely not there yet.

What does probably hold a lot of value for Hyundai is getting Spot, Pick, and perhaps even Handle into that “manufacturing, logistics, construction” stuff. This is the bread and butter for robots right now, and Boston Dynamics has plenty of valuable technology to offer in those spaces.

Photo: Bob O’Connor Boston Dynamics is selling Spot for $74,500, shipping included. Betting on Spot and Pick

With Boston Dynamics founder Marc Raibert’s transition to Chairman of the company, the CEO position is now occupied by Robert Playter, the long-time VP of engineering and more recently COO at Boston Dynamics. Here’s his statement from the release:

“Boston Dynamics’ commercial business has grown rapidly as we’ve brought to market the first robot that can automate repetitive and dangerous tasks in workplaces designed for human-level mobility. We and Hyundai share a view of the transformational power of mobility and look forward to working together to accelerate our plans to enable the world with cutting edge automation, and to continue to solve the world’s hardest robotics challenges for our customers.”

Whether Spot is in fact “the first robot that can automate repetitive and dangerous tasks in workplaces designed for human-level mobility” on the market is perhaps something that could be argued against, although I won’t. Whether or not it was the first robot that can do these kinds of things, it’s definitely not the only robot that do these kinds of things, and going forward, it’s going to be increasingly challenging for Spot to maintain its uniqueness.

For a long time, Boston Dynamics totally owned the quadruped space. Now, they’re one company among many—ANYbotics and Unitree are just two examples of other quadrupeds that are being successfully commercialized. Spot is certainly very capable and easy to use, and we shouldn’t underestimate the effort required to create a robot as complex as Spot that can be commercially used and supported. But it’s not clear how long they’ll maintain that advantage, with much more affordable platforms coming out of Asia, and other companies offering some unique new capabilities.

Photo: Boston Dynamics Boston Dynamics’ Handle is an all-electric robot featuring a leg-wheel hybrid mobility system, a manipulator arm with a vacuum gripper, and a counterbalancing tail.

Boston Dynamics’ picking system, which stemmed from their 2019 acquisition of Kinema Systems, faces the same kinds of challenges—it’s very good, but it’s not totally unique.

Boston Dynamics produces highly capable mobile robots with advanced mobility, dexterity and intelligence, enabling automation in difficult, dangerous, or unstructured environments. The company launched sales of its first commercial robot, Spot in June of 2020 and has since sold hundreds of robots in a variety of industries, such as power utilities, construction, manufacturing, oil and gas, and mining. Boston Dynamics plans to expand the Spot product line early next year with an enterprise version of the robot with greater levels of autonomy and remote inspection capabilities, and the release of a robotic arm, which will be a breakthrough in mobile manipulation.

Boston Dynamics is also entering the logistics automation market with the industry leading Pick, a computer vision-based depalletizing solution, and will introduce a mobile robot for warehouses in 2021.

Huh. We’ll be trying to figure out what “greater levels of autonomy” means, as well as whether the “mobile robot for warehouses” is Handle, or something more like an autonomous mobile robot (AMR) platform. I’d honestly be surprised if Handle was ready for work outside of Boston Dynamics next year, and it’s hard to imagine how Boston Dynamics could leverage their expertise into the AMR space with something that wouldn’t just seem… Dull, compared to what they usually do. I hope to be surprised, though!

A new deep-pocketed benefactor

Hyundai Motor Group’s decision to acquire Boston Dynamics is based on its growth potential and wide range of capabilities.

“Wide range of capabilities” we get, but that other phrase, “growth potential,” has a heck of a lot wrapped up in it. At the moment, Boston Dynamics is nowhere near profitable, as far as we know. SoftBank acquired Boston Dynamics in 2017 for between one hundred and two hundred million, and over the last three years they’ve poured hundreds of millions more into Boston Dynamics.

Hyundai’s 80 percent stake just means that they’ll need to take over the majority of that support, and perhaps even increase it if Boston Dynamics’ growth is one of their primary goals. Hyundai can’t have a reasonable expectation that Boston Dynamics will be profitable any time soon; they’re selling Spots now, but it’s an open question whether Spot will manage to find a scalable niche in which it’ll be useful in the sort of volume that will make it a sustainable commercial success. And even if it does become a success, it seems unlikely that Spot by itself will make a significant dent in Boston Dynamics’ burn rate anytime soon. Boston Dynamics will have more products of course, but it’s going to take a while, and Hyundai will need to support them in the interim.

Depending on whether Hyundai views Boston Dynamics as a company that does research or a company that makes robots that are useful and profitable, it may be difficult for Boston Dynamics to justify the cost to develop the  next Atlas, when the  current one still seems so far from commercialization

It’s become clear that to sustain itself, Boston Dynamics needs a benefactor with very deep pockets and a long time horizon. Initially, Boston Dynamics’ business model (or whatever you want to call it) was to do bespoke projects for defense-ish folks like DARPA, but from what we understand Boston Dynamics stopped that sort of work after Google acquired them back in 2013. From one perspective, that government funding did exactly what it was supposed to do, which was to fund the development of legged robots through low TRLs (technology readiness levels) to the point where they could start to explore commercialization.

The question now, though, is whether Hyundai is willing to let Boston Dynamics undertake the kinds of low-TRL, high-risk projects that led from BigDog to LS3 to Spot, and from PETMAN to DRC Atlas to the current Atlas. So will Hyundai be cool about the whole thing and be the sort of benefactor that’s willing to give Boston Dynamics the resources that they need to keep doing what they’re doing, without having to answer too many awkward questions about things like practicality and profitability? Hyundai can certainly afford to do this, but so could SoftBank, and Google—the question is whether Hyundai will want to, over the length of time that’s required for the development of the kind of ultra-sophisticated robotics hardware that Boston Dynamics specializes in.

To put it another way: Depending whether Hyundai’s perspective on Boston Dynamics is as a company that does research or a company that makes robots that are useful and profitable, it may be difficult for Boston Dynamics to justify the cost to develop the next Atlas, when the current one still seems so far from commercialization.

Google, SoftBank, now Hyundai

Boston Dynamics possesses multiple key technologies for high-performance robots equipped with perception, navigation, and intelligence. 

Hyundai Motor Group’s AI and Human Robot Interaction (HRI) expertise is highly synergistic with Boston Dynamics’s 3D vision, manipulation, and bipedal/quadruped expertise.

As it turns out, Hyundai Motors does have its own robotics lab, called Hyundai Motors Robotics Lab. Their website is not all that great, but here’s a video from last year:

I’m not entirely clear on what Hyundai means when they use the word “synergistic” when they talk about their robotics lab and Boston Dynamics, but it’s a little bit concerning. Usually, when a big company buys a little company that specializes in something that the big company is interested in, the idea is that the little company, to some extent, will be absorbed into the big company to give them some expertise in that area. Historically, however, Boston Dynamics has been highly resistant to this, maintaining its post-acquisition independence and appearing to be very reluctant to do anything besides what it wants to do, at whatever pace it wants to do it, and as by itself as possible.

From what we understand, Boston Dynamics didn’t integrate particularly well with Google’s robotics push in 2013, and we haven’t seen much evidence that SoftBank’s experience was much different. The most direct benefit to SoftBank (or at least the most visible one) was the addition of a fleet of Spot robots to the SoftBank Hawks baseball team cheerleading squad, along with a single (that we know about) choreographed gymnastics routine from an Atlas robot that was only shown on video.

And honestly, if you were a big manufacturing company with a bunch of money and you wanted to build up your own robotics program quickly, you’d probably have much better luck picking up some smaller robotics companies who were a bit less individualistic and would probably be more amenable to integration and would cost way less than a billion dollars-ish. And if integration is ultimately Hyundai’s goal, we’ll be very sad, because it’ll likely signal the end of Boston Dynamics doing the unfettered crazy stuff that we’ve grown to love.

Photo: Bob O’Connor Possibly the most agile humanoid robot ever built, Atlas can run, climb, jump over obstacles, and even get up after a fall. Boston Dynamics contemplates its future

The release ends by saying that the transaction is “subject to regulatory approvals and other customary closing conditions” and “is expected to close by June of 2021.” Again, you can read the whole thing here.

My initial reaction is that, despite the “synergies” described by Hyundai, it’s certainly not immediately obvious why the company wants to own 80 percent of Boston Dynamics. I’d also like a better understanding of how they arrived at the $1.1 billion valuation. I’m not saying this because I don’t believe in what Boston Dynamics is doing or in the inherent value of the company, because I absolutely do, albeit perhaps in a slightly less tangible sense. But when you start tossing around numbers like these, a big pile of expectations inevitably comes along with them. I hope that Boston Dynamics is unique enough that the kinds of rules that normally apply to robotics companies (or companies in general) can be set aside, at least somewhat, but I also worry that what made Boston Dynamics great was the explicit funding for the kinds of radical ideas that eventually resulted in robots like Atlas and Spot.

Can Hyundai continue giving Boston Dynamics the support and freedom that they need to keep doing the kinds of things that have made them legendary? I certainly hope so.

As much as we like to go on about bio-inspired robots (and we do go on about them), there are some things that nature hasn’t quite figured out yet. Wheels are almost one of those things—while some animals do roll, and have inspired robots based on that rolling, true wheeled motion isn’t found in nature above the microscopic level. When humans figured out how useful wheels were, we (among other things) strapped them to our feet to make our motion more efficient under certain conditions, which really showed nature who was boss. Our smug wheeled superiority hasn’t lasted very long, though, because robots are rapidly becoming more skilled with wheels than we can ever hope to be.

The key difference between a human on roller skates and a robot on actuated wheels is that the robot, if it’s engineered properly, can exert control over its wheels with a nuance that we’ll never be able to match. We’ve seen this in action with Boston Dynamics’ Handle, Handle, although so far, Handle hasn’t seemed to take full advantage of the fact that it’s got legs, too. To understand why wheels and legs together are such a game-changer for robotic mobility, we can take a look at ANYmal, which seamlessly blends four legs and four wheels together with every movement it makes.

The really cool thing here is that ANYmal is dynamically choosing an optimal hybrid gait that’s a fusion of powered rolling and legged stepping. It’s doing this “blind,” without any camera or lidar inputs, just based on the feel of the terrain underneath its wheels. You can see how it transitions seamlessly between rolling and stepping, even mid-stride, based on how much utility the wheeled motion has on a per-leg basis—if a wheel stops being efficient, the controller switches that leg to a stepping motion instead, while maintaining coordination with the other legs. Overall, this makes ANYmal move more quickly without reducing its ability to handle challenging terrain, and reduces its cost of transport since rolling is much more efficient than walking.

For more details, we spoke with Marko Bjelonic from ETH Zurich.

IEEE Spectrum: Are there certain kinds of terrain that make these ANYmal’s gait transitions particularly challenging?

Marko Bjelonic: Aperiodic gait sequences are automatically found through kinematic leg utilities without the need for predefined gait timings. Based on the robot's current situation, each leg can reason on its own when it is a good time to lift off the ground. Our approach works quite well in rough terrain, but more considerable obstacles, e.g., stairs, are challenging.

How much of a difference do you think incorporating sensors to identify terrain would make to ANYmal’s capability?

Our submitted publication is only based on proprioceptive signals, i.e., no terrain perception is used to make gait transitions based on the perceived environment. We are surprised how well this framework already works on flat and uneven terrain. However, we are currently working on an extension that considers the terrain upfront for the robot to plan the stepping sequences. This terrain-responsive extension is capable of handling also larger obstacles like stairs.

“My experience shows me that the current version of ANYmal with actuated wheels improves mobility drastically. And I believe that these kinds of robots will outperform nature first. There is no animal or human being that can exploit such a concept.” —Marko Bjelonic, ETH Zurich

How many degrees of freedom do you think are optimal for a hybrid robot like ANYmal? For example, if the wheels could be steerable, would that be beneficial? 

It is a nice challenge to have no steerable wheels, because then the robot is forced to explore hybrid roller-walking motions. From an application perspective, it would be beneficial to have the possibility of steering the wheels. We already analyzed the leg configuration and the amount of actuation per leg and found that no additional degrees of freedom are necessary to achieve this. We can rotate the first actuator, the hip adduction/abduction, and without increasing the complexity, we increase the robot's mobility and add the possibility of steering the wheels.

What are the disadvantages of hybrid mobility? Why shouldn’t every legged robot also have wheels?

Every legged robot should have wheels! I think it’s going to be more common in the future. There are currently only a few hybrid mobility concepts out there, e.g., the roller-walking ANYmal, the CENTAURO robot, and Handle from Boston Dynamics. The additional degrees of freedom and missing counterparts in nature make designing locomotion capabilities for wheeled-legged robots more challenging. This is one reason why we do not see more of these creatures. But I am sure that more concepts will follow with the current advancements in this field.

What are you working on next?

We are working on an artistic framework enabling the robot more complex motions on the ground and over challenging obstacles. The challenge here is how to find optimal maneuvers for such high-dimensional problems and how to execute these motions on the real robot robustly.

“Whole-Body MPC and Online Gait Sequence Generation for Wheeled-Legged Robots,” by Marko Bjelonic, Ruben Grandia, Oliver Harley, Cla Galliard, Samuel Zimmermann, and Marco Hutter from ETH Zürich, is available on arXiv.​

Sonar, which measures the time it takes for sound waves to bounce off objects and travel back to a receiver, is the best way to visualize underwater terrain or inspect marine-based structures. Sonar systems, though, have to be deployed on ships or buoys, making them slow and limiting the area they can cover.

However, engineers at Stanford University have developed a new hybrid technique combining light and sound. Aircraft, they suggest, could use this combined laser/sonar technology to sweep the ocean surface for high-resolution images of submerged objects. The proof-of-concept airborne sonar system, presented recently in the journal IEEE Access, could make it easier and faster to find sunken wrecks, investigate marine habitats, and spot enemy submarines.

“Our system could be on a drone, airplane or helicopter,” says Amin Arbabian, an electrical engineering professor at Stanford University. “It could be deployed rapidly…and cover larger areas.”

Airborne radar and lidar are used to map the Earth’s surface at high resolution. Both can penetrate clouds and forest cover, making them especially useful in the air and on the ground. But peering into water from the air is a different challenge. Sound, radio, and light waves all quickly lose their energy when traveling from air into water and back. This attenuation is even worse in turbid water, Arbabian says.

So he and his students combined the two modalities—laser and sonar. Their system relies on the well-known photoacoustic effect, which turns pulses of light into sound. “When you shine a pulse of light on an object it heats up and expands and that leads to a sound wave because it moves molecules of air around the object,” he says.

The group’s new photoacoustic sonar system begins by shooting laser pulses at the water surface. Water absorbs most of the energy, creating ultrasound waves that move through it much like conventional sonar. These waves bounce off objects, and some of the reflected waves go back out from the water into the air.

At this point, the acoustic echoes lose a tremendous amount of energy as they cross that water-air barrier and then travel through the air. Here is where another critical part of the team’s design comes in.

Image: Aidan Fitzpatrick

To detect the weak acoustic waves in air, the team uses an ultra-sensitive microelectromechanical device with the mouthful name of an air-coupled capacitive micromachined ultrasonic transducer (CMUT). These devices are simple capacitors with a thin plate that vibrates when hit by ultrasound waves, causing a detectable change in capacitance. They are known to be efficient at detecting sound waves in air, and Arbabian has been investigating the use of CMUT sensors for remote ultrasound imaging. Special software processes the detected ultrasound signals to reconstruct a high-resolution 3D image of the underwater object.

Gif: Aidan Fitzpatrick An animation showing the 3D image of the submerged object recreated using reflected ultrasound waves.

The researchers tested the system by imaging metal bars of different heights and diameters placed in a large 25cm-deep fish tank filled with clear water. The CMUT detector was 10cm above the water surface.

The system should work in murky water, Arbabian says, although they haven’t tested that yet. Next up, they plan to image objects placed in a swimming pool, for which they will have to use more powerful laser sources that work for deeper water. They also want to improve the system so it works with waves, which distort signals and make the detection and image reconstruction much harder. “This proof of concept is to show that you can see through the air-water interface” Arbabian says. “That’s the hardest part of this problem. Once we can prove it works it can scale up to greater depths and larger objects.”

Any successful implementation of artificial intelligence hinges on asking the right questions in the right way. That’s what the British AI company DeepMind (a subsidiary of Alphabet) accomplished when it used its neural network to tackle one of biology’s grand challenges, the protein-folding problem. Its neural net, known as AlphaFold, was able to predict the 3D structures of proteins based on their amino acid sequences with unprecedented accuracy. 

AlphaFold’s predictions at the 14th Critical Assessment of protein Structure Prediction (CASP14) were accurate to within an atom’s width for most of the proteins. The competition consisted of blindly predicting the structure of proteins that have only recently been experimentally determined—with some still awaiting determination.

Called the building blocks of life, proteins consist of 20 different amino acids in various combinations and sequences. A protein's biological function is tied to its 3D structure. Therefore, knowledge of the final folded shape is essential to understanding how a specific protein works—such as how they interact with other biomolecules, how they may be controlled or modified, and so on. “Being able to predict structure from sequence is the first real step towards protein design,” says Janet M. Thornton, director emeritus of the European Bioinformatics Institute. It also has enormous benefits in understanding disease-causing pathogens. For instance, at the moment only about 18 of the 26 proteins in the SARS-CoV-2 virus are known.

Predicting a protein’s 3D structure is a computational nightmare. In 1969 Cyrus Levinthal estimated that there are 10300 possible conformational combinations for a single protein, which would take longer than the age of the known universe to evaluate by brute force calculation. AlphaFold can do it in a few days.

As scientific breakthroughs go, AlphaFold’s discovery is right up there with the likes of James Watson and Francis Crick’s DNA double-helix model, or, more recently, Jennifer Doudna and Emmanuelle Charpentier’s CRISPR-Cas9 genome editing technique.

How did a team that just a few years ago was teaching an AI to master a 3,000-year-old game end up training one to answer a question plaguing biologists for five decades? That, says Briana Brownell, data scientist and founder of the AI company PureStrategy, is the beauty of artificial intelligence: The same kind of algorithm can be used for very different things. 

“Whenever you have a problem that you want to solve with AI,” she says, “you need to figure out how to get the right data into the model—and then the right  sort of output that you can translate back into the real world.” 

DeepMind’s success, she says, wasn’t so much a function of picking the right neural nets but rather “how they set up the problem in a sophisticated enough way that the neural network-based modeling [could] actually answer the question.”

AlphaFold showed promise in 2018, when DeepMind introduced a previous iteration of their AI at CASP13, achieving the highest accuracy among all participants. The team had trained its to model target shapes from scratch, without using previously solved proteins as templates.

For 2020 they deployed new deep learning architectures into the AI, using an attention-based model that was trained end-to-end. Attention in a deep learning network refers to a component that manages and quantifies the interdependence between the input and output elements, as well as between the input elements themselves. 

The system was trained on public datasets of the approximately 170,000 known experimental protein structures in addition to databases with protein sequences of unknown structures. 

“If you look at the difference between their entry two years ago and this one, the structure of the AI system was different,” says Brownell. “This time, they’ve figured out how to translate the real world into data … [and] created an output that could be translated back into the real world.”

Like any AI system, AlphaFold may need to contend with biases in the training data. For instance, Brownell says, AlphaFold is using available information about protein structure that has been measured in other ways. However, there are also many proteins with as yet unknown 3D structures. Therefore, she says, a bias could conceivably creep in toward those kinds of proteins that we have more structural data for. 

Thornton says it’s difficult to predict how long it will take for AlphaFold’s breakthrough to translate into real-world applications.

“We only have experimental structures for about 10 per cent of the 20,000 proteins [in] the human body,” she says. “A powerful AI model could unveil the structures of the other 90 per cent.”

Apart from increasing our understanding of human biology and health, she adds, “it is the first real step toward… building proteins that fulfill a specific function. From protein therapeutics to biofuels or enzymes that eat plastic, the possibilities are endless.”

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

ACRA 2020 – December 8-10, 2020 – [Online]

Let us know if you have suggestions for next week, and enjoy today’s videos.

Another BIG step for Japan’s Gundam project.

[ Gundam Factory ]

We present an interactive design system that allows users to create sculpting styles and fabricate clay models using a standard 6-axis robot arm. Given a general mesh as input, the user iteratively selects sub-areas of the mesh through decomposition and embeds the design expression into an initial set of toolpaths by modifying key parameters that affect the visual appearance of the sculpted surface finish. We demonstrate the versatility of our approach by designing and fabricating different sculpting styles over a wide range of clay models.

[ Disney Research ]

China’s Chang’e-5 completed the drilling, sampling and sealing of lunar soil at 04:53 BJT on Wednesday, marking the first automatic sampling on the Moon, the China National Space Administration (CNSA) announced Wednesday.

[ CCTV ]

Red Hat’s been putting together an excellent documentary on Willow Garage and ROS, and all five parts have just been released. We posted Part 1 a little while ago, so here’s Part 2 and Part 3.

Parts 4 and 5 are at the link below!

[ Red Hat ]

Congratulations to ANYbotics on a well-deserved raise!

ANYbotics has origins in the Robotic Systems Lab at ETH Zurich, and ANYmal’s heritage can be traced back at least as far as StarlETH, which we first met at ICRA 2013.

[ ANYbotics ]

Most conventional robots are working with 0.05-0.1mm accuracy. Such accuracy requires high-end components like low-backlash gears, high-resolution encoders, complicated CNC parts, powerful motor drives, etc. Those in combination end up an expensive solution, which is either unaffordable or unnecessary for many applications. As a result, we found the Apicoo Robotics to provide our customers solutions with a much lower cost and higher stability.

[ Apicoo Robotics ]

The Skydio 2 is an incredible drone that can take incredible footage fully autonomously, but it definitely helps if you do incredible things in incredible places.

[ Skydio ]

Jueying is the first domestic sensitive quadruped robot for industry applications and scenarios. It can coordinate (replace) humans to reach any place that can be reached. It has superior environmental adaptability, excellent dynamic balance capabilities and precise Environmental perception capabilities. By carrying functional modules for different application scenarios in the safe load area, the mobile superiority of the quadruped robot can be organically integrated with the commercialization of functional modules, providing smart factories, smart parks, scene display and public safety application solutions.

[ DeepRobotics ]

We have developed semi-autonomous quadruped robot, called LASER-D (Legged-Agile-Smart-Efficient Robot for Disinfection) for performing disinfection in cluttered environments. The robot is equipped with a spray-based disinfection system and leverages the body motion to controlling the spray action without the need for an extra stabilization mechanism. The system includes an image processing capability to verify disinfected regions with high accuracy. This system allows the robot to successfully carry out effective disinfection tasks while safely traversing through cluttered environments, climb stairs/slopes, and navigate on slippery surfaces.

[ USC Viterbi ]

We propose the “multi-vision hand”, in which a number of small high-speed cameras are mounted on the robot hand of a common 7 degrees-of-freedom robot. Also, we propose visual-servoing control by using a multi-vision system that combines the multi-vision hand and external fixed high-speed cameras. The target task was ball catching motion, which requires high-speed operation. In the proposed catching control, the catch position of the ball, which is estimated by the external fixed high-speed cameras, is corrected by the multi-vision hand in real-time.

More details available through IROS on-demand.

[ Namiki Laboratory ]

Shunichi Kurumaya wrote in to share his work on PneuFinger, a pneumatically actuated compliant robotic gripping system.

[ Nakamura Lab ]

Thanks Shunichi!

Motivated by insights into the human teaching process, we introduce a method for incorporating unstructured natural language into imitation learning. At training time, the expert can provide demonstrations along with verbal descriptions in order to describe the underlying intent, e.g., ``Go to the large green bowl’’. The training process, then, interrelates the different modalities to encode the correlations between language, perception, and motion. The resulting language-conditioned visuomotor policies can be conditioned at run time on new human commands and instructions, which allows for more fine-grained control over the trained policies while also reducing situational ambiguity.

[ ASU ]

Thanks Heni!

Gita is on sale for the holidays for only $2,000.

[ Gita ]

This video introduces a computational approach for routing thin artificial muscle actuators through hyperelastic soft robots, in order to achieve a desired deformation behavior. Provided with a robot design, and a set of example deformations, we continuously co-optimize the routing of actuators, and their actuation, to approximate example deformations as closely as possible.

[ Disney Research ]

Researchers and mountain rescuers in Switzerland are making huge progress in the field of autonomous drones as the technology becomes more in-demand for global search-and-rescue operations.

[ SWI ]

This short clip of the Ghost Robotics V60 features an interesting, if awkward looking, righting behavior at the end.

[ Ghost Robotics ]

Europe’s Rosalind Franklin ExoMars rover has a younger ’sibling’, ExoMy. The blueprints and software for this mini-version of the full-size Mars explorer are available for free so that anyone can 3D print, assemble and program their own ExoMy.

[ ESA ]

The holiday season is here, and with the added impact of Covid-19 consumer demand is at an all-time high. Berkshire Grey is the partner that today’s leading organizations turn to when it comes to fulfillment automation.

[ Berkshire Grey ]

Until very recently, the vast majority of studies and reports on the use of cargo drones for public health were almost exclusively focused on the technology. The driving interest from was on the range that these drones could travel, how much they could carry and how they worked. Little to no attention was placed on the human side of these projects. Community perception, community engagement, consent and stakeholder feedback were rarely if ever addressed. This webinar presents the findings from a very recent study that finally sheds some light on the human side of drone delivery projects.

[ WeRobotics ]

Although the in-person Systems Track event of the DARPA SubT Challenge was cancelled because of the global pandemic, the Systems Track teams still have to prepare for the Final Event in 2021, which will include a cave component. Systems Track teams have been on their own to find cave environments to test in, and many of them are running their own DARPA-style competitions to test their software and hardware.

We’ll be posting a series of interviews exploring where and how the teams are making this happen, and today we’re featuring Team CSIRO Data 61, based in Brisbane, Australia.

This interview features the following roboticists from Team CSIRO Data 61:

  • Katrina Lo Surdo—Electrical and Computer Engineer, Advanced Mechatronics Systems

  • Nicolas Hudson—Senior Principal Research Scientist, Group Leader 

  • Navinda Kottege—Principal Research Scientist, Dynamic Platforms Team Leader

  • Fletcher Talbot—Software Engineer, Dynamic Platforms and Primary Robot Operator

IEEE Spectrum: Tell me about your cave! How’d you find your cave, and what kind of cave was it?

Katrina Lo Surdo: We basically just sent a bunch of emails around to different caving clubs all across Australia asking if they knew where we could test our robots, and most of them said no. But this particular caving club in Chillagoe (a 20 hours’ drive north of Brisbane) said they knew of a good cave. The caves in Chillagoe used to be coral reefs—they were formed about 400 million years ago, and then over time the reefs turned into limestone and then that limestone eroded into caves. In the particular cave that we went to, although a lot of the formations and the actual sort of caverns themselves are formed by limestone, there’s a lot of sediment that has been deposited inside the caves so the floor is reasonably flat. And it’s got that red dirt feel that you think of when you think of Australia.

I do think this cave had a good mix of a lot of the elements that most caves would have. It did have some verticality, some massive caverns, and some really small constrained passageways. And it was really sprawling as well, so I think it was a good representation of a lot of different types of caves.

Were you looking for any cave you could find, or a cave that was particularly robot friendly?

Lo Surdo: We wanted to be able to succeed as much as possible, but the cave needed to provide enough of a challenge that it would be useful for us to go. So if it was going to be completely flat with no obstacles, I don’t think it would have been good. And another thing that would be looked at was whether the cave itself is fragile or anything, because obviously we’re rolling our robots around and we don’t want to be damaging it.

The terrain itself was quite extreme, although a human could walk through a large portion of it without difficulty.

Nicolas Hudson: We should add that Katrina is an experienced caver and an expert climber, so when she says it’s easily traversable by a human, she means that cavers find it easy. There were others on the team who were not comfortable at all in the cave.

What do you feel like the biggest new challenge was, going from an urban environment to a cave environment?

Hudson: My take going from the Urban Circuit to this cave was that at Urban, it was essentially set up so that a human, legged, or tracked system could traverse the entire thing. For example, at Urban, we flew a drone through a hole in the floor, but there was a staircase right next to it. In the cave, there were parts that were only drone-accessible.

Another good example is that our drone actually flew way beyond the course we expected at one point, because we don’t have any artificial constraints—it’s just the cave system. And it was flying through an area that we weren’t comfortable going as people. So, I think the cave system was really a place where the mobility of drones shines in certain areas even more so than urban environments. That was the most important difference from my perspective.

How did your team of robots change between Urban and Cave?

Hudson: Our robots didn’t change a lot. We kept the large Titan robots because they’re by far our most capable ground platform. In my opinion, they’re actually more capable than legs on slippery intense slopes because of the amount of grip that they have. There are things I wouldn’t walk up that the Titan can drive up. So that stayed as our primary platform.

While the larger platforms could cover a lot of ground and were very stable, the smaller tracked platforms, SuperDroid robots which are about a meter long, didn’t even function in the cave. Like, they went a meter and then the just traction wasn’t enough, because they were too small. We’ve started working a beefed-up small tracked platform that has a lot more grip. We decided not to push for legs in the cave. We have a Ghost Vision 60. And we thought about, do we go legged in this environment, and we decided not to because of how unstructured it was, and just because of the difficulty of human traversing it. 

I really think the big difference was the drone played a much larger role. Where in Urban the drone had this targeted investigation role where it would be sitting on the back of the Titans and it would take off and you’d send it up through a hole or something like that, in the cave, what we found ourselves doing was really using it to sort of scout because the ground was just so challenging. The cost to go 20 meters in a cave with a ground robot can be absurdly difficult. And so getting better situational awareness quickly with the drone was probably where the concept of operations changed more than the robots did.

Photo: Team CSIRO Data 61

With such extreme mobility challenges, why use ground robots at all? Why not just stick with drones?

Hudson: We found that perception was significantly better on the ground robots. The ground robots have four cameras, and so they’re running 360 vision the whole time for object detection. The drone was great as a scout, but it was really difficult for it to find objects because there are so many crevices that to look through every area with a drone is very time consuming and they run out of battery. And so it’s really the endurance of the ground robot and the better perception where they played their part. 

We used the drones to figure out the topological layout of the cave. We didn’t let the operators see the cave beforehand, and it’s sort of hard to comprehend—in Urban, the drone did quite well because there were these very geometric rooms and so you could sort of cover things with a gimbal camera. But in the cave there’s just so many strange structures, and you have very poor camera coverage with a single camera. 

When you’re using the drones and the ground robots together, how are the robots able to decide where it’s safe to go with that terrain variability?

Hudson: I’ll answer that with respect to our first couple mock-competition runs, where the robot operators didn’t have any prior knowledge of the cave. What happened is that once the drones did a scouting mission, the operator gets a reasonably good idea if there’s any constrictions or any large elevation changes. And then we spread out the ground robots to different areas and tried things. 

Our autonomy system went up some things we didn’t expect it to—we just thought it would say “don’t go there.” And in other cases there was a little ledge or a series of rocks that the autonomy system said “I don’t want to do that” but it looked traversable in the map. We have a sort of backup teleoperation mode where you can just command the velocity of the tracks. One time, that was beneficial, in that it actually went through something that the autonomy system didn’t. But the other two times, it ended up flipping the robot, and one of those times, it actually flipped the robot and crushed the drone.

So a real lesson learned is that it was incredibly hard for human operators to perceive what was traversable and what was not, even with 3D point clouds and cameras. My overwhelming impression was it was unbelievably difficult to predict, as a person, what was traversable by a robot. 

Lo Surdo: And the autonomy did a much better job at choosing a path.

So the autonomy was doing a better job than the human teleoperators, even in this complex environment?

Hudson: It’s a difficult question to answer. Half of the time, that’s absolutely correct: The robot was more capable than the human thought it would be. There were other times that I think a human with a teleoperation system standing right next to the robot could better understand things like crazy terrain formations or dust, and the robot just didn’t have that context. I think if I had to rank it, a person with a remote control right next to the robot is probably the gold standard. We never really had issues with that. Then but the autonomy was definitely better than someone at the base station with a little bit of latency.

And that’s much different than your experience with Tunnel or Urban, right? Where a human teleoperator could be both more efficient and safer than a fully autonomous robot?

Hudson: That’s right. 

What were some challenges that were unique to the cave?

Hudson: The cave terrain was a big mix of things. There was a dry river bed in parts of it, and then other parts of it had these rocks that look almost like coral. There were formations that drop from the ceiling, things that have grown up from the ground, and it was just this completely random distribution of obstacles that’s hard for a human to make up, if that makes sense. And we definitely saw the robots getting trapped once or twice by those kinds of things.

Every run that we had we ended up with our large ground robots flipped over at least once, and that almost always occurred because it slipped off a two meter drop when the terrain deformed underneath the robot. Because the Titans are so sturdily built, the perception pack was protected, and the entire setup could be turned back over and they kept working.

Lo Surdo: There was also quite a natural flow to the terrain, because that’s where people had traversed through, and I think in a lot of cases the autonomy did a pretty good job of picking its way through those obstacles, and following the path that the humans had taken to get to different places. That was impressive to me. 

Navinda Kottege: I think the randomness also may be related to the relatively poor performance of the operators, because in the other SubT circuits, the level of situational awareness they got from the sensors would be augmented by their prior experience. Even in Urban, if it’s a room, it’s a geometric shape, and the human operator can kind of fill in the blanks because they have some prior experience. In caves, since they haven’t experienced that kind of environment, with the patchy situational awareness they get from the sensors it’s very challenging to make assumptions about what the environment around the robot is like.

What kind of experience did you have as a robot operator during your mock Cave Circuit competition?

Fletcher Talbot: It was extremely difficult. We made some big assumptions which turned out to be very wrong about the terrain, because myself and the other operator were completely unaware of what the cave looked like—we didn’t see any photos or anything before we actually visited. And my internal idea of what it would look like was wrong initially, misinformed somewhat by some of the feedback we got back from point clouds and meshes, and then rudely awakened by going on a tour through the cave after our mock competition ended. 

“During our runs we saw some slopes that looked completely traversable and so we tried to send robots up those slopes—if I had known what those slopes actually looked like, I never would have done that. But the robots themselves were beasts, and just did stuff that we would never have thought possible.” —Fletcher Talbot, Team CSIRO Data 61

For example, during our runs we saw some slopes that looked completely traversable and so we tried to send robots up those slopes—if I had known what those slopes actually looked like, I never would have done that. But the robots themselves were beasts, and just did stuff that we would never have thought possible.

We definitely learned that operators can hamper the progress of the robots, because we don’t really know what we’re doing sometimes. My approach going through the different runs was to just let the robots be more autonomous, and just give them very high level commands rather than trying to do any kind of finessing into gaps and stuff. That was my takeaway— trust the autonomy. 

So the cave circuit has gotten you to trust your robots more?

Talbot: Yeah, some other stuff that robots did was insane. As operators we never would have expected them to be able to do it, or commanded them to do it in the first place.

Photo: Team CSIRO Data 61

What were the results of the competition that you held?

Lo Surdo: We made our best guess as to what DARPA would do in an environment like this, and hid artifacts around the cave in the way that we’ve seen them hide artifacts before. 

Hudson: We set up the staging area with a team of 14 people; we took a lot of people because it was only a 20-hour drive away [as opposed to a flight across the world]. The operators came in and only saw the staging area.

Talbot: We were blind to what the course was going to be like, or where the objects were. We only knew what objects were brought.

Kottege: We did four runs overall, dividing the cave into two courses, doing two runs each. 

Talbot: The performance was reasonably consistent, I think, throughout all the runs. It was always four or five objects detected, about half the ones that were placed on the course.

How are you feeling about the combined circuit for the SubT Final?

Kottege: I think we have some pretty good ideas of what we need to focus on, but there’s also this big question of how DARPA will set up the combined event. So, once that announcement is made, there will be some more tweaking of our approach.

This is probably true for other teams as well, but after we performed at Urban, we felt like if we got a chance to do Tunnel again, we’d be able to really ace it, because we’d improved that much. Similarly, once we did our cave testing, we’ve had a similar sentiment— that if we got a chance to do Urban again, we’d probably do far better. I think that’s a really good place to be at, but I’m sure DARPA has some interesting challenges in mind for the final.

Lo Surdo: I do think that us going to the cave gives us a bit of an advantage, because there’s some terrain that you can’t really make or simulate, and some of the stuff we learned was really valuable and I think we’ll really serve us in the next competition. One thing in particular was the way that our robots assessed risk—we went up some really crazy terrain which was amazing, but in some instances there was a really easy pathway right next to it. So assessing risk is something that we’re going to be looking at improving in the future.

Talbot: With the cave, it’s hard to gauge the difficulty level compared to what DARPA might have given us—whether we met that difficulty level or went way above it or maybe even undershot it, we don’t really know. But it’ll be very interesting to see what DARPA throws at us, or if they give us some indication of what they were going to give us the cave so we can sort of balance it and figure out whether we hit the mark.

Photo: Team CSIRO Data 61

Now that you’ve been through Tunnel and Urban and your own version of cave, do you feel like you’re approaching a generalizable solution for underground environments?

Kottege: I’m fairly confident that we are approaching a generalizable state where our robots can perform quite well in a given underground environment. 

Talbot: Yeah, I think we are getting there. I think it needs some refinement, but I think the key components are there. One of the benefits of doing these field trips, and hopefully we do more in the future, is that we don’t really know what we can’t do until we come across that obstacle in real life. And then we go, “oh crap, we’re not prepared for that!” But from all the test environments that we’ve been in, I think we have a somewhat generalizable solution.

Read more DARPA SubT coverage from IEEE Spectrum