Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Enjoy today’s videos!

[ MIT ]

[ Max Planck Institute ]

[ UT HCRL ]

[ Paper ]

Thanks, Nagamanikandan!

Animals (or at least, many animals) are squishy for a reason–it helps to manage safe environmental contact. Let’s make all robots squishy!

[ Paper ]

Thanks, Pham!

This short video shows an actuator from Ed Habtour at the University of Washington, modeled after the vertebrae of sea birds and snakes.

[ UW ]

Thanks, Sarah!

[ ARL ]

[ LANL ]

OK, once urban delivery drones can do this, maybe I’ll pay more attention to them.

[ HKUST ]

[ Verity ]

[ ACFR ]

I am including this video because growing tea is beautiful.

[ SUIND ]

[ Clearpath ]

This Tartan Planning Series talk on is from Sebastian Scherer, on “Informative Path Planning, Exploration, and Intent Prediction.”

[ Air Lab ]

This Stanford HAI Seminar is from Oussama Khatib, on “From Romeo and Juliet to OceanOnek; Deep-Sea Robotic Exploration.”

[ Stanford HAI ]

Without a lifetime of experience to build on like humans have (and totally take for granted), robots that want to learn a new skill often have to start from scratch. Reinforcement learning is a technique that lets robots learn new skills through trial and error, but especially in the case of learning end-to-end vision based control policies, it takes a lot of time because the real world is a weirdly-lit friction-filled obstacle-y mess that robots can’t understand without a frequently impractical amount of effort.

Roboticists at UC Berkeley have vastly sped up this process by doing the same kind of cheating that humans do—instead of starting from scratch, you start with some previous experience that helps get you going. By leveraging a “foundation model” that was pre-trained on robots driving themselves around, the researchers were able to get a small-scale robotic rally car to teach itself to race around indoor and outdoor tracks, matching human performance after just 20 minutes of practice.

That first pre-training stage happens at your leisure, by manually driving a robot (that isn’t necessarily the robot that will be doing the task that you care about) around different environments. The goal of doing this isn’t to teach the robot to drive fast around a course, but instead to teach it the basics of not running into stuff.

With that pre-trained “foundation model” in place, when you then move over to the little robotic rally car, it no longer has to start from scratch. Instead, you can plop it onto the course you want it to learn, drive it around once slowly to show it where you want it to go, and then let it go fully autonomous, training itself to drive faster and faster. With a low-resolution, front-facing camera and some basic state estimation, the robot attempts to reach the next checkpoint on the course as quickly as possible, leading to some interesting emergent behaviors:

The other clever bit here is the reset feature, which is necessary in real world training. When training in simulation, it’s super easy to reset a robot that fails, but outside of simulation, a failure can (by definition) end the training if the robot gets itself stuck somehow. That’s not a big deal if you want to spend all your time minding the robot while it learns, but if you have something better to do, the robot needs to be able to train autonomously from start to finish. In this case, if the robot hasn’t moved at least 0.5 meters in the previous three seconds, it knows that it’s stuck, and will execute a simple behavior of turning randomly, backing up, and then trying to drive forward again, which gets it unstuck eventually.

During indoor and outdoor experiments, the robot was able to learn aggressive driving comparable to a human expert after just 20 minutes of autonomous practice, which the researchers say “provides strong validation that deep reinforcement learning can indeed be a viable tool for learning real-world policies even from raw images, when combined with appropriate pre-training and implemented in the context of an autonomous training framework.” It’s going to take a lot more work to implement this sort of thing safely on a larger platform, but this little car is taking the first few laps in the right direction just as quickly as it possibly can.

FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing, by Kyle Stachowicz, Arjun Bhorkar, Dhruv Shah, Ilya Kostrikov, and Sergey Levine from UC Berkeley, is available on arXiv.

Without a lifetime of experience to build on like humans have (and totally take for granted), robots that want to learn a new skill often have to start from scratch. Reinforcement learning is a technique that lets robots learn new skills through trial and error, but especially in the case of learning end-to-end vision based control policies, it takes a lot of time because the real world is a weirdly-lit friction-filled obstacle-y mess that robots can’t understand without a frequently impractical amount of effort.

Roboticists at UC Berkeley have vastly sped up this process by doing the same kind of cheating that humans do—instead of starting from scratch, you start with some previous experience that helps get you going. By leveraging a “foundation model” that was pre-trained on robots driving themselves around, the researchers were able to get a small-scale robotic rally car to teach itself to race around indoor and outdoor tracks, matching human performance after just 20 minutes of practice.

That first pre-training stage happens at your leisure, by manually driving a robot (that isn’t necessarily the robot that will be doing the task that you care about) around different environments. The goal of doing this isn’t to teach the robot to drive fast around a course, but instead to teach it the basics of not running into stuff.

With that pre-trained “foundation model” in place, when you then move over to the little robotic rally car, it no longer has to start from scratch. Instead, you can plop it onto the course you want it to learn, drive it around once slowly to show it where you want it to go, and then let it go fully autonomous, training itself to drive faster and faster. With a low-resolution, front-facing camera and some basic state estimation, the robot attempts to reach the next checkpoint on the course as quickly as possible, leading to some interesting emergent behaviors:

The other clever bit here is the reset feature, which is necessary in real world training. When training in simulation, it’s super easy to reset a robot that fails, but outside of simulation, a failure can (by definition) end the training if the robot gets itself stuck somehow. That’s not a big deal if you want to spend all your time minding the robot while it learns, but if you have something better to do, the robot needs to be able to train autonomously from start to finish. In this case, if the robot hasn’t moved at least 0.5 meters in the previous three seconds, it knows that it’s stuck, and will execute a simple behavior of turning randomly, backing up, and then trying to drive forward again, which gets it unstuck eventually.

During indoor and outdoor experiments, the robot was able to learn aggressive driving comparable to a human expert after just 20 minutes of autonomous practice, which the researchers say “provides strong validation that deep reinforcement learning can indeed be a viable tool for learning real-world policies even from raw images, when combined with appropriate pre-training and implemented in the context of an autonomous training framework.” It’s going to take a lot more work to implement this sort of thing safely on a larger platform, but this little car is taking the first few laps in the right direction just as quickly as it possibly can.

FastRLAP: A System for Learning High-Speed Driving via Deep RL and Autonomous Practicing, by Kyle Stachowicz, Arjun Bhorkar, Dhruv Shah, Ilya Kostrikov, and Sergey Levine from UC Berkeley, is available on arXiv.

In the realm of artificial intelligence, bigger is supposed to be better. Neural networks with billions of parameters power everyday AI-based tools like ChatGPT and Dall-E, and each new large language model (LLM) edges out its predecessors in size and complexity. Meanwhile, at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), a group of researchers have been working on going small.

In recent research, they demonstrated the efficiency of a new kind of very small—20,000 parameter—machine-learning system called a liquid neural network. They showed that drones equipped with these excelled in navigating complex, new environments with precision, even edging out state-of-the art systems. The systems were able to make decisions that led them to a target in previously unexplored forests and city spaces, and they could do it in the presence of added noise and other difficulties.

Neural networks in typical machine-learning systems learn only during the training process. After that, their parameters are fixed. Liquid neural networks, explains Ramin Hasani, one of the CSAIL scientists, are a class of artificial intelligence systems that learn on the job, even after their training. In other words, they utilize “liquid” algorithms that continuously adapt to new information, such as a new environment, just like the brains of living organisms. “They are directly modeled after how neurons and synapses interact in biological brains,” Hasani says. In fact, their network architecture is inspired by the nervous system of living creatures called C. elegans, tiny worms commonly found in the soil.

“We can implement a liquid neural network that can drive a car, on a Raspberry Pi”. —Ramin Hasani, MIT’s CSAIL

The purpose of this experiment wasn’t just the robust autonomous navigation of a drone, Hasani says. “It was about testing the task-understanding capabilities of neural networks when they are deployed in our society as autonomous systems.”

As training data for the neural networks that would control the drone, the researchers used drone footage collected by a human pilot flying toward a target. “You expect the system to have learned to move towards the object,” Hasani says, without having defined what the object is, or provided any label about the environment. “The drone has to infer that the task is this: I want to move towards [the object].”

The team performed a series of experiments to test how learned navigational skills transferred to new, never-seen-before environments. They tested the system in many real-world settings, including in different seasons in a forest, and in an urban setting. The drones underwent range and stress tests, and the targets were rotated, occluded, set in motion, and more. Liquid neural networks were the only ones that could generalize to scenarios that they had never seen, without any fine-tuning, and could perform this task seamlessly and reliably.

The application of liquid neural networks to robotics could lead to more robust autonomous navigation systems, for search and rescue, wildlife monitoring, and deliveries, among other things. Smart mobility, according to Hasani, is going to be crucial as cities get denser, and the small size of these neural nets could be a huge advantage: “We can implement a liquid neural network that can drive a car, on a Raspberry Pi.”

But the researchers believe liquid neural networks could go even farther, becoming the future of decision making related to any kind of time series data processing, including video and language processing. Because liquid neural networks are sequence data-processing engines, they could predict financial and medical events. By processing vital signs, for example, models can be developed to predict the status of a patient in the ICU.

Over and above their other advantages, liquid neural networks also offer explainability and interpretability. In other words, they open the proverbial black box of the system’s decision-making process. “If I have only 34 neurons [in the drone system], I can literally go and figure out what is the function of each and every element,” says Hasani. That’s something that would be virtually impossible in a large-scale deep neural network. The smaller size of liquid neural nets also massively reduces the computational cost, and therefore the carbon footprints, of machine-learning models.

Hasani and his colleagues are looking for ways to improve liquid neural networks. “This paper covered a very controlled and straightforward kind of reasoning capability, but real-world interactions require more and more sophisticated reasoning problems,” he says. The team would like to design more complex tasks and test liquid neural networks to their limit, while also figuring out why liquid neural networks perform so much better than their competitors in reasoning tests.

Hasani explains liquid neural networks in this video:

Liquid Neural Networks | Ramin Hasani | TEDxMIT youtu.be

In the realm of artificial intelligence, bigger is supposed to be better. Neural networks with billions of parameters power everyday AI-based tools like ChatGPT and Dall-E, and each new large language model (LLM) edges out its predecessors in size and complexity. Meanwhile, at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL), a group of researchers have been working on going small.

In recent research, they demonstrated the efficiency of a new kind of very small—20,000 parameter—machine-learning system called a liquid neural network. They showed that drones equipped with these excelled in navigating complex, new environments with precision, even edging out state-of-the art systems. The systems were able to make decisions that led them to a target in previously unexplored forests and city spaces, and they could do it in the presence of added noise and other difficulties.

Neural networks in typical machine-learning systems learn only during the training process. After that, their parameters are fixed. Liquid neural networks, explains Ramin Hasani, one of the CSAIL scientists, are a class of artificial intelligence systems that learn on the job, even after their training. In other words, they utilize “liquid” algorithms that continuously adapt to new information, such as a new environment, just like the brains of living organisms. “They are directly modeled after how neurons and synapses interact in biological brains,” Hasani says. In fact, their network architecture is inspired by the nervous system of living creatures called C. elegans, tiny worms commonly found in the soil.

“We can implement a liquid neural network that can drive a car, on a Raspberry Pi”. —Ramin Hasani, MIT’s CSAIL

The purpose of this experiment wasn’t just the robust autonomous navigation of a drone, Hasani says. “It was about testing the task-understanding capabilities of neural networks when they are deployed in our society as autonomous systems.”

As training data for the neural networks that would control the drone, the researchers used drone footage collected by a human pilot flying toward a target. “You expect the system to have learned to move towards the object,” Hasani says, without having defined what the object is, or provided any label about the environment. “The drone has to infer that the task is this: I want to move towards [the object].”

The team performed a series of experiments to test how learned navigational skills transferred to new, never-seen-before environments. They tested the system in many real-world settings, including in different seasons in a forest, and in an urban setting. The drones underwent range and stress tests, and the targets were rotated, occluded, set in motion, and more. Liquid neural networks were the only ones that could generalize to scenarios that they had never seen, without any fine-tuning, and could perform this task seamlessly and reliably.

The application of liquid neural networks to robotics could lead to more robust autonomous navigation systems, for search and rescue, wildlife monitoring, and deliveries, among other things. Smart mobility, according to Hasani, is going to be crucial as cities get denser, and the small size of these neural nets could be a huge advantage: “We can implement a liquid neural network that can drive a car, on a Raspberry Pi.”

But the researchers believe liquid neural networks could go even farther, becoming the future of decision making related to any kind of time series data processing, including video and language processing. Because liquid neural networks are sequence data-processing engines, they could predict financial and medical events. By processing vital signs, for example, models can be developed to predict the status of a patient in the ICU.

Over and above their other advantages, liquid neural networks also offer explainability and interpretability. In other words, they open the proverbial black box of the system’s decision-making process. “If I have only 34 neurons [in the drone system], I can literally go and figure out what is the function of each and every element,” says Hasani. That’s something that would be virtually impossible in a large-scale deep neural network. The smaller size of liquid neural nets also massively reduces the computational cost, and therefore the carbon footprints, of machine-learning models.

Hasani and his colleagues are looking for ways to improve liquid neural networks. “This paper covered a very controlled and straightforward kind of reasoning capability, but real-world interactions require more and more sophisticated reasoning problems,” he says. The team would like to design more complex tasks and test liquid neural networks to their limit, while also figuring out why liquid neural networks perform so much better than their competitors in reasoning tests.

Hasani explains liquid neural networks in this video:

Liquid Neural Networks | Ramin Hasani | TEDxMIT youtu.be

Birds are able to do a lot thanks to their highly flexible necks, whether it be turning their heads around to groom their backs, looking in many different directions during flight, or accessing hard to reach nooks and crannies along the ground or in trees. Among all avian species, the ostrich stands out as one bird with a particularly sturdy and dexterous neck – qualities that are also appealing for robotic manipulators.

Using an accurate blueprint of the muscles and tendons in an ostrich’s neck, researchers in Japan have created a novel robotic manipulator called RobOstrich. They describe the device in a study published 6 April in IEEE Robotics and Automation Letters.

Researchers are interested in creating soft and flexible robotic manipulators that could easily bend into hard-to-reach places, but this comes with challenges. “From a robotics perspective, it is difficult to control such a structure,” explains Kazashi Nakano, a doctoral student at the Graduate School of Information Science and Technology, at the University of Tokyo. “We focused on the ostrich neck because of the possibility of discovering something new.”

His team first dissected the neck of an ostrich to understand the underlying network of tendons, muscle and bone that helps manipulate such a long and heavy body part, which weighs in at a hefty 3 kilograms. Whereas a human has seven vertebrae in their neck, an ostrich has more that double that. What’s more, each cervical vertebra bends in two directions, resulting in extremely high degrees of freedom.

Using this anatomical data, the researchers set about creating their RobOstrich manipulator by 3-D printing 17 vertebrae, which they connected with bearings. Bundles of piano wires were used to mimick the biological muscles between an ostrich’s vertebrae (intervertebral muscles), and rubber bands were used as ligaments at the base of the manipulator to provide tension. An electric motor then reels in the wires, generating tension to “flex” the manipulator’s muscles. In a series of experiments, RobOstrich was able to complete various reaching tasks, where it had to achieve different configurations in order to come into contact with an object.

IEEE Spectrum RobOstrich v3 www.youtube.com

Just like a real ostrich neck, the RobOstrich manipulator achieved a “rolling pattern,” where adjacent joints move in sequence while the head remains level with the ground. Nakano says he was surprised to find that this movement pattern can be achieved simply by adding tension to wires on just the underside of the neck alone, while the length of the wire on the backside remains constant — ie., doesn’t need to be pulled. In this way, the manipulator can achieve complex configurations with minimal effort.

“The flexible structure is difficult to control, but the advantage is that dexterous reaching movements can be achieved by introducing muscle arrangements and joint ranges of motion based on the ostrich’s anatomy,” says Nakano.

As currently configured, RoboOstrich can only move forward on a 2-D plane, but the researchers hope to achieve 3-D movement in the future.

“We aim to develop a controller that can perform reaching movements in an unstructured environment while colliding gently with it,” says Nakano.

Meanwhile, RobOstrich is not the only ostrich-inspired robot making headlines. Ostriches can also run at astounding speeds – covering 100 meters in just five seconds. Inspired by this feat, researchers at Oregon State University created Cassie, a two-legged speedster. Last September, Cassie set new records for the fastest bipedal robot; you can watch the video here.

Birds are able to do a lot thanks to their highly flexible necks, whether it be turning their heads around to groom their backs, looking in many different directions during flight, or accessing hard to reach nooks and crannies along the ground or in trees. Among all avian species, the ostrich stands out as one bird with a particularly sturdy and dexterous neck – qualities that are also appealing for robotic manipulators.

Using an accurate blueprint of the muscles and tendons in an ostrich’s neck, researchers in Japan have created a novel robotic manipulator called RobOstrich. They describe the device in a study published 6 April in IEEE Robotics and Automation Letters.

Researchers are interested in creating soft and flexible robotic manipulators that could easily bend into hard-to-reach places, but this comes with challenges. “From a robotics perspective, it is difficult to control such a structure,” explains Kazashi Nakano, a doctoral student at the Graduate School of Information Science and Technology, at the University of Tokyo. “We focused on the ostrich neck because of the possibility of discovering something new.”

His team first dissected the neck of an ostrich to understand the underlying network of tendons, muscle and bone that helps manipulate such a long and heavy body part, which weighs in at a hefty 3 kilograms. Whereas a human has seven vertebrae in their neck, an ostrich has more that double that. What’s more, each cervical vertebra bends in two directions, resulting in extremely high degrees of freedom.

Using this anatomical data, the researchers set about creating their RobOstrich manipulator by 3-D printing 17 vertebrae, which they connected with bearings. Bundles of piano wires were used to mimick the biological muscles between an ostrich’s vertebrae (intervertebral muscles), and rubber bands were used as ligaments at the base of the manipulator to provide tension. An electric motor then reels in the wires, generating tension to “flex” the manipulator’s muscles. In a series of experiments, RobOstrich was able to complete various reaching tasks, where it had to achieve different configurations in order to come into contact with an object.

IEEE Spectrum RobOstrich v3 www.youtube.com

Just like a real ostrich neck, the RobOstrich manipulator achieved a “rolling pattern,” where adjacent joints move in sequence while the head remains level with the ground. Nakano says he was surprised to find that this movement pattern can be achieved simply by adding tension to wires on just the underside of the neck alone, while the length of the wire on the backside remains constant — ie., doesn’t need to be pulled. In this way, the manipulator can achieve complex configurations with minimal effort.

“The flexible structure is difficult to control, but the advantage is that dexterous reaching movements can be achieved by introducing muscle arrangements and joint ranges of motion based on the ostrich’s anatomy,” says Nakano.

As currently configured, RoboOstrich can only move forward on a 2-D plane, but the researchers hope to achieve 3-D movement in the future.

“We aim to develop a controller that can perform reaching movements in an unstructured environment while colliding gently with it,” says Nakano.

Meanwhile, RobOstrich is not the only ostrich-inspired robot making headlines. Ostriches can also run at astounding speeds – covering 100 meters in just five seconds. Inspired by this feat, researchers at Oregon State University created Cassie, a two-legged speedster. Last September, Cassie set new records for the fastest bipedal robot; you can watch the video here.

Swarm robotics is a promising approach to control large groups of robots. However, designing the individual behavior of the robots so that a desired collective behavior emerges is still a major challenge. In recent years, many advances in the automatic design of control software for robot swarms have been made, thus making automatic design a promising tool to address this challenge. In this article, I highlight and discuss recent advances and trends in offline robot evolution, embodied evolution, and offline robot learning for swarm robotics. For each approach, I describe recent design methods of interest, and commonly encountered challenges. In addition to the review, I provide a perspective on recent trends and discuss how they might influence future research to help address the remaining challenges of designing robot swarms.

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Enjoy today’s videos!

THE EYE

[ Atonaton ]

Some real DARPA Grand Challenge vibes here, except where these robots are going, they don’t need roads.

UW Off-road Autonomous Driving - Complex Terrain to Fast Trail | DARPA-funded Research www.youtube.com

[ UW ]

[ IHMC ]

Thanks, William!

[ MIT ]

[ CHI 2023 ]

[ WVUIRL ]

It’s still kinda wild to me that quadrupeds are good enough now that there are multiple viable use cases for them.

[ Boston Dynamics ]

[ Neurobionics Lab ]

[ Cybathlon ]

[ Mobot ]

This CMU RI seminar is from Vandi Verma, on “Mars Robots and Robotics at NASA JPL.”

[ CMU RI ]

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Enjoy today’s videos!

THE EYE

[ Atonaton ]

Some real DARPA Grand Challenge vibes here, except where these robots are going, they don’t need roads.

UW Off-road Autonomous Driving - Complex Terrain to Fast Trail | DARPA-funded Research www.youtube.com

[ UW ]

[ IHMC ]

Thanks, William!

[ MIT ]

[ CHI 2023 ]

[ WVUIRL ]

It’s still kinda wild to me that quadrupeds are good enough now that there are multiple viable use cases for them.

[ Boston Dynamics ]

[ Neurobionics Lab ]

[ Cybathlon ]

[ Mobot ]

This CMU RI seminar is from Vandi Verma, on “Mars Robots and Robotics at NASA JPL.”

[ CMU RI ]

Introduction: Recent studies found that wearable exoskeletons can reduce physical effort and fatigue during squatting. In particular, subject-specific assistance helped to significantly reduce physical effort, shown by reduced metabolic cost, using human-in-the-loop optimization of the exoskeleton parameters. However, measuring metabolic cost using respiratory data has limitations, such as long estimation times, presence of noise, and user discomfort. A recent study suggests that foot contact forces can address those challenges and be used as an alternative metric to the metabolic cost to personalize wearable robot assistance during walking.

Methods: In this study, we propose that foot center of pressure (CoP) features can be used to estimate the metabolic cost of squatting using a machine learning method. Five subjects’ foot pressure and metabolic cost data were collected as they performed squats with an ankle exoskeleton at different assistance conditions in our prior study. In this study, we extracted statistical features from the CoP squat trajectories and fed them as input to a random forest model, with the metabolic cost as the output.

Results: The model predicted the metabolic cost with a mean error of 0.55 W/kg on unseen test data, with a high correlation (r = 0.89, p < 0.01) between the true and predicted cost. The features of the CoP trajectory in the medial-lateral direction of the foot (xCoP), which relate to ankle eversion-inversion, were found to be important and highly correlated with metabolic cost.

Conclusion: Our findings indicate that increased ankle eversion (outward roll of the ankle), which reflects a suboptimal squatting strategy, results in higher metabolic cost. Higher ankle eversion has been linked with the etiology of chronic lower limb injuries. Hence, a CoP-based cost function in human-in-the-loop optimization could offer several advantages, such as reduced estimation time, injury risk mitigation, and better user comfort.

Now that anyone, anywhere can get themselves a quadrupedal robot without having to apply for a major research grant, we’re seeing all kinds of fun research being done with our four-legged electromechanical friends. And by “fun research” I mean very serious research that is making valuable contributions towards practical robotics. But seriously, there are lots of important robotics problems that can be solved in fun and interesting ways; don’t let anyone tell you different, especially not the current United States ambassador to Turkey.

At the 2023 International Conference on Robotics and Automation (ICRA) slated to take place in London next month, three papers will be presented that report on the talents of quadrupedal robots and the researchers who teach them new things, including dribbling, catching, and traversing a balance beam.

Quadrupedal soccer robots have a long and noble history; for years, Sony Aibos were the standard platform at RoboCup. But quadrupeds have made some enormous four-legged strides since the late 1 990s and early 2000s. Now that basic quadrupedal mobility has been pretty well figured out, it’s time to get these robots doing fun stuff. In an upcoming ICRA paper, roboticists from MIT describe how they have taught a quadruped to dribble a soccer ball across rough terrain, which is actually really impressive for anyone who has tried to do this themselves.

Let’s just get this out of the way: for most of the world, we’re talking about football here. But the paper calls it soccer, so I’m going to call it soccer too. Whatever you call it, it’s the one with the round ball where most of the time a game is actually being played instead of the one with the pointy ball where most of the time people are just standing around not doing anything.

DribbleBot, a name given to an automaton whose functionality the paper describes as “Dexterous Ball Manipulation with a

Legged Robot,” is a Unitree Go1. The machine can dribble a soccer ball under the same real-world conditions as humans who don’t have access to an actual soccer field. For those of us who have experience playing zero-budget pick-up soccer wherever we won’t get yelled at, flat and smooth grass is often an unattainable luxury. The real world is unfortunately full of tree roots and rocks and gravel and snow and all kinds of other things that make soccer balls behave unpredictably—and give me knee problems. This is the kind of terrain that DribbleBot is learning to handle.

The robot is using only onboard sensing and computation for this task, and it was first trained extensively through reinforcement learning in simulation. There’s actually a lot going on with dribbling: as the paper says, “successful dribbling involves adjusting the leg swings to apply targeted forces while the robot moves, balances itself, and orients its position relative to a moving ball.” But if you can look past the soccer-specific aspect, the real problem that’s being solved here is legged locomotion while manipulating an occasionally adversarial object in the real world. This obviously opens up other potential applications. Even if soccer were the only application, though, I’d totally pick DribbleBot for my team.

DribbleBot: Dynamic Legged Manipulation in the Wild, by Yandong Ji, Gabriel B. Margolis, and Pulkit Agrawal from MIT, will be presented at ICRA 2023 in London.

I would argue that one of the most impressive things that animals (humans included) can do is catch. And we do it effortlessly—there’s a small object flying at you which you have to detect, track, estimate its trajectory, and then actuate a bunch of different muscles to make sure that your hand is in exactly the right place at the right time, and usually you only have a couple of seconds to make all of this happen. It’s amazing that we’re able to do it at all, so it’s understandable that this confluence of tasks makes catching an especially thorny problem for robots.

The biggest problem for robots in a task like this is the relatively short amount of time that they have to sense, think, and react. Conventional cameras make this problem worse, which is why the UZH researchers are instead relying on event cameras. We’ve written about event cameras a bunch, but basically, they’re a kind of camera that only detects movement, but can do so almost instantly. By drastically lowering perception latency relative to a traditional camera, the robot is able to detect, track, and estimate a catching location for a ball thrown from 4 meters away and traveling at up to 15 m/s.

The catching maneuver was trained in simulation, and run in real life on an ANYmal-C quadruped, which displays some impressive self-sacrificing behaviors like lunges. An overall success rate of 83 percent isn’t bad at all, and the researchers point out that this is just a “first working demo” and that there’s plenty of room for optimization. The really important thing here is giving quadrupedal robots new capabilities by adding event cameras to a sensing arsenal that’s been suck in stereo camera and lidar land for far too long. Especially considering the new dynamic skills that we’ve been seeing from quadrupeds recently, event cameras could unlock all kinds of new capabilities that depend on rapid perception of moving objects.

Event-based Agile Object Catching with a Quadrupedal Robot, by Benedek Forrai, Takahiro Miki, Daniel Gehrig, Marco Hutter, and Davide Scaramuzza from UZH, will be presented at ICRA 2023 in London.

Balancing is a skill that you’d think robots would excel at, because we can equip them with exquisitely sensitive pieces of hardware that can tell them how they’re moving with an astounding level of precision. But, a robot knowing exactly how out of balance it is is different from a robot robot being able to get itself back into balance. A problem that many (if not most) legged robots have when it comes to balancing is that they have a limited amount of ankle and foot actuation. Some humanoids have it, and you can see for yourself how important it is by taking off your shoes and standing on one foot—pay attention to the constant corrective motions coming from all of those teeny muscles in your ankle, foot, and toes. Even the most sophisticated humanoid robots don’t have that level of control, and with quadrupeds, they’ve usually only got pointy feet to work with. That’s why, when it comes to balancing, they need a little help.

Aww, just look at those adorable little steps! Unfortunately, the adorable little steps aren’t doing the job of keeping the robot from tipping over. For that, you can thank the reaction wheels mounted on its back. You’ll notice that the robot ambulates two legs at a time, meaning that only two legs are keeping it off the ground, and that’s not enough legs on the ground for the robot to keep itself stable. The reaction wheels compensate by spinning up and down to exert torque on the body of the robot, independently of its legs. If this seems like cheating to you, well, you can just think of the reaction wheels as the equivalent of a tail, which many animals (and a few robots) use as a supplemental control system.

The researchers suggest that a smaller and lighter version of these reaction wheels could be usefully integrated into many legged robot designs, and would help all of them to successfully cross balance beams. For the tiny minority of robots that don’t find themselves crossing balance beams full-time, reaction wheels would be an added source of stability, making them better able to (among other things) withstand the obligatory shoves and kicks that every single quadruped robot in a robotics lab has to endure.

Now that anyone, anywhere can get themselves a quadrupedal robot without having to apply for a major research grant, we’re seeing all kinds of fun research being done with our four-legged electromechanical friends. And by “fun research” I mean very serious research that is making valuable contributions towards practical robotics. But seriously, there are lots of important robotics problems that can be solved in fun and interesting ways; don’t let anyone tell you different, especially not the current United States ambassador to Turkey.

At the 2023 International Conference on Robotics and Automation (ICRA) slated to take place in London next month, three papers will be presented that report on the talents of quadrupedal robots and the researchers who teach them new things, including dribbling, catching, and traversing a balance beam.

Quadrupedal soccer robots have a long and noble history; for years, Sony Aibos were the standard platform at RoboCup. But quadrupeds have made some enormous four-legged strides since the late 1 990s and early 2000s. Now that basic quadrupedal mobility has been pretty well figured out, it’s time to get these robots doing fun stuff. In an upcoming ICRA paper, roboticists from MIT describe how they have taught a quadruped to dribble a soccer ball across rough terrain, which is actually really impressive for anyone who has tried to do this themselves.

Let’s just get this out of the way: for most of the world, we’re talking about football here. But the paper calls it soccer, so I’m going to call it soccer too. Whatever you call it, it’s the one with the round ball where most of the time a game is actually being played instead of the one with the pointy ball where most of the time people are just standing around not doing anything.

DribbleBot, a name given to an automaton whose functionality the paper describes as “Dexterous Ball Manipulation with a

Legged Robot,” is a Unitree Go1. The machine can dribble a soccer ball under the same real-world conditions as humans who don’t have access to an actual soccer field. For those of us who have experience playing zero-budget pick-up soccer wherever we won’t get yelled at, flat and smooth grass is often an unattainable luxury. The real world is unfortunately full of tree roots and rocks and gravel and snow and all kinds of other things that make soccer balls behave unpredictably—and give me knee problems. This is the kind of terrain that DribbleBot is learning to handle.

The robot is using only onboard sensing and computation for this task, and it was first trained extensively through reinforcement learning in simulation. There’s actually a lot going on with dribbling: as the paper says, “successful dribbling involves adjusting the leg swings to apply targeted forces while the robot moves, balances itself, and orients its position relative to a moving ball.” But if you can look past the soccer-specific aspect, the real problem that’s being solved here is legged locomotion while manipulating an occasionally adversarial object in the real world. This obviously opens up other potential applications. Even if soccer were the only application, though, I’d totally pick DribbleBot for my team.

DribbleBot: Dynamic Legged Manipulation in the Wild, by Yandong Ji, Gabriel B. Margolis, and Pulkit Agrawal from MIT, will be presented at ICRA 2023 in London.

I would argue that one of the most impressive things that animals (humans included) can do is catch. And we do it effortlessly—there’s a small object flying at you which you have to detect, track, estimate its trajectory, and then actuate a bunch of different muscles to make sure that your hand is in exactly the right place at the right time, and usually you only have a couple of seconds to make all of this happen. It’s amazing that we’re able to do it at all, so it’s understandable that this confluence of tasks makes catching an especially thorny problem for robots.

The biggest problem for robots in a task like this is the relatively short amount of time that they have to sense, think, and react. Conventional cameras make this problem worse, which is why the UZH researchers are instead relying on event cameras. We’ve written about event cameras a bunch, but basically, they’re a kind of camera that only detects movement, but can do so almost instantly. By drastically lowering perception latency relative to a traditional camera, the robot is able to detect, track, and estimate a catching location for a ball thrown from 4 meters away and traveling at up to 15 m/s.

The catching maneuver was trained in simulation, and run in real life on an ANYmal-C quadruped, which displays some impressive self-sacrificing behaviors like lunges. An overall success rate of 83 percent isn’t bad at all, and the researchers point out that this is just a “first working demo” and that there’s plenty of room for optimization. The really important thing here is giving quadrupedal robots new capabilities by adding event cameras to a sensing arsenal that’s been suck in stereo camera and lidar land for far too long. Especially considering the new dynamic skills that we’ve been seeing from quadrupeds recently, event cameras could unlock all kinds of new capabilities that depend on rapid perception of moving objects.

Event-based Agile Object Catching with a Quadrupedal Robot, by Benedek Forrai, Takahiro Miki, Daniel Gehrig, Marco Hutter, and Davide Scaramuzza from UZH, will be presented at ICRA 2023 in London.

Balancing is a skill that you’d think robots would excel at, because we can equip them with exquisitely sensitive pieces of hardware that can tell them how they’re moving with an astounding level of precision. But, a robot knowing exactly how out of balance it is is different from a robot robot being able to get itself back into balance. A problem that many (if not most) legged robots have when it comes to balancing is that they have a limited amount of ankle and foot actuation. Some humanoids have it, and you can see for yourself how important it is by taking off your shoes and standing on one foot—pay attention to the constant corrective motions coming from all of those teeny muscles in your ankle, foot, and toes. Even the most sophisticated humanoid robots don’t have that level of control, and with quadrupeds, they’ve usually only got pointy feet to work with. That’s why, when it comes to balancing, they need a little help.

Aww, just look at those adorable little steps! Unfortunately, the adorable little steps aren’t doing the job of keeping the robot from tipping over. For that, you can thank the reaction wheels mounted on its back. You’ll notice that the robot ambulates two legs at a time, meaning that only two legs are keeping it off the ground, and that’s not enough legs on the ground for the robot to keep itself stable. The reaction wheels compensate by spinning up and down to exert torque on the body of the robot, independently of its legs. If this seems like cheating to you, well, you can just think of the reaction wheels as the equivalent of a tail, which many animals (and a few robots) use as a supplemental control system.

The researchers suggest that a smaller and lighter version of these reaction wheels could be usefully integrated into many legged robot designs, and would help all of them to successfully cross balance beams. For the tiny minority of robots that don’t find themselves crossing balance beams full-time, reaction wheels would be an added source of stability, making them better able to (among other things) withstand the obligatory shoves and kicks that every single quadruped robot in a robotics lab has to endure.

Introduction: Laparoscopic surgery often relies on a fixed Remote Center of Motion (RCM) for robot mobility control, which assumes that the patient’s abdominal walls are immobile. However, this assumption is inaccurate, especially in collaborative surgical environments. In this paper, we present a force-based strategy for the mobility of a robotic camera-holder system for laparoscopic surgery based on a pivoting motion. This strategy re-conceptualizes the conventional mobility control paradigm of surgical robotics.

Methods: The proposed strategy involves direct control of the Tool Center Point’s (TCP) position and orientation without any constraints associated with the spatial position of the incision. It is based on pivoting motions to minimize contact forces between the abdominal walls and the laparoscope. The control directly relates the measured force and angular velocity of the laparoscope, resulting in the reallocation of the trocar, whose position becomes a consequence of the natural accommodation allowed by this pivoting.

Results: The effectiveness and safety of the proposed control were evaluated through a series of experiments. The experiments showed that the control was able to minimize an external force of 9 N to ±0.2 N in 0.7 s and reduce it to 2 N in just 0.3 s. Furthermore, the camera was able to track a region of interest by displacing the TCP as desired, leveraging the strategy’s property that dynamically constrains its orientation.

Discussion: The proposed control strategy has proven to be effective minimizing the risk caused by sudden high forces resulting from accidents and maintaining the field of view despite any movements in the surgical environment, such as physiological movements of the patient or undesired movements of other surgical instruments. This control strategy can be implemented for laparoscopic robots without mechanical RCMs, as well as commercial collaborative robots, thereby improving the safety of surgical interventions in collaborative environments.

Modern industrial applications of robotics such as small-series production and automated warehousing require versatile grippers, i.e., grippers that can pick up the widest possible variety of objects. These objects must often be grasped or placed inside a container, which limits the size of the gripper. In this article, we propose to combine the two most popular gripper technologies in order to maximise versatility: finger grippers and suction-cup (vacuum) grippers. Many researchers and a few companies have followed this same idea in the past, but their gripper designs are often overly complex or too bulky to pick up objects inside containers. Here, we develop a gripper where the suction cup is lodged inside the palm of a two-finger robotic hand. The suction cup is mounted on a retractile rod that can extend to pick up objects inside containers without interference from the two fingers. A single actuator drives both the finger and sliding-rod motions so as to minimise the gripper complexity. The opening and closing sequence of the gripper is achieved by using a planetary gear train as transmission between the actuator, the fingers and the suction cup sliding mechanism. Special attention is paid to minimise the overall gripper size; its diameter being kept to 75 mm, which is that of the end link of the common UR5 robot. A prototype of the gripper is built and its versatility is demonstrated in a short accompanying video.

Robots are not ready for the real world. It’s still an achievement for autonomous robots to merely survive in the real world, which is a long way from any kind of useful generalized autonomy. Under some fairly specific constraints, autonomous robots are starting to find a few valuable niches in semistructured environments, like offices and hospitals and warehouses. But when it comes to the unstructured nature of disaster areas or human interaction, or really any situation that requires innovation and creativity, autonomous robots are often at a loss.

For the foreseeable future, this means that humans are still necessary. It doesn’t mean that humans must be physically present, however—just that a human is in the loop somewhere. And this creates an opportunity.

In 2018, the XPrize Foundation announced a competition (sponsored by the Japanese airline ANA) to create “an avatar system that can transport human presence to a remote location in real time,” with the goal of developing robotic systems that could be used by humans to interact with the world anywhere with a decent Internet connection. The final event took place last November in Long Beach, Calif., where 17 teams from around the world competed for US $8 million in prize money.

While avatar systems are all able to move and interact with their environment, the Avatar XPrize competition showcased a variety of different hardware and software approaches to creating the most effective system.XPrize Foundation

The competition showcased the power of humans paired with robotic systems, transporting our experience and adaptability to a remote location. While the robots and interfaces were very much research projects rather than systems ready for real-world use, the Avatar XPrize provided the inspiration (as well as the structure and funding) to help some of the world’s best roboticists push the limits of what’s possible through telepresence.

A robotic avatar system is similar to virtual reality, in that both allow a person located in one place to experience and interact with a different place using technology as an interface. Like VR, an effective robotic avatar enables the user to see, hear, touch, move, and communicate in such a way that they feel like they’re actually somewhere else. But where VR puts a human into a virtual environment, a robotic avatar brings a human into a physical environment, which could be in the next room or thousands of kilometers away.

ANA Avatar XPRIZE Finals: Winning team NimbRo Day 2 Test Run youtu.be

The XPrize Foundation hopes that avatar robots could one day be used for more practical purposes: providing care to anyone instantly, regardless of distance; disaster relief in areas where it is too dangerous for human rescuers to go; and performing critical repairs, as well as maintenance and other hard-to-come-by services.

“The available methods by which we can physically transport ourselves from one place to another are not scaling rapidly enough,” says David Locke, the executive director of Avatar XPrize. “A disruption in this space is long overdue. Our aim is to bypass the barriers of distance and time by introducing a new means of physical connection, allowing anyone in the world to physically experience another location and provide on-the-ground assistance where and when it is needed.”

In the Long Beach convention center, XPrize did its best to create an atmosphere that was part rock concert, part sporting event, and part robotics research conference and expo. The course was set up in an arena with stadium seating (open to the public) and extensively decorated and dramatically lit. Live commentary accompanied each competitor’s run. Between runs, teams worked on their avatar systems in a convention hall, where they could interact with each other as well as with curious onlookers. The 17 teams hailed from France, Germany, Italy, Japan, Mexico, Singapore, South Korea, the Netherlands, the United Kingdom, and the United States. With each team preparing for several runs over three days, the atmosphere was by turns frantic and focused as team members moved around the venue and worked to repair or improve their robots. Major academic research labs set up next to small robotics startups, with each team hoping their unique approach would triumph.

The Avatar XPrize course was designed to look like a science station on an alien planet, and the avatar systems had to complete tasks that included using tools and identifying rock samples.XPrize Foundation

The competition course included a series of tasks that each robot had to perform, based around a science mission on the surface of an alien planet. Completing the course involved communicating with a human mission commander, flipping an electrical switch, moving through an obstacle course, identifying a container by weight and manipulating it, using a power drill, and finally, using touch to categorize a rock sample. Teams were ranked by the amount of time their avatar system took to successfully finish all tasks.

There are two fundamental aspects to an avatar system. The first is the robotic mobile manipulator that the human operator controls. The second is the interface that allows the operator to provide that control, and this is arguably the more difficult part of the system. In previous robotics competitions, like the DARPA Robotics Challenge and the DARPA Subterranean Challenge, the interface was generally based around a traditional computer (or multiple computers) with a keyboard and mouse, and the highly specialized job of operator required an immense amount of training and experience. This approach is not accessible or scalable, however. The competition in Long Beach thus featured avatar systems that were essentially operator-agnostic, so that anyone could effectively use them.

XPrize judge Justin Manley celebrates with NimbRo’s avatar robot after completing the course.Evan Ackerman

“Ultimately, the general public will be the end user,” explains Locke. “This competition forced teams to invest time into researching and improving the operator-experience component of the technology. They had to open their technology and labs to general users who could operate and provide feedback on the experience, and the teams who scored highest also had the most intuitive and user-friendly operating interfaces.”

During the competition, team members weren’t allowed to operate their own robots. Instead, a judge was assigned to each team, and the team had 45 minutes to train the judge on the robot and interface. The judges included experts in robotics, virtual reality, human-computer interaction, and neuroscience, but none of them had previous experience as an avatar operator.

Northeastern team member David Nguyen watches XPrize judge Peggy Wu operate the avatar system during a competition run. XPrize Foundation

Once the training was complete, the judge used the team’s interface to operate the robot through the course, while the team could do nothing but sit and watch. Two team members were allowed to remain with the judge in case of technical problems, and a live stream of the operator room captured the stress and helplessness that teams were under: After years of work and with millions of dollars at stake, it was up to a stranger they’d met an hour before to pilot their system to victory. It didn’t always go well, and occasionally it went very badly, as when a bipedal robot collided with the edge of a doorway on the course during a competition run and crashed to the ground, suffering damage that was ultimately unfixable.

The diversity of the teams was reflected in the diversity of their avatar systems. The competition imposed some basic design requirements for the robot, including mobility, manipulation, and a communication interface, but otherwise it was up to each team to design and implement their own hardware and software. Most teams favored a wheeled base with two robotic arms and a head consisting of a screen for displaying the operator’s face. A few daring teams brought bipedal humanoid robots. Stereo cameras were commonly used to provide visual and depth information to the operator, and some teams included additional sensors to convey other types of information about the remote environment.

For example, in the final competition task, the operator needed the equivalent of a sense of touch in order to differentiate a rough rock from a smooth one. While touch sensors for robots are common, translating the data that they collect into something readable by humans is not straightforward. Some teams opted for highly complex (and expensive) microfluidic gloves that transmit touch sensations from the fingertips of the robot to the fingertips of the operator. Other teams used small, finger-mounted vibrating motors to translate roughness into haptic feedback that the operator could feel. Another approach was to mount microphones on the robot’s fingers. As its fingers moved over different surfaces, rough surfaces sounded louder to the operator while smooth surfaces sounded softer.

Many teams, including i-Botics [left], relied on commercial virtual-reality headsets as part of their interfaces. Avatar interfaces were made as immersive as possible to help operators control their robots effectively.Left: Evan Ackerman; Right: XPrize Foundation

In addition to perceiving the remote environment, the operator had to efficiently and effectively control the robot. A basic control interface might be a mouse and keyboard, or a game controller. But with many degrees of freedom to control, limited operator training time, and a competition judged on speed, teams had to get creative. Some teams used motion-detecting virtual-reality systems to transfer the motion of the operator to the avatar robot. Other teams favored a physical interface, strapping the operator into hardware (almost like a robotic exoskeleton) that could read their motions and then actuate the limbs of the avatar robot to match, while simultaneously providing force feedback. With the operator’s arms and hands busy with manipulation, the robot’s movement across the floor was typically controlled with foot pedals.

Northeastern’s robot moves through the course.Evan Ackerman

Another challenge of the XPrize competition was how to use the avatar robot to communicate with a remote human. Teams were judged on how natural such communication was, which precluded using text-only or voice-only interfaces; instead, teams had to give their robot some kind of expressive face. This was easy enough for operator interfaces that used screens; a webcam that was pointed at the operator and streamed to display on the robot worked well.

But for interfaces that used VR headsets, where the operator’s face was partially obscured, teams had to find other solutions. Some teams used in-headset eye tracking and speech recognition to map the operator’s voice and facial movements onto an animated face. Other teams dynamically warped a real image of the user’s face to reflect their eye and mouth movements. The interaction wasn’t seamless, but it was surprisingly effective.

Team iCub, from the Istituto Italiano di Tecnologia, believed its bipedal avatar was the most intuitive way to transfer natural human motion to a robot.Evan Ackerman

With robotics competitions like the Avatar XPrize, there is an inherent conflict between the broader goal of generalized solutions for real-world problems and the focused objective of the competing teams, which is simply to win. Winning doesn’t necessarily lead to a solution to the problem that the competition is trying to solve. XPrize may have wanted to foster the creation of “avatar system[s] that can transport human presence to a remote location in real time,” but the winning team was the one that most efficiently completed the very specific set of competition tasks.

For example, Team iCub, from the Istituto Italiano di Tecnologia (IIT) in Genoa, Italy, believed that the best way to transport human presence to a remote location was to embody that human as closely as possible. To that end, IIT’s avatar system consisted of a small bipedal humanoid robot—the 100-centimeter-tall iCub. Getting a bipedal robot to walk reliably is a challenge, especially when that robot is under the direct control of an inexperienced human. But even under ideal conditions, there was simply no way that iCub could move as quickly as its wheeled competitors.

XPrize decided against a course that would have rewarded humanlike robots—there were no stairs on the course, for example—which prompts the question of what “human presence” actually means. If it means being able to go wherever able-bodied humans can go, then legs might be necessary. If it means accepting that robots (and some humans) have mobility limitations and consequently focusing on other aspects of the avatar experience, then perhaps legs are optional. Whatever the intent of XPrize may have been, the course itself ultimately dictated what made for a successful avatar for the purposes of the competition.

Unsurprisingly, the teams that focused on the competition and optimized their avatar systems accordingly tended to perform well. Team Northeastern won third place and $1 million using a hydrostatic force-feedback interface for the operator. The interface was based on a system of fluidic actuators first conceptualized a decade ago at Disney Research.

Second place went to Team Pollen Robotics, a French startup. Their robot, Reachy, is based on Pollen Robotics’ commercially available mobile manipulator, and it was likely one of the most affordable systems in the competition, costing a mere €20,000 (US $22,000). It used primarily 3D-printed components and an open-source design. Reachy was an exception to the strategy of optimization, because it’s intended to be a generalizable platform for real-world manipulation. But the team’s relatively simple approach helped them win the $2 million second-place prize.

In first place, completing the entire course in under 6 minutes with a perfect score, was Team NimbRo, from the University of Bonn, in Germany. NimbRo has a long history of robotics competitions; they participated in the DARPA Robotics Challenge in 2015 and have been involved in the international RoboCup competition since 2005. But the Avatar XPrize allowed them to focus on new ways of combining human intelligence with robot-control systems. “When I watch human intelligence operating a machine, I find that fascinating,” team lead Sven Behnke told IEEE Spectrum. “A human can see deviations from how they are expecting the machine to behave, and then can resolve those deviations with creativity.”

Team NimbRo’s system relied heavily on the human operator’s own senses and knowledge. “We try to take advantage of human cognitive capabilities as much as possible,” explains Behnke. “For example, our system doesn’t use sensors to estimate depth. It simply relies on the visual cortex of the operator, since humans have evolved to do this in tremendously efficient ways.” To that end, NimbRo’s robot had an unusually long and flexible neck that followed the motions of the operator’s head. During the competition, the robot’s head could be seen shifting from side to side as the operator used parallax to understand how far away objects were. It worked quite well, although NimbRo had to implement a special rendering technique to minimize latency between the operator’s head motions and the video feed from the robot, so that the operator didn’t get motion sickness.

XPrize judge Jerry Pratt [left] operates NimbRo’s robot on the course [right]. The drill task was particularly difficult, involving lifting a heavy object and manipulating it with high precision. Left: Team NimbRo; Right: Evan Ackerman

The team also put a lot of effort into making sure that using the robot to manipulate objects was as intuitive as possible. The operator’s arms were directly attached to robotic arms, which were duplicates of the arms on the avatar robot. This meant that any arm motions made by the operator would be mirrored by the robot, yielding a very consistent experience for the operator.

The operator judge for Team NimbRo’s winning run was Jerry Pratt, who spent decades as a robotics professor at the Florida Institute for Human and Machine Cognition before joining humanoid robotics startup Figure last year. Pratt had led Team IHMC (and a Boston Dynamics Atlas robot) to a second-place finish at the DARPA Robotics Challenge Finals in 2015. “I found it incredible that you can learn how to use these systems in 60 minutes,” Pratt said of his XPrize run. “And operating them is super fun!” Pratt’s winning time of 5:50 to complete the Avatar XPrize course was not much slower than human speed.

At the 2015 DARPA Robotics Challenge finals, by contrast, the Atlas robot had to be painstakingly piloted through the course by a team of experts. It took that robot 50 minutes to complete the course, which a human could have finished in about 5 minutes. “Trying to pick up things with a joystick and mouse [during the DARPA competition] is just really slow,” Pratt says. “Nothing beats being able to just go, ‘Oh, that’s an object, let me grab it’ with full telepresence. You just do it.”

Team Pollen’s robot [left] had a relatively simple operator interface [middle], but that may have been an asset during the competition [right].Pollen Robotics

Both Pratt and NimbRo’s Behnke see humans as a critical component of robots operating in the unstructured environments of the real world, at least in the short term. “You need humans for high-level decision making,” says Pratt. “As soon as there’s something novel, or something goes wrong, you need human cognition in the world. And that’s why you need telepresence.”

Behnke agrees. He hopes that what his group has learned from the Avatar XPrize competition will lead to hybrid autonomy through telepresence, in which robots are autonomous most of the time but humans can use telepresence to help the robots when they get stuck. This approach is already being implemented in simpler contexts, like sidewalk delivery robots, but not yet in the kind of complex human-in-the-loop manipulation that Behnke’s system is capable of.

“Step by step, my objective is to take the human out of that loop so that one operator can control maybe 10 robots, which would be autonomous most of the time,” Behnke says. “And as these 10 systems operate, we get more data from which we can learn, and then maybe one operator will be responsible for 100 robots, and then 1,000 robots. We’re using telepresence to learn how to do autonomy better.”

The entire Avatar XPrize event is available to watch through this live stream recording on YouTube. www.youtube.com

While the Avatar XPrize final competition was based around a space-exploration scenario, Behnke is more interested in applications in which a telepresence-mediated human touch might be even more valuable, such as personal assistance. Behnke’s group has already demonstrated how their avatar system could be used to help someone with an injured arm measure their blood pressure and put on a coat. These sound like simple tasks, but they involve exactly the kind of human interaction and creative manipulation that is exceptionally difficult for a robot on its own. Immersive telepresence makes these tasks almost trivial, and accessible to just about any human with a little training—which is what the Avatar XPrize was trying to achieve.

Still, it’s hard to know how scalable these technologies are. At the moment, avatar systems are fragile and expensive. Historically, there has been a gap of about five to 10 years between high-profile robotics competitions and the arrival of the resulting technology—such as autonomous cars and humanoid robots—at a useful place outside the lab. It’s possible that autonomy will advance quickly enough that the impact of avatar robots will be somewhat reduced for common tasks in structured environments. But it’s hard to imagine that autonomous systems will ever achieve human levels of intuition or creativity. That is, there will continue to be a need for avatars for the foreseeable future. And if these teams can leverage the lessons they’ve learned over the four years of the Avatar XPrize competition to pull this technology out of the research phase, their systems could bypass the constraints of autonomy through human cleverness, bringing us useful robots that are helpful in our daily lives.

Robots are not ready for the real world. It’s still an achievement for autonomous robots to merely survive in the real world, which is a long way from any kind of useful generalized autonomy. Under some fairly specific constraints, autonomous robots are starting to find a few valuable niches in semistructured environments, like offices and hospitals and warehouses. But when it comes to the unstructured nature of disaster areas or human interaction, or really any situation that requires innovation and creativity, autonomous robots are often at a loss.

For the foreseeable future, this means that humans are still necessary. It doesn’t mean that humans must be physically present, however—just that a human is in the loop somewhere. And this creates an opportunity.

In 2018, the XPrize Foundation announced a competition (sponsored by the Japanese airline ANA) to create “an avatar system that can transport human presence to a remote location in real time,” with the goal of developing robotic systems that could be used by humans to interact with the world anywhere with a decent Internet connection. The final event took place last November in Long Beach, Calif., where 17 teams from around the world competed for US $8 million in prize money.

While avatar systems are all able to move and interact with their environment, the Avatar XPrize competition showcased a variety of different hardware and software approaches to creating the most effective system.XPrize Foundation

The competition showcased the power of humans paired with robotic systems, transporting our experience and adaptability to a remote location. While the robots and interfaces were very much research projects rather than systems ready for real-world use, the Avatar XPrize provided the inspiration (as well as the structure and funding) to help some of the world’s best roboticists push the limits of what’s possible through telepresence.

A robotic avatar system is similar to virtual reality, in that both allow a person located in one place to experience and interact with a different place using technology as an interface. Like VR, an effective robotic avatar enables the user to see, hear, touch, move, and communicate in such a way that they feel like they’re actually somewhere else. But where VR puts a human into a virtual environment, a robotic avatar brings a human into a physical environment, which could be in the next room or thousands of kilometers away.

ANA Avatar XPRIZE Finals: Winning team NimbRo Day 2 Test Run youtu.be

The XPrize Foundation hopes that avatar robots could one day be used for more practical purposes: providing care to anyone instantly, regardless of distance; disaster relief in areas where it is too dangerous for human rescuers to go; and performing critical repairs, as well as maintenance and other hard-to-come-by services.

“The available methods by which we can physically transport ourselves from one place to another are not scaling rapidly enough,” says David Locke, the executive director of Avatar XPrize. “A disruption in this space is long overdue. Our aim is to bypass the barriers of distance and time by introducing a new means of physical connection, allowing anyone in the world to physically experience another location and provide on-the-ground assistance where and when it is needed.”

In the Long Beach convention center, XPrize did its best to create an atmosphere that was part rock concert, part sporting event, and part robotics research conference and expo. The course was set up in an arena with stadium seating (open to the public) and extensively decorated and dramatically lit. Live commentary accompanied each competitor’s run. Between runs, teams worked on their avatar systems in a convention hall, where they could interact with each other as well as with curious onlookers. The 17 teams hailed from France, Germany, Italy, Japan, Mexico, Singapore, South Korea, the Netherlands, the United Kingdom, and the United States. With each team preparing for several runs over three days, the atmosphere was by turns frantic and focused as team members moved around the venue and worked to repair or improve their robots. Major academic research labs set up next to small robotics startups, with each team hoping their unique approach would triumph.

The Avatar XPrize course was designed to look like a science station on an alien planet, and the avatar systems had to complete tasks that included using tools and identifying rock samples.XPrize Foundation

The competition course included a series of tasks that each robot had to perform, based around a science mission on the surface of an alien planet. Completing the course involved communicating with a human mission commander, flipping an electrical switch, moving through an obstacle course, identifying a container by weight and manipulating it, using a power drill, and finally, using touch to categorize a rock sample. Teams were ranked by the amount of time their avatar system took to successfully finish all tasks.

There are two fundamental aspects to an avatar system. The first is the robotic mobile manipulator that the human operator controls. The second is the interface that allows the operator to provide that control, and this is arguably the more difficult part of the system. In previous robotics competitions, like the DARPA Robotics Challenge and the DARPA Subterranean Challenge, the interface was generally based around a traditional computer (or multiple computers) with a keyboard and mouse, and the highly specialized job of operator required an immense amount of training and experience. This approach is not accessible or scalable, however. The competition in Long Beach thus featured avatar systems that were essentially operator-agnostic, so that anyone could effectively use them.

XPrize judge Justin Manley celebrates with NimbRo’s avatar robot after completing the course.Evan Ackerman

“Ultimately, the general public will be the end user,” explains Locke. “This competition forced teams to invest time into researching and improving the operator-experience component of the technology. They had to open their technology and labs to general users who could operate and provide feedback on the experience, and the teams who scored highest also had the most intuitive and user-friendly operating interfaces.”

During the competition, team members weren’t allowed to operate their own robots. Instead, a judge was assigned to each team, and the team had 45 minutes to train the judge on the robot and interface. The judges included experts in robotics, virtual reality, human-computer interaction, and neuroscience, but none of them had previous experience as an avatar operator.

Northeastern team member David Nguyen watches XPrize judge Peggy Wu operate the avatar system during a competition run. XPrize Foundation

Once the training was complete, the judge used the team’s interface to operate the robot through the course, while the team could do nothing but sit and watch. Two team members were allowed to remain with the judge in case of technical problems, and a live stream of the operator room captured the stress and helplessness that teams were under: After years of work and with millions of dollars at stake, it was up to a stranger they’d met an hour before to pilot their system to victory. It didn’t always go well, and occasionally it went very badly, as when a bipedal robot collided with the edge of a doorway on the course during a competition run and crashed to the ground, suffering damage that was ultimately unfixable.

The diversity of the teams was reflected in the diversity of their avatar systems. The competition imposed some basic design requirements for the robot, including mobility, manipulation, and a communication interface, but otherwise it was up to each team to design and implement their own hardware and software. Most teams favored a wheeled base with two robotic arms and a head consisting of a screen for displaying the operator’s face. A few daring teams brought bipedal humanoid robots. Stereo cameras were commonly used to provide visual and depth information to the operator, and some teams included additional sensors to convey other types of information about the remote environment.

For example, in the final competition task, the operator needed the equivalent of a sense of touch in order to differentiate a rough rock from a smooth one. While touch sensors for robots are common, translating the data that they collect into something readable by humans is not straightforward. Some teams opted for highly complex (and expensive) microfluidic gloves that transmit touch sensations from the fingertips of the robot to the fingertips of the operator. Other teams used small, finger-mounted vibrating motors to translate roughness into haptic feedback that the operator could feel. Another approach was to mount microphones on the robot’s fingers. As its fingers moved over different surfaces, rough surfaces sounded louder to the operator while smooth surfaces sounded softer.

Many teams, including i-Botics [left], relied on commercial virtual-reality headsets as part of their interfaces. Avatar interfaces were made as immersive as possible to help operators control their robots effectively.Left: Evan Ackerman; Right: XPrize Foundation

In addition to perceiving the remote environment, the operator had to efficiently and effectively control the robot. A basic control interface might be a mouse and keyboard, or a game controller. But with many degrees of freedom to control, limited operator training time, and a competition judged on speed, teams had to get creative. Some teams used motion-detecting virtual-reality systems to transfer the motion of the operator to the avatar robot. Other teams favored a physical interface, strapping the operator into hardware (almost like a robotic exoskeleton) that could read their motions and then actuate the limbs of the avatar robot to match, while simultaneously providing force feedback. With the operator’s arms and hands busy with manipulation, the robot’s movement across the floor was typically controlled with foot pedals.

Northeastern’s robot moves through the course.Evan Ackerman

Another challenge of the XPrize competition was how to use the avatar robot to communicate with a remote human. Teams were judged on how natural such communication was, which precluded using text-only or voice-only interfaces; instead, teams had to give their robot some kind of expressive face. This was easy enough for operator interfaces that used screens; a webcam that was pointed at the operator and streamed to display on the robot worked well.

But for interfaces that used VR headsets, where the operator’s face was partially obscured, teams had to find other solutions. Some teams used in-headset eye tracking and speech recognition to map the operator’s voice and facial movements onto an animated face. Other teams dynamically warped a real image of the user’s face to reflect their eye and mouth movements. The interaction wasn’t seamless, but it was surprisingly effective.

Team iCub, from the Istituto Italiano di Tecnologia, believed its bipedal avatar was the most intuitive way to transfer natural human motion to a robot.Evan Ackerman

With robotics competitions like the Avatar XPrize, there is an inherent conflict between the broader goal of generalized solutions for real-world problems and the focused objective of the competing teams, which is simply to win. Winning doesn’t necessarily lead to a solution to the problem that the competition is trying to solve. XPrize may have wanted to foster the creation of “avatar system[s] that can transport human presence to a remote location in real time,” but the winning team was the one that most efficiently completed the very specific set of competition tasks.

For example, Team iCub, from the Istituto Italiano di Tecnologia (IIT) in Genoa, Italy, believed that the best way to transport human presence to a remote location was to embody that human as closely as possible. To that end, IIT’s avatar system consisted of a small bipedal humanoid robot—the 100-centimeter-tall iCub. Getting a bipedal robot to walk reliably is a challenge, especially when that robot is under the direct control of an inexperienced human. But even under ideal conditions, there was simply no way that iCub could move as quickly as its wheeled competitors.

XPrize decided against a course that would have rewarded humanlike robots—there were no stairs on the course, for example—which prompts the question of what “human presence” actually means. If it means being able to go wherever able-bodied humans can go, then legs might be necessary. If it means accepting that robots (and some humans) have mobility limitations and consequently focusing on other aspects of the avatar experience, then perhaps legs are optional. Whatever the intent of XPrize may have been, the course itself ultimately dictated what made for a successful avatar for the purposes of the competition.

Unsurprisingly, the teams that focused on the competition and optimized their avatar systems accordingly tended to perform well. Team Northeastern won third place and $1 million using a hydrostatic force-feedback interface for the operator. The interface was based on a system of fluidic actuators first conceptualized a decade ago at Disney Research.

Second place went to Team Pollen Robotics, a French startup. Their robot, Reachy, is based on Pollen Robotics’ commercially available mobile manipulator, and it was likely one of the most affordable systems in the competition, costing a mere €20,000 (US $22,000). It used primarily 3D-printed components and an open-source design. Reachy was an exception to the strategy of optimization, because it’s intended to be a generalizable platform for real-world manipulation. But the team’s relatively simple approach helped them win the $2 million second-place prize.

In first place, completing the entire course in under 6 minutes with a perfect score, was Team NimbRo, from the University of Bonn, in Germany. NimbRo has a long history of robotics competitions; they participated in the DARPA Robotics Challenge in 2015 and have been involved in the international RoboCup competition since 2005. But the Avatar XPrize allowed them to focus on new ways of combining human intelligence with robot-control systems. “When I watch human intelligence operating a machine, I find that fascinating,” team lead Sven Behnke told IEEE Spectrum. “A human can see deviations from how they are expecting the machine to behave, and then can resolve those deviations with creativity.”

Team NimbRo’s system relied heavily on the human operator’s own senses and knowledge. “We try to take advantage of human cognitive capabilities as much as possible,” explains Behnke. “For example, our system doesn’t use sensors to estimate depth. It simply relies on the visual cortex of the operator, since humans have evolved to do this in tremendously efficient ways.” To that end, NimbRo’s robot had an unusually long and flexible neck that followed the motions of the operator’s head. During the competition, the robot’s head could be seen shifting from side to side as the operator used parallax to understand how far away objects were. It worked quite well, although NimbRo had to implement a special rendering technique to minimize latency between the operator’s head motions and the video feed from the robot, so that the operator didn’t get motion sickness.

XPrize judge Jerry Pratt [left] operates NimbRo’s robot on the course [right]. The drill task was particularly difficult, involving lifting a heavy object and manipulating it with high precision. Left: Team NimbRo; Right: Evan Ackerman

The team also put a lot of effort into making sure that using the robot to manipulate objects was as intuitive as possible. The operator’s arms were directly attached to robotic arms, which were duplicates of the arms on the avatar robot. This meant that any arm motions made by the operator would be mirrored by the robot, yielding a very consistent experience for the operator.

The operator judge for Team NimbRo’s winning run was Jerry Pratt, who spent decades as a robotics professor at the Florida Institute for Human and Machine Cognition before joining humanoid robotics startup Figure last year. Pratt had led Team IHMC (and a Boston Dynamics Atlas robot) to a second-place finish at the DARPA Robotics Challenge Finals in 2015. “I found it incredible that you can learn how to use these systems in 60 minutes,” Pratt said of his XPrize run. “And operating them is super fun!” Pratt’s winning time of 5:50 to complete the Avatar XPrize course was not much slower than human speed.

At the 2015 DARPA Robotics Challenge finals, by contrast, the Atlas robot had to be painstakingly piloted through the course by a team of experts. It took that robot 50 minutes to complete the course, which a human could have finished in about 5 minutes. “Trying to pick up things with a joystick and mouse [during the DARPA competition] is just really slow,” Pratt says. “Nothing beats being able to just go, ‘Oh, that’s an object, let me grab it’ with full telepresence. You just do it.”

Team Pollen’s robot [left] had a relatively simple operator interface [middle], but that may have been an asset during the competition [right].Pollen Robotics

Both Pratt and NimbRo’s Behnke see humans as a critical component of robots operating in the unstructured environments of the real world, at least in the short term. “You need humans for high-level decision making,” says Pratt. “As soon as there’s something novel, or something goes wrong, you need human cognition in the world. And that’s why you need telepresence.”

Behnke agrees. He hopes that what his group has learned from the Avatar XPrize competition will lead to hybrid autonomy through telepresence, in which robots are autonomous most of the time but humans can use telepresence to help the robots when they get stuck. This approach is already being implemented in simpler contexts, like sidewalk delivery robots, but not yet in the kind of complex human-in-the-loop manipulation that Behnke’s system is capable of.

“Step by step, my objective is to take the human out of that loop so that one operator can control maybe 10 robots, which would be autonomous most of the time,” Behnke says. “And as these 10 systems operate, we get more data from which we can learn, and then maybe one operator will be responsible for 100 robots, and then 1,000 robots. We’re using telepresence to learn how to do autonomy better.”

The entire Avatar XPrize event is available to watch through this live stream recording on YouTube. www.youtube.com

While the Avatar XPrize final competition was based around a space-exploration scenario, Behnke is more interested in applications in which a telepresence-mediated human touch might be even more valuable, such as personal assistance. Behnke’s group has already demonstrated how their avatar system could be used to help someone with an injured arm measure their blood pressure and put on a coat. These sound like simple tasks, but they involve exactly the kind of human interaction and creative manipulation that is exceptionally difficult for a robot on its own. Immersive telepresence makes these tasks almost trivial, and accessible to just about any human with a little training—which is what the Avatar XPrize was trying to achieve.

Still, it’s hard to know how scalable these technologies are. At the moment, avatar systems are fragile and expensive. Historically, there has been a gap of about five to 10 years between high-profile robotics competitions and the arrival of the resulting technology—such as autonomous cars and humanoid robots—at a useful place outside the lab. It’s possible that autonomy will advance quickly enough that the impact of avatar robots will be somewhat reduced for common tasks in structured environments. But it’s hard to imagine that autonomous systems will ever achieve human levels of intuition or creativity. That is, there will continue to be a need for avatars for the foreseeable future. And if these teams can leverage the lessons they’ve learned over the four years of the Avatar XPrize competition to pull this technology out of the research phase, their systems could bypass the constraints of autonomy through human cleverness, bringing us useful robots that are helpful in our daily lives.