Feed aggregator



About a year ago, Zipline introduced Platform 2, an approach to precision urban drone delivery that combines a large hovering drone with a smaller package-delivery “Droid.” Lowered on a tether from the belly of its parent Zip drone, the Droid contains thrusters and sensors (plus a 2.5- to 3.5-kilogram payload) to reliably navigate itself to a delivery area of just one meter in diameter. The Zip, meanwhile, safely remains hundreds of meters up. After depositing its payload, the Droid rises back up to the drone on its tether, and off they go.

At first glance, the sensor and thruster-packed Droid seems complicated enough to be bordering on impractical, especially when you consider the relative simplicity of other drone delivery solutions, which commonly just drop the package itself on a tether from a hovering drone. I’ve been writing about robots long enough that I’m suspicious of robotic solutions that appear to be overengineered, since that’s always a huge temptation with robotics. Like, is this really the best way of solving a problem, or is it just the coolest way?

We know the folks at Zipline pretty well, though, and they’ve certainly made creative engineering work for them, as we saw when we visited one of their “nests” in rural Rwanda. So as Zipline nears the official launch of Platform 2, we spoke with Zipline cofounder and CTO Keenan Wyrobek, Platform 2 lead Zoltan Laszlo, and industrial designer Gregoire Vandenbussche to understand exactly why they think this is the best way of solving precision urban drone delivery.

First, a quick refresher. Here’s what the delivery sequence with the vertical takeoff and landing (VTOL) Zip and the Droid looks like:

The system has a service radius of about 16 kilometers (10 miles), and it can make deliveries to outdoor spaces of “any meaningful size.” Visual sensors on the Droid find the delivery site and check for obstacles on the way down, while the thrusters compensate for wind and movement of the parent drone. Since the big VTOL Zip remains well out of the way, deliveries are fast, safe, and quiet. But it takes two robots to pull off the delivery rather than just one.

On the other end is the infrastructure required to load and charge these drones. Zipline’s Platform 1 drones require a dedicated base with relatively large launch and recovery systems. With Platform 2, the drone drops the Droid into a large chute attached to the side of a building so that the Droid can be reloaded, after which it pulls the Droid out again and flies off to make the delivery:

“We think it’s the best delivery experience. Not the best drone delivery experience, the best delivery experience,” Zipline’s Wyrobek tells us. That may be true, but the experience also has to be practical and sustainable for Zipline to be successful, so we asked the Zipline team to explain the company’s approach to precision urban delivery.

Zipline on:

IEEE Spectrum: What problems is Platform 2 solving, and why is it necessary to solve those problems in this specific way?

Keenan Wyrobek: There are literally billions of last-mile deliveries happening every year in [the United States] alone, and our customers have been asking for years for something that can deliver to their homes. With our long-range platform, Platform 1, we can float a package down into your yard on a parachute, but that takes some space. And so one half of the big design challenge was how to get our deliveries precise enough, while the other half was to develop a system that will bolt on to existing facilities, which Platform 1 doesn’t do.

Zoltan Laszlo: Platform 1 can deliver within an area of about two parking spaces. As we started to actually look at the data in urban areas using publicly available lidar surveys, we found that two parking spaces serves a bit more than half the market. We want to be a universal delivery service.

But with a delivery area of 1 meter in diameter, which is what we’re actually hitting in our delivery demonstrations for Platform 2, that gets us into the high 90s for the percentage of people that we can deliver to.

Wyrobek: When we say “urban,” what we’re talking about is three-story sprawl, which is common in many large cities around the world. And we wanted to make sure that our deliveries could be precise enough for places like that.

There are some existing solutions for precision aerial delivery that have been operating at scale with some success, typically by winching packages to the ground from a VTOL drone. Why develop your own technique rather than just going with something that has already been shown to work?

Laszlo: Winching down is the natural extension of being able to hover in place, and when we first started, we were like, “Okay, we’re just going to winch down. This will be great, super easy.”

So we went to our test site in Half Moon Bay [on the Northern California coast] and built a quick prototype of a winch system. But as soon as we lowered a box down on the winch, the wind started blowing it all over the place. And this was from the height of our lift, which is less than 10 meters up. We weren’t even able to stay inside two parking spaces, which told us that something was broken with our approach.

The aircraft can sense the wind, so we thought we’d be able to find the right angle for the delivery and things like that. But the wind where the aircraft is may be different from the wind nearer the ground. We realized that unless we’re delivering to an open field, a package that does not have active wind compensation is going to be very hard to control. We’re targeting high-90th percentile in terms of availability due to weather—even if it’s a pretty blustery day, we still want to be able to deliver.

Wyrobek: This was a wild insight when we really understood that unless it’s a perfect day, using a winch actually takes almost as much space as we use for Platform 1 floating a package down on a parachute.

Engineering test footage of Zipline’s Platform 2 docking system at their test site in Half Moon Bay in California.

How did you arrive at this particular delivery solution for Platform 2?

Laszlo: I don’t remember whose idea it was, but we were playing with a bunch of different options. Putting thrusters on the tether wasn’t even the craziest idea. We had our Platform 1 aircraft, which was reliable, so we started with looking at ways to just make that aircraft deliver more precisely. There was only so much more we could do with passive parachutes, but what does an active, steerable parachute look like? There are remote-controlled paragliding toys out there that we tested, with mixed results—the challenge is to minimize the smarts in your parachute, because there’s a chance you won’t get it back. So then we started some crazy brainstorming about how to reliably retrieve the parachute.

Wyrobek: One idea was that the parachute would come with a self-return envelope that you could stick in the mail. Another idea was that the parachute would be steered by a little drone, and when the package got dropped off, the drone would reel the parachute in and then fly back up into the Zip.

Laszlo: But when we realized that the package has to be able to steer itself, that meant the Zip doesn’t need to be active. The Zip doesn’t need to drive the package, it doesn’t even need to see the package, it just needs to be a point up in the sky that’s holding the package. That let us move from having the Zip 50 feet up, to having it 300 feet up, which is important because it’s a big, heavy drone that we don’t want in our customer’s space. And the final step was adding enough smarts to the thing coming down into your space to figure out where exactly to deliver to, and of course to handle the wind.

Once you knew what you needed to do, how did you get to the actual design of the droid?

Gregoire Vandenbussche: Zipline showed me pretty early on that they were ready to try crazy ideas, and from my experience, that’s extremely rare. When the idea of having this controllable tether with a package attached to it came up, one of my first thoughts was that from a user standpoint, nothing like this exists. And the difficulty of designing something that doesn’t exist is that people will try to identify it according to what they know. So we had to find a way to drive that thinking towards something positive.

Early Droid concept sketches by designer Gregoire Vandenbussche featured legs that would fold up after delivery.Zipline

First we thought about putting words onto it, like “hello” or something, but the reality is that we’re an international company and we need to be able to work everywhere. But there’s one thing that’s common to everyone, and that’s emotions—people are able to recognize certain things as being approachable and adorable, so going in that direction felt like the right thing to do. However, being able to design a robot that gives you that kind of emotion but also flies was quite a challenge. We took inspiration from other things that move in 3D, like sea mammals—things that people will recognize even without thinking about it.

Vandenbussche’s sketches show how the design of the Droid was partially inspired by dolphins.Zipline

Now that you say it, I can definitely see the sea mammal inspiration in the drone.

Vandenbussche: There are two aspects of sea mammals that work really well for our purpose. One of them is simplicity of shape; sea mammals don’t have all that many details. Also, they tend to be optimized for performance. Ultimately, we need that, because we need to be able to fly. And we need to be able to convey to people that the drone is under control. So having something you can tell is moving forward or turning or moving away was very helpful.

Wyrobek: One other insight that we had is that Platform 2 needs to be small to fit into tight delivery spaces, and it needs to feel small when it comes into your personal space, but it also has to be big enough inside to be a useful delivery platform. We tried to leverage the chubby but cute look that baby seals have going on.

The design journey was pretty fun. Gregoire would spend two or three days coming up with a hundred different concept sketches. We’d do a bunch of brainstorming, and then Gregoire would come up with a whole bunch of new directions, and we’d keep exploring. To be clear, no one would describe our functional prototypes from back then as “cute.” But through all this iteration eventually we ended up in an awesome place.

And how do you find that place? When do you know that your robot is just cute enough?

One iteration of the Droid, Vandenbussche determined, looked too technical and intimidating.Zipline

Vandenbussche: It’s finding the balance around what’s realistic and functional. I like to think of industrial design as taking all of the constraints and kind of playing Tetris with them until you get a result that ideally satisfies everybody. I remember at one point looking at where we were, and feeling like we were focusing too much on performance and missing that emotional level. So, we went back a little bit to say, where can we bring this back from looking like a highly technical machine to something that can give you a feeling of approachability?

Laszlo: We spent a fair bit of time on the controls and behaviors of the droid to make sure that it moves in a very approachable and predictable way, so that you know where it’s going ahead of time and it doesn’t behave in unexpected ways. That’s pretty important for how people perceive it.

We did a lot of work on how the droid would descend and approach the delivery site. One concept had the droid start to lower down well before the Zip was hovering directly overhead. We had simulations and renderings, and it looked great. We could do the whole delivery in barely over 20 seconds. But even if the package is far away from you, it still looks scary because [the Zip is] moving faster than you would expect, and you can’t tell exactly where it’s going to deliver. So we deleted all that code, and now it just comes straight down, and people don’t back away from the Droid anymore. They’re just like, “Oh, okay, cool.”

How did you design the thrusters to enable these pinpoint deliveries?

Early tests of the Droid centered around a two-fan version.Zipline

Laszlo: With the thrusters, we knew we wanted to maximize the size of at least one of the fans, because we were almost always going to have to deal with wind. We’re trying to be as quiet as we can, so the key there is to maximize the area of the propeller. Our leading early design was just a box with two fans on it:

Two fans with unobstructed flow meant that it moved great, but the challenge of fitting it inside another aircraft was going to be painful. And it looked big, even though it wasn’t actually that big.

Vandenbussche: It was also pretty intimidating when you had those two fans facing you and the Droid coming toward you.

A single steerable fan [left] that acted like a rudder was simpler in some ways, but as the fan got larger, the gyroscopic effects became hard to manage. Instead of one steerable fan, how about two steerable fans? [right] Omnidirectional motion was possible with this setup, but packaging it inside of a Zip didn’t work.Zipline

Laszlo: We then started looking at configurations with a main fan and a second smaller fan, with the bigger fan at the back pushing forward and the smaller fan at the front providing thrust for turning. The third fan we added relatively late because we didn’t want to add it at all. But we found that [with two fans] the droid would have to spin relatively quickly to align to shifting winds, whereas with a third fan we can just push sideways in the direction that we need.

What kind of intelligence does the Droid have?

The current design of Zipline’s Platform 2 Droid is built around a large thruster in the rear and two smaller thrusters at the front and back.Zipline

Wyrobek: The Droid has its own little autopilot, and there’s a very simple communications system between the two vehicles. You may think that it’s a really complex coordinated control problem, but it’s not: The Zip just kind of hangs out, and the Droid takes care of the delivery. The sensing challenge is for the Droid to find trees and powerlines and things like that, and then find a good delivery site.

Was there ever a point at which you were concerned that the size and weight and complexity would not be worth it?

Wyrobek: Our mindset was to fail fast, to try things and do what we needed to do to convince ourselves that it wasn’t a good path. What’s fun about this kind of iterative process is oftentimes, you try things and you realize that actually, this is better than we thought.

Laszlo: We first thought about the Droid as a little bit of a tax, in that it’s costing us extra weight. But if your main drone can stay high enough up that it avoids trees and buildings, then it can just float around up there. If it gets pushed around by the wind, it doesn’t matter because the Droid can compensate.

Wyrobek: Keeping the Zip at altitude is a big win in many ways. It doesn’t have to spend energy station-keeping, descending, and then ascending again. We just do that with the much smaller Droid, which also makes the hovering phase much shorter. It’s also much more efficient to control the small droid than the large Zip. And having all of the sensors on the Droid very close to the area that you’re delivering to makes that problem easier as well. It may look like a more complex system from the outside, but from the inside, it’s basically making all the hardest problems much easier.

Over the past year, Zipline has set up a bunch of partnerships to make residential deliveries to consumers using Droid starting in 2024, including prescriptions from Cleveland Clinic in Ohio, medical products from WellSpan Health in Pennsylvania, tasty food from Mendocino Farms in California, and a little bit of everything from Walmart starting in Dallas. Zipline’s plan is to kick things off with Platform 2 later this year.



Legendary MIT roboticist Daniela Rus has published a new book called The Heart and the Chip: Our Bright Future with Robots. “There is a robotics revolution underway,” Rus says in the book’s introduction, “one that is already causing massive changes in our society and in our lives.” She’s quite right, of course, and although some of us have been feeling that this is true for decades, it’s arguably more true right now than it ever has been. But robots are difficult and complicated, and the way that their progress is intertwined with the humans that make them and work with them means that these changes won’t come quickly or easily. Rus’ experience gives her a deep and nuanced perspective on robotics’ past and future, and we’re able to share a little bit of that with you here.

Daniela Rus: Should roboticists consider subscribing to their own Hippocratic oath?

The following excerpt is from Chapter 14, entitled “What Could Go Wrong?” Which, let’s be honest, is the right question to ask (and then attempt to conclusively answer) whenever you’re thinking about sending a robot out into the real world.

At several points in this book I’ve mentioned the fictional character Tony Stark, who uses technology to transform himself into the superhero Iron Man. To me this character is a tremendous inspiration, yet I often remind myself that in the story, he begins his career as an MIT-­trained weapons manufacturer and munitions developer. In the 2008 film Iron Man, he changes his ways because he learns that his company’s specialized weapons are being used by terrorists.

Remember, robots are tools. Inherently, they are neither good nor bad; it’s how we choose to use them that matters. In 2022, aerial drones were used as weapons on both sides of devastating wars. Anyone can purchase a drone, but there are regulations for using drones that vary between and within different countries. In the United States, the Federal Aviation Administration requires that all drones be registered, with a few exceptions, including toy models weighing less than 250 grams. The rules also depend on whether the drone is flown for fun or for business. Regardless of regulations, anyone could use a flying robot to inflict harm, just like anyone can swing a hammer to hurt someone instead of driving a nail into a board. Yet drones are also being used to deliver critical medical supplies in hard-­to-­reach areas, track the health of forests, and help scientists like Roger Payne monitor and advocate for at-­risk species. My group collaborated with the modern dance company Pilobolus to stage the first theatrical performance featuring a mix of humans and drones back in 2012, with a robot called Seraph. So, drones can be dancers, too. In Kim Stanley Robinson’s prescient science fiction novel The Ministry for the Future, a swarm of unmanned aerial vehicles is deployed to crash an airliner. I can imagine a flock of these mechanical birds being used in many good ways, too. At the start of its war against Ukraine, Russia limited its citizens’ access to unbiased news and information in hopes of controlling and shaping the narrative around the conflict. The true story of the invasion was stifled, and I wondered whether we could have dispatched a swarm of flying video screens capable of arranging themselves into one giant aerial monitor in the middle of popular city squares across Russia, showing real footage of the war, not merely clips approved by the government. Or, even simpler: swarms of flying digital projectors could have broadcasted the footage on the sides of buildings and walls for all to see. If we had deployed enough, there would have been too many of them to shut down.

There may be variations of Tony Stark passing through my university or the labs of my colleagues around the world, and we need to do whatever we can to ensure these talented young individuals endeavor to have a positive impact on humanity.

The Tony Stark character is shaped by his experiences and steered toward having a positive impact on the world, but we cannot wait for all of our technologists to endure harrowing, life-­changing experiences. Nor can we expect everyone to use these intelligent machines for good once they are developed and moved out into circulation. Yet that doesn’t mean we should stop working on these technologies—­the potential benefits are too great. What we can do is think harder about the consequences and put in place the guardrails to ensure positive benefits. My contemporaries and I can’t necessarily control how these tools are used in the world, but we can do more to influence the people making them.

There may be variations of Tony Stark passing through my university or the labs of my colleagues around the world, and we need to do whatever we can to ensure these talented young individuals endeavor to have a positive impact on humanity. We absolutely must have diversity in our university labs and research centers, but we may be able to do more to shape the young people who study with us. For example, we could require study of the Manhattan Project and the moral and ethical quandaries associated with the phenomenal effort to build and use the atomic bomb. At this point, ethics courses are not a widespread requirement for an advanced degree in robotics or AI, but perhaps they should be. Or why not require graduates to swear to a robotics-­ and AI-­attuned variation on the Hippocratic oath?

The oath comes from an early Greek medical text, which may or may not have been written by the philosopher Hippocrates, and it has evolved over the centuries. Fundamentally, it represents a standard of medical ethics to which doctors are expected to adhere. The most famous of these is the promise to do no harm, or to avoid intentional wrongdoing. I also applaud the oath’s focus on committing to the community of doctors and the necessity of maintaining the sacred bond between teacher and pupils. The more we remain linked as a robotics community, the more we foster and maintain our relationships as our students move out into the world, the more we can do to steer the technology toward a positive future. Today the Hippocratic oath is not a universal requirement for certification as a doctor, and I do not see it functioning that way for roboticists, either. Nor am I the first roboticist or AI leader to suggest this possibility. But we should seriously consider making it standard practice.

In the aftermath of the development of the atomic bomb, when the potential of scientists to do harm was made suddenly and terribly evident, there was some discussion of a Hippocratic oath for scientific researchers. The idea has resurfaced from time to time and rarely gains traction. But science is fundamentally about the pursuit of knowledge; in that sense it is pure. In robotics and AI, we are building things that will have an impact on the world and its people and other forms of life. In this sense, our field is somewhat closer to medicine, as doctors are using their training to directly impact the lives of individuals. Asking technologists to formally recite a version of the Hippocratic oath could be a way to continue nudging our field in the right direction, and perhaps serve as a check on individuals who are later asked to develop robots or AI expressly for nefarious purposes.

Of course, the very idea of what is good or bad, in terms of how a robot is used, depends on where you sit. I am steadfastly opposed to giving armed or weaponized robots autonomy. We cannot and should not trust machine intelligences to make decisions about whether to inflict harm on a person or group of people on their own. Personally, I would prefer that robots never be used to do harm to anyone, but this is now unrealistic. Robots are being used as tools of war, and it is our responsibility to do whatever we can to shape their ethical use. So, I do not separate or divorce myself from reality and operate solely in some utopian universe of happy, helpful robots. In fact, I teach courses on artificial intelligence to national security officials and advise them on the strengths, weaknesses, and capabilities of the technology. I see this as a patriotic duty, and I’m honored to be helping our leaders understand the limitations, strengths, and possibilities of robots and other AI-­enhanced physical systems—­what they can and cannot do, what they should and should not do, and what I believe they must do.

Ultimately, no matter how much we teach and preach about the limitations of technology, the ethics of AI, or the potential dangers of developing such powerful tools, people will make their own choices, whether they are recently graduated students or senior national security leaders. What I hope and teach is that we should choose to do good. Despite the efforts of life extension companies, we all have a limited time on this planet, what the scientist Carl Sagan called our “pale blue dot,” and we should do whatever we can to make the most of that time and have a positive impact on our beautiful environment, and the many people and other species with which we share it. My decades-­long quest to build more intelligent and capable robots has only strengthened my appreciation for—­no, wonder at—­the marvelous creatures that crawl, walk, swim, run, slither, and soar across and around our planet, and the fantastic plants, too. We should not busy ourselves with the work of developing robots that can eliminate these cosmically rare creations. We should focus instead on building technologies to preserve them, and even help them thrive. That applies to all living entities, including the one species that is especially concerned about the rise of intelligent machines.


Excerpted from “The Heart and the Chip: Our Bright Future with Robots”. Copyright 2024 by Daniela Rus, Gregory Mone. Used with permission of the publisher, W.W. Norton & Company. All rights reserved.



Legendary MIT roboticist Daniela Rus has published a new book called The Heart and the Chip: Our Bright Future with Robots. “There is a robotics revolution underway,” Rus says in the book’s introduction, “one that is already causing massive changes in our society and in our lives.” She’s quite right, of course, and although some of us have been feeling that this is true for decades, it’s arguably more true right now than it ever has been. But robots are difficult and complicated, and the way that their progress is intertwined with the humans that make them and work with them means that these changes won’t come quickly or easily. Rus’ experience gives her a deep and nuanced perspective on robotics’ past and future, and we’re able to share a little bit of that with you here.

Daniela Rus: Should roboticists consider subscribing to their own Hippocratic oath?

The following excerpt is from Chapter 14, entitled “What Could Go Wrong?” Which, let’s be honest, is the right question to ask (and then attempt to conclusively answer) whenever you’re thinking about sending a robot out into the real world.

At several points in this book I’ve mentioned the fictional character Tony Stark, who uses technology to transform himself into the superhero Iron Man. To me this character is a tremendous inspiration, yet I often remind myself that in the story, he begins his career as an MIT-­trained weapons manufacturer and munitions developer. In the 2008 film Iron Man, he changes his ways because he learns that his company’s specialized weapons are being used by terrorists.

Remember, robots are tools. Inherently, they are neither good nor bad; it’s how we choose to use them that matters. In 2022, aerial drones were used as weapons on both sides of devastating wars. Anyone can purchase a drone, but there are regulations for using drones that vary between and within different countries. In the United States, the Federal Aviation Administration requires that all drones be registered, with a few exceptions, including toy models weighing less than 250 grams. The rules also depend on whether the drone is flown for fun or for business. Regardless of regulations, anyone could use a flying robot to inflict harm, just like anyone can swing a hammer to hurt someone instead of driving a nail into a board. Yet drones are also being used to deliver critical medical supplies in hard-­to-­reach areas, track the health of forests, and help scientists like Roger Payne monitor and advocate for at-­risk species. My group collaborated with the modern dance company Pilobolus to stage the first theatrical performance featuring a mix of humans and drones back in 2012, with a robot called Seraph. So, drones can be dancers, too. In Kim Stanley Robinson’s prescient science fiction novel The Ministry for the Future, a swarm of unmanned aerial vehicles is deployed to crash an airliner. I can imagine a flock of these mechanical birds being used in many good ways, too. At the start of its war against Ukraine, Russia limited its citizens’ access to unbiased news and information in hopes of controlling and shaping the narrative around the conflict. The true story of the invasion was stifled, and I wondered whether we could have dispatched a swarm of flying video screens capable of arranging themselves into one giant aerial monitor in the middle of popular city squares across Russia, showing real footage of the war, not merely clips approved by the government. Or, even simpler: swarms of flying digital projectors could have broadcasted the footage on the sides of buildings and walls for all to see. If we had deployed enough, there would have been too many of them to shut down.

There may be variations of Tony Stark passing through my university or the labs of my colleagues around the world, and we need to do whatever we can to ensure these talented young individuals endeavor to have a positive impact on humanity.

The Tony Stark character is shaped by his experiences and steered toward having a positive impact on the world, but we cannot wait for all of our technologists to endure harrowing, life-­changing experiences. Nor can we expect everyone to use these intelligent machines for good once they are developed and moved out into circulation. Yet that doesn’t mean we should stop working on these technologies—­the potential benefits are too great. What we can do is think harder about the consequences and put in place the guardrails to ensure positive benefits. My contemporaries and I can’t necessarily control how these tools are used in the world, but we can do more to influence the people making them.

There may be variations of Tony Stark passing through my university or the labs of my colleagues around the world, and we need to do whatever we can to ensure these talented young individuals endeavor to have a positive impact on humanity. We absolutely must have diversity in our university labs and research centers, but we may be able to do more to shape the young people who study with us. For example, we could require study of the Manhattan Project and the moral and ethical quandaries associated with the phenomenal effort to build and use the atomic bomb. At this point, ethics courses are not a widespread requirement for an advanced degree in robotics or AI, but perhaps they should be. Or why not require graduates to swear to a robotics-­ and AI-­attuned variation on the Hippocratic oath?

The oath comes from an early Greek medical text, which may or may not have been written by the philosopher Hippocrates, and it has evolved over the centuries. Fundamentally, it represents a standard of medical ethics to which doctors are expected to adhere. The most famous of these is the promise to do no harm, or to avoid intentional wrongdoing. I also applaud the oath’s focus on committing to the community of doctors and the necessity of maintaining the sacred bond between teacher and pupils. The more we remain linked as a robotics community, the more we foster and maintain our relationships as our students move out into the world, the more we can do to steer the technology toward a positive future. Today the Hippocratic oath is not a universal requirement for certification as a doctor, and I do not see it functioning that way for roboticists, either. Nor am I the first roboticist or AI leader to suggest this possibility. But we should seriously consider making it standard practice.

In the aftermath of the development of the atomic bomb, when the potential of scientists to do harm was made suddenly and terribly evident, there was some discussion of a Hippocratic oath for scientific researchers. The idea has resurfaced from time to time and rarely gains traction. But science is fundamentally about the pursuit of knowledge; in that sense it is pure. In robotics and AI, we are building things that will have an impact on the world and its people and other forms of life. In this sense, our field is somewhat closer to medicine, as doctors are using their training to directly impact the lives of individuals. Asking technologists to formally recite a version of the Hippocratic oath could be a way to continue nudging our field in the right direction, and perhaps serve as a check on individuals who are later asked to develop robots or AI expressly for nefarious purposes.

Of course, the very idea of what is good or bad, in terms of how a robot is used, depends on where you sit. I am steadfastly opposed to giving armed or weaponized robots autonomy. We cannot and should not trust machine intelligences to make decisions about whether to inflict harm on a person or group of people on their own. Personally, I would prefer that robots never be used to do harm to anyone, but this is now unrealistic. Robots are being used as tools of war, and it is our responsibility to do whatever we can to shape their ethical use. So, I do not separate or divorce myself from reality and operate solely in some utopian universe of happy, helpful robots. In fact, I teach courses on artificial intelligence to national security officials and advise them on the strengths, weaknesses, and capabilities of the technology. I see this as a patriotic duty, and I’m honored to be helping our leaders understand the limitations, strengths, and possibilities of robots and other AI-­enhanced physical systems—­what they can and cannot do, what they should and should not do, and what I believe they must do.

Ultimately, no matter how much we teach and preach about the limitations of technology, the ethics of AI, or the potential dangers of developing such powerful tools, people will make their own choices, whether they are recently graduated students or senior national security leaders. What I hope and teach is that we should choose to do good. Despite the efforts of life extension companies, we all have a limited time on this planet, what the scientist Carl Sagan called our “pale blue dot,” and we should do whatever we can to make the most of that time and have a positive impact on our beautiful environment, and the many people and other species with which we share it. My decades-­long quest to build more intelligent and capable robots has only strengthened my appreciation for—­no, wonder at—­the marvelous creatures that crawl, walk, swim, run, slither, and soar across and around our planet, and the fantastic plants, too. We should not busy ourselves with the work of developing robots that can eliminate these cosmically rare creations. We should focus instead on building technologies to preserve them, and even help them thrive. That applies to all living entities, including the one species that is especially concerned about the rise of intelligent machines.


Excerpted from “The Heart and the Chip: Our Bright Future with Robots”. Copyright 2024 by Daniela Rus, Gregory Mone. Used with permission of the publisher, W.W. Norton & Company. All rights reserved.



Your weekly selection of awesome robot videos

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPAN

Enjoy today’s videos!

How many quadrupeds can you control with one single locomotion policy? Apparently, the answer is “all of the quadrupeds.”

Look for this at ICRA 2024 in a couple of months!

[ EPFL ]

Thanks, Milad!

Very impressive performance from Figure 01, I think, although as is frequently the case, it’s hard to tell exactly how impressive without more information about exactly what’s going on here.

[ Figure ]

That awesome ANYmal Parkour research is now published, which means that there’s a new video, well worth watching all the way to the end.

[ Science ] via [ ETHZ RSL ]

Robotic vision can be pretty tricky when you’re cooking, because things can significantly change how they look over time, like with melting butter or an egg being fried. Some new research is tackling this, using a (now ancient?) PR2.

[ JSK Lab ]

Thanks, Kento!

Filmed in January of 2020, this video shows Atlas clearing debris and going through a doorway. Uses a combination of simple footstep planning, teleoperation, and autonomous behaviors through a single virtual reality operator interface. Robot built by Boston Dynamics for the DARPA Robotics Challenge in 2013.

[ IHMC ]

Sustainable fashion enabled by smart textiles shaped by a robot and a heat gun. Multiple styles, multiple sizes, all in one garment!

[ MIT ]

Video of Boston Dynamics’ Stretch from MODEX, with a little sneak peak at the end of what the robot’s next warehouse task might be.

[ Boston Dynamics ]

Pickle Robots autonomously unload trucks and import containers. The system is in production use at customer warehoues handling floor-loaded freight at human scale or better.

[ Pickle Robot ]

The ROBDEKON robotics competence center is dedicated to the development of robotic systems for hazardous environments that pose a potential risk to humans. As part of the consortium, the FZI Research Center for Information Technology developed robotic systems, technologies, and Artificial Intelligence (AI) methods that can be used to handle hazardous materials–for example, to sort potentially dangerous used batteries for recycling.

[ FZI ]

This research project with Ontario Power Generation involves adapting Boston Dynamics Spot’s localization system to longterm changes in the environment. During this testing, we mounted a GoPro camera on the back of Spot and took a video of each walk for a year from Spot’s point of view. We put the footage together as a moving time-lapse video where the day changes as Spot completes the Autowalk around the campus.

[ MARS Lab ]



Your weekly selection of awesome robot videos

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPAN

Enjoy today’s videos!

How many quadrupeds can you control with one single locomotion policy? Apparently, the answer is “all of the quadrupeds.”

Look for this at ICRA 2024 in a couple of months!

[ EPFL ]

Thanks, Milad!

Very impressive performance from Figure 01, I think, although as is frequently the case, it’s hard to tell exactly how impressive without more information about exactly what’s going on here.

[ Figure ]

That awesome ANYmal Parkour research is now published, which means that there’s a new video, well worth watching all the way to the end.

[ Science ] via [ ETHZ RSL ]

Robotic vision can be pretty tricky when you’re cooking, because things can significantly change how they look over time, like with melting butter or an egg being fried. Some new research is tackling this, using a (now ancient?) PR2.

[ JSK Lab ]

Thanks, Kento!

Filmed in January of 2020, this video shows Atlas clearing debris and going through a doorway. Uses a combination of simple footstep planning, teleoperation, and autonomous behaviors through a single virtual reality operator interface. Robot built by Boston Dynamics for the DARPA Robotics Challenge in 2013.

[ IHMC ]

Sustainable fashion enabled by smart textiles shaped by a robot and a heat gun. Multiple styles, multiple sizes, all in one garment!

[ MIT ]

Video of Boston Dynamics’ Stretch from MODEX, with a little sneak peak at the end of what the robot’s next warehouse task might be.

[ Boston Dynamics ]

Pickle Robots autonomously unload trucks and import containers. The system is in production use at customer warehoues handling floor-loaded freight at human scale or better.

[ Pickle Robot ]

The ROBDEKON robotics competence center is dedicated to the development of robotic systems for hazardous environments that pose a potential risk to humans. As part of the consortium, the FZI Research Center for Information Technology developed robotic systems, technologies, and Artificial Intelligence (AI) methods that can be used to handle hazardous materials–for example, to sort potentially dangerous used batteries for recycling.

[ FZI ]

This research project with Ontario Power Generation involves adapting Boston Dynamics Spot’s localization system to longterm changes in the environment. During this testing, we mounted a GoPro camera on the back of Spot and took a video of each walk for a year from Spot’s point of view. We put the footage together as a moving time-lapse video where the day changes as Spot completes the Autowalk around the campus.

[ MARS Lab ]

Robots have tremendous potential, and have recently been introduced not only for simple operations in factories, but also in workplaces where customer service communication is required. However, communication robots have not always been accepted. This study proposes a three-stage (first contact, interaction, and decision) model for robot acceptance based on the human cognitive process flow to design preferred robots and clarifies the elements of the robot and the processes that affect robot acceptance decision-making. Unlike previous robot acceptance models, the current model focuses on a sequential account of how people decide to accept, considering the interaction (or carry-over) effect between impressions established at each stage. According to the model, this study conducted a scenario-based experiment focusing on the impression of the first contact (a robot’s appearance) and that formed during the interaction with robot (politeness of its conversation and behavior) on robot acceptance in both successful and slightly failed situations. The better the appearance of the robot and the more polite its behavior, the greater the acceptance rate. Importantly, there was no interaction between these two factors. The results indicating that the impressions of the first contact and interaction are additively processed suggest that we should accumulate findings that improving the appearance of the robot and making its communication behavior more human-like in politeness will lead to a more acceptable robot design.

Actuator failure on a remotely deployed robot results in decreased efficiency or even renders it inoperable. Robustness to these failures will become critical as robots are required to be more independent and operate out of the range of repair. To address these challenges, we present two approaches based on modular robotic architecture to improve robustness to actuator failure of both fixed-configuration robots and modular reconfigurable robots. Our work uses modular reconfigurable robots capable of modifying their style of locomotion and changing their designed morphology through ejecting modules. This framework improved the distance travelled and decreased the effort to move through the environment of simulated and physical robots. When the deployed robot was allowed to change its locomotion style, it showed improved robustness to actuator failure when compared to a robot with a fixed controller. Furthermore, a robot capable of changing its locomotion and design morphology statistically outlasted both tests with a fixed morphology. Testing was carried out using a gazebo simulation and validated in multiple tests in the field. We show for the first time that ejecting modular failed components can improve the overall mission length.

One of the greatest challenges to the automated production of goods is equipment malfunction. Ideally, machines should be able to automatically predict and detect operational faults in order to minimize downtime and plan for timely maintenance. While traditional condition-based maintenance (CBM) involves costly sensor additions and engineering, machine learning approaches offer the potential to learn from already existing sensors. Implementations of data-driven CBM typically use supervised and semi-supervised learning to classify faults. In addition to a large collection of operation data, records of faulty operation are also necessary, which are often costly to obtain. Instead of classifying faults, we use an approach to detect abnormal behaviour within the machine’s operation. This approach is analogous to semi-supervised anomaly detection in machine learning (ML), with important distinctions in experimental design and evaluation specific to the problem of industrial fault detection. We present a novel method of machine fault detection using temporal-difference learning and General Value Functions (GVFs). Using GVFs, we form a predictive model of sensor data to detect faulty behaviour. As sensor data from machines is not i.i.d. but closer to Markovian sampling, temporal-difference learning methods should be well suited for this data. We compare our GVF outlier detection (GVFOD) algorithm to a broad selection of multivariate and temporal outlier detection methods, using datasets collected from a tabletop robot emulating the movement of an industrial actuator. We find that not only does GVFOD achieve the same recall score as other multivariate OD algorithms, it attains significantly higher precision. Furthermore, GVFOD has intuitive hyperparameters which can be selected based upon expert knowledge of the application. Together, these findings allow for a more reliable detection of abnormal machine behaviour to allow ideal timing of maintenance; saving resources, time and cost.

Introduction: Children and adolescents with neurological impairments face reduced participation and independence in daily life activities due to walking difficulties. Existing assistive devices often offer insufficient support, potentially leading to wheelchair dependence and limiting physical activity and daily life engagement. Mobile wearable robots, such as exoskeletons and exosuits, have shown promise in supporting adults during activities of daily living but are underexplored for children.

Methods: We conducted a cross-sectional study to examine the potential of a cable-driven exosuit, the Myosuit, to enhance walking efficiency in adolescents with diverse ambulatory impairments. Each participant walked a course including up-hill, down-hill, level ground walking, and stairs ascending and descending, with and without the exosuit’s assistance. We monitored the time and step count to complete the course and the average heart rate and muscle activity. Additionally, we assessed the adolescents’ perspective on the exosuit’s utility using a visual analog scale.

Results: Six adolescents completed the study. Although not statistically significant, five participants completed the course with the exosuit’s assistance in reduced time (time reduction range: [-3.87, 17.42]%, p-value: 0.08, effect size: 0.88). The number of steps taken decreased significantly with the Myosuit’s assistance (steps reduction range: [1.07, 15.71]%, p-value: 0.04, effect size: 0.90). Heart rate and muscle activity did not differ between Myosuit-assisted and unassisted conditions (p-value: 0.96 and 0.35, effect size: 0.02 and 0.42, respectively). Participants generally perceived reduced effort and increased safety with the Myosuit’s assistance, especially during tasks involving concentric contractions (e.g., walking uphill). Three participants expressed a willingness to use the Myosuit in daily life, while the others found it heavy or too conspicuous.

Discussion: Increased walking speed without increasing physical effort when performing activities of daily living could lead to higher levels of participation and increased functional independence. Despite perceiving the benefits introduced by the exosuit’s assistance, adolescents reported the need for further modification of the device design before using it extensively at home and in the community.

Heterogeneous multi-agent systems can be deployed to complete a variety of tasks, including some that are impossible using a single generic modality. This paper introduces an approach to solving the problem of cooperative behavior planning in small heterogeneous robot teams where members can both function independently as well as physically interact with each other in ways that give rise to additional functionality. This approach enables, for the first time, the cooperative completion of tasks that are infeasible when using any single modality from those agents comprising the team.

Introduction: It is crucial to identify neurodevelopmental disorders in infants early on for timely intervention to improve their long-term outcomes. Combining natural play with quantitative measurements of developmental milestones can be an effective way to swiftly and efficiently detect infants who are at risk of neurodevelopmental delays. Clinical studies have established differences in toy interaction behaviors between full-term infants and pre-term infants who are at risk for cerebral palsy and other developmental disorders.

Methods: The proposed toy aims to improve the quantitative assessment of infant-toy interactions and fully automate the process of detecting those infants at risk of developing motor delays. This paper describes the design and development of a toy that uniquely utilizes a collection of soft lossy force sensors which are developed using optical fibers to gather play interaction data from infants laying supine in a gym. An example interaction database was created by having 15 adults complete a total of 2480 interactions with the toy consisting of 620 touches, 620 punches—“kick substitute,” 620 weak grasps and 620 strong grasps.

Results: The data is analyzed for patterns of interaction with the toy face using a machine learning model developed to classify the four interactions present in the database. Results indicate that the configuration of 6 soft force sensors on the face created unique activation patterns.

Discussion: The machine learning algorithm was able to identify the distinct action types from the data, suggesting the potential usability of the toy. Next steps involve sensorizing the entire toy and testing with infants.

The environmental pollution caused by various sources has escalated the climate crisis making the need to establish reliable, intelligent, and persistent environmental monitoring solutions more crucial than ever. Mobile sensing systems are a popular platform due to their cost-effectiveness and adaptability. However, in practice, operation environments demand highly intelligent and robust systems that can cope with an environment’s changing dynamics. To achieve this reinforcement learning has become a popular tool as it facilitates the training of intelligent and robust sensing agents that can handle unknown and extreme conditions. In this paper, a framework that formulates active sensing as a reinforcement learning problem is proposed. This framework allows unification with multiple essential environmental monitoring tasks and algorithms such as coverage, patrolling, source seeking, exploration and search and rescue. The unified framework represents a step towards bridging the divide between theoretical advancements in reinforcement learning and real-world applications in environmental monitoring. A critical review of the literature in this field is carried out and it is found that despite the potential of reinforcement learning for environmental active sensing applications there is still a lack of practical implementation and most work remains in the simulation phase. It is also noted that despite the consensus that, multi-agent systems are crucial to fully realize the potential of active sensing there is a lack of research in this area.



When IEEE Spectrum first wrote about Covariant in 2020, it was a new-ish robotics startup looking to apply robotics to warehouse picking at scale through the magic of a single end-to-end neural network. At the time, Covariant was focused on this picking use case, because it represents an application that could provide immediate value—warehouse companies pay Covariant for its robots to pick items in their warehouses. But for Covariant, the exciting part was that picking items in warehouses has, over the last four years, yielded a massive amount of real-world manipulation data—and you can probably guess where this is going.

Today, Covariant is announcing RFM-1, which the company describes as a robotics foundation model that gives robots the “human-like ability to reason.” That’s from the press release, and while I wouldn’t necessarily read too much into “human-like” or “reason,” what Covariant has going on here is pretty cool.

“Foundation model” means that RFM-1 can be trained on more data to do more things—at the moment, it’s all about warehouse manipulation because that’s what it’s been trained on, but its capabilities can be expanded by feeding it more data. “Our existing system is already good enough to do very fast, very variable pick and place,” says Covariant co-founder Pieter Abbeel. “But we’re now taking it quite a bit further. Any task, any embodiment—that’s the long-term vision. Robotics foundation models powering billions of robots across the world.” From the sound of things, Covariant’s business of deploying a large fleet of warehouse automation robots was the fastest way for them to collect the tens of millions of trajectories (how a robot moves during a task) that they needed to train the 8 billion parameter RFM-1 model.

Covariant

“The only way you can do what we’re doing is by having robots deployed in the world collecting a ton of data,” says Abbeel. “Which is what allows us to train a robotics foundation model that’s uniquely capable.”

There have been other attempts at this sort of thing: The RTX project is one recent example. But while RT-X depends on research labs sharing what data they have to create a dataset that’s large enough to be useful, Covariant is doing it alone, thanks to its fleet of warehouse robots. “RT-X is about a million trajectories of data,” Abbeel says, “but we’re able to surpass it because we’re getting a million trajectories every few weeks.”

“By building a valuable picking robot that’s deployed across 15 countries with dozens of customers, we essentially have a data collection machine.” —Pieter Abbeel, Covariant

You can think of the current execution of RFM-1 as a prediction engine for suction-based object manipulation in warehouse environments. The model incorporates still images, video, joint angles, force reading, suction cup strength—everything involved in the kind of robotic manipulation that Covariant does. All of these things are interconnected within RFM-1, which means that you can put any of those things into one end of RFM-1, and out of the other end of the model will come a prediction. That prediction can be in the form of an image, a video, or a series of commands for a robot.

What’s important to understand about all of this is that RFM-1 isn’t restricted to picking only things it’s seen before, or only working on robots it has direct experience with. This is what’s nice about foundation models—they can generalize within the domain of their training data, and it’s how Covariant has been able to scale their business as successfully as they have, by not having to retrain for every new picking robot or every new item. What’s counter-intuitive about these large models is that they’re actually better at dealing with new situations than models that are trained specifically for those situations.

For example, let’s say you want to train a model to drive a car on a highway. The question, Abbeel says, is whether it would be worth your time to train on other kinds of driving anyway. The answer is yes, because highway driving is sometimes not highway driving. There will be accidents or rush hour traffic that will require you to drive differently. If you’ve also trained on driving on city streets, you’re effectively training on highway edge cases, which will come in handy at some point and improve performance overall. With RFM-1, it’s the same idea: Training on lots of different kinds of manipulation—different robots, different objects, and so on—means that any single kind of manipulation will be that much more capable.

In the context of generalization, Covariant talks about RFM-1’s ability to “understand” its environment. This can be a tricky word with AI, but what’s relevant is to ground the meaning of “understand” in what RFM-1 is capable of. For example, you don’t need to understand physics to be able to catch a baseball, you just need to have a lot of experience catching baseballs, and that’s where RFM-1 is at. You could also reason out how to catch a baseball with no experience but an understanding of physics, and RFM-1 is not doing this, which is why I hesitate to use the word “understand” in this context.

But this brings us to another interesting capability of RFM-1: it operates as a very effective, if constrained, simulation tool. As a prediction engine that outputs video, you can ask it to generate what the next couple seconds of an action sequence will look like, and it’ll give you a result that’s both realistic and accurate, being grounded in all of its data. The key here is that RFM-1 can effectively simulate objects that are challenging to simulate traditionally, like floppy things.

Covariant’s Abbeel explains that the “world model” that RFM-1 bases its predictions on is effectively a learned physics engine. “Building physics engines turns out to be a very daunting task to really cover every possible thing that can happen in the world,” Abbeel says. “Once you get complicated scenarios, it becomes very inaccurate, very quickly, because people have to make all kinds of approximations to make the physics engine run on a computer. We’re just doing the large-scale data version of this with a world model, and it’s showing really good results.”

Abbeel gives an example of asking a robot to simulate (or predict) what would happen if a cylinder is placed vertically on a conveyor belt. The prediction accurately shows the cylinder falling over and rolling when the belt starts to move—not because the cylinder is being simulated, but because RFM-1 has seen a lot of things being placed on a lot of conveyor belts.

“Five years from now, it’s not unlikely that what we are building here will be the only type of simulator anyone will ever use.” —Pieter Abbeel, Covariant

This only works if there’s the right kind of data for RFM-1 to train on, so unlike most simulation environments, it can’t currently generalize to completely new objects or situations. But Abbeel believes that with enough data, useful world simulation will be possible. “Five years from now, it’s not unlikely that what we are building here will be the only type of simulator anyone will ever use. It’s a more capable simulator than one built from the ground up with collision checking and finite elements and all that stuff. All those things are so hard to build into your physics engine in any kind of way, not to mention the renderer to make things look like they look in the real world—in some sense, we’re taking a shortcut.”

RFM-1 also incorporates language data to be able to communicate more effectively with humans. Covariant

For Covariant to expand the capabilities of RFM-1 towards that long-term vision of foundation models powering “billions of robots across the world,” the next step is to feed it more data from a wider variety of robots doing a wider variety of tasks. “We’ve built essentially a data ingestion engine,” Abbeel says. “If you’re willing to give us data of a different type, we’ll ingest that too.”

“We have a lot of confidence that this kind of model could power all kinds of robots—maybe with more data for the types of robots and types of situations it could be used in.” —Pieter Abbeel, Covariant

One way or another, that path is going to involve a heck of a lot of data, and it’s going to be data that Covariant is not currently collecting with its own fleet of warehouse manipulation robots. So if you’re, say, a humanoid robotics company, what’s your incentive to share all the data you’ve been collecting with Covariant? “The pitch is that we’ll help them get to the real world,” Covariant co-founder Peter Chen says. “I don’t think there are really that many companies that have AI to make their robots truly autonomous in a production environment. If they want AI that’s robust and powerful and can actually help them enter the real world, we are really their best bet.”

Covariant’s core argument here is that while it’s certainly possible for every robotics company to train up their own models individually, the performance—for anybody trying to do manipulation, at least—would be not nearly as good as using a model that incorporates all of the manipulation data that Covariant already has within RFM-1. “It has always been our long term plan to be a robotics foundation model company,” says Chen. “There was just not sufficient data and compute and algorithms to get to this point—but building a universal AI platform for robots, that’s what Covariant has been about from the very beginning.”



When IEEE Spectrum first wrote about Covariant in 2020, it was a new-ish robotics startup looking to apply robotics to warehouse picking at scale through the magic of a single end-to-end neural network. At the time, Covariant was focused on this picking use case, because it represents an application that could provide immediate value—warehouse companies pay Covariant for its robots to pick items in their warehouses. But for Covariant, the exciting part was that picking items in warehouses has, over the last four years, yielded a massive amount of real-world manipulation data—and you can probably guess where this is going.

Today, Covariant is announcing RFM-1, which the company describes as a robotics foundation model that gives robots the “human-like ability to reason.” That’s from the press release, and while I wouldn’t necessarily read too much into “human-like” or “reason,” what Covariant has going on here is pretty cool.

“Foundation model” means that RFM-1 can be trained on more data to do more things—at the moment, it’s all about warehouse manipulation because that’s what it’s been trained on, but its capabilities can be expanded by feeding it more data. “Our existing system is already good enough to do very fast, very variable pick and place,” says Covariant co-founder Pieter Abbeel. “But we’re now taking it quite a bit further. Any task, any embodiment—that’s the long-term vision. Robotics foundation models powering billions of robots across the world.” From the sound of things, Covariant’s business of deploying a large fleet of warehouse automation robots was the fastest way for them to collect the tens of millions of trajectories (how a robot moves during a task) that they needed to train the 8 billion parameter RFM-1 model.

Covariant

“The only way you can do what we’re doing is by having robots deployed in the world collecting a ton of data,” says Abbeel. “Which is what allows us to train a robotics foundation model that’s uniquely capable.”

There have been other attempts at this sort of thing: The RTX project is one recent example. But while RT-X depends on research labs sharing what data they have to create a dataset that’s large enough to be useful, Covariant is doing it alone, thanks to its fleet of warehouse robots. “RT-X is about a million trajectories of data,” Abbeel says, “but we’re able to surpass it because we’re getting a million trajectories every few weeks.”

“By building a valuable picking robot that’s deployed across 15 countries with dozens of customers, we essentially have a data collection machine.” —Pieter Abbeel, Covariant

You can think of the current execution of RFM-1 as a prediction engine for suction-based object manipulation in warehouse environments. The model incorporates still images, video, joint angles, force reading, suction cup strength—everything involved in the kind of robotic manipulation that Covariant does. All of these things are interconnected within RFM-1, which means that you can put any of those things into one end of RFM-1, and out of the other end of the model will come a prediction. That prediction can be in the form of an image, a video, or a series of commands for a robot.

What’s important to understand about all of this is that RFM-1 isn’t restricted to picking only things it’s seen before, or only working on robots it has direct experience with. This is what’s nice about foundation models—they can generalize within the domain of their training data, and it’s how Covariant has been able to scale their business as successfully as they have, by not having to retrain for every new picking robot or every new item. What’s counter-intuitive about these large models is that they’re actually better at dealing with new situations than models that are trained specifically for those situations.

For example, let’s say you want to train a model to drive a car on a highway. The question, Abbeel says, is whether it would be worth your time to train on other kinds of driving anyway. The answer is yes, because highway driving is sometimes not highway driving. There will be accidents or rush hour traffic that will require you to drive differently. If you’ve also trained on driving on city streets, you’re effectively training on highway edge cases, which will come in handy at some point and improve performance overall. With RFM-1, it’s the same idea: Training on lots of different kinds of manipulation—different robots, different objects, and so on—means that any single kind of manipulation will be that much more capable.

In the context of generalization, Covariant talks about RFM-1’s ability to “understand” its environment. This can be a tricky word with AI, but what’s relevant is to ground the meaning of “understand” in what RFM-1 is capable of. For example, you don’t need to understand physics to be able to catch a baseball, you just need to have a lot of experience catching baseballs, and that’s where RFM-1 is at. You could also reason out how to catch a baseball with no experience but an understanding of physics, and RFM-1 is not doing this, which is why I hesitate to use the word “understand” in this context.

But this brings us to another interesting capability of RFM-1: it operates as a very effective, if constrained, simulation tool. As a prediction engine that outputs video, you can ask it to generate what the next couple seconds of an action sequence will look like, and it’ll give you a result that’s both realistic and accurate, being grounded in all of its data. The key here is that RFM-1 can effectively simulate objects that are challenging to simulate traditionally, like floppy things.

Covariant’s Abbeel explains that the “world model” that RFM-1 bases its predictions on is effectively a learned physics engine. “Building physics engines turns out to be a very daunting task to really cover every possible thing that can happen in the world,” Abbeel says. “Once you get complicated scenarios, it becomes very inaccurate, very quickly, because people have to make all kinds of approximations to make the physics engine run on a computer. We’re just doing the large-scale data version of this with a world model, and it’s showing really good results.”

Abbeel gives an example of asking a robot to simulate (or predict) what would happen if a cylinder is placed vertically on a conveyor belt. The prediction accurately shows the cylinder falling over and rolling when the belt starts to move—not because the cylinder is being simulated, but because RFM-1 has seen a lot of things being placed on a lot of conveyor belts.

“Five years from now, it’s not unlikely that what we are building here will be the only type of simulator anyone will ever use.” —Pieter Abbeel, Covariant

This only works if there’s the right kind of data for RFM-1 to train on, so unlike most simulation environments, it can’t currently generalize to completely new objects or situations. But Abbeel believes that with enough data, useful world simulation will be possible. “Five years from now, it’s not unlikely that what we are building here will be the only type of simulator anyone will ever use. It’s a more capable simulator than one built from the ground up with collision checking and finite elements and all that stuff. All those things are so hard to build into your physics engine in any kind of way, not to mention the renderer to make things look like they look in the real world—in some sense, we’re taking a shortcut.”

RFM-1 also incorporates language data to be able to communicate more effectively with humans. Covariant

For Covariant to expand the capabilities of RFM-1 towards that long-term vision of foundation models powering “billions of robots across the world,” the next step is to feed it more data from a wider variety of robots doing a wider variety of tasks. “We’ve built essentially a data ingestion engine,” Abbeel says. “If you’re willing to give us data of a different type, we’ll ingest that too.”

“We have a lot of confidence that this kind of model could power all kinds of robots—maybe with more data for the types of robots and types of situations it could be used in.” —Pieter Abbeel, Covariant

One way or another, that path is going to involve a heck of a lot of data, and it’s going to be data that Covariant is not currently collecting with its own fleet of warehouse manipulation robots. So if you’re, say, a humanoid robotics company, what’s your incentive to share all the data you’ve been collecting with Covariant? “The pitch is that we’ll help them get to the real world,” Covariant co-founder Peter Chen says. “I don’t think there are really that many companies that have AI to make their robots truly autonomous in a production environment. If they want AI that’s robust and powerful and can actually help them enter the real world, we are really their best bet.”

Covariant’s core argument here is that while it’s certainly possible for every robotics company to train up their own models individually, the performance—for anybody trying to do manipulation, at least—would be not nearly as good as using a model that incorporates all of the manipulation data that Covariant already has within RFM-1. “It has always been our long term plan to be a robotics foundation model company,” says Chen. “There was just not sufficient data and compute and algorithms to get to this point—but building a universal AI platform for robots, that’s what Covariant has been about from the very beginning.”



The global ocean is difficult to explore—the common refrain is that we know less about the deep ocean than we do about the surface of the moon. Australian company Advanced Navigation wants to change that with a pint-sized autonomous underwater vehicle (AUV) that it hopes will become the maritime equivalent of a consumer drone. And the new AUV is already getting to work mapping and monitoring Australia’s coral reefs and diving for shipwrecks.

The Sydney-based company has been developing underwater navigation technology for more than a decade. In 2022, Advanced Navigation unveiled its first in-house AUV, called Hydrus. At less than half a meter long, the vehicle is considerably smaller than most alternatives. Even so, it’s fully autonomous and carries a 4k-resolution camera capable of 60 frames per second that can both capture high-definition video and construct detailed 3D photogrammetry models.

Advanced Navigation says Hydrus—with a depth rating of 3,000 meters, a range of 9 kilometers, and a battery that lasts up to three hours—is capable of a wide variety of missions. The company recently sold two units to the Australian Institute of Marine Science (AIMS), the country’s tropical marine science agency, which will use them to survey coral reefs in the North West Shelf region off the country’s west coast. Hydrus has also recently collaborated with the Western Australian Museum to produce a detailed 3D model of a shipwreck off the coast near Perth.

“If people can go and throw one of these off the boat, just like they can throw a drone up in the air, that will obviously benefit the exploration of the sea.” —Ross Anderson, Western Australian Museum

After many years of supplying components to other robotics companies, Peter Baker, subsea product manager at Advanced Navigation, says they company spotted a gap in the market. “We wanted to take the user experience that someone would have with an aerial drone and bring that underwater,” he says. “It’s very expensive to get images and data of the seabed. So by being able to miniaturize this system, and have it drastically simplified from the user’s point of view, it makes data a lot more accessible to people.”

But building a compact and low-cost AUV is not simple. The deep ocean is not a friendly place for electronics, says Baker, due to a combination of high pressure and corrosive seawater. The traditional way of dealing with this is to stick all the critical components in a sealed titanium tube that can maintain ambient pressure and keep moisture out. However, this requires you to add buoyancy to compensate for the extra weight, says Baker, which increases the bulk of the vehicle. That means bigger motors and bigger batteries. “The whole thing spirals up and up until you’ve got something the size of a minibus,” he says.

Advanced Navigation got around the spiral by designing bespoke pressure-tolerant electronics. They built all of their circuit boards from scratch, carefully selecting components that had been tested to destruction in a hydrostatic pressure chamber. These were then encapsulated in a water-proof composite shell, and to further reduce the risk of water ingress the drone operates completely wirelessly. Batteries are recharged using inductive charging and data transfer is either over Wi-Fi when above water or via an optical modem when below the surface.

Hydrus AUVs are charged using induction to keep corrosive seawater from leaking in through charging ports.Advanced Navigation

This has allowed the company to significantly miniaturize the system, says Baker, which has a drastic impact on the overall cost of operations. “You don’t need a crane or a winch or anything like that to recover the vehicle, you can pick it up with a fishing net,” he says. “You can get away with using a much smaller boat, and the rule of thumb in the industry is if you double the size of your boat, you quadruple the cost.”

Just as important, though, is the vehicle’s ease of use. Most underwater robotics systems still operate with a tether, says Baker, but Hydrus carries all the hardware required to support autonomous navigation on board. The company’s “bread and butter” is inertial navigation technology, which uses accelerometers and gyroscopes to track the vehicle from a known starting point. But the drone also features a sonar system that allows it to stay a set distance from the seabed and also judge its speed by measuring the Doppler shift on echoes as they bounce back.

This means that users can simply program in a set of way points on a map, toss the vehicle overboard and leave it to its own devices, says Baker. The Hydrus does have a low-bandwidth acoustic communication channel that allows the operator to send basic commands like “stop” or “come home,” he says, but Hydrus is designed to be a set-and-forget AUV. “That really lowers the thresholds of what a user needs to be able to operate it,” he says. “If you can fly a DJI drone you could fly a Hydrus.”

The company estimates for a typical seabed investigation in water shallow enough for human divers, the Hydrus could be 75 percent cheaper than alternatives. And the savings would go up significantly at greater depths. What’s more, says Baker, the drone’s precise navigation means it can produce much more consistent and repeatable data.

To demonstrate its capabilities, Hydrus’ designers went hunting for shipwrecks in the Rottnest ships graveyard just off the coast near Perth, in Western Australia. The site was a designated spot for scuttling aging ships, says Ross Anderson, curator at Western Australian Museum, but has yet to be fully explored due to the depth of many of the wrecks.

The Advanced Navigation team used the Hydrus to create a detailed 3D model of a sunken “coal hulk”—one of a category of old iron sailing ships that were later converted to floating coal warehouses for steamships. The Western Australian Museum has been unable to identify the vessel so far, but Anderson says these kind of models can be hugely beneficial for carrying out maritime archaeology research, as well as educating people about what’s below the waves.


Advanced Navigation used its new Hydrus drone to create a detailed 3D image of an unidentified “coal hulk” ship in the Rottnest ships graveyard off the western coast of Australia.

Advanced Navigation

Any technology that can simplify the process is greatly welcomed, Anderson adds. “If people can go and throw one of these off the boat, just like they can throw a drone up in the air, that will obviously benefit the exploration of the sea,” he says.

Ease of use was also a big driver behind AIMS’s purchase of two Hydrus drones, says technology development program lead Melanie Olsen, who is also an IEEE senior member. Most of the technology available for marine science is still research-grade and a long way from a polished, professional product.

“When you’re an operational agency like AIMS, you typically don’t have the luxury of spending a lot of time on the back of the boat getting equipment ready,” says Olsen. “You need something that users can turn on and go and it’s just working, as time is of the essence.”

Another benefit of the Hydrus for AIMS is that the drone can operate at greater depths than divers and in conditions that would be dangerous for humans. “Its enabling our researchers to see further down in the water and also operate in more dangerous situations such as at night, or in the presence of threats such as crocodiles or sharks, places where we just wouldn’t be able to collect that data,” says Olsen.

The agency will initially use the drones to survey reefs on Australia’s North West Shelf, including Scott Reef and Ashmore Reef. The goal is to collect regular data data on coral health to monitor the state of the reefs, investigate how they’re being effected by climate change, and hopefully get early warning of emerging problems. But Olsen says they expect that the Hydrus will become standard part of their ocean monitoring toolkit going forward.

This story was updated on 11 March 2024 to correct the year when Advanced Navigation unveiled Hydrus.



The global ocean is difficult to explore—the common refrain is that we know less about the deep ocean than we do about the surface of the moon. Australian company Advanced Navigation wants to change that with a pint-sized autonomous underwater vehicle (AUV) that it hopes will become the maritime equivalent of a consumer drone. And the new AUV is already getting to work mapping and monitoring Australia’s coral reefs and diving for shipwrecks.

The Sydney-based company has been developing underwater navigation technology for more than a decade. In 2022, Advanced Navigation unveiled its first in-house AUV, called Hydrus. At less than half a meter long, the vehicle is considerably smaller than most alternatives. Even so, it’s fully autonomous and carries a 4k-resolution camera capable of 60 frames per second that can both capture high-definition video and construct detailed 3D photogrammetry models.

Advanced Navigation says Hydrus—with a depth rating of 3,000 meters, a range of 9 kilometers, and a battery that lasts up to three hours—is capable of a wide variety of missions. The company recently sold two units to the Australian Institute of Marine Science (AIMS), the country’s tropical marine science agency, which will use them to survey coral reefs in the North West Shelf region off the country’s west coast. Hydrus has also recently collaborated with the Western Australian Museum to produce a detailed 3D model of a shipwreck off the coast near Perth.

“If people can go and throw one of these off the boat, just like they can throw a drone up in the air, that will obviously benefit the exploration of the sea.” —Ross Anderson, Western Australian Museum

After many years of supplying components to other robotics companies, Peter Baker, subsea product manager at Advanced Navigation, says they company spotted a gap in the market. “We wanted to take the user experience that someone would have with an aerial drone and bring that underwater,” he says. “It’s very expensive to get images and data of the seabed. So by being able to miniaturize this system, and have it drastically simplified from the user’s point of view, it makes data a lot more accessible to people.”

But building a compact and low-cost AUV is not simple. The deep ocean is not a friendly place for electronics, says Baker, due to a combination of high pressure and corrosive seawater. The traditional way of dealing with this is to stick all the critical components in a sealed titanium tube that can maintain ambient pressure and keep moisture out. However, this requires you to add buoyancy to compensate for the extra weight, says Baker, which increases the bulk of the vehicle. That means bigger motors and bigger batteries. “The whole thing spirals up and up until you’ve got something the size of a minibus,” he says.

Advanced Navigation got around the spiral by designing bespoke pressure-tolerant electronics. They built all of their circuit boards from scratch, carefully selecting components that had been tested to destruction in a hydrostatic pressure chamber. These were then encapsulated in a water-proof composite shell, and to further reduce the risk of water ingress the drone operates completely wirelessly. Batteries are recharged using inductive charging and data transfer is either over Wi-Fi when above water or via an optical modem when below the surface.

Hydrus AUVs are charged using induction to keep corrosive seawater from leaking in through charging ports.Advanced Navigation

This has allowed the company to significantly miniaturize the system, says Baker, which has a drastic impact on the overall cost of operations. “You don’t need a crane or a winch or anything like that to recover the vehicle, you can pick it up with a fishing net,” he says. “You can get away with using a much smaller boat, and the rule of thumb in the industry is if you double the size of your boat, you quadruple the cost.”

Just as important, though, is the vehicle’s ease of use. Most underwater robotics systems still operate with a tether, says Baker, but Hydrus carries all the hardware required to support autonomous navigation on board. The company’s “bread and butter” is inertial navigation technology, which uses accelerometers and gyroscopes to track the vehicle from a known starting point. But the drone also features a sonar system that allows it to stay a set distance from the seabed and also judge its speed by measuring the Doppler shift on echoes as they bounce back.

This means that users can simply program in a set of way points on a map, toss the vehicle overboard and leave it to its own devices, says Baker. The Hydrus does have a low-bandwidth acoustic communication channel that allows the operator to send basic commands like “stop” or “come home,” he says, but Hydrus is designed to be a set-and-forget AUV. “That really lowers the thresholds of what a user needs to be able to operate it,” he says. “If you can fly a DJI drone you could fly a Hydrus.”

The company estimates for a typical seabed investigation in water shallow enough for human divers, the Hydrus could be 75 percent cheaper than alternatives. And the savings would go up significantly at greater depths. What’s more, says Baker, the drone’s precise navigation means it can produce much more consistent and repeatable data.

To demonstrate its capabilities, Hydrus’ designers went hunting for shipwrecks in the Rottnest ships graveyard just off the coast near Perth, in Western Australia. The site was a designated spot for scuttling aging ships, says Ross Anderson, curator at Western Australian Museum, but has yet to be fully explored due to the depth of many of the wrecks.

The Advanced Navigation team used the Hydrus to create a detailed 3D model of a sunken “coal hulk”—one of a category of old iron sailing ships that were later converted to floating coal warehouses for steamships. The Western Australian Museum has been unable to identify the vessel so far, but Anderson says these kind of models can be hugely beneficial for carrying out maritime archaeology research, as well as educating people about what’s below the waves.


Advanced Navigation used its new Hydrus drone to create a detailed 3D image of an unidentified “coal hulk” ship in the Rottnest ships graveyard off the western coast of Australia.

Advanced Navigation

Any technology that can simplify the process is greatly welcomed, Anderson adds. “If people can go and throw one of these off the boat, just like they can throw a drone up in the air, that will obviously benefit the exploration of the sea,” he says.

Ease of use was also a big driver behind AIMS’s purchase of two Hydrus drones, says technology development program lead Melanie Olsen, who is also an IEEE senior member. Most of the technology available for marine science is still research-grade and a long way from a polished, professional product.

“When you’re an operational agency like AIMS, you typically don’t have the luxury of spending a lot of time on the back of the boat getting equipment ready,” says Olsen. “You need something that users can turn on and go and it’s just working, as time is of the essence.”

Another benefit of the Hydrus for AIMS is that the drone can operate at greater depths than divers and in conditions that would be dangerous for humans. “Its enabling our researchers to see further down in the water and also operate in more dangerous situations such as at night, or in the presence of threats such as crocodiles or sharks, places where we just wouldn’t be able to collect that data,” says Olsen.

The agency will initially use the drones to survey reefs on Australia’s North West Shelf, including Scott Reef and Ashmore Reef. The goal is to collect regular data data on coral health to monitor the state of the reefs, investigate how they’re being effected by climate change, and hopefully get early warning of emerging problems. But Olsen says they expect that the Hydrus will become standard part of their ocean monitoring toolkit going forward.

This story was updated on 11 March 2024 to correct the year when Advanced Navigation unveiled Hydrus.



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPANRoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS

Enjoy today’s videos!

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time, whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. We successfully achieve teleoperation of dynamic, whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based, real-time, whole-body humanoid teleoperation.

[ CMU ]

Legged robots have the potential to traverse complex terrain and access confined spaces beyond the reach of traditional platforms thanks to their ability to carefully select footholds and flexibly adapt their body posture while walking. However, robust deployment in real-world applications is still an open challenge. In this paper, we present a method for legged locomotion control using reinforcement learning and 3D volumetric representations to enable robust and versatile locomotion in confined and unstructured environments.

[ Takahiro Miki ]

Sure, 3.3 meters per second is fast for a humanoid, but I’m more impressed by the spinning around while walking downstairs.

[ Unitree ]

Improving the safety of collaborative manipulators necessitates the reduction of inertia in the moving part. We introduce a novel approach in the form of a passive, 3D wire aligner, serving as a lightweight and low-friction power transmission mechanism, thus achieving the desired low inertia in the manipulator’s operation.

[ SAQIEL ]

Thanks, Temma!

Robot Era just launched Humanoid-Gym, an open-source reinforcement learning framework for bipedal humanoids. As you can see from the video, RL algorithms have given the robot, called Xiao Xing, or XBot, the ability to climb up and down haphazardly stacked boxes with relative stability and ease.

[ Robot Era ]

“Impact-Aware Bimanual Catching of Large-Momentum Objects.” Need I say more?

[ SLMC ]

More than 80% of stroke survivors experience walking difficulty, significantly impacting their daily lives, independence, and overall quality of life. Now, new research from the University of Massachusetts Amherst pushes forward the bounds of stroke recovery with a unique robotic hip exoskeleton, designed as a training tool to improve walking function. This invites the possibility of new therapies that are more accessible and easier to translate from practice to daily life, compared to current rehabilitation methods.

[ UMass Amherst ]

Thanks, Julia!

The manipulation here is pretty impressive, but it’s hard to know how impressive without also knowing how much the video was sped up.

[ Somatic ]

DJI drones work to make the world a better place and one of the ways that we do this is through conservation work. We partnered with Halo Robotics and the OFI Orangutan Foundation International to showcase just how these drones can make an impact.

[ DJI ]

The aim of the test is to demonstrate the removal and replacement of satellite modules into a 27U CubeSat format using augmented reality control of a robot. In this use case, the “client” satellite is being upgraded and refueled using modular componentry. The robot will then remove the failed computer module and place it in a fixture. It will then do the same with the propellant tank. The robot will then place these correctly back into the satellite.

[ Extend Robotics ]

This video features some of the highlights and favorite moments from the CYBATHLON Challenges 2024 that took place on 2 February, showing so many diverse types of assistive technology taking on discipline tasks and displaying pilots’ tenacity and determination. The Challenges saw new teams, new tasks, and new formats for many of the CYBATHLON disciplines.

[ Cybathlon ]

It’s been a long road to electrically powered robots.

[ ABB ]

Small drones for catastrophic wildfires (ones covering more than [40,470 hectares]) are like bringing a flashlight to light up a football field. This short video describes the major uses for drones of all sizes and why and when they are used, or why not.

[ CRASAR ]

It probably will not surprise you that there are a lot of robots involved in building Rivian trucks and vans.

[ Kawasaki Robotics ]

DARPA’s Learning Introspective Control (LINC) program is developing machine learning methods that show promise in making that scenario closer to reality. LINC aims to fundamentally improve the safety of mechanical systems—specifically in ground vehicles, ships, drone swarms, and robotics—using various methods that require minimal computing power. The result is an AI-powered controller the size of a cell phone.

[ DARPA ]



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPANRoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS

Enjoy today’s videos!

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time, whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. We successfully achieve teleoperation of dynamic, whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based, real-time, whole-body humanoid teleoperation.

[ CMU ]

Legged robots have the potential to traverse complex terrain and access confined spaces beyond the reach of traditional platforms thanks to their ability to carefully select footholds and flexibly adapt their body posture while walking. However, robust deployment in real-world applications is still an open challenge. In this paper, we present a method for legged locomotion control using reinforcement learning and 3D volumetric representations to enable robust and versatile locomotion in confined and unstructured environments.

[ Takahiro Miki ]

Sure, 3.3 meters per second is fast for a humanoid, but I’m more impressed by the spinning around while walking downstairs.

[ Unitree ]

Improving the safety of collaborative manipulators necessitates the reduction of inertia in the moving part. We introduce a novel approach in the form of a passive, 3D wire aligner, serving as a lightweight and low-friction power transmission mechanism, thus achieving the desired low inertia in the manipulator’s operation.

[ SAQIEL ]

Thanks, Temma!

Robot Era just launched Humanoid-Gym, an open-source reinforcement learning framework for bipedal humanoids. As you can see from the video, RL algorithms have given the robot, called Xiao Xing, or XBot, the ability to climb up and down haphazardly stacked boxes with relative stability and ease.

[ Robot Era ]

“Impact-Aware Bimanual Catching of Large-Momentum Objects.” Need I say more?

[ SLMC ]

More than 80% of stroke survivors experience walking difficulty, significantly impacting their daily lives, independence, and overall quality of life. Now, new research from the University of Massachusetts Amherst pushes forward the bounds of stroke recovery with a unique robotic hip exoskeleton, designed as a training tool to improve walking function. This invites the possibility of new therapies that are more accessible and easier to translate from practice to daily life, compared to current rehabilitation methods.

[ UMass Amherst ]

Thanks, Julia!

The manipulation here is pretty impressive, but it’s hard to know how impressive without also knowing how much the video was sped up.

[ Somatic ]

DJI drones work to make the world a better place and one of the ways that we do this is through conservation work. We partnered with Halo Robotics and the OFI Orangutan Foundation International to showcase just how these drones can make an impact.

[ DJI ]

The aim of the test is to demonstrate the removal and replacement of satellite modules into a 27U CubeSat format using augmented reality control of a robot. In this use case, the “client” satellite is being upgraded and refueled using modular componentry. The robot will then remove the failed computer module and place it in a fixture. It will then do the same with the propellant tank. The robot will then place these correctly back into the satellite.

[ Extend Robotics ]

This video features some of the highlights and favorite moments from the CYBATHLON Challenges 2024 that took place on 2 February, showing so many diverse types of assistive technology taking on discipline tasks and displaying pilots’ tenacity and determination. The Challenges saw new teams, new tasks, and new formats for many of the CYBATHLON disciplines.

[ Cybathlon ]

It’s been a long road to electrically powered robots.

[ ABB ]

Small drones for catastrophic wildfires (ones covering more than [40,470 hectares]) are like bringing a flashlight to light up a football field. This short video describes the major uses for drones of all sizes and why and when they are used, or why not.

[ CRASAR ]

It probably will not surprise you that there are a lot of robots involved in building Rivian trucks and vans.

[ Kawasaki Robotics ]

DARPA’s Learning Introspective Control (LINC) program is developing machine learning methods that show promise in making that scenario closer to reality. LINC aims to fundamentally improve the safety of mechanical systems—specifically in ground vehicles, ships, drone swarms, and robotics—using various methods that require minimal computing power. The result is an AI-powered controller the size of a cell phone.

[ DARPA ]

This survey reviews advances in 3D object detection approaches for autonomous driving. A brief introduction to 2D object detection is first discussed and drawbacks of the existing methodologies are identified for highly dynamic environments. Subsequently, this paper reviews the state-of-the-art 3D object detection techniques that utilizes monocular and stereo vision for reliable detection in urban settings. Based on depth inference basis, learning schemes, and internal representation, this work presents a method taxonomy of three classes: model-based and geometrically constrained approaches, end-to-end learning methodologies, and hybrid methods. There is highlighted segment for current trend of multi-view detectors as end-to-end methods due to their boosted robustness. Detectors from the last two kinds were specially selected to exploit the autonomous driving context in terms of geometry, scene content and instances distribution. To prove the effectiveness of each method, 3D object detection datasets for autonomous vehicles are described with their unique features, e. g., varying weather conditions, multi-modality, multi camera perspective and their respective metrics associated to different difficulty categories. In addition, we included multi-modal visual datasets, i. e., V2X that may tackle the problems of single-view occlusion. Finally, the current research trends in object detection are summarized, followed by a discussion on possible scope for future research in this domain.

Pages