IEEE Spectrum Automation

IEEE Spectrum
Subscribe to IEEE Spectrum Automation feed IEEE Spectrum Automation


About a year ago, Zipline introduced Platform 2, an approach to precision urban drone delivery that combines a large hovering drone with a smaller package-delivery “Droid.” Lowered on a tether from the belly of its parent Zip drone, the Droid contains thrusters and sensors (plus a 2.5- to 3.5-kilogram payload) to reliably navigate itself to a delivery area of just one meter in diameter. The Zip, meanwhile, safely remains hundreds of meters up. After depositing its payload, the Droid rises back up to the drone on its tether, and off they go.

At first glance, the sensor and thruster-packed Droid seems complicated enough to be bordering on impractical, especially when you consider the relative simplicity of other drone delivery solutions, which commonly just drop the package itself on a tether from a hovering drone. I’ve been writing about robots long enough that I’m suspicious of robotic solutions that appear to be overengineered, since that’s always a huge temptation with robotics. Like, is this really the best way of solving a problem, or is it just the coolest way?

We know the folks at Zipline pretty well, though, and they’ve certainly made creative engineering work for them, as we saw when we visited one of their “nests” in rural Rwanda. So as Zipline nears the official launch of Platform 2, we spoke with Zipline cofounder and CTO Keenan Wyrobek, Platform 2 lead Zoltan Laszlo, and industrial designer Gregoire Vandenbussche to understand exactly why they think this is the best way of solving precision urban drone delivery.

First, a quick refresher. Here’s what the delivery sequence with the vertical takeoff and landing (VTOL) Zip and the Droid looks like:

The system has a service radius of about 16 kilometers (10 miles), and it can make deliveries to outdoor spaces of “any meaningful size.” Visual sensors on the Droid find the delivery site and check for obstacles on the way down, while the thrusters compensate for wind and movement of the parent drone. Since the big VTOL Zip remains well out of the way, deliveries are fast, safe, and quiet. But it takes two robots to pull off the delivery rather than just one.

On the other end is the infrastructure required to load and charge these drones. Zipline’s Platform 1 drones require a dedicated base with relatively large launch and recovery systems. With Platform 2, the drone drops the Droid into a large chute attached to the side of a building so that the Droid can be reloaded, after which it pulls the Droid out again and flies off to make the delivery:

“We think it’s the best delivery experience. Not the best drone delivery experience, the best delivery experience,” Zipline’s Wyrobek tells us. That may be true, but the experience also has to be practical and sustainable for Zipline to be successful, so we asked the Zipline team to explain the company’s approach to precision urban delivery.

Zipline on:

IEEE Spectrum: What problems is Platform 2 solving, and why is it necessary to solve those problems in this specific way?

Keenan Wyrobek: There are literally billions of last-mile deliveries happening every year in [the United States] alone, and our customers have been asking for years for something that can deliver to their homes. With our long-range platform, Platform 1, we can float a package down into your yard on a parachute, but that takes some space. And so one half of the big design challenge was how to get our deliveries precise enough, while the other half was to develop a system that will bolt on to existing facilities, which Platform 1 doesn’t do.

Zoltan Laszlo: Platform 1 can deliver within an area of about two parking spaces. As we started to actually look at the data in urban areas using publicly available lidar surveys, we found that two parking spaces serves a bit more than half the market. We want to be a universal delivery service.

But with a delivery area of 1 meter in diameter, which is what we’re actually hitting in our delivery demonstrations for Platform 2, that gets us into the high 90s for the percentage of people that we can deliver to.

Wyrobek: When we say “urban,” what we’re talking about is three-story sprawl, which is common in many large cities around the world. And we wanted to make sure that our deliveries could be precise enough for places like that.

There are some existing solutions for precision aerial delivery that have been operating at scale with some success, typically by winching packages to the ground from a VTOL drone. Why develop your own technique rather than just going with something that has already been shown to work?

Laszlo: Winching down is the natural extension of being able to hover in place, and when we first started, we were like, “Okay, we’re just going to winch down. This will be great, super easy.”

So we went to our test site in Half Moon Bay [on the Northern California coast] and built a quick prototype of a winch system. But as soon as we lowered a box down on the winch, the wind started blowing it all over the place. And this was from the height of our lift, which is less than 10 meters up. We weren’t even able to stay inside two parking spaces, which told us that something was broken with our approach.

The aircraft can sense the wind, so we thought we’d be able to find the right angle for the delivery and things like that. But the wind where the aircraft is may be different from the wind nearer the ground. We realized that unless we’re delivering to an open field, a package that does not have active wind compensation is going to be very hard to control. We’re targeting high-90th percentile in terms of availability due to weather—even if it’s a pretty blustery day, we still want to be able to deliver.

Wyrobek: This was a wild insight when we really understood that unless it’s a perfect day, using a winch actually takes almost as much space as we use for Platform 1 floating a package down on a parachute.

Engineering test footage of Zipline’s Platform 2 docking system at their test site in Half Moon Bay in California.

How did you arrive at this particular delivery solution for Platform 2?

Laszlo: I don’t remember whose idea it was, but we were playing with a bunch of different options. Putting thrusters on the tether wasn’t even the craziest idea. We had our Platform 1 aircraft, which was reliable, so we started with looking at ways to just make that aircraft deliver more precisely. There was only so much more we could do with passive parachutes, but what does an active, steerable parachute look like? There are remote-controlled paragliding toys out there that we tested, with mixed results—the challenge is to minimize the smarts in your parachute, because there’s a chance you won’t get it back. So then we started some crazy brainstorming about how to reliably retrieve the parachute.

Wyrobek: One idea was that the parachute would come with a self-return envelope that you could stick in the mail. Another idea was that the parachute would be steered by a little drone, and when the package got dropped off, the drone would reel the parachute in and then fly back up into the Zip.

Laszlo: But when we realized that the package has to be able to steer itself, that meant the Zip doesn’t need to be active. The Zip doesn’t need to drive the package, it doesn’t even need to see the package, it just needs to be a point up in the sky that’s holding the package. That let us move from having the Zip 50 feet up, to having it 300 feet up, which is important because it’s a big, heavy drone that we don’t want in our customer’s space. And the final step was adding enough smarts to the thing coming down into your space to figure out where exactly to deliver to, and of course to handle the wind.

Once you knew what you needed to do, how did you get to the actual design of the droid?

Gregoire Vandenbussche: Zipline showed me pretty early on that they were ready to try crazy ideas, and from my experience, that’s extremely rare. When the idea of having this controllable tether with a package attached to it came up, one of my first thoughts was that from a user standpoint, nothing like this exists. And the difficulty of designing something that doesn’t exist is that people will try to identify it according to what they know. So we had to find a way to drive that thinking towards something positive.

Early Droid concept sketches by designer Gregoire Vandenbussche featured legs that would fold up after delivery.Zipline

First we thought about putting words onto it, like “hello” or something, but the reality is that we’re an international company and we need to be able to work everywhere. But there’s one thing that’s common to everyone, and that’s emotions—people are able to recognize certain things as being approachable and adorable, so going in that direction felt like the right thing to do. However, being able to design a robot that gives you that kind of emotion but also flies was quite a challenge. We took inspiration from other things that move in 3D, like sea mammals—things that people will recognize even without thinking about it.

Vandenbussche’s sketches show how the design of the Droid was partially inspired by dolphins.Zipline

Now that you say it, I can definitely see the sea mammal inspiration in the drone.

Vandenbussche: There are two aspects of sea mammals that work really well for our purpose. One of them is simplicity of shape; sea mammals don’t have all that many details. Also, they tend to be optimized for performance. Ultimately, we need that, because we need to be able to fly. And we need to be able to convey to people that the drone is under control. So having something you can tell is moving forward or turning or moving away was very helpful.

Wyrobek: One other insight that we had is that Platform 2 needs to be small to fit into tight delivery spaces, and it needs to feel small when it comes into your personal space, but it also has to be big enough inside to be a useful delivery platform. We tried to leverage the chubby but cute look that baby seals have going on.

The design journey was pretty fun. Gregoire would spend two or three days coming up with a hundred different concept sketches. We’d do a bunch of brainstorming, and then Gregoire would come up with a whole bunch of new directions, and we’d keep exploring. To be clear, no one would describe our functional prototypes from back then as “cute.” But through all this iteration eventually we ended up in an awesome place.

And how do you find that place? When do you know that your robot is just cute enough?

One iteration of the Droid, Vandenbussche determined, looked too technical and intimidating.Zipline

Vandenbussche: It’s finding the balance around what’s realistic and functional. I like to think of industrial design as taking all of the constraints and kind of playing Tetris with them until you get a result that ideally satisfies everybody. I remember at one point looking at where we were, and feeling like we were focusing too much on performance and missing that emotional level. So, we went back a little bit to say, where can we bring this back from looking like a highly technical machine to something that can give you a feeling of approachability?

Laszlo: We spent a fair bit of time on the controls and behaviors of the droid to make sure that it moves in a very approachable and predictable way, so that you know where it’s going ahead of time and it doesn’t behave in unexpected ways. That’s pretty important for how people perceive it.

We did a lot of work on how the droid would descend and approach the delivery site. One concept had the droid start to lower down well before the Zip was hovering directly overhead. We had simulations and renderings, and it looked great. We could do the whole delivery in barely over 20 seconds. But even if the package is far away from you, it still looks scary because [the Zip is] moving faster than you would expect, and you can’t tell exactly where it’s going to deliver. So we deleted all that code, and now it just comes straight down, and people don’t back away from the Droid anymore. They’re just like, “Oh, okay, cool.”

How did you design the thrusters to enable these pinpoint deliveries?

Early tests of the Droid centered around a two-fan version.Zipline

Laszlo: With the thrusters, we knew we wanted to maximize the size of at least one of the fans, because we were almost always going to have to deal with wind. We’re trying to be as quiet as we can, so the key there is to maximize the area of the propeller. Our leading early design was just a box with two fans on it:

Two fans with unobstructed flow meant that it moved great, but the challenge of fitting it inside another aircraft was going to be painful. And it looked big, even though it wasn’t actually that big.

Vandenbussche: It was also pretty intimidating when you had those two fans facing you and the Droid coming toward you.

A single steerable fan [left] that acted like a rudder was simpler in some ways, but as the fan got larger, the gyroscopic effects became hard to manage. Instead of one steerable fan, how about two steerable fans? [right] Omnidirectional motion was possible with this setup, but packaging it inside of a Zip didn’t work.Zipline

Laszlo: We then started looking at configurations with a main fan and a second smaller fan, with the bigger fan at the back pushing forward and the smaller fan at the front providing thrust for turning. The third fan we added relatively late because we didn’t want to add it at all. But we found that [with two fans] the droid would have to spin relatively quickly to align to shifting winds, whereas with a third fan we can just push sideways in the direction that we need.

What kind of intelligence does the Droid have?

The current design of Zipline’s Platform 2 Droid is built around a large thruster in the rear and two smaller thrusters at the front and back.Zipline

Wyrobek: The Droid has its own little autopilot, and there’s a very simple communications system between the two vehicles. You may think that it’s a really complex coordinated control problem, but it’s not: The Zip just kind of hangs out, and the Droid takes care of the delivery. The sensing challenge is for the Droid to find trees and powerlines and things like that, and then find a good delivery site.

Was there ever a point at which you were concerned that the size and weight and complexity would not be worth it?

Wyrobek: Our mindset was to fail fast, to try things and do what we needed to do to convince ourselves that it wasn’t a good path. What’s fun about this kind of iterative process is oftentimes, you try things and you realize that actually, this is better than we thought.

Laszlo: We first thought about the Droid as a little bit of a tax, in that it’s costing us extra weight. But if your main drone can stay high enough up that it avoids trees and buildings, then it can just float around up there. If it gets pushed around by the wind, it doesn’t matter because the Droid can compensate.

Wyrobek: Keeping the Zip at altitude is a big win in many ways. It doesn’t have to spend energy station-keeping, descending, and then ascending again. We just do that with the much smaller Droid, which also makes the hovering phase much shorter. It’s also much more efficient to control the small droid than the large Zip. And having all of the sensors on the Droid very close to the area that you’re delivering to makes that problem easier as well. It may look like a more complex system from the outside, but from the inside, it’s basically making all the hardest problems much easier.

Over the past year, Zipline has set up a bunch of partnerships to make residential deliveries to consumers using Droid starting in 2024, including prescriptions from Cleveland Clinic in Ohio, medical products from WellSpan Health in Pennsylvania, tasty food from Mendocino Farms in California, and a little bit of everything from Walmart starting in Dallas. Zipline’s plan is to kick things off with Platform 2 later this year.



Legendary MIT roboticist Daniela Rus has published a new book called The Heart and the Chip: Our Bright Future with Robots. “There is a robotics revolution underway,” Rus says in the book’s introduction, “one that is already causing massive changes in our society and in our lives.” She’s quite right, of course, and although some of us have been feeling that this is true for decades, it’s arguably more true right now than it ever has been. But robots are difficult and complicated, and the way that their progress is intertwined with the humans that make them and work with them means that these changes won’t come quickly or easily. Rus’ experience gives her a deep and nuanced perspective on robotics’ past and future, and we’re able to share a little bit of that with you here.

Daniela Rus: Should roboticists consider subscribing to their own Hippocratic oath?

The following excerpt is from Chapter 14, entitled “What Could Go Wrong?” Which, let’s be honest, is the right question to ask (and then attempt to conclusively answer) whenever you’re thinking about sending a robot out into the real world.

At several points in this book I’ve mentioned the fictional character Tony Stark, who uses technology to transform himself into the superhero Iron Man. To me this character is a tremendous inspiration, yet I often remind myself that in the story, he begins his career as an MIT-­trained weapons manufacturer and munitions developer. In the 2008 film Iron Man, he changes his ways because he learns that his company’s specialized weapons are being used by terrorists.

Remember, robots are tools. Inherently, they are neither good nor bad; it’s how we choose to use them that matters. In 2022, aerial drones were used as weapons on both sides of devastating wars. Anyone can purchase a drone, but there are regulations for using drones that vary between and within different countries. In the United States, the Federal Aviation Administration requires that all drones be registered, with a few exceptions, including toy models weighing less than 250 grams. The rules also depend on whether the drone is flown for fun or for business. Regardless of regulations, anyone could use a flying robot to inflict harm, just like anyone can swing a hammer to hurt someone instead of driving a nail into a board. Yet drones are also being used to deliver critical medical supplies in hard-­to-­reach areas, track the health of forests, and help scientists like Roger Payne monitor and advocate for at-­risk species. My group collaborated with the modern dance company Pilobolus to stage the first theatrical performance featuring a mix of humans and drones back in 2012, with a robot called Seraph. So, drones can be dancers, too. In Kim Stanley Robinson’s prescient science fiction novel The Ministry for the Future, a swarm of unmanned aerial vehicles is deployed to crash an airliner. I can imagine a flock of these mechanical birds being used in many good ways, too. At the start of its war against Ukraine, Russia limited its citizens’ access to unbiased news and information in hopes of controlling and shaping the narrative around the conflict. The true story of the invasion was stifled, and I wondered whether we could have dispatched a swarm of flying video screens capable of arranging themselves into one giant aerial monitor in the middle of popular city squares across Russia, showing real footage of the war, not merely clips approved by the government. Or, even simpler: swarms of flying digital projectors could have broadcasted the footage on the sides of buildings and walls for all to see. If we had deployed enough, there would have been too many of them to shut down.

There may be variations of Tony Stark passing through my university or the labs of my colleagues around the world, and we need to do whatever we can to ensure these talented young individuals endeavor to have a positive impact on humanity.

The Tony Stark character is shaped by his experiences and steered toward having a positive impact on the world, but we cannot wait for all of our technologists to endure harrowing, life-­changing experiences. Nor can we expect everyone to use these intelligent machines for good once they are developed and moved out into circulation. Yet that doesn’t mean we should stop working on these technologies—­the potential benefits are too great. What we can do is think harder about the consequences and put in place the guardrails to ensure positive benefits. My contemporaries and I can’t necessarily control how these tools are used in the world, but we can do more to influence the people making them.

There may be variations of Tony Stark passing through my university or the labs of my colleagues around the world, and we need to do whatever we can to ensure these talented young individuals endeavor to have a positive impact on humanity. We absolutely must have diversity in our university labs and research centers, but we may be able to do more to shape the young people who study with us. For example, we could require study of the Manhattan Project and the moral and ethical quandaries associated with the phenomenal effort to build and use the atomic bomb. At this point, ethics courses are not a widespread requirement for an advanced degree in robotics or AI, but perhaps they should be. Or why not require graduates to swear to a robotics-­ and AI-­attuned variation on the Hippocratic oath?

The oath comes from an early Greek medical text, which may or may not have been written by the philosopher Hippocrates, and it has evolved over the centuries. Fundamentally, it represents a standard of medical ethics to which doctors are expected to adhere. The most famous of these is the promise to do no harm, or to avoid intentional wrongdoing. I also applaud the oath’s focus on committing to the community of doctors and the necessity of maintaining the sacred bond between teacher and pupils. The more we remain linked as a robotics community, the more we foster and maintain our relationships as our students move out into the world, the more we can do to steer the technology toward a positive future. Today the Hippocratic oath is not a universal requirement for certification as a doctor, and I do not see it functioning that way for roboticists, either. Nor am I the first roboticist or AI leader to suggest this possibility. But we should seriously consider making it standard practice.

In the aftermath of the development of the atomic bomb, when the potential of scientists to do harm was made suddenly and terribly evident, there was some discussion of a Hippocratic oath for scientific researchers. The idea has resurfaced from time to time and rarely gains traction. But science is fundamentally about the pursuit of knowledge; in that sense it is pure. In robotics and AI, we are building things that will have an impact on the world and its people and other forms of life. In this sense, our field is somewhat closer to medicine, as doctors are using their training to directly impact the lives of individuals. Asking technologists to formally recite a version of the Hippocratic oath could be a way to continue nudging our field in the right direction, and perhaps serve as a check on individuals who are later asked to develop robots or AI expressly for nefarious purposes.

Of course, the very idea of what is good or bad, in terms of how a robot is used, depends on where you sit. I am steadfastly opposed to giving armed or weaponized robots autonomy. We cannot and should not trust machine intelligences to make decisions about whether to inflict harm on a person or group of people on their own. Personally, I would prefer that robots never be used to do harm to anyone, but this is now unrealistic. Robots are being used as tools of war, and it is our responsibility to do whatever we can to shape their ethical use. So, I do not separate or divorce myself from reality and operate solely in some utopian universe of happy, helpful robots. In fact, I teach courses on artificial intelligence to national security officials and advise them on the strengths, weaknesses, and capabilities of the technology. I see this as a patriotic duty, and I’m honored to be helping our leaders understand the limitations, strengths, and possibilities of robots and other AI-­enhanced physical systems—­what they can and cannot do, what they should and should not do, and what I believe they must do.

Ultimately, no matter how much we teach and preach about the limitations of technology, the ethics of AI, or the potential dangers of developing such powerful tools, people will make their own choices, whether they are recently graduated students or senior national security leaders. What I hope and teach is that we should choose to do good. Despite the efforts of life extension companies, we all have a limited time on this planet, what the scientist Carl Sagan called our “pale blue dot,” and we should do whatever we can to make the most of that time and have a positive impact on our beautiful environment, and the many people and other species with which we share it. My decades-­long quest to build more intelligent and capable robots has only strengthened my appreciation for—­no, wonder at—­the marvelous creatures that crawl, walk, swim, run, slither, and soar across and around our planet, and the fantastic plants, too. We should not busy ourselves with the work of developing robots that can eliminate these cosmically rare creations. We should focus instead on building technologies to preserve them, and even help them thrive. That applies to all living entities, including the one species that is especially concerned about the rise of intelligent machines.


Excerpted from “The Heart and the Chip: Our Bright Future with Robots”. Copyright 2024 by Daniela Rus, Gregory Mone. Used with permission of the publisher, W.W. Norton & Company. All rights reserved.



Your weekly selection of awesome robot videos

Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPAN

Enjoy today’s videos!

How many quadrupeds can you control with one single locomotion policy? Apparently, the answer is “all of the quadrupeds.”

Look for this at ICRA 2024 in a couple of months!

[ EPFL ]

Thanks, Milad!

Very impressive performance from Figure 01, I think, although as is frequently the case, it’s hard to tell exactly how impressive without more information about exactly what’s going on here.

[ Figure ]

That awesome ANYmal Parkour research is now published, which means that there’s a new video, well worth watching all the way to the end.

[ Science ] via [ ETHZ RSL ]

Robotic vision can be pretty tricky when you’re cooking, because things can significantly change how they look over time, like with melting butter or an egg being fried. Some new research is tackling this, using a (now ancient?) PR2.

[ JSK Lab ]

Thanks, Kento!

Filmed in January of 2020, this video shows Atlas clearing debris and going through a doorway. Uses a combination of simple footstep planning, teleoperation, and autonomous behaviors through a single virtual reality operator interface. Robot built by Boston Dynamics for the DARPA Robotics Challenge in 2013.

[ IHMC ]

Sustainable fashion enabled by smart textiles shaped by a robot and a heat gun. Multiple styles, multiple sizes, all in one garment!

[ MIT ]

Video of Boston Dynamics’ Stretch from MODEX, with a little sneak peak at the end of what the robot’s next warehouse task might be.

[ Boston Dynamics ]

Pickle Robots autonomously unload trucks and import containers. The system is in production use at customer warehoues handling floor-loaded freight at human scale or better.

[ Pickle Robot ]

The ROBDEKON robotics competence center is dedicated to the development of robotic systems for hazardous environments that pose a potential risk to humans. As part of the consortium, the FZI Research Center for Information Technology developed robotic systems, technologies, and Artificial Intelligence (AI) methods that can be used to handle hazardous materials–for example, to sort potentially dangerous used batteries for recycling.

[ FZI ]

This research project with Ontario Power Generation involves adapting Boston Dynamics Spot’s localization system to longterm changes in the environment. During this testing, we mounted a GoPro camera on the back of Spot and took a video of each walk for a year from Spot’s point of view. We put the footage together as a moving time-lapse video where the day changes as Spot completes the Autowalk around the campus.

[ MARS Lab ]



When IEEE Spectrum first wrote about Covariant in 2020, it was a new-ish robotics startup looking to apply robotics to warehouse picking at scale through the magic of a single end-to-end neural network. At the time, Covariant was focused on this picking use case, because it represents an application that could provide immediate value—warehouse companies pay Covariant for its robots to pick items in their warehouses. But for Covariant, the exciting part was that picking items in warehouses has, over the last four years, yielded a massive amount of real-world manipulation data—and you can probably guess where this is going.

Today, Covariant is announcing RFM-1, which the company describes as a robotics foundation model that gives robots the “human-like ability to reason.” That’s from the press release, and while I wouldn’t necessarily read too much into “human-like” or “reason,” what Covariant has going on here is pretty cool.

“Foundation model” means that RFM-1 can be trained on more data to do more things—at the moment, it’s all about warehouse manipulation because that’s what it’s been trained on, but its capabilities can be expanded by feeding it more data. “Our existing system is already good enough to do very fast, very variable pick and place,” says Covariant co-founder Pieter Abbeel. “But we’re now taking it quite a bit further. Any task, any embodiment—that’s the long-term vision. Robotics foundation models powering billions of robots across the world.” From the sound of things, Covariant’s business of deploying a large fleet of warehouse automation robots was the fastest way for them to collect the tens of millions of trajectories (how a robot moves during a task) that they needed to train the 8 billion parameter RFM-1 model.

Covariant

“The only way you can do what we’re doing is by having robots deployed in the world collecting a ton of data,” says Abbeel. “Which is what allows us to train a robotics foundation model that’s uniquely capable.”

There have been other attempts at this sort of thing: The RTX project is one recent example. But while RT-X depends on research labs sharing what data they have to create a dataset that’s large enough to be useful, Covariant is doing it alone, thanks to its fleet of warehouse robots. “RT-X is about a million trajectories of data,” Abbeel says, “but we’re able to surpass it because we’re getting a million trajectories every few weeks.”

“By building a valuable picking robot that’s deployed across 15 countries with dozens of customers, we essentially have a data collection machine.” —Pieter Abbeel, Covariant

You can think of the current execution of RFM-1 as a prediction engine for suction-based object manipulation in warehouse environments. The model incorporates still images, video, joint angles, force reading, suction cup strength—everything involved in the kind of robotic manipulation that Covariant does. All of these things are interconnected within RFM-1, which means that you can put any of those things into one end of RFM-1, and out of the other end of the model will come a prediction. That prediction can be in the form of an image, a video, or a series of commands for a robot.

What’s important to understand about all of this is that RFM-1 isn’t restricted to picking only things it’s seen before, or only working on robots it has direct experience with. This is what’s nice about foundation models—they can generalize within the domain of their training data, and it’s how Covariant has been able to scale their business as successfully as they have, by not having to retrain for every new picking robot or every new item. What’s counter-intuitive about these large models is that they’re actually better at dealing with new situations than models that are trained specifically for those situations.

For example, let’s say you want to train a model to drive a car on a highway. The question, Abbeel says, is whether it would be worth your time to train on other kinds of driving anyway. The answer is yes, because highway driving is sometimes not highway driving. There will be accidents or rush hour traffic that will require you to drive differently. If you’ve also trained on driving on city streets, you’re effectively training on highway edge cases, which will come in handy at some point and improve performance overall. With RFM-1, it’s the same idea: Training on lots of different kinds of manipulation—different robots, different objects, and so on—means that any single kind of manipulation will be that much more capable.

In the context of generalization, Covariant talks about RFM-1’s ability to “understand” its environment. This can be a tricky word with AI, but what’s relevant is to ground the meaning of “understand” in what RFM-1 is capable of. For example, you don’t need to understand physics to be able to catch a baseball, you just need to have a lot of experience catching baseballs, and that’s where RFM-1 is at. You could also reason out how to catch a baseball with no experience but an understanding of physics, and RFM-1 is not doing this, which is why I hesitate to use the word “understand” in this context.

But this brings us to another interesting capability of RFM-1: it operates as a very effective, if constrained, simulation tool. As a prediction engine that outputs video, you can ask it to generate what the next couple seconds of an action sequence will look like, and it’ll give you a result that’s both realistic and accurate, being grounded in all of its data. The key here is that RFM-1 can effectively simulate objects that are challenging to simulate traditionally, like floppy things.

Covariant’s Abbeel explains that the “world model” that RFM-1 bases its predictions on is effectively a learned physics engine. “Building physics engines turns out to be a very daunting task to really cover every possible thing that can happen in the world,” Abbeel says. “Once you get complicated scenarios, it becomes very inaccurate, very quickly, because people have to make all kinds of approximations to make the physics engine run on a computer. We’re just doing the large-scale data version of this with a world model, and it’s showing really good results.”

Abbeel gives an example of asking a robot to simulate (or predict) what would happen if a cylinder is placed vertically on a conveyor belt. The prediction accurately shows the cylinder falling over and rolling when the belt starts to move—not because the cylinder is being simulated, but because RFM-1 has seen a lot of things being placed on a lot of conveyor belts.

“Five years from now, it’s not unlikely that what we are building here will be the only type of simulator anyone will ever use.” —Pieter Abbeel, Covariant

This only works if there’s the right kind of data for RFM-1 to train on, so unlike most simulation environments, it can’t currently generalize to completely new objects or situations. But Abbeel believes that with enough data, useful world simulation will be possible. “Five years from now, it’s not unlikely that what we are building here will be the only type of simulator anyone will ever use. It’s a more capable simulator than one built from the ground up with collision checking and finite elements and all that stuff. All those things are so hard to build into your physics engine in any kind of way, not to mention the renderer to make things look like they look in the real world—in some sense, we’re taking a shortcut.”

RFM-1 also incorporates language data to be able to communicate more effectively with humans. Covariant

For Covariant to expand the capabilities of RFM-1 towards that long-term vision of foundation models powering “billions of robots across the world,” the next step is to feed it more data from a wider variety of robots doing a wider variety of tasks. “We’ve built essentially a data ingestion engine,” Abbeel says. “If you’re willing to give us data of a different type, we’ll ingest that too.”

“We have a lot of confidence that this kind of model could power all kinds of robots—maybe with more data for the types of robots and types of situations it could be used in.” —Pieter Abbeel, Covariant

One way or another, that path is going to involve a heck of a lot of data, and it’s going to be data that Covariant is not currently collecting with its own fleet of warehouse manipulation robots. So if you’re, say, a humanoid robotics company, what’s your incentive to share all the data you’ve been collecting with Covariant? “The pitch is that we’ll help them get to the real world,” Covariant co-founder Peter Chen says. “I don’t think there are really that many companies that have AI to make their robots truly autonomous in a production environment. If they want AI that’s robust and powerful and can actually help them enter the real world, we are really their best bet.”

Covariant’s core argument here is that while it’s certainly possible for every robotics company to train up their own models individually, the performance—for anybody trying to do manipulation, at least—would be not nearly as good as using a model that incorporates all of the manipulation data that Covariant already has within RFM-1. “It has always been our long term plan to be a robotics foundation model company,” says Chen. “There was just not sufficient data and compute and algorithms to get to this point—but building a universal AI platform for robots, that’s what Covariant has been about from the very beginning.”



The global ocean is difficult to explore—the common refrain is that we know less about the deep ocean than we do about the surface of the moon. Australian company Advanced Navigation wants to change that with a pint-sized autonomous underwater vehicle (AUV) that it hopes will become the maritime equivalent of a consumer drone. And the new AUV is already getting to work mapping and monitoring Australia’s coral reefs and diving for shipwrecks.

The Sydney-based company has been developing underwater navigation technology for more than a decade. In 2022, Advanced Navigation unveiled its first in-house AUV, called Hydrus. At less than half a meter long, the vehicle is considerably smaller than most alternatives. Even so, it’s fully autonomous and carries a 4k-resolution camera capable of 60 frames per second that can both capture high-definition video and construct detailed 3D photogrammetry models.

Advanced Navigation says Hydrus—with a depth rating of 3,000 meters, a range of 9 kilometers, and a battery that lasts up to three hours—is capable of a wide variety of missions. The company recently sold two units to the Australian Institute of Marine Science (AIMS), the country’s tropical marine science agency, which will use them to survey coral reefs in the North West Shelf region off the country’s west coast. Hydrus has also recently collaborated with the Western Australian Museum to produce a detailed 3D model of a shipwreck off the coast near Perth.

“If people can go and throw one of these off the boat, just like they can throw a drone up in the air, that will obviously benefit the exploration of the sea.” —Ross Anderson, Western Australian Museum

After many years of supplying components to other robotics companies, Peter Baker, subsea product manager at Advanced Navigation, says they company spotted a gap in the market. “We wanted to take the user experience that someone would have with an aerial drone and bring that underwater,” he says. “It’s very expensive to get images and data of the seabed. So by being able to miniaturize this system, and have it drastically simplified from the user’s point of view, it makes data a lot more accessible to people.”

But building a compact and low-cost AUV is not simple. The deep ocean is not a friendly place for electronics, says Baker, due to a combination of high pressure and corrosive seawater. The traditional way of dealing with this is to stick all the critical components in a sealed titanium tube that can maintain ambient pressure and keep moisture out. However, this requires you to add buoyancy to compensate for the extra weight, says Baker, which increases the bulk of the vehicle. That means bigger motors and bigger batteries. “The whole thing spirals up and up until you’ve got something the size of a minibus,” he says.

Advanced Navigation got around the spiral by designing bespoke pressure-tolerant electronics. They built all of their circuit boards from scratch, carefully selecting components that had been tested to destruction in a hydrostatic pressure chamber. These were then encapsulated in a water-proof composite shell, and to further reduce the risk of water ingress the drone operates completely wirelessly. Batteries are recharged using inductive charging and data transfer is either over Wi-Fi when above water or via an optical modem when below the surface.

Hydrus AUVs are charged using induction to keep corrosive seawater from leaking in through charging ports.Advanced Navigation

This has allowed the company to significantly miniaturize the system, says Baker, which has a drastic impact on the overall cost of operations. “You don’t need a crane or a winch or anything like that to recover the vehicle, you can pick it up with a fishing net,” he says. “You can get away with using a much smaller boat, and the rule of thumb in the industry is if you double the size of your boat, you quadruple the cost.”

Just as important, though, is the vehicle’s ease of use. Most underwater robotics systems still operate with a tether, says Baker, but Hydrus carries all the hardware required to support autonomous navigation on board. The company’s “bread and butter” is inertial navigation technology, which uses accelerometers and gyroscopes to track the vehicle from a known starting point. But the drone also features a sonar system that allows it to stay a set distance from the seabed and also judge its speed by measuring the Doppler shift on echoes as they bounce back.

This means that users can simply program in a set of way points on a map, toss the vehicle overboard and leave it to its own devices, says Baker. The Hydrus does have a low-bandwidth acoustic communication channel that allows the operator to send basic commands like “stop” or “come home,” he says, but Hydrus is designed to be a set-and-forget AUV. “That really lowers the thresholds of what a user needs to be able to operate it,” he says. “If you can fly a DJI drone you could fly a Hydrus.”

The company estimates for a typical seabed investigation in water shallow enough for human divers, the Hydrus could be 75 percent cheaper than alternatives. And the savings would go up significantly at greater depths. What’s more, says Baker, the drone’s precise navigation means it can produce much more consistent and repeatable data.

To demonstrate its capabilities, Hydrus’ designers went hunting for shipwrecks in the Rottnest ships graveyard just off the coast near Perth, in Western Australia. The site was a designated spot for scuttling aging ships, says Ross Anderson, curator at Western Australian Museum, but has yet to be fully explored due to the depth of many of the wrecks.

The Advanced Navigation team used the Hydrus to create a detailed 3D model of a sunken “coal hulk”—one of a category of old iron sailing ships that were later converted to floating coal warehouses for steamships. The Western Australian Museum has been unable to identify the vessel so far, but Anderson says these kind of models can be hugely beneficial for carrying out maritime archaeology research, as well as educating people about what’s below the waves.


Advanced Navigation used its new Hydrus drone to create a detailed 3D image of an unidentified “coal hulk” ship in the Rottnest ships graveyard off the western coast of Australia.

Advanced Navigation

Any technology that can simplify the process is greatly welcomed, Anderson adds. “If people can go and throw one of these off the boat, just like they can throw a drone up in the air, that will obviously benefit the exploration of the sea,” he says.

Ease of use was also a big driver behind AIMS’s purchase of two Hydrus drones, says technology development program lead Melanie Olsen, who is also an IEEE senior member. Most of the technology available for marine science is still research-grade and a long way from a polished, professional product.

“When you’re an operational agency like AIMS, you typically don’t have the luxury of spending a lot of time on the back of the boat getting equipment ready,” says Olsen. “You need something that users can turn on and go and it’s just working, as time is of the essence.”

Another benefit of the Hydrus for AIMS is that the drone can operate at greater depths than divers and in conditions that would be dangerous for humans. “Its enabling our researchers to see further down in the water and also operate in more dangerous situations such as at night, or in the presence of threats such as crocodiles or sharks, places where we just wouldn’t be able to collect that data,” says Olsen.

The agency will initially use the drones to survey reefs on Australia’s North West Shelf, including Scott Reef and Ashmore Reef. The goal is to collect regular data data on coral health to monitor the state of the reefs, investigate how they’re being effected by climate change, and hopefully get early warning of emerging problems. But Olsen says they expect that the Hydrus will become standard part of their ocean monitoring toolkit going forward.

This story was updated on 11 March 2024 to correct the year when Advanced Navigation unveiled Hydrus.



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPANRoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS

Enjoy today’s videos!

We present Human to Humanoid (H2O), a reinforcement learning (RL) based framework that enables real-time, whole-body teleoperation of a full-sized humanoid robot with only an RGB camera. We successfully achieve teleoperation of dynamic, whole-body motions in real-world scenarios, including walking, back jumping, kicking, turning, waving, pushing, boxing, etc. To the best of our knowledge, this is the first demonstration to achieve learning-based, real-time, whole-body humanoid teleoperation.

[ CMU ]

Legged robots have the potential to traverse complex terrain and access confined spaces beyond the reach of traditional platforms thanks to their ability to carefully select footholds and flexibly adapt their body posture while walking. However, robust deployment in real-world applications is still an open challenge. In this paper, we present a method for legged locomotion control using reinforcement learning and 3D volumetric representations to enable robust and versatile locomotion in confined and unstructured environments.

[ Takahiro Miki ]

Sure, 3.3 meters per second is fast for a humanoid, but I’m more impressed by the spinning around while walking downstairs.

[ Unitree ]

Improving the safety of collaborative manipulators necessitates the reduction of inertia in the moving part. We introduce a novel approach in the form of a passive, 3D wire aligner, serving as a lightweight and low-friction power transmission mechanism, thus achieving the desired low inertia in the manipulator’s operation.

[ SAQIEL ]

Thanks, Temma!

Robot Era just launched Humanoid-Gym, an open-source reinforcement learning framework for bipedal humanoids. As you can see from the video, RL algorithms have given the robot, called Xiao Xing, or XBot, the ability to climb up and down haphazardly stacked boxes with relative stability and ease.

[ Robot Era ]

“Impact-Aware Bimanual Catching of Large-Momentum Objects.” Need I say more?

[ SLMC ]

More than 80% of stroke survivors experience walking difficulty, significantly impacting their daily lives, independence, and overall quality of life. Now, new research from the University of Massachusetts Amherst pushes forward the bounds of stroke recovery with a unique robotic hip exoskeleton, designed as a training tool to improve walking function. This invites the possibility of new therapies that are more accessible and easier to translate from practice to daily life, compared to current rehabilitation methods.

[ UMass Amherst ]

Thanks, Julia!

The manipulation here is pretty impressive, but it’s hard to know how impressive without also knowing how much the video was sped up.

[ Somatic ]

DJI drones work to make the world a better place and one of the ways that we do this is through conservation work. We partnered with Halo Robotics and the OFI Orangutan Foundation International to showcase just how these drones can make an impact.

[ DJI ]

The aim of the test is to demonstrate the removal and replacement of satellite modules into a 27U CubeSat format using augmented reality control of a robot. In this use case, the “client” satellite is being upgraded and refueled using modular componentry. The robot will then remove the failed computer module and place it in a fixture. It will then do the same with the propellant tank. The robot will then place these correctly back into the satellite.

[ Extend Robotics ]

This video features some of the highlights and favorite moments from the CYBATHLON Challenges 2024 that took place on 2 February, showing so many diverse types of assistive technology taking on discipline tasks and displaying pilots’ tenacity and determination. The Challenges saw new teams, new tasks, and new formats for many of the CYBATHLON disciplines.

[ Cybathlon ]

It’s been a long road to electrically powered robots.

[ ABB ]

Small drones for catastrophic wildfires (ones covering more than [40,470 hectares]) are like bringing a flashlight to light up a football field. This short video describes the major uses for drones of all sizes and why and when they are used, or why not.

[ CRASAR ]

It probably will not surprise you that there are a lot of robots involved in building Rivian trucks and vans.

[ Kawasaki Robotics ]

DARPA’s Learning Introspective Control (LINC) program is developing machine learning methods that show promise in making that scenario closer to reality. LINC aims to fundamentally improve the safety of mechanical systems—specifically in ground vehicles, ships, drone swarms, and robotics—using various methods that require minimal computing power. The result is an AI-powered controller the size of a cell phone.

[ DARPA ]



You’ve seen this before: a truck unloading robot that’s made up of a mobile base with an arm on it that drives up into the back of a trailer and then uses suction to grab stacked boxes and put them onto a conveyor belt. We’ve written about a couple of the companies doing this, and there are even more out there. It’s easy to understand why—trailer unloading involves a fairly structured and controlled environment with a very repetitive task, it’s a hard job that sucks for humans, and there’s an enormous amount of demand.

While it’s likely true that there’s enough room for a whole bunch of different robotics companies in the trailer unloading space, a given customer is probably only going to pick one, and they’re going to pick the one that offers the right combination of safety, capability, and cost. Anyware Robotics thinks they have that mix, aided by a box handling solution that is both very clever and so obvious that I’m wondering why I didn’t think of it myself.

The overall design of Pixmo itself is fairly standard as far as trailer unloading robots go, but some of the details are interesting. We’re told that Pixmo is the only trailer unloading system that integrates a heavy-payload collaborative arm, actually a fairly new commercial arm from Fanuc. This means that Anyware Robotics doesn’t have to faff about with their own hardware, and also that their robot is arguably safer, being ISO certified safe to work directly with people. The base is custom, but Anyware is contracting it out to a big robotics OEM.

“We’ve put a lot of effort into making sure that most of the components of our robot are off-the-shelf,” co-founder and CEO Thomas Tang tells us. “There are already so many mature and cost-efficient suppliers that we want to offload the supply chain, the certification, the reliability testing onto someone else’s shoulders.” And while there are a selection of automated mobile robots (AMRs) out there that seem like they could get the job done, the problem is that they’re all designed for flat surfaces, and getting into and out of the back of the trailer often involves a short, steep ramp, hence the need for their own design. Even with the custom base, Tang says that Pixmo is very cost efficient, and the company predicts that it will be approximately one third the cost of other solutions with a payback of about 24 months.

But here’s the really clever bit:

Anyware Robotics Pixmo Trailer Unloading

That conveyor system in front of the boxes is an add-on that’s used in support of Pixmo. There are two benefits here: first, having the conveyor add-on aligned with the base of a box minimizes the amount of lifting that Pixmo has to do. This allows Pixmo to handle boxes of up to 65 lbs with a lift-and-slide technique, putting it at the top end of trailer unloading robot payload. And the second benefit is that the add-on system decreases the distance that Pixmo has to move the box to just about as small as it can possibly be, eliminating the need for the arm to rotate around to place a box on a conveyor next to or behind itself. Lowering this cycle time means that Pixmo can achieve a throughput of up to 1,000 boxes per hour—about one box every four seconds, which the Internet suggests is quite fast, even for a professional human. Anyware Robotics is introducing this add-on system at MODEX next week, and they have a patent pending on the idea.

This seems like such a simple, useful idea that I asked Tang why they were the first ones to come up with it. “In robotics startups, there tends to be a legacy mindset issue,” Tang told me. “When people have been working on robot arms for so many years, we just think about how to use robot arms to solve everything. That’s maybe that’s the reason why other companies didn’t come up with this solution.” Tang says that Anyware started with much more complicated add-on designs before finding this solution. “Usually it’s the most simple solution that has the most trial and error behind it.”

Anyware Robotics is focused on trailer unloading for now, but Pixmo could easily be adapted for palletizing and depalletizing or somewhat less easily for other warehouse tasks like order picking or machine tending. But why stop there? A mobile manipulator can (theoretically) do it all (almost), and that’s exactly what Tang wants:

In our long-term vision, we believe that the future will have two different types of general purpose robots. In one direction is the humanoid form, which is a really flexible solution for jobs where you want to replace a human. But there are so many jobs that are just not reasonable for a human body to do. So we believe there should be another form of general purpose robot, which is designed for industrial tasks. Our design philosophy is in that direction—it’s also general purpose, but for industrial applications.

At just over one year old, Anyware has already managed to complete a pilot program (and convert it to a purchase order). They’re currently in the middle of several other pilot programs with leading third-party logistics providers, and they expect to spend the next several months focusing on productization with the goal of releasing the first commercial version of Pixmo by July of this year.



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLORADO, USAEurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPANRoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS

Enjoy today’s videos!

Figure has raised a US $675 million Series B, valuing the company at $2.6 billion.

[ Figure ]

Meanwhile, here’s how things are going at Agility Robotics, whose last raise was a $150 million Series B in April of 2022.

[ Agility Robotics ]

Also meanwhile, here’s how things are going at Sanctuary AI, whose last raise was a $58.5 million Series A in March of 2022.

[ Sanctuary AI ]

The time has come for humanoid robots to enter industrial production lines and learn how to assist humans by undertaking repetitive, tedious, and potentially dangerous tasks for them. Recently, UBTECH’s humanoid robot Walker S was introduced into the assembly line of NIO’s advanced vehicle-manufacturing center, as an “intern” assisting in the car production. Walker S is the first bipedal humanoid robot to complete a specific workstation’s tasks on a mobile EV production line.

[ UBTECH ]

Henry Evans keeps working hard to make robots better, this time with the assistance of researchers from Carnegie Mellon University.

Henry said he preferred using head-worn assistive teleoperation (HAT) with a robot for certain tasks rather than depending on a caregiver. “Definitely scratching itches,” he said. “I would be happy to have it stand next to me all day, ready to do that or hold a towel to my mouth. Also, feeding me soft foods, operating the blinds, and doing odd jobs around the room.”
One innovation in particular, software called Driver Assistance that helps align the robot’s gripper with an object the user wants to pick up, was “awesome,” Henry said. Driver Assistance leaves the user in control while it makes the fine adjustments and corrections that can make controlling a robot both tedious and demanding. “That’s better than anything I have tried for grasping,” Henry said, adding that he would like to see Driver Assistance used for every interface that controls Stretch robots.

[ HAT2 ] via [ CMU ]

Watch this video for the three glorious seconds at the end.

[ Tech United ]

Get ready to rip, shear, mow, and tear, as DOOM is back! This April, we’re making the legendary game playable on our robotic mowers as a tribute to 30 years of mowing down demons.

Oh, it’s HOOSKvarna, not HUSKvarna.

[ Husqvarna ] via [ Engadget ]

Latest developments demonstrated on the Ameca Desktop platform. Having fun with vision- and voice-cloning capabilities.

[ Engineered Arts ]

Could an artificial-intelligence system learn language from a child? New York University researchers supported by the National Science Foundation, using first-person video from a head-mounted camera, trained AI models to learn language through the eyes and ears of a child.

[ NYU ]

The world’s leaders in manufacturing, natural resources, power, and utilities are using our autonomous robots to gather data of higher quality and higher quantities of data than ever before. Thousands of Spots have been deployed around the world—more than any other walking robot—to tackle this challenge. This release helps maintenance teams tap into the power of AI with new software capabilities and Spot enhancements.

[ Boston Dynamics ]

Modular self-reconfigurable robotic systems are more adaptive than conventional systems. This article proposes a novel free-form and truss-structured modular self-reconfigurable robot called FreeSN, containing node and strut modules. This article presents a novel configuration identification system for FreeSN, including connection point magnetic localization, module identification, module orientation fusion, and system-configuration fusion.

[ Freeform Robotics ]

The OOS-SIM (On-Orbit Servicing Simulator) is a simulator for on-orbit servicing tasks such as repair, maintenance and assembly that have to be carried out on satellites orbiting the earth. It simulates the operational conditions in orbit, such as the felt weightlessness and the harsh illumination.

[ DLR ]

The next CYBATHLON competition, which will take place again in 2024, breaks down barriers between the public, people with disabilities, researchers and technology developers. From 25 to 27 October 2024, the CYBATHLON will take place in a global format in the Arena Schluefweg in Kloten near Zurich and in local hubs all around the world.

[ CYBATHLON ]

George’s story is a testament to the incredible journey that unfolds when passion, opportunity and community converge. His journey from a drone enthusiast to someone actively contributing to making a difference not only to his local community but also globally; serves as a beacon of hope for all who dare to dream and pursue their passions.

[ WeRobotics ]

In case you’d forgotten, Amazon has a lot of robots.

[ Amazon Robotics ]

ABB’s fifty-year story of robotic innovation that began in 1974 with the sale of the world’s first commercial all-electric robot, the IRB 6. Björn Weichbrodt was a key figure in the development of the IRB 6.

[ ABB ]

Robotics Debate of the Ingenuity Labs Robotics and AI Symposium (RAIS2023) from October 12, 2023: Is robotics helping or hindering our progress on UN Sustainable Development Goals?

[ Ingenuity Labs ]



Today, Figure is announcing an astonishing US $675 million Series B raise, which values the company at an even more astonishing $2.6 billion. Figure is one of the companies working towards a multi or general purpose (depending on who you ask) bipedal or humanoid (depending on who you ask) robot. The astonishing thing about this valuation is that Figure’s robot is still very much in the development phase—although they’re making rapid progress, which they demonstrate in a new video posted this week.

This round of funding comes from Microsoft, OpenAI Startup Fund, Nvidia, Jeff Bezos (through Bezos Expeditions), Parkway Venture Capital, Intel Capital, Align Ventures, and ARK Invest. Figure says that they’re going to use this new capital “for scaling up AI training, robot manufacturing, expanding engineering headcount, and advancing commercial deployment efforts.” In addition, Figure and OpenAI will be collaborating on the development of “next generation AI models for humanoid robots” which will “help accelerate Figure’s commercial timeline by enhancing the capabilities of humanoid robots to process and reason from language.”

As far as that commercial timeline goes, here’s the most recent update:

Figure

And to understand everything that’s going on here, we sent a whole bunch of questions to Jenna Reher, Senior Robotics/AI Engineer at Figure.

What does “fully autonomous” mean, exactly?

Jenna Reher: In this case, we simply put the robot on the ground and hit go on the task with no other user input. What you see is using a learned vision model for bin detection that allows us to localize the robot relative to the target bin and get the bin pose. The robot can then navigate itself to within reach of the bin, determine grasp points based on the bin pose, and detect grasp success through the measured forces on the hands. Once the robot turns and sees the conveyor the rest of the task rolls out in a similar manner. By doing things in this way we can move the bins and conveyor around in the test space or start the robot from a different position and still complete the task successfully.

How many takes did it take to get this take?

Reher: We’ve been running this use case consistently for some time now as part of our work in the lab, so we didn’t really have to change much for the filming here. We did two or three practice runs in the morning and then three filming takes. All of the takes were successful, so the extras were to make sure we got the cleanest one to show.

What’s back in the Advanced Actuator Lab?

Reher: We have an awesome team of folks working on some exciting custom actuator designs for our future robots, as well as supporting and characterizing the actuators that went into our current robots.

That’s a very specific number for “speed vs human.” Which human did you measure the robot’s speed against?

Reher: We timed Brett [Adcock, founder of Figure] and a few poor engineers doing the task and took the average to get a rough baseline. If you are observant, that seemingly over-specific number is just saying we’re at 1/6 human speed. The main point that we’re trying to make here is that we are aware we are currently below human speed, and it’s an important metric to track as we improve.

What’s the tether for?

Reher: For this task we currently process the camera data off-robot while all of the behavior planning and control happens onboard in the computer that’s in the torso. Our robots should be fully tetherless in the near future as we finish packaging all of that onboard. We’ve been developing behaviors quickly in the lab here at Figure in parallel to all of the other systems engineering and integration efforts happening, so hopefully folks notice all of these subtle parallel threads converging as we try to release regular updates.

How the heck do you keep your robotics lab so clean?

Reher: Everything we’ve filmed so far is in our large robot test lab, so it’s a lot easier to keep the area clean when people’s desks aren’t intruding in the space. Definitely no guarantees on that level of cleanliness if the camera were pointed in the other direction!

Is the robot in the background doing okay?

Reher: Yes! The other robot was patiently standing there in the background, waiting for the filming to finish up so that our manipulation team could get back to training it to do more manipulation tasks. We hope we can share some more developments with that robot as the main star in the near future.

What would happen if I put a single bowling ball into that tote?

Reher: A bowling ball is particularly menacing to this task primarily due to the moving mass, in addition to the impact if you are throwing it in. The robot would in all likelihood end up dropping the tote, stay standing, and abort the task. With what you see here, we assume that the mass of the tote is known a-priori so that our whole body controller can compensate for the external forces while tracking the manipulation task. Reacting to and estimating larger unknown disturbances such as this is a challenging problem, but we’re definitely working on it.

Tell me more about that very zen arm and hand pose that the robot adopts after putting the tote on the conveyor.

Reher: It does look kind of zen! If you re-watch our coffee video you’ll notice the same pose after the robot gets things brewing. This is a reset pose that our controller will go into between manipulation tasks while the robot is awaiting commands to execute either an engineered behavior or a learned policy.

Are the fingers less fragile than they look?

Reher: They are more robust than they look, but not impervious to damage by any means. The design is pretty modular which is great, meaning that if we damage one or two fingers there is a small number of parts to swap to get everything back up and running. The current fingers won’t necessarily survive a direct impact from a bad fall, but can pick up totes and do manipulation tasks all day without issues.

Is the Figure logo footsteps?

Reher: One of the reasons I really like the figure logo is that it has a bunch of different interpretations depending on how you look at it. In some cases it’s just an F that looks like a footstep plan rollout, while some of the logo animations we have look like active stepping. One other possible interpretation could be an occupancy grid.


Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

HRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPANRoboCup 2024: 17–22 July 2024, EINDHOVEN, NETHERLANDS

Enjoy today’s videos!

Legged robots have the potential to become vital in maintenance, home support, and exploration scenarios. In order to interact with and manipulate their environments, most legged robots are equipped with a dedicated robot arm, which means additional mass and mechanical complexity compared to standard legged robots. In this work, we explore pedipulation—using the legs of a legged robot for manipulation.

This work, by Philip Arm, Mayank Mittal, Hendrik Kolvenbach, and Marco Hutter from ETHZ RSL, will be presented at the IEEE International Conference on Robotics and Automation (ICRA 2024) in May in Japan (see events calendar above).

[ Pedipulate ]

I learned a new word today! “Stigmergy.” Stigmergy is a kind of group coordination that’s based on environmental modification. Like, when insects leave pheromone trails, they’re not directly sending messages to other individuals, but as a group the ants are able to manifest surprisingly complex coordinated behaviors. Cool, right? Researchers are IRIDIA are exploring the possibilities for robots using stigmergy with a cool ‘artificial pheromone’ system using a UV-sensitive surface.

“Automatic design of stigmergy-based behaviors for robot swarms,” by Muhammad Salman, David Garzón Ramos, and Mauro Birattari, is published in the journal Communications Engineering.

[ Nature ] via [ IRIDIA ]

Thanks, David!

Filmed in July 2017, this video shows Atlas walking through a “hatch” on a pitching surface. This uses autonomous behaviors, with the robot not knowing about the rocking world. Robot built by Boston Dynamics for the DARPA Robotics Challenge in 2013. Software by IHMC Robotics.

[ IHMC ]

That IHMC video reminded me of the SAFFiR program for Shipboard Autonomous Firefighting Robots, which is responsible for a bunch of really cool research in partnership with the United States Naval Research Laboratory. NRL did some interesting stuff with Nexi robots from MIT and made their own videos. That effort I think didn’t get nearly enough credit for being very entertaining while communicating important robotics research.

[ NRL ]

I want more robot videos with this energy.

[ MIT CSAIL ]

Large industrial asset operators increasingly use robotics to automate hazardous work at their facilities. This has led to soaring demand for autonomous inspection solutions like ANYmal. Series production by our partner Zollner enables ANYbotics to supply our customers with the required quantities of robots.

[ ANYbotics ]

This week is Grain Bin Safety Week, and Grain Weevil is here to help.

[ Grain Weevil ]

Oof, this is some heavy, heavy deep-time stuff.

[ Onkalo ]

And now, this.

[ RozenZebet ]

Hawkeye is a real time multimodal conversation and interaction agent for the Boston Dynamics’ mobile robot Spot. Leveraging OpenAI’s experimental GPT-4 Turbo and Vision AI models, Hawkeye aims to empower everyone, from seniors to healthcare professionals in forming new and unique interactions with the world around them.

That moment at 1:07 is so relatable.

[ Hawkeye ]

Wing would really prefer that if you find one of their drones on the ground, you don’t run off with it.

[ Wing ]

The rover Artemis, developed at the DFKI Robotics Innovation Center, has been equipped with a penetrometer that measures the soil’s penetration resistance to obtain precise information about soil strength. The video showcases an initial test run with the device mounted on the robot. During this test, the robot was remotely controlled, and the maximum penetration depth was limited to 15 mm.

[ DFKI ]

To efficiently achieve complex humanoid loco-manipulation tasks in industrial contexts, we propose a combined vision-based tracker-localization interplay integrated as part of a task-space whole-body optimization control. Our approach allows humanoid robots, targeted for industrial manufacturing, to manipulate and assemble large-scale objects while walking.

[ Paper ]

We developed a novel multi-body robot (called the Two-Body Bot) consisting of two small-footprint mobile bases connected by a four bar linkage where handlebars are mounted. Each base measures only 29.2 cm wide, making the robot likely the slimmest ever developed for mobile postural assistance.

[ MIT ]

Lex Fridman interviews Marc Raibert.

[ Lex Fridman ]



Dina Genkina: Hi. I’m Dina Genkina for IEEE Spectrum‘s Fixing the Future. Before we start, I want to tell you that you can get the latest coverage from some of Spectrum’s most important beeps, including AI, Change, and Robotics, by signing up for one of our free newsletters. Just go to spectrum.ieee.org\newsletters to subscribe. Today, a guest is Dr. Benji Maruyama, a Principal Materials Research Engineer at the Air Force Research Laboratory, or AFRL. Dr. Maruyama is a materials scientist, and his research focuses on carbon nanotubes and making research go faster. But he’s also a man with a dream, a dream of a world where science isn’t something done by a select few locked away in an ivory tower, but something most people can participate in. He hopes to start what he calls the billion scientist movement by building AI-enabled research robots that are accessible to all. Benji, thank you for coming on the show.

Benji Maruyama: Thanks, Dina. Great to be with you. I appreciate the invitation.

Genkina: Yeah. So let’s set the scene a little bit for our listeners. So you advocate for this billion scientist movement. If everything works amazingly, what would this look like? Paint us a picture of how AI will help us get there.

Maruyama: Right, great. Thanks. Yeah. So one of the things as you set the scene there is right now, to be a scientist, most people need to have access to a big lab with very expensive equipment. So I think top universities, government labs, industry folks, lots of equipment. It’s like a million dollars, right, to get one of them. And frankly, just not that many of us have access to those kinds of instruments. But at the same time, there’s probably a lot of us who want to do science, right? And so how do we make it so that anyone who wants to do science can try, can have access to instruments so that they can contribute to it. So that’s the basics behind citizen science or democratization of science so that everyone can do it. And one way to think of it is what happened with 3D printing. It used to be that in order to make something, you had to have access to a machine shop or maybe get fancy tools and dyes that could cost tens of thousands of dollars a pop. Or if you wanted to do electronics, you had to have access to very expensive equipment or services. But when 3D printers came along and became very inexpensive, all of a sudden now, anyone with access to a 3D printer, so maybe in a school or a library or a makerspace could print something out. And it could be something fun, like a game piece, but it could also be something that got you to an invention, something that was maybe useful to the community, was either a prototype or an actual working device.

And so really, 3D printing democratized manufacturing, right? It made it so that many more of us could do things that before only a select few could. And so that’s where we’re trying to go with science now, is that instead of only those of us who have access to big labs, we’re building research robots. And when I say we, we’re doing it, but now there are a lot of others who are doing it as well, and I’ll get into that. But the example that we have is that we took a 3D printer that you can buy off the internet for less than $300. Plus a couple of extra parts, a webcam, a Raspberry Pi board, and a tripod really, so only four components. You can get them all for $300. Load them with open-source software that was developed by AFIT, the Air Force Institute of Technology. So Burt Peterson and Greg Captain [inaudible]. We worked together to build this fully autonomous 3D printing robot that taught itself how to print to better than manufacturer’s specifications. So that was a really fun advance for us, and now we’re trying to take that same idea and broaden it. So I’ll turn it back over to you.

Genkina: Yeah, okay. So maybe let’s talk a little bit about this automated research robot that you’ve made. So right now, it works with a 3D printer, but is the big picture that one day it’s going to give people access to that million dollar lab? How would that look like?

Maruyama: Right, so there are different models out there. One, we just did a workshop at the University of— sorry, North Carolina State University about that very problem, right? So there’s two models. One is to get low-cost scientific tools like the 3D printer. There’s a couple of different chemistry robots, one out of University of Maryland and NIST, one out of University of Washington that are in the sort of 300 to 1,000 dollars range that makes it accessible. The other part is kind of the user facility model. So in the US, the Department of Energy National Labs have many user facilities where you can apply to get time on very expensive instruments. Now we’re talking tens of millions. For example, Brookhaven has a synchrotron light source where you can sign up and it doesn’t cost you any money to use the facility. And you can get days on that facility. And so that’s already there, but now the advances are that by using this, autonomy, autonomous closed loop experimentation, that the work that you do will be much faster and much more productive. So, for example, on ARES, our Autonomous Research System at AFRL, we actually were able to do experiments so fast that a professor who came into my lab said, it just took me aside and said, “Hey, Benji, in a week’s worth of time, I did a dissertation’s worth of research.” So maybe five years worth of research in a week. So imagine if you keep doing that week after week after week, how fast research goes. So it’s very exciting.

Genkina: Yeah, so tell us a little bit about how that works. So what’s this system that has sped up five years of research into a week and made graduate students obsolete? Not yet, not yet. How does that work? Is that the 3D printer system or is that a—

Maruyama: So we started with our system to grow carbon nanotubes. And I’ll say, actually, when we first thought about it, your comment about graduate students being absolute— obsolete, sorry, is interesting and important because, when we first built our system that worked it 100 times faster than normal, I thought that might be the case. We called it sort of graduate student out of the loop. But when I started talking with people who specialize in autonomy, it’s actually the opposite, right? It’s actually empowering graduate students to go faster and also to do the work that they want to do, right? And so just to digress a little bit, if you think about farmers before the Industrial Revolution, what were they doing? They were plowing fields with oxen and beasts of burden and hand plows. And it was hard work. And now, of course, you wouldn’t ask a farmer today to give up their tractor or their combine harvester, right? They would say, of course not. So very soon, we expect it to be the same for researchers, that if you asked a graduate student to give up their autonomous research robot five years from now, they’ll say, “Are you crazy? This is how I get my work done.”

But for our original ARES system, it worked on the synthesis of carbon nanotubes. So that meant that what we’re doing is trying to take this system that’s been pretty well studied, but we haven’t figured out how to make it at scale. So at hundreds of millions of tons per year, sort of like polyethylene production. And part of that is because it’s slow, right? One experiment takes a day, but also because there are just so many different ways to do a reaction, so many different combinations of temperature and pressure and a dozen different gases and half the periodic table as far as the catalyst. It’s just too much to just brute force your way through. So even though we went from experiments where we could do 100 experiments a day instead of one experiment a day, just that combinatorial space was vastly overwhelmed our ability to do it, even with many research robots or many graduate students. So the idea of having artificial intelligence algorithms that drive the research is key. And so that ability to do an experiment, see what happened, and then analyze it, iterate, and constantly be able to choose the optimal next best experiment to do is where ARES really shines. And so that’s what we did. ARES taught itself how to grow carbon nanotubes at controlled rates. And we were the first ones to do that for material science in our 2016 publication.

Genkina: That’s very exciting. So maybe we can peer under the hood a little bit of this AI model. How does the magic work? How does it pick the next best point to take and why it’s better than you could do as a graduate student or researcher?

Maruyama: Yeah, and so I think it’s interesting, right? In science, a lot of times we’re taught to hold everything constant, change one variable at a time, search over that entire space, see what happened, and then go back and try something else, right? So we reduce it to one variable at a time. It’s a reductionist approach. And that’s worked really well, but a lot of the problems that we want to go after are simply too complex for that reductionist approach. And so the benefit of being able to use artificial intelligence is that high dimensionality is no problem, right? Tens of dimensions search over very complex high-dimensional parameter space, which is overwhelming to humans, right? Is just basically bread and butter for AI. The other part to it is the iterative part. The beauty of doing autonomous experimentation is that you’re constantly iterating. You’re constantly learning over what just happened. You might also say, well, not only do I know what happened experimentally, but I have other sources of prior knowledge, right? So for example, ideal gas law says that this should happen, right? Or Gibbs phase rule might say, this can happen or this can’t happen. So you can use that prior knowledge to say, “Okay, I’m not going to do those experiments because that’s not going to work. I’m going to try here because this has the best chance of working.”

And within that, there are many different machine learning or artificial intelligence algorithms. Bayesian optimization is a popular one to help you choose what experiment is best. There’s also new AI that people are trying to develop to get better search.

Genkina: Cool. And so the software part of this autonomous robot is available for anyone to download, which is also really exciting. So what would someone need to do to be able to use that? Do they need to get a 3D printer and a Raspberry Pi and set it up? And what would they be able to do with it? Can they just build carbon nanotubes or can they do more stuff?

Maruyama: Right. So what we did, we built ARES OS, which is our open source software, and we’ll make sure to get you the GitHub link so that anyone can download it. And the idea behind ARES OS is that it provides a software framework for anyone to build their own autonomous research robot. And so the 3D printing example will be out there soon. But it’s the starting point. Of course, if you want to build your own new kind of robot, you still have to do the software development, for example, to link the ARES framework, the core, if you will, to your particular hardware, maybe your particular camera or 3D printer, or pipetting robot, or spectrometer, whatever that is. We have examples out there and we’re hoping to get to a point where it becomes much more user-friendly. So having direct Python connects so that you don’t— currently it’s programmed in C#. But to make it more accessible, we’d like it to be set up so that if you can do Python, you can probably have good success in building your own research robot.

Genkina: Cool. And you’re also working on a educational version of this, I understand. So what’s the status of that and what’s different about that version?

Maruyama: Yeah, right. So the educational version is going to be-- its sort of composition of a combination of hardware and software. So what we’re starting with is a low-cost 3D printer. And we’re collaborating now with the University at Buffalo, Materials Design Innovation Department. And we’re hoping to build up a robot based on a 3D printer. And we’ll see how it goes. It’s still evolving. But for example, it could be based on this very inexpensive $200 3D printer. It’s an Ender 3D printer. There’s another printer out there that’s based on University of Washington’s Jubilee printer. And that’s a very exciting development as well. So professors Lilo Pozzo and Nadya Peek at the University of Washington built this Jubilee robot with that idea of accessibility in mind. And so combining our ARES OS software with their Jubilee robot hardware is something that I’m very excited about and hope to be able to move forward on.

Genkina: What’s this Jubilee 3D printer? How is it different from a regular 3D printer?

Maruyama: It’s very open source. Not all 3D printers are open source and it’s based on a gantry system with interchangeable heads. So for example, you can get not just a 3D printing head, but other heads that might do things like do indentation, see how stiff something is, or maybe put a camera on there that can move around. And so it’s the flexibility of being able to pick different heads dynamically that I think makes it super useful. For the software, right, we have to have a good, accessible, user-friendly graphical user interface, a GUI. That takes time and effort, so we want to work on that. But again, that’s just the hardware software. Really to make ARES a good educational platform, we need to make it so that a teacher who’s interested can have the lowest activation barrier possible, right? We want she or he to be able to pull a lesson plan off of the internet, have supporting YouTube videos, and actually have the material that is a fully developed curriculum that’s mapped against state standards.

So that, right now, if you’re a teacher who— let’s face it, teachers are already overwhelmed with all that they have to do, putting something like this into their curriculum can be a lot of work, especially if you have to think about, well, I’m going to take all this time, but I also have to meet all of my teaching standards, all the state curriculum standards. And so if we build that out so that it’s a matter of just looking at the curriculum and just checking off the boxes of what state standards it maps to, then that makes it that much easier for the teacher to teach.

Genkina: Great. And what do you think is the timeline? Do you expect to be able to do this sometime in the coming year?

Maruyama: That’s right. These things always take longer than hoped for than expected, but we’re hoping to do it within this calendar year and very excited to get it going. And I would say for your listeners, if you’re interested in working together, please let me know. We’re very excited about trying to involve as many people as we can.

Genkina: Great. Okay, so you have the educational version, and you have the more research geared version, and you’re working on making this educational version more accessible. Is there something with the research version that you’re working on next, how you’re hoping to upgrade it, or is there something you’re using it for right now that you’re excited about?

There’s a number of things that we are very excited about the possibility of carbon nanotubes being produced at very large scale. So right now, people may remember carbon nanotubes as that great material that sort of never made it and was very overhyped. But there’s a core group of us who are still working on it because of the important promise of that material. So it’s material that is super strong, stiff, lightweight, electrically conductive. Much better than silicon as a digital electronics compute material. All of those great things, except we’re not making it at large enough scale. It’s actually used pretty significantly in lithium-ion batteries. It’s an important application. But other than that, it’s sort of like where’s my flying car? It’s never panned out. But there’s, as I said, a group of us who are working to really produce carbon nanotubes at much larger scale. So large scale for nanotubes now is sort of in the kilogram or ton scale. But what we need to get to is hundreds of millions of tons per year production rates. And why is that? Well, there’s a great effort that came out of ARPA-E. So the Department of Energy Advanced Research Projects Agency and the E is for Energy in that case.

So they funded a collaboration between Shell Oil and Rice University to pyrolyze methane, so natural gas into hydrogen for the hydrogen economy. So now that’s a clean burning fuel plus carbon. And instead of burning the carbon to CO2, which is what we now do, right? We just take natural gas and feed it through a turbine and generate electric power instead of— and that, by the way, generates so much CO2 that it’s causing global climate change. So if we can do that pyrolysis at scale, at hundreds of millions of tons per year, it’s literally a save the world proposition, meaning that we can avoid so much CO2 emissions that we can reduce global CO2 emissions by 20 to 40 percent. And that is the save the world proposition. It’s a huge undertaking, right? That’s a big problem to tackle, starting with the science. We still don’t have the science to efficiently and effectively make carbon nanotubes at that scale. And then, of course, we have to take the material and turn it into useful products. So the batteries is the first example, but thinking about replacing copper for electrical wire, replacing steel for structural materials, aluminum, all those kinds of applications. But we can’t do it. We can’t even get to that kind of development because we haven’t been able to make the carbon nanotubes at sufficient scale.

So I would say that’s something that I’m working on now that I’m very excited about and trying to get there, but it’s going to take some good developments in our research robots and some very smart people to get us there.

Genkina: Yeah, it seems so counterintuitive that making everything out of carbon is good for lowering carbon emissions, but I guess that’s the break.

Maruyama: Yeah, it is interesting, right? So people talk about carbon emissions, but really, the molecule that’s causing global warming is carbon dioxide, CO2, which you get from burning carbon. And so if you take that methane and parallelize it to carbon nanotubes, that carbon is now sequestered, right? It’s not going off as CO2. It’s staying in solid state. And not only is it just not going up into the atmosphere, but now we’re using it to replace steel, for example, which, by the way, steel, aluminum, copper production, all of those things emit lots of CO2 in their production, right? They’re energy intensive as a material production. So it’s kind of ironic.

Genkina: Okay, and are there any other research robots that you’re excited about that you think are also contributing to this democratization of science process?

Maruyama: Yeah, so we talked about Jubilee, the NIST robot, which is from Professor Ichiro Takeuchi at Maryland and Gilad Kusne at NIST, National Institute of Standards and Technology. Theirs is fun too. It’s LEGO as. So it’s actually based on a LEGO robotics platform. So it’s an actual chemistry robot built out of Legos. So I think that’s fun as well. And you can imagine, just like we have LEGO robot competitions, we can have autonomous research robot competitions where we try and do research through these robots or competitions where everybody sort of starts with the same robot, just like with LEGO robotics. So that’s fun as well. But I would say there’s a growing number of people doing these kinds of, first of all, low-cost science, accessible science, but in particular low-cost autonomous experimentation.

Genkina: So how far are we from a world where a high school student has an idea and they can just go and carry it out on some autonomous research system at some high-end lab?

Maruyama: That’s a really good question. I hope that it’s going to be in 5 to 10 years, that it becomes reasonably commonplace. But it’s going to take still some significant investment to get this going. And so we’ll see how that goes. But I don’t think there are any scientific impediments to getting this done. There is a significant amount of engineering to be done. And sometimes we hear, oh, it’s just engineering. The engineering is a significant problem. And it’s work to get some of these things accessible, low cost. But there are lots of great efforts. There are people who have used CDs, compact discs to make spectrometers out of. There are lots of good examples of citizen science out there. But it’s, I think, at this point, going to take investment in software, in hardware to make it accessible, and then importantly, getting students really up to speed on what AI is and how it works and how it can help them. And so I think it’s actually really important. So again, that’s the democratization of science is if we can make it available to everyone and accessible, then that helps people, everyone contribute to science. And I do believe that there are important contributions to be made by ordinary citizens, by people who aren’t you know PhDs working in a lab.

And I think there’s a lot of science out there to be done. If you ask working scientists, almost no one has run out of ideas or things they want to work on. There’s many more scientific problems to work on than we have the time where people are funding to work on. And so if we make science cheaper to do, then all of a sudden, more people can do science. And so those questions start to be resolved. And so I think that’s super important. And now we have, instead of, just those of us who work in big labs, you have millions, tens of millions, up to a billion people, that’s the billion scientist idea, who are contributing to the scientific community. And that, to me, is so powerful that many more of us can contribute than just the few of us who do it right now.

Genkina: Okay, that’s a great place to end on, I think. So, today we spoke to Dr. Benji Maruyama, a material scientist at AFRL, about his efforts to democratize scientific discovery through automated research robots. For IEEE Spectrum, I’m Dina Genkina, and I hope you’ll join us next time on Fixing the Future.



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Cybathlon Challenges: 2 February 2024, ZURICHHRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPAN

Enjoy today’s videos!

Just like a real human, Acrobot will sometimes kick you in the face.

[ Acrobotics ]

Thanks, Elizabeth!

You had me at “wormlike, limbless robots.”

[ GitHub ] via [ Georgia Tech ]

Filmed in July 2017, this video shows us using Atlas to put out a “fire” on our loading dock. This uses a combination of teleoperation and autonomous behaviors through a single, remote computer. Robot built by Boston Dynamics for the DARPA Robotics Challenge in 2013. Software by IHMC Robotics.

I would say that in the middle of a rainstorm is probably the best time to start a fire that you expect to be extinguished by a robot.

[ IHMC ]

We’re hard at work, but Atlas still has time for a dance break.

[ Boston Dynamics ]

This is pretty cool: BruBotics is testing its self-healing robotics gripper technology on commercial grippers from Festo.

[ Paper ] via [ BruBotics ]

Thanks, Bram!

You should read our in-depth article on Stretch 3, so if you haven’t yet, consider this as just a teaser.

[ Hello Robot ]

Inspired by caregiving experts, we proposed a bimanual interactive robotic dressing assistance scheme, which is unprecedented in previous research. In the scheme, an interactive robot joins hands with the human thus supporting/guiding the human in the dressing process, while the dressing robot performs the dressing task. This work represents a paradigm shift of thinking of the dressing assistance task from one-robot-to-one-arm to two-robot-to-one-arm.

[ Project ]

Thanks, Jihong!

Tony Punnoose Valayil from the Bulgarian Academy of Sciences Institute of Robotics wrote in to share some very low-cost hand-rehabilitation robots for home use.

In this video, we present a robot-assisted rehabilitation of the wrist joint which can aid in restoring the strength that has been lost across the upper limb due to stroke. This robot is very cost-effective and can be used for home rehabilitation.

In this video, we present an exoskeleton robot which can be used at home for rehabilitating the index and middle fingers of stroke-affected patients. This robot is built at a cost of 50 euros for patients who are not financially independent to get better treatment.

[ BAS ]

Some very impressive work here from the Norwegian University of Science and Technology (NTNU), showing a drone tracking its position using radar and lidar-based odometry in some nightmare (for robots) environments, including a long tunnel that looks the same everywhere and a hallway full of smoke.

[ Paper ] via [ GitHub ]

I’m sorry, but people should really know better than to make videos like this for social robot crowdfunding by now.

It’s on Kickstarter for about $300, and the fact that it’s been funded so quickly tells me that people have already forgotten about the social robotpocalypse.

[ Kickstarter ]

Introducing Orbit, your portal for managing asset-intensive facilities through real-time and predictive intelligence. Orbit brings a whole new suite of fleet management capabilities and will unify your ecosystem of Boston Dynamics robots, starting with Spot.

[ Boston Dynamics ]



A lot has happened in robotics over the last year. Everyone is wondering how AI will transform robotics, and everyone else is wondering whether humanoids are going to blow it or not, and the rest of us are busy trying not to get completely run over as things shake out however they’re going to shake out.

Meanwhile, over at Hello Robot, they’ve been focused on making their Stretch robot do useful things while also being affordable and reliable and affordable and expandable and affordable and community-friendly and affordable. Which are some really hard and important problems that can sometimes get overwhelmed by flashier things.

Today, Hello Robot is announcing Stretch 3, which provides a suite of upgrades to what they (quite accurately) call “the world’s only lightweight, capable, developer-friendly mobile manipulator.” And impressively, they’ve managed to do it without forgetting about that whole “affordable” part.

Hello Robot

Stretch 3 looks about the same as the previous versions, but there are important upgrades that are worth highlighting. The most impactful: Stretch 3 now comes with the dexterous wrist kit that used to be an add-on, and it now also includes an Intel Realsense D405 camera mounted right behind the gripper, which is a huge help for both autonomy and remote teleoperation—a useful new feature shipping with Stretch 3 that’s based on research out of Maya Cakmak’s lab at the University of Washington, in Seattle. This is an example of turning innovation from the community of Stretch users into product features, a product-development approach that seems to be working well for Hello Robot.

“We’ve really been learning from our community,” says Hello Robot cofounder and CEO Aaron Edsinger. “In the past year, we’ve seen a real uptick in publications, and it feels like we’re getting to this critical-mass moment with Stretch. So with Stretch 3, it’s about implementing features that our community has been asking us for.”

“When we launched, we didn’t have a dexterous wrist at the end as standard, because we were trying to start with truly the minimum viable product,” says Hello Robot cofounder and CTO Charlie Kemp. “And what we found is that almost every order was adding the dexterous wrist, and by actually having it come in standard, we’ve been able to devote more attention to it and make it a much more robust and capable system.”

Kemp says that having Stretch do everything right out of the box (with Hello Robot support) makes a big difference for their research customers. “Making it easier for people to try things—we’ve learned to really value that, because the more steps that people have to go through to experience it, the less likely they are to build on it.” In a research context, this is important because what you’re really talking about is time: The more time people spend just trying to make the robot function, the less time they’ll spend getting the robot to do useful things.

Hello Robot

At this point, you may be thinking of Stretch as a research platform. Or you may be thinking of Stretch as a robot for people with disabilities, if you read our November 2023 cover story about Stretch and Henry and Jane Evans. And the robot is definitely both of those things. But Hello Robot stresses that these specific markets are not their end goal—they see Stretch as a generalist mobile manipulator with a future in the home, as suggested by this Stretch 3 promo video:

Hello Robot

Dishes, laundry, bubble cannons: All of these are critical to the functionality of any normal household. “Stretch is an inclusive robot,” says Kemp. “It’s not just for older adults or people with disabilities. We want a robot that can be beneficial for everyone. Our vision, and what we believe will really happen, whether it’s us or someone else, is that there is going to be a versatile, general-purpose home robot. Right now, clearly, our market is not yet consumers in the home. But that’s where we want to go.”

Robots in the home have been promised for decades, and with the notable exception of the Roomba, there has not been a lot of success. The idea of a robot that could handle dishes or laundry is tempting, but is it near-term or medium-term realistic? Edsinger, who has been at this whole robots thing for a very long time, is an optimist about this, and about the role that Stretch will play. “There are so many places where you can see the progress happening—in sensing, in manipulation,” Edsinger says. “I can imagine those things coming together now in a way that I could not have 5 to 10 years ago, when it seemed so incredibly hard.”

“We’re very pragmatic about what is possible. And I think that we do believe that things are changing faster than we anticipated—10 years ago, I had a pretty clear linear path in mind for robotics, but it’s hard to really imagine where we’ll be in terms of robot capabilities 10 years from now.” —Aaron Edsinger, Hello Robot

I’d say that it’s still incredibly hard, but Edsinger is right that a lot of the pieces do seem to be coming together. Arguably, the hardware is the biggest challenge here, because working in a home puts heavy constraints on what kind of hardware you’re able to use. You’re not likely to see a humanoid in a home anytime soon, because they’d actually be dangerous, and even a quadruped is likely to be more trouble than it’s worth in a home environment. Hello Robot is conscious of this, and that’s been one of the main drivers of the design of Stretch.

“I think the portability of Stretch is really worth highlighting because there’s just so much value in that which is maybe not obvious,” Edsinger tells us. Being able to just pick up and move a mobile manipulator is not normal. Stretch’s weight (24.5 kilograms) is almost trivial to work with, in sharp contrast with virtually every other mobile robot with an arm: Stretch fits into places that humans fit into, and manages to have a similar workspace as well, and its bottom-heavy design makes it safe for humans to be around. It can’t climb stairs, but it can be carried upstairs, which is a bigger deal than it may seem. It’ll fit in the back of a car, too. Stretch is built to explore the world—not just some facsimile of the world in a research lab.

NYU students have been taking Stretch into tens of homes around New York,” says Edsinger. “They carried one up a four-story walk-up. This enables real in-home data collection. And this is where home robots will start to happen—when you can have hundreds of these out there in homes collecting data for machine learning.”

“That’s where the opportunity is,” adds Kemp. “It’s that engagement with the world about where to apply the technology beneficially. And if you’re in a lab, you’re not going to find it.”

We’ve seen some compelling examples of this recently, with Mobile ALOHA. These are robots learning to be autonomous by having humans teleoperate them through common household skills. But the system isn’t particularly portable, and it costs nearly US $32,000 in parts alone. Don’t get me wrong: I love the research. It’s just going to be difficult to scale, and in order to collect enough data to effectively tackle the world, scale is critical. Stretch is much easier to scale, because you can just straight up buy one.

Or two! You may have noticed that some of the Stretch 3 videos have two robots in them, collaborating with each other. This is not yet autonomous, but with two robots, a single human (or a pair of humans) can teleoperate them as if they were effectively a single two-armed robot:

Hello Robot

Essentially, what you’ve got here is a two-armed robot that (very intentionally) has nothing to do with humanoids. As Kemp explains: “We’re trying to help our community and the world see that there is a different path from the human model. We humans tend to think of the preexisting solution: People have two arms, so we think, well, I’m going to need to have two arms on my robot or it’s going to have all these issues.” Kemp points out that robots like Stretch have shown that really quite a lot of things can be done with only one arm, but two arms can still be helpful for a substantial subset of common tasks. “The challenge for us, which I had just never been able to find a solution for, was how you get two arms into a portable, compact, affordable lightweight mobile manipulator. You can’t!”

But with two Stretches, you have not only two arms but also two shoulders that you can put wherever you want. Washing a dish? You’ll probably want two arms close together for collaborative manipulation. Making a bed? Put the two arms far apart to handle both sides of a sheet at once. It’s a sort of distributed on-demand bimanual manipulation, which certainly adds a little bit of complexity but also solves a bunch of problems when it comes to practical in-home manipulation. Oh—and if those teleop tools look like modified kitchen tongs, that’s because they’re modified kitchen tongs.

Of course, buying two Stretch robots is twice as expensive as buying a single Stretch robot, and even though Stretch 3’s cost of just under $25,000 is very inexpensive for a mobile manipulator and very affordable in a research or education context, we’re still pretty far from something that most people would be able to afford for themselves. Hello Robot says that producing robots at scale is the answer here, which I’m sure is true, but it can be a difficult thing for a small company to achieve.

Moving slowly toward scale is at least partly intentional, Kemp tells us. “We’re still in the process of discovering Stretch’s true form—what the robot really should be. If we tried to scale to make lots and lots of robots at a much lower cost before we fundamentally understood what the needs and challenges were going to be, I think it would be a mistake. And there are many gravestones out there for various home-robotics companies, some of which I truly loved. We don’t want to become one of those.”

This is not to say that Hello Robot isn’t actively trying to make Stretch more affordable, and Edsinger suggests that the next iteration of the robot will be more focused on that. But—and this is super important—Kemp tells us that Stretch has been, is, and will continue to be sustainable for Hello Robot: “We actually charge what we should be charging to be able to have a sustainable business.” In other words, Hello Robot is not relying on some nebulous scale-defined future to transition into a business model that can develop, sell, and support robots. They can do that right now while keeping the lights on. “Our sales have enough margin to make our business work,” says Kemp. “That’s part of our discipline.”

Stretch 3 is available now for $24,950, which is just about the same as the cost of Stretch 2 with the optional add-ons included. There are lots and lots of other new features that we couldn’t squeeze into this article, including FCC certification, a more durable arm, and off-board GPU support. You’ll find a handy list of all the upgrades here.


Odorigui is a type of Japanese cuisine in which people consume live seafood while it’s still moving, making movement part of the experience. You may have some feelings about this (I definitely do), but from a research perspective, getting into what those feelings are and what they mean isn’t really practical. To do so in a controlled way would be both morally and technically complicated, which is why Japanese researchers have started developing robots that can be eaten as they move, wriggling around in your mouth as you chomp down on them. Welcome to HERI: Human-Edible Robot Interaction.

That happy little robot that got its head ripped off by a hungry human (who, we have to say, was exceptionally polite about it) is made primarily of gelatin, along with sugar and apple juice for taste. After all the ingredients were mixed, it was poured into a mold and refrigerated for 12 hours to set, with the resulting texture ending up like a chewy gummy candy. The mold incorporated a couple of air chambers into the structure of the robot, which were hooked up to pneumatics that got the robot to wiggle back and forth.

Sixteen students at Osaka University got the chance to eat one of these wiggly little robots. The process was to put your mouth around the robot, let the robot move around in there for 10 seconds for the full experience, and then bite it off, chew, and swallow. Japanese people were chosen partly because this research was done in Japan, but also because, according to the paper, “of the cultural influences on the use of onomatopoeic terms.” In Japanese, there are terms that are useful in communicating specific kinds of textures that can’t easily be quantified.

The participants were asked a series of questions about their experience, including some heavy ones:

  • Did you think what you just ate had animateness?
  • Did you feel an emotion in what you just ate?
  • Did you think what you just ate had intelligence?
  • Did you feel guilty about what you just ate?

Oof.

Compared to a control group of students who ate the robot when it was not moving, the students who ate the moving robot were more likely to interpret it as having a “munya-munya” or “mumbly” texture, showing that movement can influence the eating experience. Analysis of question responses showed that the moving robot also caused people to perceive it as emotive and intelligent, and caused more feelings of guilt when it was consumed. The paper summarizes it pretty well: “In the stationary condition, participants perceived the robot as ‘food,’ whereas in the movement condition, they perceived it as a ‘creature.’”

The good news here is that since these robots are more like living things than non-robots, they could potentially stand in for eating live critters in a research context, say the researchers: “The utilization of edible robots in this study enabled us to examine the effects of subtle movement variations in human eating behavior under controlled conditions, a task that would be challenging to accomplish with real organisms.” There’s still more work to do to make the robots more like specific living things, but that’s the plan going forward:

Our proposed edible robot design does not specifically mimic any particular biological form. To address these limitations, we will focus on the field by designing edible robots that imitate forms relevant to ongoing discussions on food shortages and cultural delicacies. Specifically, in future studies, we will emulate creatures consumed in contexts such as insect-based diets, which are being considered as a solution to food scarcity issues, and traditional Japanese dishes like “Odorigui” or “Ikizukuri (live fish sashimi).” These imitations are expected to provide deep insights into the psychological and cognitive responses elicited when consuming moving robots, merging technology with necessities and culinary traditions.

Exploring the eating experience of a pneumatically-driven edible robot: Perception, taste, and texture, by Yoshihiro NakataI, Midori Ban, Ren Yamaki, Kazuya Horibe, Hideyuki Takahashi, and Hiroshi Ishiguro from The University of Electro-Communications and Osaka University, is published in PLOS One.



Just last month, Oslo, Norway-based 1X (formerly Halodi Robotics) announced a massive $100 million Series B, and clearly they’ve been putting the work in. A new video posted last week shows a [insert collective noun for humanoid robots here] of EVE android-ish mobile manipulators doing a wide variety of tasks leveraging end-to-end neural networks (pixels to actions). And best of all, the video seems to be more or less an honest one: a single take, at (appropriately) 1X speed, and full autonomy. But we still had questions! And 1X has answers.

If, like me, you had some very important questions after watching this video, including whether that plant is actually dead and the fate of the weighted companion cube, you’ll want to read this Q&A with Eric Jang, Vice President of Artificial Intelligence at 1X.

IEEE Spectrum: How many takes did it take to get this take?

Eric Jang: About 10 takes that lasted more than a minute; this was our first time doing a video like this, so it was more about learning how to coordinate the film crew and set up the shoot to look impressive.

Did you train your robots specifically on floppy things and transparent things?

Jang: Nope! We train our neural network to pick up all kinds of objects—both rigid and deformable and transparent things. Because we train manipulation end-to-end from pixels, picking up deformables and transparent objects is much easier than a classical grasping pipeline, where you have to figure out the exact geometry of what you are trying to grasp.

What keeps your robots from doing these tasks faster?

Jang: Our robots learn from demonstrations, so they go at exactly the same speed the human teleoperators demonstrate the task at. If we gathered demonstrations where we move faster, so would the robots.

How many weighted companion cubes were harmed in the making of this video?

Jang: At 1X, weighted companion cubes do not have rights.

That’s a very cool method for charging, but it seems a lot more complicated than some kind of drive-on interface directly with the base. Why use manipulation instead?

Jang: You’re right that this isn’t the simplest way to charge the robot, but if we are going to succeed at our mission to build generally capable and reliable robots that can manipulate all kinds of objects, our neural nets have to be able to do this task at the very least. Plus, it reduces costs quite a bit and simplifies the system!

What animal is that blue plush supposed to be?

Jang: It’s an obese shark, I think.

How many different robots are in this video?

Jang: 17? And more that are stationary.

How do you tell the robots apart?

Jang: They have little numbers printed on the base.

Is that plant dead?

Jang: Yes, we put it there because no CGI / 3D rendered video would ever go through the trouble of adding a dead plant.

What sort of existential crisis is the robot at the window having?

Jang: It was supposed to be opening and closing the window repeatedly (good for testing statistical significance).

If one of the robots was actually a human in a helmet and a suit holding grippers and standing on a mobile base, would I be able to tell?

Jang: I was super flattered by this comment on the Youtube video:

But if you look at the area where the upper arm tapers at the shoulder, it’s too thin for a human to fit inside while still having such broad shoulders:

Why are your robots so happy all the time? Are you planning to do more complex HRI stuff with their faces?

Jang: Yes, more complex HRI stuff is in the pipeline!

Are your robots able to autonomously collaborate with each other?

Jang: Stay tuned!

Is the skew tetromino the most difficult tetromino for robotic manipulation?

Jang: Good catch! Yes, the green one is the worst of them all because there are many valid ways to pinch it with the gripper and lift it up. In robotic learning, if there are multiple ways to pick something up, it can actually confuse the machine learning model. Kind of like asking a car to turn left and right at the same time to avoid a tree.

Everyone else’s robots are making coffee. Can your robots make coffee?

Jang: Yep! We were planning to throw in some coffee making on this video as an easter egg, but the coffee machine broke right before the film shoot and it turns out it’s impossible to get a Keurig K-Slim in Norway via next day shipping.

1X is currently hiring both AI researchers (imitation learning, reinforcement learning, large-scale training, etc) and android operators (!) which actually sounds like a super fun and interesting job. More here.



This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

If Disney’s history of storytelling has taught us anything, it’s to never underestimate the power of a great sidekick. Even though sidekicks aren’t the stars of the show, they provide life and energy and move the story along in important ways. It’s hard to imagine Aladdin without the Genie, or Peter Pan without Tinker Bell.

In robotics, however, solo acts proliferate. Even when multiple robots are used, they usually act in parallel. One key reason for this is that most robots are designed in ways that make direct collaboration with other robots difficult. Stiff, strong robots are more repeatable and easier to control, but those designs have very little forgiveness for the imperfections and mismatches that are inherent in coming into contact with another robot.

Having robots work together–especially if they have complementary skill sets–can open up some exciting opportunities, especially in the entertainment robotics space. At Walt Disney Imagineering, our research and development teams have been working on this idea of collaboration between robots, and we were able to show off the result of one such collaboration in Shanghai this week, when a little furry character interrupted the opening moments for the first-ever Zootopia land.

Our newest robotic character, Duke Weaselton, rolled onstage at the Shanghai Disney Resort for the first time last December, pushing a purple kiosk and blasting pop music. As seen in the video below, the audience got a kick out of watching him hop up on top of the kiosk and try to negotiate with the Chairman of Disney Experiences, Josh D’Amaro, for a new job. And of course, some new perks. After a few moments of wheeling and dealing, Duke gets gently escorted offstage by team members Richard Landon and Louis Lambie.

What might not be obvious at first is that the moment you just saw was enabled not by one robot, but by two. Duke Weaselton is the star of the show, but his dynamic motion wouldn’t be possible without the kiosk, which is its own independent, actuated robot. While these two robots are very different, by working together as one system, they’re able to do things that neither could do alone.

The character and the kiosk bring two very different kinds of motion together, and create something more than the sum of their parts in the process. The character is an expressive, bipedal robot with an exaggerated, animated motion style. It looks fantastic, but it’s not optimized for robust, reliable locomotion. The kiosk, meanwhile, is a simple wheeled system that behaves in a highly predictable way. While that’s great for reliability, it means that by itself it’s not likely to surprise you. But when we combine these two robots, we get the best of both worlds. The character robot can bring a zany, unrestrained energy and excitement as it bounces up, over, and alongside the kiosk, while the kiosk itself ensures that both robots reliably get to wherever they are going.

Harout Jarchafjian, Sophie Bowe, Tony Dohi, Bill West, Marcela de los Rios, Bob Michel, and Morgan Pope.Morgan Pope

The collaboration between the two robots is enabled by designing them to be robust and flexible, and with motions that can tolerate a large amount of uncertainty while still delivering a compelling show. This is a direct result from lessons learned from an earlier robot, one that tumbled across the stage at SXSW earlier this year. Our basic insight is that a small, lightweight robot can be surprisingly tough, and that this toughness enables new levels of creative freedom in the design and execution of a show.

This level of robustness also makes collaboration between robots easier. Because the character robot is tough and because there is some flexibility built into its motors and joints, small errors in placement and pose don’t create big problems like they might for a more conventional robot. The character can lean on the motorized kiosk to create the illusion that it is pushing it across the stage. The kiosk then uses a winch to hoist the character onto a platform, where electromagnets help stabilize its feet. Essentially, the kiosk is compensating for the fact that Duke himself can’t climb, and might be a little wobbly without having his feet secured. The overall result is a free-ranging bipedal robot that moves in a way that feels natural and engaging, but that doesn’t require especially complicated controls or highly precise mechanical design. Here’s a behind-the-scenes look at our development of these systems:

Disney Imagineering

To program Duke’s motions, our team uses an animation pipeline that was originally developed for the SXSW demo, where a designer can pose the robot by hand to create new motions. We have since developed an interface which can also take motions from conventional animation software tools. Motions can then be adjusted to adapt to the real physical constraints of the robots, and that information can be sent back to the animation tool. As animations are developed, it’s critical to retain a tight synchronization between the kiosk and the character. The system is designed so that the motion of both robots is always coordinated, while simultaneously supporting the ability to flexibly animate individual robots–or individual parts of the robot, like the mouth and eyes.

Over the past nine months, we explored a few different kinds of collaborative locomotion approaches. The GIFs below show some early attempts at riding a tricycle, skateboarding, and pushing a crate. In each case, the idea is for a robotic character to eventually collaborate with another robotic system that helps bring that character’s motions to life in a stable and repeatable way.

Disney hopes that their Judy Hopps robot will soon be able to use the help of a robotic tricycle, crate, or skateboard to enable new forms of locomotion.Morgan Pope

This demo with Duke Weaselton and his kiosk is just the beginning, says Principal R&D Imagineer Tony Dohi, who leads the project for us. “Ultimately, what we showed today is an important step towards a bigger vision. This project is laying the groundwork for robots that can interact with each other in surprising and emotionally satisfying ways. Today it’s a character and a kiosk, but moving forward we want to have multiple characters that can engage with each other and with our guests.”

Walt Disney Imagineering R&D is exploring a multi-pronged development strategy for our robotic characters. Engaging character demonstrations like Duke Weasleton focus on quickly prototyping complete experiences using immediately accessible techniques. In parallel, our research group is developing new technologies and capabilities that become the building blocks for both elevating existing experiences, and designing and delivering completely new shows. The robotics team led by Moritz Bächer shared one such building block–embodied in a highly expressive and stylized robotic walking character–at IROS in October. The capabilities demonstrated there can eventually be used to help robots like Duke Weaselton perform more flexibly, more reliably, and more spectacularly.

“Authentic character demonstrations are useful because they help inform what tools are the most valuable for us to develop,” explains Bächer. “In the end our goal is to create tools that enable our teams to produce and deliver these shows rapidly and efficiently.” This ties back to the fundamental technical idea behind the Duke Weaselton show moment–collaboration is key!



Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Cybathlon Challenges: 02 February 2024, ZURICHHRI 2024: 11–15 March 2024, BOULDER, COLO.Eurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPAN

Enjoy today’s videos!

In this video, we present Ringbot, a novel leg-wheel transformer robot incorporating a monocycle mechanism with legs. Ringbot aims to provide versatile mobility by replacing the driver and driving components of a conventional monocycle vehicle with legs mounted on compact driving modules inside the wheel.

[ Paper ] via [ KIMLAB ]

Making money with robots has always been a struggle, but I think ALOHA 2 has figured it out.

Seriously, though, that is some impressive manipulation capability. I don’t know what that freakish panda thing is, but getting a contact lens from the package onto its bizarre eyeball was some wild dexterity.

[ ALOHA 2 ]

Highlights from testing our new arms built by Boardwalk Robotics. Installed in October of 2023, these new arms are not just for boxing, and are provide much greater speed and power. This matches the mobility and manipulation goals we have for Nadia!

The least dramatic but possibly most important bit of that video is when Nadia uses her arms to help her balance against a wall, which is one of those things that humans do all the time without thinking about it. And we always appreciate being shown things that don’t go perfectly alongside things that do. The bit at the end there was Nadia not quite managing to do lateral arm raises. I can relate; that’s my reaction when I lift weights, too.

[ IHMC ]

Thanks, Robert!

The recent progress in commercial humanoids is just exhausting.

[ Unitree ]

We present an avatar system designed to facilitate the embodiment of humanoid robots by human operators, validated through iCub3, a humanoid developed at the Istituto Italiano di Tecnologia.

[ Science Robotics ]

Have you ever seen a robot skiing?! Ascento robot enjoying a day in the ski slopes of Davos.

[ Ascento ]

Can’t trip Atlas up! Our humanoid robot gets ready for real work combining strength, perception, and mobility.

Notable that Boston Dynamics is now saying that Atlas “gets ready for real work.” Wonder how much to read into that?

[ Boston Dynamics ]

You deserve to be free from endless chores! YOU! DESERVE! CHORE! FREEDOM!

Pretty sure this is teleoperated, so someone is still doing the chores, sadly.

[ MagicLab ]

Multimodal UAVs (Unmanned Aerial Vehicles) are rarely capable of more than two modalities, i.e., flying and walking or flying and perching. However, being able to fly, perch, and walk could further improve their usefulness by expanding their operating envelope. For instance, an aerial robot could fly a long distance, perch in a high place to survey the surroundings, then walk to avoid obstacles that could potentially inhibit flight. Birds are capable of these three tasks, and so offer a practical example of how a robot might be developed to do the same.

[ Paper ] via [ EPFL LIS ]

Nissan announces the concept model of “Iruyo”, a robot that supports babysitting while driving. Ilyo relieves the anxiety of the mother, father, and baby in the driver’s seat. We support safe and secure driving for parents and children. Nissan and Akachan Honpo are working on a project to make life better with cars and babies. Iruyo was born out of the voices of mothers and fathers who said, “I can’t hold my baby while driving alone.”

[ Nissan ]

Building 937 houses the coolest robots at CERN. This is where the action happens to build and program robots that can tackle the unconventional challenges presented by the Laboratory’s unique facilities. Recently, a new type of robot called CERNquadbot has entered CERN’s robot pool and successfully completed its first radiation protection test in the North Area.

[ CERN ]

Congrats to Starship, the OG robotic delivery service, on their $90m raise.

[ Starship ]

By blending 2D images with foundation models to build 3D feature fields, a new MIT method helps robots understand and manipulate nearby objects with open-ended language prompts.

[ GitHub ] via [ MIT ]

This is one of those things that’s far more difficult than it might look.

[ ROAM Lab ]

Our current care system does not scale and our populations are ageing fast. Robodies are multipliers for care staff, allowing them to work together with local helpers to provide protection and assistance around the clock while maintaining personal contact with people in the community.

[ DEVANTHRO ]

It’s the world’s smallest humanoid robot, until someone comes out with slightly smaller servos!

[ Guinness ]

Deep Robotics wishes you a happy year of the dragon!

[ Deep Robotics ]

SEAS researchers are helping develop resilient and autonomous deep space and extraterrestrial habitations by developing technologies to let autonomous robots repair or replace damaged components in a habitat. The research is part of the Resilient ExtraTerrestrial Habitats institute (RETHi) is led by Purdue University, in partnership with SEAS, the University of Connecticut and the University of Texas at San Antonio. Its goal is to “design and operate resilient deep space habitats that can adapt, absorb and rapidly recover from expected and unexpected disruptions.”

[ Harvard ]

Find out how a bold vision became a success story! The DLR Institute of Robotics and Mechatronics has been researching robotic arms since the 1990s - originally for use in space. It was a long and ambitious journey before these lightweight robotic arms could be used on earth and finally in operating theaters, a journey that required concentrated robotics expertise, interdisciplinary cooperation and ultimately a successful technology transfer.]

[ DLR MIRO ]

Robotics is changing the world, driven by focused teams of diverse experts. Willow Garage operated with the mantra “Impact first, return on capital second” and through ROS and the PR2 had enormous impact. Autonomous mobile robots are finally being accepted in the service industry, and Savioke (now Relay Robotics) was created to drive that impact. This talk will trace the evolution of Relay robots and their deployment in hotels, hospitals and other service industries, starting with roots at Willow Garage. As robotics technology is poised for the next round of advances, how do we create and maintain the organizations that continue to drive progress?

[ Northwestern ]



It’s kind of astonishing how quadrotors have scaled over the past decade. Like, we’re now at the point where they’re verging on disposable, at least from a commercial or research perspective—for a bit over US $200, you can buy a little 27-gram, completely open-source drone, and all you have to do is teach it to fly. That’s where things do get a bit more challenging, though, because teaching drones to fly is not a straightforward process. Thanks to good simulation and techniques like reinforcement learning, it’s much easier to imbue drones with autonomy than it used to be. But it’s not typically a fast process, and it can be finicky to make a smooth transition from simulation to reality.

New York University’s Agile Robotics and Perception Lab has managed to streamline the process of getting basic autonomy to work on drones, and streamline it by a lot: The lab’s system is able to train a drone in simulation from nothing up to stable and controllable flying in 18 seconds flat on a MacBook Pro. And it actually takes longer to compile and flash the firmware onto the drone itself than it does for the entire training process.

ARPL NYU

So not only is the drone able to keep a stable hover while rejecting pokes and nudges and wind, but it’s also able to fly specific trajectories. Not bad for 18 seconds, right?

One of the things that typically slows down training times is the need to keep refining exactly what you’re training for, without refining it so much that you’re only training your system to fly in your specific simulation rather than the real world. The strategy used here is what the researchers call a curriculum (you can also think of it as a sort of lesson plan) to adjust the reward function used to train the system through reinforcement learning. The curriculum starts things off being more forgiving and gradually increasing the penalties to emphasize robustness and reliability. This is all about efficiency: Doing that training that you need to do in the way that it needs to be done to get the results you want, and no more.

There are other, more straightforward, tricks that optimize this technique for speed as well. The deep-reinforcement learning algorithms are particularly efficient, and leverage the hardware acceleration that comes along with Apple’s M-series processors. The simulator efficiency multiplies the benefits of the curriculum-driven sample efficiency of the reinforcement-learning pipeline, leading to that wicked-fast training time.

This approach isn’t limited to simple tiny drones—it’ll work on pretty much any drone, including bigger and more expensive ones, or even a drone that you yourself build from scratch.

Jonas Eschmann

We’re told that it took minutes rather than seconds to train a policy for the drone in the video above, although the researchers expect that 18 seconds is achievable even for a more complex drone like this in the near future. And it’s all open source, so you can, in fact, build a drone and teach it to fly with this system. But if you wait a little bit, it’s only going to get better: The researchers tell us that they’re working on integrating with the PX4 open source drone autopilot. Longer term, the idea is to have a single policy that can adapt to different environmental conditions, as well as different vehicle configurations, meaning that this could work on all kinds of flying robots rather than just quadrotors.

Everything you need to run this yourself is available on GitHub, and the paper is on ArXiv here.



About a decade ago, there was a lot of excitement in the robotics world around gecko-inspired directional adhesives, which are materials that stick without being sticky using the same van der Waals forces that allow geckos to scamper around on vertical panes of glass. They were used extensively in different sorts of climbing robots, some of them quite lovely. Gecko adhesives are uniquely able to stick to very smooth things where your only other option might be suction, which requires all kinds of extra infrastructure to work.

We haven’t seen gecko adhesives around as much of late, for a couple of reasons. First, the ability to only stick to smooth surfaces (which is what gecko adhesives are best at) is a bit of a limitation for mobile robots. And second, the gap between research and useful application is wide and deep and full of crocodiles. I’m talking about the mean kind of crocodiles, not the cuddly kind. But Flexiv Robotics has made gecko adhesives practical for robotic grasping in a commercial environment, thanks in part to a sort of robotic tongue that licks the gecko tape clean.

If you zoom way, way in on a gecko’s foot, you’ll see that each toe is covered in millions of hair-like nanostructures called setae. Each setae branches out at the end into hundreds of more hairs with flat bits at the end called spatulas. The result of this complex arrangement of setae and spatulas is that gecko toes have a ridiculous amount of surface area, meaning that they can leverage the extremely weak van der Waals forces between molecules to stick themselves to perfectly flat and smooth surfaces. This technique works exceptionally well: Geckos can hang from glass by a single toe, and a fully adhered gecko can hold something like 140 kg (which, unfortunately, seems to be an extrapolation rather than an experimental result). And luckily for the gecko, the structure of the spatulas makes the adhesion directional, so that when its toes are no longer being loaded, they can be easily peeled off of whatever they’re attached to.

Natural gecko adhesive structure, along with a synthetic adhesive (f).Gecko adhesion: evolutionary nanotechnology, by Kellar Autumn and Nick Gravish

Since geckos don’t “stick” to things in the sense that we typically use the word “sticky,” a better way of characterizing what geckos can do is as “dry adhesion,” as opposed to something that involves some sort of glue. You can also think about gecko toes as just being very, very high friction, and it’s this perspective that is particularly interesting in the context of robotic grippers.

This is Flexiv’s “Grav Enhanced” gripper, which uses a combination of pinch grasping and high friction gecko adhesive to lift heavy and delicate objects without having to squeeze them. When you think about a traditional robotic grasping system trying to lift something like a water balloon, you have to squeeze that balloon until the friction between the side of the gripper and the side of the balloon overcomes the weight of the balloon itself. The higher the friction, the lower the squeeze required, and although a water balloon might be an extreme example, maximizing gripper friction can make a huge difference when it comes to fragile or deformable objects.

There are a couple of problems with dry adhesive, however. The tiny structures that make the adhesive adhere can be prone to damage, and the fact that dry adhesive will stick to just about anything it can make good contact with means that it’ll rapidly accumulate dirt outside of a carefully controlled environment. In research contexts, these problems aren’t all that significant, but for a commercial system, you can’t have something that requires constant attention.

Flexiv says that the microstructure material that makes up their gecko adhesive was able to sustain two million gripping cycles without any visible degradation in performance, suggesting that as long as you use the stuff within the tolerances that it’s designed for, it should keep on adhering to things indefinitely—although trying to lift too much weight will tear the microstructures, ruining the adhesive properties after just a few cycles. And to keep the adhesive from getting clogged up with debris, Flexiv came up with this clever little cleaning station that acts like a little robotic tongue of sorts:

Interestingly, geckos themselves don’t seem to use their own tongues to clean their toes. They lick their eyeballs on the regular, like all normal humans do, but gecko toes appear to be self-cleaning, which is a pretty neat trick. It’s certainly possible to make self-cleaning synthetic gecko adhesive, but Flexiv tells us that “due to technical and practical limitations, replicating this process in our own gecko adhesive material is not possible. Essentially, we replicate the microstructure of a gecko’s footpad, but not its self-cleaning process.” This likely goes back to that whole thing about what works in a research context versus what works in a commercial context, and Flexiv needs their gecko adhesive to handle all those millions of cycles.

Flexiv says that they were made aware of the need for a system like this when one of their clients started using the gripper for the extra-dirty task of sorting trash from recycling, and that the solution was inspired by a lint roller. And I have to say, I appreciate the simplicity of the system that Flexiv came up with to solve the problem directly and efficiently. Maybe one day, they’ll be able to replicate a real gecko’s natural self-cleaning toes with a durable and affordable artificial dry adhesive, but until that happens, an artificial tongue does the trick.


Video Friday is your weekly selection of awesome robotics videos, collected by your friends at IEEE Spectrum robotics. We also post a weekly calendar of upcoming robotics events for the next few months. Please send us your events for inclusion.

Cybathlon Challenges: 2 February 2024, ZURICHEurobot Open 2024: 8–11 May 2024, LA ROCHE-SUR-YON, FRANCEICRA 2024: 13–17 May 2024, YOKOHAMA, JAPAN

Enjoy today’s videos!

Is “scamperiest” a word? If not, it should be, because this is the scamperiest robot I’ve ever seen.

[ ABS ]

GITAI is pleased to announce that its 1.5-meter-long autonomous dual robotic arm system (S2) has successfully arrived at the International Space Station (ISS) aboard the SpaceX Falcon 9 rocket (NG-20) to conduct an external demonstration of in-space servicing, assembly, and manufacturing (ISAM) while onboard the ISS. The success of the S2 tech demo will be a major milestone for GITAI, confirming the feasibility of this technology as a fully operational system in space.

[ GITAI ]

This work presents a comprehensive study on using deep reinforcement learning (RL) to create dynamic locomotion controllers for bipedal robots. Going beyond focusing on a single locomotion skill, we develop a general control solution that can be used for a range of dynamic bipedal skills, from periodic walking and running to aperiodic jumping and standing.

And if you want to get exhausted on behalf of a robot, the full 400-meter dash is below.

[ Hybrid Robotics ]

NASA’s Ingenuity Mars Helicopter pushed aerodynamic limits during the final months of its mission, setting new records for speed, distance, and altitude. Hear from Ingenuity chief engineer Travis Brown on how the data the team collected could eventually be used in future rotorcraft designs.

[ NASA ]

BigDog: 15 years of solving mobility problems its own way.

[ Boston Dynamics ]

[Harvard School of Engineering and Applied Sciences] researchers are helping develop resilient and autonomous deep space and extraterrestrial habitations by developing technologies to let autonomous robots repair or replace damaged components in a habitat. The research is part of the Resilient ExtraTerrestrial Habitats institute (RETHi) led by Purdue University, in partnership with [Harvard] SEAS, the University of Connecticut and the University of Texas at San Antonio. Its goal is to “design and operate resilient deep space habitats that can adapt, absorb and rapidly recover from expected and unexpected disruptions.”

[ Harvard SEAS ]

Researchers from Huazhong University of Science and Technology (HUST) in a recent T-RO paper describe and construct a novel variable stiffness spherical joint motor that enables dexterous motion and joint compliance in omni-directions.

[ Paper ]

Thanks, Ram!

We are told that this new robot from HEBI is called “Mark Suckerberg” and that they’ve got a pretty cool application in mind for it, to be revealed later this year.

[ HEBI Robotics ]

Thanks, Dave!

Dive into the first edition of our new Real-World-Robotics class at ETH Zürich! Our students embarked on an incredible journey, creating their human-like robotic hands from scratch. In just three months, the teams designed, built, and programmed their tendon-driven robotic hands, mastering dexterous manipulation with reinforcement learning! The result? A spectacular display of innovation and skill during our grand final.

[ SRL ETHZ ]

Carnegie Mellon researchers have built a system with a robotic arm atop a RangerMini 2.0 robotic cart from AgileX robotics to make what they’re calling a platform for “intelligent movement and processing.”

[ CMU ] via [ AgileX ]

Picassnake is our custom-made robot that paints pictures from music. Picassnake consists of an arm and a head, embedded in a plush snake doll. The robot is connected to a laptop for control and music processing, which can be fed through a microphone or an MP3 file. To open the media source, an operator can use the graphical user interface or place a text QR code in front of a webcam. Once the media source is opened, Picassnake generates unique strokes based on the music and translates the strokes to physical movement to paint them on canvas.

[ Picassnake ]

In April 2021, NASA’s Ingenuity Mars Helicopter became the first spacecraft to achieve powered, controlled flight on another world. With 72 successful flights, Ingenuity has far surpassed its originally planned technology demonstration of up to five flights. On Jan. 18, Ingenuity flew for the final time on the Red Planet. Join Tiffany Morgan, NASA’s Mars Exploration Program Deputy Director, and Teddy Tzanetos, Ingenuity Project Manager, as they discuss these historic flights and what they could mean for future extraterrestrial aerial exploration.

[ NASA ]

Pages