IEEE Spectrum Automation

IEEE Spectrum Automaton blog recent content
Subscribe to IEEE Spectrum Automation feed
aside.inlay.CoronaVirusCoverage.xlrg { font-family: "Helvetica", sans-serif; text-transform: uppercase; text-align: center; border-width: 4px 0; border-top: 2px solid #666; border-bottom: 2px solid #666; padding: 10px 0; font-size: 18px; font-weight: bold; } span.LinkHereRed { color: #cc0000; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; }

A year ago, we visited Rwanda to see how Zipline’s autonomous, fixed-wing delivery drones were providing blood to hospitals and clinics across the country. We were impressed with both Zipline’s system design (involving dramatic catapult launches, parachute drops, and mid-air drone catching), as well as their model of operations, which minimizes waste while making critical supplies available in minutes almost anywhere in the country.

Since then, Zipline has expanded into Ghana, and has plans to start flying in India as well, but the COVID-19 pandemic is changing everything. Africa is preparing for the worst, while in the United States, Zipline is working with the Federal Aviation Administration to try and expedite safety and regulatory approvals for an emergency humanitarian mission with the goal of launching a medical supply delivery network that could help people maintain social distancing or quarantine when necessary by delivering urgent medication nearly to their doorsteps.

In addition to its existing role delivering blood products and medication, Zipline is acting as a centralized distribution network for COVID-19 supplies in Ghana and Rwanda. Things like personal protective equipment (PPE) will be delivered as needed by drone, ensuring that demand is met across the entire healthcare network. This has been a problem in the United States—getting existing supplies where they’re needed takes a lot of organization and coordination, which the US government is finding to be a challenge.

Photo: Zipline

Zipline says that their drones are able to reduce human involvement in the supply chain (a vector for infection), while reducing hospital overcrowding by making it more practical for non-urgent patients to receive care in local clinics closer to home. COVID-19 is also having indirect effects on healthcare, with social distancing and community lockdowns straining blood supplies. With its centralized distribution model, Zipline has helped Rwanda to essentially eliminate wasted (expired) blood products. “We probably waste more blood [in the United States] than is used in all of Rwanda,” Zipline CEO Keller Rinaudo told us. But it’s going to take more than blood supply to fight COVID-19, and it may hit Africa particularly hard.

Click here for additional coronavirus coverage

“Things are earlier in Africa, you don’t see infections at the scale that we’re seeing in the U.S.,” says Rinaudo. “I also think Africa is responding much faster. Part of that is the benefit of seeing what’s happening in countries that didn’t take it seriously in the first few months where community spreading gets completely out of control. But it’s quite possible that COVID is going to be much more severe in countries that are less capable of locking down, where you have densely populated areas with people who can’t just stay in their house for 45 days.” 

In an attempt to prepare for things getting worse, Rinaudo says that Zipline is stocking as many COVID-related products as possible, and they’re also looking at whether they’ll be able to deliver to neighborhood drop-off points, or perhaps directly to homes. “That’s something that Zipline has been on track to do for quite some time, and we’re considering ways of accelerating that. When everyone’s staying at home, that’s the ideal time for robots to be making deliveries in a contactless way.” This kind of system, Rinaudo points out, would also benefit people with non-COVID healthcare needs, who need to do their best to avoid hospitals. If a combination of telemedicine and home or neighborhood delivery of medical supplies means they can stay home, it would be a benefit for everyone. “This is a transformation of the healthcare system that’s already happening and needs to happen anyway. COVID is just accelerating it.”

“When everyone’s staying at home, that’s the ideal time for robots to be making deliveries in a contactless way” —Keller Rinaudo, Zipline

For the past year, Zipline, working closely with the FAA, has been planning on a localized commercial trial of a medical drone delivery service that was scheduled to begin in North Carolina this fall. While COVID is more urgent, the work that’s already been done towards this trial puts Zipline in a good position to move quickly, says Rinaudo.

“All of the work that we did with the IPP [UAS Integration Pilot Program] is even more important, given this crisis. It means that we’ve already been working with the FAA in detail, and that’s made it possible for us to have a foundation to build on to help with the COVID-19 response.” Assuming that Zipline and the FAA can find a regulatory path forward, the company could begin setting up distribution centers that can support hospital networks for both interfacility delivery as well as contactless delivery to (eventually) neighborhood points and perhaps even homes. “It’s exactly the use case and value proposition that I was describing for Africa,” Rinaudo says.

Leveraging rapid deployment experience that it has from work with the U.S. Department of Defense, Zipline would launch one distribution center within just a few months of a go-ahead from the FAA. This single distribution center could cover an area representing up to 10 million people. “We definitely want to move quickly here,” Rinaudo tells us. Within 18 months, Zipline could theoretically cover the entire US, although he admits “that would be an insanely fast roll-out.”

The question, at this point, is how fast the FAA can take action to make innovative projects like this happen. Zipline, as far as we can tell, is ready to go. We did also ask Rinaudo if he thought that hospitals specifically, and the medical system in general, has the bandwidth to adopt a system like Zipline’s in the middle of a pandemic that’s already stretching people and resources to the limit.

“In the U.S. there’s this sense that this technology is impossible, whereas it’s already operating at multi-national scale, serving thousands of hospitals and health facilities, and it’s completely boring to the people who are benefiting from it,” Rinaudo says. “People in the U.S. have really not caught on that this is something that’s reliable and can dramatically improve our response to crises like this.”

[ Zipline ]

aside.inlay.CoronaVirusCoverage.xlrg { font-family: "Helvetica", sans-serif; text-transform: uppercase; text-align: center; border-width: 4px 0; border-top: 2px solid #666; border-bottom: 2px solid #666; padding: 10px 0; font-size: 18px; font-weight: bold; } span.LinkHereRed { color: #cc0000; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; }

For the past two months, the vegetables have arrived on the back of a robot. That’s how 16 communities in Zibo, in eastern China, have received fresh produce during the coronavirus pandemic. The robot is an autonomous van that uses lidars, cameras, and deep-learning algorithms to drive itself, carrying up to 1,000 kilograms on its cargo compartment.

The unmanned vehicle provides a “contactless” alternative to regular deliveries, helping reduce the risk of person-to-person infection, says Professor Ming Liu, a computer scientist at the Hong Kong University of Science and Technology (HKUST) and cofounder of Unity Drive Innovation, or UDI, the Shenzhen-based startup that developed the self-driving van.

Since February, UDI has been operating a small fleet of vehicles in Zibo and two other cities, Suzhou and Shenzhen, where they deliver meal boxes to checkpoint workers and spray disinfectant near hospitals. Combined, the vans have made more than 2,500 autonomous trips, often encountering busy traffic conditions despite the lockdown.

“It’s like Uber for packages—you use your phone to call a robot to pick up and deliver your boxes,” Professor Liu told IEEE Spectrum in an interview via Zoom.

Even before the pandemic, package shipments had been skyrocketing in China and elsewhere. Alibaba founder Jack Ma has said that his company is preparing to handle 1 billion packages per day. With the logistics sector facing major labor shortages, a 2016 McKinsey report predicted that autonomous vehicles will deliver 80 percent of parcels within 10 years.

That’s the future UDI is betting on. Unlike robocars developed by Waymo, Cruise, Zoox, and others, UDI’s vehicles are designed to transport goods, not people. They are similar to those of Nuro, a Silicon Valley startup, and Neolix, based in Beijing, which has deployed 50 robot vans in 10 Chinese cities to do mobile delivery and disinfection service.

Photo: UDI A self-driving vehicle delivers lunch boxes to workers in Pingshan District in Shenzhen. Since February, UDI’s autonomous fleet has made more than 800 meal deliveries.

Professor Liu, an IEEE Senior Member and director of the Intelligent Autonomous Driving Center at HKUST, is unfazed by the competition. He says UDI is ready to operate its vehicles on public roads thanks to the real-world experience it has gained from a string of recent projects. These involve large companies testing the robot vans inside their industrial parks.

One of them is Taiwanese electronics giant Foxconn. Since late 2018, it has used UDI vans to transport electronic parts and other items within its vast Shenzhen campus where some 200,000 workers reside. The robots have to navigate labyrinthine routes while avoiding an unpredictable mass of pedestrians, bicycles, and trucks.

Autonomous driving powered by deep learning

UDI’s vehicle, called Hercules, uses an industrial-grade PC running the Robot Operating System, or ROS. It’s also equipped with a drive-by-wire chassis with electric motors powered by a 8.4-kWh lithium-ion battery. Sensors include a main lidar, three auxiliary lidars, a stereo camera, four fisheye cameras, 16 sonars, redundant satellite navigation systems, an inertial measurement unit (IMU), and two wheel encoders.

The PC receives the lidar point-clouds and feeds them into the main perception algorithm, which consists of a convolutional neural network trained to detect and classify objects. The neural net outputs a set of 3D bounding boxes representing vehicles and other obstacles on the road. This process repeats 100 times per second.

Image: UDI UDI’s vehicle is equipped with a main lidar and three auxiliary lidars, a stereo camera, and various other sensors [top]. The cargo compartment can be modified based on the items to be transported and is not shown. The chassis [bottom] includes an electric motor, removable lithium-ion battery, vehicle control unit (VCU), motor control unit (MCU), electric power steering (EPS), electro-hydraulic brake (EHB), electronic parking brake (EPB), on-board charger (OBC), and direct-current-to-direct-current (DCDC) converter.

Another algorithm processes images from forward-facing cameras to identify road signs and traffic lights, and a third matches the point-clouds and IMU data to a global map, allowing the vehicle to self-localize. To accelerate, brake, and steer, the PC sends commands to two secondary computers running real-time operating systems and connected to the drive-by-wire modules.

Professor Liu says UDI faces more challenging driving conditions than competitors like Waymo and Nuro that conduct their tests in suburban areas in the United States. In Shenzhen, for example, the UDI vans have to navigate through narrow streets with double parked cars and aggressive motorcycles that whiz by narrowly missing the robot.

Click here for additional coronavirus coverage

Over the past couple of months, UDI has monitored its fleet from its headquarters. Using 5G, a remote operator can receive data from a vehicle with just 10 milliseconds of delay. In Shenzhen, human intervention was required about two dozen times when the robots encountered situations they didn’t know how to handle—too many vehicles on the road, false detections of traffic lights at night, or in one case, a worker coming out of a manhole.

Photo: UDI One of UDI’s autonomous vehicles equipped with a device that sprays disinfectant operates near a hospital in Shenzhen.

For safety, UDI programmed the vans to drive at low speeds of up to 30 kilometers per hour, though they can go faster. On a few occasions, remote operators took control because the vehicles were driving too slowly, becoming a road hazard and annoying nearby drivers. Professor Liu says it’s a challenge to balance cautiousness and aggressiveness in self-driving vehicles that will operate in the real world.

He notes that UDI vehicles have been collecting huge amounts of video and sensor data during their autonomous runs. This information will be useful to improve computer simulations of the vehicles and, later, the real vehicles themselves. UDI says it plans to open source part of the data.

Mass produced robot vans

Professor Liu has been working on advanced vehicles for nearly two decades. His projects include robotic cars, buses, and boats, with a focus on applying deep reinforcement learning to enable autonomous behaviors. He says UDI’s vehicles are not cars, and they aren’t unmanned ground robots, either—they are something in between. He likes to call them “running robots.”

Liu’s cofounders are Professor Xiaorui Zhu at Harbin Institute of Technology, in Shenzhen, and Professor Lujia Wang at the Shenzhen Institutes of Advanced Technology, part of the Chinese Academy of Sciences. “We want to be the first company in the world to achieve mass production of autonomous logistics vehicles,” says Wang, who is the CTO of UDI.

To do that, the startup has hired 100 employees and is preparing to put its assembly line into high gear in the next several months. “I’m not saying we solved all the problems,” Professor Liu says, citing system integration and cost as the biggest challenges. “Can we do better? Yes, it can always be better.”

aside.inlay.CoronaVirusCoverage.xlrg { font-family: "Helvetica", sans-serif; text-transform: uppercase; text-align: center; border-width: 4px 0; border-top: 2px solid #666; border-bottom: 2px solid #666; padding: 10px 0; font-size: 18px; font-weight: bold; } span.LinkHereRed { color: #cc0000; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; }

For the last several years, Diligent Robotics has been testing out its robot, Moxi, in hospitals in Texas. Diligent isn’t the only company working on hospital robots, but Moxi is unique in that it’s doing commercial mobile manipulation, picking supplies out of supply closets and delivering them to patient rooms, all completely autonomously.

A few weeks ago, Diligent announced US $10 million in new funding, which comes at a critical time, as the company addressed in their press release:

Now more than ever hospitals are under enormous stress, and the people bearing the most risk in this pandemic are the nurses and clinicians at the frontlines of patient care. Our mission with Moxi has always been focused on relieving tasks from nurses, giving them more time to focus on patients, and today that mission has a newfound meaning and purpose. Time and again, we hear from our hospital partners that Moxi not only returns time back to their day but also brings a smile to their face.  

We checked in with Diligent CEO Andrea Thomaz last week to get a better sense of how Moxi is being used at hospitals. “As our hospital customers are implementing new protocols to respond to the [COVID-19] crisis, we are working with them to identify the best ways for Moxi to be deployed as a resource,” Thomaz told us. “The same kinds of delivery tasks we have been doing are still just as needed as ever, but we are also working with them to identify use cases where having Moxi do a delivery task also reduces infection risk to people in the environment.”

Click here for additional coronavirus coverage

Since this is still something that Diligent and their hospital customers are actively working on, it’s a little early for them to share details. But in general, robots making deliveries means that people aren’t making deliveries, which has several immediate benefits. First, it means that overworked hospital staff can spend their time doing other things (like interacting with patients), and second, the robot is less likely to infect other people. It’s not just that the robot can’t get a virus (not that kind of virus, at any rate), but it’s also much easier to keep robots clean in ways that aren’t an option for humans. Besides wiping them down with chemicals, without too much trouble you could also have them autonomously disinfect themselves with UV, which is both efficient and effective.

While COVID-19 only emphasizes the importance of robots in healthcare, Diligent is tackling a particularly difficult set of problems with Moxi, involving full autonomy, manipulation, and human-robot interaction. Earlier this year, we spoke with Thomaz about how Moxi is starting to make a difference to hospital staff.

IEEE Spectrum: Last time we talked, Moxi was in beta testing. What’s different about Moxi now that it’s ready for full-time deployment?

Andrew Thomaz: During our beta trial, Moxi was deployed for over 120 days total, in four different hospitals (one of them was a children’s hospital, the other three were adult acute-care units), working alongside more than 125 nurses and clinicians. The people we were working with were so excited to be part of this kind of innovative research, and how this new technology is going to actually impact workloads. Our focus on the beta trials was to try any idea that a customer had of how Moxi could provide value—if it seemed at all reasonable, then we would quickly try to mock something up and try it.

I think it validates our human-robot interaction approach to building the company, of getting the technology out there in front of customers to make sure that we’re building the product that they really need. We started to see common workflows across hospitals—there are different kinds of patient care that’s happening, but the kinds of support and supplies and things that are moving around the hospital are similar—and so then we felt that we had learned what we needed to learn from the beta trial and we were ready to launch with our first customers.

Photo: Diligent Robotics

The primary function that Moxi has right now, of restocking and delivery, was that there from the beginning? Or was that something that people asked for and you realized, oh, well, this is how a robot can actually be the most useful.

We knew from the beginning that our goal was to provide the kind of operational support that an end-to-end mobile manipulation platform can do, where you can go somewhere autonomously, pick something up, and bring it to another location and put it down. With each of our beta customers, we were very focused on opportunities where that was the case, where nurses were wasting time.

We did a lot of that kind of discovery, and then you just start seeing that it’s not rocket science—there are central supply places where things are kept around the hospital, and nurses are running back and forth to these places multiple times a day. We’d look at some particular task like admission buckets, or something else that nurses have to do everyday, and then we say, where are the places that automation can really fit in? Some of that support is just navigation tasks, like going from one place to another, some actually involves manipulation, like you need to press this button or you need to pick up this thing. But with Moxi, we have a mobility and a manipulation component that we can put to work, to redefine workflows to include automation.

You mentioned that as part of the beta program that you were mocking the robot up to try all kinds of customer ideas. Was there something that hospitals really wanted the robot to do, that you mocked up and tried but just didn’t work at all?

We were pretty good at not setting ourselves up for failure. I think the biggest thing would be, if there was something that was going to be too heavy for the Kinova arm, or the Robotiq gripper, that’s something we just can’t do right now. But honestly, it was a pretty small percentage of things that we were kind of asked to manipulate that we had to say, oh no, sorry, we can’t lift that much or we can’t grip that wide. The other reason that things that we tried in the beta didn’t make it into our roadmap is if there was an idea that came up with only one of the beta sites. One example is delivering water: One of the beta sites was super excited about having water delivered to the patients every day, ahead of medication deliveries, which makes a lot of sense, but when we start talking to hospital leadership or other people, in other hospitals, it’s definitely just a “nice to have.” So for us, from a technical standpoint, it doesn’t make as much sense to devote a lot of resources into making water delivery a real task if it’s just going to be kind of a “nice to have” for a small percentage of our hospitals. That’s more how that R&D went—if we heard it from one hospital we’d ask, is this something that everybody wants, or just an idea that one person had. 

Let’s talk about how Moxi does what it does. How does the picking process work?

We’re focused on very structured manipulation; we’re not doing general purpose manipulation, and so we have a process for teaching Moxi a particular supply room. There are visual cues that are used to orient the robot to that supply room, and then once you are oriented you know where a bin is. Things don’t really move around a great deal in the supply room, the bigger variability is just how full each of the bins are.

The things that the robot is picking out of the bins are very well known, and we make sure that hospitals have a drop off location outside the patient’s room. In about half the hospitals we were in, they already had a drawer where the robot could bring supplies, but sometimes they didn’t have anything, and then we would install something like a mailbox on the wall. That’s something that we’re still working out exactly—it was definitely a prototype for the beta trials, and we’re working out how much that’s going to be needed in our future roll out.

“A robot needs to do something functional, be a utility, and provide value, but also be socially acceptable and something that people want to have around” —Andrea Thomaz, Diligent Robotics

These aren’t supply rooms that are dedicated to the robot—they’re also used by humans who may move things around unpredictably. How does Moxi deal with the added uncertainty?

That’s really the entire focus of our human-guided learning approach—having the robot build manipulation skills with perceptual cues that are telling it about different anchor points to do that manipulation skill with respect to, and learning particular grasp strategies for a particular category of objects. Those kinds of strategies are going to make that grasp into that bin more successful, and then also learning the sensory feedback that’s expected on a successful grasp versus an unsuccessful one, so that you have the ability to retry until you get the expected sensory feedback.

There must also be plenty of uncertainty when Moxi is navigating around the hospital, which is probably full of people who’ve never seen it before and want to interact with it. To what extent is Moxi designed for those kinds of interactions? And if Moxi needs to be somewhere because it has a job to do, how do you mitigate or avoid them?

One of the things that we liked about hospitals as a semi-structured environment is that even the human interaction that you’re going to run into is structured as well, more so than somewhere like a shopping mall. In a hospital you have a kind of idea of the kind of people that are going to be interacting with the robot, and you can have some expectations about who they are and why they’re there and things, so that’s nice.

We had gone into the beta trial thinking, okay, we’re not doing any patient care, we’re not going into patients’ rooms, we’re bringing things to right outside the patient rooms, we’re mostly going to be interacting with nurses and staff and doctors. We had developed a lot of the social capabilities, little things that Moxi would do with the eyes or little sounds that would be made occasionally, really thinking about nurses and doctors that were going to be in the hallways interacting with Moxi. Within the first couple weeks at the first beta site, the patients and general public in the hospital were having so many more interactions with the robot than we expected. There were people who were, like, grandma is in the hospital, so the entire family comes over on the weekend, to see the robot that happens to be on grandma’s unit, and stuff like that. It was fascinating.

We always knew that being socially acceptable and fitting into the social fabric of the team was important to focus on. A robot needs to have both sides of that coin—it needs to do something functional, be a utility, and provide value, but also be socially acceptable and something that people want to have around. But in the first couple weeks in our first beta trial, we quickly had to ramp up and say, okay, what else can Moxi do to be social? We had the robot, instead of just going to the charger in between tasks, taking an extra social lap to see if there’s anybody that wants to take a selfie. We added different kinds of hot word detections, like for when people say “hi Moxi,” “good morning, Moxi,” or “how are you?” Just all these things that people were saying to the robot that we wanted to turn into fun interactions.

I would guess that this could sometimes be a little problematic, especially at a children’s hospital where you’re getting lots of new people coming in who haven’t seen a robot before—people really want to interact with robots and that’s independent of whether or not the robot has something else it’s trying to do. How much of a problem is that for Moxi?

That’s on our technical roadmap. We still have to figure out socially appropriate ways to disengage. But what we did learn in our beta trials is that there are even just different navigation paths that you can take, by understanding where crowds tend to be at different times. Like, maybe don’t take a path right by the cafeteria at noon, instead take the back hallway at noon. There are always different ways to get to where you’re going. Houston was a great example—in that hospital, there was this one skyway where you knew the robot was going to get held up for 10 or 15 minutes taking selfies with people, but there was another hallway two floors down that was always empty. So you can kind of optimize navigation time for the number of selfies expected, things like that.

Photo: Diligent Robotics

To what extent is the visual design of Moxi intended to give people a sense of what its capabilities are, or aren’t?

For us, it started with the functional things that Moxie needs. We knew that we’re doing mobile manipulation, so we’d need a base, and we’d need an arm. And we knew we also wanted it to have a social presence, and so from those constraints, we worked with our amazing head of design, Carla Diana, on the look and feel of the robot. For this iteration, we wanted to make sure it didn’t have an overly humanoid look.

Some of the previous platforms that I used in academia, like the Simon robot or the Curie robot, had very realistic eyes. But when you start to talk about taking that to a commercial setting, now you have these eyeballs and eyelids and each of those is a motor that has to work every day all day long, so we realized that you can get a lot out of some simplified LED eyes, and it’s actually endearing to people to have this kind of simplified version of it. The eyes are a big component—that’s always been a big thing for me because of the importance of attention, and being able to communicate to people what the robot is paying attention to. Even if you don’t put eyeballs on a robot, people will find a thing to attribute attention to: They’ll find the camera and say, “oh, those are its eyes!” So I find it’s better to give the robot a socially expressive focus of attention.

I would say speech is the biggest one that we have drawn the line on. We want to make sure people don’t get the sense that Moxi can understand the full English language, because I think people are getting to be really used to speech interfaces, and we don’t have an Alexa or anything like that integrated yet. That could happen in the future, but we don’t have a real need for that right now, so it’s not there, so we want to make sure people don’t think of the robot as an Alexa or a Google Home or a Siri that you can just talk to, so we make sure that it just does beeps and whistles, and then that kind of makes sense to people. So they get that you can say stuff like “hi Moxi,” but that’s about it. 

Otherwise, I think the design is really meant to be socially acceptable, we want to make sure people are comfortable, because like you’re saying, this is a robot that a lot of people are going to see for the first time, and we have to be really sensitive to the fact that the hospital is a stressful place for a lot of people, you’re already there with a sick family member and you might have a lot going on, and we want to make sure that we aren’t contributing to additional stress in your day.

You mentioned that you have a vision for human-robot teaming. Longer term, how do you feel like people should be partnering more directly with robots?

Right now, we’re really focused on looking at operational processes that hit two or three different departments in the hospital and require a nurse to do this and a patient care technician to do that and a pharmacy or a materials supply person to do something else. We’re working with hospitals to understand how that whole team of people is making some big operational workflow happen and where Moxi could fit in. 

Some places where Moxi fits in, it’s a completely independent task. Other places, it might be a nurse on a unit calling Moxi over to do something, and so there might be a more direct interaction sometimes. Other times it might be that we’re able to connect to the electronic health record and infer automatically that something’s needed and then it really is just happening more in the background. We’re definitely open to both explicit interaction with the team where Moxi’s being called to do something in particular by someone, but I think some of the more powerful examples from our beta trials were ones that really take that cognitive burden off of people—where Moxi could just infer what could happen in the background.

In terms of direct collaboration, like side-by-side working together kind of thing, I do think there’s just such vast differences between—if you’re talking about a human and a robot cooperating on some manipulation task, robots are just—it’s going to be awhile before a robot is going to be as capable. If you already have a person there, doing some kind of manipulation task, it’s going to be hard for a robot to compete, and so I think it’s better to think about places where the person could be used for better things and you could hand something else off entirely to the robot.

So how feasible in the near-term is a nurse saying, “Moxi, could you hold this for me?” How complicated or potentially useful is that?

I think that’s a really interesting example. So then a question is, is the value of the resource and whether being always available to be like a third hand for any particular clinician is the most valuable thing that this mobile manipulation platform could be doing, and what, we did a little bit of that kind on-demand, you know, hey Moxi come over here and do this thing, in some of our beta trials just to kind of look at that on demand versus pre planned activities, and if you can find things in workflows that can be automated and inferred what the robot’s gonna be doing, we think that’s gonna be the biggest bang for your buck, in terms of the value that the robot’s able to deliver, 

I think that there may come a day where every clinician’s walking around and there’s always a robot available to respond to “hey, hold this for me,” and I think that would be amazing. But for now, the question is whether the robot being like a third hand for any particular clinician is the most valuable thing that this mobile manipulation platform could be doing, when it could instead be working all night long to get things ready for the next shift.

[ Diligent Robotics ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

ICARSC 2020 – April 15-17, 2020 – [Online Conference] ICRA 2020 – May 31-4, 2020 – [TBD] ICUAS 2020 – June 9-12, 2020 – Athens, Greece RSS 2020 – July 12-16, 2020 – [Online Conference] CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

You need this dancing robot right now.

By Vanessa Weiß at UPenn.

[ KodLab ]

Remember Qoobo the headless robot cat? There’s a TINY QOOBO NOW!

It’s available now on a Japanese crowdfunding site, but I can’t tell if it’ll ship to other countries.

[ Qoobo ]

Just what we need, more of this thing.

[ Vstone ]

HiBot, which just received an influx of funding, is adding new RaaS (robotics as a service) offerings to its collection of robot arms and snakebots.

HiBot ]

If social distancing already feels like too much work, Misty is like that one-in-a-thousand child that enjoys cleaning. See her in action here as a robot disinfector and sanitizer for common and high-touch surfaces. Alcohol reservoir, servo actuator, and nozzle not (yet) included. But we will provide the support to help you build the skill.

[ Misty Robotics ]

After seeing this tweet from Kate Darling that mentions an MIT experiment in which “a group of gerbils inhabited an architectural environment made of modular blocks, which were manipulated by a robotic arm in response to the gerbils’ movements,” I had to find a video of the robot arm gerbil habitat. The best I could do was this 2007 German remake, but it’s pretty good:

[ Lutz Dammbeck ]

We posted about this research almost a year ago when it came out in RA-L, but I’m not tired of watching the video yet.

Today’s autonomous drones have reaction times of tens of milliseconds, which is not enough for navigating fast in complex dynamic environments. To safely avoid fast moving objects, drones need low-latency sensors and algorithms. We depart from state of the art approaches by using event cameras, which are novel bioinspired sensors with reaction times of microseconds. We demonstrate the effectiveness of our approach on an autonomous quadrotor using only onboard sensing and computation. Our drone was capable of avoiding multiple obstacles of different sizes and shapes at relative speeds up to 10 meters/second, both indoors and outdoors.

[ UZH ]

In this video we present the autonomous exploration of a staircase with four sub-levels and the transition between two floors of the Satsop Nuclear Power Plant during the DARPA Subterranean Challenge Urban Circuit. The utilized system is a collision-tolerant flying robot capable of multi-modal Localization And Mapping fusing LiDAR, vision and inertial sensing. Autonomous exploration and navigation through the staircase is enabled through a Graph-based Exploration Planner implementing a specific mode for vertical exploration. The collision-tolerance of the platform was of paramount importance especially due to the thin features of the involved geometry such as handrails. The whole mission was conducted fully autonomously.


At Cognizant’s Inclusion in Tech: Work of Belonging conference, Cognizant VP and Managing Director of the Center for the Future of Work, Ben Pring, sits down with Mary “Mary” Cummings. Missy is currently a Professor at Duke University and the Director of the Duke Robotics Labe. Interestingly, Missy began her career as one of the first female fighter pilots in the U.S. Navy. Working in predominantly male fields – the military, tech, academia – Missy understands the prevalence of sexism, bias and gender discrimination.

Let’s hear more from Missy Cummings on, like, everything.

[ Duke ] via [ Cognizant ]

You don’t need to mountain bike for the Skydio 2 to be worth it, but it helps.

[ Skydio ]

Here’s a look at one of the preliminary simulated cave environments for the DARPA SubT Challenge.

[ Robotika ]

SherpaUW is a hybrid walking and driving exploration rover for subsea applications. The locomotive system consists of four legs with 5 active DoF each. Additionally, a 6 DoF manipulation arm is available. All joints of the legs and the manipulation arm are sealed against water. The arm is pressure compensated, allowing the deployment in deep sea applications.

SherpaUW’s hybrid crawler-design is intended to allow for extended long-term missions on the sea floor. Since it requires no extra energy to maintain its posture and position compared to traditional underwater ROVs (Remotely Operated Vehicles), SherpaUW is well suited for repeated and precise sampling operations, for example monitoring black smockers over a longer period of time.

[ DFKI ]

In collaboration with the Army and Marines, 16 active-duty Army soldiers and Marines used Near Earth’s technology to safely execute 64 resupply missions in an operational demonstration at Fort AP Hill, Virginia in Sep 2019. This video shows some of the modes used during the demonstration.

[ NEA ]

For those of us who aren’t either lucky enough or cursed enough to live with our robotic co-workers, HEBI suggests that now might be a great time to try simulation.

[ GitHub ]

DJI Phantom 4 Pro V2.0 is a complete aerial imaging solution, designed for the professional creator. Featuring a 1-inch CMOS sensor that can shoot 4K/60fps videos and 20MP photos, the Phantom 4 Pro V2.0 grants filmmakers absolute creative freedom. The OcuSync 2.0 HD transmission system ensures stable connectivity and reliability, five directions of obstacle sensing ensures additional safety, and a dedicated remote controller with a built-in screen grants even greater precision and control.

US $1600, or $2k with VR goggles.

[ DJI ]

Not sure why now is the right time to introduce the Fetch research robot, but if you forgot it existed, here’s a reminder.

[ Fetch ]

Two keynotes from the MBZIRC Symposium, featuring Oussama Khatib and Ron Arkin.


And here are a couple of talks from the 2020 ROS-I Consortium.

Roger Barga, GM of AWS Robotics and Autonomous Services at Amazon shares some of the latest developments around ROS and advanced robotics in the cloud.

Alex Shikany, VP of Membership and Business Intelligence for A3 shares insights from his organization on the relationship between robotics growth and employment.

[ ROS-I ]

Many tech companies are trying to build machines that detect people’s emotions, using techniques from artificial intelligence. Some companies claim to have succeeded already. Dr. Lisa Feldman Barrett evaluates these claims against the latest scientific evidence on emotion. What does it mean to “detect” emotion in a human face? How often do smiles express happiness and scowls express anger? And what are emotions, scientifically speaking?

[ Microsoft ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – [ONLINE EVENT] ICARSC 2020 – April 15-17, 2020 – [ONLINE EVENT] ICRA 2020 – May 31-4, 2020 – [SEE ATTENDANCE SURVEY] ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

UBTECH Robotics’ ATRIS, AIMBOT, and Cruzr robots were deployed at a Shenzhen hospital specialized in treating COVID-19 patients. The company says the robots, which are typically used in retail and hospitality scenarios, were modified to perform tasks that can help keep the hospital safer for everyone, especially front-line healthcare workers. The tasks include providing videoconferencing services between patients and doctors, monitoring the body temperatures of visitors and patients, and disinfecting designated areas.

The Third People’s Hospital of Shenzhen (TPHS), the only designated hospital for treating COVID-19 in Shenzhen, a metropolis with a population of more than 12.5 million, has introduced an intelligent anti-epidemic solution to combat the coronavirus.

AI robots are playing a key role. The UBTECH-developed robot trio, namely ATRIS, AIMBOT, and Cruzr, are giving a helping hand to monitor body temperature, detect people without masks, spray disinfectants and provide medical inquiries.


Someone has spilled gold all over the place! Probably one of those St. Paddy’s leprechauns... Anyways... It happened near a Robotiq Wrist Camera and Epick setup so it only took a couple of minutes to program and ’’pick and place’’ the mess up.

Even in situations like these, it’s important to stay positive and laugh a little, we had this ready and though we’d still share. Stay safe!

[ Robotiq ]

HEBI Robotics is helping out with social distancing by controlling a robot arm in Austria from their lab in Pittsburgh.

Can’t be too careful!

[ HEBI Robotics ]

Thanks Dave!

SLIDER, a new robot under development at Imperial College London, reminds us a little bit of what SCHAFT was working on with its straight-legged design.

[ Imperial ]

Imitation learning is an effective and safe technique to train robot policies in the real world because it does not depend on an expensive random exploration process. However, due to the lack of exploration, learning policies that generalize beyond the demonstrated behaviors is still an open challenge. We present a novel imitation learning framework to enable robots to 1) learn complex real world manipulation tasks efficiently from a small number of human demonstrations, and 2) synthesize new behaviors not contained in the collected demonstrations. Our key insight is that multi-task domains often present a latent structure, where demonstrated trajectories for different tasks intersect at common regions of the state space. We present Generalization Through Imitation (GTI), a two-stage offline imitation learning algorithm that exploits this intersecting structure to train goal-directed policies that generalize to unseen start and goal state combinations.

[ GTI ]

Here are two excellent videos from UPenn’s Kod*lab showing the capabilities of their programmable compliant origami spring things.

[ Kod*lab ]

We met Bornlove when we were reporting on drones in Tanzania in 2018, and it’s good to see that he’s still improving on his built-from-scratch drone.

[ ADF ]

Laser. Guided. Sandwich. Stacking.

[ Kawasaki ]

The Self-Driving Car Research Studio is a highly expandable and powerful platform designed specifically for academic research. It includes the tools and components researchers need to start testing and validating their concepts and technologies on the first day, without spending time and resources on building DYI platforms or implementing hobby-level vehicles. The research studio includes a fleet of vehicles, software tools enabling researchers to work in Simulink, C/C++, Python, or ROS, with pre-built libraries and models and simulated environments support, even a set of reconfigurable floor panels with road patterns and a set of traffic signs. The research studio’s feature vehicle, QCar, is a 1/10 scale model vehicle powered by NVIDIA Jetson TX2 supercomputer and equipped with LIDAR, 360-degree vision, depth sensor, IMU, encoders, and other sensors, as well as user-expandable IO.

[ Quanser ]

Thanks Zuzana!

The Swarm-Probe Enabling ATEG Reactor, or SPEAR, is a nuclear electric propulsion spacecraft that uses a new, lightweight reactor moderator and advanced thermoelectric generators (ATEGs) to greatly reduce overall core mass. If the total mass of an NEP system could be reduced to levels that were able to be launched on smaller vehicles, these devices could deliver scientific payloads to anywhere in the solar system.

One major destination of recent importance is Europa, one of the moons of Jupiter, which may contain traces of extraterrestrial life deep beneath the surface of its icy crust. Occasionally, the subsurface water on Europa violently breaks through the icy crust and bursts into the space above, creating a large water plume. One proposed method of searching for evidence of life on Europa is to orbit the moon and scan these plumes for ejected organic material. By deploying a swarm of Cubesats, these plumes can be flown through and analyzed multiple times to find important scientific data.


This hydraulic cyborg hand costs just $35.

Available next month in Japan.

[ Elekit ]

Microsoft is collaborating with researchers from Carnegie Mellon University and Oregon State University to compete in the DARPA Subterranean (SubT) challenges, collectively named Team Explorer. These challenges are designed to test drones and robots on how they perform in hazardous physical environments where humans can’t access safely. By participating in these challenges, these teams hope to find a solution that will assist emergency first responders to help find survivors more quickly.

[ Team Explorer ]

Aalborg University Hospital is the largest hospital in the North Jutland region of Denmark. Up to 3,000 blood samples arrive here in the lab every day. They must be tested and sorted – a time-consuming and monotonous process which was done manually until now. The university hospital has now automated the procedure: a robot-based system and intelligent transport boxes ensure the quality of the samples – and show how workflows in hospitals can be simplified by automation.

[ Kuka ]

This video shows human-robot collaboration for assembly of a gearbox mount in a realistic replica of a production line of Volkswagen AG. Knowledge-based robot skills enable autonomous operation of a mobile dual arm robot side-by-side of a worker.

[ DFKI ]

A brief overview of what’s going on in Max Likhachev’s lab at CMU.

Always good to see PR2 keeping busy!

[ CMU ]

The Intelligent Autonomous Manipulation (IAM) Lab at the Carnegie Mellon University (CMU) Robotics Institute brings together researchers to address the challenges of creating general purpose robots that are capable of performing manipulation tasks in unstructured and everyday environments. Our research focuses on developing learning methods for robots to model tasks and acquire versatile and robust manipulation skills in a sample-efficient manner.

[ IAM Lab ]

Jesse Hostetler is an Advanced Computer Scientist in the Vision and Learning org at SRI International in Princeton, NJ. In this episode of The Dish TV they explore the different aspects of artificial intelligence, and creating robots that use sleep and dream states to prevent catastrophic forgetting.

[ SRI ]

On the latest episode of the AI Podcast, Lex interviews Anca Dragan from UC Berkeley.

Anca Dragan is a professor at Berkeley, working on human-robot interaction -- algorithms that look beyond the robot’s function in isolation, and generate robot behavior that accounts for interaction and coordination with human beings.

[ AI Podcast ]

aside.inlay.CoronaVirusCoverage.xlrg { font-family: "Helvetica", sans-serif; text-transform: uppercase; text-align: center; border-width: 4px 0; border-top: 2px solid #666; border-bottom: 2px solid #666; padding: 10px 0; font-size: 18px; font-weight: bold; } span.LinkHereRed { color: #cc0000; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; }

When I reached Professor Guang-Zhong Yang on the phone last week, he was cooped up in a hotel room in Shanghai, where he had self-isolated after returning from a trip abroad. I wanted to hear from Yang, a widely respected figure in the robotics community, about the role that robots are playing in fighting the coronavirus pandemic. He’d been monitoring the situation from his room over the previous week, and during that time his only visitors were a hotel employee, who took his temperature twice a day, and a small wheeled robot, which delivered his meals autonomously.

An IEEE Fellow and founding editor of the journal Science Robotics, Yang is the former director and co-founder of the Hamlyn Centre for Robotic Surgery at Imperial College London. More recently, he became the founding dean of the Institute of Medical Robotics at Shanghai Jiao Tong University, often called the MIT of China. Yang wants to build the new institute into a robotics powerhouse, recruiting 500 faculty members and graduate students over the next three years to explore areas like surgical and rehabilitation robots, image-guided systems, and precision mechatronics.

“I ran a lot of the operations for the institute from my hotel room using Zoom,” he told me.

Yang is impressed by the different robotic systems being deployed as part of the COVID-19 response. There are robots checking patients for fever, robots disinfecting hospitals, and robots delivering medicine and food. But he thinks robotics can do even more.

Photo: Shanghai Jiao Tong University Professor Guang-Zhong Yang, founding dean of the Institute of Medical Robotics at Shanghai Jiao Tong University.

“Robots can be really useful to help you manage this kind of situation, whether to minimize human-to-human contact or as a front-line tool you can use to help contain the outbreak,” he says. While the robots currently being used rely on technologies that are mature enough to be deployed, he argues that roboticists should work more closely with medical experts to develop new types of robots for fighting infectious diseases.

“What I fear is that, there is really no sustained or coherent effort in developing these types of robots,” he says. “We need an orchestrated effort in the medical robotics community, and also the research community at large, to really look at this more seriously.”

Yang calls for a global effort to tackle the problem. “In terms of the way to move forward, I think we need to be more coordinated globally,” he says. “Because many of the challenges require that we work collectively to deal with them.”

Our full conversation, edited for clarity and length, is below.

IEEE Spectrum: How is the situation in Shanghai?

Guang-Zhong Yang: I came back to Shanghai about 10 days ago, via Hong Kong, so I’m now under self-imposed isolation in a hotel room just to be cautious, for two weeks. The general feeling in Shanghai is that it’s really calm and orderly. Everything seems well under control. And as you probably know, in recent days the number of new cases is steadily dropping. So the main priority for the government is to restore normal routines, and also for companies to go back to work. Of course, people are still very cautious, and there are systematic checks in place. In my hotel, for instance, I get checked twice a day for my temperature to make sure that all the people in the hotel are well.

Are most people staying inside, are the streets empty?

No, the streets are not empty. In fact, in Minhang, next to Shanghai Jiao Tong University, things are going back to normal. Not at full capacity, but stores and restaurants are gradually opening. And people are thinking about the essential travels they need to do, what they can do remotely. As you know in China we have very good online order and delivery services, so people use them a lot more. I was really impressed by how the whole thing got under control, really.

Has Shanghai Jiao Tong University switched to online classes?

Yes. Since last week, the students are attending online lectures. The university has 1449 courses for undergrads and 657 for graduate students. I participated in some of them. It’s really well run. You can have the typical format with a presenter teaching the class, but you can also have part of the lecture with the students divided into groups and having discussions. Of course what’s really affected is laboratory-based work. So we’ll need to wait for some more time to get back into action.

What do you think of the robots being used to help fight the outbreak?

I’ve seen reports showing a variety of robots being deployed. Disinfection robots that use UV light in hospitals. Drones being used for transporting samples. There’s a prototype robot, developed by the Chinese Academy of Sciences, to remotely collect oropharyngeal swabs from patients for testing, so a medical worker doesn’t have to directly swab the patient. In my hotel, there’s a robot that brings my meals to my door. This little robot can manage to get into the lift, go to your room, and call you to open the door. I’m a roboticist myself and I find it striking how well this robot works every time! [Laughs.]

Photo: UVD Robots UVD Robots has shipped hundreds of ultraviolet-C disinfection robots like the one above to Chinese hospitals. 

After Japan’s Fukushima nuclear emergency, the robotics community realized that it needed to be better prepared. It seems that we’ve made progress with disaster-response robots, but what about dealing with pandemics?

I think that for events involving infectious diseases, like this coronavirus outbreak, when they happen, everybody realizes the importance of robots. The challenge is that at most research institutions, people are more concerned with specific research topics, and that’s indeed the work of a scientist—to dig deep into the scientific issues and solve those specific problems. But we also need to have a global view to deal with big challenges like this pandemic.

So I think what we need to do, starting now, is to have a more systematic effort to make sure those robots can be deployed when we need them. We just need to recompose ourselves and work to identify the technologies that are ready to be deployed, and what are the key directions we need to pursue. There’s a lot we can do. It’s not too late. Because this is not going to disappear. We have to see the worst before it gets better.

Click here for additional coronavirus coverage

So what should we do to be better prepared?

After a major crisis, when everything is under control, people’s priority is to go back to our normal routines. The last thing in people’s minds is, What should we do to prepare for the next crisis? And the thing is, you can’t predict when the next crisis will happen. So I think we need three levels of action, and it really has to be a global effort. One is at the government level, in particular funding agencies: How to make sure we can plan ahead and to prepare for the worst.

Another level is the robotics community, including organizations like the IEEE, we need leadership to advocate for these issues and promote activities like robotics challenges. We see challenges for disasters, logistics, drones—how about a robotic challenge for infectious diseases. I was surprised, and a bit disappointed in myself, that we didn’t think about this before. So for the editorial board of Science Robotics, for instance, this will become an important topic for us to rethink.

And the third level is our interaction with front-line clinicians—our interaction with them needs to be stronger. We need to understand the requirements and not be obsessed with pure technologies, so we can ensure that our systems are effective, safe, and can be rapidly deployed. I think that if we can mobilize and coordinate our effort at all these three levels, that would be transformative. And we’ll be better prepared for the next crisis.

Are there projects taking place at the Institute of Medical Robotics that could help with this pandemic?

The institute has been in full operation for just over a year now. We have three main areas of research: The first is surgical robotics, which is my main area of research. The second area is in rehabilitation and assistive robots. The third area is hospital and laboratory automation. One important lesson that we learned from the coronavirus is that, if we can detect and intervene early, we have a better chance of containing it. And for other diseases, it’s the same. For cancer, early detection based on imaging and other sensing technologies, is critical. So that’s something we want to explore—how robotics, including technologies like laboratory automation, can help with early detection and intervention.

“One area we are working on is automated intensive-care unit wards. The idea it to build negative-pressure ICU wards for infectious diseases equipped with robotic capabilities that can take care of certain critical care tasks”

One area we are working on is automated intensive-care unit wards. The idea it to build negative-pressure ICU wards for infectious diseases equipped with robotic capabilities that can take care of certain critical care tasks. Some tasks could be performed remotely by medical personnel, while other tasks could be fully automated. A lot of the technologies that we already use in surgical robotics can be translated into this area. We’re hoping to work with other institutions and share our expertise to continue developing this further. Indeed, this technology is not just for emergency situations. It will also be useful for routine management of infectious disease patients. We really need to rethink how hospitals are organized in the future to avoid unnecessary exposure and cross-infection.

Photo: Shanghai Jiao Tong University Shanghai Jiao Tong University’s Institute of Medical Robotics is researching areas like micro/nano systems, surgical and rehabilitation robotics, and human-robot interaction.

I’ve seen some recent headlines—“China’s tech fights back,” “Coronavirus is the first big test for futuristic tech”—many people expect technology to save the day.

When there’s a major crisis like this pandemic, in the general public’s mind, people want to find a magic cure that will solve all the problems. I completely understand that expectation. But technology can’t always do that, of course. What technology can do is to help us to be better prepared. For example, it’s clear that in the last few years self-navigating robots with localization and mapping are becoming a mature technology, so we should see more of those used for situations like this. I’d also like to see more technologies developed for front-line management of patients, like the robotic ICU I mentioned earlier. Another area is public transportation systems—can they have an element of disease prevention, using technology to minimize the spread of diseases so that lockdowns are only imposed as a last resort?

And then there’s the problem of people being isolated. You probably saw that Italy has imposed a total lockdown. That could have a major psychological impact, particularly for people who are vulnerable and living alone. There is one area of robotics, called social robotics, that could play a part in this as well. I’ve been in this hotel room by myself for days now—I’m really starting to feel the isolation…

We should have done a Zoom call.

Yes, we should. [Laughs.] I guess this isolation, or quarantine for various people, also provides the opportunity for us to reflect on our lives, our work, our daily routines. That’s the silver lining that we may see from this crisis.

Photo: Unity Drive Innovation Unity Drive, a startup spun out of Hong Kong University of Science and Technology, is deploying self-driving vehicles to carry out contactless deliveries in three Chinese cities.

While some people say we need more technology during emergencies like this, others worry that companies and governments will use things like cameras and facial recognition to increase surveillance of individuals.

A while ago we published an article listing the 10 grand challenges for robotics in Science Robotics. One of the grand challenges is concerned with legal and ethical issues, which include what you mentioned in your question. Respecting privacy, and also being sensitive about individual and citizens’ rights—these are very, very important. Because we must operate within this legal ethical boundary. We should not use technologies that will intrude in people’s lives. You mentioned that some people say that we don’t have enough technology, and that others say we have too much. And I think both have a point. What we need to do is to develop technologies that are appropriate to be deployed in the right situation and for the right tasks.

Many researchers seem eager to help. What would you say to roboticists interested in helping fight this outbreak or prepare for the next one?

For medical robotics research, my experience is that for your technology to be effective, it has to be application oriented. You need to ensure that end-users like the clinicians who will use your robot, or in the case of assistive robots, the patients, that they are deeply involved in the development of the technology. And the second thing is really to think out of the box—how to develop radically different new technologies. Because robotics research is very hands on and there’s a tendency of adapting what’s readily available out there. For your technology to have a major impact, you need to fundamentally rethink your research and innovation, not just follow the waves.

For example, at our institute we’re investing a lot of effort on the development of micro and nano systems and also new materials that could one day be used in robots. Because for micro robotic systems, we can’t rely on the more traditional approach of using motors and gears that we use in larger systems. So my suggestion is to work on technologies that not only have a deep science element but can also become part of a real-world application. Only then we can be sure to have strong technologies to deal with future crises.

We’ve been writing about the musical robots from Georgia Tech’s Center for Music Technology for many, many years. Over that time, Gil Weinberg’s robots have progressed from being able to dance along to music that they hear, to being able to improvise along with it, to now being able to compose, play, and sing completely original songs.

Shimon, the marimba-playing robot that has performed in places like the Kennedy Center, will be going on a new tour to promote an album that will be released on Spotify next month, featuring songs written (and sung) entirely by the robot.

Deep learning is famous for producing results that seem like they sort of make sense, but actually don’t at all. Key to Shimon’s composing ability is its semantic knowledge—the ability to make thematic connections between things, which is a step beyond just throwing some deep learning at a huge database of music composed by humans (although that’s Shimon’s starting point, a dataset of 50,000 lyrics from jazz, prog rock, and hip-hop). So rather than just training a neural network that relates specific words that tend to be found together in lyrics, Shimon can recognize more general themes and build on them to create a coherent piece of music.

Fans of Shimon may have noticed that the robot has had its head almost completely replaced. It may be tempting to say “upgraded,” since the robot now has eyes, eyebrows, and a mouth, but I’ll always have a liking for Shimon’s older design, which had just one sort of abstract eye thing ( that functions as a mouth on the current design). Personally, I very much appreciate robots that are able to be highly expressive without resorting to anthropomorphism, but in its new career as a pop sensation, I guess having eyes and a mouth are, like, important, or something?

To find out more about Shimon’s new talents (and new face), we spoke with Georgia Tech professor Gil Weinberg and his PhD student Richard Savery.

IEEE Spectrum: What makes Shimon’s music fundamentally different from music that could have been written by a human? 

Richard Savery: Shimon’s musical knowledge is drawn from training on huge datasets of lyrics, around 20,000 prog rock songs and another 20,000 jazz songs. With this level of data Shimon is able to draw on far more sources of inspiration than than a human would ever be able to. At a fundamental level Shimon is able to take in huge amounts of new material very rapidly, so within a day it can change from focusing on jazz lyrics, to hip hop to prog rock, or a hybrid combination of them all. 

How much human adjustment is involved in developing coherent melodies and lyrics with Shimon?

Savery: Just like working with a human collaborator, there’s many different ways Shimon can interact. Shimon can perform a range of musical tasks from composing a full song by itself or just playing a part composed by a human. For the new album we focused on human-robot collaboration so every song has some elements that were created by a human and some by Shimon. More than human adjustment from Shimon’s generation we try and have a musical dialogue where we get inspired and build on Shimon’s creation. Like any band, each of us has our own strengths and weaknesses, in our case no one else writes lyrics, so it was natural for Shimon to take responsibility for the lyrics. As a lyricist there’s a few ways Shimon can work, firstly Shimon can be given some keywords or ideas, like “earth” and “humanity” and then generate a full song of lyrics around those words. In addition to keywords Shimon can also take a musical and write lyrics that fit over that melody. 

The press release mentions that Shimon is able to “decide what’s good.” What does that mean?

Richard Savery: When Shimon writes lyrics the first step is generating thousands of phrases. So for those keywords Shimon will generate lots of material about “earth,” and then also generate related synonyms and antonyms like “world,” and “ocean.” Like a human composer Shimon has to parse through lots of ideas to choose what’s good from the creations. Shimon has preferences towards maintaining the same sentiment, or gradually shifting sentiment as well as trying to keep rhymes going between lines. For Shimon good lyrics should rhyme, keep some core thematic ideas going, maintain a similar sentiment and have some similarity to existing lyrics. 

I would guess that Shimon’s voice could have been almost anything—why choose this particular voice?

Gil Weinberg: Since we did not have singing voice synthesis expertise in our Robotic Musicianship group at Georgia Tech, we looked to collaborate with other groups. The Music Technology Group at Pompeu Fabra University developed a remarkable deep learning-based singing voice synthesizer and was excited to collaborate. As part of the process, we sent them audio files of songs recorded by one of our students to be used as a dataset to train their neural network. At the end, we decided to use another voice that was trained on a different dataset, since we felt it better represented Shimon’s genderless personality and was a better fit to the melodic register of our songs. 

“We hope both audiences and musicians will see Shimon as an expressive and creative musician, who can understand and connect to music like we humans do, but also has a strange and unique mind that can surprise and inspire us” —Gil Weinberg, Georgia Tech

Can you tell us about the changes made to Shimon’s face?

Weinberg: We are big fans of avoiding exaggerated anthropomorphism and using too many degrees of freedom in our robots. We feel that this might push robots into the uncanny valley. But after much deliberation, we decided that a singing robot should have a mouth to represent the embodiment of singing and to look believable. It was important to us, though, not to add DoFs for this purpose, rather to replace the old eye DoF with a mouth to minimize complexity. Originally, we thought to repurpose both DoFs of the old eye (bottom eyelid and top eye lid) to represent top lip and bottom lip. But we felt this might be too anthropomorphic, and that it would be more challenging and interesting to use only one DoF to automatically control mouth size based on the lyric’s phonemes. For this purpose, we looked at examples as varied as parrot vocalization and Muppets animation, to learn how animals and animators go about mouth actuation. Once we were happy with what we developed, we decided to use the old top eyelid DoFs as an eyebrow, to add more emotion to Shimon’s expression. 

Are you able to take advantage of any inherently robotic capabilities of Shimon?

Weinberg: One of the most important new features of the new Shimon, in addition to its singing song-writing capabilities, is a total redesign of its striking arms. As part of the process we replaced the old solenoid-based actuators with new brushless DC motors that can support a much faster striking (up to 30 hits per second) as well as a wider and more linear dynamic range—from very soft pianissimo to much louder fortissimo. This not only allows for a much richer musical expression, but also supports the ability to create new humanly impossible timbres and sonorities by using 8 novel virtuosic actuators. We hope and believe that these new abilities would push human collaborators to new uncharted directions that could not be achieved in human-to-human collaboration.

How do you hope audiences will react to Shimon?

Weinberg: We hope both audiences and musicians will see Shimon as an expressive and creative musician, who can understand and connect to music like we humans do, but also has a strange and unique mind that can surprise and inspire us to listen to, play, and think about music in new ways.

What are you working on next?

Gil Weinberg: We are currently working on new capabilities that would allow Shimon to listen to, understand, and respond to lyrics in real time. The first genre we are exploring for this functionality is rap battles. We plan to release a new album on Spotify April 10th featuring songs where Shimon not only sings but raps in real time as well.

[ Georgia Tech ]

As much as we love soft robots (and we really love soft robots), the vast majority of them operate pneumatically (or hydraulically) at larger scales, especially when they need to exert significant amounts of force. This causes complications, because pneumatics and hydraulics generally require a pump somewhere to move fluid around, so you often see soft robots tethered to external and decidedly non-soft power sources. There’s nothing wrong with this, really, because there are plenty of challenges that you can still tackle that way, and there are some up-and-coming technologies that might result in soft pumps or gas generators.

Researchers at Stanford have developed a new kind of (mostly) soft robot based around a series of compliant, air-filled tubes. It’s human scale, moves around, doesn’t require a pump or tether, is more or less as safe as large robots get, and even manages to play a little bit of basketball.

Image: Stanford/Science Robotics

Stanford’s soft robot consists of a set of identical robotic roller modules mounted onto inflated fabric tubes (A). The rollers pinch the fabric tube between rollers, creating an effective joint (B) that can be relocated by driving the rollers. The roller modules actuate the robot by driving along the tube, simultaneously lengthening one edge while shortening another (C). The roller modules connect to each other at nodes using three-degree-of-freedom universal joints that are composed of a clevis joint that couples two rods, each free to spin about its axis (D). The robot moves untethered outdoors using a rolling gait (E).

This thing looks a heck of a lot like the tensegrity robots that NASA Ames has been working on forever, and which are now being commercialized (hopefully?) by Squishy Robotics. Stanford’s model is not technically a tensegrity robot, though, because it doesn’t use structural components that are under tension (like cables). The researchers refer to this kind of robot as “isoperimetric,” which means while discrete parts of the structure may change length, the overall length of all the parts put together stays the same. This means it’s got a similar sort of inherent compliance across the structure to tensegrity robots, which is one of the things that makes them so appealing. 

While the compliance of Stanford’s robot comes from a truss-like structure made of air-filled tubes, its motion relies on powered movable modules. These modules pinch the tube that they’re located on through two cylindrical rollers (without creating a seal), and driving the rollers moves the module back and forth along the tube, effectively making one section of the tube longer and the other one shorter. Although this is just one degree of freedom, having a whole bunch of tubes each with an independently controlled roller module means that the robot as a whole can exhibit complex behaviors, like drastic shape changes, movement, and even manipulation.

There are numerous advantages to a design like this. You get all the advantages of pneumatic robots (compliance, flexibility, collapsibility, durability, high strength to weight ratio) without requiring some way of constantly moving air around, since the volume of air inside the robot stays constant. Each individual triangular module is self-contained (with one tube, two active roller modules, and one passive anchor module) and easy to combine with similar modules—the video shows an octahedron, but you can easily add or subtract modules to make a variety of differently shaped robots with different capabilities.

Since the robot is inherently so modular, there are all kinds of potential applications for this thing, as the researchers speculate in a paper published today in Science Robotics:

The compliance and shape change of the robot could make it suitable for several tasks involving humans. For example, the robot could work alongside workers, holding parts in place as the worker bolts them in place. In the classroom, the modularity and soft nature of the robotic system make it a potentially valuable educational tool. Students could create many different robots with a single collection of hardware and then physically interact with the robot. By including a much larger number of roller modules in a robot, the robot could function as a shape display, dynamically changing shape as a sort of high–refresh rate 3D printer. Incorporating touch-sensitive fabric into the structure could allow users to directly interact with the displayed shapes. More broadly, the modularity allows the same hardware to build a diverse family of robots—the same roller modules can be used with new tube routings to create new robots. If the user needed a robot to reach through a long, narrow passageway, they could assemble a chain-like robot; then, for a locomoting robot, they could reassemble into a spherical shape.

Image: Farrin Abbott

I’m having trouble picturing some of that stuff, but the rest of it sounds like fun.

We’re obligated to point out that because of the motorized roller modules, this soft robot is really only semi-soft, and you could argue that it’s not fundamentally all that much better than hydraulic or pneumatic soft robots with embedded rigid components like batteries and pumps. Calling this robot “inherently human-safe,” as the researchers do, might be overselling it slightly, in that it has hard edges, pokey bits, and what look to be some serious finger-munchers. It does sound like there might be some potential to replace the roller modules with something softer and more flexible, which will be a focus of future work.

An untethered isoperimetric soft robot,” by Nathan S. Usevitch, Zachary M. Hammond, Mac Schwager, Allison M. Okamura, Elliot W. Hawkes, and Sean Follmer from Stanford University and UCSB, was published in Science Robotics.

Editor’s Note: When we asked Rodney Brooks if he’d write an article for IEEE Spectrum on his definition of robot, he wrote back right away. “I recently learned that Warren McCulloch”—one of the pioneers of computational neuroscience—“wrote sonnets,” Brooks told us. “He, and your request, inspired me. Here is my article—a little shorter than you might have desired.” Included in his reply were 14 lines composed in iambic pentameter. Brooks titled it “What Is a Robot?” Later, after a few tweaks to improve the metric structure of some of the lines, he added, “I am no William Shakespeare, but I think it is now a real sonnet, if a little clunky in places.”

What Is a Robot?*
By Rodney Brooks

Shall I compare thee to creatures of God?
Thou art more simple and yet more remote.
You move about, but still today, a clod,
You sense and act but don’t see or emote.

You make fast maps with laser light all spread,
Then compare shapes to object libraries,
And quickly plan a path, to move ahead,
Then roll and touch and grasp so clumsily.

You learn just the tiniest little bit,
And start to show some low intelligence,
But we, your makers, Gods not, we admit,
All pledge to quest for genuine sentience.

    So long as mortals breathe, or eyes can see,
    We shall endeavor to give life to thee.

* With thanks to William Shakespeare

Rodney Brooks is the Panasonic Professor of Robotics (emeritus) at MIT, where he was director of the AI Lab and then CSAIL. He has been cofounder of iRobot, Rethink Robotics, and Robust AI, where he is currently CTO.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – Cambridge, U.K. [CANCELED] ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

Having robots learn dexterous tasks requiring real-time hand-eye coordination is hard. Many tasks that we would consider simple, like hanging up a baseball cap on a rack, would be very challenging for most robot software. What’s more, for a robot to learn each new task, it typically takes significant amounts of engineering time to program the robot. Pete Florence and Lucas Manuelli in the Robot Locomotion Group took a step closer to that goal with their work.

[ Paper ]

Octo-Bouncer is not a robot that bounces an octopus. But it’s almost as good. Almost.

[ Electron Dust ]

D’Kitty (pronounced as “The Kitty”) is a 12-degree-of-freedom platform for exploring learning-based techniques in locomotion and it’s adooorable!

[ D’Kitty ]

Knightscope Autonomous Security Robot meets Tesla Model 3 in Summon Mode!  See, nothing to fear, Elon. :-)

The robots also have a message for us:

[ Knightscope ]

If you missed the robots vs. humans match at RoboCup 2019, here are the highlights.

Tech United ]

Fraunhofer developed this cute little demo of autonomously navigating, cooperating mobile robots executing a miniaturized logistics scenario involving chocolate for the LogiMAT trade show. Which was canceled. But enjoy the video!

[ Fraunhofer ]

Thanks Thilo!

Drones can potentially be used for taking soil samples in awkward areas by dropping darts equipped with accelerometers. But the really clever bit is how the drone can retrieve the dart on its own.

[ UH ]

Rope manipulation is one of those human-easy robot-hard things that’s really, really robot-hard.

[ UC Berkeley ]

Autonomous landing on a moving platform presents unique challenges for multirotor vehicles, including the need to accurately localize the platform, fast trajectory planning, and precise/robust control. This work presents a fully autonomous vision-based system that addresses these limitations by tightly coupling the localization, planning, and control, thereby enabling fast and accurate landing on a moving platform. The platform’s position, orientation, and velocity are estimated by an extended Kalman filter using simulated GPS measurements when the quadrotor-platform distance is large, and by a visual fiducial system when the platform is nearby. To improve the performance, the characteristics of the turbulent conditions are accounted for in the controller. The landing trajectory is fast, direct, and does not require hovering over the platform, as is typical of most state-of-the-art approaches. Simulations and hardware experiments are presented to validate the robustness of the approach.


And now, this.

[ Soft Robotics ]

The EPRI (Electric Power Research Institute) recently worked with Exyn Technologies, a pioneer in autonomous aerial robot systems, for a safety and data collection demonstration at Exelon’s Peach Bottom Atomic Power Station in Pennsylvania. Exyn’s drone was able to autonomously inspect components in elevated hard to access areas, search for temperature anomalies, and collect dose rate surveys in radiological areas— without the need for a human operator.

[ Exyn ]

Thanks Zach!

Relax: Pepper is here to help with all of your medical problems.

[ Softbank ]

Amir Shapiro at BGU, along with Yoav Golan (whose work on haptic control of dogs we covered last year), have developed an interesting new kind of robotic finger with passively adjustable friction.

Paper ] via [ BGU ]

Thanks Andy!

UBTECH’s Alpha Mini Robot with Smart Robot’s “Maatje” software is expected to offer healthcare services to children at Sint Maartenskliniek in the Netherlands. Before that, three of them have been trained to have exercise, empathy and cognition capabilities.


Get ready for CYBATHLON, postponed to September 2020!

[ Cybathlon ]

In partnership with the World Mosquito Program (WMP), WeRobotics has led the development and deployment of a drone-based release mechanism that has been shown to help prevent the incidence of Dengue fever.

[ WeRobotics ]

Sadly, koalas today face a dire outlook across Australia due to human development, droughts, and forest fires. Events like these and a declining population make conservation and research more important than ever. Drones offer a more efficient way to count koalas from above, covering more ground than was possible in the past. Dr. Hamilton and his team at the Queensland University of Technology use DJI drones to count koalas, using the data obtained to better help these furry friends from down under.

[ DJI ]

Fostering the Next Generation of Robotics Startups | TC Sessions: Robotics

Robotics and AI are the future of many or most industries, but the barrier of entry is still difficult to surmount for many startups. Speakers will discuss the challenges of serving robotics startups and companies that require robotics labor, from bootstrapped startups to large scale enterprises.

[ TechCrunch ]

aside.inlay.CoronaVirusCoverage.xlrg { font-family: "Helvetica", sans-serif; text-transform: uppercase; text-align: center; border-width: 4px 0; border-top: 2px solid #666; border-bottom: 2px solid #666; padding: 10px 0; font-size: 18px; font-weight: bold; } span.LinkHereRed { color: #cc0000; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; }

The absolute best way of dealing with the coronavirus pandemic is to just not get coronavirus in the first place. By now, you’ve (hopefully) had all of the strategies for doing this drilled into your skull—wash your hands, keep away from large groups of people, wash your hands, stay home when sick, wash your hands, avoid travel when possible, and please, please wash your hands

At the top of the list of the places to avoid right now are hospitals, because that’s where all the really sick people go. But for healthcare workers, and the sick people themselves, there’s really no other option. To prevent the spread of coronavirus (and everything else) through hospitals, keeping surfaces disinfected is incredibly important, but it’s also dirty, dull, and (considering what you can get infected with) dangerous. And that’s why it’s an ideal task for autonomous robots.

  Photo: UVD Robots The robots can travel through hallways, up and down elevators if necessary, and perform the disinfection without human intervention before returning to recharge.

UVD Robots is a Danish company making robots that are able to disinfect patient rooms and operating theaters in hospitals. They’re able to disinfect pretty much anything you point them at—each robot is a mobile array of powerful short wavelength ultraviolet-C (UVC) lights that emit enough energy to literally shred the DNA or RNA of any microorganisms that have the misfortune of being exposed to them. 

The company’s robots have been operating in China for the past two or three weeks, and UVD Robots CEO Per Juul Nielsen says they are sending more to China as fast as they can. “The initial volume is in the hundreds of robots; the first ones went to Wuhan where the situation is the most severe,” Nielsen told IEEE Spectrum. “We’re shipping every week—they’re going air freight into China because they’re so desperately needed.” The goal is to supply the robots to over 2,000 hospitals and medical facilities in China.

UV disinfecting technology has been around for something like a century, and it’s commonly used to disinfect drinking water. You don’t see it much outside of fixed infrastructure because you have to point a UV lamp directly at a surface for a couple of minutes in order to be effective, and since it can cause damage to skin and eyes, humans have to be careful around it. Mobile UVC disinfection systems are a bit more common—UV lamps on a cart that a human can move from place to place to disinfect specific areas, like airplanes. For large environments like a hospital with dozens of rooms, operating UV systems manually can be costly and have mixed results—humans can inadvertently miss certain areas, or not expose them long enough.

“And then came the coronavirus, accelerating the situation—spreading more than anything we’ve seen before on a global basis” —Per Juul Nielsen, UVD Robots

UVD Robots spent four years developing a robotic UV disinfection system, which it started selling in 2018. The robot consists of a mobile base equipped with multiple lidar sensors and an array of UV lamps mounted on top. To deploy a robot, you drive it around once using a computer. The robot scans the environment using its lidars and creates a digital map. You then annotate the map indicating all the rooms and points the robot should stop to perform disinfecting tasks. 

After that, the robot relies on simultaneous localization and mapping (SLAM) to navigate, and it operates completely on its own. It’ll travel from its charging station, through hallways, up and down elevators if necessary, and perform the disinfection without human intervention before returning to recharge. For safety, the robot operates when people are not around, using its sensors to detect motion and shutting the UV lights off if a person enters the area.


It takes between 10 and 15 minutes to disinfect a typical room, with the robot spending 1 or 2 minutes in five or six different positions around the room to maximize the number of surfaces that it disinfects. The robot’s UV array emits 20 joules per square meter per second (at 1 meter distance) of 254-nanometer light, which will utterly wreck 99.99 percent of germs in just a few minutes without the robot having to do anything more complicated than just sit there. The process is more consistent than a human cleaning since the robot follows the same path each time, and its autonomy means that human staff can be freed up to do more interesting tasks, like interacting with patients. 

Originally, the robots were developed to address hospital acquired infections, which are a significant problem globally. According to Nielsen, between 5 and 10 percent of hospital patients worldwide will acquire a new infection while in the hospital, and tens of thousands of people die from these infections every year. The goal of the UVD robots was to help hospitals prevent these infections in the first place.

Photo: UVD Robots A shipment of robots from UVD Robots arrives at a hospital in Wuhan, where the first coronavirus cases were reported in December.

“And then came the coronavirus, accelerating the situation—spreading more than anything we’ve seen before on a global basis,” Nielsen says. “That’s why there’s a big need for our robots all over the world now, because they can be used in fighting coronavirus, and for fighting all of the other infections that are still there.”

The robots, which cost between US $80,000 and $90,000, are relatively affordable for medical equipment, and as you might expect, recent interest in them has been substantial. “Once [hospitals] see it, it’s a no-brainer,” Nielsen says. “If they want this type of disinfection solution, then the robot is much smarter and more cost-effective than what’s available in the market today.” Hundreds of these robots are at work in more than 40 countries, and they’ve recently completed hospital trials in Florida. Over the next few weeks, they’ll be tested at other medical facilities around the United States, and Nielsen points out that they could be useful in schools, cruise ships, or any other relatively structured spaces. I’ll take one for my apartment, please.

UVD Robots ]

aside.inlay.xlrg.XploreFree { font-family: "Georgia", serif; border-width: 4px 0; border-top: solid #888; border-bottom: solid #888; padding: 10px 0; font-size: 19px; font-weight: bold; text-align: center; } span.FreeRed { color: red; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; } span.XploreBlue { color: #03a6e3; font-family: "Theinhardt-Medium", sans-serif; }

Swarms of small, inexpensive robots are a compelling research area in robotics. With a swarm, you can often accomplish tasks that would be impractical (or impossible) for larger robots to do, in a way that’s much more resilient and cost effective than larger robots could ever be.

The tricky thing is getting a swarm of robots to work together to do what you want them to do, especially if what you want them to do is a task that’s complicated or highly structured. It’s not too bad if you have some kind of controller that can see all the robots at once and tell them where to go, but that’s a luxury that you’re not likely to find outside of a robotics lab.

Researchers at Northwestern University, in Evanston, have been working on a way to provide decentralized control for a swarm of 100 identically programmed small robots, which allows them to collectively work out a way to transition from one shape to another without running into each other even a little bit.

The process that the robots use to figure out where to go seems like it should be mostly straightforward: They’re given a shape to form, so each robot picks its goal location (where it wants to end up as part of the shape), and then plans a path to get from where it is to where it needs to go, following a grid pattern to make things a little easier. But using this method, you immediately run into two problems: First, since there’s no central control, you may end up with two (or more) robots with the same goal; and second, there’s no way for any single robot to path plan all the way to its goal in a way that it can be certain won’t run into another robot.

To solve these problems, the robots are all talking to each other as they move, not just to avoid colliding with its friends, but also to figure out where its friends are going and whether it might be worth swapping destinations. Since the robots are all the same, they don’t really care where exactly they end up, as long as all of the goal positions are filled up. And if one robot talks to another robot and they agree that a goal swap would result in both of them having to move less, they go ahead and swap. The algorithm makes sure that all goal positions are filled eventually, and also helps robots avoid running into each other through judicious use of a “wait” command.

What’s novel about this approach is that despite the fully distributed nature of the algorithm, it’s also provably correct, and will result in the guaranteed formation of an entire shape without collisions or deadlocks. As far as the researchers know, it’s the first algorithm to do this.

What’s really novel about this approach is that despite the fully distributed nature of the algorithm, it’s also provably correct, and will result in the guaranteed formation of an entire shape without collisions or deadlocks. As far as the researchers know, it’s the first algorithm to do this. And it means that since it’s effective with no centralized control at all, you can think of “the swarm” as a sort of Borg-like collective entity of its own, which is pretty cool.

The Northwestern researchers behind this are Michael Rubenstein, assistant professor of electrical engineering and computer science, and his PhD student Hanlin Wang. You might remember Mike from his work on Kilobots at Harvard, which we wrote about in 2011, 2013, and again in 2014, when Mike and his fellow researchers managed to put together a thousand (!) of them. As awesome as it is to have a thousand robots, when you start thinking about what it takes to charge, fix, and modify them, a thousand robots (a thousand robots!), it makes sense why they’ve updated the platform a bit (now called Coachbot) and reduced the swarm size to 100 physical robots, making up the rest in simulation.

These robots, we’re told, are “much better behaved.”

Image: Northwestern University

The hardware used by the researchers in their experiments. 1. The Coachbot V2.0 mobile robots (height of 12 cm and a diameter of 10 cm) are equipped with a localization system based on the HTC Vive (a), Raspberry Pi b+ computer (b), electronics motherboard (c), and rechargeable battery (d). The robot arena used in experiments has an overhead camera only used for recording videos (e) and an overhead HTC Vive base station (f). The experiments relied on a swarm of 100 robots (g). 2. The Coachbot V2.0 swarm communication network consists of an ethernet connection between the base station and a Wi-Fi router (green link), TCP/IP connections (blue links), and layer 2 broadcasting connections (black links). 3. A swarm of 100 robots. 4. The robots recharge their batteries by connecting to two metal strips attached to the wall.

For more details on this work, we spoke with Mike Rubenstein via email.

IEEE Spectrum: Why switch to the new hardware platform instead of Kilobots?

Mike Rubenstein: We wanted to make a platform more capable and extendable than Kilobot, and improve on lessons learned with Kilobot. These robots have far better locomotion capabilities that Kilobot, and include absolute position sensing, which makes operating the robots easier. They have truly “hands free” operations. For example with Kilobot to start an experiment you had to place the robots in their starting position by hand (sometimes taking an hour or two), while with these robots, a user just specifies a set of positions for all the robots and presses the “go” button. With Kilobot it was also hard to see what the state of all the robots were, for example it was difficult to see if 999 robots are powered on or 1000 robots are powered on. These new robots send state information back to a user display, making it easy to understand the full state of the swarm. 
How much of a constraint is grid-ifying the goal points and motion planning?

The grid constraint obviously makes motion less efficient as they must move in Manhattan-type paths, not straight line paths, so most of the time they move a bit farther. The reason we constrain the motions to move in a discrete grid is that it makes the robot algorithm less computationally complex and reasoning about collisions and deadlock becomes a lot easier, which allowed us to provide guarantees that the shape will form successfully. 

Image: Northwestern University

Still images of a 100 robot shape formation experiment. The robots start in a random configuration, and move to form the desired “N” shape. Once this shape is formed, they then form the shape “U.” The entire sequence is fully autonomous. (a) T = 0 s; (b) T = 20 s; (c) T = 64 s; (d) T = 72 s; (e)  T = 80 s; (f) T = 112 s.

Can you tell us about those couple of lonely wandering robots at the end of the simulated “N” formation in the video?

In our algorithm, we don’t assign goal locations to all the robots at the start, they have to figure out on their own which robot goes where. The last few robots you pointed out happened to be far away from the goal location the swarm figured they should have. Instead of having that robot move around the whole shape to its goal, you see a subset of robots all shift over by one to make room for the robot in the shape closer to its current position.
What are some examples of ways in which this research could be applied to real-world useful swarms of robots?

One example could be the shape formation in modular self-reconfigurable robots. The hope is that this shape formation algorithm could allow these self-reconfigurable systems to automatically change their shape in a simple and reliable way. Another example could be warehouse robots, where robots need to move to assigned goals to pick up items. This algorithm would help them move quickly and reliably.
What are you working on next?

I’m looking at trying to understand how to enable large groups of simple individuals to behave in a controlled and reliable way as a group. I’ve started looking at this question in a wide range of settings; from swarms of ground robots, to reconfigurable robots that attach together by melting conductive plastic, to swarms of flying vehicles, to satellite swarms. 

Shape Formation in Homogeneous Swarms Using Local Task Swapping,” by Hanlin Wang and Michael Rubenstein from Northwestern, is published in IEEE Transactions on Robotics. < Back to IEEE Journal Watch

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

NASA Curiosity Project Scientist Ashwin Vasavada guides this tour of the rover’s view of the Martian surface. Composed of more than 1,000 images and carefully assembled over the ensuing months, the larger version of this composite contains nearly 1.8 billion pixels of Martian landscape.

This panorama showcases "Glen Torridon," a region on the side of Mount Sharp that Curiosity is exploring. The panorama was taken between Nov. 24 and Dec. 1, 2019, when the Curiosity team was out for the Thanksgiving holiday. Since the rover would be sitting still with few other tasks to do while it waited for the team to return and provide its next commands, the rover had a rare chance to image its surroundings several days in a row without moving.

[ MSL ]

Sarcos has been making progress with its Guardian XO powered exoskeleton, which we got to see late last year in prototype stage:

The Sarcos Guardian XO full-body, powered exoskeleton is a first-of-its-kind wearable robot that enhances human productivity while keeping workers safe from strain or injury. Set to transform the way work gets done, the Guardian XO exoskeleton augments operator strength without restricting freedom of movement to boost productivity while dramatically reducing injuries.

[ Sarcos ]

Professor Hooman Samani, director of the Artificial Intelligence and Robotics Technology Laboratory (AIART Lab) at National Taipei University, Taiwan, writes in to share some ideas on how robots could be used to fight the coronavirus outbreak. 

Time is a critical issue when dealing with people affected by Coronavirus. Also due to the current emergency disaster, doctors could be far away from the patients. Additionally, avoiding direct contact with infected person is a medical priority. An immediate monitoring and treatment using specific kits must be administered to the victim. We have designed and developed the Ambulance Robot (AmbuBot) which could be a solution to address those issues. AmbuBot could be placed in various locations especially in busy, remote or quarantine areas to assist in above mentioned scenario. The AmbuBot also brings along an AED in a sudden event of cardiac arrest and facilitates various modes of operation from manual to semi-autonomous to autonomous functioning.

[ AIART Lab ]

IEEE Spectrum is interested in exploring how robotics and related technologies can help to fight the coronavirus (COVID-19) outbreak. If you are involved with actual deployments of robots to hospitals and high risk areas or have experience working with robots, drones, or other autonomous systems designed for this kind of emergency, please contact  IEEE Spectrum senior editor Erico Guizzo (

Digit is launching later this month alongside a brand new sim that’s a 1:1 match to both the API and physics of the actual robot. Here, we show off the ability to train a learned policy against the validated physics of the robot. We have a LOT more to say about RL with real hardware... stay tuned.

Staying tuned!

Agility Robotics ]

This video presents simulations and experiments highlighting the functioning of the proposed Trapezium Line Theta* planner, as well as its improvements over our previous work namely the Obstacle Negotiating A* planner. First, we briefly present a comparison of our previous and new planners. We then show two simulations. The first shows the robot traversing an inclined corridor to reach a goal near the low-lying obstacle. This demonstrates the omnidirectional and any-angle motion planning improvement achieved by the new planner, as well as the independent planning for the front and back wheel pairs. The second simulation further demonstrates the key improvements mentioned above by having the robot traverse tight right-angled corridors. Finally, we present two real experiments on the CENTAURO robot. In the first experiment, the robot has to traverse into a narrow passage and then expand over a low lying obstacle. The second experiment has the robot first expand over a wide obstacle and then move into a narrow passage.

To be presented at ICRA 2020.

Dimitrios Kanoulas ]

We’re contractually obligated to post any video with “adverse events” in the title.


Waymo advertises their self-driving system in this animated video that features a robot car making a right turn without indicating. Also pretty sure that it ends up in the wrong lane for a little bit after a super wide turn and blocks a crosswalk to pick up a passenger. Oops!

I’d still ride in one, though.

Waymo ]

Exyn is building the world’s most advanced, autonomous aerial robots. Today, we launched our latest capability, Scoutonomy. Our pilotless robot can now ‘scout’ freely within a desired volume, such as a tunnel, or this parking garage. The robot sees the white boxes as ‘unknown’ space, and flies to explore them. The orange boxes are mapped obstacles. It also intelligently avoids obstacles in its path and identifies objects, such as people or cars. Scoutonomy can be used to safely and quickly finding survivors in natural, or man-made, disasters.

Exyn ]

I don’t know what soma blocks are, but this robot is better with them than I am.

This work presents a planner that can automatically find an optimal assembly sequence for a dual-arm robot to assemble the soma blocks. The planner uses the mesh model of objects and the final state of the assembly to generate all possible assembly sequence and evaluate the optimal assembly sequence by considering the stability, graspability, assemblability, as well as the need for a second arm. Especially, the need for a second arm is considered when supports from worktables and other workpieces are not enough to produce a stable assembly.

[ Harada Lab ]

Semantic grasping is the problem of selecting stable grasps that are functionally suitable for specific object manipulation tasks. In order for robots to effectively perform object manipulation, a broad sense of contexts, including object and task constraints, needs to be accounted for. We introduce the Context-Aware Grasping Engine, which combines a novel semantic representation of grasp contexts with a neural network structure based on the Wide & Deep model, capable of capturing complex reasoning patterns. We quantitatively validate our approach against three prior methods on a novel dataset consisting of 14,000 semantic grasps for 44 objects, 7 tasks, and 6 different object states. Our approach outperformed all baselines by statistically significant margins, producing new insights into the importance of balancing memorization and generalization of contexts for semantic grasping. We further demonstrate the effectiveness of our approach on robot experiments in which the presented model successfully achieved 31 of 32 suitable grasps.

[ RAIL Lab ]

I’m not totally convinced that bathroom cleaning is an ideal job for autonomous robots at this point, just because of the unstructured nature of a messy bathroom (if not of the bathroom itself). But this startup is giving it a shot anyway.

The cost target is $1,000 per month.

[ Somatic ] via [ TechCrunch ]

IHMC is designing, building, and testing a mobility assistance research device named Quix. The main function of Quix is to restore mobility to those stricken with lower limb paralysis. In order to achieve this the device has motors at the pelvis, hips, knees, and ankles and an onboard computer controlling the motors and various sensors incorporated into the system.

[ IHMC ]

In this major advance for mind-controlled prosthetics, U-M research led by Paul Cederna and Cindy Chestek demonstrates an ultra-precise prosthetic interface technology that taps faint latent signals from nerves in the arm and amplifies them to enable real-time, intuitive, finger-level control of a robotic hand.

[ University of Michigan ]

Coral reefs represent only 1% of the seafloor, but are home to more than 25% of all marine life. Reefs are declining worldwide. Yet, critical information remains unknown about basic biological, ecological, and chemical processes that sustain coral reefs because of the challenges to access their narrow crevices and passageways. A robot that grows through its environment would be well suited to this challenge as there is no relative motion between the exterior of the robot and its surroundings. We design and develop a soft growing robot that operates underwater and take a step towards navigating the complex terrain of a coral reef.

[ UCSD ]

What goes on inside those package lockers, apparently.

[ Dorabot ]

In the future robots could track the progress of construction projects. As part of the MEMMO H2020 project, we recently carried out an autonomous inspection of the Costain High Speed Rail site in London with our ANYmal robot, in collaboration with Edinburgh Robotics.

[ ORI ]

Soft Robotics technology enables seafood handling at high speed even with amorphous products like mussels, crab legs, and lobster tails.

[ Soft Robotics ]

Pepper and Nao had a busy 2019:

[ SoftBank Robotics ]

Chris Atkeson, a professor at the Robotics Institute at Carnegie Mellon University, watches a variety of scenes featuring robots from movies and television and breaks down how accurate their depictions really are. Would the Terminator actually have dialogue options? Are the "three laws" from I, Robot a real thing? Is it actually hard to erase a robot’s memory (a la Westworld)?

[ Chris Atkeson ] via [ Wired ]

This week’s CMU RI Seminar comes from Anca Dragan at UC Berkeley, on “Optimizing for Coordination With People.”

From autonomous cars to quadrotors to mobile manipulators, robots need to co-exist and even collaborate with humans. In this talk, we will explore how our formalism for decision making needs to change to account for this interaction, and dig our heels into the subtleties of modeling human behavior — sometimes strategic, often irrational, and nearly always influenceable. Towards the end, I’ll try to convince you that every robotics task is actually a human-robot interaction task (its specification lies with a human!) and how this view has shaped our more recent work.

[ CMU RI ]

When the group of high schoolers arrived for the coding camp, the idea of spending the day staring at a computer screen didn’t seem too exciting to them. But then Pepper rolled into the room.

“All of a sudden everyone wanted to become a robot coder,” says Kass Dawson, head of marketing and business strategy at SoftBank Robotics America, in San Francisco. He saw the same thing happen in other classrooms, where the friendly humanoid was an instant hit with students.

“What we realized very quickly was, we need to take advantage of the fact that this robot can get kids excited about computer science,” Dawson says.

Today SoftBank is launching Tethys, a visual programming tool designed to teach students how to code by creating applications for Pepper. The company is hoping that its humanoid robot, which has been deployed in homes, retail stores, and research labs, can also play a role in schools, helping to foster the next generation of engineers and roboticists.

Tethys is based on an intuitive, graphical approach to coding. To create a program, you drag boxes (representing different robot behaviors) on the screen and connect them with wires. You can run your program instantly on a Pepper to see how it works. You can also run it on a virtual robot on the screen.

As part of a pilot program, more than 1,000 students in about 20 public schools in Boston, San Francisco, and Vancouver, Canada, are already using the tool. SoftBank plans to continue expanding to more locations. (Educators interested in bringing Tethys and Pepper to their schools should reach out to the company by email.)

Bringing robots to the classroom

The idea of using robots to teach coding, logic, and problem-solving skills is not new (in fact, in the United States it goes back nearly half a century). Lego robotics kits like Mindstorms, Boost, and WeDo are widely used in STEM education today. Other popular robots and kits include Dash and Dot, Cubelets, Sphero, VEX, Parallax, and Ozobot. Last year, iRobot acquired Root, a robotics education startup founded by Harvard researchers.

Photo: SoftBank Robotics Using the Tethys visual programming tool, students can program Pepper to move, gesticulate, talk, and display graphics on its tablet. They can run their programs on a real robot or a virtual one on their computers.

So SoftBank is entering a crowded market, although one that has a lot of growth potential. And to be fair, SoftBank is not entirely new to the educational space—its experience goes back to the acquisition of French company Aldebaran Robotics, whose Nao humanoid has long been used in classrooms. Pepper, also originally developed by Aldebaran, is Nao’s newer, bigger sibling, and it, too, has been used in classrooms before.

Pepper’s size is probably one of its main advantages over the competition. It’s a 1.2-meter tall humanoid that can move around a room, dance, and have conversations and play games with people—not just a small wheeled robot beeping and driving on a tabletop.

On the other hand, Pepper’s size also means it costs several times as much as those other robots. That’s a challenge if SoftBank wants to get lots of them out to schools, which may not be able to afford them. So far the company has addressed the issue by donating Peppers—over 100 robots in the past two years.

How Tethys work

When SoftBank first took Pepper to classrooms, it discovered that the robot’s original software development platform, called Choregraphe, wasn’t designed as an educational tool. It was hard to use by non engineers, and was glitchy. SoftBank then partnered with Finger Food Advanced Technology Group, a Vancouver-based software company, to develop Tethys.

Image: SoftBank Robotics While Tethys is based on a visual programming environment, students can inspect the underlying Python scripts and modify them or write their own code.

Tethys is an integrated development environment, or IDE, that runs on a web browser (it works on regular laptops and also Chromebooks, popular in schools). It features a user-friendly visual programming interface, and in that sense it is similar to other visual programming languages like Blockly and Scratch.

But students aren’t limited to dragging blocks and wires on the screen; they can inspect the underlying Python scripts and modify them, or write their own code.

SoftBank says the new initiative is focused on “STREAM” education, or Science, Technology, Robotics, Engineering, Art, and Mathematics. Accordingly, Tethys is named after the Greek Titan goddess of streams, says SoftBank’s Dawson, who heads its STREAM Education program.

“It’s really important to make sure that more people are getting involved in robotics,” he says, “and that means not just the existing engineers who are out there, but trying to encourage the engineers of the future.”

Today, Boston Dynamics and OTTO Motors (a division of Clearpath Robotics) are announcing a partnership to “coordinate mobile robots in the warehouse” as part of “the future of warehouse automation.” It’s a collaboration between OTTO’s autonomous mobile robots and Boston Dynamics’s Handle, showing how a heterogeneous robot team can be faster and more efficient in a realistic warehouse environment.

As much as we love Handle, it doesn’t really seem like the safest robot for humans to be working around. Its sheer size, dynamic motion, and heavy payloads mean that the kind of sense-and-avoid hardware and software you’d really want to have on it for humans to able to move through its space without getting smushed would likely be impractical, so you need another way of moving stuff in an out of its work zone. The Handle logistics video Boston Dynamics released about a year ago showed the robot working mostly with conveyor belts, but that kind of fixed infrastructure may not be ideal for warehouses that want to remain flexible.

This is where OTTO Motors comes in—its mobile robots (essentially autonomous mobile cargo pallets) can safely interact with Handles carrying boxes, moving stuff from where the Handles are working to where it needs to go without requiring intervention from a fragile and unpredictable human who would likely only get in the way of the whole process. 

From the press release:

“We’ve built a proof of concept demonstration of a heterogeneous fleet of robots building distribution center orders to provide a more flexible warehouse automation solution,” said Boston Dynamics VP of Product Engineering Kevin Blankespoor. “To meet the rates that our customers expect, we’re continuing to expand Handle’s capabilities and optimizing its interactions with other robots like the OTTO 1500 for warehouse applications.”

This sort of suggests that OTTO Motors might not be the only partner that Boston Dynamics is working with. There are certainly other companies who make autonomous mobile robots for warehouses like OTTO does, but it’s more fun to think about fleets of warehouse robots that are as heterogeneous as possible: drones, blimps, snake robots, hexapods—I wouldn’t put anything past them.

[ OTTO Motors ]

For the past two weeks, teams of robots (and their humans) have been exploring an unfinished nuclear power plant in Washington State as part of DARPA’s Subterranean Challenge. The SubT Challenge consists of three separate circuits, each representing a distinct underground environment: tunnel systems, urban underground, and cave networks.

The Urban Circuit portion of the challenge ended last Thursday, and DARPA live streamed all of the course runs and put together some great video recaps of the competition itself. But that footage represents just a small portion of what actually went on at the challenge, as teams raced to implement fixes and improvements in hardware and software in between runs, often staying up all night in weird places trying to get their robots to work better (or work at all).

We visited the SubT Urban Challenge during the official media day last week, and also spent some time off-site with the teams themselves, as they solved problems and tested their robots wherever they could, from nearby high schools to empty malls to hotel stairwells at 5 a.m. 

And the winner of the SubT Urban Circuit is...

The winner of the SubT Urban Circuit was Team CoSTAR, a collaboration between NASA JPL, MIT, Caltech, KAIST, LTU, and industry partners, including Clearpath Robotics and Boston Dynamics. Second place went to Carnegie Mellon’s Team Explorer, which took first at the previous SubT Tunnel Circuit six months ago, setting up a highly competitive Cave Circuit event which will take place six months from now.

We’ll have some more details on the teams’ final scores, but first here’s a brief post-challenge overview video from DARPA to get you caught up:

The Urban Circuit location: an unfinished nuclear power plant

The Urban Circuit of the DARPA Subterranean Challenge was held at the Satsop Nuclear Power Plant, about an hour and a half south of Seattle. 

Photo: DARPA Aerial photo of the unfinished Satsop nuclear power plant.

Started in 1977, the plant was about 80 percent complete when state funding fell through, and after nothing happened for a couple of decades, ownership was transferred to the Satsop Redevelopment Project to try and figure out how to turn the aging dystopian infrastructure into something useful. Something useful includes renting the space for people to film action movies, and for DARPA to host challenges.

The biggest difference between Tunnel and Urban is that while Tunnel was mostly, you know, tunnels (mostly long straight-ish passages connected with each other), Urban included a variety of large spaces and interconnected small rooms spread out across multiple levels. This is a 5-minute long walkthrough from DARPA that shows one of the course configurations; you don’t need to watch the whole thing, but it should give you a pretty good idea of the sort of environment that these robots had to deal with:

The biggest challenge: Communications, or stairs?

While communications were an enormous challenge at the Tunnel Circuit, from talking with the teams it sounded like comms was not nearly as much of an issue at Urban, because of a combination of a slightly friendlier environment (concrete walls instead of meters of solid rock) and teams taking comms very, very seriously as they prepared their systems for this event. More teams used deployable networking nodes to build up a mesh network as their robots progressed farther into the course (more on this later), and there was also more of an emphasis on fully autonomous exploration where robots were comfortable operating for extended periods outside of communication range completely. 

Photo: Evan Ackerman/IEEE Spectrum Team garages at the event. You can’t see how cold it is, but if you could, you’d understand why they’re mostly empty.

When we talked to DARPA SubT Program Manager Tim Chung a few weeks ago, he was looking forward to creating an atmosphere of warm camaraderie between teams:

I’m super excited about how we set up the team garages at the Urban Circuit. It’ll be like pit row, in a way that really highlights how much I value the interactions between teams, it’ll be an opportunity to truly capitalize on having a high concentration of enthusiastic and ambitious roboticists in one area. 

Another challenge: Finding a warm place to test the robots

Having all the teams gathered at their garages would have been pretty awesome, except that the building somehow functioned as a giant heat sink, and while it was in the mid-30s Fahrenheit outside, it felt like the mid-20s inside! Neither humans nor robots had any particular desire to spend more time in the garages than was strictly necessary—most teams would arrive immediately before the start of their run staging time, and then escape to somewhere warmer immediately after their run ended. 

It wasn’t just a temperature thing that kept teams out of the garages—to test effectively, most teams needed a lot more dedicated space than was available on-site. Teams understood how important test environments were after the Tunnel Circuit, and most of them scrounged up spaces well in advance. Team CSIRO DATA61 found an indoor horse paddock at the local fairgrounds. Team CERBERUS set up in an empty storefront in a half dead mall about 20 miles away. And Team CoSTAR took over the conference center at a local hotel, which turned out to be my hotel, as I discovered when I met Spot undergoing testing in the hallway outside of my room right after I checked in:

Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR’s Spot robot (on loan from Boston Dynamics) undergoing testing in a hotel hallway.

Spot is not exactly the stealthiest of robots, and the hotel testing was not what you’d call low-key. I can tell you that CoSTAR finished their testing at around 5:15 a.m., when Spot’s THUMP THUMP THUMP THUMP THUMPing gait woke up pretty much the entire hotel as the robot made its way back to its hotel room. Spot did do a very good job on the stairs, though:

Photo: Evan Ackerman/IEEE Spectrum Even with its top-heavy JPL autonomy and mapping payload, Spot was able to climb stairs without too much trouble.

After the early morning quadrupedal wake-up call, I put on every single layer of clothing I’d brought and drove up to the competition site for the DARPA media day. We were invited to watch the beginning of a few competition runs, take a brief course tour (after being sworn to secrecy), and speak with teams at the garages before and after their runs. During the Tunnel circuit, I’d focused on the creative communications strategies that each team was using, but for Urban, I asked teams to tell me about some of the clever hacks they’d come up with to solve challenges specific to the Urban circuit.

Here’s some of what teams came up with:


Team NCTU from Taiwan has some of the most consistently creative approaches to the DARPA SubT courses we’ve seen. They’re probably best known for their “Duckiefloat” blimps, which had some trouble fitting through narrow tunnels during the Tunnel circuit six months ago. Knowing that passages would be even slimmer for the Urban Circuit, NCTU built a carbon fiber frame around the Duckiefloats to squish their sides in a bit.

Photo: Evan Ackerman/IEEE Spectrum Duckiefloat is much slimmer (if a bit less pleasingly spherical) thanks to a carbon fiber framework that squeezes it into a more streamlined shape to better fit through narrow corridors.

NCTU also added millimeter wave radar to one of the Duckiefloats as a lighter substitute for on-board lidar or RGBD cameras, and had good results navigating with the radar alone, which (as far as I know) is a totally unique approach. We will definitely be seeing more of Duckiefloat for the cave circuit.

Photo: Evan Ackerman/IEEE Spectrum NCTU’s Anchorball droppable WiFi nodes now include a speaker, which the Husky UGV can localize with microphone arrays (the black circle with the white border).

At Tunnel, NCTU dropped mesh WiFi nodes that doubled as beacons, called Anchorballs. For Urban, the Anchorballs are 100 percent less ball-like, and incorporate a speaker, which plays chirping noises once deployed. Microphone arrays on the Husky UGVs can localize this chirping, allowing multiple robots to use the nodes as tie points to coordinate their maps.

Photo: Evan Ackerman/IEEE Spectrum NCTU is developing mobile mesh network nodes in the form of autonomous robot balls.

Also under development at NCTU is this mobile Anchorball, which is basically a big Sphero with a bunch of networking gear packed into it that can move itself around to optimize signal strength.


Team NUS SEDS accidentally burned out a couple of the onboard computers driving their robots. The solution was to run out and buy a laptop, and then 3D print some mounts to attach the laptop to the top of the robot and run things from there.

Photo: Evan Ackerman/IEEE Spectrum When an onboard computer burned out, NUS SEDS bought a new laptop to power their mobile robot, because what else are you going to do?

They also had a larger tracked vehicle that was able to go up and down stairs, but it got stuck in customs and didn’t make it to the competition at all.

Team Explorer

Team Explorer did extensive testing in an abandoned hospital in Pittsburgh, which I’m sure wasn’t creepy at all. While they brought along some drones that were used very successfully, getting their beefy wheeled robots up and down stairs wasn’t easy. To add some traction, Explorer cut chunks out of the wheels on one of their robots to help it grip the edges of stairs. 

Photo: Evan Ackerman/IEEE Spectrum Team Explorer’s robot has wedges cut out of its wheels to help it get a grip on stairways.

It doesn’t look especially sophisticated, but the team lead Sebastian Scherer told me that this was the result of 14 (!) iterations of wheel and track modifications. 


Six months ago, we checked out a bunch of different creative communications strategies that teams used at SubT Tunnel. MARBLE improved on their droppable wireless repeater nodes with a powered, extending antenna (harvested from a Miata, apparently).

Photo: Evan Ackerman/IEEE Spectrum After being dropped from its carrier robot, this mesh networking node extends its antennas half a meter into the air to maximize signal strength.

This is more than just a neat trick: We were told that the extra height that the antennas have once fully deployed does significantly improve their performance.

Team Robotika

Based on their experience during the Tunnel Circuit, Team Robotika decided that there was no such thing as having too much light in the tunnels, so they brought along a robot with the most enormous light-to-robot ratio that we saw at SubT.

Photo: Evan Ackerman/IEEE Spectrum No such thing as too much light during DARPA SubT.

Like many other teams, Robotika was continually making minor hardware adjustments to refine the performance of their robots and make them more resilient to the environment. These last-minute plastic bumpers would keep the robot from driving up walls and potentially flipping itself over.

Photo: Evan Ackerman/IEEE Spectrum A bumper hacked together from plastic and duct tape keeps this robot from flipping itself over against walls. Team CSIRO Data61

I met CSIRO Data61 (based in Australia) at the testing location they’d found in a building at the Grays Harbor County Fairgrounds, right next to an indoor horse arena that provided an interesting environment, especially for their drones. During their first run, one of their large tracked robots (an ex-police robot called Titan) had the misfortune to get its track caught on an obstacle that was exactly the wrong size, and it burned out a couple motors trying to get free.

Photo: Evan Ackerman/IEEE Spectrum A burned motor, crispy on the inside.

You can practically smell that through the screen, right? And these are fancy Maxon motors, which you can’t just pick up at your local hardware store. CSIRO didn’t have spares with them, so the most expedient way to get new motors that were sure to work turned out to be flying another team member over from Australia (!) with extra motors in their carry-on luggage. And by Tuesday morning, the Titan was up and running again.

Photo: Evan Ackerman/IEEE Spectrum A fully operational Titan beside a pair of commercial SuperDroid robots at CSIRO’s off-site testing area. Team CERBERUS

Team CERBERUS didn’t have a run scheduled during the SubT media day, but they invited me to visit their testing area in an empty store next to an Extreme Fun Center in a slightly depressing mall in Aberdeen (Kurt Cobain’s hometown), about 20 miles down the road from Satsop. CERBERUS was using a mix of wheeled vehicles, collision-tolerant drones, and ANYmal legged robots.

Photo: Evan Ackerman/IEEE Spectrum Team CERBERUS doing some off-site testing of their robots with the lights off.

CERBERUS had noticed during a DARPA course pre-briefing that the Alpha course had an almost immediate 90-degree turn before a long passage, which would block any directional antennas placed in the staging area. To try to maximize communication range, they developed this dumb antenna robot: Dumb in the sense that it has no sensing or autonomy, but instead is designed to carry a giant tethered antenna just around that first corner.

Photo: Evan Ackerman/IEEE Spectrum Basically just a remote-controlled directional antenna, CERBERUS developed this robot to extend communications from their base station around the first corner of Alpha Course.

Another communications challenge was how to talk to robots after they traversed down a flight of stairs. Alpha Course featured a flight of stairs going downwards just past the starting gate, and CERBERUS wanted a way of getting a mesh networking node down those stairs to be able to reliably talk to robots exploring the lower level. Here’s what they came up with:

Photo: Evan Ackerman/IEEE Spectrum A mesh network node inside of a foam ball covered in duct tape can be thrown by a human into hard-to-reach spots near the starting area.

The initial idea was to put a node into a soccer ball which would then be kicked from the staging area, off the far wall, and down the stairs, but they ended up finding some hemispheres of green foam used for flower arrangements at Walmart, hollowed them out, put in a node, and then wrapped the whole thing in duct tape. With the addition of a tether, the node in a ball could be thrown from the staging area into the stairwell, and brought back up with the tether if it didn’t land in the right spot.

Plan B for stairwell communications was a bit more of a brute force approach, using a directional antenna on a stick that could be poked out of the starting area and angled over the stairwell.

Photo: Evan Ackerman/IEEE Spectrum If your antenna balls don’t work? Just add a directional antenna to a stick.

Since DARPA did allow tethers, CERBERUS figured that this was basically just a sort of rigid tether. Sounds good to me!


Team CoSTAR surprised everyone by showing up to the SubT Urban Circuit with a pair of Spot quadrupeds from Boston Dynamics. The Spots were very much a last-minute addition to the team, and CoSTAR only had about six weeks to get them up and (metaphorically) running. Consequently, the Spots were a little bit overburdened with a payload that CoSTAR hasn’t had much of a chance to optimize. The payload takes care of all of the higher level autonomy and map making and stuff, while Spot’s own sensors handle the low-level motion planning. 

Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR’s Spot robots carried a payload that was almost too heavy for the robot to manage, and included sensors, lights, computers, batteries, and even two mesh network node droppers.

In what would be a spectacular coincidence were both of these teams not packed full of brilliant roboticists, Team CoSTAR independently came up with something very similar to the throwable network node that Team CERBERUS was messing around with.

Photo: Evan Ackerman/IEEE Spectrum A throwable mesh network node embedded in a foam ball that could be bounced into a stairwell to extend communications.

One of the early prototypes of this thing was a Mars lander-style “airbag” system, consisting of a pyramid of foam balls with a network node embedded in the very center of the pile. They showed me a video of this thing, and it was ridiculously cool, but they found that carving out the inside of a foam ball worked just as well and was far easier to manage.

There was only so much testing that CoSTAR was able to do in the hotel and conference center, since a better match for the Urban Circuit would be a much larger area with long hallways, small rooms, and multiple levels that could be reached by ramps and stairs. So every evening, the team and their robots drove 10 minutes down the road to Elma High School, which seemed to be just about the perfect place for testing SubT robots. CoSTAR very kindly let me tag along one night to watch their Huskies and Spots explore the school looking for artifacts, and here are some pictures that I took.

Photo: Evan Ackerman/IEEE Spectrum The Elma High School cafeteria became the staging area for Team CoSTAR’s SubT test course. Two Boston Dynamics Spot robots and two Clearpath Robotics Huskies made up CoSTAR’s team of robots. The yellow total station behind the robots is used for initial location calibration, and many other teams relied on them as well. Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR hid artifacts all over the school to test the robots’ ability to autonomously recognize and locate them. That’s a survivor dummy down the hall.

JPL put together this video of one of the test runs, which cuts out the three hours of setup and calibration and condenses all the good stuff into a minute and a half:

DARPA SubT Urban Circuit: Final scores

In their final SubT Urban run, CoSTAR scored a staggering 9 points, giving them a total of 16 for the Urban Circuit, 5 more than Team Explorer, which came in second. Third place went to Team CTU-CRAS-NORLAB, and as a self-funded (as opposed to DARPA-funded) team, they walked away with a $500,000 prize.

Image: DARPA DARPA SubT Urban Circuit final scores.

Six months from now, all of these teams will meet again to compete at the SubT Cave Circuit, the last (and perhaps most challenging) domain that DARPA has in store. We don’t yet know exactly when or where Cave will take place, but we do know that we'll be there to see what six more months of hard work and creativity can do for these teams and their robots.

[ DARPA SubT Urban Results ]

Special thanks to DARPA for putting on this incredible event, and thanks also to the teams that let me follow them around and get (ever so slightly) in the way for a day or two.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

We’ll have more on the DARPA Subterranean Challenge Urban Circuit next week, but here’s a quick compilation from DARPA of some of the competition footage.

[ SubT ]

ABB set up a global competition in 2019 to assess 20 leading AI technology start-ups on how they could approach solutions for 26 real-world picking, packing and sorting challenges. The aim was to understand if AI is mature enough to fully unlock the potential for robotics and automation. ABB was also searching for a technology partner to co-develop robust AI solutions with. Covariant won the challenge by successfully completing each of the 26 challenges; on February 25, ABB and Covariant announced a partnership to bring AI-enabled robotic solutions to market.

We wrote about Covariant and its AI-based robot picking system last month. The most interesting part of the video above is probably the apple picking, where the system has to deal with irregular, shiny, rolling objects. The robot has a hard time picking upside-down apples, and after several failures in a row, it nudges the last one to make it easier to pick up. Impressive! And here’s one more video of real-time picking mostly transparent water bottles:

[ Covariant ]

Osaka University’s Affetto robot, which we’ve written about before, is looking somewhat more realistic than when we first wrote about it.

Those are some weird noises that it’s making though, right? Affetto, as it turns out, also doesn’t like getting poked in its (disembodied) tactile sensor:

They’re working on a body for it, too:

[ Osaka University ]

University of Washington students reimagine today’s libraries.

[ UW ]

Thanks Elcee!

Astrobee will be getting a hand up on the ISS, from Columbia’s ROAM Lab.

I think this will be Astrobee’s second hand, in addition to its perching arm. Maybe not designed for bimanual tasks, but still, pretty cool!

[ ROAM Lab ]

In this paper, we tackle the problem of pushing piles of small objects into a desired target set using visual feedback. Unlike conventional single-object manipulation pipelines, which estimate the state of the system parametrized by pose, the underlying physical state of this system is difficult to observe from images. Thus, we take the approach of reasoning directly in the space of images, and acquire the dynamics of visual measurements in order to synthesize a visual-feedback policy.

[ MIT ]

In this project we are exploring ways of interacting with terrain using hardware already present on exploration rovers - wheels! By using wheels for manipulation, we can expand the capabilities of space robots without the need for adding hardware. Nonprehensile terrain manipulation can be used many applications such as removing soil to sample below the surface or making terrain easier to cross for another robot. Watch until the end to see MiniRHex and the rover working together!

[ Robomechanics Lab ]

Dundee Precious Metals reveals how Exyn’s fully autonomous aerial drones are transforming their cavity monitoring systems with increased safety and maximum efficiency.

[ Exyn ]

Thanks Rachel!

Dragonfly is a NASA mission to explore the chemistry and habitability of Saturn’s largest moon, Titan. The fourth mission in the New Frontiers line, Dragonfly will send an autonomously-operated rotorcraft to visit dozens of sites on Titan, investigating the moon’s surface and shallow subsurface for organic molecules and possible biosignatures.

Dragonfly is scheduled to launch in 2026 and arrive at Titan in 2034.

[ NASA ]

Researchers at the Max Planck Institute for Intelligent Systems in Stuttgart in cooperation with Tampere University in Finland developed a gel-like robot inspired by sea slugs and snails they are able to steer with light. Much like the soft body of these aquatic invertebrates, the bioinspired robot is able to deform easily inside water when exposed to this energy source.

Due to specifically aligned molecules of liquid crystal gels – its building material – and illumination of specific parts of the robot, it is able to crawl, walk, jump, and swim inside water. The scientists see their research project as an inspiration for other roboticists who struggle to design untethered soft robots that are able to move freely in a fluidic environment.

[ Max Planck Institute ]

Forests are a very challenging environment for drones, especially if you want to both avoid and map trees at the same time.

[ Kumar Lab ]

Some highlights from the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) that took place in Abu Dhabi, UAE last week.


I never get tired of hearing technical presentations from Skydio, and here’s Ryan Kennedy giving at talk at the GRASP Lab.

The technology for intelligent and trustworthy navigation of autonomous UAVs has reached an inflection point to provide transformative gains in capability, efficiency, and safety to major industries. Drones are starting to save lives of first responders, automate dangerous infrastructure inspection, digitize the physical world with millimeter precision, and capture Hollywood quality video - all on affordable consumer hardware.

At Skydio, we have invested five years of R&D in the ability to handle difficult unknown scenarios in real-time based on visual sensing, and shipped two generations of fully autonomous drone. In this talk, I will discuss the close collaboration of geometry, learning, and modeling within our system, our experience putting robots into production, and the challenges still ahead.

[ Skydio ]

This week’s CMU RI Seminar comes from Sarjoun Skaff at Bossa Nova Robotics: “Yes, That’s a Robot in Your Grocery Store. Now what?”

Retail stores are becoming ground zero for indoor robotics. Fleet of different robots have to coexist with each others and humans every day, navigating safely, coordinating missions, and interacting appropriately with people, all at large scale. For us roboticists, stores are giant labs where we’re learning what doesn’t work and iterating. If we get it right, it will serve as an example for other industries, and robots will finally become ubiquitous in our lives.

[ CMU RI ]

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Artificial intelligence and robotic technologies with semi-autonomous learning, reasoning, and decision-making capabilities are increasingly being incorporated into defense, military, and security systems. Unsurprisingly, there is increasing concern about the stability and safety of these systems. In a different sector, runaway interactions between autonomous trading systems in financial markets have produced a series of stock market “flash crashes,” and as a result, those markets now have rules to prevent such interactions from having a significant impact1.

Could the same kinds of unexpected interactions and feedback loops lead to similar instability with defense or security AIs?

Adversarial attacks on AI systems

General concerns about the impacts of defense AIs and robots on stability, whether in isolation or through interaction, have only been exacerbated by recent demonstrations of adversarial attacks against these systems2. Perhaps the most widely-discussed attack cases involve image classification algorithms that are deceived into “seeing” images in noise3, or are easily tricked by pixel-level changes so they classify, say, a turtle as a rifle4. Similarly, game-playing systems that outperform any human (e.g., AlphaGo) can suddenly fail if the game structure or rules are even slightly altered in ways that would not affect a human5. Autonomous vehicles that function reasonably well in ordinary conditions can, with the application of a few pieces of tape, be induced to swerve into the wrong lane or speed through a stop sign6. And the list of adversarial attacks continues to grow and grow over time.

Adversarial attacks pose a tangible threat to the stability and safety of AI and robotic technologies. The exact conditions for such attacks are typically quite unintuitive for humans, so it is difficult to predict when and where the attacks could occur. And even if we could estimate the likelihood of an adversarial attack, the exact response of the AI system can be difficult to predict as well, leading to further surprises and less stable, less safe military engagements and interactions. Even overall assessments of reliability are difficult in the face of adversarial attacks.

We might hope that adversarial attacks would be relatively rare in the everyday world, since “random noise” that targets image classification algorithms is actually far from random: The tape on the stop sign must be carefully placed, the pixel-level perturbations added to the image must be carefully calculated, and so on. Significant effort is required to construct an adversarial attack, and so we might simply deploy our AI and robotic systems with the hope that the everyday world will not conspire to deceive them. 

Unfortunately, this confidence is almost certainly unwarranted for defense or security technologies. These systems will invariably be deployed in contexts where the other side has the time, energy, and ability to develop and construct exactly these types of adversarial attacks. AI and robotic technologies are particularly appealing for deployment in enemy-controlled or enemy-contested areas since those environments are riskiest for our human soldiers, in large part because the other side has the most control over the environment. 

Defenses against adversarial attacks

Although adversarial attacks on defense and military AIs and robots are likely, they are not necessarily destabilizing, particularly since humans are typically unaffected by these attacks. We can easily recognize that a turtle is not a rifle even with random noise, we view tape on a stop sign as an annoyance rather than something that disrupts our ability to follow the rules of the road, and so on. Of course, there are complexities, but we can safely say that human performance is strongly robust to adversarial attacks against AIs. Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

This recommendation is attractive, but is also necessarily limited in scope to applications where a human can be directly involved. In the case of intelligence, surveillance, and reconnaissance (ISR) systems, however, substantive human interaction might not be possible. AI technologies are being increasingly used to handle the enormous volumes of data generated for ISR purposes. AI technologies for ISR now play a significant role in the creation and maintenance of situational awareness for human decision-makers, and in such situations, the destabilizing risks of adversarial attacks again rear their heads.

As an extreme example, consider the intersection of AI and nuclear weapons. One might think that these two technologies should never meet, since we ought not delegate the decision to use nuclear force to an AI. Regardless, AI systems potentially (or perhaps actually) do play a role in nuclear weapons, namely in the ISR that informs human decisions about whether to use such weapons. The worldwide sensor and data input streams almost certainly cannot be processed entirely by human beings. We will need to use (or perhaps already do use) AI technologies without a human in the loop to help us understand our world, and so there may not always be a human to intercept adversarial attacks against those systems.

Our situational awareness can therefore be affected or degraded due to deliberately distorted “perceptions” coming from the AI analyses. These problems are not limited to the extreme case of nuclear weapons—any military or security action where situational awareness depends partly on unmonitored ISR AI will be vulnerable to adversarial attacks in ways that a human cannot necessarily recognize and rectify.

Perhaps we could simply monitor the ISR AI by requiring it to provide evidence or explanations of its analyses that are sufficiently detailed for a human to be able to recognize an adversarial attack. However, if we consider only “explainable AIs” in these contexts, then we are restricting the space of possible models, and so arguably9 placing an artificial upper bound on system performance. Moreover, many AI systems are moving some computation onto the sensors themselves to help overcome processing and memory constraints.

For example, AI on the sensors might perform anomaly detection, leaving a higher level AI system to process only potential outliers. These distributed systems might not be able to retain the evidence (for example, the original image) required for human recognition of an adversarial attack. And in real-world cases, we might not have the time to look at the evidence even if it were provided, and so would not be able to respond to destabilizing adversarial attacks on the ISR AI.

This all might seem to be much ado about nothing new: After all, information gathering has always been susceptible to deception, manipulation, and misinformation. But adversarial attacks can lead to completely bizarre and ridiculous (from a human perspective) behavior from an AI. No ordinary deception would ever lead a human intelligence officer to see a turtle as a rifle, and the use of ISR AI opens the door to much different types of deception, with much different results. Without proper understanding of these potential impacts, the world is likely to be a less stable and less safe place. 

Adversarial attacks can destabilize AI technologies, rendering them less safe, predictable, or reliable. However, we do not necessarily need to worry about them as direct attacks on the decision-making machinery of the system. Instead, we should worry about the corruption of human situational awareness through adversarial AI, which can be equally effective in undermining the safety, stability, and trust in the AI and robotic technologies.

David Danks is L.L. Thurstone professor of philosophy and psychology, and head of the department of philosophy, at Carnegie Mellon University. He is also the chief ethicist of CMU’s Block Center for Technology & Society; co-director of CMU’s Center for Informed Democracy and Social Cybersecurity (IDeaS); and an adjunct member of the Heinz College of Information Systems and Public Policy. His research interests are at the intersection of philosophy, cognitive science, and machine learning. Most recently, Danks has been examining the ethical, psychological, and policy issues around AI and robotics in transportation, healthcare, privacy, and security.


1. Serritella, D. M. (2010). “High speed trading begets high speed regulation: SEC response to flash crash, rash.” Illinois Journal of Law, Technology & Policy, 2010 (2).

2. Biggio, B. & Roli, F. (2018). “Wild patterns: Ten years after the rise of adversarial machine learning.” Pattern Recognition, 84, 317-331.

3. Nguyen, A., Yosinski, J., & Clune, J. (2015). “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 427-436).

4. Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2018). “Synthesizing robust adversarial examples.” In Proceedings of the 35th International Conference on Machine Learning (pp. 284-293).

5. Raghu, M., Irpan, A., Andreas, J., Kleinberg, R., Le, Q. V., & Kleinberg, J. (2018). “Can deep reinforcement learning solve Erdos-Selfridge-Spencer games?” Proceedings of ICML.

6. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018). “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1625-1634).

7. Rudin, C. (2018). “Please stop explaining black box models for high stakes decisions.” NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning. arXiv:1811.10154v2

This article was originally published on LinkedIn. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Build a rover, send it to the Moon, sell the movie rights.

That was our first business model at iRobot. Way back in 1990. We thought it would be how we’d first change the world. It’s ironic, of course, that through that model, changing the world meant sending a robot to another one. Sadly, that business model failed. And it wouldn’t be our last failed business model. Not by a long shot.

Photo: iRobot

Why? Because changing the world through robots, it turns out, is no easy task.

Perhaps the biggest challenge back when we started in 1990 was that there existed no rule book on how to do it. There weren’t many robots, let alone robot companies, let alone any kind of robot industry. We would have to build it. All of it.

Walking that path meant being comfortable with ambiguity, and comfortable with the knowledge that not everything we tried was going to work–at least not in the way we originally conceived. It was and continues to be the cost of inventing the future.

But walking that trying path also meant learning from our mistakes, dusting ourselves off, trying again, and eventually, yes, doing what we set out to do: Change the world through robots.

We’ve learned so much along the way–what have we learned in our 30-year journey building the robot industry?

Robots are hard

I’ve said it before, and I’ll say it again: When we first started iRobot we had to invent every element of the robot. Spatial navigation was a robot problem, voice recognition was a robot problem, machine vision was a robot problem, just to name a few. Back then, no one else had set out to solve these hard problems. Because so many of these problems existed, the robot industry, if it could be called that, moved in anti-dog years. Fortunately, times have changed and the ecosystem around the technologies that make robots possible is much richer… But back then… it was just us.

But even today, with a much larger ecosystem of bright minds solving for the hard tech problems, getting a robot to work successfully still means getting just the right mix of mechanical, electrical, and software engineering, connectivity, and data science into a robot form factor that people trust and want to invite into their home.

Photo: iRobot

Speaking of trust, therein lied another challenge. Even when we did invent a robot that worked extraordinarily well–Roomba–consumers simply didn’t believe a robot could do what we said Roomba was capable of. It turns out that the principal objection to purchasing a robot for much of the last 30 years is a lack of belief that it could possibly work.

For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. We were no exception: We tried 14 business models before we arrived at one that sustainably worked.

But that’s not all: Even when you build a robot right, you can still somehow build it wrong. We experienced this with Roomba. We built it to match the reliability standards of European upright vacuums, something of which we were very proud. Of course, we didn’t anticipate that our customers would run their Roomba once per day, rather than the once per week average the European standard set. And as the first generation of Roomba robots broke down two years ahead of schedule, we learned that reverse logistics, great customer service, and a generous return policy were a very important part of a good robot–as was the realization that we couldn’t compare usage to whatever traditional means of action a good robot might take the place of.

And yet while building a robot that was durable, that people wanted and trusted was hard enough, 30 years building robots has also taught us that…

Good business models are harder to build than good robots

Let’s state this one right off the bat: For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. It turns out that a business model is as important as the tech, but much more rarely found in a robot company. And for a long time we were no exception: We tried 14 business models before we arrived at one that sustainably worked.

Image: iRobot

But the tenuous nature of our business models did teach us the value of extending the runway for our business until we found one that worked. And how does one extend the runway most effectively? By managing risk.

It’s one of the great misunderstandings of entrepreneurship–that great entrepreneurs are risk takers. Great entrepreneurs are not great risk takers… they’re great risk managers. And this was something we at iRobot were and are exceptionally good at.

How did we manage risk early on? Through partnerships. The kind of partnership we looked for were ones in which there was a big company–one that had a lot of money, a channel to the marketplace, and knowledge of that marketplace, but for whatever reason lacked belief that they themselves were innovative. We were a small company with no money, but believed ourselves to have cool technology, and be highly capable of innovation.

Image: iRobot

What we’d do was give our partner, the big company, absolute control. By doing this, it allowed us to say that since they could cancel the partnership at any time, we needed them to cover our costs… which they did. But we also didn’t ask them to pay us profit upfront. By not having the pay profit upfront, it makes obvious that we’re sharing the value that the partnership would ultimately create, and in a worst-case scenario for our partner, if the partnership didn’t result in a successful product, they got very inexpensive high-quality research.

This “asymmetric strategic partnership” approach not only provided the funds needed to sustain our business when we didn’t have a sustainable business model–the “failure” of those partnerships actually led to our ultimate success. Why? Because…

Innovation and failure come hand-in-hand

While this is far from a groundbreaking realization, its applicability to iRobot is quite unique. Because for us to become successful, it turns out that we had to learn the lessons from failing to earn royalties on robot toys (business model #3), failing to license technology for industrial floor-cleaning robots (business model #8), and failing to sell land mine clearance robots (business model #11).

Image: iRobot

Why? Because #3 taught us to manufacture at scale, #8 taught us how to clean floors, and #11 taught us how to navigate and cover large spaces.  All of which gave us the knowledge and capability to build… Roomba.

Image: iRobot Yes, you can change the world through robots

We did. In more ways the one. We changed the world by eliminating the need for people to vacuum the house themselves. By IPOing, we showed that a robotics company could be successful–which gave investors more reason to put money into robotics companies around the world.

But perhaps the most important way we’ve succeeded in changing the world is by making robots a daily reality for it. And how do we know that robots are now a reality? Because for the better part of the first 30 years of iRobot, what people said to me about robots–and Roomba specifically–was, “I can’t believe it actually works.”

But now, the question they ask me is, “Why can’t robots do more?

It is a great question. And that is what the next 30 years of iRobot will be about.

Colin Angle is chairman of the board, chief executive officer, and founder of iRobot. Celebrating its 30th year, iRobot has grown from an MIT startup to become a global leader in consumer robots, with more than 30 million sold worldwide. You can follow him on Twitter at @ColinAngle.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.

I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.

AI deception: How to define it?

If we take a rather long view of history, deception may be as old as the world itself, and it is certainly not the sole provenance of human beings. Adaptation and evolution for survival with traits like camouflage are deceptive acts, as are forms of mimicry commonly seen in animals. But pinning down exactly what constitutes deception for an AI agent is not an easy task—it requires quite a bit of thinking about acts, outcomes, agents, targets, means and methods, and motives. What we include or exclude in that calculation may then have wide ranging implications about what needs immediate regulation, policy guidance, or technological solutions. I will only focus on a couple of items here, namely intent and act type, to highlight this point.

What is deception? Bond and Robinson argue that deception is “false communication to the benefit of the communicator.”1 Whaley argues that deception is also the communication of information provided with the intent to manipulate another.2 These seem pretty straightforward approaches, except when you try to press on the idea of what constitutes “intent” and what is required to meet that threshold, as well as whether or not the false communication requires the intent to be explicitly beneficial to the deceiver. Moreover, depending on which stance you take, deception for altruistic reasons may be excluded entirely. Imagine if you asked your AI-enabled robot butler, “How do I look?” To which it answers, “Very nice.”

Let’s start with intent. Intent requires a theory of mind, meaning that the agent has some understanding of itself, and that it can reason about other external entities and their intentions, desires, states, and potential behaviors.3 If deception requires intent in the ways described above, then true AI deception would require an AI to possess a theory of mind. We might kick the can on that conclusion for a bit and claim that current forms of AI deception instead rely on human intent—where some human is using AI as a tool or means to carry out that person’s intent to deceive.

Or, we may not: Just because current AI agents lack a theory of mind doesn’t mean that they cannot learn to deceive. In multi-agent AI systems, some agents can learn deceptive behaviors without having a true appreciation or comprehension of what “deception” actually is. This could be as simple as hiding resources or information, or providing false information to achieve some goal. If we then put aside the theory of mind for the moment and instead posit that intention is not a prerequisite for deception and that an agent can unintentionally deceive, then we really have opened the aperture for existing AI agents to deceive in many ways. 

What about the way in which deception occurs? That is, what are the deceptive act types? We can identify two broad categories here: 1) acts of commission, where an agent actively engages in a behavior like sending misinformation; and 2) acts of omission, where an agent is passive but may be withholding information or hiding. AI agents can learn all sorts of these types of behaviors given the right conditions.4 Just consider how AI agents used for cyber defense may learn to signal various forms of misinformation, or how swarms of AI-enabled robotic systems could learn deceptive behaviors on a battlefield to escape adversary detection. In more pedestrian examples, perhaps a rather poorly specified or corrupted AI tax assistant omits various types of income on a tax return to minimize the likelihood of owing money to the relevant authorities.

Preparing ourselves against AI deception

The first step towards preparing for our coming AI future is to recognize that such systems already do deceive, and are likely to continue to deceive. How that deception occurs, whether it is a desirable trait (such as with our adaptive swarms), and whether we can actually detect when it is occurring are going to be ongoing challenges. Once we acknowledge this simple but true fact, we can begin to undergo the requisite analysis of what exactly constitutes deception, whether and to whom it is beneficial, and how it may pose risks. 

This is no small task, and it will require not only interdisciplinary work from AI experts, but also input from sociologists, psychologists, political scientists, lawyers, ethicists, and policy wonks. For military AI systems, it will also require domain and mission knowledge. In short, developing a comprehensive framework for AI deception is a crucial step if we are not to find ourselves on the back foot. 

We need to begin thinking about how to engineer novel solutions to mitigate unwanted deception by AI agents. This goes beyond current detection research, and requires thinking about environments, optimization problems, and how AI agents model other AI agents and their emergent effects could yield undesirable deceptive behaviors.

Furthermore, once this framework is in place, we need to begin thinking about how to engineer novel solutions to identify and mitigate unwanted deception by AI agents. This goes beyond current detection research, and moving forward requires thinking about environments, optimization problems, and how AI agents model other AI agents and their interactive or emergent effects could yield risky or undesirable deceptive behaviors. 

We presently face a myriad of challenges related to AI deception, and these challenges are only going to increase as the cognitive capacities of AI increase. The desire of some to create AI systems with a rudimentary theory of mind and social intelligence is a case in point to be socially intelligent one must be able to understand and to “manage” the actions of others5, and if this ability to understand another’s feelings, beliefs, emotions, and intentions exists, along with the ability to act to influence those feelings, beliefs, or actions, then deception is much more likely to occur.

However, we do not need to wait for artificial agents to possess a theory of mind or social intelligence for deception with and from AI systems. We should instead begin thinking about potential technological, policy, legal, and ethical solutions to these coming problems before AI gets more advanced than it already is. With a clearer understanding of the landscape, we can analyze potential responses to AI deception, and begin designing AI systems for truth.

Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.


1. Bond CF, Robinson M (1988), “The evolution of deception.” J Nonverbal Behav 12(4):295–307. Note also that this definition precludes certain forms of deception from altruistic or paternalistic reasons.

2. B. Whaley, “Toward a general theory of deception,” Journal of Strategic Studies, vol. 5, no. 1, pp. 178–192, Mar. 1982. 

3. Cheney DL, Seyfarth RM, “Baboon metaphysics: the evolution of a social mind.” University of Chicago Press, Chicago, 2008.

4. J. F. Dunnigan and A. A. Nofi, “Victory and deceit, 2nd edition: Deception and trickery in war,” Writers Press Books, 2001. J. Shim and R.C. Arkin, “A Taxonomy of Robot Deception and Its Benefits in HRI” IEEE International Conference on Systems, Man, and Cybernetics, 2013. S. Erat and U. Gneezy, “White lies,” Rady Working paper, Rady School of Management, UC San Diego, 2009. N. C. Rowe, “Designing good deceptions in defense of information systems,” in Proceedings of the 20th Annual Computer Security Applications Conference, ser. ACSAC ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 418–427.

5. E.L. Thorndike. “Intelligence and Its Use.” Harpers Magazine, Vol. 140, 1920: p. 228. Thorndike’s early definition of social intelligence has been widely used and updated for the past 100 years. Even current attempts in cognitive science have looked at separating out the tasks of “understanding” and “acting,” which maps directly to Thorndike’s language of “understand” and “manage”. Cf: M.I. Brown, A. Ratajska, S.L. Hughes, J.B. Fishman, E. Huerta, and C.F. Chabris. “The Social Shape Tests: A New Measure of Social Intelligence, Mentalizing and Theory of Mind.” Personality and Individual Differences, vol. 143, 2019: 107-117.