Feed aggregator

aside.inlay.CoronaVirusCoverage.xlrg { font-family: "Helvetica", sans-serif; text-transform: uppercase; text-align: center; border-width: 4px 0; border-top: 2px solid #666; border-bottom: 2px solid #666; padding: 10px 0; font-size: 18px; font-weight: bold; } span.LinkHereRed { color: #cc0000; text-transform: uppercase; font-family: "Theinhardt-Medium", sans-serif; }

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

NASA Curiosity Project Scientist Ashwin Vasavada guides this tour of the rover’s view of the Martian surface. Composed of more than 1,000 images and carefully assembled over the ensuing months, the larger version of this composite contains nearly 1.8 billion pixels of Martian landscape.

This panorama showcases "Glen Torridon," a region on the side of Mount Sharp that Curiosity is exploring. The panorama was taken between Nov. 24 and Dec. 1, 2019, when the Curiosity team was out for the Thanksgiving holiday. Since the rover would be sitting still with few other tasks to do while it waited for the team to return and provide its next commands, the rover had a rare chance to image its surroundings several days in a row without moving.

[ MSL ]

Sarcos has been making progress with its Guardian XO powered exoskeleton, which we got to see late last year in prototype stage:

The Sarcos Guardian XO full-body, powered exoskeleton is a first-of-its-kind wearable robot that enhances human productivity while keeping workers safe from strain or injury. Set to transform the way work gets done, the Guardian XO exoskeleton augments operator strength without restricting freedom of movement to boost productivity while dramatically reducing injuries.

[ Sarcos ]

Professor Hooman Samani, director of the Artificial Intelligence and Robotics Technology Laboratory (AIART Lab) at National Taipei University, Taiwan, writes in to share some ideas on how robots could be used to fight the coronavirus outbreak. 

Time is a critical issue when dealing with people affected by Coronavirus. Also due to the current emergency disaster, doctors could be far away from the patients. Additionally, avoiding direct contact with infected person is a medical priority. An immediate monitoring and treatment using specific kits must be administered to the victim. We have designed and developed the Ambulance Robot (AmbuBot) which could be a solution to address those issues. AmbuBot could be placed in various locations especially in busy, remote or quarantine areas to assist in above mentioned scenario. The AmbuBot also brings along an AED in a sudden event of cardiac arrest and facilitates various modes of operation from manual to semi-autonomous to autonomous functioning.

[ AIART Lab ]

IEEE Spectrum is interested in exploring how robotics and related technologies can help to fight the coronavirus (COVID-19) outbreak. If you are involved with actual deployments of robots to hospitals and high risk areas or have experience working with robots, drones, or other autonomous systems designed for this kind of emergency, please contact  IEEE Spectrum senior editor Erico Guizzo (e.guizzo@ieee.org) Click here for additional coronavirus coverage

Digit is launching later this month alongside a brand new sim that’s a 1:1 match to both the API and physics of the actual robot. Here, we show off the ability to train a learned policy against the validated physics of the robot. We have a LOT more to say about RL with real hardware... stay tuned.

Staying tuned!

Agility Robotics ]

This video presents simulations and experiments highlighting the functioning of the proposed Trapezium Line Theta* planner, as well as its improvements over our previous work namely the Obstacle Negotiating A* planner. First, we briefly present a comparison of our previous and new planners. We then show two simulations. The first shows the robot traversing an inclined corridor to reach a goal near the low-lying obstacle. This demonstrates the omnidirectional and any-angle motion planning improvement achieved by the new planner, as well as the independent planning for the front and back wheel pairs. The second simulation further demonstrates the key improvements mentioned above by having the robot traverse tight right-angled corridors. Finally, we present two real experiments on the CENTAURO robot. In the first experiment, the robot has to traverse into a narrow passage and then expand over a low lying obstacle. The second experiment has the robot first expand over a wide obstacle and then move into a narrow passage.

To be presented at ICRA 2020.

Dimitrios Kanoulas ]

We’re contractually obligated to post any video with “adverse events” in the title.

JHU ]

Waymo advertises their self-driving system in this animated video that features a robot car making a right turn without indicating. Also pretty sure that it ends up in the wrong lane for a little bit after a super wide turn and blocks a crosswalk to pick up a passenger. Oops!

I’d still ride in one, though.

Waymo ]

Exyn is building the world’s most advanced, autonomous aerial robots. Today, we launched our latest capability, Scoutonomy. Our pilotless robot can now ‘scout’ freely within a desired volume, such as a tunnel, or this parking garage. The robot sees the white boxes as ‘unknown’ space, and flies to explore them. The orange boxes are mapped obstacles. It also intelligently avoids obstacles in its path and identifies objects, such as people or cars. Scoutonomy can be used to safely and quickly finding survivors in natural, or man-made, disasters.

Exyn ]

I don’t know what soma blocks are, but this robot is better with them than I am.

This work presents a planner that can automatically find an optimal assembly sequence for a dual-arm robot to assemble the soma blocks. The planner uses the mesh model of objects and the final state of the assembly to generate all possible assembly sequence and evaluate the optimal assembly sequence by considering the stability, graspability, assemblability, as well as the need for a second arm. Especially, the need for a second arm is considered when supports from worktables and other workpieces are not enough to produce a stable assembly.

[ Harada Lab ]

Semantic grasping is the problem of selecting stable grasps that are functionally suitable for specific object manipulation tasks. In order for robots to effectively perform object manipulation, a broad sense of contexts, including object and task constraints, needs to be accounted for. We introduce the Context-Aware Grasping Engine, which combines a novel semantic representation of grasp contexts with a neural network structure based on the Wide & Deep model, capable of capturing complex reasoning patterns. We quantitatively validate our approach against three prior methods on a novel dataset consisting of 14,000 semantic grasps for 44 objects, 7 tasks, and 6 different object states. Our approach outperformed all baselines by statistically significant margins, producing new insights into the importance of balancing memorization and generalization of contexts for semantic grasping. We further demonstrate the effectiveness of our approach on robot experiments in which the presented model successfully achieved 31 of 32 suitable grasps.

[ RAIL Lab ]

I’m not totally convinced that bathroom cleaning is an ideal job for autonomous robots at this point, just because of the unstructured nature of a messy bathroom (if not of the bathroom itself). But this startup is giving it a shot anyway.

The cost target is $1,000 per month.

[ Somatic ] via [ TechCrunch ]

IHMC is designing, building, and testing a mobility assistance research device named Quix. The main function of Quix is to restore mobility to those stricken with lower limb paralysis. In order to achieve this the device has motors at the pelvis, hips, knees, and ankles and an onboard computer controlling the motors and various sensors incorporated into the system.

[ IHMC ]

In this major advance for mind-controlled prosthetics, U-M research led by Paul Cederna and Cindy Chestek demonstrates an ultra-precise prosthetic interface technology that taps faint latent signals from nerves in the arm and amplifies them to enable real-time, intuitive, finger-level control of a robotic hand.

[ University of Michigan ]

Coral reefs represent only 1% of the seafloor, but are home to more than 25% of all marine life. Reefs are declining worldwide. Yet, critical information remains unknown about basic biological, ecological, and chemical processes that sustain coral reefs because of the challenges to access their narrow crevices and passageways. A robot that grows through its environment would be well suited to this challenge as there is no relative motion between the exterior of the robot and its surroundings. We design and develop a soft growing robot that operates underwater and take a step towards navigating the complex terrain of a coral reef.

[ UCSD ]

What goes on inside those package lockers, apparently.

[ Dorabot ]

In the future robots could track the progress of construction projects. As part of the MEMMO H2020 project, we recently carried out an autonomous inspection of the Costain High Speed Rail site in London with our ANYmal robot, in collaboration with Edinburgh Robotics.

[ ORI ]

Soft Robotics technology enables seafood handling at high speed even with amorphous products like mussels, crab legs, and lobster tails.

[ Soft Robotics ]

Pepper and Nao had a busy 2019:

[ SoftBank Robotics ]

Chris Atkeson, a professor at the Robotics Institute at Carnegie Mellon University, watches a variety of scenes featuring robots from movies and television and breaks down how accurate their depictions really are. Would the Terminator actually have dialogue options? Are the "three laws" from I, Robot a real thing? Is it actually hard to erase a robot’s memory (a la Westworld)?

[ Chris Atkeson ] via [ Wired ]

This week’s CMU RI Seminar comes from Anca Dragan at UC Berkeley, on “Optimizing for Coordination With People.”

From autonomous cars to quadrotors to mobile manipulators, robots need to co-exist and even collaborate with humans. In this talk, we will explore how our formalism for decision making needs to change to account for this interaction, and dig our heels into the subtleties of modeling human behavior — sometimes strategic, often irrational, and nearly always influenceable. Towards the end, I’ll try to convince you that every robotics task is actually a human-robot interaction task (its specification lies with a human!) and how this view has shaped our more recent work.

[ CMU RI ]

When the group of high schoolers arrived for the coding camp, the idea of spending the day staring at a computer screen didn’t seem too exciting to them. But then Pepper rolled into the room.

“All of a sudden everyone wanted to become a robot coder,” says Kass Dawson, head of marketing and business strategy at SoftBank Robotics America, in San Francisco. He saw the same thing happen in other classrooms, where the friendly humanoid was an instant hit with students.

“What we realized very quickly was, we need to take advantage of the fact that this robot can get kids excited about computer science,” Dawson says.

Today SoftBank is launching Tethys, a visual programming tool designed to teach students how to code by creating applications for Pepper. The company is hoping that its humanoid robot, which has been deployed in homesretail stores, and research labs, can also play a role in schools, helping to foster the next generation of engineers and roboticists.

As part of a pilot program, more than 1,000 students in about 20 public schools in Boston, San Francisco, and Vancouver, Canada

Tethys is based on an intuitive, graphical approach to coding. To create a program, you drag boxes (representing different robot behaviors) on the screen and connect them with wires. You can run your program instantly on a Pepper to see how it works. You can also run it on a virtual robot on the screen.

As part of a pilot program, more than 1,000 students in about 20 public schools in Boston, San Francisco, and Vancouver, Canada, are already using the tool. SoftBank plans to continue expanding to more locations. (Educators interested in bringing Tethys and Pepper to their schools should reach out to the company by email.)

Bringing robots to the classroom

The idea of using robots to teach coding, logic, and problem-solving skills is not new (in fact, in the United States it goes back nearly half a century). Lego robotics kits like Mindstorms, Boost, and WeDo are widely used in STEM education today. Other popular robots and kits include Dash and Dot, Cubelets, Sphero, VEX, Parallax, and Ozobot. Last year, iRobot acquired Root, a robotics education startup founded by Harvard researchers.

So SoftBank is entering a crowded market, although one that has a lot of growth potential. And to be fair, SoftBank is not entirely new to the educational space—its experience goes back to the acquisition of French company Aldebaran Robotics, whose Nao humanoid has long been used in classrooms. Pepper, also originally developed by Aldebaran, is Nao’s newer, bigger sibling, and it, too, has been used in classrooms before.

Photo: SoftBank Robotics Using the Tethys visual programming tool, students can program Pepper to move, gesticulate, talk, and display graphics on its tablet. They can run their programs on a real robot or a virtual one on their computers.

Pepper’s size is probably one of its main advantages over the competition. It’s a 1.2-meter tall humanoid that can move around a room, dance, and have conversations and play games with people—not just a small wheeled robot beeping and driving on a tabletop.

On the other hand, Pepper’s size also means it costs several times as much as those other robots. That’s a challenge if SoftBank wants to get lots of them out to schools, which may not be able to afford them. So far the company has addressed the issue by donating Peppers—over 100 robots in the past two years.

How Tethys work

When SoftBank first took Pepper to classrooms, it discovered that the robot’s original software development platform, called Choregraphe, wasn’t designed as an educational tool. It was hard to use by non engineers, and was glitchy. SoftBank then partnered with Finger Food Advanced Technology Group, a Vancouver-based software company, to develop Tethys.

Image: SoftBank Robotics While Tethys is based on a visual programming environment, students can inspect the underlying Python scripts and modify them or write their own code.

Tethys is an integrated development environment, or IDE, that runs on a web browser (it works on regular laptops and also Chromebooks, popular in schools). It features a user-friendly visual programming interface, and in that sense it is similar to other visual programming languages like Blockly and Scratch.

But students aren’t limited to dragging blocks and wires on the screen; they can inspect the underlying Python scripts and modify them, or write their own code.

SoftBank says the new initiative is focused on “STREAM” education, or Science, Technology, Robotics, Engineering, Art, and Mathematics. Accordingly, Tethys is named after the Greek Titan goddess of streams, says SoftBank’s Dawson, who heads its STREAM Education program.

“It’s really important to make sure that more people are getting involved in robotics,” he says, “and that means not just the existing engineers who are out there, but trying to encourage the engineers of the future.”

When the group of high schoolers arrived for the coding camp, the idea of spending the day staring at a computer screen didn’t seem too exciting to them. But then Pepper rolled into the room.

“All of a sudden everyone wanted to become a robot coder,” says Kass Dawson, head of marketing and business strategy at SoftBank Robotics America, in San Francisco. He saw the same thing happen in other classrooms, where the friendly humanoid was an instant hit with students.

“What we realized very quickly was, we need to take advantage of the fact that this robot can get kids excited about computer science,” Dawson says.

Today SoftBank is launching Tethys, a visual programming tool designed to teach students how to code by creating applications for Pepper. The company is hoping that its humanoid robot, which has been deployed in homesretail stores, and research labs, can also play a role in schools, helping to foster the next generation of engineers and roboticists.

As part of a pilot program, more than 1,000 students in about 20 public schools in Boston, San Francisco, and Vancouver, Canada

Tethys is based on an intuitive, graphical approach to coding. To create a program, you drag boxes (representing different robot behaviors) on the screen and connect them with wires. You can run your program instantly on a Pepper to see how it works. You can also run it on a virtual robot on the screen.

As part of a pilot program, more than 1,000 students in about 20 public schools in Boston, San Francisco, and Vancouver, Canada, are already using the tool. SoftBank plans to continue expanding to more locations. (Educators interested in bringing Tethys and Pepper to their schools should reach out to the company by email.)

Bringing robots to the classroom

The idea of using robots to teach coding, logic, and problem-solving skills is not new (in fact, in the United States it goes back nearly half a century). Lego robotics kits like Mindstorms, Boost, and WeDo are widely used in STEM education today. Other popular robots and kits include Dash and Dot, Cubelets, Sphero, VEX, Parallax, and Ozobot. Last year, iRobot acquired Root, a robotics education startup founded by Harvard researchers.

So SoftBank is entering a crowded market, although one that has a lot of growth potential. And to be fair, SoftBank is not entirely new to the educational space—its experience goes back to the acquisition of French company Aldebaran Robotics, whose Nao humanoid has long been used in classrooms. Pepper, also originally developed by Aldebaran, is Nao’s newer, bigger sibling, and it, too, has been used in classrooms before.

Photo: SoftBank Robotics Using the Tethys visual programming tool, students can program Pepper to move, gesticulate, talk, and display graphics on its tablet. They can run their programs on a real robot or a virtual one on their computers.

Pepper’s size is probably one of its main advantages over the competition. It’s a 1.2-meter tall humanoid that can move around a room, dance, and have conversations and play games with people—not just a small wheeled robot beeping and driving on a tabletop.

On the other hand, Pepper’s size also means it costs several times as much as those other robots. That’s a challenge if SoftBank wants to get lots of them out to schools, which may not be able to afford them. So far the company has addressed the issue by donating Peppers—over 100 robots in the past two years.

How Tethys work

When SoftBank first took Pepper to classrooms, it discovered that the robot’s original software development platform, called Choregraphe, wasn’t designed as an educational tool. It was hard to use by non engineers, and was glitchy. SoftBank then partnered with Finger Food Advanced Technology Group, a Vancouver-based software company, to develop Tethys.

Image: SoftBank Robotics While Tethys is based on a visual programming environment, students can inspect the underlying Python scripts and modify them or write their own code.

Tethys is an integrated development environment, or IDE, that runs on a web browser (it works on regular laptops and also Chromebooks, popular in schools). It features a user-friendly visual programming interface, and in that sense it is similar to other visual programming languages like Blockly and Scratch.

But students aren’t limited to dragging blocks and wires on the screen; they can inspect the underlying Python scripts and modify them, or write their own code.

SoftBank says the new initiative is focused on “STREAM” education, or Science, Technology, Robotics, Engineering, Art, and Mathematics. Accordingly, Tethys is named after the Greek Titan goddess of streams, says SoftBank’s Dawson, who heads its STREAM Education program.

“It’s really important to make sure that more people are getting involved in robotics,” he says, “and that means not just the existing engineers who are out there, but trying to encourage the engineers of the future.”

Today, Boston Dynamics and OTTO Motors (a division of Clearpath Robotics) are announcing a partnership to “coordinate mobile robots in the warehouse” as part of “the future of warehouse automation.” It’s a collaboration between OTTO’s autonomous mobile robots and Boston Dynamics’s Handle, showing how a heterogeneous robot team can be faster and more efficient in a realistic warehouse environment.

As much as we love Handle, it doesn’t really seem like the safest robot for humans to be working around. Its sheer size, dynamic motion, and heavy payloads mean that the kind of sense-and-avoid hardware and software you’d really want to have on it for humans to able to move through its space without getting smushed would likely be impractical, so you need another way of moving stuff in an out of its work zone. The Handle logistics video Boston Dynamics released about a year ago showed the robot working mostly with conveyor belts, but that kind of fixed infrastructure may not be ideal for warehouses that want to remain flexible.

This is where OTTO Motors comes in—its mobile robots (essentially autonomous mobile cargo pallets) can safely interact with Handles carrying boxes, moving stuff from where the Handles are working to where it needs to go without requiring intervention from a fragile and unpredictable human who would likely only get in the way of the whole process. 

From the press release:

“We’ve built a proof of concept demonstration of a heterogeneous fleet of robots building distribution center orders to provide a more flexible warehouse automation solution,” said Boston Dynamics VP of Product Engineering Kevin Blankespoor. “To meet the rates that our customers expect, we’re continuing to expand Handle’s capabilities and optimizing its interactions with other robots like the OTTO 1500 for warehouse applications.”

This sort of suggests that OTTO Motors might not be the only partner that Boston Dynamics is working with. There are certainly other companies who make autonomous mobile robots for warehouses like OTTO does, but it’s more fun to think about fleets of warehouse robots that are as heterogeneous as possible: drones, blimps, snake robots, hexapods—I wouldn’t put anything past them.

[ OTTO Motors ]

Today, Boston Dynamics and OTTO Motors (a division of Clearpath Robotics) are announcing a partnership to “coordinate mobile robots in the warehouse” as part of “the future of warehouse automation.” It’s a collaboration between OTTO’s autonomous mobile robots and Boston Dynamics’s Handle, showing how a heterogeneous robot team can be faster and more efficient in a realistic warehouse environment.

As much as we love Handle, it doesn’t really seem like the safest robot for humans to be working around. Its sheer size, dynamic motion, and heavy payloads mean that the kind of sense-and-avoid hardware and software you’d really want to have on it for humans to able to move through its space without getting smushed would likely be impractical, so you need another way of moving stuff in an out of its work zone. The Handle logistics video Boston Dynamics released about a year ago showed the robot working mostly with conveyor belts, but that kind of fixed infrastructure may not be ideal for warehouses that want to remain flexible.

This is where OTTO Motors comes in—its mobile robots (essentially autonomous mobile cargo pallets) can safely interact with Handles carrying boxes, moving stuff from where the Handles are working to where it needs to go without requiring intervention from a fragile and unpredictable human who would likely only get in the way of the whole process. 

From the press release:

“We’ve built a proof of concept demonstration of a heterogeneous fleet of robots building distribution center orders to provide a more flexible warehouse automation solution,” said Boston Dynamics VP of Product Engineering Kevin Blankespoor. “To meet the rates that our customers expect, we’re continuing to expand Handle’s capabilities and optimizing its interactions with other robots like the OTTO 1500 for warehouse applications.”

This sort of suggests that OTTO Motors might not be the only partner that Boston Dynamics is working with. There are certainly other companies who make autonomous mobile robots for warehouses like OTTO does, but it’s more fun to think about fleets of warehouse robots that are as heterogeneous as possible: drones, blimps, snake robots, hexapods—I wouldn’t put anything past them.

[ OTTO Motors ]

For the past two weeks, teams of robots (and their humans) have been exploring an unfinished nuclear power plant in Washington State as part of DARPA’s Subterranean Challenge. The SubT Challenge consists of three separate circuits, each representing a distinct underground environment: tunnel systems, urban underground, and cave networks.

The Urban Circuit portion of the challenge ended last Thursday, and DARPA live streamed all of the course runs and put together some great video recaps of the competition itself. But that footage represents just a small portion of what actually went on at the challenge, as teams raced to implement fixes and improvements in hardware and software in between runs, often staying up all night in weird places trying to get their robots to work better (or work at all).

We visited the SubT Urban Challenge during the official media day last week, and also spent some time off-site with the teams themselves, as they solved problems and tested their robots wherever they could, from nearby high schools to empty malls to hotel stairwells at 5 a.m. 

And the winner of the SubT Urban Circuit is...

The winner of the SubT Urban Circuit was Team CoSTAR, a collaboration between NASA JPL, MIT, Caltech, KAIST, LTU, and industry partners, including Clearpath Robotics and Boston Dynamics. Second place went to Carnegie Mellon’s Team Explorer, which took first at the previous SubT Tunnel Circuit six months ago, setting up a highly competitive Cave Circuit event which will take place six months from now.

We’ll have some more details on the teams’ final scores, but first here’s a brief post-challenge overview video from DARPA to get you caught up:

The Urban Circuit location: an unfinished nuclear power plant

The Urban Circuit of the DARPA Subterranean Challenge was held at the Satsop Business Park, about an hour and a half south of Seattle. 

Photo: DARPA Aerial photo of the unfinished Satsop nuclear power plant.

Started in 1977, the plant was about 80 percent complete when state funding fell through, and after nothing happened for a couple of decades, ownership was transferred to the Satsop Redevelopment Project to try and figure out how to turn the aging dystopian infrastructure into something useful. Something useful includes renting the space for people to film action movies, and for DARPA to host challenges.

The biggest difference between Tunnel and Urban is that while Tunnel was mostly, you know, tunnels (mostly long straight-ish passages connected with each other), Urban included a variety of large spaces and interconnected small rooms spread out across multiple levels. This is a 5-minute long walkthrough from DARPA that shows one of the course configurations; you don’t need to watch the whole thing, but it should give you a pretty good idea of the sort of environment that these robots had to deal with:

The biggest challenge: Communications, or stairs?

While communications were an enormous challenge at the Tunnel Circuit, from talking with the teams it sounded like comms was not nearly as much of an issue at Urban, because of a combination of a slightly friendlier environment (concrete walls instead of meters of solid rock) and teams taking comms very, very seriously as they prepared their systems for this event. More teams used deployable networking nodes to build up a mesh network as their robots progressed farther into the course (more on this later), and there was also more of an emphasis on fully autonomous exploration where robots were comfortable operating for extended periods outside of communication range completely. 

Photo: Evan Ackerman/IEEE Spectrum Team garages at the event. You can’t see how cold it is, but if you could, you’d understand why they’re mostly empty.

When we talked to DARPA SubT Program Manager Tim Chung a few weeks ago, he was looking forward to creating an atmosphere of warm camaraderie between teams:

I’m super excited about how we set up the team garages at the Urban Circuit. It’ll be like pit row, in a way that really highlights how much I value the interactions between teams, it’ll be an opportunity to truly capitalize on having a high concentration of enthusiastic and ambitious roboticists in one area. 

Another challenge: Finding a warm place to test the robots

Having all the teams gathered at their garages would have been pretty awesome, except that the building somehow functioned as a giant heat sink, and while it was in the mid-30s Fahrenheit outside, it felt like the mid-20s inside! Neither humans nor robots had any particular desire to spend more time in the garages than was strictly necessary—most teams would arrive immediately before the start of their run staging time, and then escape to somewhere warmer immediately after their run ended. 

It wasn’t just a temperature thing that kept teams out of the garages—to test effectively, most teams needed a lot more dedicated space than was available on-site. Teams understood how important test environments were after the Tunnel Circuit, and most of them scrounged up spaces well in advance. Team CSIRO DATA61 found an indoor horse paddock at the local fairgrounds. Team CERBERUS set up in an empty storefront in a half dead mall about 20 miles away. And Team CoSTAR took over the conference center at a local hotel, which turned out to be my hotel, as I discovered when I met Spot undergoing testing in the hallway outside of my room right after I checked in:

Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR’s Spot robot (on loan from Boston Dynamics) undergoing testing in a hotel hallway.

Spot is not exactly the stealthiest of robots, and the hotel testing was not what you’d call low-key. I can tell you that CoSTAR finished their testing at around 5:15 a.m., when Spot’s THUMP THUMP THUMP THUMP THUMPing gait woke up pretty much the entire hotel as the robot made its way back to its hotel room. Spot did do a very good job on the stairs, though:

Photo: Evan Ackerman/IEEE Spectrum Even with its top-heavy JPL autonomy and mapping payload, Spot was able to climb stairs without too much trouble.

After the early morning quadrupedal wake-up call, I put on every single layer of clothing I’d brought and drove up to the competition site for the DARPA media day. We were invited to watch the beginning of a few competition runs, take a brief course tour (after being sworn to secrecy), and speak with teams at the garages before and after their runs. During the Tunnel circuit, I’d focused on the creative communications strategies that each team was using, but for Urban, I asked teams to tell me about some of the clever hacks they’d come up with to solve challenges specific to the Urban circuit.

Here’s some of what teams came up with:

Team NCTU

Team NCTU from Taiwan has some of the most consistently creative approaches to the DARPA SubT courses we’ve seen. They’re probably best known for their “Duckiefloat” blimps, which had some trouble fitting through narrow tunnels during the Tunnel circuit six months ago. Knowing that passages would be even slimmer for the Urban Circuit, NCTU built a carbon fiber frame around the Duckiefloats to squish their sides in a bit.

Photo: Evan Ackerman/IEEE Spectrum Duckiefloat is much slimmer (if a bit less pleasingly spherical) thanks to a carbon fiber framework that squeezes it into a more streamlined shape to better fit through narrow corridors.

NCTU also added millimeter wave radar to one of the Duckiefloats as a lighter substitute for on-board lidar or RGBD cameras, and had good results navigating with the radar alone, which (as far as I know) is a totally unique approach. We will definitely be seeing more of Duckiefloat for the cave circuit.

Photo: Evan Ackerman/IEEE Spectrum NCTU’s Anchorball droppable WiFi nodes now include a speaker, which the Husky UGV can localize with microphone arrays (the black circle with the white border).

At Tunnel, NCTU dropped mesh WiFi nodes that doubled as beacons, called Anchorballs. For Urban, the Anchorballs are 100 percent less ball-like, and incorporate a speaker, which plays chirping noises once deployed. Microphone arrays on the Husky UGVs can localize this chirping, allowing multiple robots to use the nodes as tie points to coordinate their maps.

Photo: Evan Ackerman/IEEE Spectrum NCTU is developing mobile mesh network nodes in the form of autonomous robot balls.

Also under development at NCTU is this mobile Anchorball, which is basically a big Sphero with a bunch of networking gear packed into it that can move itself around to optimize signal strength.

Team NUS SEDS

Team NUS SEDS accidentally burned out a couple of the onboard computers driving their robots. The solution was to run out and buy a laptop, and then 3D print some mounts to attach the laptop to the top of the robot and run things from there.

Photo: Evan Ackerman/IEEE Spectrum When an onboard computer burned out, NUS SEDS bought a new laptop to power their mobile robot, because what else are you going to do?

They also had a larger tracked vehicle that was able to go up and down stairs, but it got stuck in customs and didn’t make it to the competition at all.

Team Explorer

Team Explorer did extensive testing in an abandoned hospital in Pittsburgh, which I’m sure wasn’t creepy at all. While they brought along some drones that were used very successfully, getting their beefy wheeled robots up and down stairs wasn’t easy. To add some traction, Explorer cut chunks out of the wheels on one of their robots to help it grip the edges of stairs. 

Photo: Evan Ackerman/IEEE Spectrum Team Explorer’s robot has wedges cut out of its wheels to help it get a grip on stairways.

It doesn’t look especially sophisticated, but the team lead Sebastian Scherer told me that this was the result of 14 (!) iterations of wheel and track modifications. 

Team MARBLE

Six months ago, we checked out a bunch of different creative communications strategies that teams used at SubT Tunnel. MARBLE improved on their droppable wireless repeater nodes with a powered, extending antenna (harvested from a Miata, apparently).

Photo: Evan Ackerman/IEEE Spectrum After being dropped from its carrier robot, this mesh networking node extends its antennas half a meter into the air to maximize signal strength.

This is more than just a neat trick: We were told that the extra height that the antennas have once fully deployed does significantly improve their performance.

Team Robotika

Based on their experience during the Tunnel Circuit, Team Robotika decided that there was no such thing as having too much light in the tunnels, so they brought along a robot with the most enormous light-to-robot ratio that we saw at SubT.

Photo: Evan Ackerman/IEEE Spectrum No such thing as too much light during DARPA SubT.

Like many other teams, Robotika was continually making minor hardware adjustments to refine the performance of their robots and make them more resilient to the environment. These last-minute plastic bumpers would keep the robot from driving up walls and potentially flipping itself over.

Photo: Evan Ackerman/IEEE Spectrum A bumper hacked together from plastic and duct tape keeps this robot from flipping itself over against walls. Team CSIRO Data61

I met CSIRO Data61 (based in Australia) at the testing location they’d found in a building at the Grays Harbor County Fairgrounds, right next to an indoor horse arena that provided an interesting environment, especially for their drones. During their first run, one of their large tracked robots (an ex-police robot called Titan) had the misfortune to get its track caught on an obstacle that was exactly the wrong size, and it burned out a couple motors trying to get free.

Photo: Evan Ackerman/IEEE Spectrum A burned motor, crispy on the inside.

You can practically smell that through the screen, right? And these are fancy Maxon motors, which you can’t just pick up at your local hardware store. CSIRO didn’t have spares with them, so the most expedient way to get new motors that were sure to work turned out to be flying another team member over from Australia (!) with extra motors in their carry-on luggage. And by Tuesday morning, the Titan was up and running again.

Photo: Evan Ackerman/IEEE Spectrum A fully operational Titan beside a pair of commercial SuperDroid robots at CSIRO’s off-site testing area. Team CERBERUS

Team CERBERUS didn’t have a run scheduled during the SubT media day, but they invited me to visit their testing area in an empty store next to an Extreme Fun Center in a slightly depressing mall in Aberdeen (Kurt Cobain’s hometown), about 20 miles down the road from Satsop. CERBERUS was using a mix of wheeled vehicles, collision-tolerant drones, and ANYmal legged robots.

Photo: Evan Ackerman/IEEE Spectrum Team CERBERUS doing some off-site testing of their robots with the lights off.

CERBERUS had noticed during a DARPA course pre-briefing that the Alpha course had an almost immediate 90-degree turn before a long passage, which would block any directional antennas placed in the staging area. To try to maximize communication range, they developed this dumb antenna robot: Dumb in the sense that it has no sensing or autonomy, but instead is designed to carry a giant tethered antenna just around that first corner.

Photo: Evan Ackerman/IEEE Spectrum Basically just a remote-controlled directional antenna, CERBERUS developed this robot to extend communications from their base station around the first corner of Alpha Course.

Another communications challenge was how to talk to robots after they traversed down a flight of stairs. Alpha Course featured a flight of stairs going downwards just past the starting gate, and CERBERUS wanted a way of getting a mesh networking node down those stairs to be able to reliably talk to robots exploring the lower level. Here’s what they came up with:

Photo: Evan Ackerman/IEEE Spectrum A mesh network node inside of a foam ball covered in duct tape can be thrown by a human into hard-to-reach spots near the starting area.

The initial idea was to put a node into a soccer ball which would then be kicked from the staging area, off the far wall, and down the stairs, but they ended up finding some hemispheres of green foam used for flower arrangements at Walmart, hollowed them out, put in a node, and then wrapped the whole thing in duct tape. With the addition of a tether, the node in a ball could be thrown from the staging area into the stairwell, and brought back up with the tether if it didn’t land in the right spot.

Plan B for stairwell communications was a bit more of a brute force approach, using a directional antenna on a stick that could be poked out of the starting area and angled over the stairwell.

Photo: Evan Ackerman/IEEE Spectrum If your antenna balls don’t work? Just add a directional antenna to a stick.

Since DARPA did allow tethers, CERBERUS figured that this was basically just a sort of rigid tether. Sounds good to me!

Team CoSTAR

Team CoSTAR surprised everyone by showing up to the SubT Urban Circuit with a pair of Spot quadrupeds from Boston Dynamics. The Spots were very much a last-minute addition to the team, and CoSTAR only had about six weeks to get them up and (metaphorically) running. Consequently, the Spots were a little bit overburdened with a payload that CoSTAR hasn’t had much of a chance to optimize. The payload takes care of all of the higher level autonomy and map making and stuff, while Spot’s own sensors handle the low-level motion planning. 

Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR’s Spot robots carried a payload that was almost too heavy for the robot to manage, and included sensors, lights, computers, batteries, and even two mesh network node droppers.

In what would be a spectacular coincidence were both of these teams not packed full of brilliant roboticists, Team CoSTAR independently came up with something very similar to the throwable network node that Team CERBERUS was messing around with.

Photo: Evan Ackerman/IEEE Spectrum A throwable mesh network node embedded in a foam ball that could be bounced into a stairwell to extend communications.

One of the early prototypes of this thing was a Mars lander-style “airbag” system, consisting of a pyramid of foam balls with a network node embedded in the very center of the pile. They showed me a video of this thing, and it was ridiculously cool, but they found that carving out the inside of a foam ball worked just as well and was far easier to manage.

There was only so much testing that CoSTAR was able to do in the hotel and conference center, since a better match for the Urban Circuit would be a much larger area with long hallways, small rooms, and multiple levels that could be reached by ramps and stairs. So every evening, the team and their robots drove 10 minutes down the road to Elma High School, which seemed to be just about the perfect place for testing SubT robots. CoSTAR very kindly let me tag along one night to watch their Huskies and Spots explore the school looking for artifacts, and here are some pictures that I took.

Photo: Evan Ackerman/IEEE Spectrum The Elma High School cafeteria became the staging area for Team CoSTAR’s SubT test course. Two Boston Dynamics Spot robots and two Clearpath Robotics Huskies made up CoSTAR’s team of robots. The yellow total station behind the robots is used for initial location calibration, and many other teams relied on them as well. Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR hid artifacts all over the school to test the robots’ ability to autonomously recognize and locate them. That’s a survivor dummy down the hall.

JPL put together this video of one of the test runs, which cuts out the three hours of setup and calibration and condenses all the good stuff into a minute and a half:

DARPA SubT Urban Circuit: Final scores

In their final SubT Urban run, CoSTAR scored a staggering 9 points, giving them a total of 16 for the Urban Circuit, 5 more than Team Explorer, which came in second. Third place went to Team CTU-CRAS-NORLAB, and as a self-funded (as opposed to DARPA-funded) team, they walked away with a $500,000 prize.

Image: DARPA DARPA SubT Urban Circuit final scores.

Six months from now, all of these teams will meet again to compete at the SubT Cave Circuit, the last (and perhaps most challenging) domain that DARPA has in store. We don’t yet know exactly when or where Cave will take place, but we do know that we'll be there to see what six more months of hard work and creativity can do for these teams and their robots.

[ DARPA SubT Urban Results ]

Special thanks to DARPA for putting on this incredible event, and thanks also to the teams that let me follow them around and get (ever so slightly) in the way for a day or two.

For the past two weeks, teams of robots (and their humans) have been exploring an unfinished nuclear power plant in Washington State as part of DARPA’s Subterranean Challenge. The SubT Challenge consists of three separate circuits, each representing a distinct underground environment: tunnel systems, urban underground, and cave networks.

The Urban Circuit portion of the challenge ended last Thursday, and DARPA live streamed all of the course runs and put together some great video recaps of the competition itself. But that footage represents just a small portion of what actually went on at the challenge, as teams raced to implement fixes and improvements in hardware and software in between runs, often staying up all night in weird places trying to get their robots to work better (or work at all).

We visited the SubT Urban Challenge during the official media day last week, and also spent some time off-site with the teams themselves, as they solved problems and tested their robots wherever they could, from nearby high schools to empty malls to hotel stairwells at 5 a.m. 

And the winner of the SubT Urban Circuit is...

The winner of the SubT Urban Circuit was Team CoSTAR, a collaboration between NASA JPL, MIT, Caltech, KAIST, LTU, and industry partners, including Clearpath Robotics and Boston Dynamics. Second place went to Carnegie Mellon’s Team Explorer, which took first at the previous SubT Tunnel Circuit six months ago, setting up a highly competitive Cave Circuit event which will take place six months from now.

We’ll have some more details on the teams’ final scores, but first here’s a brief post-challenge overview video from DARPA to get you caught up:

The Urban Circuit location: an unfinished nuclear power plant

The Urban Circuit of the DARPA Subterranean Challenge was held at the Satsop Business Park, about an hour and a half south of Seattle. 

Photo: DARPA Aerial photo of the unfinished Satsop nuclear power plant.

Started in 1977, the plant was about 80 percent complete when state funding fell through, and after nothing happened for a couple of decades, ownership was transferred to the Satsop Redevelopment Project to try and figure out how to turn the aging dystopian infrastructure into something useful. Something useful includes renting the space for people to film action movies, and for DARPA to host challenges.

The biggest difference between Tunnel and Urban is that while Tunnel was mostly, you know, tunnels (mostly long straight-ish passages connected with each other), Urban included a variety of large spaces and interconnected small rooms spread out across multiple levels. This is a 5-minute long walkthrough from DARPA that shows one of the course configurations; you don’t need to watch the whole thing, but it should give you a pretty good idea of the sort of environment that these robots had to deal with:

The biggest challenge: Communications, or stairs?

While communications were an enormous challenge at the Tunnel Circuit, from talking with the teams it sounded like comms was not nearly as much of an issue at Urban, because of a combination of a slightly friendlier environment (concrete walls instead of meters of solid rock) and teams taking comms very, very seriously as they prepared their systems for this event. More teams used deployable networking nodes to build up a mesh network as their robots progressed farther into the course (more on this later), and there was also more of an emphasis on fully autonomous exploration where robots were comfortable operating for extended periods outside of communication range completely. 

Photo: Evan Ackerman/IEEE Spectrum Team garages at the event. You can’t see how cold it is, but if you could, you’d understand why they’re mostly empty.

When we talked to DARPA SubT Program Manager Tim Chung a few weeks ago, he was looking forward to creating an atmosphere of warm camaraderie between teams:

I’m super excited about how we set up the team garages at the Urban Circuit. It’ll be like pit row, in a way that really highlights how much I value the interactions between teams, it’ll be an opportunity to truly capitalize on having a high concentration of enthusiastic and ambitious roboticists in one area. 

Another challenge: Finding a warm place to test the robots

Having all the teams gathered at their garages would have been pretty awesome, except that the building somehow functioned as a giant heat sink, and while it was in the mid-30s Fahrenheit outside, it felt like the mid-20s inside! Neither humans nor robots had any particular desire to spend more time in the garages than was strictly necessary—most teams would arrive immediately before the start of their run staging time, and then escape to somewhere warmer immediately after their run ended. 

It wasn’t just a temperature thing that kept teams out of the garages—to test effectively, most teams needed a lot more dedicated space than was available on-site. Teams understood how important test environments were after the Tunnel Circuit, and most of them scrounged up spaces well in advance. Team CSIRO DATA61 found an indoor horse paddock at the local fairgrounds. Team CERBERUS set up in an empty storefront in a half dead mall about 20 miles away. And Team CoSTAR took over the conference center at a local hotel, which turned out to be my hotel, as I discovered when I met Spot undergoing testing in the hallway outside of my room right after I checked in:

Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR’s Spot robot (on loan from Boston Dynamics) undergoing testing in a hotel hallway.

Spot is not exactly the stealthiest of robots, and the hotel testing was not what you’d call low-key. I can tell you that CoSTAR finished their testing at around 5:15 a.m., when Spot’s THUMP THUMP THUMP THUMP THUMPing gait woke up pretty much the entire hotel as the robot made its way back to its hotel room. Spot did do a very good job on the stairs, though:

Photo: Evan Ackerman/IEEE Spectrum Even with its top-heavy JPL autonomy and mapping payload, Spot was able to climb stairs without too much trouble.

After the early morning quadrupedal wake-up call, I put on every single layer of clothing I’d brought and drove up to the competition site for the DARPA media day. We were invited to watch the beginning of a few competition runs, take a brief course tour (after being sworn to secrecy), and speak with teams at the garages before and after their runs. During the Tunnel circuit, I’d focused on the creative communications strategies that each team was using, but for Urban, I asked teams to tell me about some of the clever hacks they’d come up with to solve challenges specific to the Urban circuit.

Here’s some of what teams came up with:

Team NCTU

Team NCTU from Taiwan has some of the most consistently creative approaches to the DARPA SubT courses we’ve seen. They’re probably best known for their “Duckiefloat” blimps, which had some trouble fitting through narrow tunnels during the Tunnel circuit six months ago. Knowing that passages would be even slimmer for the Urban Circuit, NCTU built a carbon fiber frame around the Duckiefloats to squish their sides in a bit.

Photo: Evan Ackerman/IEEE Spectrum Duckiefloat is much slimmer (if a bit less pleasingly spherical) thanks to a carbon fiber framework that squeezes it into a more streamlined shape to better fit through narrow corridors.

NCTU also added millimeter wave radar to one of the Duckiefloats as a lighter substitute for on-board lidar or RGBD cameras, and had good results navigating with the radar alone, which (as far as I know) is a totally unique approach. We will definitely be seeing more of Duckiefloat for the cave circuit.

Photo: Evan Ackerman/IEEE Spectrum NCTU’s Anchorball droppable WiFi nodes now include a speaker, which the Husky UGV can localize with microphone arrays (the black circle with the white border).

At Tunnel, NCTU dropped mesh WiFi nodes that doubled as beacons, called Anchorballs. For Urban, the Anchorballs are 100 percent less ball-like, and incorporate a speaker, which plays chirping noises once deployed. Microphone arrays on the Husky UGVs can localize this chirping, allowing multiple robots to use the nodes as tie points to coordinate their maps.

Photo: Evan Ackerman/IEEE Spectrum NCTU is developing mobile mesh network nodes in the form of autonomous robot balls.

Also under development at NCTU is this mobile Anchorball, which is basically a big Sphero with a bunch of networking gear packed into it that can move itself around to optimize signal strength.

Team NUS SEDS

Team NUS SEDS accidentally burned out a couple of the onboard computers driving their robots. The solution was to run out and buy a laptop, and then 3D print some mounts to attach the laptop to the top of the robot and run things from there.

Photo: Evan Ackerman/IEEE Spectrum When an onboard computer burned out, NUS SEDS bought a new laptop to power their mobile robot, because what else are you going to do?

They also had a larger tracked vehicle that was able to go up and down stairs, but it got stuck in customs and didn’t make it to the competition at all.

Team Explorer

Team Explorer did extensive testing in an abandoned hospital in Pittsburgh, which I’m sure wasn’t creepy at all. While they brought along some drones that were used very successfully, getting their beefy wheeled robots up and down stairs wasn’t easy. To add some traction, Explorer cut chunks out of the wheels on one of their robots to help it grip the edges of stairs. 

Photo: Evan Ackerman/IEEE Spectrum Team Explorer’s robot has wedges cut out of its wheels to help it get a grip on stairways.

It doesn’t look especially sophisticated, but the team lead Sebastian Scherer told me that this was the result of 14 (!) iterations of wheel and track modifications. 

Team MARBLE

Six months ago, we checked out a bunch of different creative communications strategies that teams used at SubT Tunnel. MARBLE improved on their droppable wireless repeater nodes with a powered, extending antenna (harvested from a Miata, apparently).

Photo: Evan Ackerman/IEEE Spectrum After being dropped from its carrier robot, this mesh networking node extends its antennas half a meter into the air to maximize signal strength.

This is more than just a neat trick: We were told that the extra height that the antennas have once fully deployed does significantly improve their performance.

Team Robotika

Based on their experience during the Tunnel Circuit, Team Robotika decided that there was no such thing as having too much light in the tunnels, so they brought along a robot with the most enormous light-to-robot ratio that we saw at SubT.

Photo: Evan Ackerman/IEEE Spectrum No such thing as too much light during DARPA SubT.

Like many other teams, Robotika was continually making minor hardware adjustments to refine the performance of their robots and make them more resilient to the environment. These last-minute plastic bumpers would keep the robot from driving up walls and potentially flipping itself over.

Photo: Evan Ackerman/IEEE Spectrum A bumper hacked together from plastic and duct tape keeps this robot from flipping itself over against walls. Team CSIRO Data61

I met CSIRO Data61 (based in Australia) at the testing location they’d found in a building at the Grays Harbor County Fairgrounds, right next to an indoor horse arena that provided an interesting environment, especially for their drones. During their first run, one of their large tracked robots (an ex-police robot called Titan) had the misfortune to get its track caught on an obstacle that was exactly the wrong size, and it burned out a couple motors trying to get free.

Photo: Evan Ackerman/IEEE Spectrum A burned motor, crispy on the inside.

You can practically smell that through the screen, right? And these are fancy Maxon motors, which you can’t just pick up at your local hardware store. CSIRO didn’t have spares with them, so the most expedient way to get new motors that were sure to work turned out to be flying another team member over from Australia (!) with extra motors in their carry-on luggage. And by Tuesday morning, the Titan was up and running again.

Photo: Evan Ackerman/IEEE Spectrum A fully operational Titan beside a pair of commercial SuperDroid robots at CSIRO’s off-site testing area. Team CERBERUS

Team CERBERUS didn’t have a run scheduled during the SubT media day, but they invited me to visit their testing area in an empty store next to an Extreme Fun Center in a slightly depressing mall in Aberdeen (Kurt Cobain’s hometown), about 20 miles down the road from Satsop. CERBERUS was using a mix of wheeled vehicles, collision-tolerant drones, and ANYmal legged robots.

Photo: Evan Ackerman/IEEE Spectrum Team CERBERUS doing some off-site testing of their robots with the lights off.

CERBERUS had noticed during a DARPA course pre-briefing that the Alpha course had an almost immediate 90-degree turn before a long passage, which would block any directional antennas placed in the staging area. To try to maximize communication range, they developed this dumb antenna robot: Dumb in the sense that it has no sensing or autonomy, but instead is designed to carry a giant tethered antenna just around that first corner.

Photo: Evan Ackerman/IEEE Spectrum Basically just a remote-controlled directional antenna, CERBERUS developed this robot to extend communications from their base station around the first corner of Alpha Course.

Another communications challenge was how to talk to robots after they traversed down a flight of stairs. Alpha Course featured a flight of stairs going downwards just past the starting gate, and CERBERUS wanted a way of getting a mesh networking node down those stairs to be able to reliably talk to robots exploring the lower level. Here’s what they came up with:

Photo: Evan Ackerman/IEEE Spectrum A mesh network node inside of a foam ball covered in duct tape can be thrown by a human into hard-to-reach spots near the starting area.

The initial idea was to put a node into a soccer ball which would then be kicked from the staging area, off the far wall, and down the stairs, but they ended up finding some hemispheres of green foam used for flower arrangements at Walmart, hollowed them out, put in a node, and then wrapped the whole thing in duct tape. With the addition of a tether, the node in a ball could be thrown from the staging area into the stairwell, and brought back up with the tether if it didn’t land in the right spot.

Plan B for stairwell communications was a bit more of a brute force approach, using a directional antenna on a stick that could be poked out of the starting area and angled over the stairwell.

Photo: Evan Ackerman/IEEE Spectrum If your antenna balls don’t work? Just add a directional antenna to a stick.

Since DARPA did allow tethers, CERBERUS figured that this was basically just a sort of rigid tether. Sounds good to me!

Team CoSTAR

Team CoSTAR surprised everyone by showing up to the SubT Urban Circuit with a pair of Spot quadrupeds from Boston Dynamics. The Spots were very much a last-minute addition to the team, and CoSTAR only had about six weeks to get them up and (metaphorically) running. Consequently, the Spots were a little bit overburdened with a payload that CoSTAR hasn’t had much of a chance to optimize. The payload takes care of all of the higher level autonomy and map making and stuff, while Spot’s own sensors handle the low-level motion planning. 

Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR’s Spot robots carried a payload that was almost too heavy for the robot to manage, and included sensors, lights, computers, batteries, and even two mesh network node droppers.

In what would be a spectacular coincidence were both of these teams not packed full of brilliant roboticists, Team CoSTAR independently came up with something very similar to the throwable network node that Team CERBERUS was messing around with.

Photo: Evan Ackerman/IEEE Spectrum A throwable mesh network node embedded in a foam ball that could be bounced into a stairwell to extend communications.

One of the early prototypes of this thing was a Mars lander-style “airbag” system, consisting of a pyramid of foam balls with a network node embedded in the very center of the pile. They showed me a video of this thing, and it was ridiculously cool, but they found that carving out the inside of a foam ball worked just as well and was far easier to manage.

There was only so much testing that CoSTAR was able to do in the hotel and conference center, since a better match for the Urban Circuit would be a much larger area with long hallways, small rooms, and multiple levels that could be reached by ramps and stairs. So every evening, the team and their robots drove 10 minutes down the road to Elma High School, which seemed to be just about the perfect place for testing SubT robots. CoSTAR very kindly let me tag along one night to watch their Huskies and Spots explore the school looking for artifacts, and here are some pictures that I took.

Photo: Evan Ackerman/IEEE Spectrum The Elma High School cafeteria became the staging area for Team CoSTAR’s SubT test course. Two Boston Dynamics Spot robots and two Clearpath Robotics Huskies made up CoSTAR’s team of robots. The yellow total station behind the robots is used for initial location calibration, and many other teams relied on them as well. Photo: Evan Ackerman/IEEE Spectrum Team CoSTAR hid artifacts all over the school to test the robots’ ability to autonomously recognize and locate them. That’s a survivor dummy down the hall.

JPL put together this video of one of the test runs, which cuts out the three hours of setup and calibration and condenses all the good stuff into a minute and a half:

DARPA SubT Urban Circuit: Final scores

In their final SubT Urban run, CoSTAR scored a staggering 9 points, giving them a total of 16 for the Urban Circuit, 5 more than Team Explorer, which came in second. Third place went to Team CTU-CRAS-NORLAB, and as a self-funded (as opposed to DARPA-funded) team, they walked away with a $500,000 prize.

Image: DARPA DARPA SubT Urban Circuit final scores.

Six months from now, all of these teams will meet again to compete at the SubT Cave Circuit, the last (and perhaps most challenging) domain that DARPA has in store. We don’t yet know exactly when or where Cave will take place, but we do know that we'll be there to see what six more months of hard work and creativity can do for these teams and their robots.

[ DARPA SubT Urban Results ]

Special thanks to DARPA for putting on this incredible event, and thanks also to the teams that let me follow them around and get (ever so slightly) in the way for a day or two.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

We’ll have more on the DARPA Subterranean Challenge Urban Circuit next week, but here’s a quick compilation from DARPA of some of the competition footage.

[ SubT ]

ABB set up a global competition in 2019 to assess 20 leading AI technology start-ups on how they could approach solutions for 26 real-world picking, packing and sorting challenges. The aim was to understand if AI is mature enough to fully unlock the potential for robotics and automation. ABB was also searching for a technology partner to co-develop robust AI solutions with. Covariant won the challenge by successfully completing each of the 26 challenges; on February 25, ABB and Covariant announced a partnership to bring AI-enabled robotic solutions to market.

We wrote about Covariant and its AI-based robot picking system last month. The most interesting part of the video above is probably the apple picking, where the system has to deal with irregular, shiny, rolling objects. The robot has a hard time picking upside-down apples, and after several failures in a row, it nudges the last one to make it easier to pick up. Impressive! And here’s one more video of real-time picking mostly transparent water bottles:

[ Covariant ]

Osaka University’s Affetto robot, which we’ve written about before, is looking somewhat more realistic than when we first wrote about it.

Those are some weird noises that it’s making though, right? Affetto, as it turns out, also doesn’t like getting poked in its (disembodied) tactile sensor:

They’re working on a body for it, too:

[ Osaka University ]

University of Washington students reimagine today’s libraries.

[ UW ]

Thanks Elcee!

Astrobee will be getting a hand up on the ISS, from Columbia’s ROAM Lab.

I think this will be Astrobee’s second hand, in addition to its perching arm. Maybe not designed for bimanual tasks, but still, pretty cool!

[ ROAM Lab ]

In this paper, we tackle the problem of pushing piles of small objects into a desired target set using visual feedback. Unlike conventional single-object manipulation pipelines, which estimate the state of the system parametrized by pose, the underlying physical state of this system is difficult to observe from images. Thus, we take the approach of reasoning directly in the space of images, and acquire the dynamics of visual measurements in order to synthesize a visual-feedback policy.

[ MIT ]

In this project we are exploring ways of interacting with terrain using hardware already present on exploration rovers - wheels! By using wheels for manipulation, we can expand the capabilities of space robots without the need for adding hardware. Nonprehensile terrain manipulation can be used many applications such as removing soil to sample below the surface or making terrain easier to cross for another robot. Watch until the end to see MiniRHex and the rover working together!

[ Robomechanics Lab ]

Dundee Precious Metals reveals how Exyn’s fully autonomous aerial drones are transforming their cavity monitoring systems with increased safety and maximum efficiency.

[ Exyn ]

Thanks Rachel!

Dragonfly is a NASA mission to explore the chemistry and habitability of Saturn’s largest moon, Titan. The fourth mission in the New Frontiers line, Dragonfly will send an autonomously-operated rotorcraft to visit dozens of sites on Titan, investigating the moon’s surface and shallow subsurface for organic molecules and possible biosignatures.

Dragonfly is scheduled to launch in 2026 and arrive at Titan in 2034.

[ NASA ]

Researchers at the Max Planck Institute for Intelligent Systems in Stuttgart in cooperation with Tampere University in Finland developed a gel-like robot inspired by sea slugs and snails they are able to steer with light. Much like the soft body of these aquatic invertebrates, the bioinspired robot is able to deform easily inside water when exposed to this energy source.

Due to specifically aligned molecules of liquid crystal gels – its building material – and illumination of specific parts of the robot, it is able to crawl, walk, jump, and swim inside water. The scientists see their research project as an inspiration for other roboticists who struggle to design untethered soft robots that are able to move freely in a fluidic environment.

[ Max Planck Institute ]

Forests are a very challenging environment for drones, especially if you want to both avoid and map trees at the same time.

[ Kumar Lab ]

Some highlights from the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) that took place in Abu Dhabi, UAE last week.

[ MBZ IRC ]

I never get tired of hearing technical presentations from Skydio, and here’s Ryan Kennedy giving at talk at the GRASP Lab.

The technology for intelligent and trustworthy navigation of autonomous UAVs has reached an inflection point to provide transformative gains in capability, efficiency, and safety to major industries. Drones are starting to save lives of first responders, automate dangerous infrastructure inspection, digitize the physical world with millimeter precision, and capture Hollywood quality video - all on affordable consumer hardware.

At Skydio, we have invested five years of R&D in the ability to handle difficult unknown scenarios in real-time based on visual sensing, and shipped two generations of fully autonomous drone. In this talk, I will discuss the close collaboration of geometry, learning, and modeling within our system, our experience putting robots into production, and the challenges still ahead.

[ Skydio ]

This week’s CMU RI Seminar comes from Sarjoun Skaff at Bossa Nova Robotics: “Yes, That’s a Robot in Your Grocery Store. Now what?”

Retail stores are becoming ground zero for indoor robotics. Fleet of different robots have to coexist with each others and humans every day, navigating safely, coordinating missions, and interacting appropriately with people, all at large scale. For us roboticists, stores are giant labs where we’re learning what doesn’t work and iterating. If we get it right, it will serve as an example for other industries, and robots will finally become ubiquitous in our lives.

[ CMU RI ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

We’ll have more on the DARPA Subterranean Challenge Urban Circuit next week, but here’s a quick compilation from DARPA of some of the competition footage.

[ SubT ]

ABB set up a global competition in 2019 to assess 20 leading AI technology start-ups on how they could approach solutions for 26 real-world picking, packing and sorting challenges. The aim was to understand if AI is mature enough to fully unlock the potential for robotics and automation. ABB was also searching for a technology partner to co-develop robust AI solutions with. Covariant won the challenge by successfully completing each of the 26 challenges; on February 25, ABB and Covariant announced a partnership to bring AI-enabled robotic solutions to market.

We wrote about Covariant and its AI-based robot picking system last month. The most interesting part of the video above is probably the apple picking, where the system has to deal with irregular, shiny, rolling objects. The robot has a hard time picking upside-down apples, and after several failures in a row, it nudges the last one to make it easier to pick up. Impressive! And here’s one more video of real-time picking mostly transparent water bottles:

[ Covariant ]

Osaka University’s Affetto robot, which we’ve written about before, is looking somewhat more realistic than when we first wrote about it.

Those are some weird noises that it’s making though, right? Affetto, as it turns out, also doesn’t like getting poked in its (disembodied) tactile sensor:

They’re working on a body for it, too:

[ Osaka University ]

University of Washington students reimagine today’s libraries.

[ UW ]

Thanks Elcee!

Astrobee will be getting a hand up on the ISS, from Columbia’s ROAM Lab.

I think this will be Astrobee’s second hand, in addition to its perching arm. Maybe not designed for bimanual tasks, but still, pretty cool!

[ ROAM Lab ]

In this paper, we tackle the problem of pushing piles of small objects into a desired target set using visual feedback. Unlike conventional single-object manipulation pipelines, which estimate the state of the system parametrized by pose, the underlying physical state of this system is difficult to observe from images. Thus, we take the approach of reasoning directly in the space of images, and acquire the dynamics of visual measurements in order to synthesize a visual-feedback policy.

[ MIT ]

In this project we are exploring ways of interacting with terrain using hardware already present on exploration rovers - wheels! By using wheels for manipulation, we can expand the capabilities of space robots without the need for adding hardware. Nonprehensile terrain manipulation can be used many applications such as removing soil to sample below the surface or making terrain easier to cross for another robot. Watch until the end to see MiniRHex and the rover working together!

[ Robomechanics Lab ]

Dundee Precious Metals reveals how Exyn’s fully autonomous aerial drones are transforming their cavity monitoring systems with increased safety and maximum efficiency.

[ Exyn ]

Thanks Rachel!

Dragonfly is a NASA mission to explore the chemistry and habitability of Saturn’s largest moon, Titan. The fourth mission in the New Frontiers line, Dragonfly will send an autonomously-operated rotorcraft to visit dozens of sites on Titan, investigating the moon’s surface and shallow subsurface for organic molecules and possible biosignatures.

Dragonfly is scheduled to launch in 2026 and arrive at Titan in 2034.

[ NASA ]

Researchers at the Max Planck Institute for Intelligent Systems in Stuttgart in cooperation with Tampere University in Finland developed a gel-like robot inspired by sea slugs and snails they are able to steer with light. Much like the soft body of these aquatic invertebrates, the bioinspired robot is able to deform easily inside water when exposed to this energy source.

Due to specifically aligned molecules of liquid crystal gels – its building material – and illumination of specific parts of the robot, it is able to crawl, walk, jump, and swim inside water. The scientists see their research project as an inspiration for other roboticists who struggle to design untethered soft robots that are able to move freely in a fluidic environment.

[ Max Planck Institute ]

Forests are a very challenging environment for drones, especially if you want to both avoid and map trees at the same time.

[ Kumar Lab ]

Some highlights from the Mohamed Bin Zayed International Robotics Challenge (MBZIRC) that took place in Abu Dhabi, UAE last week.

[ MBZ IRC ]

I never get tired of hearing technical presentations from Skydio, and here’s Ryan Kennedy giving at talk at the GRASP Lab.

The technology for intelligent and trustworthy navigation of autonomous UAVs has reached an inflection point to provide transformative gains in capability, efficiency, and safety to major industries. Drones are starting to save lives of first responders, automate dangerous infrastructure inspection, digitize the physical world with millimeter precision, and capture Hollywood quality video - all on affordable consumer hardware.

At Skydio, we have invested five years of R&D in the ability to handle difficult unknown scenarios in real-time based on visual sensing, and shipped two generations of fully autonomous drone. In this talk, I will discuss the close collaboration of geometry, learning, and modeling within our system, our experience putting robots into production, and the challenges still ahead.

[ Skydio ]

This week’s CMU RI Seminar comes from Sarjoun Skaff at Bossa Nova Robotics: “Yes, That’s a Robot in Your Grocery Store. Now what?”

Retail stores are becoming ground zero for indoor robotics. Fleet of different robots have to coexist with each others and humans every day, navigating safely, coordinating missions, and interacting appropriately with people, all at large scale. For us roboticists, stores are giant labs where we’re learning what doesn’t work and iterating. If we get it right, it will serve as an example for other industries, and robots will finally become ubiquitous in our lives.

[ CMU RI ]

Photo: Science and Society Picture Library/Getty Images Neurophysiologist W. Grey Walter built his cybernetic tortoises to elucidate the functions of the brain.

In the robotics family tree, Roomba’s ancestors were probably Elmer and Elsie, a pair of cybernetic tortoises invented in the 1940s by neurophysiologist W. Grey Walter. The robots could “see” by means of a rotating photocell that steered them toward a light source. If the light was too bright, they would retreat and continue their exploration in a new direction. Likewise, when they ran into obstacles, a touch sensor would compel the tortoises to reverse and change course. In this way, Elmer and Elsie slowly explored their surroundings.

Walter was an early researcher into electroencephalography (EEG), a technique for detecting the electrical activity of the brain using electrodes attached to the scalp. Among his notable clinical breakthroughs was the first diagnosis of a brain tumor by EEG. In 1939 he joined the newly established Burden Neurological Institute in Bristol, England, as head of its physiology department, and he remained at the Burden for the rest of his career.

Norbert Wiener’s cybernetics movement gave birth to a menagerie of cybernetic creatures

In the late 1940s, Walter became involved in the emerging community of scientists who were interested in cybernetics. The field’s founder, Norbert Wiener, defined cybernetics as “the scientific study of control and communication in the animal and the machine.” In the first wave of cybernetics, people were keen on building machines to model animal behavior. Claude Shannon played around with a robotic mouse named Theseus that could navigate mazes. W. Ross Ashby built the Homeostat, a machine that automatically adapted to inputs so as to remain in a stable state.

Walter’s contribution to this cybernetic menagerie was an electromechanical tortoise, which he began working on in the spring of 1948 in his spare time. His first attempts were inelegant. In 1951 W. J. “Bunny” Warren, an electrical engineer at the Burden, constructed six tortoises for Walter that were more solidly engineered. Two of these six tortoises became Elmer and Elsie, their names taken from Grey’s somewhat contrived acronym: ELectro MEchanical Robots, Light Sensitive, with Internal and External stability.

Photo: Larry Burrows/The LIFE Picture Collection/Getty Images In this time-lapse photo from 1950, Walter has a smoke while one of his cybernetic tortoises roams about the living room.

Walter considered Elmer and Elsie to be the Adam and Eve of a new species, Machina speculatrix. The scientific nomenclature reflected the robots’ exploratory or speculative behavior. The creatures each had a smooth protective shell and a protruding neck, so Walter put them in the Linnaean genus Testudo, or tortoise. Extending his naming scheme, he dubbed Shannon’s maze-crawling mouse Machina labyrinthia and Ashby’s Homestat Machina sopora (sleeping machine).

Did W. Grey Walter’s cybernetic tortoises exhibit free will?

Each tortoise moved on three wheels with two sets of motors, one for locomotion and the other for steering. Its “brain” consisted of two vacuum tubes, which Walter said gave it the equivalent of two functioning neurons.

Despite such limited equipment, the tortoises displayed free will, he claimed. In the May 1950 issue of Scientific American [PDF], he described how the photocell atop the tortoise’s neck scanned the surroundings for a light source. The photocell was attached to the steering mechanism, and as the tortoise searched, it moved forward in a circular pattern. Walter compared this to the alpha rhythm of the electric pulses in the brain, which sweeps over the visual areas and at the same time releases impulses for the muscles to move.

In a dark room, the tortoise wandered aimlessly. When it detected a light, the tortoise moved directly toward the source. But if the light surpassed a certain brightness, it retreated. Presented with two distinct light sources, it would trace a path back and forth between the pair. “Like a moth to a flame,” Walter wrote, the tortoise oscillated between seeking and withdrawing from the lights.

The tortoise had a running light that came on when it was searching for a light source. Originally, this was just to signal to observers what command the robot was processing, but it had some unintended consequences. If Elmer happened to catch a glimpse of itself in a mirror, it would begin moving closer to the image until the reflected light became too bright, and then it would retreat. In his 1953 book The Living Brain, Walter compared the robot to “a clumsy Narcissus.”

Similarly, if Elmer and Elsie were in the same area and saw the other’s light, they would lock onto the source and approach, only to veer away when they got too close. Ever willing to describe the machines in biological terms, Walter called this a mating dance where the unfortunate lovers could never “consummate their ‘desire.’ ”

The tortoise’s shell did much more than just protect the machine’s electromechanical insides. If the robot bumped into an obstacle, a touch sensor in the shell caused it to reverse and change direction. In this manner it could explore its surroundings despite being effectively blind.

M. speculatrix was powered by a hearing-aid battery and a 6-volt battery. When its wanderings were done—that is, when its battery levels were low—it made its way to its hutch. There, it could connect its circuits, turn off its motors, and recharge.

Elmer and Elsie were a huge hit at the 1951 Festival of Britain

During the summer of 1951, Elmer and Elsie performed daily in the science exhibition at the Festival of Britain. Held at sites throughout the United Kingdom, the festival drew millions of visitors. The tortoises were a huge hit. Attendees wondered at their curious activity as they navigated their pen, moved toward and away from light sources, and avoided obstacles in their path. A third tortoise with a transparent shell was on display to showcase the inner workings and to advertise the component parts.

Even as M. speculatrix was wowing the public, Walter was investigating the next evolution of the species. Elmer and Elsie successfully demonstrated unpredictable behavior that could be compared with a basic animal reaction to stimuli, but they never learned from their experience. They had no memory and could not adapt to their environment.

Walter dubbed his next experimental tortoise M. docilis, from the Latin for teachable, and he attempted to build a robot that could mimic Pavlovian conditioned responses. Where the Russian psychologist used dogs, food, and some sort of sound, Walter used his cybernetic tortoises, light, and a whistle. That is, he taught his M. docilis tortoises that the sound of a whistle was the same as a light source and that the tortoise would move toward the sound even if no light was present.

Walter published his findings on M. docilis in a second Scientific American article, “A Machine That Learns” [PDF]. This follow-up article had much to offer electrical engineers, including circuit diagrams and a technical discussion of some of the challenges in constructing the robots, such as amplifying the sound of the whistle to overcome the noise of the motors.

Photo: Larry Burrows/The LIFE Picture Collection/Getty Images The robo-tortoise returns to its hutch to recharge its battery.

The brain of M. docilis was CORA (short for COnditioned Reflex Analog) circuitry, which detected repeated coincidental sensory inputs on separate channels, such as light and sound that happened at the same time. After CORA logged a certain number of repetitions, often between 10 and 20 instances, it linked the resulting behavior, which Walter described as a conditioned response. Just as CORA could learn a behavior, it could also forget it. If the operator teased the tortoise by withholding the light from the sound of the whistle, CORA would delink the response.

At the end of his article, Walter acknowledged that future experiments with more circuits and inputs were feasible, but the increase in complexity would come at the cost of stability. Eventually, scientists would find it too difficult to model the behavior and understand the reactions to multiple stimuli.

Walter discontinued his experiments with robotic tortoises after CORA, and the research was not picked up by others. As the historian of science Andrew Pickering noted in his 2009 book, The Cybernetic Brain, “CORA remains an unexploited resource in the history of cybernetics.”

Walter’s legacy lives on in his tortoises. The late Rueben Hoggett compiled a treasure trove of archival research on Walter’s tortoises, which can be found on Hoggett’s website, Cybernetic Zoo. The three tortoises from the Festival of Britain were auctioned off, and the winner, Wes Clutterbuck, nicknamed them Slo, Mo, and Shun. Although two were later destroyed in a fire, the Clutterbuck family donated the one with a transparent shell to the Smithsonian Institution. The only other known surviving tortoise from the original six crafted by Bunny Warren is at the Science Museum in London. It is currently on exhibit in the Making the Modern World Gallery.

An abridged version of this article appears in the March 2020 print issue as “The Proto-Roomba.”

Part of a continuing series looking at photographs of historical artifacts that embrace the boundless potential of technology.

About the Author

Allison Marsh is an associate professor of history at the University of South Carolina and codirector of the university’s Ann Johnson Institute for Science, Technology & Society.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Artificial intelligence and robotic technologies with semi-autonomous learning, reasoning, and decision-making capabilities are increasingly being incorporated into defense, military, and security systems. Unsurprisingly, there is increasing concern about the stability and safety of these systems. In a different sector, runaway interactions between autonomous trading systems in financial markets have produced a series of stock market “flash crashes,” and as a result, those markets now have rules to prevent such interactions from having a significant impact1.

Could the same kinds of unexpected interactions and feedback loops lead to similar instability with defense or security AIs?

Adversarial attacks on AI systems

General concerns about the impacts of defense AIs and robots on stability, whether in isolation or through interaction, have only been exacerbated by recent demonstrations of adversarial attacks against these systems2. Perhaps the most widely-discussed attack cases involve image classification algorithms that are deceived into “seeing” images in noise3, or are easily tricked by pixel-level changes so they classify, say, a turtle as a rifle4. Similarly, game-playing systems that outperform any human (e.g., AlphaGo) can suddenly fail if the game structure or rules are even slightly altered in ways that would not affect a human5. Autonomous vehicles that function reasonably well in ordinary conditions can, with the application of a few pieces of tape, be induced to swerve into the wrong lane or speed through a stop sign6. And the list of adversarial attacks continues to grow and grow over time.

Adversarial attacks pose a tangible threat to the stability and safety of AI and robotic technologies. The exact conditions for such attacks are typically quite unintuitive for humans, so it is difficult to predict when and where the attacks could occur. And even if we could estimate the likelihood of an adversarial attack, the exact response of the AI system can be difficult to predict as well, leading to further surprises and less stable, less safe military engagements and interactions. Even overall assessments of reliability are difficult in the face of adversarial attacks.

We might hope that adversarial attacks would be relatively rare in the everyday world, since “random noise” that targets image classification algorithms is actually far from random: The tape on the stop sign must be carefully placed, the pixel-level perturbations added to the image must be carefully calculated, and so on. Significant effort is required to construct an adversarial attack, and so we might simply deploy our AI and robotic systems with the hope that the everyday world will not conspire to deceive them. 

Unfortunately, this confidence is almost certainly unwarranted for defense or security technologies. These systems will invariably be deployed in contexts where the other side has the time, energy, and ability to develop and construct exactly these types of adversarial attacks. AI and robotic technologies are particularly appealing for deployment in enemy-controlled or enemy-contested areas since those environments are riskiest for our human soldiers, in large part because the other side has the most control over the environment. 

Defenses against adversarial attacks

Although adversarial attacks on defense and military AIs and robots are likely, they are not necessarily destabilizing, particularly since humans are typically unaffected by these attacks. We can easily recognize that a turtle is not a rifle even with random noise, we view tape on a stop sign as an annoyance rather than something that disrupts our ability to follow the rules of the road, and so on. Of course, there are complexities, but we can safely say that human performance is strongly robust to adversarial attacks against AIs. Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

This recommendation is attractive, but is also necessarily limited in scope to applications where a human can be directly involved. In the case of intelligence, surveillance, and reconnaissance (ISR) systems, however, substantive human interaction might not be possible. AI technologies are being increasingly used to handle the enormous volumes of data generated for ISR purposes. AI technologies for ISR now play a significant role in the creation and maintenance of situational awareness for human decision-makers, and in such situations, the destabilizing risks of adversarial attacks again rear their heads.

As an extreme example, consider the intersection of AI and nuclear weapons. One might think that these two technologies should never meet, since we ought not delegate the decision to use nuclear force to an AI. Regardless, AI systems potentially (or perhaps actually) do play a role in nuclear weapons, namely in the ISR that informs human decisions about whether to use such weapons. The worldwide sensor and data input streams almost certainly cannot be processed entirely by human beings. We will need to use (or perhaps already do use) AI technologies without a human in the loop to help us understand our world, and so there may not always be a human to intercept adversarial attacks against those systems.

Our situational awareness can therefore be affected or degraded due to deliberately distorted “perceptions” coming from the AI analyses. These problems are not limited to the extreme case of nuclear weapons—any military or security action where situational awareness depends partly on unmonitored ISR AI will be vulnerable to adversarial attacks in ways that a human cannot necessarily recognize and rectify.

Perhaps we could simply monitor the ISR AI by requiring it to provide evidence or explanations of its analyses that are sufficiently detailed for a human to be able to recognize an adversarial attack. However, if we consider only “explainable AIs” in these contexts, then we are restricting the space of possible models, and so arguably9 placing an artificial upper bound on system performance. Moreover, many AI systems are moving some computation onto the sensors themselves to help overcome processing and memory constraints.

For example, AI on the sensors might perform anomaly detection, leaving a higher level AI system to process only potential outliers. These distributed systems might not be able to retain the evidence (for example, the original image) required for human recognition of an adversarial attack. And in real-world cases, we might not have the time to look at the evidence even if it were provided, and so would not be able to respond to destabilizing adversarial attacks on the ISR AI.

This all might seem to be much ado about nothing new: After all, information gathering has always been susceptible to deception, manipulation, and misinformation. But adversarial attacks can lead to completely bizarre and ridiculous (from a human perspective) behavior from an AI. No ordinary deception would ever lead a human intelligence officer to see a turtle as a rifle, and the use of ISR AI opens the door to much different types of deception, with much different results. Without proper understanding of these potential impacts, the world is likely to be a less stable and less safe place. 

Adversarial attacks can destabilize AI technologies, rendering them less safe, predictable, or reliable. However, we do not necessarily need to worry about them as direct attacks on the decision-making machinery of the system. Instead, we should worry about the corruption of human situational awareness through adversarial AI, which can be equally effective in undermining the safety, stability, and trust in the AI and robotic technologies.

David Danks is L.L. Thurstone professor of philosophy and psychology, and head of the department of philosophy, at Carnegie Mellon University. He is also the chief ethicist of CMU’s Block Center for Technology & Society; co-director of CMU’s Center for Informed Democracy and Social Cybersecurity (IDeaS); and an adjunct member of the Heinz College of Information Systems and Public Policy. His research interests are at the intersection of philosophy, cognitive science, and machine learning. Most recently, Danks has been examining the ethical, psychological, and policy issues around AI and robotics in transportation, healthcare, privacy, and security.

References

1. Serritella, D. M. (2010). “High speed trading begets high speed regulation: SEC response to flash crash, rash.” Illinois Journal of Law, Technology & Policy, 2010 (2).

2. Biggio, B. & Roli, F. (2018). “Wild patterns: Ten years after the rise of adversarial machine learning.” Pattern Recognition, 84, 317-331.

3. Nguyen, A., Yosinski, J., & Clune, J. (2015). “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 427-436).

4. Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2018). “Synthesizing robust adversarial examples.” In Proceedings of the 35th International Conference on Machine Learning (pp. 284-293).

5. Raghu, M., Irpan, A., Andreas, J., Kleinberg, R., Le, Q. V., & Kleinberg, J. (2018). “Can deep reinforcement learning solve Erdos-Selfridge-Spencer games?” Proceedings of ICML.

6. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018). “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1625-1634).

7. Rudin, C. (2018). “Please stop explaining black box models for high stakes decisions.” NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning. arXiv:1811.10154v2

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Artificial intelligence and robotic technologies with semi-autonomous learning, reasoning, and decision-making capabilities are increasingly being incorporated into defense, military, and security systems. Unsurprisingly, there is increasing concern about the stability and safety of these systems. In a different sector, runaway interactions between autonomous trading systems in financial markets have produced a series of stock market “flash crashes,” and as a result, those markets now have rules to prevent such interactions from having a significant impact1.

Could the same kinds of unexpected interactions and feedback loops lead to similar instability with defense or security AIs?

Adversarial attacks on AI systems

General concerns about the impacts of defense AIs and robots on stability, whether in isolation or through interaction, have only been exacerbated by recent demonstrations of adversarial attacks against these systems2. Perhaps the most widely-discussed attack cases involve image classification algorithms that are deceived into “seeing” images in noise3, or are easily tricked by pixel-level changes so they classify, say, a turtle as a rifle4. Similarly, game-playing systems that outperform any human (e.g., AlphaGo) can suddenly fail if the game structure or rules are even slightly altered in ways that would not affect a human5. Autonomous vehicles that function reasonably well in ordinary conditions can, with the application of a few pieces of tape, be induced to swerve into the wrong lane or speed through a stop sign6. And the list of adversarial attacks continues to grow and grow over time.

Adversarial attacks pose a tangible threat to the stability and safety of AI and robotic technologies. The exact conditions for such attacks are typically quite unintuitive for humans, so it is difficult to predict when and where the attacks could occur. And even if we could estimate the likelihood of an adversarial attack, the exact response of the AI system can be difficult to predict as well, leading to further surprises and less stable, less safe military engagements and interactions. Even overall assessments of reliability are difficult in the face of adversarial attacks.

We might hope that adversarial attacks would be relatively rare in the everyday world, since “random noise” that targets image classification algorithms is actually far from random: The tape on the stop sign must be carefully placed, the pixel-level perturbations added to the image must be carefully calculated, and so on. Significant effort is required to construct an adversarial attack, and so we might simply deploy our AI and robotic systems with the hope that the everyday world will not conspire to deceive them. 

Unfortunately, this confidence is almost certainly unwarranted for defense or security technologies. These systems will invariably be deployed in contexts where the other side has the time, energy, and ability to develop and construct exactly these types of adversarial attacks. AI and robotic technologies are particularly appealing for deployment in enemy-controlled or enemy-contested areas since those environments are riskiest for our human soldiers, in large part because the other side has the most control over the environment. 

Defenses against adversarial attacks

Although adversarial attacks on defense and military AIs and robots are likely, they are not necessarily destabilizing, particularly since humans are typically unaffected by these attacks. We can easily recognize that a turtle is not a rifle even with random noise, we view tape on a stop sign as an annoyance rather than something that disrupts our ability to follow the rules of the road, and so on. Of course, there are complexities, but we can safely say that human performance is strongly robust to adversarial attacks against AIs. Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

This recommendation is attractive, but is also necessarily limited in scope to applications where a human can be directly involved. In the case of intelligence, surveillance, and reconnaissance (ISR) systems, however, substantive human interaction might not be possible. AI technologies are being increasingly used to handle the enormous volumes of data generated for ISR purposes. AI technologies for ISR now play a significant role in the creation and maintenance of situational awareness for human decision-makers, and in such situations, the destabilizing risks of adversarial attacks again rear their heads.

As an extreme example, consider the intersection of AI and nuclear weapons. One might think that these two technologies should never meet, since we ought not delegate the decision to use nuclear force to an AI. Regardless, AI systems potentially (or perhaps actually) do play a role in nuclear weapons, namely in the ISR that informs human decisions about whether to use such weapons. The worldwide sensor and data input streams almost certainly cannot be processed entirely by human beings. We will need to use (or perhaps already do use) AI technologies without a human in the loop to help us understand our world, and so there may not always be a human to intercept adversarial attacks against those systems.

Our situational awareness can therefore be affected or degraded due to deliberately distorted “perceptions” coming from the AI analyses. These problems are not limited to the extreme case of nuclear weapons—any military or security action where situational awareness depends partly on unmonitored ISR AI will be vulnerable to adversarial attacks in ways that a human cannot necessarily recognize and rectify.

Perhaps we could simply monitor the ISR AI by requiring it to provide evidence or explanations of its analyses that are sufficiently detailed for a human to be able to recognize an adversarial attack. However, if we consider only “explainable AIs” in these contexts, then we are restricting the space of possible models, and so arguably9 placing an artificial upper bound on system performance. Moreover, many AI systems are moving some computation onto the sensors themselves to help overcome processing and memory constraints.

For example, AI on the sensors might perform anomaly detection, leaving a higher level AI system to process only potential outliers. These distributed systems might not be able to retain the evidence (for example, the original image) required for human recognition of an adversarial attack. And in real-world cases, we might not have the time to look at the evidence even if it were provided, and so would not be able to respond to destabilizing adversarial attacks on the ISR AI.

This all might seem to be much ado about nothing new: After all, information gathering has always been susceptible to deception, manipulation, and misinformation. But adversarial attacks can lead to completely bizarre and ridiculous (from a human perspective) behavior from an AI. No ordinary deception would ever lead a human intelligence officer to see a turtle as a rifle, and the use of ISR AI opens the door to much different types of deception, with much different results. Without proper understanding of these potential impacts, the world is likely to be a less stable and less safe place. 

Adversarial attacks can destabilize AI technologies, rendering them less safe, predictable, or reliable. However, we do not necessarily need to worry about them as direct attacks on the decision-making machinery of the system. Instead, we should worry about the corruption of human situational awareness through adversarial AI, which can be equally effective in undermining the safety, stability, and trust in the AI and robotic technologies.

David Danks is L.L. Thurstone professor of philosophy and psychology, and head of the department of philosophy, at Carnegie Mellon University. He is also the chief ethicist of CMU’s Block Center for Technology & Society; co-director of CMU’s Center for Informed Democracy and Social Cybersecurity (IDeaS); and an adjunct member of the Heinz College of Information Systems and Public Policy. His research interests are at the intersection of philosophy, cognitive science, and machine learning. Most recently, Danks has been examining the ethical, psychological, and policy issues around AI and robotics in transportation, healthcare, privacy, and security.

References

1. Serritella, D. M. (2010). “High speed trading begets high speed regulation: SEC response to flash crash, rash.” Illinois Journal of Law, Technology & Policy, 2010 (2).

2. Biggio, B. & Roli, F. (2018). “Wild patterns: Ten years after the rise of adversarial machine learning.” Pattern Recognition, 84, 317-331.

3. Nguyen, A., Yosinski, J., & Clune, J. (2015). “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 427-436).

4. Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2018). “Synthesizing robust adversarial examples.” In Proceedings of the 35th International Conference on Machine Learning (pp. 284-293).

5. Raghu, M., Irpan, A., Andreas, J., Kleinberg, R., Le, Q. V., & Kleinberg, J. (2018). “Can deep reinforcement learning solve Erdos-Selfridge-Spencer games?” Proceedings of ICML.

6. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018). “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1625-1634).

7. Rudin, C. (2018). “Please stop explaining black box models for high stakes decisions.” NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning. arXiv:1811.10154v2

This article was originally published on LinkedIn. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Build a rover, send it to the Moon, sell the movie rights.

That was our first business model at iRobot. Way back in 1990. We thought it would be how we’d first change the world. It’s ironic, of course, that through that model, changing the world meant sending a robot to another one. Sadly, that business model failed. And it wouldn’t be our last failed business model. Not by a long shot.

Photo: iRobot

Why? Because changing the world through robots, it turns out, is no easy task.

Perhaps the biggest challenge back when we started in 1990 was that there existed no rule book on how to do it. There weren’t many robots, let alone robot companies, let alone any kind of robot industry. We would have to build it. All of it.

Walking that path meant being comfortable with ambiguity, and comfortable with the knowledge that not everything we tried was going to work–at least not in the way we originally conceived. It was and continues to be the cost of inventing the future.

But walking that trying path also meant learning from our mistakes, dusting ourselves off, trying again, and eventually, yes, doing what we set out to do: Change the world through robots.

We’ve learned so much along the way–what have we learned in our 30-year journey building the robot industry?

Robots are hard

I’ve said it before, and I’ll say it again: When we first started iRobot we had to invent every element of the robot. Spatial navigation was a robot problem, voice recognition was a robot problem, machine vision was a robot problem, just to name a few. Back then, no one else had set out to solve these hard problems. Because so many of these problems existed, the robot industry, if it could be called that, moved in anti-dog years. Fortunately, times have changed and the ecosystem around the technologies that make robots possible is much richer… But back then… it was just us.

But even today, with a much larger ecosystem of bright minds solving for the hard tech problems, getting a robot to work successfully still means getting just the right mix of mechanical, electrical, and software engineering, connectivity, and data science into a robot form factor that people trust and want to invite into their home.

Photo: iRobot

Speaking of trust, therein lied another challenge. Even when we did invent a robot that worked extraordinarily well–Roomba–consumers simply didn’t believe a robot could do what we said Roomba was capable of. It turns out that the principal objection to purchasing a robot for much of the last 30 years is a lack of belief that it could possibly work.

For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. We were no exception: We tried 14 business models before we arrived at one that sustainably worked.

But that’s not all: Even when you build a robot right, you can still somehow build it wrong. We experienced this with Roomba. We built it to match the reliability standards of European upright vacuums, something of which we were very proud. Of course, we didn’t anticipate that our customers would run their Roomba once per day, rather than the once per week average the European standard set. And as the first generation of Roomba robots broke down two years ahead of schedule, we learned that reverse logistics, great customer service, and a generous return policy were a very important part of a good robot–as was the realization that we couldn’t compare usage to whatever traditional means of action a good robot might take the place of.

And yet while building a robot that was durable, that people wanted and trusted was hard enough, 30 years building robots has also taught us that…

Good business models are harder to build than good robots

Let’s state this one right off the bat: For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. It turns out that a business model is as important as the tech, but much more rarely found in a robot company. And for a long time we were no exception: We tried 14 business models before we arrived at one that sustainably worked.

Image: iRobot

But the tenuous nature of our business models did teach us the value of extending the runway for our business until we found one that worked. And how does one extend the runway most effectively? By managing risk.

It’s one of the great misunderstandings of entrepreneurship–that great entrepreneurs are risk takers. Great entrepreneurs are not great risk takers… they’re great risk managers. And this was something we at iRobot were and are exceptionally good at.

How did we manage risk early on? Through partnerships. The kind of partnership we looked for were ones in which there was a big company–one that had a lot of money, a channel to the marketplace, and knowledge of that marketplace, but for whatever reason lacked belief that they themselves were innovative. We were a small company with no money, but believed ourselves to have cool technology, and be highly capable of innovation.

Image: iRobot

What we’d do was give our partner, the big company, absolute control. By doing this, it allowed us to say that since they could cancel the partnership at any time, we needed them to cover our costs… which they did. But we also didn’t ask them to pay us profit upfront. By not having the pay profit upfront, it makes obvious that we’re sharing the value that the partnership would ultimately create, and in a worst-case scenario for our partner, if the partnership didn’t result in a successful product, they got very inexpensive high-quality research.

This “asymmetric strategic partnership” approach not only provided the funds needed to sustain our business when we didn’t have a sustainable business model–the “failure” of those partnerships actually led to our ultimate success. Why? Because…

Innovation and failure come hand-in-hand

While this is far from a groundbreaking realization, its applicability to iRobot is quite unique. Because for us to become successful, it turns out that we had to learn the lessons from failing to earn royalties on robot toys (business model #3), failing to license technology for industrial floor-cleaning robots (business model #8), and failing to sell land mine clearance robots (business model #11).

Image: iRobot

Why? Because #3 taught us to manufacture at scale, #8 taught us how to clean floors, and #11 taught us how to navigate and cover large spaces.  All of which gave us the knowledge and capability to build… Roomba.

Image: iRobot Yes, you can change the world through robots

We did. In more ways the one. We changed the world by eliminating the need for people to vacuum the house themselves. By IPOing, we showed that a robotics company could be successful–which gave investors more reason to put money into robotics companies around the world.

But perhaps the most important way we’ve succeeded in changing the world is by making robots a daily reality for it. And how do we know that robots are now a reality? Because for the better part of the first 30 years of iRobot, what people said to me about robots–and Roomba specifically–was, “I can’t believe it actually works.”

But now, the question they ask me is, “Why can’t robots do more?

It is a great question. And that is what the next 30 years of iRobot will be about.

Colin Angle is chairman of the board, chief executive officer, and founder of iRobot. Celebrating its 30th year, iRobot has grown from an MIT startup to become a global leader in consumer robots, with more than 30 million sold worldwide. You can follow him on Twitter at @ColinAngle.

This article was originally published on LinkedIn. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Build a rover, send it to the Moon, sell the movie rights.

That was our first business model at iRobot. Way back in 1990. We thought it would be how we’d first change the world. It’s ironic, of course, that through that model, changing the world meant sending a robot to another one. Sadly, that business model failed. And it wouldn’t be our last failed business model. Not by a long shot.

Photo: iRobot

Why? Because changing the world through robots, it turns out, is no easy task.

Perhaps the biggest challenge back when we started in 1990 was that there existed no rule book on how to do it. There weren’t many robots, let alone robot companies, let alone any kind of robot industry. We would have to build it. All of it.

Walking that path meant being comfortable with ambiguity, and comfortable with the knowledge that not everything we tried was going to work–at least not in the way we originally conceived. It was and continues to be the cost of inventing the future.

But walking that trying path also meant learning from our mistakes, dusting ourselves off, trying again, and eventually, yes, doing what we set out to do: Change the world through robots.

We’ve learned so much along the way–what have we learned in our 30-year journey building the robot industry?

Robots are hard

I’ve said it before, and I’ll say it again: When we first started iRobot we had to invent every element of the robot. Spatial navigation was a robot problem, voice recognition was a robot problem, machine vision was a robot problem, just to name a few. Back then, no one else had set out to solve these hard problems. Because so many of these problems existed, the robot industry, if it could be called that, moved in anti-dog years. Fortunately, times have changed and the ecosystem around the technologies that make robots possible is much richer… But back then… it was just us.

But even today, with a much larger ecosystem of bright minds solving for the hard tech problems, getting a robot to work successfully still means getting just the right mix of mechanical, electrical, and software engineering, connectivity, and data science into a robot form factor that people trust and want to invite into their home.

Photo: iRobot

Speaking of trust, therein lied another challenge. Even when we did invent a robot that worked extraordinarily well–Roomba–consumers simply didn’t believe a robot could do what we said Roomba was capable of. It turns out that the principal objection to purchasing a robot for much of the last 30 years is a lack of belief that it could possibly work.

For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. We were no exception: We tried 14 business models before we arrived at one that sustainably worked.

But that’s not all: Even when you build a robot right, you can still somehow build it wrong. We experienced this with Roomba. We built it to match the reliability standards of European upright vacuums, something of which we were very proud. Of course, we didn’t anticipate that our customers would run their Roomba once per day, rather than the once per week average the European standard set. And as the first generation of Roomba robots broke down two years ahead of schedule, we learned that reverse logistics, great customer service, and a generous return policy were a very important part of a good robot–as was the realization that we couldn’t compare usage to whatever traditional means of action a good robot might take the place of.

And yet while building a robot that was durable, that people wanted and trusted was hard enough, 30 years building robots has also taught us that…

Good business models are harder to build than good robots

Let’s state this one right off the bat: For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. It turns out that a business model is as important as the tech, but much more rarely found in a robot company. And for a long time we were no exception: We tried 14 business models before we arrived at one that sustainably worked.

Image: iRobot

But the tenuous nature of our business models did teach us the value of extending the runway for our business until we found one that worked. And how does one extend the runway most effectively? By managing risk.

It’s one of the great misunderstandings of entrepreneurship–that great entrepreneurs are risk takers. Great entrepreneurs are not great risk takers… they’re great risk managers. And this was something we at iRobot were and are exceptionally good at.

How did we manage risk early on? Through partnerships. The kind of partnership we looked for were ones in which there was a big company–one that had a lot of money, a channel to the marketplace, and knowledge of that marketplace, but for whatever reason lacked belief that they themselves were innovative. We were a small company with no money, but believed ourselves to have cool technology, and be highly capable of innovation.

Image: iRobot

What we’d do was give our partner, the big company, absolute control. By doing this, it allowed us to say that since they could cancel the partnership at any time, we needed them to cover our costs… which they did. But we also didn’t ask them to pay us profit upfront. By not having the pay profit upfront, it makes obvious that we’re sharing the value that the partnership would ultimately create, and in a worst-case scenario for our partner, if the partnership didn’t result in a successful product, they got very inexpensive high-quality research.

This “asymmetric strategic partnership” approach not only provided the funds needed to sustain our business when we didn’t have a sustainable business model–the “failure” of those partnerships actually led to our ultimate success. Why? Because…

Innovation and failure come hand-in-hand

While this is far from a groundbreaking realization, its applicability to iRobot is quite unique. Because for us to become successful, it turns out that we had to learn the lessons from failing to earn royalties on robot toys (business model #3), failing to license technology for industrial floor-cleaning robots (business model #8), and failing to sell land mine clearance robots (business model #11).

Image: iRobot

Why? Because #3 taught us to manufacture at scale, #8 taught us how to clean floors, and #11 taught us how to navigate and cover large spaces.  All of which gave us the knowledge and capability to build… Roomba.

Image: iRobot Yes, you can change the world through robots

We did. In more ways the one. We changed the world by eliminating the need for people to vacuum the house themselves. By IPOing, we showed that a robotics company could be successful–which gave investors more reason to put money into robotics companies around the world.

But perhaps the most important way we’ve succeeded in changing the world is by making robots a daily reality for it. And how do we know that robots are now a reality? Because for the better part of the first 30 years of iRobot, what people said to me about robots–and Roomba specifically–was, “I can’t believe it actually works.”

But now, the question they ask me is, “Why can’t robots do more?

It is a great question. And that is what the next 30 years of iRobot will be about.

Colin Angle is chairman of the board, chief executive officer, and founder of iRobot. Celebrating its 30th year, iRobot has grown from an MIT startup to become a global leader in consumer robots, with more than 30 million sold worldwide. You can follow him on Twitter at @ColinAngle.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.

I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.

AI deception: How to define it?

If we take a rather long view of history, deception may be as old as the world itself, and it is certainly not the sole provenance of human beings. Adaptation and evolution for survival with traits like camouflage are deceptive acts, as are forms of mimicry commonly seen in animals. But pinning down exactly what constitutes deception for an AI agent is not an easy task—it requires quite a bit of thinking about acts, outcomes, agents, targets, means and methods, and motives. What we include or exclude in that calculation may then have wide ranging implications about what needs immediate regulation, policy guidance, or technological solutions. I will only focus on a couple of items here, namely intent and act type, to highlight this point.

What is deception? Bond and Robinson argue that deception is “false communication to the benefit of the communicator.”1 Whaley argues that deception is also the communication of information provided with the intent to manipulate another.2 These seem pretty straightforward approaches, except when you try to press on the idea of what constitutes “intent” and what is required to meet that threshold, as well as whether or not the false communication requires the intent to be explicitly beneficial to the deceiver. Moreover, depending on which stance you take, deception for altruistic reasons may be excluded entirely. Imagine if you asked your AI-enabled robot butler, “How do I look?” To which it answers, “Very nice.”

Let’s start with intent. Intent requires a theory of mind, meaning that the agent has some understanding of itself, and that it can reason about other external entities and their intentions, desires, states, and potential behaviors.3 If deception requires intent in the ways described above, then true AI deception would require an AI to possess a theory of mind. We might kick the can on that conclusion for a bit and claim that current forms of AI deception instead rely on human intent—where some human is using AI as a tool or means to carry out that person’s intent to deceive.

Or, we may not: Just because current AI agents lack a theory of mind doesn’t mean that they cannot learn to deceive. In multi-agent AI systems, some agents can learn deceptive behaviors without having a true appreciation or comprehension of what “deception” actually is. This could be as simple as hiding resources or information, or providing false information to achieve some goal. If we then put aside the theory of mind for the moment and instead posit that intention is not a prerequisite for deception and that an agent can unintentionally deceive, then we really have opened the aperture for existing AI agents to deceive in many ways. 

What about the way in which deception occurs? That is, what are the deceptive act types? We can identify two broad categories here: 1) acts of commission, where an agent actively engages in a behavior like sending misinformation; and 2) acts of omission, where an agent is passive but may be withholding information or hiding. AI agents can learn all sorts of these types of behaviors given the right conditions.4 Just consider how AI agents used for cyber defense may learn to signal various forms of misinformation, or how swarms of AI-enabled robotic systems could learn deceptive behaviors on a battlefield to escape adversary detection. In more pedestrian examples, perhaps a rather poorly specified or corrupted AI tax assistant omits various types of income on a tax return to minimize the likelihood of owing money to the relevant authorities.

Preparing ourselves against AI deception

The first step towards preparing for our coming AI future is to recognize that such systems already do deceive, and are likely to continue to deceive. How that deception occurs, whether it is a desirable trait (such as with our adaptive swarms), and whether we can actually detect when it is occurring are going to be ongoing challenges. Once we acknowledge this simple but true fact, we can begin to undergo the requisite analysis of what exactly constitutes deception, whether and to whom it is beneficial, and how it may pose risks. 

This is no small task, and it will require not only interdisciplinary work from AI experts, but also input from sociologists, psychologists, political scientists, lawyers, ethicists, and policy wonks. For military AI systems, it will also require domain and mission knowledge. In short, developing a comprehensive framework for AI deception is a crucial step if we are not to find ourselves on the back foot. 

We need to begin thinking about how to engineer novel solutions to mitigate unwanted deception by AI agents. This goes beyond current detection research, and requires thinking about environments, optimization problems, and how AI agents model other AI agents and their emergent effects could yield undesirable deceptive behaviors.

Furthermore, once this framework is in place, we need to begin thinking about how to engineer novel solutions to identify and mitigate unwanted deception by AI agents. This goes beyond current detection research, and moving forward requires thinking about environments, optimization problems, and how AI agents model other AI agents and their interactive or emergent effects could yield risky or undesirable deceptive behaviors. 

We presently face a myriad of challenges related to AI deception, and these challenges are only going to increase as the cognitive capacities of AI increase. The desire of some to create AI systems with a rudimentary theory of mind and social intelligence is a case in point to be socially intelligent one must be able to understand and to “manage” the actions of others5, and if this ability to understand another’s feelings, beliefs, emotions, and intentions exists, along with the ability to act to influence those feelings, beliefs, or actions, then deception is much more likely to occur.

However, we do not need to wait for artificial agents to possess a theory of mind or social intelligence for deception with and from AI systems. We should instead begin thinking about potential technological, policy, legal, and ethical solutions to these coming problems before AI gets more advanced than it already is. With a clearer understanding of the landscape, we can analyze potential responses to AI deception, and begin designing AI systems for truth.

Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.

References

1. Bond CF, Robinson M (1988), “The evolution of deception.” J Nonverbal Behav 12(4):295–307. Note also that this definition precludes certain forms of deception from altruistic or paternalistic reasons.

2. B. Whaley, “Toward a general theory of deception,” Journal of Strategic Studies, vol. 5, no. 1, pp. 178–192, Mar. 1982. 

3. Cheney DL, Seyfarth RM, “Baboon metaphysics: the evolution of a social mind.” University of Chicago Press, Chicago, 2008.

4. J. F. Dunnigan and A. A. Nofi, “Victory and deceit, 2nd edition: Deception and trickery in war,” Writers Press Books, 2001. J. Shim and R.C. Arkin, “A Taxonomy of Robot Deception and Its Benefits in HRI” IEEE International Conference on Systems, Man, and Cybernetics, 2013. S. Erat and U. Gneezy, “White lies,” Rady Working paper, Rady School of Management, UC San Diego, 2009. N. C. Rowe, “Designing good deceptions in defense of information systems,” in Proceedings of the 20th Annual Computer Security Applications Conference, ser. ACSAC ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 418–427.

5. E.L. Thorndike. “Intelligence and Its Use.” Harpers Magazine, Vol. 140, 1920: p. 228. Thorndike’s early definition of social intelligence has been widely used and updated for the past 100 years. Even current attempts in cognitive science have looked at separating out the tasks of “understanding” and “acting,” which maps directly to Thorndike’s language of “understand” and “manage”. Cf: M.I. Brown, A. Ratajska, S.L. Hughes, J.B. Fishman, E. Huerta, and C.F. Chabris. “The Social Shape Tests: A New Measure of Social Intelligence, Mentalizing and Theory of Mind.” Personality and Individual Differences, vol. 143, 2019: 107-117.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.

I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.

AI deception: How to define it?

If we take a rather long view of history, deception may be as old as the world itself, and it is certainly not the sole provenance of human beings. Adaptation and evolution for survival with traits like camouflage are deceptive acts, as are forms of mimicry commonly seen in animals. But pinning down exactly what constitutes deception for an AI agent is not an easy task—it requires quite a bit of thinking about acts, outcomes, agents, targets, means and methods, and motives. What we include or exclude in that calculation may then have wide ranging implications about what needs immediate regulation, policy guidance, or technological solutions. I will only focus on a couple of items here, namely intent and act type, to highlight this point.

What is deception? Bond and Robinson argue that deception is “false communication to the benefit of the communicator.”1 Whaley argues that deception is also the communication of information provided with the intent to manipulate another.2 These seem pretty straightforward approaches, except when you try to press on the idea of what constitutes “intent” and what is required to meet that threshold, as well as whether or not the false communication requires the intent to be explicitly beneficial to the deceiver. Moreover, depending on which stance you take, deception for altruistic reasons may be excluded entirely. Imagine if you asked your AI-enabled robot butler, “How do I look?” To which it answers, “Very nice.”

Let’s start with intent. Intent requires a theory of mind, meaning that the agent has some understanding of itself, and that it can reason about other external entities and their intentions, desires, states, and potential behaviors.3 If deception requires intent in the ways described above, then true AI deception would require an AI to possess a theory of mind. We might kick the can on that conclusion for a bit and claim that current forms of AI deception instead rely on human intent—where some human is using AI as a tool or means to carry out that person’s intent to deceive.

Or, we may not: Just because current AI agents lack a theory of mind doesn’t mean that they cannot learn to deceive. In multi-agent AI systems, some agents can learn deceptive behaviors without having a true appreciation or comprehension of what “deception” actually is. This could be as simple as hiding resources or information, or providing false information to achieve some goal. If we then put aside the theory of mind for the moment and instead posit that intention is not a prerequisite for deception and that an agent can unintentionally deceive, then we really have opened the aperture for existing AI agents to deceive in many ways. 

What about the way in which deception occurs? That is, what are the deceptive act types? We can identify two broad categories here: 1) acts of commission, where an agent actively engages in a behavior like sending misinformation; and 2) acts of omission, where an agent is passive but may be withholding information or hiding. AI agents can learn all sorts of these types of behaviors given the right conditions.4 Just consider how AI agents used for cyber defense may learn to signal various forms of misinformation, or how swarms of AI-enabled robotic systems could learn deceptive behaviors on a battlefield to escape adversary detection. In more pedestrian examples, perhaps a rather poorly specified or corrupted AI tax assistant omits various types of income on a tax return to minimize the likelihood of owing money to the relevant authorities.

Preparing ourselves against AI deception

The first step towards preparing for our coming AI future is to recognize that such systems already do deceive, and are likely to continue to deceive. How that deception occurs, whether it is a desirable trait (such as with our adaptive swarms), and whether we can actually detect when it is occurring are going to be ongoing challenges. Once we acknowledge this simple but true fact, we can begin to undergo the requisite analysis of what exactly constitutes deception, whether and to whom it is beneficial, and how it may pose risks. 

This is no small task, and it will require not only interdisciplinary work from AI experts, but also input from sociologists, psychologists, political scientists, lawyers, ethicists, and policy wonks. For military AI systems, it will also require domain and mission knowledge. In short, developing a comprehensive framework for AI deception is a crucial step if we are not to find ourselves on the back foot. 

We need to begin thinking about how to engineer novel solutions to mitigate unwanted deception by AI agents. This goes beyond current detection research, and requires thinking about environments, optimization problems, and how AI agents model other AI agents and their emergent effects could yield undesirable deceptive behaviors.

Furthermore, once this framework is in place, we need to begin thinking about how to engineer novel solutions to identify and mitigate unwanted deception by AI agents. This goes beyond current detection research, and moving forward requires thinking about environments, optimization problems, and how AI agents model other AI agents and their interactive or emergent effects could yield risky or undesirable deceptive behaviors. 

We presently face a myriad of challenges related to AI deception, and these challenges are only going to increase as the cognitive capacities of AI increase. The desire of some to create AI systems with a rudimentary theory of mind and social intelligence is a case in point to be socially intelligent one must be able to understand and to “manage” the actions of others5, and if this ability to understand another’s feelings, beliefs, emotions, and intentions exists, along with the ability to act to influence those feelings, beliefs, or actions, then deception is much more likely to occur.

However, we do not need to wait for artificial agents to possess a theory of mind or social intelligence for deception with and from AI systems. We should instead begin thinking about potential technological, policy, legal, and ethical solutions to these coming problems before AI gets more advanced than it already is. With a clearer understanding of the landscape, we can analyze potential responses to AI deception, and begin designing AI systems for truth.

Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.

References

1. Bond CF, Robinson M (1988), “The evolution of deception.” J Nonverbal Behav 12(4):295–307. Note also that this definition precludes certain forms of deception from altruistic or paternalistic reasons.

2. B. Whaley, “Toward a general theory of deception,” Journal of Strategic Studies, vol. 5, no. 1, pp. 178–192, Mar. 1982. 

3. Cheney DL, Seyfarth RM, “Baboon metaphysics: the evolution of a social mind.” University of Chicago Press, Chicago, 2008.

4. J. F. Dunnigan and A. A. Nofi, “Victory and deceit, 2nd edition: Deception and trickery in war,” Writers Press Books, 2001. J. Shim and R.C. Arkin, “A Taxonomy of Robot Deception and Its Benefits in HRI” IEEE International Conference on Systems, Man, and Cybernetics, 2013. S. Erat and U. Gneezy, “White lies,” Rady Working paper, Rady School of Management, UC San Diego, 2009. N. C. Rowe, “Designing good deceptions in defense of information systems,” in Proceedings of the 20th Annual Computer Security Applications Conference, ser. ACSAC ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 418–427.

5. E.L. Thorndike. “Intelligence and Its Use.” Harpers Magazine, Vol. 140, 1920: p. 228. Thorndike’s early definition of social intelligence has been widely used and updated for the past 100 years. Even current attempts in cognitive science have looked at separating out the tasks of “understanding” and “acting,” which maps directly to Thorndike’s language of “understand” and “manage”. Cf: M.I. Brown, A. Ratajska, S.L. Hughes, J.B. Fishman, E. Huerta, and C.F. Chabris. “The Social Shape Tests: A New Measure of Social Intelligence, Mentalizing and Theory of Mind.” Personality and Individual Differences, vol. 143, 2019: 107-117.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today's videos.

These videos show some highlights from the Lake Kivu Challenge, which took place in Rwanda earlier this month. In addition to a conference and forum, international teams and their drones competed in emergency delivery, sample pick-up, and find and assess tasks.

[ Lake Kivu Challenge ]

The DARPA SubT Challenge Urban Circuit is ON!!!

[ SubT ]

One of the ways Microsoft trains autonomous systems is participating in research focused on solving real-world challenges, like aiding first responders in hazardous scenarios. This week, our collaborators at Carnegie Mellon University and Oregon State University, collectively named Team Explorer, are demonstrating tech breakthroughs in this area as they compete in the Feb 18-27, 2020 DARPA Subterranean (SubT) Urban Challenge in Elma, Washington.

The team is looking for another win after taking first place in round one of the DARPA SubT Challenge, the Tunnel Circuit, in August 2019. The competition continues with the Cave Circuit later in 2020, wrapping up with a final event incorporating all three underground environments in 2021.

[ Explorer ] via [ Microsoft ]

Spot can pull rickshaws now?

[ Tested ] via [ Gizmodo ]

Robot hugs!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people- connecting robots to humans and building relationships that last.

[ Roboy ]

Fabian Kung from Malaysia wrote in to share a video of a robot that he's been working on: "We designed and build this mini agile robot as part of our efforts in robotics and artificial intelligence research. It is kept small to reduce the cost and built time. Besides, there is less safety issue with small machine."

[ MMU ]

Thanks Fabian!

Happy (belated) Valentine's Day from Robotiq!

[ Robotiq ]

Happy (belated) Valentine's Day from Sphero!

C'mon dude, just pick all four. They're robots!

[ Sphero ]

Craving a bite out of a freshly grilled ballpark frank? Two robots named Jaco and Baxter can serve one up. Boston University engineers have made a jump in using machine learning to teach robots to perform complex tasks, a framework that could be applied to a host of tasks, like identifying cancerous spots on mammograms or better understanding spoken commands to play music. But first, as a proof of concept—they’ve learned how to prepare the perfect hot dog.

[ BU ]

The latest version of ETH Zurich's Ascento wheel-leg robot has gotten way more robust and capable over the last year.

[ Ascento ]

Snakes live in diverse environments ranging from unbearably hot deserts to lush tropical forests. But regardless of their habitat, they are able to slither up trees, rocks, and shrubbery with ease. By studying how the creatures move, a team of Johns Hopkins engineers have created a snake robot that can nimbly and stably climb large steps. The team's new findings, published in Journal of Experimental Biology and Royal Society Open Science, could advance the creation of search and rescue robots that can successfully navigate treacherous terrain.

[ JHU ]

In a recent demo conducted in Israel, RAFAEL’s Drone Dome C-UAS system performed interceptions of multiple drones, including maneuvering targets, using its hard-kill LASER BEAM director. The system achieved 100% success in all test scenarios. The stages of the interceptions included target detection, identification, and interception with a high-power LASER beam.

[ Rafael ]

EPFL has a little bit of robotics going on sometimes, you know?

[ EPFL ]

This video is basically an ad for ABB, but it's always fun to see robots picking stuff, especially when some of that stuff is tasty.

[ ABB ]

Hayk Martirosyan from Skydio gave a lecture as part of Pieter Abbeel's robotics course at UC Berkeley—this is where you hear about all the secret stuff Skydio is working on next.

[ UC Berkeley ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today's videos.

These videos show some highlights from the Lake Kivu Challenge, which took place in Rwanda earlier this month. In addition to a conference and forum, international teams and their drones competed in emergency delivery, sample pick-up, and find and assess tasks.

[ Lake Kivu Challenge ]

The DARPA SubT Challenge Urban Circuit is ON!!!

[ SubT ]

One of the ways Microsoft trains autonomous systems is participating in research focused on solving real-world challenges, like aiding first responders in hazardous scenarios. This week, our collaborators at Carnegie Mellon University and Oregon State University, collectively named Team Explorer, are demonstrating tech breakthroughs in this area as they compete in the Feb 18-27, 2020 DARPA Subterranean (SubT) Urban Challenge in Elma, Washington.

The team is looking for another win after taking first place in round one of the DARPA SubT Challenge, the Tunnel Circuit, in August 2019. The competition continues with the Cave Circuit later in 2020, wrapping up with a final event incorporating all three underground environments in 2021.

[ Explorer ] via [ Microsoft ]

Spot can pull rickshaws now?

[ Tested ] via [ Gizmodo ]

Robot hugs!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people- connecting robots to humans and building relationships that last.

[ Roboy ]

Fabian Kung from Malaysia wrote in to share a video of a robot that he's been working on: "We designed and build this mini agile robot as part of our efforts in robotics and artificial intelligence research. It is kept small to reduce the cost and built time. Besides, there is less safety issue with small machine."

[ MMU ]

Thanks Fabian!

Happy (belated) Valentine's Day from Robotiq!

[ Robotiq ]

Happy (belated) Valentine's Day from Sphero!

C'mon dude, just pick all four. They're robots!

[ Sphero ]

Craving a bite out of a freshly grilled ballpark frank? Two robots named Jaco and Baxter can serve one up. Boston University engineers have made a jump in using machine learning to teach robots to perform complex tasks, a framework that could be applied to a host of tasks, like identifying cancerous spots on mammograms or better understanding spoken commands to play music. But first, as a proof of concept—they’ve learned how to prepare the perfect hot dog.

[ BU ]

The latest version of ETH Zurich's Ascento wheel-leg robot has gotten way more robust and capable over the last year.

[ Ascento ]

Snakes live in diverse environments ranging from unbearably hot deserts to lush tropical forests. But regardless of their habitat, they are able to slither up trees, rocks, and shrubbery with ease. By studying how the creatures move, a team of Johns Hopkins engineers have created a snake robot that can nimbly and stably climb large steps. The team's new findings, published in Journal of Experimental Biology and Royal Society Open Science, could advance the creation of search and rescue robots that can successfully navigate treacherous terrain.

[ JHU ]

In a recent demo conducted in Israel, RAFAEL’s Drone Dome C-UAS system performed interceptions of multiple drones, including maneuvering targets, using its hard-kill LASER BEAM director. The system achieved 100% success in all test scenarios. The stages of the interceptions included target detection, identification, and interception with a high-power LASER beam.

[ Rafael ]

EPFL has a little bit of robotics going on sometimes, you know?

[ EPFL ]

This video is basically an ad for ABB, but it's always fun to see robots picking stuff, especially when some of that stuff is tasty.

[ ABB ]

Hayk Martirosyan from Skydio gave a lecture as part of Pieter Abbeel's robotics course at UC Berkeley—this is where you hear about all the secret stuff Skydio is working on next.

[ UC Berkeley ]

Illustration: Chris Philpot

Many young urbanites don’t want to own a car, and unlike earlier generations, they don’t have to rely on mass transit. Instead they treat mobility as a service: When they need to travel significant distances, say, more than 5 miles (8 kilometers), they use their phones to summon an Uber (or a car from a similar ride-sharing company). If they have less than a mile or so to go, they either walk or use various “micromobility” services, such as the increasingly ubiquitous Lime and Bird scooters or, in some cities, bike sharing.

The problem is that today’s mobility-as-a-service ecosystem often doesn’t do a good job covering intermediate distances, say a few miles. Hiring an Uber or Lyft for such short trips proves frustratingly expensive, and riding a scooter or bike more than a mile or so can be taxing to many people. So getting yourself to a destination that is from 1 to 5 miles away can be a challenge. Yet such trips account for about half of the total passenger miles traveled.

Many of these intermediate-distance trips take place in environments with limited traffic, such as university campuses and industrial parks, where it is now both economically reasonable and technologically possible to deploy small, low-speed autonomous vehicles powered by electricity. We’ve been involved with a startup that intends to make this form of transportation popular. The company, PerceptIn, hasautonomous vehicles operating at tourist sites in Nara and Fukuoka, Japan; at an industrial park in Shenzhen, China; and is just now arranging for its vehicles to shuttle people around Fishers, Ind., the location of the company’s headquarters.

Because these diminutive autonomous vehicles never exceed 20 miles (32 kilometers) per hour and don’t mix with high-speed traffic, they don’t engender the same kind of safety concerns that arise with autonomous cars that travel on regular roads and highways. While autonomous driving is a complicated endeavor, the real challenge for PerceptIn was not about making a vehicle that can drive itself in such environments—the technology to do that is now well established—but rather about keeping costs down.

Given how expensive autonomous cars still are in the quantities that they are currently being produced—an experimental model can cost you in the neighborhood of US $300,000—you might not think it possible to sell a self-driving vehicle of any kind for much less. Our experience over the past few years shows that, in fact, it is possible today to produce a self-driving passenger vehicle much more economically: PerceptIn’s vehicles currently sell for about $70,000, and the price will surely drop in the future. Here’s how we and our colleagues at PerceptIn brought the cost of autonomous driving down to earth.

Let’s start by explaining why autonomous cars are normally so expensive. In a nutshell, it’s because the sensors and computers they carry are very pricey.

The suite of sensors required for autonomous driving normally includes a high-end satellite-navigation receiver, lidar (light detection and ranging), one or more video cameras, radar, and sonar. The vehicle also requires at least one very powerful computer.

The satellite-navigation receivers used in this context aren’t the same as the one found in your phone. The kind built into autonomous vehicles have what is called real-time kinematic capabilities for high-precision position fixes—down to 10 centimeters. These devices typically cost about $4,000. Even so, such satellite-navigation receivers can’t be entirely relied on to tell the vehicle where it is. The fixes it gets could be off in situations where the satellite signals bounce off of nearby buildings, introducing noise and delays. In any case, satellite navigation requires an unobstructed view of the sky. In closed environments, such as tunnels, that just doesn’t work.

  Illustration: Chris Philpot

Fortunately, autonomous vehicles have other ways to figure out where they are. In particular they can use lidar, which determines distances to things by bouncing a laser beam off them and measuring how long it takes for the light to reflect back. A typical lidar unit for autonomous vehicles covers a range of 150 meters and samples more than 1 million spatial points per second.

Such lidar scans can be used to identify different shapes in the local environment. The vehicle’s computer then compares the observed shapes with the shapes recorded in a high-definition digital map of the area, allowing it to track the exact position of the vehicle at all times. Lidar can also be used to identify and avoid transient obstacles, such as pedestrians and other cars.

Lidar is a wonderful technology, but it suffers from two problems. First, these units are extremely expensive: A high-end lidar for autonomous driving can easily cost more than $80,000, although costs are dropping, and for low-speed applications a suitable unit can be purchased for about $4,000. Also, lidar, being an optical device, can fail to provide reasonable measurements in bad weather, such as heavy rain or fog.

The same is true for the cameras found on these vehicles, which are mostly used to recognize and track different objects, such as the boundaries of driving lanes, traffic lights, and pedestrians. Usually, multiple cameras are mounted around the vehicle. These cameras typically run at 60 frames per second, and the multiple cameras used can generate more than 1 gigabyte of raw data each second. Processing this vast amount of information, of course, places very large computational demands on the vehicle’s computer. On the plus side, cameras aren’t very expensive.

The radar and sonar systems found in autonomous vehicles are used for obstacle avoidance. The data sets they generate show the distance from the nearest object in the vehicle’s path. The major advantage of these systems is that they work in all weather conditions. Sonar usually covers a range of up to 10 meters, whereas radar typically has a range of up to 200 meters. Like cameras, these sensors are relatively inexpensive, often costing less than $1,000 each.

The many measurements such sensors supply are fed into the vehicle’s computers, which have to integrate all this information to produce an understanding of the environment. Artificial neural networks and deep learning, an approach that has grown rapidly in recent years, play a large role here. With these techniques, the computer can keep track of other vehicles moving nearby, as well as of pedestrians crossing the road, ensuring the autonomous vehicle doesn’t collide with anything or anyone.

Of course, the computers that direct autonomous vehicles have to do a lot more than just avoid hitting something. They have to make a vast number of decisions about where to steer and how fast to go. For that, the vehicle’s computers generate predictions about the upcoming movement of nearby vehicles before deciding on an action plan based on those predictions and on where the occupant needs to go.

Lastly, an autonomous vehicle needs a good map. Traditional digital maps are usually generated from satellite imagery and have meter-level accuracy. Although that’s more than sufficient for human drivers, autonomous vehicles demand higher accuracy for lane-level information. Therefore, special high-definition maps are needed.

Just like traditional digital maps, these HD maps contain many layers of information. The bottom layer is a map with grid cells that are about 5 by 5 cm; it’s generated from raw lidar data collected using special cars. This grid records elevation and reflection information about the objects in the environment.

Photos: Perceptin Slowly But Surely: The authors’ approach to autonomy has been applied to two different types of low-speed electric vehicles. One is a two-seat “pod,” shown here being demonstrated at Purdue University, where it was used to transport students from parking lots to the center of campus [top]. The other is a multipassenger bus, which is being used now at various sites around the world, including the Nara Palace historical park in Japan [bottom].

On top of that base grid, there are several layers of additional information. For instance, lane information is added to the grid map to allow autonomous vehicles to determine whether they are in the correct lane. On top of the lane information, traffic-sign labels are added to notify the autonomous vehicles of the local speed limit, whether they are approaching traffic lights, and so forth. This helps in cases where cameras on the vehicle are unable to read the signs.

Traditional digital maps are updated every 6 to 12 months. To make sure the maps that autonomous vehicles use contain up-to-date information, HD maps should be refreshed weekly. As a result, generating and maintaining HD maps can cost millions of dollars per year for a midsize city.

All that data on those HD maps has to be stored on board the vehicle in solid-state memory for ready access, adding to the cost of the computing hardware, which needs to be quite powerful. To give you a sense, an early computing system that Baidu employed for autonomous driving used an Intel Xeon E5 processor and four to eight Nvidia K80 GPU accelerators. The system was capable of delivering 64.5 trillion floating-point operations per second, but it consumed around 3,000 watts and generated an enormous amount of heat. And it cost about $30,000.

Given that the sensors and computers alone can easily cost more than $100,000, it’s not hard to understand why autonomous vehicles are so expensive, at least today. Sure, the price will come down as the total number manufactured increases. But it’s still unclear how the costs of creating and maintaining HD maps will be passed along. In any case, it will take time for better technology to address all the obvious safety concerns that come with autonomous driving on normal roads and highways.

We and our colleagues at PerceptIn have been trying to address these challenges by focusing on small, slow-speed vehicles that operate in limited areas and don’t have to mix with high-speed traffic—university campuses and industrial parks, for example.

The main tactic we’ve used to reduce costs is to do away with lidar entirely and instead use more affordable sensors: cameras, inertial measurement units, satellite positioning receivers, wheel encoders, radars, and sonars. The data that each of these sensors provides can then be combined though a process called sensor fusion.

With a balance of drawbacks and advantages, these sensors tend to complement one another. When one fails or malfunctions, others can take over to ensure that the system remains reliable. With this sensor-fusion approach, sensor costs could drop eventually to something like $2,000.

Because our vehicle runs at a low speed, it takes at the very most 7 meters to stop, making it much safer than a normal car, which can take tens of meters to stop. And with the low speed, the computing systems have less severe latency requirements than those used in high-speed autonomous vehicles.

PerceptIn’s vehicles use satellite positioning for initial localization. While not as accurate as the systems found on highway-capable autonomous cars, these satellite-navigation receivers still provide submeter accuracy. Using a combination of camera images and data from inertial measurement units (in a technique called visual inertial odometry), the vehicle’s computer further improves the accuracy, fixing position down to the decimeter level.

For imaging, PerceptIn has integrated four cameras into one hardware module. One pair faces the front of the vehicle, and another pair faces the rear. Each pair of cameras provides binocular vision, allowing it to capture the kind of spatial information normally given by lidar. What’s more, the four cameras together can capture a 360-degree view of the environment, with enough overlapping spatial regions between frames to ensure that visual odometry works in any direction.

Even if visual odometry were to fail and satellite-positioning signals were to drop out, all wouldn’t be lost. The vehicle could still work out position updates using rotary encoders attached to its wheels—following a general strategy that sailors used for centuries, called dead reckoning.

Data sets from all these sensors are combined to give the vehicle an overall understanding of its environment. Based on this understanding, the vehicle’s computer can make the decisions it requires to ensure a smooth and safe trip.

The vehicle also has an anti-collision system that operates independently of its main computer, providing a last line of defense. This uses a combination of millimeter-wave radars and sonars to sense when the vehicle is within 5 meters of objects, in which case it’s immediately stopped.

Relying on less expensive sensors is just one strategy that PerceptIn has pursued to reduce costs. Another has been to push computing to the sensors to reduce the demands on the vehicle’s main computer, a normal PC with a total cost less than $1,500 and a peak system power of just 400 W.

PerceptIn’s camera module, for example, can generate 400 megabytes of image information per second. If all this data were transferred to the main computer for processing, that computer would have to be extremely complex, which would have significant consequences in terms of reliability, power, and cost. PerceptIn instead has each sensor module perform as much computing as possible. This reduces the burden on the main computer and simplifies its design.

More specifically, a GPU is embedded into the camera module to extract features from the raw images. Then, only the extracted features are sent to the main computer, reducing the data-transfer rate a thousandfold.

Another way to limit costs involves the creation and maintenance of the HD maps. Rather than using vehicles outfitted with lidar units to provide map data, PerceptIn enhances existing digital maps with visual information to achieve decimeter-level accuracy.

The resultant high-precision visual maps, like the lidar-based HD maps they replace, consist of multiple layers. The bottom layer can be any existing digital map, such as one from the OpenStreetMap project. This bottom layer has a resolution of about 1 meter. The second layer records the visual features of the road surfaces to improve mapping resolution to the decimeter level. The third layer, also saved at decimeter resolution, records the visual features of other parts of the environment—such as signs, buildings, trees, fences, and light poles. The fourth layer is the semantic layer, which contains lane markings, traffic sign labels, and so forth.

While there’s been much progress over the past decade, it will probably be another decade or more before fully autonomous cars start taking to most roads and highways. In the meantime, a practical approach is to use low-speed autonomous vehicles in restricted settings. Several companies, including Navya, EasyMile, and May Mobility, along with PerceptIn, have been pursuing this strategy intently and are making good progress.

Eventually, as the relevant technology advances, the types of vehicles and deployments can expand, ultimately to include vehicles that can equal or surpass the performance of an expert human driver.

PerceptIn has shown that it’s possible to build small, low-speed autonomous vehicles for much less than it costs to make a highway-capable autonomous car. When the vehicles are produced in large quantities, we expect the manufacturing costs to be less than $10,000. Not too far in the future, it might be possible for such clean-energy autonomous shuttles to be carrying passengers in city centers, such as Manhattan’s central business district, where the average speed of traffic now is only 7 miles per hour[PDF]. Such a fleet would significantly reduce the cost to riders, improve traffic conditions, enhance safety, and improve air quality to boot. Tackling autonomous driving on the world’s highways can come later.

This article appears in the March 2020 print issue as “Autonomous Vehicles Lite.”

About the Authors

Shaoshan Liu is the cofounder and CEO of PerceptIn, an autonomous vehicle startup in Fishers, Ind. Jean-Luc Gaudiot is a professor of electrical engineering and computer science at the University of California, Irvine.

The Urban Circuit, the second of four robotics competitions that are part of the DARPA Subterranean Challenge, kicks off today, and you can follow along with what’s going on through DARPA’s livestream.

Round 1 happens from February 20 (Thursday) to February 22 (Saturday), and Round 2 from February 24 (Monday) to February 26 (Wednesday).

DARPA says the livestream is on a one hour time delay for competition integrity, but the good news is that the coverage includes commentary and interviews. “We intend to have both narration of the action and commentary where applicable,” DARPA SubT program manager Tim Chung told us. “We’re opening the aperture because there are a lot of really interesting features that we’d love our viewing audience to be able to witness, and that includes robots traversing interesting features and elements within the course, so you can look forward to more of that.”

Here’s today’s livestream (Urban Circuit Awards Ceremony & Technical Interchange Meeting):

Archived Livestreams

Round 1 – Day 1:

Round 1 – Day 2:

Round 1 – Day 3:

Round 2  – Day 1:

Round 2 – Day 2:

Round 2 – Day 3:

There’s a media day on the 24 that we’ll be at, so make sure and let us know if there are specific things you want us to check out.

Also, you can hear directly from the teams themselves by following #SubTChallenge on Twitter.

Pages