Feed aggregator

Photo: Science and Society Picture Library/Getty Images Neurophysiologist W. Grey Walter built his cybernetic tortoises to elucidate the functions of the brain.

In the robotics family tree, Roomba’s ancestors were probably Elmer and Elsie, a pair of cybernetic tortoises invented in the 1940s by neurophysiologist W. Grey Walter. The robots could “see” by means of a rotating photocell that steered them toward a light source. If the light was too bright, they would retreat and continue their exploration in a new direction. Likewise, when they ran into obstacles, a touch sensor would compel the tortoises to reverse and change course. In this way, Elmer and Elsie slowly explored their surroundings.

Walter was an early researcher into electroencephalography (EEG), a technique for detecting the electrical activity of the brain using electrodes attached to the scalp. Among his notable clinical breakthroughs was the first diagnosis of a brain tumor by EEG. In 1939 he joined the newly established Burden Neurological Institute in Bristol, England, as head of its physiology department, and he remained at the Burden for the rest of his career.

Norbert Wiener’s cybernetics movement gave birth to a menagerie of cybernetic creatures

In the late 1940s, Walter became involved in the emerging community of scientists who were interested in cybernetics. The field’s founder, Norbert Wiener, defined cybernetics as “the scientific study of control and communication in the animal and the machine.” In the first wave of cybernetics, people were keen on building machines to model animal behavior. Claude Shannon played around with a robotic mouse named Theseus that could navigate mazes. W. Ross Ashby built the Homeostat, a machine that automatically adapted to inputs so as to remain in a stable state.

Walter’s contribution to this cybernetic menagerie was an electromechanical tortoise, which he began working on in the spring of 1948 in his spare time. His first attempts were inelegant. In 1951 W. J. “Bunny” Warren, an electrical engineer at the Burden, constructed six tortoises for Walter that were more solidly engineered. Two of these six tortoises became Elmer and Elsie, their names taken from Grey’s somewhat contrived acronym: ELectro MEchanical Robots, Light Sensitive, with Internal and External stability.

Photo: Larry Burrows/The LIFE Picture Collection/Getty Images In this time-lapse photo from 1950, Walter has a smoke while one of his cybernetic tortoises roams about the living room.

Walter considered Elmer and Elsie to be the Adam and Eve of a new species, Machina speculatrix. The scientific nomenclature reflected the robots’ exploratory or speculative behavior. The creatures each had a smooth protective shell and a protruding neck, so Walter put them in the Linnaean genus Testudo, or tortoise. Extending his naming scheme, he dubbed Shannon’s maze-crawling mouse Machina labyrinthia and Ashby’s Homestat Machina sopora (sleeping machine).

Did W. Grey Walter’s cybernetic tortoises exhibit free will?

Each tortoise moved on three wheels with two sets of motors, one for locomotion and the other for steering. Its “brain” consisted of two vacuum tubes, which Walter said gave it the equivalent of two functioning neurons.

Despite such limited equipment, the tortoises displayed free will, he claimed. In the May 1950 issue of Scientific American [PDF], he described how the photocell atop the tortoise’s neck scanned the surroundings for a light source. The photocell was attached to the steering mechanism, and as the tortoise searched, it moved forward in a circular pattern. Walter compared this to the alpha rhythm of the electric pulses in the brain, which sweeps over the visual areas and at the same time releases impulses for the muscles to move.

In a dark room, the tortoise wandered aimlessly. When it detected a light, the tortoise moved directly toward the source. But if the light surpassed a certain brightness, it retreated. Presented with two distinct light sources, it would trace a path back and forth between the pair. “Like a moth to a flame,” Walter wrote, the tortoise oscillated between seeking and withdrawing from the lights.

The tortoise had a running light that came on when it was searching for a light source. Originally, this was just to signal to observers what command the robot was processing, but it had some unintended consequences. If Elmer happened to catch a glimpse of itself in a mirror, it would begin moving closer to the image until the reflected light became too bright, and then it would retreat. In his 1953 book The Living Brain, Walter compared the robot to “a clumsy Narcissus.”

Similarly, if Elmer and Elsie were in the same area and saw the other’s light, they would lock onto the source and approach, only to veer away when they got too close. Ever willing to describe the machines in biological terms, Walter called this a mating dance where the unfortunate lovers could never “consummate their ‘desire.’ ”

The tortoise’s shell did much more than just protect the machine’s electromechanical insides. If the robot bumped into an obstacle, a touch sensor in the shell caused it to reverse and change direction. In this manner it could explore its surroundings despite being effectively blind.

M. speculatrix was powered by a hearing-aid battery and a 6-volt battery. When its wanderings were done—that is, when its battery levels were low—it made its way to its hutch. There, it could connect its circuits, turn off its motors, and recharge.

Elmer and Elsie were a huge hit at the 1951 Festival of Britain

During the summer of 1951, Elmer and Elsie performed daily in the science exhibition at the Festival of Britain. Held at sites throughout the United Kingdom, the festival drew millions of visitors. The tortoises were a huge hit. Attendees wondered at their curious activity as they navigated their pen, moved toward and away from light sources, and avoided obstacles in their path. A third tortoise with a transparent shell was on display to showcase the inner workings and to advertise the component parts.

Even as M. speculatrix was wowing the public, Walter was investigating the next evolution of the species. Elmer and Elsie successfully demonstrated unpredictable behavior that could be compared with a basic animal reaction to stimuli, but they never learned from their experience. They had no memory and could not adapt to their environment.

Walter dubbed his next experimental tortoise M. docilis, from the Latin for teachable, and he attempted to build a robot that could mimic Pavlovian conditioned responses. Where the Russian psychologist used dogs, food, and some sort of sound, Walter used his cybernetic tortoises, light, and a whistle. That is, he taught his M. docilis tortoises that the sound of a whistle was the same as a light source and that the tortoise would move toward the sound even if no light was present.

Walter published his findings on M. docilis in a second Scientific American article, “A Machine That Learns” [PDF]. This follow-up article had much to offer electrical engineers, including circuit diagrams and a technical discussion of some of the challenges in constructing the robots, such as amplifying the sound of the whistle to overcome the noise of the motors.

Photo: Larry Burrows/The LIFE Picture Collection/Getty Images The robo-tortoise returns to its hutch to recharge its battery.

The brain of M. docilis was CORA (short for COnditioned Reflex Analog) circuitry, which detected repeated coincidental sensory inputs on separate channels, such as light and sound that happened at the same time. After CORA logged a certain number of repetitions, often between 10 and 20 instances, it linked the resulting behavior, which Walter described as a conditioned response. Just as CORA could learn a behavior, it could also forget it. If the operator teased the tortoise by withholding the light from the sound of the whistle, CORA would delink the response.

At the end of his article, Walter acknowledged that future experiments with more circuits and inputs were feasible, but the increase in complexity would come at the cost of stability. Eventually, scientists would find it too difficult to model the behavior and understand the reactions to multiple stimuli.

Walter discontinued his experiments with robotic tortoises after CORA, and the research was not picked up by others. As the historian of science Andrew Pickering noted in his 2009 book, The Cybernetic Brain, “CORA remains an unexploited resource in the history of cybernetics.”

Walter’s legacy lives on in his tortoises. The late Rueben Hoggett compiled a treasure trove of archival research on Walter’s tortoises, which can be found on Hoggett’s website, Cybernetic Zoo. The three tortoises from the Festival of Britain were auctioned off, and the winner, Wes Clutterbuck, nicknamed them Slo, Mo, and Shun. Although two were later destroyed in a fire, the Clutterbuck family donated the one with a transparent shell to the Smithsonian Institution. The only other known surviving tortoise from the original six crafted by Bunny Warren is at the Science Museum in London. It is currently on exhibit in the Making the Modern World Gallery.

An abridged version of this article appears in the March 2020 print issue as “The Proto-Roomba.”

Part of a continuing series looking at photographs of historical artifacts that embrace the boundless potential of technology.

About the Author

Allison Marsh is an associate professor of history at the University of South Carolina and codirector of the university’s Ann Johnson Institute for Science, Technology & Society.

This study aimed to investigate whether using a wearable robot applying interactive rhythmic stimulation on the upper limbs of patients with Parkinson's disease (PD) could affect their gait. The wearable robot presented tactile stimuli on the patients' upper limbs, which was mutually synchronized with the swing of their upper limbs. We conducted an evaluation experiment with PD patients (n = 30, Modified Hoehn-Yahr = 1–3, on-state) to investigate the assistance effect by the robot and the immediate after-effect of intervention. The participants were instructed to walk 30 m under four different conditions: (1) not wearing the robot before the intervention (Pre-condition), (2) wearing the robot without the rhythm assistance (RwoA condition), (3) wearing the robot with rhythm assistance (RwA condition), and (4) not wearing the robot immediately after the intervention (Post-condition). These conditions were conducted in this order over a single day. The third condition was performed three times and the others, once. The arm swing amplitude, stride length, and velocity were increased in the RwA condition compared to the RwoA condition. The coefficient of variance (CV) of the stride duration was decreased in the RwA condition compared to the RwoA condition. These results revealed that the assistance by the robot increased the gait performance of PD patients. In addition, the stride length and velocity were increased and the stride duration CV was decreased in the Post-condition compared to the Pre-condition. These results show that the effect of robot assistance on the patient's gait remained immediately after the intervention. These findings suggest that synchronized rhythmic stimulation on the upper limbs could influence the gait of PD patients and that the robot may assist with gait rehabilitation in these patients.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Artificial intelligence and robotic technologies with semi-autonomous learning, reasoning, and decision-making capabilities are increasingly being incorporated into defense, military, and security systems. Unsurprisingly, there is increasing concern about the stability and safety of these systems. In a different sector, runaway interactions between autonomous trading systems in financial markets have produced a series of stock market “flash crashes,” and as a result, those markets now have rules to prevent such interactions from having a significant impact1.

Could the same kinds of unexpected interactions and feedback loops lead to similar instability with defense or security AIs?

Adversarial attacks on AI systems

General concerns about the impacts of defense AIs and robots on stability, whether in isolation or through interaction, have only been exacerbated by recent demonstrations of adversarial attacks against these systems2. Perhaps the most widely-discussed attack cases involve image classification algorithms that are deceived into “seeing” images in noise3, or are easily tricked by pixel-level changes so they classify, say, a turtle as a rifle4. Similarly, game-playing systems that outperform any human (e.g., AlphaGo) can suddenly fail if the game structure or rules are even slightly altered in ways that would not affect a human5. Autonomous vehicles that function reasonably well in ordinary conditions can, with the application of a few pieces of tape, be induced to swerve into the wrong lane or speed through a stop sign6. And the list of adversarial attacks continues to grow and grow over time.

Adversarial attacks pose a tangible threat to the stability and safety of AI and robotic technologies. The exact conditions for such attacks are typically quite unintuitive for humans, so it is difficult to predict when and where the attacks could occur. And even if we could estimate the likelihood of an adversarial attack, the exact response of the AI system can be difficult to predict as well, leading to further surprises and less stable, less safe military engagements and interactions. Even overall assessments of reliability are difficult in the face of adversarial attacks.

We might hope that adversarial attacks would be relatively rare in the everyday world, since “random noise” that targets image classification algorithms is actually far from random: The tape on the stop sign must be carefully placed, the pixel-level perturbations added to the image must be carefully calculated, and so on. Significant effort is required to construct an adversarial attack, and so we might simply deploy our AI and robotic systems with the hope that the everyday world will not conspire to deceive them. 

Unfortunately, this confidence is almost certainly unwarranted for defense or security technologies. These systems will invariably be deployed in contexts where the other side has the time, energy, and ability to develop and construct exactly these types of adversarial attacks. AI and robotic technologies are particularly appealing for deployment in enemy-controlled or enemy-contested areas since those environments are riskiest for our human soldiers, in large part because the other side has the most control over the environment. 

Defenses against adversarial attacks

Although adversarial attacks on defense and military AIs and robots are likely, they are not necessarily destabilizing, particularly since humans are typically unaffected by these attacks. We can easily recognize that a turtle is not a rifle even with random noise, we view tape on a stop sign as an annoyance rather than something that disrupts our ability to follow the rules of the road, and so on. Of course, there are complexities, but we can safely say that human performance is strongly robust to adversarial attacks against AIs. Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

This recommendation is attractive, but is also necessarily limited in scope to applications where a human can be directly involved. In the case of intelligence, surveillance, and reconnaissance (ISR) systems, however, substantive human interaction might not be possible. AI technologies are being increasingly used to handle the enormous volumes of data generated for ISR purposes. AI technologies for ISR now play a significant role in the creation and maintenance of situational awareness for human decision-makers, and in such situations, the destabilizing risks of adversarial attacks again rear their heads.

As an extreme example, consider the intersection of AI and nuclear weapons. One might think that these two technologies should never meet, since we ought not delegate the decision to use nuclear force to an AI. Regardless, AI systems potentially (or perhaps actually) do play a role in nuclear weapons, namely in the ISR that informs human decisions about whether to use such weapons. The worldwide sensor and data input streams almost certainly cannot be processed entirely by human beings. We will need to use (or perhaps already do use) AI technologies without a human in the loop to help us understand our world, and so there may not always be a human to intercept adversarial attacks against those systems.

Our situational awareness can therefore be affected or degraded due to deliberately distorted “perceptions” coming from the AI analyses. These problems are not limited to the extreme case of nuclear weapons—any military or security action where situational awareness depends partly on unmonitored ISR AI will be vulnerable to adversarial attacks in ways that a human cannot necessarily recognize and rectify.

Perhaps we could simply monitor the ISR AI by requiring it to provide evidence or explanations of its analyses that are sufficiently detailed for a human to be able to recognize an adversarial attack. However, if we consider only “explainable AIs” in these contexts, then we are restricting the space of possible models, and so arguably9 placing an artificial upper bound on system performance. Moreover, many AI systems are moving some computation onto the sensors themselves to help overcome processing and memory constraints.

For example, AI on the sensors might perform anomaly detection, leaving a higher level AI system to process only potential outliers. These distributed systems might not be able to retain the evidence (for example, the original image) required for human recognition of an adversarial attack. And in real-world cases, we might not have the time to look at the evidence even if it were provided, and so would not be able to respond to destabilizing adversarial attacks on the ISR AI.

This all might seem to be much ado about nothing new: After all, information gathering has always been susceptible to deception, manipulation, and misinformation. But adversarial attacks can lead to completely bizarre and ridiculous (from a human perspective) behavior from an AI. No ordinary deception would ever lead a human intelligence officer to see a turtle as a rifle, and the use of ISR AI opens the door to much different types of deception, with much different results. Without proper understanding of these potential impacts, the world is likely to be a less stable and less safe place. 

Adversarial attacks can destabilize AI technologies, rendering them less safe, predictable, or reliable. However, we do not necessarily need to worry about them as direct attacks on the decision-making machinery of the system. Instead, we should worry about the corruption of human situational awareness through adversarial AI, which can be equally effective in undermining the safety, stability, and trust in the AI and robotic technologies.

David Danks is L.L. Thurstone professor of philosophy and psychology, and head of the department of philosophy, at Carnegie Mellon University. He is also the chief ethicist of CMU’s Block Center for Technology & Society; co-director of CMU’s Center for Informed Democracy and Social Cybersecurity (IDeaS); and an adjunct member of the Heinz College of Information Systems and Public Policy. His research interests are at the intersection of philosophy, cognitive science, and machine learning. Most recently, Danks has been examining the ethical, psychological, and policy issues around AI and robotics in transportation, healthcare, privacy, and security.

References

1. Serritella, D. M. (2010). “High speed trading begets high speed regulation: SEC response to flash crash, rash.” Illinois Journal of Law, Technology & Policy, 2010 (2).

2. Biggio, B. & Roli, F. (2018). “Wild patterns: Ten years after the rise of adversarial machine learning.” Pattern Recognition, 84, 317-331.

3. Nguyen, A., Yosinski, J., & Clune, J. (2015). “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 427-436).

4. Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2018). “Synthesizing robust adversarial examples.” In Proceedings of the 35th International Conference on Machine Learning (pp. 284-293).

5. Raghu, M., Irpan, A., Andreas, J., Kleinberg, R., Le, Q. V., & Kleinberg, J. (2018). “Can deep reinforcement learning solve Erdos-Selfridge-Spencer games?” Proceedings of ICML.

6. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018). “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1625-1634).

7. Rudin, C. (2018). “Please stop explaining black box models for high stakes decisions.” NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning. arXiv:1811.10154v2

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Artificial intelligence and robotic technologies with semi-autonomous learning, reasoning, and decision-making capabilities are increasingly being incorporated into defense, military, and security systems. Unsurprisingly, there is increasing concern about the stability and safety of these systems. In a different sector, runaway interactions between autonomous trading systems in financial markets have produced a series of stock market “flash crashes,” and as a result, those markets now have rules to prevent such interactions from having a significant impact1.

Could the same kinds of unexpected interactions and feedback loops lead to similar instability with defense or security AIs?

Adversarial attacks on AI systems

General concerns about the impacts of defense AIs and robots on stability, whether in isolation or through interaction, have only been exacerbated by recent demonstrations of adversarial attacks against these systems2. Perhaps the most widely-discussed attack cases involve image classification algorithms that are deceived into “seeing” images in noise3, or are easily tricked by pixel-level changes so they classify, say, a turtle as a rifle4. Similarly, game-playing systems that outperform any human (e.g., AlphaGo) can suddenly fail if the game structure or rules are even slightly altered in ways that would not affect a human5. Autonomous vehicles that function reasonably well in ordinary conditions can, with the application of a few pieces of tape, be induced to swerve into the wrong lane or speed through a stop sign6. And the list of adversarial attacks continues to grow and grow over time.

Adversarial attacks pose a tangible threat to the stability and safety of AI and robotic technologies. The exact conditions for such attacks are typically quite unintuitive for humans, so it is difficult to predict when and where the attacks could occur. And even if we could estimate the likelihood of an adversarial attack, the exact response of the AI system can be difficult to predict as well, leading to further surprises and less stable, less safe military engagements and interactions. Even overall assessments of reliability are difficult in the face of adversarial attacks.

We might hope that adversarial attacks would be relatively rare in the everyday world, since “random noise” that targets image classification algorithms is actually far from random: The tape on the stop sign must be carefully placed, the pixel-level perturbations added to the image must be carefully calculated, and so on. Significant effort is required to construct an adversarial attack, and so we might simply deploy our AI and robotic systems with the hope that the everyday world will not conspire to deceive them. 

Unfortunately, this confidence is almost certainly unwarranted for defense or security technologies. These systems will invariably be deployed in contexts where the other side has the time, energy, and ability to develop and construct exactly these types of adversarial attacks. AI and robotic technologies are particularly appealing for deployment in enemy-controlled or enemy-contested areas since those environments are riskiest for our human soldiers, in large part because the other side has the most control over the environment. 

Defenses against adversarial attacks

Although adversarial attacks on defense and military AIs and robots are likely, they are not necessarily destabilizing, particularly since humans are typically unaffected by these attacks. We can easily recognize that a turtle is not a rifle even with random noise, we view tape on a stop sign as an annoyance rather than something that disrupts our ability to follow the rules of the road, and so on. Of course, there are complexities, but we can safely say that human performance is strongly robust to adversarial attacks against AIs. Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

Adversarial attacks will thus not be destabilizing if we follow a straightforward policy recommendation: Keep humans in (or on) the loop for these technologies. If there is human-AI teaming, then people can (hopefully!) recognize that an adversarial attack has occurred, and guide the system to appropriate behaviors.

This recommendation is attractive, but is also necessarily limited in scope to applications where a human can be directly involved. In the case of intelligence, surveillance, and reconnaissance (ISR) systems, however, substantive human interaction might not be possible. AI technologies are being increasingly used to handle the enormous volumes of data generated for ISR purposes. AI technologies for ISR now play a significant role in the creation and maintenance of situational awareness for human decision-makers, and in such situations, the destabilizing risks of adversarial attacks again rear their heads.

As an extreme example, consider the intersection of AI and nuclear weapons. One might think that these two technologies should never meet, since we ought not delegate the decision to use nuclear force to an AI. Regardless, AI systems potentially (or perhaps actually) do play a role in nuclear weapons, namely in the ISR that informs human decisions about whether to use such weapons. The worldwide sensor and data input streams almost certainly cannot be processed entirely by human beings. We will need to use (or perhaps already do use) AI technologies without a human in the loop to help us understand our world, and so there may not always be a human to intercept adversarial attacks against those systems.

Our situational awareness can therefore be affected or degraded due to deliberately distorted “perceptions” coming from the AI analyses. These problems are not limited to the extreme case of nuclear weapons—any military or security action where situational awareness depends partly on unmonitored ISR AI will be vulnerable to adversarial attacks in ways that a human cannot necessarily recognize and rectify.

Perhaps we could simply monitor the ISR AI by requiring it to provide evidence or explanations of its analyses that are sufficiently detailed for a human to be able to recognize an adversarial attack. However, if we consider only “explainable AIs” in these contexts, then we are restricting the space of possible models, and so arguably9 placing an artificial upper bound on system performance. Moreover, many AI systems are moving some computation onto the sensors themselves to help overcome processing and memory constraints.

For example, AI on the sensors might perform anomaly detection, leaving a higher level AI system to process only potential outliers. These distributed systems might not be able to retain the evidence (for example, the original image) required for human recognition of an adversarial attack. And in real-world cases, we might not have the time to look at the evidence even if it were provided, and so would not be able to respond to destabilizing adversarial attacks on the ISR AI.

This all might seem to be much ado about nothing new: After all, information gathering has always been susceptible to deception, manipulation, and misinformation. But adversarial attacks can lead to completely bizarre and ridiculous (from a human perspective) behavior from an AI. No ordinary deception would ever lead a human intelligence officer to see a turtle as a rifle, and the use of ISR AI opens the door to much different types of deception, with much different results. Without proper understanding of these potential impacts, the world is likely to be a less stable and less safe place. 

Adversarial attacks can destabilize AI technologies, rendering them less safe, predictable, or reliable. However, we do not necessarily need to worry about them as direct attacks on the decision-making machinery of the system. Instead, we should worry about the corruption of human situational awareness through adversarial AI, which can be equally effective in undermining the safety, stability, and trust in the AI and robotic technologies.

David Danks is L.L. Thurstone professor of philosophy and psychology, and head of the department of philosophy, at Carnegie Mellon University. He is also the chief ethicist of CMU’s Block Center for Technology & Society; co-director of CMU’s Center for Informed Democracy and Social Cybersecurity (IDeaS); and an adjunct member of the Heinz College of Information Systems and Public Policy. His research interests are at the intersection of philosophy, cognitive science, and machine learning. Most recently, Danks has been examining the ethical, psychological, and policy issues around AI and robotics in transportation, healthcare, privacy, and security.

References

1. Serritella, D. M. (2010). “High speed trading begets high speed regulation: SEC response to flash crash, rash.” Illinois Journal of Law, Technology & Policy, 2010 (2).

2. Biggio, B. & Roli, F. (2018). “Wild patterns: Ten years after the rise of adversarial machine learning.” Pattern Recognition, 84, 317-331.

3. Nguyen, A., Yosinski, J., & Clune, J. (2015). “Deep neural networks are easily fooled: High confidence predictions for unrecognizable images.” In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 427-436).

4. Athalye, A., Engstrom, L., Ilyas, A., & Kwok, K. (2018). “Synthesizing robust adversarial examples.” In Proceedings of the 35th International Conference on Machine Learning (pp. 284-293).

5. Raghu, M., Irpan, A., Andreas, J., Kleinberg, R., Le, Q. V., & Kleinberg, J. (2018). “Can deep reinforcement learning solve Erdos-Selfridge-Spencer games?” Proceedings of ICML.

6. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., & Song, D. (2018). “Robust physical-world attacks on deep learning visual classification.” Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (pp. 1625-1634).

7. Rudin, C. (2018). “Please stop explaining black box models for high stakes decisions.” NeurIPS 2018 Workshop on Critiquing and Correcting Trends in Machine Learning. arXiv:1811.10154v2

This article was originally published on LinkedIn. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Build a rover, send it to the Moon, sell the movie rights.

That was our first business model at iRobot. Way back in 1990. We thought it would be how we’d first change the world. It’s ironic, of course, that through that model, changing the world meant sending a robot to another one. Sadly, that business model failed. And it wouldn’t be our last failed business model. Not by a long shot.

Photo: iRobot

Why? Because changing the world through robots, it turns out, is no easy task.

Perhaps the biggest challenge back when we started in 1990 was that there existed no rule book on how to do it. There weren’t many robots, let alone robot companies, let alone any kind of robot industry. We would have to build it. All of it.

Walking that path meant being comfortable with ambiguity, and comfortable with the knowledge that not everything we tried was going to work–at least not in the way we originally conceived. It was and continues to be the cost of inventing the future.

But walking that trying path also meant learning from our mistakes, dusting ourselves off, trying again, and eventually, yes, doing what we set out to do: Change the world through robots.

We’ve learned so much along the way–what have we learned in our 30-year journey building the robot industry?

Robots are hard

I’ve said it before, and I’ll say it again: When we first started iRobot we had to invent every element of the robot. Spatial navigation was a robot problem, voice recognition was a robot problem, machine vision was a robot problem, just to name a few. Back then, no one else had set out to solve these hard problems. Because so many of these problems existed, the robot industry, if it could be called that, moved in anti-dog years. Fortunately, times have changed and the ecosystem around the technologies that make robots possible is much richer… But back then… it was just us.

But even today, with a much larger ecosystem of bright minds solving for the hard tech problems, getting a robot to work successfully still means getting just the right mix of mechanical, electrical, and software engineering, connectivity, and data science into a robot form factor that people trust and want to invite into their home.

Photo: iRobot

Speaking of trust, therein lied another challenge. Even when we did invent a robot that worked extraordinarily well–Roomba–consumers simply didn’t believe a robot could do what we said Roomba was capable of. It turns out that the principal objection to purchasing a robot for much of the last 30 years is a lack of belief that it could possibly work.

For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. We were no exception: We tried 14 business models before we arrived at one that sustainably worked.

But that’s not all: Even when you build a robot right, you can still somehow build it wrong. We experienced this with Roomba. We built it to match the reliability standards of European upright vacuums, something of which we were very proud. Of course, we didn’t anticipate that our customers would run their Roomba once per day, rather than the once per week average the European standard set. And as the first generation of Roomba robots broke down two years ahead of schedule, we learned that reverse logistics, great customer service, and a generous return policy were a very important part of a good robot–as was the realization that we couldn’t compare usage to whatever traditional means of action a good robot might take the place of.

And yet while building a robot that was durable, that people wanted and trusted was hard enough, 30 years building robots has also taught us that…

Good business models are harder to build than good robots

Let’s state this one right off the bat: For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. It turns out that a business model is as important as the tech, but much more rarely found in a robot company. And for a long time we were no exception: We tried 14 business models before we arrived at one that sustainably worked.

Image: iRobot

But the tenuous nature of our business models did teach us the value of extending the runway for our business until we found one that worked. And how does one extend the runway most effectively? By managing risk.

It’s one of the great misunderstandings of entrepreneurship–that great entrepreneurs are risk takers. Great entrepreneurs are not great risk takers… they’re great risk managers. And this was something we at iRobot were and are exceptionally good at.

How did we manage risk early on? Through partnerships. The kind of partnership we looked for were ones in which there was a big company–one that had a lot of money, a channel to the marketplace, and knowledge of that marketplace, but for whatever reason lacked belief that they themselves were innovative. We were a small company with no money, but believed ourselves to have cool technology, and be highly capable of innovation.

Image: iRobot

What we’d do was give our partner, the big company, absolute control. By doing this, it allowed us to say that since they could cancel the partnership at any time, we needed them to cover our costs… which they did. But we also didn’t ask them to pay us profit upfront. By not having the pay profit upfront, it makes obvious that we’re sharing the value that the partnership would ultimately create, and in a worst-case scenario for our partner, if the partnership didn’t result in a successful product, they got very inexpensive high-quality research.

This “asymmetric strategic partnership” approach not only provided the funds needed to sustain our business when we didn’t have a sustainable business model–the “failure” of those partnerships actually led to our ultimate success. Why? Because…

Innovation and failure come hand-in-hand

While this is far from a groundbreaking realization, its applicability to iRobot is quite unique. Because for us to become successful, it turns out that we had to learn the lessons from failing to earn royalties on robot toys (business model #3), failing to license technology for industrial floor-cleaning robots (business model #8), and failing to sell land mine clearance robots (business model #11).

Image: iRobot

Why? Because #3 taught us to manufacture at scale, #8 taught us how to clean floors, and #11 taught us how to navigate and cover large spaces.  All of which gave us the knowledge and capability to build… Roomba.

Image: iRobot Yes, you can change the world through robots

We did. In more ways the one. We changed the world by eliminating the need for people to vacuum the house themselves. By IPOing, we showed that a robotics company could be successful–which gave investors more reason to put money into robotics companies around the world.

But perhaps the most important way we’ve succeeded in changing the world is by making robots a daily reality for it. And how do we know that robots are now a reality? Because for the better part of the first 30 years of iRobot, what people said to me about robots–and Roomba specifically–was, “I can’t believe it actually works.”

But now, the question they ask me is, “Why can’t robots do more?

It is a great question. And that is what the next 30 years of iRobot will be about.

Colin Angle is chairman of the board, chief executive officer, and founder of iRobot. Celebrating its 30th year, iRobot has grown from an MIT startup to become a global leader in consumer robots, with more than 30 million sold worldwide. You can follow him on Twitter at @ColinAngle.

This article was originally published on LinkedIn. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Build a rover, send it to the Moon, sell the movie rights.

That was our first business model at iRobot. Way back in 1990. We thought it would be how we’d first change the world. It’s ironic, of course, that through that model, changing the world meant sending a robot to another one. Sadly, that business model failed. And it wouldn’t be our last failed business model. Not by a long shot.

Photo: iRobot

Why? Because changing the world through robots, it turns out, is no easy task.

Perhaps the biggest challenge back when we started in 1990 was that there existed no rule book on how to do it. There weren’t many robots, let alone robot companies, let alone any kind of robot industry. We would have to build it. All of it.

Walking that path meant being comfortable with ambiguity, and comfortable with the knowledge that not everything we tried was going to work–at least not in the way we originally conceived. It was and continues to be the cost of inventing the future.

But walking that trying path also meant learning from our mistakes, dusting ourselves off, trying again, and eventually, yes, doing what we set out to do: Change the world through robots.

We’ve learned so much along the way–what have we learned in our 30-year journey building the robot industry?

Robots are hard

I’ve said it before, and I’ll say it again: When we first started iRobot we had to invent every element of the robot. Spatial navigation was a robot problem, voice recognition was a robot problem, machine vision was a robot problem, just to name a few. Back then, no one else had set out to solve these hard problems. Because so many of these problems existed, the robot industry, if it could be called that, moved in anti-dog years. Fortunately, times have changed and the ecosystem around the technologies that make robots possible is much richer… But back then… it was just us.

But even today, with a much larger ecosystem of bright minds solving for the hard tech problems, getting a robot to work successfully still means getting just the right mix of mechanical, electrical, and software engineering, connectivity, and data science into a robot form factor that people trust and want to invite into their home.

Photo: iRobot

Speaking of trust, therein lied another challenge. Even when we did invent a robot that worked extraordinarily well–Roomba–consumers simply didn’t believe a robot could do what we said Roomba was capable of. It turns out that the principal objection to purchasing a robot for much of the last 30 years is a lack of belief that it could possibly work.

For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. We were no exception: We tried 14 business models before we arrived at one that sustainably worked.

But that’s not all: Even when you build a robot right, you can still somehow build it wrong. We experienced this with Roomba. We built it to match the reliability standards of European upright vacuums, something of which we were very proud. Of course, we didn’t anticipate that our customers would run their Roomba once per day, rather than the once per week average the European standard set. And as the first generation of Roomba robots broke down two years ahead of schedule, we learned that reverse logistics, great customer service, and a generous return policy were a very important part of a good robot–as was the realization that we couldn’t compare usage to whatever traditional means of action a good robot might take the place of.

And yet while building a robot that was durable, that people wanted and trusted was hard enough, 30 years building robots has also taught us that…

Good business models are harder to build than good robots

Let’s state this one right off the bat: For a long time the robot industry was unfundable. Why? Because no robot company had a business model worth funding. It turns out that a business model is as important as the tech, but much more rarely found in a robot company. And for a long time we were no exception: We tried 14 business models before we arrived at one that sustainably worked.

Image: iRobot

But the tenuous nature of our business models did teach us the value of extending the runway for our business until we found one that worked. And how does one extend the runway most effectively? By managing risk.

It’s one of the great misunderstandings of entrepreneurship–that great entrepreneurs are risk takers. Great entrepreneurs are not great risk takers… they’re great risk managers. And this was something we at iRobot were and are exceptionally good at.

How did we manage risk early on? Through partnerships. The kind of partnership we looked for were ones in which there was a big company–one that had a lot of money, a channel to the marketplace, and knowledge of that marketplace, but for whatever reason lacked belief that they themselves were innovative. We were a small company with no money, but believed ourselves to have cool technology, and be highly capable of innovation.

Image: iRobot

What we’d do was give our partner, the big company, absolute control. By doing this, it allowed us to say that since they could cancel the partnership at any time, we needed them to cover our costs… which they did. But we also didn’t ask them to pay us profit upfront. By not having the pay profit upfront, it makes obvious that we’re sharing the value that the partnership would ultimately create, and in a worst-case scenario for our partner, if the partnership didn’t result in a successful product, they got very inexpensive high-quality research.

This “asymmetric strategic partnership” approach not only provided the funds needed to sustain our business when we didn’t have a sustainable business model–the “failure” of those partnerships actually led to our ultimate success. Why? Because…

Innovation and failure come hand-in-hand

While this is far from a groundbreaking realization, its applicability to iRobot is quite unique. Because for us to become successful, it turns out that we had to learn the lessons from failing to earn royalties on robot toys (business model #3), failing to license technology for industrial floor-cleaning robots (business model #8), and failing to sell land mine clearance robots (business model #11).

Image: iRobot

Why? Because #3 taught us to manufacture at scale, #8 taught us how to clean floors, and #11 taught us how to navigate and cover large spaces.  All of which gave us the knowledge and capability to build… Roomba.

Image: iRobot Yes, you can change the world through robots

We did. In more ways the one. We changed the world by eliminating the need for people to vacuum the house themselves. By IPOing, we showed that a robotics company could be successful–which gave investors more reason to put money into robotics companies around the world.

But perhaps the most important way we’ve succeeded in changing the world is by making robots a daily reality for it. And how do we know that robots are now a reality? Because for the better part of the first 30 years of iRobot, what people said to me about robots–and Roomba specifically–was, “I can’t believe it actually works.”

But now, the question they ask me is, “Why can’t robots do more?

It is a great question. And that is what the next 30 years of iRobot will be about.

Colin Angle is chairman of the board, chief executive officer, and founder of iRobot. Celebrating its 30th year, iRobot has grown from an MIT startup to become a global leader in consumer robots, with more than 30 million sold worldwide. You can follow him on Twitter at @ColinAngle.

This work presents a review and discussion of the challenges that must be solved in order to successfully develop swarms of Micro Air Vehicles (MAVs) for real world operations. From the discussion, we extract constraints and links that relate the local level MAV capabilities to the global operations of the swarm. These should be taken into account when designing swarm behaviors in order to maximize the utility of the group. At the lowest level, each MAV should operate safely. Robustness is often hailed as a pillar of swarm robotics, and a minimum level of local reliability is needed for it to propagate to the global level. An MAV must be capable of autonomous navigation within an environment with sufficient trustworthiness before the system can be scaled up. Once the operations of the single MAV are sufficiently secured for a task, the subsequent challenge is to allow the MAVs to sense one another within a neighborhood of interest. Relative localization of neighbors is a fundamental part of self-organizing robotic systems, enabling behaviors ranging from basic relative collision avoidance to higher level coordination. This ability, at times taken for granted, also must be sufficiently reliable. Moreover, herein lies a constraint: the design choice of the relative localization sensor has a direct link to the behaviors that the swarm can (and should) perform. Vision-based systems, for instance, force MAVs to fly within the field of view of their camera. Range or communication-based solutions, alternatively, provide omni-directional relative localization, yet can be victim to unobservable conditions under certain flight behaviors, such as parallel flight, and require constant relative excitation. At the swarm level, the final outcome is thus intrinsically influenced by the on-board abilities and sensors of the individual. The real-world behavior and operations of an MAV swarm intrinsically follow in a bottom-up fashion as a result of the local level limitations in cognition, relative knowledge, communication, power, and safety. Taking these local limitations into account when designing a global swarm behavior is key in order to take full advantage of the system, enabling local limitations to become true strengths of the swarm.

Innovating on the design and function of the chemical bench remains a quintessential challenge of the ages. It requires a deep understanding of the important role chemistry plays in scientific discovery as well a first principles approach to addressing the gaps in how work gets done at the bench. This perspective examines how one might explore designing and creating a sustainable new standard for advancing automated chemistry bench itself. We propose how this might be done by leveraging recent advances in laboratory automation whereby integrating the latest synthetic, analytical and information technologies, and AI/ML algorithms within a standardized framework, maximizes the value of the data generated and the broader utility of such systems. Although the context of this perspective focuses on the design of advancing molecule of potential therapeutic value, it would not be a stretch to contemplate how such systems could be applied to other applied disciplines like advanced materials, foodstuffs, or agricultural product development.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.

I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.

AI deception: How to define it?

If we take a rather long view of history, deception may be as old as the world itself, and it is certainly not the sole provenance of human beings. Adaptation and evolution for survival with traits like camouflage are deceptive acts, as are forms of mimicry commonly seen in animals. But pinning down exactly what constitutes deception for an AI agent is not an easy task—it requires quite a bit of thinking about acts, outcomes, agents, targets, means and methods, and motives. What we include or exclude in that calculation may then have wide ranging implications about what needs immediate regulation, policy guidance, or technological solutions. I will only focus on a couple of items here, namely intent and act type, to highlight this point.

What is deception? Bond and Robinson argue that deception is “false communication to the benefit of the communicator.”1 Whaley argues that deception is also the communication of information provided with the intent to manipulate another.2 These seem pretty straightforward approaches, except when you try to press on the idea of what constitutes “intent” and what is required to meet that threshold, as well as whether or not the false communication requires the intent to be explicitly beneficial to the deceiver. Moreover, depending on which stance you take, deception for altruistic reasons may be excluded entirely. Imagine if you asked your AI-enabled robot butler, “How do I look?” To which it answers, “Very nice.”

Let’s start with intent. Intent requires a theory of mind, meaning that the agent has some understanding of itself, and that it can reason about other external entities and their intentions, desires, states, and potential behaviors.3 If deception requires intent in the ways described above, then true AI deception would require an AI to possess a theory of mind. We might kick the can on that conclusion for a bit and claim that current forms of AI deception instead rely on human intent—where some human is using AI as a tool or means to carry out that person’s intent to deceive.

Or, we may not: Just because current AI agents lack a theory of mind doesn’t mean that they cannot learn to deceive. In multi-agent AI systems, some agents can learn deceptive behaviors without having a true appreciation or comprehension of what “deception” actually is. This could be as simple as hiding resources or information, or providing false information to achieve some goal. If we then put aside the theory of mind for the moment and instead posit that intention is not a prerequisite for deception and that an agent can unintentionally deceive, then we really have opened the aperture for existing AI agents to deceive in many ways. 

What about the way in which deception occurs? That is, what are the deceptive act types? We can identify two broad categories here: 1) acts of commission, where an agent actively engages in a behavior like sending misinformation; and 2) acts of omission, where an agent is passive but may be withholding information or hiding. AI agents can learn all sorts of these types of behaviors given the right conditions.4 Just consider how AI agents used for cyber defense may learn to signal various forms of misinformation, or how swarms of AI-enabled robotic systems could learn deceptive behaviors on a battlefield to escape adversary detection. In more pedestrian examples, perhaps a rather poorly specified or corrupted AI tax assistant omits various types of income on a tax return to minimize the likelihood of owing money to the relevant authorities.

Preparing ourselves against AI deception

The first step towards preparing for our coming AI future is to recognize that such systems already do deceive, and are likely to continue to deceive. How that deception occurs, whether it is a desirable trait (such as with our adaptive swarms), and whether we can actually detect when it is occurring are going to be ongoing challenges. Once we acknowledge this simple but true fact, we can begin to undergo the requisite analysis of what exactly constitutes deception, whether and to whom it is beneficial, and how it may pose risks. 

This is no small task, and it will require not only interdisciplinary work from AI experts, but also input from sociologists, psychologists, political scientists, lawyers, ethicists, and policy wonks. For military AI systems, it will also require domain and mission knowledge. In short, developing a comprehensive framework for AI deception is a crucial step if we are not to find ourselves on the back foot. 

We need to begin thinking about how to engineer novel solutions to mitigate unwanted deception by AI agents. This goes beyond current detection research, and requires thinking about environments, optimization problems, and how AI agents model other AI agents and their emergent effects could yield undesirable deceptive behaviors.

Furthermore, once this framework is in place, we need to begin thinking about how to engineer novel solutions to identify and mitigate unwanted deception by AI agents. This goes beyond current detection research, and moving forward requires thinking about environments, optimization problems, and how AI agents model other AI agents and their interactive or emergent effects could yield risky or undesirable deceptive behaviors. 

We presently face a myriad of challenges related to AI deception, and these challenges are only going to increase as the cognitive capacities of AI increase. The desire of some to create AI systems with a rudimentary theory of mind and social intelligence is a case in point to be socially intelligent one must be able to understand and to “manage” the actions of others5, and if this ability to understand another’s feelings, beliefs, emotions, and intentions exists, along with the ability to act to influence those feelings, beliefs, or actions, then deception is much more likely to occur.

However, we do not need to wait for artificial agents to possess a theory of mind or social intelligence for deception with and from AI systems. We should instead begin thinking about potential technological, policy, legal, and ethical solutions to these coming problems before AI gets more advanced than it already is. With a clearer understanding of the landscape, we can analyze potential responses to AI deception, and begin designing AI systems for truth.

Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.

References

1. Bond CF, Robinson M (1988), “The evolution of deception.” J Nonverbal Behav 12(4):295–307. Note also that this definition precludes certain forms of deception from altruistic or paternalistic reasons.

2. B. Whaley, “Toward a general theory of deception,” Journal of Strategic Studies, vol. 5, no. 1, pp. 178–192, Mar. 1982. 

3. Cheney DL, Seyfarth RM, “Baboon metaphysics: the evolution of a social mind.” University of Chicago Press, Chicago, 2008.

4. J. F. Dunnigan and A. A. Nofi, “Victory and deceit, 2nd edition: Deception and trickery in war,” Writers Press Books, 2001. J. Shim and R.C. Arkin, “A Taxonomy of Robot Deception and Its Benefits in HRI” IEEE International Conference on Systems, Man, and Cybernetics, 2013. S. Erat and U. Gneezy, “White lies,” Rady Working paper, Rady School of Management, UC San Diego, 2009. N. C. Rowe, “Designing good deceptions in defense of information systems,” in Proceedings of the 20th Annual Computer Security Applications Conference, ser. ACSAC ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 418–427.

5. E.L. Thorndike. “Intelligence and Its Use.” Harpers Magazine, Vol. 140, 1920: p. 228. Thorndike’s early definition of social intelligence has been widely used and updated for the past 100 years. Even current attempts in cognitive science have looked at separating out the tasks of “understanding” and “acting,” which maps directly to Thorndike’s language of “understand” and “manage”. Cf: M.I. Brown, A. Ratajska, S.L. Hughes, J.B. Fishman, E. Huerta, and C.F. Chabris. “The Social Shape Tests: A New Measure of Social Intelligence, Mentalizing and Theory of Mind.” Personality and Individual Differences, vol. 143, 2019: 107-117.

This piece was written as part of the Artificial Intelligence and International Stability Project at the Center for a New American Security, an independent, nonprofit organization based in Washington, D.C. Funded by Carnegie Corporation of New York, the project promotes thinking and analysis on AI and international stability. Given the likely importance that advances in artificial intelligence could play in shaping our future, it is critical to begin a discussion about ways to take advantage of the benefits of AI and autonomous systems, while mitigating the risks. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In artificial intelligence circles, we hear a lot about adversarial attacks, especially ones that attempt to “deceive” an AI into believing, or to be more accurate, classifying, something incorrectly. Self-driving cars being fooled into “thinking” stop signs are speed limit signs, pandas being identified as gibbons, or even having your favorite voice assistant be fooled by inaudible acoustic commands—these are examples that populate the narrative around AI deception. One can also point to using AI to manipulate the perceptions and beliefs of a person through “deepfakes” in video, audio, and images. Major AI conferences are more frequently addressing the subject of AI deception too. And yet, much of the literature and work around this topic is about how to fool AI and how we can defend against it through detection mechanisms.

I’d like to draw our attention to a different and more unique problem: Understanding the breadth of what “AI deception” looks like, and what happens when it is not a human’s intent behind a deceptive AI, but instead the AI agent’s own learned behavior. These may seem somewhat far-off concerns, as AI is still relatively narrow in scope and can be rather stupid in some ways. To have some analogue of an “intent” to deceive would be a large step for today’s systems. However, if we are to get ahead of the curve regarding AI deception, we need to have a robust understanding of all the ways AI could deceive. We require some conceptual framework or spectrum of the kinds of deception an AI agent may learn on its own before we can start proposing technological defenses.

AI deception: How to define it?

If we take a rather long view of history, deception may be as old as the world itself, and it is certainly not the sole provenance of human beings. Adaptation and evolution for survival with traits like camouflage are deceptive acts, as are forms of mimicry commonly seen in animals. But pinning down exactly what constitutes deception for an AI agent is not an easy task—it requires quite a bit of thinking about acts, outcomes, agents, targets, means and methods, and motives. What we include or exclude in that calculation may then have wide ranging implications about what needs immediate regulation, policy guidance, or technological solutions. I will only focus on a couple of items here, namely intent and act type, to highlight this point.

What is deception? Bond and Robinson argue that deception is “false communication to the benefit of the communicator.”1 Whaley argues that deception is also the communication of information provided with the intent to manipulate another.2 These seem pretty straightforward approaches, except when you try to press on the idea of what constitutes “intent” and what is required to meet that threshold, as well as whether or not the false communication requires the intent to be explicitly beneficial to the deceiver. Moreover, depending on which stance you take, deception for altruistic reasons may be excluded entirely. Imagine if you asked your AI-enabled robot butler, “How do I look?” To which it answers, “Very nice.”

Let’s start with intent. Intent requires a theory of mind, meaning that the agent has some understanding of itself, and that it can reason about other external entities and their intentions, desires, states, and potential behaviors.3 If deception requires intent in the ways described above, then true AI deception would require an AI to possess a theory of mind. We might kick the can on that conclusion for a bit and claim that current forms of AI deception instead rely on human intent—where some human is using AI as a tool or means to carry out that person’s intent to deceive.

Or, we may not: Just because current AI agents lack a theory of mind doesn’t mean that they cannot learn to deceive. In multi-agent AI systems, some agents can learn deceptive behaviors without having a true appreciation or comprehension of what “deception” actually is. This could be as simple as hiding resources or information, or providing false information to achieve some goal. If we then put aside the theory of mind for the moment and instead posit that intention is not a prerequisite for deception and that an agent can unintentionally deceive, then we really have opened the aperture for existing AI agents to deceive in many ways. 

What about the way in which deception occurs? That is, what are the deceptive act types? We can identify two broad categories here: 1) acts of commission, where an agent actively engages in a behavior like sending misinformation; and 2) acts of omission, where an agent is passive but may be withholding information or hiding. AI agents can learn all sorts of these types of behaviors given the right conditions.4 Just consider how AI agents used for cyber defense may learn to signal various forms of misinformation, or how swarms of AI-enabled robotic systems could learn deceptive behaviors on a battlefield to escape adversary detection. In more pedestrian examples, perhaps a rather poorly specified or corrupted AI tax assistant omits various types of income on a tax return to minimize the likelihood of owing money to the relevant authorities.

Preparing ourselves against AI deception

The first step towards preparing for our coming AI future is to recognize that such systems already do deceive, and are likely to continue to deceive. How that deception occurs, whether it is a desirable trait (such as with our adaptive swarms), and whether we can actually detect when it is occurring are going to be ongoing challenges. Once we acknowledge this simple but true fact, we can begin to undergo the requisite analysis of what exactly constitutes deception, whether and to whom it is beneficial, and how it may pose risks. 

This is no small task, and it will require not only interdisciplinary work from AI experts, but also input from sociologists, psychologists, political scientists, lawyers, ethicists, and policy wonks. For military AI systems, it will also require domain and mission knowledge. In short, developing a comprehensive framework for AI deception is a crucial step if we are not to find ourselves on the back foot. 

We need to begin thinking about how to engineer novel solutions to mitigate unwanted deception by AI agents. This goes beyond current detection research, and requires thinking about environments, optimization problems, and how AI agents model other AI agents and their emergent effects could yield undesirable deceptive behaviors.

Furthermore, once this framework is in place, we need to begin thinking about how to engineer novel solutions to identify and mitigate unwanted deception by AI agents. This goes beyond current detection research, and moving forward requires thinking about environments, optimization problems, and how AI agents model other AI agents and their interactive or emergent effects could yield risky or undesirable deceptive behaviors. 

We presently face a myriad of challenges related to AI deception, and these challenges are only going to increase as the cognitive capacities of AI increase. The desire of some to create AI systems with a rudimentary theory of mind and social intelligence is a case in point to be socially intelligent one must be able to understand and to “manage” the actions of others5, and if this ability to understand another’s feelings, beliefs, emotions, and intentions exists, along with the ability to act to influence those feelings, beliefs, or actions, then deception is much more likely to occur.

However, we do not need to wait for artificial agents to possess a theory of mind or social intelligence for deception with and from AI systems. We should instead begin thinking about potential technological, policy, legal, and ethical solutions to these coming problems before AI gets more advanced than it already is. With a clearer understanding of the landscape, we can analyze potential responses to AI deception, and begin designing AI systems for truth.

Dr. Heather M. Roff is a senior research analyst at the Johns Hopkins Applied Physics Laboratory (APL) in the National Security Analysis Department. She is also a nonresident fellow in foreign policy at Brookings Institution, and an associate fellow at the Leverhulme Centre for the Future of Intelligence at the University of Cambridge. She has held numerous faculty posts, as well as fellowships at New America. Before joining APL, she was a senior research scientist in the ethics and society team at DeepMind and a senior research fellow in the department of international relations at the University of Oxford.

References

1. Bond CF, Robinson M (1988), “The evolution of deception.” J Nonverbal Behav 12(4):295–307. Note also that this definition precludes certain forms of deception from altruistic or paternalistic reasons.

2. B. Whaley, “Toward a general theory of deception,” Journal of Strategic Studies, vol. 5, no. 1, pp. 178–192, Mar. 1982. 

3. Cheney DL, Seyfarth RM, “Baboon metaphysics: the evolution of a social mind.” University of Chicago Press, Chicago, 2008.

4. J. F. Dunnigan and A. A. Nofi, “Victory and deceit, 2nd edition: Deception and trickery in war,” Writers Press Books, 2001. J. Shim and R.C. Arkin, “A Taxonomy of Robot Deception and Its Benefits in HRI” IEEE International Conference on Systems, Man, and Cybernetics, 2013. S. Erat and U. Gneezy, “White lies,” Rady Working paper, Rady School of Management, UC San Diego, 2009. N. C. Rowe, “Designing good deceptions in defense of information systems,” in Proceedings of the 20th Annual Computer Security Applications Conference, ser. ACSAC ’04. Washington, DC, USA: IEEE Computer Society, 2004, pp. 418–427.

5. E.L. Thorndike. “Intelligence and Its Use.” Harpers Magazine, Vol. 140, 1920: p. 228. Thorndike’s early definition of social intelligence has been widely used and updated for the past 100 years. Even current attempts in cognitive science have looked at separating out the tasks of “understanding” and “acting,” which maps directly to Thorndike’s language of “understand” and “manage”. Cf: M.I. Brown, A. Ratajska, S.L. Hughes, J.B. Fishman, E. Huerta, and C.F. Chabris. “The Social Shape Tests: A New Measure of Social Intelligence, Mentalizing and Theory of Mind.” Personality and Individual Differences, vol. 143, 2019: 107-117.

  Secure your ticket today!

Being the only event worldwide that brings together all visionary key technologies in one place, automatica is the leading exhibition for smart automation and robotics. Benefit from know-how transfer with leading experts and expand your network with people who can show you the to Industry 4.0.

automatica will open its doors June 16-19, 2020 to 890 international exhibitors and over 45,584 visitors from all over the world in Munich, Germany.

Save time and money now and conveniently purchase your ticket in advance online!

Tickets

Patrick Schwarzkopf, Managing Director of VDMA Robotics + Automation, explained: “Robotics and automation is the key technology for increased competitiveness, quality and sustainability. If you want to make the best use of intelligent automation and robotics as well as find out about all new trends, you will find answers at automatica in Munich. It is clearly the leader in this topic area.”Learn more

Exhibition sectors Your contact Mrs. Dijana Baljidemic Manager Phone: 646-437-1017 Email:  dbaljidemic@tssworldwide.com STAY IN TOUCH  ‌   ‌   ‌

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today's videos.

These videos show some highlights from the Lake Kivu Challenge, which took place in Rwanda earlier this month. In addition to a conference and forum, international teams and their drones competed in emergency delivery, sample pick-up, and find and assess tasks.

[ Lake Kivu Challenge ]

The DARPA SubT Challenge Urban Circuit is ON!!!

[ SubT ]

One of the ways Microsoft trains autonomous systems is participating in research focused on solving real-world challenges, like aiding first responders in hazardous scenarios. This week, our collaborators at Carnegie Mellon University and Oregon State University, collectively named Team Explorer, are demonstrating tech breakthroughs in this area as they compete in the Feb 18-27, 2020 DARPA Subterranean (SubT) Urban Challenge in Elma, Washington.

The team is looking for another win after taking first place in round one of the DARPA SubT Challenge, the Tunnel Circuit, in August 2019. The competition continues with the Cave Circuit later in 2020, wrapping up with a final event incorporating all three underground environments in 2021.

[ Explorer ] via [ Microsoft ]

Spot can pull rickshaws now?

[ Tested ] via [ Gizmodo ]

Robot hugs!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people- connecting robots to humans and building relationships that last.

[ Roboy ]

Fabian Kung from Malaysia wrote in to share a video of a robot that he's been working on: "We designed and build this mini agile robot as part of our efforts in robotics and artificial intelligence research. It is kept small to reduce the cost and built time. Besides, there is less safety issue with small machine."

[ MMU ]

Thanks Fabian!

Happy (belated) Valentine's Day from Robotiq!

[ Robotiq ]

Happy (belated) Valentine's Day from Sphero!

C'mon dude, just pick all four. They're robots!

[ Sphero ]

Craving a bite out of a freshly grilled ballpark frank? Two robots named Jaco and Baxter can serve one up. Boston University engineers have made a jump in using machine learning to teach robots to perform complex tasks, a framework that could be applied to a host of tasks, like identifying cancerous spots on mammograms or better understanding spoken commands to play music. But first, as a proof of concept—they’ve learned how to prepare the perfect hot dog.

[ BU ]

The latest version of ETH Zurich's Ascento wheel-leg robot has gotten way more robust and capable over the last year.

[ Ascento ]

Snakes live in diverse environments ranging from unbearably hot deserts to lush tropical forests. But regardless of their habitat, they are able to slither up trees, rocks, and shrubbery with ease. By studying how the creatures move, a team of Johns Hopkins engineers have created a snake robot that can nimbly and stably climb large steps. The team's new findings, published in Journal of Experimental Biology and Royal Society Open Science, could advance the creation of search and rescue robots that can successfully navigate treacherous terrain.

[ JHU ]

In a recent demo conducted in Israel, RAFAEL’s Drone Dome C-UAS system performed interceptions of multiple drones, including maneuvering targets, using its hard-kill LASER BEAM director. The system achieved 100% success in all test scenarios. The stages of the interceptions included target detection, identification, and interception with a high-power LASER beam.

[ Rafael ]

EPFL has a little bit of robotics going on sometimes, you know?

[ EPFL ]

This video is basically an ad for ABB, but it's always fun to see robots picking stuff, especially when some of that stuff is tasty.

[ ABB ]

Hayk Martirosyan from Skydio gave a lecture as part of Pieter Abbeel's robotics course at UC Berkeley—this is where you hear about all the secret stuff Skydio is working on next.

[ UC Berkeley ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today's videos.

These videos show some highlights from the Lake Kivu Challenge, which took place in Rwanda earlier this month. In addition to a conference and forum, international teams and their drones competed in emergency delivery, sample pick-up, and find and assess tasks.

[ Lake Kivu Challenge ]

The DARPA SubT Challenge Urban Circuit is ON!!!

[ SubT ]

One of the ways Microsoft trains autonomous systems is participating in research focused on solving real-world challenges, like aiding first responders in hazardous scenarios. This week, our collaborators at Carnegie Mellon University and Oregon State University, collectively named Team Explorer, are demonstrating tech breakthroughs in this area as they compete in the Feb 18-27, 2020 DARPA Subterranean (SubT) Urban Challenge in Elma, Washington.

The team is looking for another win after taking first place in round one of the DARPA SubT Challenge, the Tunnel Circuit, in August 2019. The competition continues with the Cave Circuit later in 2020, wrapping up with a final event incorporating all three underground environments in 2021.

[ Explorer ] via [ Microsoft ]

Spot can pull rickshaws now?

[ Tested ] via [ Gizmodo ]

Robot hugs!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people- connecting robots to humans and building relationships that last.

[ Roboy ]

Fabian Kung from Malaysia wrote in to share a video of a robot that he's been working on: "We designed and build this mini agile robot as part of our efforts in robotics and artificial intelligence research. It is kept small to reduce the cost and built time. Besides, there is less safety issue with small machine."

[ MMU ]

Thanks Fabian!

Happy (belated) Valentine's Day from Robotiq!

[ Robotiq ]

Happy (belated) Valentine's Day from Sphero!

C'mon dude, just pick all four. They're robots!

[ Sphero ]

Craving a bite out of a freshly grilled ballpark frank? Two robots named Jaco and Baxter can serve one up. Boston University engineers have made a jump in using machine learning to teach robots to perform complex tasks, a framework that could be applied to a host of tasks, like identifying cancerous spots on mammograms or better understanding spoken commands to play music. But first, as a proof of concept—they’ve learned how to prepare the perfect hot dog.

[ BU ]

The latest version of ETH Zurich's Ascento wheel-leg robot has gotten way more robust and capable over the last year.

[ Ascento ]

Snakes live in diverse environments ranging from unbearably hot deserts to lush tropical forests. But regardless of their habitat, they are able to slither up trees, rocks, and shrubbery with ease. By studying how the creatures move, a team of Johns Hopkins engineers have created a snake robot that can nimbly and stably climb large steps. The team's new findings, published in Journal of Experimental Biology and Royal Society Open Science, could advance the creation of search and rescue robots that can successfully navigate treacherous terrain.

[ JHU ]

In a recent demo conducted in Israel, RAFAEL’s Drone Dome C-UAS system performed interceptions of multiple drones, including maneuvering targets, using its hard-kill LASER BEAM director. The system achieved 100% success in all test scenarios. The stages of the interceptions included target detection, identification, and interception with a high-power LASER beam.

[ Rafael ]

EPFL has a little bit of robotics going on sometimes, you know?

[ EPFL ]

This video is basically an ad for ABB, but it's always fun to see robots picking stuff, especially when some of that stuff is tasty.

[ ABB ]

Hayk Martirosyan from Skydio gave a lecture as part of Pieter Abbeel's robotics course at UC Berkeley—this is where you hear about all the secret stuff Skydio is working on next.

[ UC Berkeley ]

As social robots continue to show promise as assistive technologies, the exploration of appropriate and impactful robot behaviors is key to their eventual success. Teens are a unique population given their vulnerability to stress leading to both mental and physical illness. Much of teen stress stems from school, making the school environment an ideal location for a stress reducing technology. The goal of this mixed-methods study was to understand teens' operation of, and responsiveness to, a robot only capable of movement compared to a robot only capable of speech. Stemming from a human-centered approach, we introduce a Participatory Wizard of Oz (PWoz) interaction method that engaged teens as operators, users, and witnesses in a uniquely transparent interaction. In this paper, we illustrate the use of the PWoz interaction method as well as how it helps identify engaging robot interactions. Using this technique, we present results from a study with 62 teens that includes details of the complexity of teen stress and a significant reduction in negative attitudes toward robots after interactions. We analyzed the teens' interactions with both the verbal and non-verbal robots and identified strong themes of (1) authenticity, (2) empathy, (3) emotional engagement, and (4) imperfection creates connection. Finally, we reflect on the benefits and limitations of the PWoz method and our study to identify next steps toward the design and development of our social robot.

For effective virtual realities, “presence,” the feeling of “being there” in a virtual environment (VR), is deemed an essential prerequisite. Several studies have assessed the effect of the (non-)availability of auditory stimulation on presence, but due to differences in study design (e.g., virtual realities used, types of sounds included, rendering technologies employed), generalizing the results and estimating the effect of the auditory component is difficult. In two experiments, the influence of an ambient nature soundscape and movement-triggered step sounds were investigated regarding their effects on presence. In each experiment, approximately forty participants walked on a treadmill, thereby strolling through a virtual park environment reproduced via a stereoscopic head-mounted display (HMD), while the acoustical environment was delivered via noise-canceling headphones. In Experiment 1, conditions with the ambient soundscape and the step sounds either present or absent were combined in a 2 × 2 within-subjects design, supplemented with an additional “no-headphones” control condition. For the synchronous playback of step sounds, the probability of a step being taken was estimated by an algorithm using the HMD's sensor data. The results of Experiment 1 show that questionnaire-based measures of presence and realism were influenced by the soundscape but not by the reproduction of steps, which might be confounded with the fact that the perceived synchronicity of the sensor-triggered step sounds was rated rather low. Therefore, in Experiment 2, the step-reproduction algorithm was improved and judged to be more synchronous by participants. Consequently, large and statistically significant effects of both kinds of audio manipulations on perceived presence and realism were observed, with the effect of the soundscape being larger than that of including footstep sounds, possibly due to the remaining imperfections in the reproduction of steps. Including an appropriate soundscape or self-triggered footsteps had differential effects on subscales of presence, in that both affected overall presence and realism, while involvement was improved and distraction reduced by the ambient soundscape only.

The extension of the sense of self to the avatar during experiences of avatar embodiment requires thorough ethical and legal consideration, especially in light of potential scenarios involving physical or psychological harm caused to, or by, embodied avatars. We provide researchers and developers working in the field of virtual and robot embodiment technologies with a self-guidance tool based on the principles of Responsible Research and Innovation (RRI). This tool will help them engage in ethical and responsible research and innovation in the area of embodiment technologies in a way that guarantees all the rights of the embodied users and their interactors, including safety, privacy, autonomy, and dignity.

Illustration: Chris Philpot

Many young urbanites don’t want to own a car, and unlike earlier generations, they don’t have to rely on mass transit. Instead they treat mobility as a service: When they need to travel significant distances, say, more than 5 miles (8 kilometers), they use their phones to summon an Uber (or a car from a similar ride-sharing company). If they have less than a mile or so to go, they either walk or use various “micromobility” services, such as the increasingly ubiquitous Lime and Bird scooters or, in some cities, bike sharing.

The problem is that today’s mobility-as-a-service ecosystem often doesn’t do a good job covering intermediate distances, say a few miles. Hiring an Uber or Lyft for such short trips proves frustratingly expensive, and riding a scooter or bike more than a mile or so can be taxing to many people. So getting yourself to a destination that is from 1 to 5 miles away can be a challenge. Yet such trips account for about half of the total passenger miles traveled.

Many of these intermediate-distance trips take place in environments with limited traffic, such as university campuses and industrial parks, where it is now both economically reasonable and technologically possible to deploy small, low-speed autonomous vehicles powered by electricity. We’ve been involved with a startup that intends to make this form of transportation popular. The company, PerceptIn, hasautonomous vehicles operating at tourist sites in Nara and Fukuoka, Japan; at an industrial park in Shenzhen, China; and is just now arranging for its vehicles to shuttle people around Fishers, Ind., the location of the company’s headquarters.

Because these diminutive autonomous vehicles never exceed 20 miles (32 kilometers) per hour and don’t mix with high-speed traffic, they don’t engender the same kind of safety concerns that arise with autonomous cars that travel on regular roads and highways. While autonomous driving is a complicated endeavor, the real challenge for PerceptIn was not about making a vehicle that can drive itself in such environments—the technology to do that is now well established—but rather about keeping costs down.

Given how expensive autonomous cars still are in the quantities that they are currently being produced—an experimental model can cost you in the neighborhood of US $300,000—you might not think it possible to sell a self-driving vehicle of any kind for much less. Our experience over the past few years shows that, in fact, it is possible today to produce a self-driving passenger vehicle much more economically: PerceptIn’s vehicles currently sell for about $70,000, and the price will surely drop in the future. Here’s how we and our colleagues at PerceptIn brought the cost of autonomous driving down to earth.

Let’s start by explaining why autonomous cars are normally so expensive. In a nutshell, it’s because the sensors and computers they carry are very pricey.

The suite of sensors required for autonomous driving normally includes a high-end satellite-navigation receiver, lidar (light detection and ranging), one or more video cameras, radar, and sonar. The vehicle also requires at least one very powerful computer.

The satellite-navigation receivers used in this context aren’t the same as the one found in your phone. The kind built into autonomous vehicles have what is called real-time kinematic capabilities for high-precision position fixes—down to 10 centimeters. These devices typically cost about $4,000. Even so, such satellite-navigation receivers can’t be entirely relied on to tell the vehicle where it is. The fixes it gets could be off in situations where the satellite signals bounce off of nearby buildings, introducing noise and delays. In any case, satellite navigation requires an unobstructed view of the sky. In closed environments, such as tunnels, that just doesn’t work.

  Illustration: Chris Philpot

Fortunately, autonomous vehicles have other ways to figure out where they are. In particular they can use lidar, which determines distances to things by bouncing a laser beam off them and measuring how long it takes for the light to reflect back. A typical lidar unit for autonomous vehicles covers a range of 150 meters and samples more than 1 million spatial points per second.

Such lidar scans can be used to identify different shapes in the local environment. The vehicle’s computer then compares the observed shapes with the shapes recorded in a high-definition digital map of the area, allowing it to track the exact position of the vehicle at all times. Lidar can also be used to identify and avoid transient obstacles, such as pedestrians and other cars.

Lidar is a wonderful technology, but it suffers from two problems. First, these units are extremely expensive: A high-end lidar for autonomous driving can easily cost more than $80,000, although costs are dropping, and for low-speed applications a suitable unit can be purchased for about $4,000. Also, lidar, being an optical device, can fail to provide reasonable measurements in bad weather, such as heavy rain or fog.

The same is true for the cameras found on these vehicles, which are mostly used to recognize and track different objects, such as the boundaries of driving lanes, traffic lights, and pedestrians. Usually, multiple cameras are mounted around the vehicle. These cameras typically run at 60 frames per second, and the multiple cameras used can generate more than 1 gigabyte of raw data each second. Processing this vast amount of information, of course, places very large computational demands on the vehicle’s computer. On the plus side, cameras aren’t very expensive.

The radar and sonar systems found in autonomous vehicles are used for obstacle avoidance. The data sets they generate show the distance from the nearest object in the vehicle’s path. The major advantage of these systems is that they work in all weather conditions. Sonar usually covers a range of up to 10 meters, whereas radar typically has a range of up to 200 meters. Like cameras, these sensors are relatively inexpensive, often costing less than $1,000 each.

The many measurements such sensors supply are fed into the vehicle’s computers, which have to integrate all this information to produce an understanding of the environment. Artificial neural networks and deep learning, an approach that has grown rapidly in recent years, play a large role here. With these techniques, the computer can keep track of other vehicles moving nearby, as well as of pedestrians crossing the road, ensuring the autonomous vehicle doesn’t collide with anything or anyone.

Of course, the computers that direct autonomous vehicles have to do a lot more than just avoid hitting something. They have to make a vast number of decisions about where to steer and how fast to go. For that, the vehicle’s computers generate predictions about the upcoming movement of nearby vehicles before deciding on an action plan based on those predictions and on where the occupant needs to go.

Lastly, an autonomous vehicle needs a good map. Traditional digital maps are usually generated from satellite imagery and have meter-level accuracy. Although that’s more than sufficient for human drivers, autonomous vehicles demand higher accuracy for lane-level information. Therefore, special high-definition maps are needed.

Just like traditional digital maps, these HD maps contain many layers of information. The bottom layer is a map with grid cells that are about 5 by 5 cm; it’s generated from raw lidar data collected using special cars. This grid records elevation and reflection information about the objects in the environment.

Photos: Perceptin Slowly But Surely: The authors’ approach to autonomy has been applied to two different types of low-speed electric vehicles. One is a two-seat “pod,” shown here being demonstrated at Purdue University, where it was used to transport students from parking lots to the center of campus [top]. The other is a multipassenger bus, which is being used now at various sites around the world, including the Nara Palace historical park in Japan [bottom].

On top of that base grid, there are several layers of additional information. For instance, lane information is added to the grid map to allow autonomous vehicles to determine whether they are in the correct lane. On top of the lane information, traffic-sign labels are added to notify the autonomous vehicles of the local speed limit, whether they are approaching traffic lights, and so forth. This helps in cases where cameras on the vehicle are unable to read the signs.

Traditional digital maps are updated every 6 to 12 months. To make sure the maps that autonomous vehicles use contain up-to-date information, HD maps should be refreshed weekly. As a result, generating and maintaining HD maps can cost millions of dollars per year for a midsize city.

All that data on those HD maps has to be stored on board the vehicle in solid-state memory for ready access, adding to the cost of the computing hardware, which needs to be quite powerful. To give you a sense, an early computing system that Baidu employed for autonomous driving used an Intel Xeon E5 processor and four to eight Nvidia K80 GPU accelerators. The system was capable of delivering 64.5 trillion floating-point operations per second, but it consumed around 3,000 watts and generated an enormous amount of heat. And it cost about $30,000.

Given that the sensors and computers alone can easily cost more than $100,000, it’s not hard to understand why autonomous vehicles are so expensive, at least today. Sure, the price will come down as the total number manufactured increases. But it’s still unclear how the costs of creating and maintaining HD maps will be passed along. In any case, it will take time for better technology to address all the obvious safety concerns that come with autonomous driving on normal roads and highways.

We and our colleagues at PerceptIn have been trying to address these challenges by focusing on small, slow-speed vehicles that operate in limited areas and don’t have to mix with high-speed traffic—university campuses and industrial parks, for example.

The main tactic we’ve used to reduce costs is to do away with lidar entirely and instead use more affordable sensors: cameras, inertial measurement units, satellite positioning receivers, wheel encoders, radars, and sonars. The data that each of these sensors provides can then be combined though a process called sensor fusion.

With a balance of drawbacks and advantages, these sensors tend to complement one another. When one fails or malfunctions, others can take over to ensure that the system remains reliable. With this sensor-fusion approach, sensor costs could drop eventually to something like $2,000.

Because our vehicle runs at a low speed, it takes at the very most 7 meters to stop, making it much safer than a normal car, which can take tens of meters to stop. And with the low speed, the computing systems have less severe latency requirements than those used in high-speed autonomous vehicles.

PerceptIn’s vehicles use satellite positioning for initial localization. While not as accurate as the systems found on highway-capable autonomous cars, these satellite-navigation receivers still provide submeter accuracy. Using a combination of camera images and data from inertial measurement units (in a technique called visual inertial odometry), the vehicle’s computer further improves the accuracy, fixing position down to the decimeter level.

For imaging, PerceptIn has integrated four cameras into one hardware module. One pair faces the front of the vehicle, and another pair faces the rear. Each pair of cameras provides binocular vision, allowing it to capture the kind of spatial information normally given by lidar. What’s more, the four cameras together can capture a 360-degree view of the environment, with enough overlapping spatial regions between frames to ensure that visual odometry works in any direction.

Even if visual odometry were to fail and satellite-positioning signals were to drop out, all wouldn’t be lost. The vehicle could still work out position updates using rotary encoders attached to its wheels—following a general strategy that sailors used for centuries, called dead reckoning.

Data sets from all these sensors are combined to give the vehicle an overall understanding of its environment. Based on this understanding, the vehicle’s computer can make the decisions it requires to ensure a smooth and safe trip.

The vehicle also has an anti-collision system that operates independently of its main computer, providing a last line of defense. This uses a combination of millimeter-wave radars and sonars to sense when the vehicle is within 5 meters of objects, in which case it’s immediately stopped.

Relying on less expensive sensors is just one strategy that PerceptIn has pursued to reduce costs. Another has been to push computing to the sensors to reduce the demands on the vehicle’s main computer, a normal PC with a total cost less than $1,500 and a peak system power of just 400 W.

PerceptIn’s camera module, for example, can generate 400 megabytes of image information per second. If all this data were transferred to the main computer for processing, that computer would have to be extremely complex, which would have significant consequences in terms of reliability, power, and cost. PerceptIn instead has each sensor module perform as much computing as possible. This reduces the burden on the main computer and simplifies its design.

More specifically, a GPU is embedded into the camera module to extract features from the raw images. Then, only the extracted features are sent to the main computer, reducing the data-transfer rate a thousandfold.

Another way to limit costs involves the creation and maintenance of the HD maps. Rather than using vehicles outfitted with lidar units to provide map data, PerceptIn enhances existing digital maps with visual information to achieve decimeter-level accuracy.

The resultant high-precision visual maps, like the lidar-based HD maps they replace, consist of multiple layers. The bottom layer can be any existing digital map, such as one from the OpenStreetMap project. This bottom layer has a resolution of about 1 meter. The second layer records the visual features of the road surfaces to improve mapping resolution to the decimeter level. The third layer, also saved at decimeter resolution, records the visual features of other parts of the environment—such as signs, buildings, trees, fences, and light poles. The fourth layer is the semantic layer, which contains lane markings, traffic sign labels, and so forth.

While there’s been much progress over the past decade, it will probably be another decade or more before fully autonomous cars start taking to most roads and highways. In the meantime, a practical approach is to use low-speed autonomous vehicles in restricted settings. Several companies, including Navya, EasyMile, and May Mobility, along with PerceptIn, have been pursuing this strategy intently and are making good progress.

Eventually, as the relevant technology advances, the types of vehicles and deployments can expand, ultimately to include vehicles that can equal or surpass the performance of an expert human driver.

PerceptIn has shown that it’s possible to build small, low-speed autonomous vehicles for much less than it costs to make a highway-capable autonomous car. When the vehicles are produced in large quantities, we expect the manufacturing costs to be less than $10,000. Not too far in the future, it might be possible for such clean-energy autonomous shuttles to be carrying passengers in city centers, such as Manhattan’s central business district, where the average speed of traffic now is only 7 miles per hour[PDF]. Such a fleet would significantly reduce the cost to riders, improve traffic conditions, enhance safety, and improve air quality to boot. Tackling autonomous driving on the world’s highways can come later.

This article appears in the March 2020 print issue as “Autonomous Vehicles Lite.”

About the Authors

Shaoshan Liu is the cofounder and CEO of PerceptIn, an autonomous vehicle startup in Fishers, Ind. Jean-Luc Gaudiot is a professor of electrical engineering and computer science at the University of California, Irvine.

The Urban Circuit, the second of four robotics competitions that are part of the DARPA Subterranean Challenge, kicks off today, and you can follow along with what’s going on through DARPA’s livestream.

From what we know, the livestream is on a one hour time delay, but the good news is that it has commentary. “We intend to have both narration of the action and commentary where applicable,” DARPA SubT program manager Tim Chung told us. “We’re opening the aperture because there are a lot of really interesting features that we’d love our viewing audience to be able to witness, and that includes robots traversing interesting features and elements within the course, so you can look forward to more of that.”

The livestream starts at 9:45 am PT/12:45 pm ET. Watch below:

Scored runs (the thing that’ll be the most fun to watch) happen from February 20 (today) to February 22 (Saturday), and also February 24 (Monday) to February 26 (Wednesday).

There’s a media day on the 24 that we’ll be at, so make sure and let us know if there are specific things you want us to check out.

Also, you can hear directly from the teams themselves by following #SubTChallenge on Twitter.

The Urban Circuit, the second of four robotics competitions that are part of the DARPA Subterranean Challenge, kicks off today, and you can follow along with what’s going on through DARPA’s livestream.

From what we know, the livestream is on a one hour time delay, but the good news is that it has commentary. “We intend to have both narration of the action and commentary where applicable,” DARPA SubT program manager Tim Chung told us. “We’re opening the aperture because there are a lot of really interesting features that we’d love our viewing audience to be able to witness, and that includes robots traversing interesting features and elements within the course, so you can look forward to more of that.”

The livestream starts at 9:45 am PT/12:45 pm ET. Watch below:

Scored runs (the thing that’ll be the most fun to watch) happen from February 20 (today) to February 22 (Saturday), and also February 24 (Monday) to February 26 (Wednesday).

There’s a media day on the 24 that we’ll be at, so make sure and let us know if there are specific things you want us to check out.

Also, you can hear directly from the teams themselves by following #SubTChallenge on Twitter.

Pages