Feed aggregator

This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

At an early age, as we take our first steps into the world of math and numbers, we learn that one apple plus another apple equals two apples. We learn to count real things. Only later are we introduced to a weird concept: zero… or the number of apples in an empty box.

The concept of “zero” revolutionized math after Hindu-Arabic scholars and then the Italian mathematician Fibonacci introduced it into our modern numbering system. While today we comfortably use zero in all our mathematical operations, the concept of “nothing” has yet to enter the realm of artificial intelligence.

In a sense, AI and deep learning still need to learn how to recognize and reason with nothing.

Is it an apple or a banana? Neither!

Traditionally, deep learning algorithms such as deep neural networks (DNNs) are trained in a supervised fashion to recognize specific classes of things.

In a typical task, a DNN might be trained to visually recognize a certain number of classes, say pictures of apples and bananas. Deep learning algorithms, when fed a good quantity and quality of data, are really good at coming up with precise, low error, confident classifications.

The problem arises when a third, unknown object appears in front of the DNN. If an unknown object that was not present in the training set is introduced, such as an orange, then the network will be forced to “guess” and classify the orange as the closest class that captures the unknown object—an apple!

Basically, the world for a DNN trained on apples and bananas is completely made of apples and bananas. It can’t conceive the whole fruit basket.

Enter the world of nothing

While its usefulness is not immediately clear in all applications, the idea of “nothing” or a “class zero” is extremely useful in several ways when training and deploying a DNN.

During the training process, if a DNN has the ability to classify items as “apple,” “banana,” or “nothing,” the algorithm’s developers can determine if it hasn’t effectively learned to recognize a particular class. That said, if pictures of fruit continue to yield “nothing” responses, perhaps the developers need to add another “class” of fruit to identify, such as oranges.

Meanwhile, in a deployment scenario, a DNN trained to recognize healthy apples and bananas can answer “nothing” if there is a deviation from the prototypical fruit it has learned to recognize. In this sense, the DNN may act as an anomaly detection network—aside from classifying apples and bananas, it can also, without further changes, signal when it sees something that deviates from the norm.

As of today, there are no easy ways to train a standard DNN so that it can provide the functionality above.

One new approach called a lifelong DNN naturally incorporates the concept of “nothing” in its architecture. A lifelong DNN does this by cleverly utilizing feedback mechanisms to determine whether an input is a close match or instead a mismatch with what it has learned in the past.

This mechanism resembles how humans learn: we subconsciously and continuously check if our predictions match our world. For example, if somebody plays a trick on you and changes the height of your office chair, you’ll immediately notice it. That’s because you have a “model” of the height of your office chair that you’ve learned over time—if that model is disconfirmed, you realize the anomaly right away. We humans continuously check that our classifications match reality. If they don’t, our brains notice and emit an alert. For us, there are not only apples and bananas; there’s also the ability to reason that “I thought it was an apple, but it isn’t.”

A lifelong DNN captures this mechanism in its functioning, so it can output “nothing” when the model it has learned is disconfirmed.

Nothing to work with, no problem

Armed with a basic understanding of “nothing” using the example of apples and bananas, let’s now consider how this would play out in real-world applications beyond fruit identification.

Consider the manufacturing sector, where machines are tasked with producing massive volumes of products. Training a traditional computer-vision system to recognize different abnormalities in a product—say, surface scratches—is very challenging. On a well-run manufacturing line there aren’t many examples of what “bad” products look like, and “bad” can take an endless number of forms. There simply isn’t an abundance of data about bad products that can be used to train the system.

But with a lifelong DNN, a developer could train the computer-vision system to recognize what different examples of “good” products look like. Then, when the system detects a product that doesn’t match its definition of good, it can categorize that item as an anomaly to be handled appropriately.

For manufacturers, lifelong DNNs and the ability to detect anomalies can save time and improve efficiency in the production line. There may be similar benefits for countless other industries that are increasingly relying on AI.

Who knew that “nothing” could be so important?

Max Versace is CEO and co-founder of AI-powered visual inspections company Neurala.

The more dynamic robots get, the more likely they are to break. Or rather, all robots are 100 percent guaranteed to break eventually (this is one of their defining characteristics). More dynamic robots will also break more violently. While they’re in the lab, this isn’t a big deal, but for long term real-world use, wouldn’t it be great if we could rely on robots to repair themselves?

Rather than give a robot a screwdriver and expect it to replace its own parts, though, a much more elegant solution is robots that can heal themselves more like animals, where for many common injuries, all you have to do is sit around for a little while and your body will magically fix itself. We’ve seen a few examples of this before using self-healing polymers, but for dynamic robots that run and jump, you need the strength of metal.

At the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) last month, roboticists from the University of Tokyo’s JSK Lab presented a prototype for a robot leg with a tendon “fuse” made out of a metal that can repair fractures. It does that by autonomously melting itself down and reforming into a single piece. It’s still a work in progress, but it’s basically a tiny little piece of the T-1000 Terminator. Great!

Image: University of Tokyo/JSK Lab In one test, the researchers equipped a robotic leg with the self-healing bolt and dropped it to the ground. The bolt broke, as designed, avoiding damage to other parts of the robot. At that point, internal heating elements were activated and the two halves of the bolt liquified and then melted back together again, healing the robotic leg within about half an hour. The leg wasn’t able to fully stand up, but that’s what the researchers want to achieve with future prototypes.

This is a life-sized robotic leg with an Achilles tendon made up of a cable that transmits force from the foot around the ankle to the lower leg bone. The cable is bisected by a module containing a bolt made out of a metallic alloy that will snap under stress lower than any other point in the system, meaning that it acts like a mechanical fuse—it’ll be the first thing that breaks, sacrificing itself to protect the robot’s other joints. 

Image: University of Tokyo/JSK Lab The self-healing module (top left) consists of two halves connected by magnets and springs. Each half has a cartridge that the researchers fill with a low melting point alloy (U-47). When the cartridges heat up, the alloy melts, fusing the two halves together.

The alloy has a very low melting point (just 50° Celsius), and the module around it is made up of two halves connected by magnets and springs. If the bolt breaks, the magnets and springs will come apart also, but then snap back together, realigning the two broken halves of the bolt. At that point, internal heaters fire up, the two halves of the bolt liquify, and then melt back together again, healing the tendon within about half an hour. This video shows the robot falling, the tendon breaking, and then the robot self-healing and starting to stand up again:

In the video, it’s not quite good as new—it turns out that passive melting reduces the strength of the self-healing bolt to just 30 percent of where it was before the break. But after some additional experiments, the researchers discovered that gentle vibration during the melting and reforming process can bring the healed strength up above 90 percent of the original strength, and there’s likely even more optimization that can be done.

The researchers feel like this is a practical system to have in a real robot, and the plan is to refine it to the point where it’s a realistic feature to have on a dynamic legged robot.

“An Approach of Facilitated Investigation of Active Self-healing Tension Transmission System Oriented for Legged Robots,” by Shinsuke Nakashima, Takuma Shirai, Kento Kawaharazuka, Yuki Asano, Yohei Kakiuchi, Kei Okada, and Masayuki Inaba from the University of Tokyo, was presented at IROS 2019 in Macau.

The more dynamic robots get, the more likely they are to break. Or rather, all robots are 100 percent guaranteed to break eventually (this is one of their defining characteristics). More dynamic robots will also break more violently. While they’re in the lab, this isn’t a big deal, but for long term real-world use, wouldn’t it be great if we could rely on robots to repair themselves?

Rather than give a robot a screwdriver and expect it to replace its own parts, though, a much more elegant solution is robots that can heal themselves more like animals, where for many common injuries, all you have to do is sit around for a little while and your body will magically fix itself. We’ve seen a few examples of this before using self-healing polymers, but for dynamic robots that run and jump, you need the strength of metal.

At the 2019 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) last month, roboticists from the University of Tokyo’s JSK Lab presented a prototype for a robot leg with a tendon “fuse” made out of a metal that can repair fractures. It does that by autonomously melting itself down and reforming into a single piece. It’s still a work in progress, but it’s basically a tiny little piece of the T-1000 Terminator. Great!

Image: University of Tokyo/JSK Lab In one test, the researchers equipped a robotic leg with the self-healing bolt and dropped it to the ground. The bolt broke, as designed, avoiding damage to other parts of the robot. At that point, internal heating elements were activated and the two halves of the bolt liquified and then melted back together again, healing the robotic leg within about half an hour. The leg wasn’t able to fully stand up, but that’s what the researchers want to achieve with future prototypes.

This is a life-sized robotic leg with an Achilles tendon made up of a cable that transmits force from the foot around the ankle to the lower leg bone. The cable is bisected by a module containing a bolt made out of a metallic alloy that will snap under stress lower than any other point in the system, meaning that it acts like a mechanical fuse—it’ll be the first thing that breaks, sacrificing itself to protect the robot’s other joints. 

Image: University of Tokyo/JSK Lab The self-healing module (top left) consists of two halves connected by magnets and springs. Each half has a cartridge that the researchers fill with a low melting point alloy (U-47). When the cartridges heat up, the alloy melts, fusing the two halves together.

The alloy has a very low melting point (just 50° Celsius), and the module around it is made up of two halves connected by magnets and springs. If the bolt breaks, the magnets and springs will come apart also, but then snap back together, realigning the two broken halves of the bolt. At that point, internal heaters fire up, the two halves of the bolt liquify, and then melt back together again, healing the tendon within about half an hour. This video shows the robot falling, the tendon breaking, and then the robot self-healing and starting to stand up again:

In the video, it’s not quite good as new—it turns out that passive melting reduces the strength of the self-healing bolt to just 30 percent of where it was before the break. But after some additional experiments, the researchers discovered that gentle vibration during the melting and reforming process can bring the healed strength up above 90 percent of the original strength, and there’s likely even more optimization that can be done.

The researchers feel like this is a practical system to have in a real robot, and the plan is to refine it to the point where it’s a realistic feature to have on a dynamic legged robot.

“An Approach of Facilitated Investigation of Active Self-healing Tension Transmission System Oriented for Legged Robots,” by Shinsuke Nakashima, Takuma Shirai, Kento Kawaharazuka, Yuki Asano, Yohei Kakiuchi, Kei Okada, and Masayuki Inaba from the University of Tokyo, was presented at IROS 2019 in Macau.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

Robotic Arena – January 25, 2020 – Wrocław, Poland DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores

Let us know if you have suggestions for next week, and enjoy today’s videos.

One of Digit’s autonomy layers ensures a minimum distance from obstacles, even mobile ones like pesky engineers. In this video, the vision system is active and Digit is operating under full autonomy.

I would pay money to watch a second video that’s just like this one except Agility has given Digit a voice and it’s saying this stuff dynamically.

[ Agility Robotics ]

The Intel RealSense lidar camera L515 is the world’s smallest and most power efficient hi-resolution lidar, featuring unparalleled depth and accuracy that makes it perfect for indoor uses cases. The L515 is designed with proprietary technology that creates entirely new ways to incorporate lidar into smart devices to perceive the world in 3D.

[ Intel ]

This project investigates a design space, a fabrication system and applications of creating fluidic chambers and channels at millimeter scale for tangible actuated interfaces. The ability to design and fabricate millifluidic chambers allows one to create high frequency actuation, sequential control of flows and high resolution design on thin film materials. We propose a four dimensional design space of creating these fluidic chambers, a novel heat sealing system that enables easy and precise millifluidics fabrication, and application demonstrations of the fabricated materials for haptics, ambient devices and robotics.

[ MIT ]

This looks like it could be relaxing.

[ Eura Shin ]

This is only sort of robotics related, but it’s cool: changing the direction of a ping pong ball by nudging it with controllable ultrasound.

[ University of Tokyo ]

Check out the natural gait on this little running robot from Nagoya Institute of Technology:

[ Nagoya ]

UAV Turbines announced that it successfully demonstrated its Monarch Hybrid Range Extender (HREX), a microturbine powered generator technology that extends the range of electrically powered medium-sized unmanned aircraft.

In UAV Turbines’ HREX system, the engine extracts energy from the fuel and uses it to power the propulsion motor directly, with any excess electric power used to top off the battery charge. This greatly increases range before the weight of the added fuel becomes uneconomical. There are many tradeoffs in optimizing power for any specific system; for example, some energy is lost in the extraction process, but as the fuel is consumed, the net weight of the aircraft drops. It is this flexibility that enables engine optimizations not otherwise possible with a single power source.

[ UAV Turbines ]

Happy Holidays from Sphero!

[ Sphero ]

Happy Holidays from Yaskawa, which, despite having access to lots of real robots, stubbornly refuses to use them in their holiday videos.

[ Yaskawa ]

Join us in celebrating the life of Woodie Flowers, professor emeritus of mechanical engineering at MIT and co-founder of the FIRST Robotics Competition. A beloved teacher and pioneer in hands-on engineering education, Flowers developed design and robotics competitions at MIT, FIRST, and beyond, while promoting his concept of “gracious professionalism.”

[ MIT ]

I still really like the design of EMYS, although I admit that it looks a little strange when viewed from the side.

[ EMYS ]

Japanese college students compete to make the best laundry hanging robot, where speed and efficiency are a hilarious priority.

[ ROBOCON ]

The U in U-Drone stands for: Underground, Unlimited, Unjammable, User-Friendly and Unearthing. A prototype of a compact cable drone (U-Drone) has been developed in the one year project. It has been tested and demonstrated in underground tunnels (without GPS reception), whereby the drone can be operated at distances of up to 100 meters.

The commands to the U-Drone, and the images and data from the drone to the operator, run through an (unjammable) lightweight cable. The replaceable spool with the 100 meter cable is connected to the U-Drone and therefore unwinds from the drone during the flight in such a way that the drone is not stopped when the cable gets stuck.

[ Delft Dynamics ]

Interesting tiltrotor design for a drone that can apply a probe to a structure at any angle you want.

[ Skygauge ]

NASA has developed a flexible way to test new designs for aircraft that use multiple rotors to fly. The Multirotor Test Bed, or MTB, will let researchers study a wide variety of rotor configurations for different vehicles, including tiltrotor aircraft, mid-sized drones and even air taxis planned for the coming era of air travel called Urban Air Mobility.

[ NASA ]

Here’s a robot not to get on the wrong side of.

The Javelin Joint Venture team, a partnership of Lockheed Martin and Raytheon Company, successfully fired Javelin missiles from a Kongsberg remote weapon station integrated onto an unmanned vehicle platform. The demonstrations, conducted at the Redstone Test Center, Ala., validated the integration of the weapon station, missile and vehicle.

Raytheon ]

From Paul Scharre, who knows what he’s talking about more than most, a nuanced look at the lethal autonomous robots question.

[ Freethink ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

Robotic Arena – January 25, 2020 – Wrocław, Poland DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores

Let us know if you have suggestions for next week, and enjoy today’s videos.

One of Digit’s autonomy layers ensures a minimum distance from obstacles, even mobile ones like pesky engineers. In this video, the vision system is active and Digit is operating under full autonomy.

I would pay money to watch a second video that’s just like this one except Agility has given Digit a voice and it’s saying this stuff dynamically.

[ Agility Robotics ]

The Intel RealSense lidar camera L515 is the world’s smallest and most power efficient hi-resolution lidar, featuring unparalleled depth and accuracy that makes it perfect for indoor uses cases. The L515 is designed with proprietary technology that creates entirely new ways to incorporate lidar into smart devices to perceive the world in 3D.

[ Intel ]

This project investigates a design space, a fabrication system and applications of creating fluidic chambers and channels at millimeter scale for tangible actuated interfaces. The ability to design and fabricate millifluidic chambers allows one to create high frequency actuation, sequential control of flows and high resolution design on thin film materials. We propose a four dimensional design space of creating these fluidic chambers, a novel heat sealing system that enables easy and precise millifluidics fabrication, and application demonstrations of the fabricated materials for haptics, ambient devices and robotics.

[ MIT ]

This looks like it could be relaxing.

[ Eura Shin ]

This is only sort of robotics related, but it’s cool: changing the direction of a ping pong ball by nudging it with controllable ultrasound.

[ University of Tokyo ]

Check out the natural gait on this little running robot from Nagoya Institute of Technology:

[ Nagoya ]

UAV Turbines announced that it successfully demonstrated its Monarch Hybrid Range Extender (HREX), a microturbine powered generator technology that extends the range of electrically powered medium-sized unmanned aircraft.

In UAV Turbines’ HREX system, the engine extracts energy from the fuel and uses it to power the propulsion motor directly, with any excess electric power used to top off the battery charge. This greatly increases range before the weight of the added fuel becomes uneconomical. There are many tradeoffs in optimizing power for any specific system; for example, some energy is lost in the extraction process, but as the fuel is consumed, the net weight of the aircraft drops. It is this flexibility that enables engine optimizations not otherwise possible with a single power source.

[ UAV Turbines ]

Happy Holidays from Sphero!

[ Sphero ]

Happy Holidays from Yaskawa, which, despite having access to lots of real robots, stubbornly refuses to use them in their holiday videos.

[ Yaskawa ]

Join us in celebrating the life of Woodie Flowers, professor emeritus of mechanical engineering at MIT and co-founder of the FIRST Robotics Competition. A beloved teacher and pioneer in hands-on engineering education, Flowers developed design and robotics competitions at MIT, FIRST, and beyond, while promoting his concept of “gracious professionalism.”

[ MIT ]

I still really like the design of EMYS, although I admit that it looks a little strange when viewed from the side.

[ EMYS ]

Japanese college students compete to make the best laundry hanging robot, where speed and efficiency are a hilarious priority.

[ ROBOCON ]

The U in U-Drone stands for: Underground, Unlimited, Unjammable, User-Friendly and Unearthing. A prototype of a compact cable drone (U-Drone) has been developed in the one year project. It has been tested and demonstrated in underground tunnels (without GPS reception), whereby the drone can be operated at distances of up to 100 meters.

The commands to the U-Drone, and the images and data from the drone to the operator, run through an (unjammable) lightweight cable. The replaceable spool with the 100 meter cable is connected to the U-Drone and therefore unwinds from the drone during the flight in such a way that the drone is not stopped when the cable gets stuck.

[ Delft Dynamics ]

Interesting tiltrotor design for a drone that can apply a probe to a structure at any angle you want.

[ Skygauge ]

NASA has developed a flexible way to test new designs for aircraft that use multiple rotors to fly. The Multirotor Test Bed, or MTB, will let researchers study a wide variety of rotor configurations for different vehicles, including tiltrotor aircraft, mid-sized drones and even air taxis planned for the coming era of air travel called Urban Air Mobility.

[ NASA ]

Here’s a robot not to get on the wrong side of.

The Javelin Joint Venture team, a partnership of Lockheed Martin and Raytheon Company, successfully fired Javelin missiles from a Kongsberg remote weapon station integrated onto an unmanned vehicle platform. The demonstrations, conducted at the Redstone Test Center, Ala., validated the integration of the weapon station, missile and vehicle.

Raytheon ]

From Paul Scharre, who knows what he’s talking about more than most, a nuanced look at the lethal autonomous robots question.

[ Freethink ]

Due to their energy density and softness that are comparable to human muscles dielectric elastomer (DE) transducers are well-suited for soft-robotic applications. This kind of transducer combines actuator and sensor functionality within one transducer so that no external senors to measure the deformation or to detect collisions are required. Within this contribution we present a novel self-sensing control for a DE stack-transducer that allows to control several different quantities of the DE transducer with the same controller. This flexibility is advantageous e.g., for the development of human machine interfaces with soft-bodied robots. After introducing the DE stack-transducer that is driven by a bidirectional flyback converter, the development of the self-sensing state and disturbance estimator based on an extended Kalman-filter is explained. Compared to known estimators designed for DE transducers supplied by bulky high-voltage amplifiers this one does not require any superimposed excitation to enable the sensor capability so that it also can be used with economic and competitive power electronics like the flyback converter. Due to the behavior of this converter a sliding mode energy controller is designed afterwards. By introducing different feed-forward controls the voltage, force or deformation can be controlled. The validation proofs that both the developed self-sensing estimator as well as the self-sensing control yield comparable results as previously published sensor-based approaches.

Data scientists and software engineers who work with big data are in high demand. Thinknum Media called this field the hottest profession in 2019. Job search site Indeed earlier this year reported that job listings for data scientists jumped 31 percent between 2017 and 2018, while searches only increased 14 percent.

But what skills do you need to fill this lucrative niche?

Indeed set out to answer that question by looking at 500 tech skill terms related to data science that appeared in tech jobs posted on the site during the past five years. The analysis determined that, while Python dominates, Spark is on the fastest growth path and demand for engineers familiar with the statistical programming language R is also growing fast. Also on the radar: Hadoop, Tableau, SAS, Matlab, Redshift, and TensorFlow. [See graph, below, which omits Python because demand is literally off the charts, and because it is not strictly a data science skill.]

In terms of exactly how these skills are being applied, Indeed looked four fields that require data scientists. Machine learning came out on top—and is growing the fastest—followed by artificial intelligence, deep learning, and natural language processing. [See graph, below.]

In September, Facebook sent out a strange casting call: We need all types of people to look into a webcam or phone camera and say very mundane things. The actors stood in bedrooms, hallways, and backyards, and they talked about topics such as the perils of junk food and the importance of arts education. It was a quick and easy gig—with an odd caveat. Facebook researchers would be altering the videos, extracting each person’s face and fusing it onto another person’s head. In other words, the participants had to agree to become deepfake characters. 

Facebook’s artificial intelligence (AI) division put out this casting call so it could ethically produce deepfakes—a term that originally referred to videos that had been modified using a certain face-swapping technique but is now a catchall for manipulated video. The Facebook videos are part of a training data set that the company assembled for a global competition called the Deepfake Detection Challenge. In this competition—produced in cooperation with Amazon, Microsoft, the nonprofit Partnership on AI, and academics from eight universities—researchers around the world are vying to create automated tools that can spot fraudulent media.

The competition launched today, with an announcement at the AI conference NeurIPS, and will accept entries through March 2020. Facebook has dedicated more than US $10 million for awards and grants.

Cristian Canton Ferrer helped organize the challenge as research manager for Facebook’s AI Red Team, which analyzes the threats that AI poses to the social media giant. He says deepfakes are a growing danger not just to Facebook but to democratic societies. Manipulated videos that make politicians appear to do and say outrageous things could go viral before fact-checkers have a chance to step in.

“We’re thinking about what will be happening a year from now. It’s a cat-and-mouse approach.”  —Cristian Canton Ferrer, Facebook AI

While such a full-blown synthetic scandal has yet to occur, the Italian public recently got a taste of the possibilities. In September, a satirical news show aired a deepfake video featuring a former Italian prime minister apparently lavishing insults on other politicians. Most viewers realized it was a parody, but a few did not.

The U.S. presidential elections in 2020 are an added incentive to get ahead of the problem, says Canton Ferrer. He believes that media manipulation will become much more common over the coming year, and that the deepfakes will get much more sophisticated and believable. “We’re thinking about what will be happening a year from now,” he says. “It’s a cat-and-mouse approach.” Canton ­Ferrer’s team aims to give the cat a head start, so it will be ready to pounce.

The growing threat of deepfakes

Just how easy is it to make deepfakes? A recent audit of online resources for altering videos found that the available open-source software still requires a good amount of technical expertise. However, the audit also turned up apps and services that are making it easier for almost anyone to get in on the action. In China, a deepfake app called Zao took the country by storm in September when it offered people a simple way to superimpose their own faces onto those of actors like Leonardo DiCaprio and Marilyn Monroe.

It may seem odd that the data set compiled for Facebook’s competition is filled with unknown people doing unremarkable things. But a deepfake detector that works on those mundane videos should work equally well for videos featuring politicians. To make the Facebook challenge as realistic as possible, Canton Ferrer says his team used the most common open-source techniques to alter the videos—but he won’t name the methods, to avoid tipping off contestants. “In real life, they will not be able to ask the bad actors, ‘Can you tell me what method you used to make this deepfake?’” he says.

In the current competition, detectors will be scanning for signs of facial manipulation. However, the Facebook team is keeping an eye on new and emerging attack methods, such as full-body swaps that change the appearance and actions of a person from head to toe. “There are some of those out there, but they’re pretty obvious now,” ­Canton Ferrer says. “As they get better, we’ll add them to the data set.” Even after the detection challenge concludes in March, he says, the Facebook team will keep working on the problem of deepfakes.

As for how the winning detection methods will be used and whether they’ll be integrated into Facebook’s operations, Canton Ferrer says those decisions aren’t up to him. The Partnership on AI’s steering committee on AI and media integrity, which is overseeing the competition, will decide on the next steps, he says. Claire Leibowicz, who leads that steering committee, says the group will consider “coordinated efforts” to fight back against the global challenge of synthetic and manipulated media.

DARPA’s efforts on deepfake detection

The Facebook challenge is far from the only effort to counter deepfakes. DARPA’s Media Forensics program launched in 2016, a year before the first deepfake videos surfaced on Reddit. Program manager Matt Turek says that as the technology took off, the researchers working under the program developed a number of detection technologies, generally looking for “digital integrity, physical integrity, or semantic integrity.”

Digital integrity is defined by the patterns in an image’s pixels that are invisible to the human eye. These patterns can arise from cameras and video processing software, and any inconsistencies that appear are a tip-off that a video has been altered. Physical integrity refers to the consistency in lighting, shadows, and other physical attributes in an image. Semantic integrity considers the broader context. If a video shows an outdoor scene, for example, a deepfake detector might check the time stamp and location to look up the weather report from that time and place. The best automated detector, Turek says, would “use all those techniques to produce a single integrity score that captures everything we know about a digital asset.”

DARPA’s Media Forensics program created deepfake detectors that look at digital, physical, and semantic integrity. 

Turek says his team has created a prototype Web portal (restricted to its government partners) to demonstrate a sampling of the detectors developed during the program. When the user uploads a piece of media via the Web portal, more than 20 detectors employ a range of different approaches to try to determine whether an image or video has been manipulated. Turek says his team continues to add detectors to the system, which is already better than humans at spotting fakes.

A successor to the Media Forensics program will launch in mid-2020: the Semantic Forensics program. This broader effort will cover all types of media—text, images, videos, and audio—and will go beyond simply detecting manipulation. It will also seek methods to understand the importance of the manipulations, which could help organizations decide which content requires human review. “If you manipulate a vacation photo by adding a beach ball, it really doesn’t matter,” Turek says. “But if you manipulate an image about a protest and add an object like a flag, that could change people’s understanding of who was involved.”

The Semantic Forensics program will also try to develop tools to determine if a piece of media really comes from the source it claims. Eventually, Turek says, he’d like to see the tech community embrace a system of watermarking, in which a digital signature would be embedded in the media itself to help with the authentication process. One big challenge of this idea is that every software tool that interacts with the image, video, or other piece of media would have to “respect that watermark, or add its own,” Turek says. “It would take a long time for the ecosystem to support that.”

A deepfake detection tool for consumers

In the meantime, the AI Foundation has a plan. This nonprofit is building a tool called Reality Defender that’s due to launch in early 2020. “It will become your personal AI guardian who’s watching out for you,” says Rob Meadows, president and chief technology officer for the foundation.

Reality Defender “will become your personal AI guardian who’s watching out for you.” —Rob Meadows, AI Foundation

Reality Defender is a plug-in for Web browsers and an app for mobile phones. It scans everything on the screen using a suite of automatic detectors, then alerts the user about altered media. Detection alone won’t make for a useful tool, since ­Photoshop and other editing tools are widely used in fashion, advertising, and entertainment. If Reality Defender draws attention to every altered piece of content, Meadows notes, “it will flood consumers to the point where they say, ‘We don’t care anymore, we have to tune it out.’”

To avoid that problem, users will be able to dial the tool’s sensitivity up or down, depending on how many alerts they want. Meadows says beta testers are currently training the system, giving it feedback on which types of manipulations they care about. Once Reality Defender launches, users will be able to personalize their AI guardian by giving it a thumbs-up or thumbs-down on alerts, until it learns their preferences. “A user can say, ‘For my level of paranoia, this is what works for me,’ ” Meadows says.

He sees the software as a useful stopgap solution, but ultimately he hopes that his group’s technologies will be integrated into platforms such as Facebook, YouTube, and Twitter. He notes that Biz Stone, cofounder of Twitter, is a member of the AI Foundation’s board. To truly protect society from fake media, Meadows says, we need tools that prevent falsehoods from getting hosted on platforms and spread via social media. Debunking them after they’ve already spread is too late.

The researchers at Jigsaw, a unit of Alphabet that works on technology solutions for global challenges, would tend to agree. Technical research manager Andrew Gully says his team identified synthetic media as a societal threat some years back. To contribute to the fight, Jigsaw teamed up with sister company Google AI to produce a deepfake data set of its own in late 2018, which they contributed to the FaceForensics data set hosted by the Technical University of Munich.

Gully notes that while we haven’t yet seen a political crisis triggered by a deepfake, these videos are also used for bullying and “revenge porn,” in which a targeted woman’s face is pasted onto the face of an actor in a porno. (While pornographic deepfakes could in theory target men, a recent audit of deepfake content found that 100 percent of the pornographic videos focused on women.) What’s more, Gully says people are more likely to be credulous of videos featuring unknown individuals than famous politicians.

But it’s the threat to free and fair elections that feels most crucial in this U.S. election year. Gully says systems that detect deepfakes must take a careful approach in communicating the results to users. “We know already how difficult it is to convince people in the face of their own biases,” Gully says. “Detecting a deepfake video is hard enough, but that’s easy compared to how difficult it is to convince people of things they don’t want to believe.”

State of the art algorithms for many pattern recognition problems rely on data-driven deep network models. Training these models requires a large labeled dataset and considerable computational resources. Also, it is difficult to understand the working of these learned models, limiting their use in some critical applications. Toward addressing these limitations, our architecture draws inspiration from research in cognitive systems, and integrates the principles of commonsense logical reasoning, inductive learning, and deep learning. As a motivating example of a task that requires explainable reasoning and learning, we consider Visual Question Answering in which, given an image of a scene, the objective is to answer explanatory questions about objects in the scene, their relationships, or the outcome of executing actions on these objects. In this context, our architecture uses deep networks for extracting features from images and for generating answers to queries. Between these deep networks, it embeds components for non-monotonic logical reasoning with incomplete commonsense domain knowledge, and for decision tree induction. It also incrementally learns and reasons with previously unknown constraints governing the domain's states. We evaluated the architecture in the context of datasets of simulated and real-world images, and a simulated robot computing, executing, and providing explanatory descriptions of plans and experiences during plan execution. Experimental results indicate that in comparison with an “end to end” architecture of deep networks, our architecture provides better accuracy on classification problems when the training dataset is small, comparable accuracy with larger datasets, and more accurate answers to explanatory questions. Furthermore, incremental acquisition of previously unknown constraints improves the ability to answer explanatory questions, and extending non-monotonic logical reasoning to support planning and diagnostics improves the reliability and efficiency of computing and executing plans on a simulated robot.

Expectation learning is a unsupervised learning process which uses multisensory bindings to enhance unisensory perception. For instance, as humans, we learn to associate a barking sound with the visual appearance of a dog, and we continuously fine-tune this association over time, as we learn, e.g., to associate high-pitched barking with small dogs. In this work, we address the problem of developing a computational model that addresses important properties of expectation learning, in particular focusing on the lack of explicit external supervision other than temporal co-occurrence. To this end, we present a novel hybrid neural model based on audio-visual autoencoders and a recurrent self-organizing network for multisensory bindings that facilitate stimulus reconstructions across different sensory modalities. We refer to this mechanism as stimulus prediction across modalities and demonstrate that the proposed model is capable of learning concept bindings by evaluating it on unisensory classification tasks for audio-visual stimuli using the 43,500 Youtube videos from the animal subset of the AudioSet corpus.

Yoshua Bengio is known as one of the “three musketeers” of deep learning, the type of artificial intelligence (AI) that dominates the field today. 

Bengio, a professor at the University of Montreal, is credited with making key breakthroughs in the use of neural networks—and just as importantly, with persevering with the work through the long cold AI winter of the late 1980s and the 1990s, when most people thought that neural networks were a dead end. 

He was rewarded for his perseverance in 2018, when he and his fellow musketeers (Geoffrey Hinton and Yann LeCun) won the Turing Award, which is often called the Nobel Prize of computing.

Today, there’s increasing discussion about the shortcomings of deep learning. In that context, IEEE Spectrum spoke to Bengio about where the field should go from here. He’ll speak on a similar subject tomorrow at NeurIPS, the biggest and buzziest AI conference in the world; his talk is titled “From System 1 Deep Learning to System 2 Deep Learning.”

Yoshua Bengio on . . .

  1. Deep learning and its discontents
  2. The dawn of brain-inspired computation
  3. Learning to learn
  4. “This is not ready for industry”
  5. Physics, language, and common sense
  1. Deep learning and its discontents

    IEEE Spectrum: What do you think about all the discussion of deep learning’s limitations?

    Yoshua Bengio: Too many public-facing venues don’t understand a central thing about the way we do research, in AI and other disciplines: We try to understand the limitations of the theories and methods we currently have, in order to extend the reach of our intellectual tools. So deep learning researchers are looking to find the places where it’s not working as well as we’d like, so we can figure out what needs to be added and what needs to be explored.

    This is picked up by people like Gary Marcus, who put out the message: “Look, deep learning doesn’t work.” But really, what researchers like me are doing is expanding its reach. When I talk about things like the need for AI systems to understand causality, I’m not saying that this will replace deep learning. I’m trying to add something to the toolbox.

    What matters to me as a scientist is what needs to be explored in order to solve the problems. Not who’s right, who’s wrong, or who’s praying at which chapel.

    Spectrum: How do you assess the current state of deep learning?

    Bengio: In terms of how much progress we’ve made in this work over the last two decades: I don’t think we’re anywhere close today to the level of intelligence of a two-year-old child. But maybe we have algorithms that are equivalent to lower animals, for perception. And we’re gradually climbing this ladder in terms of tools that allow an entity to explore its environment.

    One of the big debates these days is: What are the elements of higher-level cognition? Causality is one element of it, and there’s also reasoning and planning, imagination, and credit assignment (“what should I have done?”). In classical AI, they tried to obtain these things with logic and symbols. Some people say we can do it with classic AI, maybe with improvements.

    Then there are people like me, who think that we should take the tools we’ve built in last few years to create these functionalities in a way that’s similar to the way humans do reasoning, which is actually quite different from the way a purely logical system based on search does it.

    BACK TO TOP↑

    The dawn of brain-inspired computation

    Spectrum: How can we create functions similar to human reasoning?

    Bengio: Attention mechanisms allow us to learn how to focus our computation on a few elements, a set of computations. Humans do that—it’s a particularly important part of conscious processing. When you’re conscious of something, you’re focusing on a few elements, maybe a certain thought, then you move on to another thought. This is very different from standard neural networks, which are instead parallel processing on a big scale. We’ve had big breakthroughs on computer vision, translation, and memory thanks to these attention mechanisms, but I believe it’s just the beginning of a different style of brain-inspired computation.

    It’s not that we have solved the problem, but I think we have a lot of the tools to get started. And I’m not saying it’s going to be easy. I wrote a paper in 2017 called “The Consciousness Prior” that laid out the issue. I have several students working on this and I know it is a long-term endeavor.

    Spectrum: What other aspects of human intelligence would you like to replicate in AI?

    Bengio: We also talk about the ability of neural nets to imagine: Reasoning, memory, and imagination are three aspects of the same thing going on in your mind. You project yourself into the past or the future, and when you move along these projections, you’re doing reasoning. If you anticipate something bad happening in the future, you change course—that’s how you do planning. And you’re using memory too, because you go back to things you know in order to make judgments. You select things from the present and things from the past that are relevant.

    Attention is the crucial building block here. Let’s say I’m translating a book into another language. For every word, I have to carefully look at a very small part of the book. Attention allows you abstract out a lot of irrelevant details and focus what matters. Being able to pick out the relevant elements—that’s what attention does.

    Spectrum: How does that translate to machine learning?

    Bengio: You don’t have to tell the neural net what to pay attention to—that’s the beauty of it. It learns it on its own. The neural net learns how much attention, or weight, it should give to each element in a set of possible elements to consider.

    BACK TO TOP↑ Learning to learn

    Spectrum: How is your recent work on causality related to these ideas?

    Bengio: The kind of high-level concepts that you reason with tend to be variables that are cause and/or effect. You don’t reason based on pixels. You reason based on concepts like door or knob or open or closed. Causality is very important for the next steps of progress of machine learning.

    And it’s related to another topic that is much on the minds of people in deep learning. Systematic generalization is the ability humans have to generalize the concepts we know, so they can be combined in new ways that are unlike anything else we’ve seen. Today’s machine learning doesn’t know how to do that. So you often have problems relating to training on a particular data set. Say you train in one country, and then deploy in another country. You need generalization and transfer learning. How do you train a neural net so that if you transfer it into a new environment, it continues to work well or adapts quickly?

    Spectrum: What's the key to that kind of adaptability?

    Bengio: Meta-learning is a very hot topic these days: Learning to learn. I wrote an early paper on this in 1991, but only recently did we get the computational power to implement this kind of thing. It’s computationally expensive. The idea: In order to generalize to a new environment, you have to practice generalizing to a new environment. It’s so simple when you think about it. Children do it all the time. When they move from one room to another room, the environment is not static, it keeps changing. Children train themselves to be good at adaptation. To do that efficiently, they have to use the pieces of knowledge they’ve acquired in the past. We’re starting to understand this ability, and to build tools to replicate it.

    One critique of deep learning is that it requires a huge amount of data. That’s true if you just train it on one task. But children have the ability to learn based on very little data. They capitalize on the things they’ve learned before. But more importantly, they’re capitalizing on their ability to adapt and generalize.

    BACK TO TOP↑ “This is not ready for industry”

    Spectrum: Will any of these ideas be used in the real world anytime soon?

    Bengio: No. This is all very basic research using toy problems. That’s fine, that’s where we’re at. We can debug these ideas, move on to new hypotheses. This is not ready for industry tomorrow morning.

    But there are two practical limitations that industry cares about, and that this research may help. One is building systems that are more robust to changes in the environment. Two: How do we build natural language processing systems, dialogue systems, virtual assistants? The problem with the current state of the art systems that use deep learning is that they’re trained on huge quantities of data, but they don’t really understand well what they’re talking about. People like Gary Marcus pick up on this and say, “That’s proof that deep learning doesn’t work.” People like me say, “That’s interesting, let’s tackle the challenge.”

    BACK TO TOP↑ Physics, language, and common sense

    Spectrum: How could chatbots do better?

    Bengio: There’s an idea called grounded language learning which is attracting new attention recently. The idea is, an AI system should not learn only from text. It should learn at the same time how the world works, and how to describe the world with language. Ask yourself: Could a child understand the world if they were only interacting with the world via text? I suspect they would have a hard time.

    This has to do with conscious versus unconscious knowledge, the things we know but can’t name. A good example of that is intuitive physics. A two-year-old understands intuitive physics. They don’t know Newton’s equations, but they understand concepts like gravity in a concrete sense. Some people are now trying to build systems that interact with their environment and discover the basic laws of physics.

    Spectrum: Why would a basic grasp of physics help with conversation?

    Bengio: The issue with language is that often the system doesn’t really understand the complexity of what the words are referring to. For example, the statements used in the Winograd schema; in order to make sense of them, you have to capture physical knowledge. There are sentences like: “Jim wanted to put the lamp into his luggage, but it was too large.” You know that if this object is too large for putting in the luggage, it must be the “it,” the subject of the second phrase. You can communicate that kind of knowledge in words, but it’s not the kind of thing we go around saying: “The typical size of a piece of luggage is x by x.”

    We need language understanding systems that also understand the world. Currently, AI researchers are looking for shortcuts. But they won’t be enough. AI systems also need to acquire a model of how the world works.

    BACK TO TOP↑

One year ago, for IEEE Spectrum’s special report on the Top Tech for 2019, Sarcos Robotics promised that by the end of the year they’d be ready to ship a powered exoskeleton that would be the future of industrial work. And late last month, Sarcos invited us to Salt Lake City, Utah, to see what that future looks like.

Sarcos has been developing powered exoskeletons and the robotic technologies that make them possible for decades, and the lobby of the company’s headquarters is a resting place for concepts and prototype hardware that’s been abandoned along the way. But now, Sarcos is ready to unveil the prototype of the Guardian XO, a strength-multiplying exoskeleton that’s about to begin shipping.

As our introductory briefing concludes, Sarcos CEO Ben Wolff is visibly excited to be able to show off what they’ve been working on in their lab. “If you were to ask the question, What does 30 years and $300 million look like,” Wolff tells us, “you're going to see it downstairs.”

This is what we see downstairs:

GIF: Evan Ackerman/IEEE Spectrum Guardian XO operator Fletcher Garrison demonstrates the company’s exosuit by lifting a 125-pound payload. Sarcos says this task usually requires three people. How the Guardian XO Works

The Sarcos Guardian XO is a 24-degrees-of-freedom full-body robotic exoskeleton. While wearing it, a human can lift  200 pounds (90 kilograms) while feeling like they’re lifting just 10 lbs (4.5 kg). The Guardian XO is fully electrical and untethered with a runtime of 2 hours, and hot-swappable battery packs can keep it going for a full work day. It takes seconds to put on and take off, and Sarcos says new users can be trained to use the system in minutes. One Guardian XO costs $100,000 per year to rent, and the company will be shipping its first batch of alpha units to customers (including both heavy industry and the U.S. military) in January.

Photo: Evan Ackerman/IEEE Spectrum The prototype that Sarcos demonstrated had all of the functionality of the version that will ship in January, but latter models will include plastic fairings over the suit as well as quick-change end-effectors.

In a practical sense, the Guardian XO is a humanoid robot that uses a real human as its command and control system. As companies of all kinds look towards increasing efficiency through automation, Sarcos believes that the most effective solution is a direct combination of humans and machines, enhancing the intelligence and judgement of humans with the strength and endurance of robots. (Investors in the company include Caterpillar, GE Ventures, Microsoft, and Schlumberger.)

The first thing to understand about the Guardian XO is that like a humanoid robot, it’s self-supporting. Since it has its own legs and feet, the 150 lb weight of the suit (and whatever it’s carrying) bypasses its user and is transferred directly into the ground. You don’t strap the robot to you—you strap yourself to the robot, a process that takes less than a minute. So although it looks heavy and bulky (and it is definitely both of those things), at least the weight of the system isn’t something that the user experiences directly. You can see how that works by watching Guardian XO operator Fletcher Garrison lifting all kinds of payloads in the video below.

Hands On With the Guardian XO

When Sarcos reached out and asked if we wanted to come to Salt Lake City to try out the XO, we immediately said yes (disclosure: Sarcos covered our costs to attend a media event last month). But we were disappointed when, in the end, we were only allowed to try out a one-armed version of the exoskeleton. I even offered to sign additional waivers but, alas, the company wouldn’t let me into the full suit. So my experience with the exo was pretty limited—a hands-on, literally, of a single XO arm.

Photo: Evan Ackerman/IEEE Spectrum That’s me trying out the one-arm XO system. It’s not quite like the full-body suit, but Sarcos still required me to sign a “waiver of liability, assumption of risk, and indemnity agreement.”

Still, it was an amazing sensation. The arm I tested, which Sarcos says uses the same control system as the full-body suit, was incredibly easy to operate. In terms of control, all the exo tries to do is get out of the way of your limbs: It uses force sensors to detect every motion that you make, and then moves its own limbs in parallel, smoothly matching your body with its own hardware. If you take a step, it takes a step with you. If you swing your arm back and forth, it swings its arm back and forth in the same way, right next to yours. There’s no discernible lag to this process, and it’s so intuitive that Sarcos says most people take just a minute or two to get comfortable using the system, and just an hour or two to be comfortable doing work in it.

The Guardian XO can augment the strength of the user all the way up to making a 200-pound load feel like it weighs zero pounds. Typically, this is not how the exoskeleton works, though, since it can be disconcerting to be lifting something heavy and not feel like you’re lifting anything at all. It’s better to think of the exo as a tool that makes you stronger rather than a tool that makes objects weightless, especially since you still have to deal with inertia. Remember, even if something has no apparent weight (either because you’re in space or because you’re holding it with a powered exoskeleton), it still has mass, which you have to be aware of when trying to move it or stop it from moving. The amount of help that the exo gives you is easy to adjust; it’s got a graphical control panel on the left wrist.

GIF: Evan Ackerman/IEEE Spectrum This ammo crate weighs 110 pounds, but the exoskeleton makes it feel like each arm is lifting just 6 pounds. The Guardian XO is designed for loads of up to 200 lbs. How Safe Is the Exoskeleton?

With a robotic system this powerful (XO has a peak torque of about 4000 inch-pounds, or 450 newton-meters), Sarcos made safety a top priority. For example, to move the exo’s arms, your hands need to be holding down triggers. If you let go of the triggers (for whatever reason), the arms will lock in place, which has the added benefit of letting the exo hold stuff up for you while you, say, check your phone. All of the joints are speed limited, meaning that you can’t throw a punch with the exo—they told me this during my demo, so of course I tried it, and the joints locked themselves as soon as I exceeded their safety threshold. If the system loses power for any reason, current shunts back through the motors, bringing them down gradually rather than abruptly. And by design the joints are not capable of exceeding a human range of motion, which means that the exoskeleton can’t bend or twist in a way that would injure you. Interestingly, the Guardian XO’s joint speeds are easily fast enough to allow you to run, although that’s been limited for safety reasons as well.

We asked about whether falling down was much of a risk, but it turns out that having a human in the loop for control makes that problem much simpler. Sarcos hasn’t had to program the Guardian XO to balance itself, because the human inside does all of that naturally. Having someone try to push you over while you’re in the exoskeleton is no different than having someone try to push you over while you’re out of it, because you’ll keep your own balance in either case. If you do end up falling over, Sarcos claims that the exoskeleton is designed as a roll cage, so odds are you’ll be fine, although it’s not clear how easy it would be to get out of it afterwards (or get it off of you).

More of a concern is how the XO will operate around other people. While its mass and bulk may not make all that much of a different to the user, it seems like working collaboratively could be a problem, as could working in small spaces or around anything fragile. The suit does have force feedback so that you’ll feel if you contact something, but by then it might be too late to prevent an accident.

GIF: Evan Ackerman/IEEE Spectrum With a pair of 12 lb 500 watt-hour battery packs, the exoskeleton can operate for over 2 hours during normal use. Energy Efficiency and Reliability

Efficiency might not seem like a big deal for an exoskeleton like this, but what Sarcos has managed is very impressive. The Guardian XO uses about 500 watts while fully operational—that is, while carrying 160 lbs and walking at 3 mph. To put that in context, SRI’s DURUS robot, which was designed specifically for efficiency (and is significantly smaller and lighter than the Guardian XO), used 350 watts while just walking. “That’s really one of our key innovations,” says Sarcos COO Chris Beaufait. “There aren’t many robots in the world that are as efficient as what we’re doing.” These innovations come in the form of energy recovery mechanisms, reductions in the number of individual computers on-board, and getting everything as tightly integrated as possible. With a pair of 12 lb 500 watt-hour battery packs, the exoskeleton can operate for over 2 hours during normal use, and Sarcos expects to improve the efficiency from 500 watts to 425 watts or better by January.

Since the Guardian XO is a commercial product, it has to be reliable enough to be a practical tool that’s cost effective to use. “The difference between being an R&D shop that can prove a concept versus making a commercially viable product that’s robust—it takes an entirely different skill set and mind set,” Wolff, the CEO, told us. “That’s been a challenge. I think it’s the biggest challenge that robotics companies have, and we’ve put a lot of blood, sweat, and tears into that.”

Wolff says that future XO versions (not the alpha model that will ship in January) will be able to walk outdoors over challenging terrain, through a foot of mud, and in the rain or snow. It will be able to go up and down stairs, although they’re currently working on making sure that this will be safe. The expectation, Wolff tells us, is that there won’t be much ongoing service or maintenance required for the exo’s customers. We’re not sure we share Sarcos’ confidence yet—this is a complex system that’s going to be used by non-engineers in semi and unstructured environments. A lot of unexpected scenarios can happen, and until they do, we won’t know for sure how well the Guardian XO will stand up to real-world use.

Guardian XO Applications

The Guardian XO has been designed to target some specific (but also very common) types of jobs that require humans to repetitively lift heavy things. These jobs are generally not automatable, or at least not automatable in a way that’s cost effective—the skill of a human is required. These jobs are also labor intensive, which creates both short term and long term problems for human workers. Short term, acute injuries (like back injuries) lead to lost productivity. Long term, these injuries add up to serious medical problems for workers, many of whom can only function for between five and eight years before their bodies become permanently damaged.

Wolff believes that this is where there’s an opportunity for powered exoskeletons. Using the Guardian XO to offload the kinds of tasks that put strain on a worker’s body means that humans can work at a job longer without injury. And they can keep working at that same job as they age, since the exoskeleton takes jobs that used to be about strength and instead makes them about skill and experience.

Photo: Evan Ackerman/IEEE Spectrum Sarcos says that one worker in an exoskeleton can handle tasks that would otherwise take between 4 and 10 people.

Of course, the sad fact is that none of this stuff about worker health would matter all that much if companies couldn’t be convinced that exoskeletons could also save them money. Fortunately for workers, it’s an easy argument to make. Since the Guardian XO can lift 200 pounds, Wolff says that it can improve the productivity of its user by up to an order of magnitude: “Overall, we’re seeing across the board improved productivity of somewhere between 4 and 10 times in use cases that we’ve looked at. So what that means is, one worker in an exoskeleton can do the work of between 4 and 10 employees without any stress or strain on their body.”

On the 4x end of the scale, it’s just about being able to lift more, and for longer. OSHA recommends a maximum one person load of 51 pounds, a number that gets adjusted downwards if the object has to be lifted repetitively, held for a long time, or moved. The Guardian XO allows a worker to lift four times that, for hours, while walking at up to 3 mph. Things are a little more complicated on the 10x end of the scale, but you can imagine a single 200 pound object that requires an overhead crane plus several people to manage it. It’s not just about the extra people—it’s also about the extra time and infrastructure required, when a single worker in a Guardian XO could just pick up that same object and move it by themselves.

The obvious question at this point is whether introducing powered exoskeletons is going to put people out of work. Wolff insists that is not the reality of the industry right now, since the real problem is finding qualified workers to hire in the first place. “None of our customers are talking about firing people,” Wolff says. “All of them are talking about simply not being able to produce enough of their products or services to keep their customers happy.” It should keep workers happy as well. Wolff tells us that they’ve had “enthusiastic responses” from workers who’ve tried the Guardian XO out, with their only concern being whether the exoskeleton can be adjusted to fit folks of different shapes and sizes. While initial units will be adjustable for people ranging in height from 5’4” to 6’, by next year, Sarcos promises that they’ll be adjustable enough to cover 90 percent of the American workforce. 

Image: Sarcos A rendering of how the Guardian XO will look with fairings applied. Cost and Availability

“We could not have made this an economically viable product three years ago,” Wolff says. “The size, power, weight, and cost of all of the components that we use—all of that has now gotten to a point where this is commercially feasible.” What that means, for Sarcos and the companies that they’re partnering with, is that each exoskeleton costs about $100,000 per year. The alpha units will be going to companies that can afford at least 10 of them at once, and Sarcos will send a dedicated engineer along with each batch. The Guardian XO is being sold as a service rather than a product—at least for now, it’s more of a rental with dedicated customer support. “The goal is this has to be stupid simple to manage and use,” says Wolff, adding that Sarcos expects to learn a lot over the next few months once the exoskeletons start being deployed. Commercial versions should ship later in 2020.

I made sure to ask Wolff when I might be able to rent one of these things from my local hardware store for the next time I have to move, but disappointingly, he doesn’t see that happening anytime soon. Sarcos still has a lot to learn about how to make a business out of exoskeletons, and they’d rather keep expectations realistic than promise anyone an Iron Man suit. It’s too late for me, though—I’ve seen what the Guardian XO can do. And I want one.

[ Sarcos ]

One year ago, for IEEE Spectrum’s special report on the Top Tech for 2019, Sarcos Robotics promised that by the end of the year they’d be ready to ship a powered exoskeleton that would be the future of industrial work. And late last month, Sarcos invited us to Salt Lake City, Utah, to see what that future looks like.

Sarcos has been developing powered exoskeletons and the robotic technologies that make them possible for decades, and the lobby of the company’s headquarters is a resting place for concepts and prototype hardware that’s been abandoned along the way. But now, Sarcos is ready to unveil the prototype of the Guardian XO, a strength-multiplying exoskeleton that’s about to begin shipping.

As our introductory briefing concludes, Sarcos CEO Ben Wolff is visibly excited to be able to show off what they’ve been working on in their lab. “If you were to ask the question, What does 30 years and $300 million look like,” Wolff tells us, “you're going to see it downstairs.”

This is what we see downstairs:

GIF: Evan Ackerman/IEEE Spectrum Guardian XO operator Fletcher Garrison demonstrates the company’s exosuit by lifting a 125-pound payload. Sarcos says this task usually requires three people. How the Guardian XO Works

The Sarcos Guardian XO is a 24-degrees-of-freedom full-body robotic exoskeleton. While wearing it, a human can lift  200 pounds (90 kilograms) while feeling like they’re lifting just 10 lbs (4.5 kg). The Guardian XO is fully electrical and untethered with a runtime of 2 hours, and hot-swappable battery packs can keep it going for a full work day. It takes seconds to put on and take off, and Sarcos says new users can be trained to use the system in minutes. One Guardian XO costs $100,000 per year to rent, and the company will be shipping its first batch of alpha units to customers (including both heavy industry and the U.S. military) in January.

Photo: Evan Ackerman/IEEE Spectrum The prototype that Sarcos demonstrated had all of the functionality of the version that will ship in January, but latter models will include plastic fairings over the suit as well as quick-change end-effectors.

In a practical sense, the Guardian XO is a humanoid robot that uses a real human as its command and control system. As companies of all kinds look towards increasing efficiency through automation, Sarcos believes that the most effective solution is a direct combination of humans and machines, enhancing the intelligence and judgement of humans with the strength and endurance of robots. (Investors in the company include Caterpillar, GE Ventures, Microsoft, and Schlumberger.)

The first thing to understand about the Guardian XO is that like a humanoid robot, it’s self-supporting. Since it has its own legs and feet, the 150 lb weight of the suit (and whatever it’s carrying) bypasses its user and is transferred directly into the ground. You don’t strap the robot to you—you strap yourself to the robot, a process that takes less than a minute. So although it looks heavy and bulky (and it is definitely both of those things), at least the weight of the system isn’t something that the user experiences directly. You can see how that works by watching Guardian XO operator Fletcher Garrison lifting all kinds of payloads in the video below.

Hands On With the Guardian XO

When Sarcos reached out and asked if we wanted to come to Salt Lake City to try out the XO, we immediately said yes (disclosure: Sarcos covered our costs to attend a media event last month). But we were disappointed when, in the end, we were only allowed to try out a one-armed version of the exoskeleton. I even offered to sign additional waivers but, alas, the company wouldn’t let me into the full suit. So my experience with the exo was pretty limited—a hands-on, literally, of a single XO arm.

Photo: Evan Ackerman/IEEE Spectrum That’s me trying out the one-arm XO system. It’s not quite like the full-body suit, but Sarcos still required me to sign a “waiver of liability, assumption of risk, and indemnity agreement.”

Still, it was an amazing sensation. The arm I tested, which Sarcos says uses the same control system as the full-body suit, was incredibly easy to operate. In terms of control, all the exo tries to do is get out of the way of your limbs: It uses force sensors to detect every motion that you make, and then moves its own limbs in parallel, smoothly matching your body with its own hardware. If you take a step, it takes a step with you. If you swing your arm back and forth, it swings its arm back and forth in the same way, right next to yours. There’s no discernible lag to this process, and it’s so intuitive that Sarcos says most people take just a minute or two to get comfortable using the system, and just an hour or two to be comfortable doing work in it.

The Guardian XO can augment the strength of the user all the way up to making a 200-pound load feel like it weighs zero pounds. Typically, this is not how the exoskeleton works, though, since it can be disconcerting to be lifting something heavy and not feel like you’re lifting anything at all. It’s better to think of the exo as a tool that makes you stronger rather than a tool that makes objects weightless, especially since you still have to deal with inertia. Remember, even if something has no apparent weight (either because you’re in space or because you’re holding it with a powered exoskeleton), it still has mass, which you have to be aware of when trying to move it or stop it from moving. The amount of help that the exo gives you is easy to adjust; it’s got a graphical control panel on the left wrist.

GIF: Evan Ackerman/IEEE Spectrum This ammo crate weighs 110 pounds, but the exoskeleton makes it feel like each arm is lifting just 6 pounds. The Guardian XO is designed for loads of up to 200 lbs. How Safe Is the Exoskeleton?

With a robotic system this powerful (XO has a peak torque of about 4000 inch-pounds, or 450 newton-meters), Sarcos made safety a top priority. For example, to move the exo’s arms, your hands need to be holding down triggers. If you let go of the triggers (for whatever reason), the arms will lock in place, which has the added benefit of letting the exo hold stuff up for you while you, say, check your phone. All of the joints are speed limited, meaning that you can’t throw a punch with the exo—they told me this during my demo, so of course I tried it, and the joints locked themselves as soon as I exceeded their safety threshold. If the system loses power for any reason, current shunts back through the motors, bringing them down gradually rather than abruptly. And by design the joints are not capable of exceeding a human range of motion, which means that the exoskeleton can’t bend or twist in a way that would injure you. Interestingly, the Guardian XO’s joint speeds are easily fast enough to allow you to run, although that’s been limited for safety reasons as well.

We asked about whether falling down was much of a risk, but it turns out that having a human in the loop for control makes that problem much simpler. Sarcos hasn’t had to program the Guardian XO to balance itself, because the human inside does all of that naturally. Having someone try to push you over while you’re in the exoskeleton is no different than having someone try to push you over while you’re out of it, because you’ll keep your own balance in either case. If you do end up falling over, Sarcos claims that the exoskeleton is designed as a roll cage, so odds are you’ll be fine, although it’s not clear how easy it would be to get out of it afterwards (or get it off of you).

More of a concern is how the XO will operate around other people. While its mass and bulk may not make all that much of a different to the user, it seems like working collaboratively could be a problem, as could working in small spaces or around anything fragile. The suit does have force feedback so that you’ll feel if you contact something, but by then it might be too late to prevent an accident.

GIF: Evan Ackerman/IEEE Spectrum With a pair of 12 lb 500 watt-hour battery packs, the exoskeleton can operate for over 2 hours during normal use. Energy Efficiency and Reliability

Efficiency might not seem like a big deal for an exoskeleton like this, but what Sarcos has managed is very impressive. The Guardian XO uses about 500 watts while fully operational—that is, while carrying 160 lbs and walking at 3 mph. To put that in context, SRI’s DURUS robot, which was designed specifically for efficiency (and is significantly smaller and lighter than the Guardian XO), used 350 watts while just walking. “That’s really one of our key innovations,” says Sarcos COO Chris Beaufait. “There aren’t many robots in the world that are as efficient as what we’re doing.” These innovations come in the form of energy recovery mechanisms, reductions in the number of individual computers on-board, and getting everything as tightly integrated as possible. With a pair of 12 lb 500 watt-hour battery packs, the exoskeleton can operate for over 2 hours during normal use, and Sarcos expects to improve the efficiency from 500 watts to 425 watts or better by January.

Since the Guardian XO is a commercial product, it has to be reliable enough to be a practical tool that’s cost effective to use. “The difference between being an R&D shop that can prove a concept versus making a commercially viable product that’s robust—it takes an entirely different skill set and mind set,” Wolff, the CEO, told us. “That’s been a challenge. I think it’s the biggest challenge that robotics companies have, and we’ve put a lot of blood, sweat, and tears into that.”

Wolff says that future XO versions (not the alpha model that will ship in January) will be able to walk outdoors over challenging terrain, through a foot of mud, and in the rain or snow. It will be able to go up and down stairs, although they’re currently working on making sure that this will be safe. The expectation, Wolff tells us, is that there won’t be much ongoing service or maintenance required for the exo’s customers. We’re not sure we share Sarcos’ confidence yet—this is a complex system that’s going to be used by non-engineers in semi and unstructured environments. A lot of unexpected scenarios can happen, and until they do, we won’t know for sure how well the Guardian XO will stand up to real-world use.

Guardian XO Applications

The Guardian XO has been designed to target some specific (but also very common) types of jobs that require humans to repetitively lift heavy things. These jobs are generally not automatable, or at least not automatable in a way that’s cost effective—the skill of a human is required. These jobs are also labor intensive, which creates both short term and long term problems for human workers. Short term, acute injuries (like back injuries) lead to lost productivity. Long term, these injuries add up to serious medical problems for workers, many of whom can only function for between five and eight years before their bodies become permanently damaged.

Wolff believes that this is where there’s an opportunity for powered exoskeletons. Using the Guardian XO to offload the kinds of tasks that put strain on a worker’s body means that humans can work at a job longer without injury. And they can keep working at that same job as they age, since the exoskeleton takes jobs that used to be about strength and instead makes them about skill and experience.

Photo: Evan Ackerman/IEEE Spectrum Sarcos says that one worker in an exoskeleton can handle tasks that would otherwise take between 4 and 10 people.

Of course, the sad fact is that none of this stuff about worker health would matter all that much if companies couldn’t be convinced that exoskeletons could also save them money. Fortunately for workers, it’s an easy argument to make. Since the Guardian XO can lift 200 pounds, Wolff says that it can improve the productivity of its user by up to an order of magnitude: “Overall, we’re seeing across the board improved productivity of somewhere between 4 and 10 times in use cases that we’ve looked at. So what that means is, one worker in an exoskeleton can do the work of between 4 and 10 employees without any stress or strain on their body.”

On the 4x end of the scale, it’s just about being able to lift more, and for longer. OSHA recommends a maximum one person load of 51 pounds, a number that gets adjusted downwards if the object has to be lifted repetitively, held for a long time, or moved. The Guardian XO allows a worker to lift four times that, for hours, while walking at up to 3 mph. Things are a little more complicated on the 10x end of the scale, but you can imagine a single 200 pound object that requires an overhead crane plus several people to manage it. It’s not just about the extra people—it’s also about the extra time and infrastructure required, when a single worker in a Guardian XO could just pick up that same object and move it by themselves.

The obvious question at this point is whether introducing powered exoskeletons is going to put people out of work. Wolff insists that is not the reality of the industry right now, since the real problem is finding qualified workers to hire in the first place. “None of our customers are talking about firing people,” Wolff says. “All of them are talking about simply not being able to produce enough of their products or services to keep their customers happy.” It should keep workers happy as well. Wolff tells us that they’ve had “enthusiastic responses” from workers who’ve tried the Guardian XO out, with their only concern being whether the exoskeleton can be adjusted to fit folks of different shapes and sizes. While initial units will be adjustable for people ranging in height from 5’4” to 6’, by next year, Sarcos promises that they’ll be adjustable enough to cover 90 percent of the American workforce. 

Image: Sarcos A rendering of how the Guardian XO will look with fairings applied. Cost and Availability

“We could not have made this an economically viable product three years ago,” Wolff says. “The size, power, weight, and cost of all of the components that we use—all of that has now gotten to a point where this is commercially feasible.” What that means, for Sarcos and the companies that they’re partnering with, is that each exoskeleton costs about $100,000 per year. The alpha units will be going to companies that can afford at least 10 of them at once, and Sarcos will send a dedicated engineer along with each batch. The Guardian XO is being sold as a service rather than a product—at least for now, it’s more of a rental with dedicated customer support. “The goal is this has to be stupid simple to manage and use,” says Wolff, adding that Sarcos expects to learn a lot over the next few months once the exoskeletons start being deployed. Commercial versions should ship later in 2020.

I made sure to ask Wolff when I might be able to rent one of these things from my local hardware store for the next time I have to move, but disappointingly, he doesn’t see that happening anytime soon. Sarcos still has a lot to learn about how to make a business out of exoskeletons, and they’d rather keep expectations realistic than promise anyone an Iron Man suit. It’s too late for me, though—I’ve seen what the Guardian XO can do. And I want one.

[ Sarcos ]

This study focuses on category formation for individual agents and the dynamics of symbol emergence in a multi-agent system through semiotic communication. In this study, the semiotic communication refers to exchanging signs composed of the signifier (i.e., words) and the signified (i.e., categories). We define the generation and interpretation of signs associated with the categories formed through the agent's own sensory experience or by exchanging signs with other agents as basic functions of the semiotic communication. From the viewpoint of language evolution and symbol emergence, organization of a symbol system in a multi-agent system (i.e., agent society) is considered as a bottom-up and dynamic process, where individual agents share the meaning of signs and categorize sensory experience. A constructive computational model can explain the mutual dependency of the two processes and has mathematical support that guarantees a symbol system's emergence and sharing within the multi-agent system. In this paper, we describe a new computational model that represents symbol emergence in a two-agent system based on a probabilistic generative model for multimodal categorization. It models semiotic communication via a probabilistic rejection based on the receiver's own belief. We have found that the dynamics by which cognitively independent agents create a symbol system through their semiotic communication can be regarded as the inference process of a hidden variable in an interpersonal multimodal categorizer, i.e., the complete system can be regarded as a single agent performing multimodal categorization using the sensors of all agents, if we define the rejection probability based on the Metropolis-Hastings algorithm. The validity of the proposed model and algorithm for symbol emergence, i.e., forming and sharing signs and categories, is also verified in an experiment with two agents observing daily objects in the real-world environment. In the experiment, we compared three communication algorithms: no communication, no rejection, and the proposed algorithm. The experimental results demonstrate that our model reproduces the phenomena of symbol emergence, which does not require a teacher who would know a pre-existing symbol system. Instead, the multi-agent system can form and use a symbol system without having pre-existing categories.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

Robotic Arena – January 25, 2020 – Wrocław, Poland DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA

Let us know if you have suggestions for next week, and enjoy today’s videos.

In case you somehow missed the massive Skydio 2 review we posted earlier this week, the first batches of the drone are now shipping. Each drone gets a lot of attention before it goes out the door, and here’s a behind-the-scenes clip of the process.

[ Skydio ]

Sphero RVR is one of the 15 robots on our robot gift guide this year. Here’s a new video Sphero just released showing some of the things you can do with the robot.

[ RVR ]

NimbRo-OP2 has some impressive recovery skills from the obligatory research-motivated robot abuse.

NimbRo ]

Teams seeking to qualify for the Virtual Urban Circuit of the Subterranean Challenge can access practice worlds to test their approaches prior to submitting solutions for the competition. This video previews three of the practice environments.

[ DARPA SubT ]

Stretchable skin-like robots that can be rolled up and put in your pocket have been developed by a University of Bristol team using a new way of embedding artificial muscles and electrical adhesion into soft materials.

[ Bristol ]

Happy Holidays from ABB!

Helping New York celebrate the festive season, twelve ABB robots are interacting with visitors to Bloomingdale’s iconic holiday celebration at their 59th Street flagship store. ABB’s robots are the main attraction in three of Bloomingdale’s twelve-holiday window displays at Lexington and Third Avenue, as ABB demonstrates the potential for its robotics and automation technology to revolutionize visual merchandising and make the retail experience more dynamic and whimsical.

[ ABB ]

We introduce pelican eel–inspired dual-morphing architectures that embody quasi-sequential behaviors of origami unfolding and skin stretching in response to fluid pressure. In the proposed system, fluid paths were enclosed and guided by a set of entirely stretchable origami units that imitate the morphing principle of the pelican eel’s stretchable and foldable frames. This geometric and elastomeric design of fluid networks, in which fluid pressure acts in the direction that the whole body deploys first, resulted in a quasi-sequential dual-morphing response. To verify the effectiveness of our design rule, we built an artificial creature mimicking a pelican eel and reproduced biomimetic dual-morphing behavior.

And here’s a real pelican eel:

[ Science Robotics ]

Delft Dynamics’ updated anti-drone system involves a tether, mid-air net gun, and even a parachute.

[ Delft Dynamics ]

Teleoperation is a great way of helping robots with complex tasks, especially if you can do it through motion capture. But what if you’re teleoperating a non-anthropomorphic robot? Columbia’s ROAM Lab is working on it.

[ Paper ] via [ ROAM Lab ]

I don’t know how I missed this video last year because it’s got a steely robot hand squeezing a cute lil’ chick.

[ MotionLib ] via [ RobotStart ]

In this video we present results of a trajectory generation method for autonomous overtaking of unexpected obstacles in a dynamic urban environment. In these settings, blind spots can arise from perception limitations. For example when overtaking unexpected objects on the vehicle’s ego lane on a two-way street. In this case, a human driver would first make sure that the opposite lane is free and that there is enough room to successfully execute the maneuver, and then it would cut into the opposite lane in order to execute the maneuver successfully. We consider the practical problem of autonomous overtaking when the coverage of the perception system is impaired due to occlusion.

[ Paper ]

New weirdness from Toio!

[ Toio ]

Palo Alto City Library won a technology innovation award! Watch to see how Senior Librarian Dan Lou is using Misty to enhance their technology programs to inspire and educate customers.

[ Misty Robotics ]

We consider the problem of reorienting a rigid object with arbitrary known shape on a table using a two-finger pinch gripper. Reorienting problem is challenging because of its non-smoothness and high dimensionality. In this work, we focus on solving reorienting using pivoting, in which we allow the grasped object to rotate between fingers. Pivoting decouples the gripper rotation from the object motion, making it possible to reorient an object under strict robot workspace constraints.

[ CMU ]

How can a mobile robot be a good pedestrian without bumping into you on the sidewalk? It must be hard for a robot to navigate in crowded environments since the flow of traffic follows implied social rules. But researchers from MIT developed an algorithm that teaches mobile robots to maneuver in crowds of people, respecting their natural behaviour.

[ Roboy Research Reviews ]

What happens when humans and robots make art together? In this awe-inspiring talk, artist Sougwen Chung shows how she "taught" her artistic style to a machine -- and shares the results of their collaboration after making an unexpected discovery: robots make mistakes, too. "Part of the beauty of human and machine systems is their inherent, shared fallibility," she says.

[ TED ]

Last month at the Cooper Union in New York City, IEEE TechEthics hosted a public panel session on the facts and misperceptions of autonomous vehicles, part of the IEEE TechEthics Conversations Series. The speakers were: Jason Borenstein from Georgia Tech; Missy Cummings from Duke University; Jack Pokrzywa from SAE; and Heather M. Roff from Johns Hopkins Applied Physics Laboratory. The panel was moderated by Mark A. Vasquez, program manager for IEEE TechEthics.


[ IEEE TechEthics ]

Two videos this week from Lex Fridman’s AI podcast: Noam Chomsky, and Whitney Cummings.

[ AI Podcast ]

This week’s CMU RI Seminar comes from Jeff Clune at the University of Wyoming, on “Improving Robot and Deep Reinforcement Learning via Quality Diversity and Open-Ended Algorithms.”

Quality Diversity (QD) algorithms are those that seek to produce a diverse set of high-performing solutions to problems. I will describe them and a number of their positive attributes. I will then summarize our Nature paper on how they, when combined with Bayesian Optimization, produce a learning algorithm that enables robots, after being damaged, to adapt in 1-2 minutes in order to continue performing their mission, yielding state-of-the-art robot damage recovery. I will next describe our QD-based Go-Explore algorithm, which dramatically improves the ability of deep reinforcement learning algorithms to solve previously unsolvable problems wherein reward signals are sparse, meaning that intelligent exploration is required. Go-Explore solves Montezuma’s Revenge, considered by many to be a major AI research challenge. Finally, I will motivate research into open-ended algorithms, which seek to innovate endlessly, and introduce our POET algorithm, which generates its own training challenges while learning to solve them, automatically creating a curricula for robots to learn an expanding set of diverse skills. POET creates and solves challenges that are unsolvable with traditional deep reinforcement learning techniques.

[ CMU RI ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

Robotic Arena – January 25, 2020 – Wrocław, Poland DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA

Let us know if you have suggestions for next week, and enjoy today’s videos.

In case you somehow missed the massive Skydio 2 review we posted earlier this week, the first batches of the drone are now shipping. Each drone gets a lot of attention before it goes out the door, and here’s a behind-the-scenes clip of the process.

[ Skydio ]

Sphero RVR is one of the 15 robots on our robot gift guide this year. Here’s a new video Sphero just released showing some of the things you can do with the robot.

[ RVR ]

NimbRo-OP2 has some impressive recovery skills from the obligatory research-motivated robot abuse.

NimbRo ]

Teams seeking to qualify for the Virtual Urban Circuit of the Subterranean Challenge can access practice worlds to test their approaches prior to submitting solutions for the competition. This video previews three of the practice environments.

[ DARPA SubT ]

Stretchable skin-like robots that can be rolled up and put in your pocket have been developed by a University of Bristol team using a new way of embedding artificial muscles and electrical adhesion into soft materials.

[ Bristol ]

Happy Holidays from ABB!

Helping New York celebrate the festive season, twelve ABB robots are interacting with visitors to Bloomingdale’s iconic holiday celebration at their 59th Street flagship store. ABB’s robots are the main attraction in three of Bloomingdale’s twelve-holiday window displays at Lexington and Third Avenue, as ABB demonstrates the potential for its robotics and automation technology to revolutionize visual merchandising and make the retail experience more dynamic and whimsical.

[ ABB ]

We introduce pelican eel–inspired dual-morphing architectures that embody quasi-sequential behaviors of origami unfolding and skin stretching in response to fluid pressure. In the proposed system, fluid paths were enclosed and guided by a set of entirely stretchable origami units that imitate the morphing principle of the pelican eel’s stretchable and foldable frames. This geometric and elastomeric design of fluid networks, in which fluid pressure acts in the direction that the whole body deploys first, resulted in a quasi-sequential dual-morphing response. To verify the effectiveness of our design rule, we built an artificial creature mimicking a pelican eel and reproduced biomimetic dual-morphing behavior.

And here’s a real pelican eel:

[ Science Robotics ]

Delft Dynamics’ updated anti-drone system involves a tether, mid-air net gun, and even a parachute.

[ Delft Dynamics ]

Teleoperation is a great way of helping robots with complex tasks, especially if you can do it through motion capture. But what if you’re teleoperating a non-anthropomorphic robot? Columbia’s ROAM Lab is working on it.

[ Paper ] via [ ROAM Lab ]

I don’t know how I missed this video last year because it’s got a steely robot hand squeezing a cute lil’ chick.

[ MotionLib ] via [ RobotStart ]

In this video we present results of a trajectory generation method for autonomous overtaking of unexpected obstacles in a dynamic urban environment. In these settings, blind spots can arise from perception limitations. For example when overtaking unexpected objects on the vehicle’s ego lane on a two-way street. In this case, a human driver would first make sure that the opposite lane is free and that there is enough room to successfully execute the maneuver, and then it would cut into the opposite lane in order to execute the maneuver successfully. We consider the practical problem of autonomous overtaking when the coverage of the perception system is impaired due to occlusion.

[ Paper ]

New weirdness from Toio!

[ Toio ]

Palo Alto City Library won a technology innovation award! Watch to see how Senior Librarian Dan Lou is using Misty to enhance their technology programs to inspire and educate customers.

[ Misty Robotics ]

We consider the problem of reorienting a rigid object with arbitrary known shape on a table using a two-finger pinch gripper. Reorienting problem is challenging because of its non-smoothness and high dimensionality. In this work, we focus on solving reorienting using pivoting, in which we allow the grasped object to rotate between fingers. Pivoting decouples the gripper rotation from the object motion, making it possible to reorient an object under strict robot workspace constraints.

[ CMU ]

How can a mobile robot be a good pedestrian without bumping into you on the sidewalk? It must be hard for a robot to navigate in crowded environments since the flow of traffic follows implied social rules. But researchers from MIT developed an algorithm that teaches mobile robots to maneuver in crowds of people, respecting their natural behaviour.

[ Roboy Research Reviews ]

What happens when humans and robots make art together? In this awe-inspiring talk, artist Sougwen Chung shows how she "taught" her artistic style to a machine -- and shares the results of their collaboration after making an unexpected discovery: robots make mistakes, too. "Part of the beauty of human and machine systems is their inherent, shared fallibility," she says.

[ TED ]

Last month at the Cooper Union in New York City, IEEE TechEthics hosted a public panel session on the facts and misperceptions of autonomous vehicles, part of the IEEE TechEthics Conversations Series. The speakers were: Jason Borenstein from Georgia Tech; Missy Cummings from Duke University; Jack Pokrzywa from SAE; and Heather M. Roff from Johns Hopkins Applied Physics Laboratory. The panel was moderated by Mark A. Vasquez, program manager for IEEE TechEthics.


[ IEEE TechEthics ]

Two videos this week from Lex Fridman’s AI podcast: Noam Chomsky, and Whitney Cummings.

[ AI Podcast ]

This week’s CMU RI Seminar comes from Jeff Clune at the University of Wyoming, on “Improving Robot and Deep Reinforcement Learning via Quality Diversity and Open-Ended Algorithms.”

Quality Diversity (QD) algorithms are those that seek to produce a diverse set of high-performing solutions to problems. I will describe them and a number of their positive attributes. I will then summarize our Nature paper on how they, when combined with Bayesian Optimization, produce a learning algorithm that enables robots, after being damaged, to adapt in 1-2 minutes in order to continue performing their mission, yielding state-of-the-art robot damage recovery. I will next describe our QD-based Go-Explore algorithm, which dramatically improves the ability of deep reinforcement learning algorithms to solve previously unsolvable problems wherein reward signals are sparse, meaning that intelligent exploration is required. Go-Explore solves Montezuma’s Revenge, considered by many to be a major AI research challenge. Finally, I will motivate research into open-ended algorithms, which seek to innovate endlessly, and introduce our POET algorithm, which generates its own training challenges while learning to solve them, automatically creating a curricula for robots to learn an expanding set of diverse skills. POET creates and solves challenges that are unsolvable with traditional deep reinforcement learning techniques.

[ CMU RI ]

When mobile manipulators eventually make it into our homes, self-repair is going to be a very important function. Hopefully, these robots will be durable enough that they won’t need to be repaired very often, but from time to time they’ll almost certainly need minor maintenance. At Humanoids 2019 in Toronto, researchers from the University of Tokyo showed how they taught a PR2 to perform simple repairs on itself by tightening its own screws. And using that skill, the robot was also able to augment itself, adding accessories like hooks to help it carry more stuff. Clever robot!

To keep things simple, the researchers provided the robot with CAD data that tells it exactly where all of its screws are. 

At the moment, the robot can’t directly detect on its own whether a particular screw needs tightening, although it can tell if its physical pose doesn’t match its digital model, which suggests that something has gone wonky. It can also check its screws autonomously from time to time, or rely on a human physically pointing out that it has a screw loose, using the human’s finger location to identify which screw it is. Another challenge is that most robots, like most humans, are limited in the areas on themselves that they can comfortably reach. So to tighten up everything, they might have to find themselves a robot friend to help, just like humans help each other put on sunblock.

The actual tightening is either super easy or quite complicated, depending on the location and orientation of the screw. If the robot is lucky, it can just use its continuous wrist rotation for tightening, but if a screw is located in a tight position that requires an Allen wrench, the robot has to regrasp the tool over and over as it incrementally tightens the screw. 

Image: University of Tokyo In one experiment, the researchers taught a PR2 robot to attach a hook to one of its shoulders. The robot uses one hand to grasp the hook and another hand to grasp a screwdriver. The researchers tested the hook by hanging a tote bag on it.

The other neat trick that a robot can do once it can tighten screws on its own body is to add new bits of hardware to itself. PR2 was thoughtfully designed with mounting points on its shoulders (or maybe technically its neck) and head, and it turns out that it can reach these points with its manipulators, allowing to modify itself, as the researchers explain:

When PR2 wants to have a lot of things, the only two hands are not enough to realize that. So we let PR2 to use a bag the same as we put it on our shoulder. PR2 started attaching the hook whose pose is calculated with self CAD data with a driver on his shoulder in order to put a bag on his shoulder. PR2 finished attaching the hook, and the people put a lot of cans in a tote bag and put it on PR2’s shoulder.

“Self-Repair and Self-Extension by Tightening Screws based on Precise Calculation of Screw Pose of Self-Body with CAD Data and Graph Search with Regrasping a Driver,” by Takayuki Murooka, Kei Okada, and Masayuki Inaba from the University of Tokyo, was presented at Humanoids 2019 in Toronto, Canada.

When mobile manipulators eventually make it into our homes, self-repair is going to be a very important function. Hopefully, these robots will be durable enough that they won’t need to be repaired very often, but from time to time they’ll almost certainly need minor maintenance. At Humanoids 2019 in Toronto, researchers from the University of Tokyo showed how they taught a PR2 to perform simple repairs on itself by tightening its own screws. And using that skill, the robot was also able to augment itself, adding accessories like hooks to help it carry more stuff. Clever robot!

To keep things simple, the researchers provided the robot with CAD data that tells it exactly where all of its screws are. 

At the moment, the robot can’t directly detect on its own whether a particular screw needs tightening, although it can tell if its physical pose doesn’t match its digital model, which suggests that something has gone wonky. It can also check its screws autonomously from time to time, or rely on a human physically pointing out that it has a screw loose, using the human’s finger location to identify which screw it is. Another challenge is that most robots, like most humans, are limited in the areas on themselves that they can comfortably reach. So to tighten up everything, they might have to find themselves a robot friend to help, just like humans help each other put on sunblock.

The actual tightening is either super easy or quite complicated, depending on the location and orientation of the screw. If the robot is lucky, it can just use its continuous wrist rotation for tightening, but if a screw is located in a tight position that requires an Allen wrench, the robot has to regrasp the tool over and over as it incrementally tightens the screw. 

Image: University of Tokyo In one experiment, the researchers taught a PR2 robot to attach a hook to one of its shoulders. The robot uses one hand to grasp the hook and another hand to grasp a screwdriver. The researchers tested the hook by hanging a tote bag on it.

The other neat trick that a robot can do once it can tighten screws on its own body is to add new bits of hardware to itself. PR2 was thoughtfully designed with mounting points on its shoulders (or maybe technically its neck) and head, and it turns out that it can reach these points with its manipulators, allowing to modify itself, as the researchers explain:

When PR2 wants to have a lot of things, the only two hands are not enough to realize that. So we let PR2 to use a bag the same as we put it on our shoulder. PR2 started attaching the hook whose pose is calculated with self CAD data with a driver on his shoulder in order to put a bag on his shoulder. PR2 finished attaching the hook, and the people put a lot of cans in a tote bag and put it on PR2’s shoulder.

“Self-Repair and Self-Extension by Tightening Screws based on Precise Calculation of Screw Pose of Self-Body with CAD Data and Graph Search with Regrasping a Driver,” by Takayuki Murooka, Kei Okada, and Masayuki Inaba from the University of Tokyo, was presented at Humanoids 2019 in Toronto, Canada.

Pages