IEEE Spectrum Automation

IEEE Spectrum Automaton blog recent content
Subscribe to IEEE Spectrum Automation feed

With the DARPA Subterranean Challenge Urban Circuit kicking off on Thursday, we made sure to have a chat in advance with Dr. Timothy Chung, DARPA SubT program manager. We last spoke with Tim nearly a year ago, just after SubT was announced, to get his perspective on the Subterranean Challenge in general, and we took the opportunity in this interview to ask about how DARPA felt about the Tunnel Circuit, and what we have to look forward to in the Urban Circuit.

For more details about the SubT Urban Circuit, make sure to check out our course preview post, and check back tomorrow for a Q&A with the systems track teams.

This interview has been edited for length and clarity.

IEEE Spectrum: What turned out to be the biggest challenge for teams during the Tunnel Circuit?

Tim Chung: Where to begin? And I say that from the perspective of every obstacle being a good learning opportunity for the teams! For example, mobility was a huge challenge for the teams—going from cement and gravel into areas with rails that were problematic, to muddy places with big ruts… And that diversity was just on one level of the mine. But I think what we saw from the robots, at least from the ground vehicles, was the ability to deal with the unexpected, at least as far as the terrain was concerned.

As you saw, there were some different platforms that came up with different ways of solving the terrain challenges as well, including wheels on legs, and that was another great opportunity to see under what conditions wheels make sense, or legs make sense, or treads make sense. Again, that was only with the ground vehicles, and there were a lot of other ways to address the terrain, as with air vehicles, but they had their own sets of challenges. Mobility was a good reminder that the real world doesn’t have well-defined terrain parameters.

What impressed you the most during the Tunnel Circuit?

First off, I think the energy and enthusiasm that the teams brought. As a roboticist myself, I remember getting sad at times when my robot would break, but the fact that the teams were so enthusiastic about learning in this challenging environment was a really great takeaway. From a technology point of view, I was very impressed with how teams were able to deal with a lot of the uncertainty of the course itself, in terms of—I won’t call them hacks, but overnight fixes. The ability to say, “We saw that this other team had this innovation, let’s capitalize on that and see if we can retrofit our system to be able to do that as well.” 

“Increasing the autonomous capability of these robots can overcome the limitations of some other technology areas. An example of that is communications—we know communications are really hard. The teams definitely experienced this, and they were able to develop autonomous approaches to address or mitigate some of the communications deficiencies”

It turned out that both speed and agility are important, but so is deliberate decision making. I think what that speaks to is that increasing the autonomous capability of these robots can overcome the limitations of some other technology areas. An example of that is communications—we know communications are really hard. The teams definitely experienced this, and they were able to develop autonomous approaches to address or mitigate some of the communications deficiencies.

Can you share any feedback that you got from teams about the Tunnel Circuit?

The primary feedback that we’ve received has been very positive—teams have expressed that the courses were really, really hard, and hard in different ways than they anticipated. Essentially, that DARPA has set a very high bar. But teams are pleased that the bar is so high. I think what that says to me is that we’re satisfying the role that DARPA plays within the robotics community, of setting the bar really high while inspiring teams to reach it, but I also think that it shows how the community is thirsty for the opportunity to reach a new level of technology. From that perspective, the feedback was, “It was too hard for our robots, but something we still want to overcome.” And that’s great, because ultimately, DARPA’s role is to give the community a chance to surprise itself with how innovative it can be.

Did the results of the Tunnel Circuit cause you to recalibrate the difficulty of the Urban Circuit?

I can say pretty succinctly that the answer is no, in the sense that each of these environments are so wildly different that it’s difficult to say something like, “Tunnel was a difficulty 9, and Urban is going to be a 10.” And the type of challenge elements that we’re introducing in Urban that weren’t at Tunnel also make that comparison hard. The verticality with stairs, for example, will pose totally new challenges for the teams. So I’d say, we didn’t change the difficultly level because the environment just does it intrinsically for us.

In terms of expectations, we know that the teams have come to appreciate what DARPA wants to see out of the challenge, and so they’re better equipped from that perspective. We’ve adjusted by introducing new elements into the environment that will keep them on their toes for sure.

Can you highlight some of the fundamental differences between Tunnel environments and Urban environments?

The environments that are built for people tend not to have been designed with robots in mind, so having these types of systems—systems that can make an impact in everyday lives, and particularly in the first responder case we’re interested in—will necessarily mean that they’ll have to be able to operate in these challenging urban settings where we expect people to be.

Verticality is certainly one of the key distinctions that I think will be really exciting for the Urban Circuit. In Tunnel, the emphasis was not on verticality given the single level in the courses. What we’ve previewed for Urban is that there are sections of the courses where having the ability to navigate vertically will be very beneficial to teams.

Verticality includes not just the need to move in three dimensions, but the ability to perceive in three dimensions as well. For example, we place exit signs above eye level to help them stand out, and we look for indicators like handrails and signs. I think context plays a big role in these urban settings, which we take for granted as humans, but if you’re thinking about using these environmental cues to aid in localization and navigation, there may be some interesting advantages.

Are there other particular challenges for robots in urban environments?

I can give a couple of examples where there are some unique elements. Most of our construction materials tend to involve rebar, or metal in general, that are very bad for communications for sure but also for things like compasses. Another example is the regularity and the oftentimes featureless qualities of urban environments make it difficult for feature-matching technologies to help you with your localization. When one door looks like another door, or a wall looks like every other wall—humans may like that in our urban architecture, but for robots, the lack of discriminating features makes many localization approaches challenging. 

Why did you choose this specific urban environment, as opposed to something like a subway station?

We can certainly appreciate the subway scenario as one attribute of urban underground settings, but as we did research and worked with stakeholders, and also visited a number of various sites in urban centers, I think we learned that there’s just so much to the urban underground, so many types of environments, that to only use a subway wouldn’t have allowed us to implement many of the features that we were interested in for the SubT challenge. Going beyond that, I think the environment we found in Satsop is quite representative of many of the elements of the urban underground.

It’s also a place where we know first responders like the Seattle fire department conduct urban training. Being able to find a place that has value for first responders also contributed to our selection of the site, since it had that realistic feel for what they’re interested in and training for.

Can you talk a little bit about what DARPA’s position is on teams using tethers to communicate with their robots during the SubT Challenge?

[Laughs] What does DARPA have against tethers? I’d say, the idea here is that we’re interested in the capabilities that can be manifested throughout the challenge. While the specific implementation or approach may be of interest from a technology development perspective, it’s really about the capability. And so, if the best capability at present is to insert a robot that is tether-enabled, I think there’s merit there. The environment, however, is far more expansive than would allow for tethers to be the only solution. 

“If tethers showcase the ability to do additional breakthrough capabilities, then bring it! If there are ways in which breakthrough technologies allow us to work without tethers, and that opens up the spaces in which these types of systems can work, that’s another breakthrough that we’d be very excited to see”

Our perspective, which is to neither encourage nor discourage tethers, is partially in response to teams inquiring outright if tethers are permitted, and we wanted to say that we’re not trying to prescribe any solutions, because that’s not what the challenge is designed to do—we’re trying to be as open as possible. And it’s entirely possible that tethers will work in some cases, as they might have in tunnel, and it’s entirely possible that they won’t in other cases. If tethers showcase the ability to do additional breakthrough capabilities, then bring it! If there are ways in which breakthrough technologies allow us to work without tethers, and that opens up the spaces in which these types of systems can work, that’s another breakthrough that we’d be very excited to see.

Can you tell us what we can expect for the DARPA livestream of the Urban Circuit competition?

One of the things that we took away from the Tunnel Circuit was our desire to help narrate and explain and involve the viewing audience more. We intend to have both narration of the action and commentary where applicable. We’re opening the aperture because there are a lot of really interesting features that we’d love our viewing audience to be able to witness, and that includes robots traversing interesting features and elements within the course, so you can look forward to more of that.

You should be able to follow along with what’s going on in the course a fair bit better, and beyond what we saw in Tunnel for sure. We’re balancing of course the sensitivity of the competition, but there’s just so much robot excitement to be shared, and I’m excited that we’ll be able to do more of that for the Urban Circuit.

What are you personally most excited about for the Urban Circuit?

There’s just so many things— I’m super excited about how we set up the team garages at the Urban Circuit. It’ll be like pit row, in a way that really highlights how much I value the interactions between teams, it’ll be an opportunity to truly capitalize on having a high concentration of enthusiastic and ambitious roboticists in one area. 

And the teams are already amped up. They’ve gotten a taste of the difficulty level, they’ve gotten a taste of DARPA’s aspirational goals, they know when DARPA says “challenge” we mean “Challenge” with a capital C. From a technology perspective, I’m anticipating these teams are bringing their A++ game now, not only revising their strategy but building new robots, and that’ll be really exciting too.

[ DARPA SubT ]

The Urban Circuit of the DARPA Subterranean Challenge is the second of four robotics competitions that send teams of state-of-the-art robots into challenging underground environments in an attempt to seek out artifacts while creating detailed maps. Last August, the robots explored a man-made tunnel system in the NIOSH research mine near Pittsburgh, Pennsylvania. And starting this Thursday, the teams will be taking on the urban underground, at Satsop Business Park in Elma, Wash.

If you’re not familiar with the DARPA Subterranean Challenge, here’s an overview video from DARPA to get you caught up on the whole thing:

This post will be focused on a preview of the Urban Circuit, but if you’d like to learn more about SubT we’ve got plenty more coverage on the challenge itself, the teams that are involved, and the results of the first competition, the Tunnel Circuit.

The overall objective of the SubT Challenge is for each team of robots to spend 60 minutes exploring an underground (or simulated underground) course, searching for a variety of artifacts. You can read more about how the scoring works here, but the team of robots that’s able to find the most artifacts (and report back their exact location) in the least amount of time wins.

DARPA’s intention is that the Urban Circuit “will represent human-made urban environments such as municipal infrastructure and mass transit.” We’d sort of figured that they’d choose a subway station, mostly because they’ve been using subway station graphics on the SubT website. But DARPA has chosen Satsop Business Park (just to the west of Olympia, Wash.) as the location for the event, and it’s much more of an industrial-y looking place. Here’s an environment preview video:

Environments like these can be pretty grim for autonomous robots. There’s all kinds of stuff on the floor, lots of localization-unfriendly flat surfaces, dirt, water, stairs, ledges—it sucks if you’re a robot (or roboticist). But that’s the point, because the SubT challenge is DARPA-hard. That’s what the teams and their robots are preparing for, and I can’t imagine they’d want it any other way.

The SubT challenge is DARPA-hard. That’s what the teams and their robots are preparing for, and I can’t imagine they’d want it any other way

Something in particular to look out for in the Urban Circuit is verticality: Things like stairs, ladders, ledges, and holes in the ground and ceiling that need to be explored or traversed. The Tunnel Circuit didn’t really have that; there may have been one or two optional holes, but nothing that robots were required to deal with in order to make progress. We’re expecting that verticality will be a much more significant (i.e. mandatory) part of the Urban Circuit, which will definitely add some risk. And some drama!

A key thing to remember is that all teams are allowed to do is push their robots through the entry to the Urban Circuit course, and after that, they can’t touch them. Once the robots move out of communications range, they’re completely on their own, operating fully autonomously without any human input at all.

During the Tunnel Circuit, teams tried a variety of very creative ways to keep in touch with their robots, both to give them high-level instructions and to get back artifact coordinates to score points. We’re expecting even more autonomy for the Urban Circuit, as teams have refined their communications strategies over the past six months.

New to the Urban Circuit are two different artifacts: gas, and vent. They replace the drill and fire extinguisher artifacts from the Tunnel Circuit. Here’s what we know about the new artifacts:

The gas artifact is a CO2-emitting device used to simulate a range of hazardous air quality conditions, such as a gas leak, poor ventilation, or fumes and smoke. Finding this artifact represents identifying areas that would be hazardous for personnel, including areas where breathing apparatus may be necessary.

The vent artifact is a typical supply register commonly found in homes or work environments. Finding this artifact represents identifying potential areas with fresh air or an escape route to the surface.

Image: DARPA

The vent artifact can be found on a wall or ceiling at any height, and it’ll be heated to at least 30 °C above ambient temperature. The gas artifact is a little trickier; the gas itself will be kept at 2000 ppm in a room with an open door. There won’t be any visual indicators; robots will have to sense the gas, and precisely identify the room that it’s coming from by finding the center point of the entry door threshold (at floor level) of the room where the gas is detected.

Besides the new course and new artifacts, DARPA has made some updates to the SubT rules for the Urban Circuit. We’ve gone through and highlighted some of the most relevant ones:

The scale and complexity of competition courses is expected to vary across events and may vary from run to run due to configuration changes. The design of the Urban Circuit course is intended to assess the ability of teams to address challenging urban environments that include multiple levels, degraded terrain, and austere environmental conditions.

Each of the two courses are expected to include multiple levels, requiring teams to traverse stairs, ramps, or vertical shafts to access the entire competition course. Stairs are expected to be a challenge element on both courses. The height difference between levels requires teams to traverse multiple flights of stairs to reach a different level.

Did we mention that the Urban Circuit will include multiple levels? Because there are going to be multiple levels. 

Vertical shafts and openings vary by course but may be as large as 2.5 m x 2.5 m. Each level may also include mezzanine levels. Artifacts may be located on mezzanine levels and in some cases may require elevated vantage points for line-of-site detection. It is expected that some areas could include significant drop-offs. Other areas include curbs that typically range from 0.1 to 0.2 meters high.

Interesting that there will be some artifacts that robots will have to spot without necessarily being able to get particularly close to them. And while curbs are good, the “significant drop-offs” is a bit worrisome. 

The width of passages at the Urban Circuit site vary greatly and include large open areas as well as narrow constrained passages common in urban environments (i.e., doorways). It is expected that some portions of the course will only be accessible via passages that are approximately one meter in height and/or one meter in width. For the Urban Circuit, it is expected that up to 50 percent of the competition course could be inaccessible for systems that cannot traverse these passages.

While a majority of passages at the Urban Circuit site are greater than 1 m x 1 m, some passages are as narrow as 0.8 meters and a limited number of passages are as narrow as 0.7 meters. Some artifacts (up to 10 percent) may be inaccessible without traversing the more constrained passages. 

Chubby robots are on notice!

Dispute Cards are intended to provide teams a mechanism to submit a formal dispute or request for review by the Chief Official. The Dispute Card must be completed and delivered by the Team Lead to the relevant Course Official, Team Garage Coordinator, or Chief Official. The Dispute Card must be submitted within 30 minutes of the completion of the run in question. All submissions will be reviewed by the Chief Official in a timely manner. All decisions made by the Chief Official are final.

This is a new mechanism for the Urban Circuit. We don’t know whether there was some specific incident that prompted this, or whether DARPA simply decided that it would be a good system to have in place. 

In the event that multiple teams have an identical score, tiebreakers will be applied in the following order until the tie is broken:
- Earliest time that the last artifact was successfully reported, averaged across the team’s best runs on each course
- Earliest time that the first artifact was successfully reported, averaged across the team’s best runs on each course
- Furthest distance traveled by any deployed system 

That last one (furthest distance traveled by any deployed system) is a new addition. I’d be surprised if a tiebreaker made it that far, but it does emphasize DARPA’s interest in exploration.

Teams are encouraged to provide audible and visual recovery aids such as flashing LEDs and audible cues to help Competition Staff locate deployed systems or components.

Many robots drop things like wireless repeaters as they progress through the course—DARPA was explicit that teams should not necessarily expect them all to be returned, but the agency may have gotten a little bit fed up attempting to track down all the robot droppings after every run.

There are currently no restrictions on the use of tethers for power, communications, or physical retrieval. However, teams are encouraged to consider the significant limitations imposed by the large-scale, potentially dynamic, and complex environments of interest.

This is more of a reiteration from DARPA than a new rule, but it’s worth noting that it’s being emphasized again for the Urban Circuit. Several teams who did use tethers for the Tunnel Circuit seemed to derive a significant advantage from them, and perhaps DARPA is concerned that more teams will use that approach. Which, of course, they can, it’s just that tethers can also be very limiting, and it seems like DARPA would really like to see more versatile, creative solutions.

Since the Urban Circuit is a closed course, the best way to follow along with the event will be through DARPA’s livestream, which will be on a one hour time delay (boo!) but may also have commentary this time around (yay!). Also, you can hear directly from the teams themselves by following #SubTChallenge on Twitter

Scored runs (the thing that’ll be the most fun to watch) will take place February 20 (Thursday) to February 22 (Saturday), and also February 24 (Monday) to February 26 (Wednesday). There’s a media day on the 24 that we’ll be at, so make sure and let us know if there are specific things you want us to check out.

[ DARPA Subterranean Challenge ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

Start your robots! Next week DARPA will kick off the Urban Circuit competition, the second of four robotics competitions that are part of its Subterranean Challenge.

We’re going to have a Urban Circuit preview post with all the details on Monday, a Q&A with DARPA SubT program manager Tim Chung on Tuesday, a follow-up with the teams on their preparations on Wednesday, and a post on how to watch the livestream on Thursday.

For now, watch Team Explorer from CMU and Oregon State testing some of their robots ahead of the competition.

In stage two of DARPA’s Subterranean Challenge, a team from Carnegie Mellon University and Oregon State will send robots into the depths of an incomplete nuclear power plant in a search-and-rescue scenario in Elma, Washington. Team Explorer’s machines will scale stairs and search for artifacts in the "Urban Circuit."

[ Team Explorer ]

We could watch these expandifying objects all day.

ExpandFab is a fabrication method for creating expanding objects using foam materials. The printed objects change their shape and volume, which is advantageous for reducing the printing time and transportation costs. For the fabrication of expanding objects, we investigated a basic principle of the expansion rate and developed materials by mixing a foam powder and elastic adhesive. Furthermore, we developed a fabrication method using the foam materials. A user can design expanded objects using our design software and sets the expansion areas on the surface. The software simulates and exports the 3d model into a three-dimensional (3D) printer. The 3D printer prints the expandable object by curing with ultraviolet light. Finally, the user heats the printed objects, and the objects expand to maximum approximately 2.7 times of their original size. ExpandFab allows users to prototype products that expand and morph into various shapes, such as objects changing from one shape to various shapes, and functional prototype with electronic components. In this paper, we describe the basic principle of this technique, implementation of the software and hardware, application examples, limitations and discussions, and future works.

[ Yasuaki Kakehi Laboratory ]

This new robot vacuum from Panasonic can prop itself up to drive over thick rugs and even go over small steps and bumps up to 2.5 centimeters high.

Apparently it does SLAM, though it’s not clear what kind of sensor it’s using. And at 1:05, is that a “follow me” feature?

[ Panasonic ] via [ ImpressWatch ]

Cybathlon needs you!

Robert Riener from ETH Zurich tells us that the Cybathlon organizers “still need many volunteers” to help with multiple phases of the event. Learn more about it here.

From 2nd to 3rd ETH Zurich’s CYBATHLON 2020 takes place in May. The CYBATHLON is a unique competition in which people with disabilities measure themselves when completing everyday tasks using the latest technical assistance systems.

Greet and look after international teams in the SWISS Arena in Kloten and help with the competition or with the assembly and dismantling! Are you in?

Register now as a volunteer http://www.cybathlon.com/volunteers

[ CYBATHLON ]

Happy Valentine’s Day from Robotiq!

[ Robotiq ]

In case you missed our story yesterday, Iranian researchers at the University of Tehran have unveiled a new humanoid robot called Surena IV.

[ IEEE Spectrum ]

Great, first those self-healing liquid metal robot tendons, and now this. How soon until T-1000? 

Researchers at Carnegie Mellon University and the UT Dallas have introduced a soft, multifunctional composite that remains electrically conductive when stretched and exhibits a number of other desirable properties for soft robotics and stretchable electronics.

[ CMU ]

Hey, it’s not just BotJunkie who likes to hug robots!

Roboy is not only the most human Robot, with it’s muscles and tendons - it’s also the most cuddly! At CIIE in November 2019, Roboy has been hugging more than 2800 people - connecting robots to humans and building relationships that last

[ Roboy ]

In the Dominican Republic, this initiative is using DJI drones to deliver supplies to rural communities.

Traditional methods of delivering medicine to rural communities have not been considered the most efficient solutions. Patients in smaller areas of the Dominican Republic, for example, would often go weeks without receiving the care they needed, increasing mortality rates. A reliable and cost-efficient solution became necessary. Thankfully, drone technology would answer the call. Watch how powerful equipment like the Matrice 600, and a strong collaboration between the local medical staff, Ministry of Health, WeRobotics and the Drone Innovation Center, has led to increased efficiency during important medical deliveries.

DJI ]

We’ve already seen some robots helping to fight the coronavirus outbreak. Here are some more.

This one is used for deliveries of food and medication:

[ New China TV ]

“The BOT-chelor” LOL

Who would you pick?

[ Sphero ]

Impressive demo of a real-time SLAM technique developed by SLAMcore, a startup spun out of Imperial College London.

This video was created in real-time running on the CPU of the Jetson TX2. The final map is sub cm accurate and only a few megabytes in size.

[ SLAMcore ]

We heart coffee indeed.

Kawasaki's coffee cobot, duAro, operates an espresso machine, and uses precise movements to pour steamed milk in the shape of a heart.

[ Kawasaki Robotics ]

How do you grasp hollow, deformable objects? Researchers at the Technical University of Munich are on the case.

[ Jingyi Xu ]

Some robots can use the same type of limb to walk and swim. But it would be even better if the robot’s limbs could change their shape to better adapt to different environments. Yale researchers are working on a possible implementation of this idea, inspired by sea turtles.

Most robots operate either exclusively on land or in water. Toward building an amphibious legged robot, we present a morphing limb that can adapt its structure and stiffness for amphibious operation. We draw inspiration for the limb’s design from the morphologies of sea turtle flippers and land-faring tortoise legs. Turtles and tortoises have rigid hulls that can be emulated in amphibious robots to provide a convenient, protected volume for motors, electronics, power supply, and payloads. Each of these animals’ limbs are tailored for locomotion in their respective environments. A sea turtle flipper has a streamlined profile to reduce drag, making it apt for swimming. A land tortoise leg boasts a strong, expanded cross-section conducive to load-bearing. We capture the morphological advantages of both animals’ limbs in our morphing limb via a variable stiffness composite coupled to a pneumatic actuator system that enables on-demand transitions between leg and flipper configurations. We control the degree of stiffness of the limb by varying electrical input to flexible heaters bound to the thermally responsive variable stiffness composite. The proposed morphing amphibious limb design is promising for enabling the next generation of hybrid soft-rigid robots to adapt to unstructured environments.

[ Yale Faboratory ]

Sorting recyclable waste: A job that we should definitely let robots steal from humans.

Recyclable waste sorting is mostly performed by manual labor in dull, dirty and dangerous environments. There are less and less people willing to perform these tasks and more and more recycling to be processed. A task like this is ideal for automation, but the challenge of recognizing and extracting random looking objects in a random stream of waste is very difficult. Newly available Artificial intelligence (AI) now enables these complex applications.

Current systems have limited recognition and extraction technologies that limit the quality of sorted material. Waste Robotics has developed a computer vision system that enables real-time recognition of objects in a random stream while dispatching robot commands to perform efficient and high-quality sorting to enable a circular economy.

[ FANUC ]

We aren’t sure why a flying robot is the best way of doing this task but the drone hitting the target with the rope at 0:20 is pretty cool.

This movie shows the successful test of a TugDrone, developed by Delft Dynamics and KOTUG. The drone delivers the ’heaving’ line from the the tugboat to a vessel. This improves safety on board both the vessel and the tugboat. This innovation supports the credo of KOTUG International: ’Ahead in Towage’.

[ Delft Dynamics ]

We were in Rwanda and Tanzania last year to see how drones are helping to deliver blood and medicine. See our full coverage and 3D videos here.

Three amazing days of #ADF2020 in Kigali has just wrapped up. See the highlights from the first African Drone Forum.

[ African Drone Forum ]

The IEEE Robotics and Automation Society continues to post more conference keynotes to its YouTube channel. This one is from ICRA 2018 by Louis Whitcomb from Johns Hopkins University on “Extreme Robotics: Underwater Robotic Exploration of the Karasik Seamount.”

[ IEEE RAS ]

A little over a decade ago, researchers at the University of Tehran introduced a rudimentary humanoid robot called Surena. An improved model capable of walking, Surena II, was announced not long after, followed by the more capable Surena III in 2015.

Now the Iranian roboticists have unveiled Surena IV. The new robot is a major improvement over previous designs. A video highlighting its capabilities shows the robot mimicking a person’s pose, grasping a water bottle, and writing its name on a whiteboard.

Surena is also shown taking a group selfie with its human pals.

Who developed Surena?

A team of more than 50 researchers built Surena IV at the University of Tehran’s Center for Advanced Systems and Technologies (CAST). Led by Dr. Aghil Yousefi-Koma, a professor of mechanical engineering, the group worked for the past four years to develop the new robot.

“Improving the robot’s interaction with the environment was one of the main goals of the Surena IV project,” Professor Yousefi-Koma tells IEEE Spectrum. He explains that the robot can now track objects more accurately, and new hands gave it better dexterity. These enhancements allow it to manipulate a wide variety of objects, including delicate ones, as well as operate power tools.

Surena IV is also more nimble. Custom force sensors on the bottom of its feet help the robot step over uneven surfaces by adjusting the angle and position of each foot. Walking speed increased to 0.7 kilometers per hour from 0.3 km/h in the previous generation.

While Surena IV is not a highly dynamic machine like Atlas, the famously agile humanoid from Boston Dynamics, it relies on a whole-body motion controller that continuously adjusts its posture to avoid falls. In its overall design, Surena is probably more comparable to humanoids like UBTECH’s Walker, Honda’s Asimo, Pal Robotics’ Talos, KAIST’s Hubo, and AIST’s HRP-2, although these robots have already publicly demonstrated a broader range of capabilities.

Photo: University of Tehran/CAST Surena IV is equipped with new hands that allow it to grasp a wider variety of objects.

To make his robots more widely known, Professor Yousefi-Koma has sought to take them to international conferences and trade shows, and his group has published papers on humanoid robot design, bipedal locomotion, and other topics. He has said that his group wants to “make [Surena] very competitive and may be taking part in an international challenge as well as finding real applications and users.”

In footage released as part of Surena IV’s unveiling, which happened late last year, the humanoid is shown lifting a box, inside of which is Surena Mini, a knee-high robot that the CAST team developed in 2017.

Another video gives a behind-the-scenes look at Surena’s development, showing the researchers testing new hardware, writing code, and assisting the robot as it takes its first steps.

The evolution of Surena

The latest Surena model has come a long way since the initial version, unveiled in 2008, which had only 8 degrees of freedom (DoF) and used wheels on its feet to move around. Surena II had 22 DoF and was the first version capable of walking. Surena III, with 31 DoF, demonstrated skills like climbing steps and kicking a ball.

Surena IV has even more axes of motion, with additional 12 DoF, most on the hands, bringing the total to 43 DoF. The CAST team redesigned several components, developing new lightweight structural elements and small yet powerful custom actuators. As a result, the new robot—at 1.7 meters tall and 68 kilograms—is lighter and more compact than Surena III (1.9 m and 98 kg).

Photo: University of Tehran/CAST Led by Dr. Aghil Yousefi-Koma [standing to the left of the robot], a team of more than 50 researchers developed Surena IV at the University of Tehran’s Center for Advanced Systems and Technologies (CAST). During an event late last year, part of the group posed with Surena for a photo.

Another advance is the robot’s controller. The control loop now operates at a higher frequency (200 Hz), thanks to newly added FPGA boards. Sensors include stereo cameras, 6-axis force/torque sensors in the ankles, encoders on all joints, and an inertial measurement unit (IMU). A text-to-speech system allows the robot to recognize and generate basic speech.

To make all the sensors, controllers, and actuators work together, the researchers equipped Surena with the Robot Operating System, or ROS. They also used Gazebo, Choreonoid, and MATLAB to simulate the robot’s motions and evaluate different behaviors, including walking backwards and sideways, turning around, and push recovery.

But the upgrades are not just internal. Surena IV’s exterior features new plastic covers that make the robot look sleek—and only slightly menacing. Professor Yousefi-Koma has emphasized that he views Surena as a “symbol of technology advancement in the direction of peace and humanity,” hoping it will help inspire people about the possibilities of robotics.

A version of this article was originally published on Medium. The views expressed here are solely those of the authors and do not represent positions of IEEE Spectrum or the IEEE.

We here at Skydio have been developing and deploying machine learning systems for years due to their ability to scale and improve with data. However, to date our learning systems have only been used for interpreting information about the world; in this post, we present our first machine learning system for actually acting in the world.

Using a novel learning algorithm, the Skydio autonomy engine, and only 3 hours of “off-policy” logged data, we trained a deep neural network pilot that is capable of filming and tracking a subject while avoiding obstacles.

We approached the problem of training a deep neural network pilot through the lens of imitation learning, in which the goal is to train a model that imitates an expert. Imitation learning was an appealing approach for us because we have a huge trove of flight data with an excellent drone pilot—the motion planner inside the Skydio autonomy engine. However, we quickly found that standard imitation learning performed poorly when applied to our challenging, real-world problem domain.

Standard imitation learning worked fine in easy scenarios, but did not generalize well to difficult ones. We propose that the signal of the expert’s trajectory is not rich enough to learn efficiently. Especially within our domain of flying through the air, the exact choice of flight path is a weak signal because there can be many obstacle-free paths that lead to cinematic video. The average scenario overwhelms the training signal.

CEILing can leverage acausal (future) data, which enables it to “see” farther into the future and therefore train an even smarter pilot

How can we do better? Our insight is that we don’t have just any expert, we have a computational expert: the Skydio Autonomy Engine. Therefore instead of imitating what the expert does, we understand what the expert cares about. We call this approach Computational Expert Imitation Learning, or CEILing.

Why is CEILing better than standard imitation learning? Let’s consider a didactic example in which a teacher is trying to teach a student how to do multiplication. The teacher is deciding between two possible lesson plans. The first lesson plan is to give the student a bunch of multiplication problems, along with the answer key, and leave the student alone to let them figure out how multiplication works. The second lesson plan is to let the student attempt to solve some multiplication problems, give the student feedback on the exact mistakes they made, and continue until the student has mastered the topic.

Which lesson plan should the teacher choose? The second lesson plan is likely to be more effective because the student not only learns the correct answer, but also learns why the answer is correct. This allows the student to be able to solve multiplication problems they have never encountered before.

Gif: Skydio

Simulation trials of the deep neural pilot based on the Computational Expert Imitation Learning, or CEILing, approach developed by the authors.

This same insight applies to robot navigation: Some deviations from the expert should be penalized more heavily than others. For example, deviating from the expert is generally okay in open space, but a critical mistake if it is towards an obstacle or causes visual loss of the subject. CEILing lets us convey that information from the expert instead of blindly penalizing deviations from the expert’s trajectory. This is why CEILing trains a deep neural pilot that generalizes well with little data.

Now one question to ask is why even use CEILing to train a deep neural pilot? Why not just have the computational expert be the pilot? The primary reason we are excited about CEILing is that CEILing could train a pilot that is actually better than the computational expert pilot.

How is this possible? Consider a scenario in which a drone needs to fly through a forest at high speed. This is a challenging environment because thin objects, such as tree branches, are difficult to see from far away. Although the current Skydio autonomy engine is able to perceive and avoid these thin branches, sometimes the branches can only be detected when the drone is already quite close, which forces the drone to execute an aggressive maneuver. In contrast, our deep neural pilot could be able to smoothly avoid these thin branches altogether because it will have learned that flying towards trees—which are large and easily seen—is correlated with encountering thin branches. In short, CEILing can leverage acausal (future) data, which enables it to “see” farther into the future and therefore train an even smarter pilot.

Although there is still much work to be done before the learned system will outperform our production system, we believe in pursuing leapfrog technologies. Deep reinforcement learning techniques promise to let us improve our entire system in a data-driven way, which will lead to an even smarter autonomous flying camera.

Gregory Kahn is a Ph.D. student at UC Berkeley advised by Sergey Levine and Pieter Abbeel, and was a research intern at Skydio in Spring 2019.

Abraham Bachrach is co-founder and CTO at Skydio.

Hayk Martiros leads the autonomy team at Skydio, whose work focuses on robust approaches to vision-based autonomous flight.

This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

Honda Research Institute’s (HRI) experimental social robot Haru was first introduced at the ACM/IEEE Human Robot Interaction conference in 20181. The robot is designed as a platform to investigate social presence and emotional and empathetic engagement for long-term human interaction. Envisioned as a multimodal communicative agent, Haru interacts through nonverbal sounds (paralanguage), eye, face, and body movements (kinesics), and voice (language). While some of Haru’s features connect it to a long lineage of social robots, others distinguish it and suggest new opportunities for human-robot interaction.

Haru is currently in its first iteration, with plans underway for future development. Current research with Haru is conducted with core partners of the Socially Intelligent Robotics Consortium (SIRC), described in more detail below, and it concentrates on its potential to communicate across the previously mentioned three-way modality (language, paralanguage, and kinesics). Long term, we hope Haru will drive research into robots as a new form of companion species and as a platform for creative content.

Image: Honda Research Institute Communication with Haru

The first Haru prototype is a relatively simple communication device—an open-ended platform through which researchers can explore the mechanisms of multi-modal human-robot communication. Researchers deploy various design techniques, technologies, and interaction theories to develop Haru’s basic skillset for conveying information, personality, and affect. Currently, researchers are developing Haru’s interaction capabilities to create a new form of telepresence embodiment that breaks away from the traditional notion of a tablet-on-wheels, while also exploring ways for the robot to communicate with people more directly.

Haru as a mediator

Haru’s unique form factor and its developing interaction modalities enable its telepresence capabilities to extend beyond what is possible when communicating through a screen. Ongoing research with partners involves mapping complex and nuanced human expressions to Haru. One aspect of this work developing an affective language that takes advantage of Haru’s unique LCD screen eyes and LED mouth, neck, and eye motions, and rotating base to express emotions of varying type and intensity. Another ongoing research is the development of perception capabilities for Haru to recognize different human emotions and will allow Haru to mimic the remote person’s affective cues and behaviors. 

From a practical telepresence perspective, using Haru in this way can, on the one hand, add to the enjoyment and clarity with which affect can be conveyed and read at a distance. On the other hand, Haru’s interpretation of emotional cues can help align the expectations between sender and receiver. For example, Haru could be used to enhance communication in a multi-cultural context by displaying culturally appropriate social cues without the operator controlling it. These future capabilities are being put into practice through the development of “robomoji/harumoji”—a hardware equivalent of emoji with Haru acting as a telepresence robot that transmits a remote person’s emotions and actions. 

Haru as a communicator

Building upon the work on telepresence, we will further develop Haru as a communicative agent through the development of robot agency2 consistent with its unique design and behavioral repertoire, and possibly with its own character. Ongoing research involves kinematic modeling and motion design and synthesis. We are also developing appropriate personalities for Haru with the help of human actors, who are helping us create a cohesive verbal and nonverbal language for Haru that expresses its capabilities and can guide users to interact with it in appropriate ways.

Haru will in this case autonomously frame its communication to fit that of its co-present communication partner. In this mode, the robot has more freedom to alter and interpret expressions for communication across the three modalities than in telepresence mode. The goal will be to communicate so that the receiving person not only understands the information being conveyed, but can infer Haru’s own behavioral state and presence. This will ideally make the empathic relation between Haru and the human bi-directional.     

Hybrid realism

Designing for both telepresence and direct communication means that Haru must be able to negotiate between times when it is communicating on behalf of a remote person, and times when it is communicating for itself. In the future, researchers will explore how teleoperation can be mixed with a form of autonomy, such that people around Haru know whether it is acting as a mediator or as an agent as it switches between “personalities.” Although challenging, this will open up new exciting opportunities to study the effect of social robots embodying other people, or even other robots. 

Image: Honda Research Institute Robots as a companion species

The design concept of Haru focuses on creating an emotionally expressive embodied agent that can support long-term, sustainable human-robot interaction. People already have these kinds of relationships with their pets. Pets are social creatures people relate to create bonds without sharing the same types of perception, comprehension, and expression. This bodes well for robots such as Haru, which similarly may not perceive, comprehend, or express themselves in fully human-like ways, but which nonetheless can encourage humans to empathize with them. This may also prove to be a good strategy for grounding human expectations while maximizing emotional engagement.

Recent research3,4 has shown that in some cases people can develop emotional ties to robots. Using Haru, researchers can investigate whether human-robot relations can become even more affectively rich and meaningful through informed design and better interaction strategies, as well as through more constant and varied engagement with users. 

Photo: Evan Ackerman/IEEE Spectrum Haru is sad.

Ongoing research with partners involve interaction-based dynamic interpretation of trust, design and maximization of likable traits through empathy, and investigating how people of different ages and cognitive abilities make sense of and interact with Haru. To carry out these tasks, our partners have been deploying Haru in public spaces such as in a children’s science museum, an elementary school, and an intergenerational daycare institution.

In the later stages of the project, we hope to look at how a relationship with Haru might position the robot as a new kind of companion. Haru is not intended to be a replacement for a pet or a person, but rather an entity that people can bond and grow with in a new way—a new “companion species”5. Sustained interaction with Haru will also allow us to explore how bonding can result in a relationship and a feeling of trust. We argue that a trusting relationship requires the decisional freedom not to trust and therefore, would be one in which neither human nor robot are always subordinate to each other, but rather one that supports a flexible bi-directional relationship enabling cooperative behavior and growth.  

Image: Honda Research Institute Robots as a platform for creative content

Social robots should not be designed as mere extensions to speech-centric smart devices; we need to develop meaningful and unique use cases suited to the rich modalities robots offer. For Haru, such use cases must be rooted in its rich three-way communicative modality and its unique physical and emotional affordances. Researchers plan to maximize Haru’s communication and interaction potential by developing social apps and creative content that convey information and evoke emotional responses that can enhance people’s perceptions of the robot as a social presence. 

The Haru research team (consisting of both HRI Japan and Socially Intelligent Robotics Consortium partners) will develop apps that meet societal needs through Haru’s emotional interactions with people. We envision three main pillars of research activity: entertainment and play, physically enriched social networking, and mental well-being. Entertainment and play will encompass interactive storytelling, expressive humor, and collaborative gaming, among others. Physically enriched social networking will build upon the telepresence capability of Haru. Mental well-being will explore the beneficial role of Haru’s physical manifestation of emotion and affection for coaching.

Numerous smartphone apps6 explore this application in digital and unimodal communication. We believe the much richer expressiveness and physical presence will give Haru exciting opportunities to qualitatively improve such services. Current research with partners involves collaborative problem solving and developing interactive applications for entertainment and play. Haru is currently able to play rock-paper-scissors and tower of Hanoi with people, which are being tested in an elementary school setting. Another capability that is being developed for Haru is storytelling play, in which the robot can collaborate with people to create a story. 

Haru’s research team also sees the importance of both automated and curated creative multimedia content to complement such social applications. Content drives the success of smartphones and gaming consoles, but has not yet been widely considered for robots. The team will therefore investigate how content can contribute to social human-robot interaction. Haru, by delivering creative content using its three-way communication modality, will blur the line between the physical and the animated, reinforcing its presence alongside the content it conveys.

Content drives the success of smartphones and gaming consoles, but has not yet been widely considered for robots.

In the context of Haru, creativity will be cooperative. Through sustained interaction, Haru will be able to motivate, encourage, and illustrate alternatives in creative tasks (e.g. different potential storylines in storytelling or game play) depending on the cognitive state of its interaction partner. The mixture of digital visualization (with a mini projector, for example) and physical interaction opens up new opportunities for cooperative creativity. Ongoing research with partners involves automatic affect generation for Haru, models for generating prose content for story telling in different genre, and affectively rich audio and text-to-speech to support social apps.  

Finally, Haru will be a networked device embedded into a personal or office based digital infrastructure. On one hand, Haru can serve as a physical communication platform for other robots (e.g. robotic vacuum cleaners that typically have no significant communication channels), and on the other hand, Haru can use the additional physical capabilities of other devices to complement its own limitations.

Photo: Evan Ackerman/IEEE Spectrum Haru sees you! Research with Haru

Realizing the design and development goals for Haru is an enormous task, only possible through collaboration of a multi-disciplinary group of experts in robotics, human-robot interaction, design, engineering, psychology, anthropology, philosophy, cognitive science, and other fields. To help set and achieve the goals described above, over the past two years academic researchers in Japan, Australia, the United States, Canada, and Europe have been spending their days with Haru.

This global team of roboticists, designers, and social scientists call themselves the Socially Intelligent Robotics Consortium (SIRC) and are working together to expand and actualize the potential of human-robot communication. Under the umbrella of the consortium, Haru has been adopted as a common platform enabling researchers to share data, research methods, results, and evaluation tools. As a partnership, it is possible to tackle the complex problem of developing empathic robotic mediators, companions, and content platforms by working on independent tasks under a common theme. 

Haru offers the opportunity to experiment with social robots as a new kind of trustful companion, not just based on functionality, but on emotional connection and the sense of unique social presence.

Social robotics is a promising area of research, but in a household crowded with smart devices and appliances, it is important to re-think the role of social robots to find their niche amidst competing and overlapping products while keeping users engaged over months and years instead of days and weeks. The development of Haru offers the opportunity to experiment with flexible and varied ways to communicate and interact with social robots as a new kind of trustful companion, not just based on functionality, but on emotional connection and the sense of unique social presence.

Research with Haru will explore exciting applications driven by interaction, cooperation, creativity and design, suspending people’s disbelief so that each experience with Haru is a step in a shared journey, demonstrating how people and robots might grow and find common purpose and enjoyment together in a hybrid human-robot society.

Dr. Randy Gomez is a senior scientist at Honda Research Institute Japan. He oversees the embodied communication research group, which explores the synergy between communication and interaction in order to create meaningful experiences with social robots.

References

1. Gomez R., Szapiro D., Galindo K. & Nakamura K., “Haru: Hardware Design of an Experimental Tabletop Robot Assistant,” HRI 2018: 233-240.

2. Gunkel D., “Robot Rights,” The MIT Press, 2018.

3. Haraway, “The Companion Species Manifesto: Dogs, People, and Significant Otherness,” Prickly Paradigm Press, 2003.

4. Weiss A., Wurhofer, D., & Tscheligi, M, “I love this dog-children’s emotional attachment to the robotic dog AIBO,” International Journal of Social Robotics, 2009.

5. Sabanovic S., Reeder, S. M., & Kechavarzi, B, “Designing robots in the wild: In situ prototype evaluation for a break management robot,” Journal of Human-Robot Interaction, 2014.

6. Inkster B., Shubhankar S., & Vinod S. , “An empathy-driven, conversational artificial intelligence agent (Wysa) for digital mental well-being: real-world data evaluation mixed-methods study,” JMIR mHealth and uHealth, 2018.

The Sony Aibo has been the most sophisticated home robot that you can buy for an astonishing 20 years. The first Aibo went on sale in 1999, and even though there was a dozen year-long gap between 2005’s ERS-7 and the latest ERS-1000, there was really no successful consumer robot over that intervening time that seriously challenged the Aibo.

Part of what made Aibo special was how open Sony was user customization and programmability. Aibo served as the RoboCup Standard Platform for a decade, providing an accessible hardware platform that leveled the playing field for robotic soccer. Designed to stand up to the rigors of use by unsupervised consumers (and, presumably, their kids), Aibo offered both durability and versatility that compared fairly well to later, much more expensive robots like Nao

Aibo ERS-1000: The newest model

The newest Aibo, the ERS-1000, was announced in late 2017 and is now available for US $2,900 in the United States and 198,000 yen in Japan. It’s faithful to the Aibo family, while benefiting from years of progress in robotics hardware and software. However, it wasn’t until last November that Sony opened up Aibo to programmers, by providing visual programming tools as well as access to an API (application programming interface). And over the holidays, Sony lent us an Aibo to try it out for ourselves.

This is not (I repeat not) an Aibo review: I’m not going to talk about how cute it is, how to feed it, how to teach it to play fetch, how weird it is that it pretends to pee sometimes, or how it feels to have it all snuggled up in your lap while you’re working at your computer. Instead, I’m going to talk about how to (metaphorically) rip it open and access its guts to get it to do exactly what you want.

Photo: Evan Ackerman/IEEE Spectrum The newest Aibo, the ERS-1000, was announced in late 2017 and is now available for US $2,900 in the United States and 198,000 yen in Japan.

As you read this, please keep in mind that I’m not much of a software engineer—my expertise extends about as far as Visual Basic, because as far as I’m concerned that’s the only programming language anyone needs to know. My experience here is that of someone who understands (in the abstract) how programming works, and who is willing to read documentation and ask for help, but I’m still very much a beginner at this. Fortunately, Sony has my back. For some of it, anyway.

Getting started with Aibo’s visual programming

The first thing to know about Sony’s approach to Aibo programming is that you don’t have access to everything. We’ll get into this more later, but in general, Aibo’s “personality” is completely protected and cannot be modified:

When you execute the program, Aibo has the freedom to decide which specific behavior to execute depending on his/her psychological state. The API respects Aibo's feelings so that you can enjoy programming while Aibo stays true to himself/herself.

This is a tricky thing for Sony, since each Aibo “evolves” its own unique personality, which is part of the appeal. Running a program on Aibo risks very obviously turning it from an autonomous entity into a mindless robot slave, so Sony has to be careful to maintain Aibo’s defining traits while still allowing you to customize its behavior. The compromise that they came up with is mostly effective, and when Aibo runs a program, it doesn’t disable its autonomous behaviors but rather adds the behaviors you’ve created to the existing ones. 

Aibo’s visual programming system is based on Scratch. If you’ve never used Scratch, that’s fine, because it’s a brilliantly easy and intuitive visual language to use, even for non-coders. Sony didn’t develop it—it’s a project out of MIT, and while it was originally designed for children, it’s great for adults who don’t have coding experience. Rather than having to type in code, Scratch is based around colorful blocks that graphically represent functions. The blocks are different shapes, and only fit together in a way that will yield a working bit of code. Variables appear in handy little drop-down menus, and you can just drag and drop different blocks to build as many programs as you want. You can even read through the code directly, and it’ll explain what it does in a way that makes intuitive sense, more or less:

Screenshot: Evan Ackerman/IEEE Spectrum A sample Aibo visual program from Sony.

Despite the simplicity of the visual programming language, it’s possible to create some fairly complex programs. You have access to control loops like if-then-else and wait-until, and multiple loops can run at the same time. Custom blocks allow you to nest things inside of other things, and you have access to variables and operators. Here’s a program that I put together in just a few minutes to get Aibo to entertain itself by kicking a ball around:

Screenshot: Evan Ackerman/IEEE Spectrum A program I created to make Aibo chase a ball around.

This program directs Aibo to respond to “let’s play” by making some noises and motions, locating and approaching its ball, kicking its ball, and then moving in some random directions before repeating the loop. Petting Aibo on its back will exit the loop.

Programming Aibo: What you can (and can’t) do

It’s a lot of fun to explore all of Aibo’s different behaviors, although if you’re a new user, it does minimize a bit of the magic to see this big long list of everything that Aibo is capable of doing. The granularity of some of commands is a little weird—there’s a command for “gets close to” an object, as well as a command for “gets closer to” an object. And rather than give you direct access to Aibo’s servos to convey emotions or subtle movement cues, you’re instead presented with a bewildering array of very specific options, like:

  • Aibo opens its mouth a little and closes it
  • Aibo has an “I get it” look
  • Aibo gives a high five with its right front paw
  • Aibo faces to the left petulantly
  • Aibo has a dream of becoming a human being and runs about

Unfortunately, there’s no way to “animate” Aibo directly—you don’t have servo-level control, and unlike many (if not most) programmable robots, Sony hasn’t provided a way for users to move Aibo’s servos and then have the robot play back those motions, which would have been simple and effective.

Running one of these programs can be a little frustrating at times, because there’s no indication of when (or if) Aibo transitions from its autonomous behavior to your program—you just run the program and then wait. Sony advises you to start each program with a command that puts Aibo’s autonomy on hold, but depending on what Aibo is in the middle of doing when you run your program, it may take it a little bit to finish its current behavior. My solution for this was to start each program with a sneeze command to let me know when things were actually running. This worked well enough I guess, but it’s not ideal, because sometimes Aibo sneezes by itself.

Running one of these programs can be a little frustrating at times, because there’s no indication of when (or if) Aibo transitions from its autonomous behavior to your program. My solution for this was to start each program with a sneeze command to let me know when things were actually running.

The biggest restriction of the visual programming tool is that as far as I can tell there’s no direct method of getting information back from Aibo—you can’t easily query the internal state of the robot. For example, if you want to know how much battery charge Aibo has, there’s a sensing block for that, but the best you seem to be able to do is have Aibo do specific things in response to the value of that block, like yap a set number of times to communicate what its charge is. More generally, however, it can be tough to write more interactive programs, because it’s hard to tell when, if, why, or how such programs are failing. From what I can tell, there’s no way “step” through your program, or to see which commands are being executed when, making it very hard to debug anything complicated. And this is where the API comes in handy, since it does give you explicit information back.

Aibo API: How it works

There’s a vast chasm between the Aibo visual programming language and the API. Or at least, that’s how I felt about it. The visual programming is simple and friendly, but the API just tosses you straight into the deep end of the programming pool. The good news is that the majority of the stuff that the API allows you to do can also be done visually, but there are a few things that make the API worth having a crack at, if you’re willing to put the work in.

The first step to working with the Aibo API is to get a token, which is sort of like an access password for your Sony Aibo account. There are instructions about how to do this that are clear enough, because it just involves clicking one single button. Step two is finding your Aibo’s unique device ID, and I found myself immediately out of my comfort zone with Sony’s code example of how to do that:

$ curl -X GET https://public.api.aibo.com/v1/devices \
-H "Authorization:Bearer ${accessToken}" 

As it turns out, “curl” (or cURL) is a common command line tool for sending and receiving data via various network protocols, and it’s free and included with Windows. I found my copy in C:\Windows\System32. Being able to paste my token directly into that bit of sample code and have it work would have been too easy—after a whole bunch of futzing around, I figured out that (in Windows) you need to explicitly call “curl.exe” in the command line and that you have to replace “${accessToken}” with your access token, as opposed to just the bit that says “accessToken.” This sort of thing may be super obvious to many people, but it wasn’t to me, and with the exception of some sample code and a reasonable amount of parameter-specific documentation, Sony itself offers very little hand-holding. But since figuring this stuff out is my job, on we go!

Image: Sony How the Aibo API works: Your computer doesn’t talk directly to your robot. Instead, data flows between your computer and Sony’s cloud-based servers, and from the cloud to your robot. 

I don’t have a huge amount of experience with APIs (read: almost none), but the way that the Aibo API works seems a little clunky. As far as I can tell, everything runs through Sony’s Aibo server, which completely isolates you from the Aibo itself. As an example, let’s say we want to figure out how much battery Aibo has left. Rather than just sending a query to the robot and getting a response, we instead have to ask the Aibo server to ask Aibo, and then (separately) ask the Aibo server what Aibo’s response was. Literally, the process is to send an “Execute HungryStatus” command, which returns an execution ID, and then in a second command you request the result of that execution ID, which returns the value of HungryStatus. Weirdly, HungryStatus is not a percentage or a time remaining, but rather a string that goes from “famished” (battery too low to move) to “hungry” (needs to charge) to “enough” (charged enough to move). It’s a slightly strange combination of allowing you to get deep into Aibo’s guts while seeming trying to avoid revealing that there’s a robot under there.

Screenshot: Evan Ackerman/IEEE Spectrum Example of the code required to determine Aibo’s charge. (I blurred areas showing my Aibo’s device ID and token.)

Anyway, back to the API. I think most of the unique API functionality is related to Aibo’s state—how much is Aibo charged, how sleepy is Aibo, what is Aibo perceiving, where is Aibo being touched, that sort of thing. And even then, you can kludge together ways of figuring out what’s going on in Aibo’s lil’ head if you try hard enough with the visual programming, like by turning battery state into some number of yaps.

But the API does also offer a few features that can’t be easily replicated through visual programming. Among other things, you have access to useful information like which specific voice commands Aibo is responding to and exactly where (what angle) those commands are coming from, along with estimates of distance and direction to objects that Aibo recognizes. Really, though, the value of the API for advanced users is the potential of being able to have other bits of software interact directly with Aibo.

API possibilities, and limitations

For folks who are much better at programming than I am, the Aibo API does offer the potential to hook in other services. A programming expert I consulted suggested that it would be fairly straightforward to set things up so that (for example) Aibo would bark every time someone sends you a tweet. Doing this would require writing a Python script and hosting it somewhere in the cloud, which is beyond the scope of this review, but not at all beyond the scope of a programmer with modest skills and experience, I would imagine.

Fundamentally, the API means that just about anything can be used to send commands to Aibo, and the level of control that you have could even give Aibo a way to interact with other robots. It would just be nice if it was a little bit simpler, and a little more integrated, since there are some significant limitations worth mentioning.

For example, you have only indirect access to the majority of Aibo’s sensors, like the camera. Aibo will visually recognize a few specific objects, or a general “person,” but you can’t add new objects or differentiate between people (although Aibo can do this as part of its patrol feature). You can’t command Aibo to take a picture. Aibo can’t make noises that aren’t in its existing repertoire, and there’s no way to program custom motions. You also can’t access any of Aibo’s mapping data, or command it to go to specific places. It’s unfortunate that many of the features that justify Aibo’s cost, and differentiate it from something that’s more of a toy, aren’t accessible to developers at this point.

Photo: Evan Ackerman/IEEE Spectrum Aibo’s API gives users access to, among other things, specific voice commands the robot is responding to and exactly where (what angle) those commands are coming from, along with estimates of distance and direction to objects that Aibo recognizes. Aibo’s programmability: The future

Overall, I appreciate the approach that Sony took with Aibo’s programmability, making it accessible to both absolute beginners as well as more experienced developers looking to link Aibo to other products and services. I haven’t yet seen any particularly compelling examples of folks leveraging this capability with Aibo, but the API has only been publicly available for a month or two. I would have liked to have seen more sample programs from Sony, especially more complex visual programs, and I would have really appreciated a gentler transition over to the API. Hopefully, both of these things can be addressed in the near future.

There’s a reluctance on Sony’s part to give users more control over Aibo. Some of that may be technical, and some of it may be privacy-related, but there are also omissions of functionality and limitations that don’t seem to make sense. I wonder if Sony is worried about risking an otherwise careful compromise between a robot that maintains its unique personality, and a robot that can be customized to do whatever you want it to do. As it stands, Sony is still in control of how Aibo moves, and how Aibo expresses emotions, which keeps the robot’s behavior consistent, even if it’s executing behaviors that you tell it to. 

At this point, I’m not sure that the Aibo API is full-featured and powerful enough to justify buying an Aibo purely for its developer potential, especially given the cost of the robot. If you already have an Aibo, you should definitely play with the new programming functions, because they’re free. I do feel like this is a significant step in a very positive direction for Sony, showing that they’re willing to commit resources to the nascent Aibo developer community, and I’m very much looking forward to seeing how Aibo’s capabilities continue to grow.

Photo: Evan Ackerman/IEEE Spectrum Aibo deserves a rest!

Thanks to Sony for lending us an Aibo unit for the purposes of this review. I named it Aibo, and I will miss its blue eyes. And special thanks to Kevin Finn for spending part of his holiday break helping me figure out how Aibo’s API works. If you need help with your Aibo, or help from a professional software engineer on any number of other things, you can find him here.

[ Aibo Developer Site ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France ICUAS 2020 – June 9-12, 2020 – Athens, Greece CLAWAR 2020 – August 24-26, 2020 – Moscow, Russia

Let us know if you have suggestions for next week, and enjoy today’s videos.

Since Honda decided to stop further development of the beloved robot Asimo, attention has turned to other companies building advanced humanoids. One of them is UBTECH, which appears to be making steady progress with its Walker robot. At CES early this year, the company showed Walker pushing a cart, pouring a drink, standing on one foot, and even bending its body backward like a yogi.

We had such an amazing time at CES 2020 showing you the major upgrades we’ve made to Walker. With improved flexibility, stability, precision, and speed, Walker has come a long way since its initial debut at CES a few years back.

Walker is an intelligent Humanoid Service Robot designed with outstanding hardware, excellent motion ability and AI interactive performance – the most advanced robot UBTECH has ever created.

But UBTECH wasn’t done. It also demoed its service robot Cruzr and indoor inspection robot AIMBOT.

Cruzr, UBTECH’s enterprise service robot, was on full display at CES 2020!

Cruzr is a cloud-based intelligent humanoid robot that provides a new generation of service for a variety of industrial applications. Cruzr helps enhance and personalize the guest experience in consumer facing establishments such as retail, financial institutions, and hospitality.

AT CES 2020, we showcased AIMBOT, an autonomous indoor monitoring robot. AIMBOT is used for intelligent and accurate indoor inspection, efficient inventory management, visitor verification, preventing safety hazards and more.

UBTECH ]

Generating complex movements in redundant robots like humanoids is usually done by means of multi-task controllers based on quadratic programming, where a multitude of tasks is organized according to strict or soft priorities.

Time-consuming tuning and expertise are required to choose suitable task priorities, and to optimize their gains.

Here, we automatically learn the controller configuration (soft and strict task priorities and Convergence Gains), looking for solutions that track a variety of desired task trajectories efficiently while preserving the robot’s balance.

We use multi-objective optimization to compare and choose among Pareto-optimal solutions that represent a trade-off of performance and robustness and can be transferred onto the real robot.

We experimentally validate our method by learning a control configuration for the iCub humanoid, to perform different whole-body tasks, such as picking up objects, reaching and opening doors.

[ Larsen/Inria ]

This week, roboticist and comedian Naomi Fitter wrote a fantastic guest post on her experiences with robot comedy. Here’s one of the performances she’s created, with her Nao humanoid talking and singing with comedian Sarah Hagen.

Sketch comedy duo act including the talented human/comedian Sarah Hagen and the Oregon State University SHARE Lab’s illustrious NAO robot.

[ Naomi Fitter ]

This work is part of Tim Hojnik’s PhD project, a partnership between CSIRO’s Data61 Robotics and Autonomous Systems Group and the Queensland University of Technology.

[ CSIRO ]

Who’s ready for Superbowl LIV!? The Gripper Guys are.

[ Soft Robotics ]

Researchers at the Max Planck Institute for Intelligent Systems in Stuttgart, Germany, have designed and fabricated an untethered microrobot that can slip along either a flat or curved surface in a liquid when exposed to ultrasound waves. Its propulsion force is two to three orders of magnitude stronger than the propulsion force of natural microorganisms such as bacteria or algae. Additionally, it can transport cargo while swimming. The acoustically propelled robot hence has significant potential to revolutionize the future minimally invasive treatment of patients.

[ Max Planck Institute for Intelligent Systems ]

Did you know Kuka have a giant linear robot? Now you do!

The three-axis linear robot KR 80L has Cartesian axes which are operated via the robot controller. The development of the new KR 80L benefited greatly from KUKA experience gained from many handling applications and our expertise as one of the leading suppliers of intelligent automation solutions.

The modular design allows workspaces from 0.75m³ up to 225m³ to be implemented, making the KUKA linear robot a safe investment for your automation. Minimal interference contours below the robot mean that it is ideally suited for linking work processes by carrying out loading and unloading, palletizing, handling or transfer tasks, for example. The use of proven, series-produced robotic drive components ensures utmost performance and reliability.

[ Kuka ]

Apparently Promobot brought one of its humanoids to New York City’s Bryant Park to help screen people for the coronavirus. NYC officers promptly ejected the robot from the park for lacking a permit, but not before a little robot dance party. 

[ Promobot ] via [ NY Post ]

LOVOT, which we’ve featured on our Robot Gift Guide, is very cute—at least when it has its furry skin on.

Unfortunately we don’t speak Japanese to understand the full presentation, but we applaud the fact that the company is willing to discuss—and show—what’s inside the robot. Given the high rate of consumer robot failures, more sharing and transparency could really help the industry.

[ Robot Start ]

Drones have the potential to change the African continent by revolutionizing the way deliveries are made, blood samples are processed, farmers grow their crops and more. To tackle the many challenges faced by Africa, the World Bank and partners convened the African Drone Forum in Kigali, Rwanda, from February 5-7, 2020. To welcome the audience of engineers, scientists, entrepreneurs, development experts and regulators, the World Bank and ADF team created this video.

[ African Drone Forum ]

We continue to scale our fully driverless experience -- with no one behind the wheel -- for our early riders in Metro Phoenix. We invited Arizona football legend Larry Fitzgerald to take a ride with our Waymo Driver. Watch all of Larry’s reactions in this video of his full, unedited ride.

[ Waymo ]

The humanoid Robot ARMAR-6 grasps unknown objects in a cluttered box autonomously.

[ H2T KIT ]

Quanser R&D engineers have been testing different bumper designs and materials to protect the QCar in collisions. This is a scale-speed equivalent of 120km/hr!

[ Quanser ]

Drone sales have exploded in the past few years, filling the air with millions of new aircraft. Simple modifications to these drones by criminals and terrorists have left people, privacy and physical and intellectual property totally exposed.

Fortem Technologies innovates to stay ahead of the threat, keeping pace with escalating drone threats worldwide.

With more than 3,650 captures at various attack vectors and speeds, DroneHunter is the leading, world-class interceptor drone.

[ Fortem Technologies ] via [ Engadget ]

This is an interesting application of collaborative robots at this car bumper manufacturer, where they mounted industrial cameras on FANUC cobots to perform visual quality-control checks. These visual inspections happen throughout the assembly line, with the robots operating right next to the human workers.

Discovering the many benefits a FANUC collaborative robot solution can provide.

Flex-N-Gate, a supplier of bumpers, exterior trim, lighting, chassis assemblies and other automotive products, uses inspection systems at their Ventra Ionia, Michigan plant to ensure product quality.

To help improve these processes, reduce costs and save floor space, Flex-N-Gate turned to FANUC for a collaborative robot solution, leveraging FANUC America’s 24/7/365 service network to support their cobot systems for a completely successful integration.

[ FANUC ]

In this video we present results on autonomous subterranean exploration inside an abandoned underground mine using the ANYmal legged robot. ANYmal is utilizing the proposed Graph-based Exploration Path Planner which ensures the efficient exploration of the complex underground environment, while simultaneously avoiding obstacles and respecting traversability constraints.

The designed planner first operates by engaging its local exploration mode with which guides the robot to explore along a mine corridor. When the system reaches a local dead-end, the global planning layer of the method is engaged and provides a new path to guide the robot towards a selected frontier of the explored space. The robot is thus re-positioned to this frontier and upon arrival the local planning mode is enabled again in order to enable the continuation of the exploration mission. Finally, provided a time budget for the mission, the global planner identifies the point that the robot must be commanded to return-to-home and provides an associated reference path. The presented mission is completely autonomous.

[ Robotic Systems Lab ]

Do all Roborock vacuums rock? Vacuum vlog Vacuum Wars did some extensive vacuuming tests to find out.

After testing and reviewing all of the robot vacuums Roborock has released so far, I think its time for me to do a big comparison video showing the differences their various models as well as choosing my favorite Roborock models in 3 different categories.

[ Vacuum Wars ]

Highlights from Lex Fridman’s interview with Jim Keller on Tesla, Elon Musk, Autopilot, and more.

Jim Keller is a legendary microprocessor engineer, having worked at AMD, Apple, Tesla, and now Intel. He’s known for his work on the AMD K7, K8, K12 and Zen microarchitectures, Apple A4, A5 processors, and co-author of the specifications for the x86-64 instruction set and HyperTransport interconnect.

[ Lex Fridman ]

Take a trip down the microworld as roboticists Paul McEuen and Marc Miskin explain how they design and mass-produce microrobots the size of a single cell, powered by atomically thin legs -- and show how these machines could one day be "piloted" to battle crop diseases or study your brain at the level of individual neurons.

[ TED Talks ]

This is a guest post. The views expressed here are solely those of the author and do not represent positions of IEEE Spectrum or the IEEE.

In my mythical free time outside of professorhood, I’m a stand-up comedian and improviser. As a comedian, I’ve often found myself wishing I could banter with modern commercial AI assistants. They don’t have enough comedic skills for my taste! This longing for cheeky AI eventually led me to study autonomous robot comedians, and to teach my own robot how to perform stand-up.

I’ve been fascinated with the relationship between comedy and AI even before I started doing comedy on my own in 2013. When I moved to Los Angeles in 2017 as a postdoctoral scholar for the USC Interaction Lab, I began performing in roughly two booked comedy shows per week, and I found myself with too good of an opportunity for putting a robot onstage to pass up. 

Programming a NAO robot for stand-up comedy is complicated. Some joke concepts came easily, but most were challenging to evoke. It can be tricky to write original comedy for a robot since robots have been part of television and cinema for quite some time. Despite this legacy, we wanted to come up with a perspective for the robot that was fresh and not derivative.

Another challenge was that in my human stand-up comedy, I write almost entirely from real-life experience, and I’ve never been a robot! I tried different thought exercises—imagining myself to be a robot with different annoyances, likes, dislikes, and “life” experiences. My improv comedy training with the Upright Citizens Brigade started to come in handy, as I could play-act being a robot, map classic (and even somewhat overdone) human jokes to fit robot experiences, and imagine things like, “What is a robot family?”, “What is a robot relationship like?”, and “What are drugs for a robot?”

Text-to-speech researchers would probably be astounded by the mounds of SSML that we wrote to get the robot to clearly pronounce phrases that humans have almost certainly never said, such as “I want to backpropagate all over your hidden layers”

As a robotics professor, you never quite know how thousands of dollars of improv classes will come into play in your professional life until they suddenly do! Along the way, I sought inspiration and premises from my comedy colleagues (especially fellow computer scientist/comedian Ajitesh Srivastava), although (at least for now) the robot’s final material is all written by myself and my husband, John. Early in our writing process, we made the awkward misstep of naming the robot Jon as well, and now when people ask how John’s doing, sometimes I don’t know which entity they’re talking about.

Searching for a voice for Jon was also a bit of a puzzle. We found the built-in NAO voice to be too childlike, and many modern text-to-speech voices to be too human-like for the character we were aiming to create. We sought an alternative that was distinctly robotic while still comprehensible, settling on Amazon Polly. Text-to-speech researchers would probably be astounded by the mounds of SSML (Speech Synthesis Markup Language) that we wrote to get the robot to clearly pronounce phrases that humans (or at least humans in the training dataset) have almost certainly never said, such as “I want to backpropagate all over your hidden layers” or “My only solace is re-reading Sheryl Sand-bot’s hit book, ‘Dial In.’” For now, we hand-engineered the SSML and also hand-selected robot movements to layer over each joke. Some efforts have been made by the robotics and NLP communities to automate these types of processes, but I don’t know of any foolproof solution—yet! 

During the first two performances of the robot, I encountered several cases in which the audience could not clearly hear the setup of a joke when they laughed long enough at the previous joke. This lapse in audibility is a big impediment to “getting the joke.” One way to address this problem is to lengthen the pause after each joke:

As shown in the video, this option is workable, but falls short of deftly-timed robot comedy. Luckily, my humble studio apartment contained a full battery of background noises and two expert human laughers. My husband and I modulated all aspects of apartment background noise, cued up laugh tracks, and laughed enthusiastically in search of a sensing strategy that would let the robot pause when it heard uproarious laughter, and then carry on once the crowd calmed down. The resulting audio processing tactic involved counting the number of sounds in each ~0.2-second period after the joke and watching for a moving average-filtered version of this signal to drop below an experimentally-determined threshold.

Human comics not only vie for their jokes to be heard over audience laughter, but they also read the room and adapt to joke success and failure. For maximal entertainment, we wanted our robot to be able to do this, too. By summing the laughter signal described above over the most intense 1 second of the post-joke response, we were able to obtain rudimentary estimates of joke success based on thresholding and filtering the audio signal. This experimental strategy was workable but not perfect; its joke ratings matched labels from a human rater about 60 percent of the time and were judged as different but acceptable an additional 15 percent of the time. The robot used its joke success judgements to decide between possible celebratory or reconciliatory follow-on jokes. Even when the strategy was failing, the robot produced behavior that seemed genuinely sarcastic, which the audience loved.

By this point, we were fairly sure that robot timing and adaptiveness of spoken sequences were important to comedic effectiveness, but we didn’t have any actual empirical evidence of this. As I stepped into my current role as an assistant professor at Oregon State University, it was the perfect time to design an experiment and begin gathering data! We recorded audio from 32 performances of Jon the Robot at comedy venues in Corvallis and Los Angeles, and began to crunch the numbers.

Our results showed that a robot with good timing was significantly funnier–a good confirmation of what the comedy community already expected. Adaptivity actually didn’t make the robot funnier over the course of a full performance, but it did improve the audience’s initial response to jokes about 80 percent of the time.

While this research was certainly fun to conduct, there were also some challenges and missteps along the way. One (half serious/half silly) problem was that we designed the robot to have a male voice, and as soon as I brought it to the heavily male-dominated local comedy scene, the robot quickly began to get more offers of stage time than I did. This felt like a careless oversight on my part—my own male-voiced robot was taking away my stage time! (Or sometimes I gave it up to Jon the Robot, for the sake of data.)

Some individual crowd members mildly heckled the robot. One audience member angrily left the performance, grumbling at the robot to “write your own jokes.” 

All of the robot’s audiences were very receptive, but some individual crowd members mildly heckled the robot. Because of our carefully-crafted writing, most of these hecklers were eventually won over by the robot’s active evaluation of the crowd, but a few weren’t. One audience member angrily left the performance, grumbling directly at the robot to “write your own jokes.”  While all of Jon’s jokes are original material, the robot doesn’t know how to generate its own comedy—at least, not that we’re ready to tell you about yet.

Writing comedy material for robots, especially as a roboticist myself, also can feel like a bit of a minefield. It’s easy to get people to laugh at quips about robot takeovers, and robot jokes that are R-rated are also reliably funny, if not particularly creative. Getting the attendees of a performance to learn something about robotics while also enjoying themselves is of great interest to me as a robotics professor, but comedy shows can lose momentum if they turn too instructional. My current approach to writing material for shows includes a bit of all of the above concepts—in the end, simply getting people to genuinely laugh is a great triumph. 

Hopefully by now you’re excited about robot comedy! If so, you’re in luck– Jon the Robot performs quarterly in Corvallis, Ore., and is going on tour, starting with the ACM/IEEE International Conference on Human-Robot Interaction this year in Cambridge, U.K. And trust me—there’s nothing like “live”—er, well, “physically embodied”—robot comedy!

Naomi Fitter is an assistant professor in the Collaborative Robotics and Intelligent Systems (CoRIS) Institute at Oregon State University, where her Social Haptics, Assistive Robotics, and Embodiment (SHARE) research group aims to equip robots with the ability to engage and empower people in interactions from playful high-fives to challenging physical therapy routines. She completed her doctoral work in the GRASP Laboratory’s Haptics Group and was a postdoctoral scholar in the University of Southern California’s Interaction Lab from 2017 to 2018. Naomi’s not-so-secret pastime is performing stand-up and improv comedy.

We’ve all seen drone displays—massive swarms of tiny drones, each carrying a light, that swarm together in carefully choreographed patterns to form giant (albeit very low resolution) 3D shapes in the sky at night. It’s cool, but it’s not particularly novel anymore, and without thousands of drones, the amount of detail that you can expect out of the display is not all that great.

CollMot Entertainment, a Hungarian company that puts on traditional drone shows, has been working on something a little bit different. Instead of using drones as pixels, they’ve developed a system that uses drones to generate an enormous screen in the sky, and then laser projectors draw on that screen to create “the largest 3D display you have ever seen.”

The video appears to show an array of drones carrying smoke generators, which collaboratively create a backdrop that can reflect laser light that’s projected from the ground. CollMot, based in Budapest, and Phase 7, a German company, developed the technology together. They want to keep the details under wraps for now, but we got some additional information from Csilla Vitályos, head of business development at CollMot.

IEEE Spectrum: Can you describe what the “drone-laser technology” is and how the system operates?

Drone-laser technology is a special combination of our drone swarms and a ground based or aerial laser. The intelligent drone swarm creates a giant canvas in the air with uniquely controlled smoke machines and real-time active swarm control. The laser projects onto this special aerial smoke canvas, creating the largest 2D and 3D laser displays ever seen.

What exactly are we seeing in the video?

This video shows how much more we can visualize with such technology compared to individual light dots represented by standard drone shows. The footage was taken on one of our tests out in the field, producing shiny 3D laser images of around 50 to 150 meters in width up in the air.

Image: CollMot Entertainment

What are the technical specifications of the system?

We work with a drone fleet of 10 to 50 special intelligent drones to accomplish such a production, which can last for several minutes and can contain very detailed custom visuals. Creating a stable visual without proper technology and experience is very challenging as there are several environmental parameters that affect the results. We have put a lot of time and energy into our experiments lately to find the best solutions for such holographic-like aerial displays.

What is unique about this system, and what can it do that other drone display technologies can’t?

The most stunning difference compared to standard drone shows (what we actually also provide and also like a lot) is that while in usual drone light shows each drone is a single pixel on the sky, here we can visualize colorful lines and curves as well. A point is zero dimensional, a line is one dimensional. Try to draw something with a limited number of points and try to do the same with lines. You will experience the difference immediately.

Can you share anything else about the system?

At this point we would like to keep the drone-related technical details as part of our secret formula but we are more than happy to present our technology’s scope of application at events in the future.

[ CollMot ]

David Zarrouk’s lab at Ben Gurion University, in Israel, is well known for developing creative, highly mobile robots that use a minimal number of actuators. Their latest robot is called RCTR (Reconfigurable Continuous Track Robot), and it manages to change its entire body shape on a link-by-link basis, using just one extra actuator to “build its own track in the air as it advances.”

The concept behind this robot is similar to Zarrouk’s reconfigurable robotic arm, which we wrote about a few years ago. That arm is made up of a bunch of links that are attached to each other through passive joints, and a little robotic module can travel across those links and adjust the angle of each joint separately to reconfigure the arm. 

Image: Ben Gurion University The robot’s locking mechanism (located in the front of the robot’s body) can lock the track links at a 20° angle (A) or a straight angle (B), or it can keep the track links unlocked (C).

RCTR takes this idea and flips it around, so that instead of an actuator moving along a bunch of flexible links, you have a bunch of flexible links (the track) moving across an actuator. Each link in the track has a locking pin, and depending on what the actuator is set to when that link moves across it, the locking pin can be engaged such that the following link gets fixed at a relative angle of either zero degrees or 20 degrees. It’s this ability to lock the links of the track—turning the robot from flexible to stiff—that allows RCTR to rear up to pass over an obstacle, and do the other stuff that you can see in the video. And to keep the robot from fighting against its own tracks, the rear of the robot has a passive system that disengages the locking pins on every link to reset the flexibility of the track as it passes over the top. 

The biggest downside to this robot is that it’s not able to, uh, steer. Adding steering wouldn’t be particularly difficult, although it would mean a hardware redesign: the simplest solution is likely to do what most other tracked vehicles do, and use a pair of tracks and skid-steering, although you could also attach two modules front to back with a powered hinge between them. The researchers are also working on a locomotion planning algorithm for handling a variety of terrain, presumably by working out the best combination of rigid and flexible links to apply to different obstacles.

“A Minimally Actuated Reconfigurable Continuous Track Robot,” by Tal Kislassi and David Zarrouk from Ben Gurion University in Israel, is published in IEEE Robotics and Automation Letters.

[ RA-L ] via [ BGU ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France

Let us know if you have suggestions for next week, and enjoy today’s videos.

Automaton contributor Fan Shi, who helps with our coverage of robotics in Asia, shared a few videos from China showing ways in which robots might be useful to help combat the spread of the deadly coronavirus. These include using robots to deliver medicine, food, and disinfect rooms.

And according to some reports, doctors at a Seattle area hospital are using a telepresence robot to treat a man infected with the virus, the first confirmed case of coronavirus in the United States.

Watch until 0:44 to get your mind blown by MiniCheetah.

[ MIT ]

This new video from Logistics Gliders shows more footage of how these disposable cargo UAVs land. It’s not pretty, but it’s very cost effective.

[ Logistics Gliders ]

Thanks Marti!

At the KUKA Innovation Award 2019 about 30 research teams from all over the world applied with their concepts on the topic of Healthy Living. The applicants were asked to develop an innovative concept using the KUKA LBR Med for the use in hospitals and rehabilitation centers. At MEDICA, the world's largest medical fair, the teams of the 5 finalists presented their innovative applications.

[ Kuka ]

Unlike most dogs, I think Aibo is cuter with transparent skin.

[ Aibo ] via [ RobotStart ]

We’ve written extensively about Realtime Robotics, and here’s their motion-planning software running on a couple of collision-prone picking robots at IREX.

[ Realtime Robotics ] via [ sbbit ]

Tech United is already looking hard to beat for RoboCup 2020.

[ Tech United ]

In its third field experiment, DARPA's OFFensive Swarm-Enabled Tactics (OFFSET) program deployed swarms of autonomous air and ground vehicles to demonstrate a raid in an urban area. The field experiment took place at the Combined Arms Collective Training Facility (CACTF) at the Camp Shelby Joint Forces Training Center in Mississippi.

The OFFSET program envisions swarms of up to 250 collaborative autonomous systems providing critical insights to small ground units in urban areas where limited sight lines and tight spaces can obscure hazards, as well as constrain mobility and communications.

[ DARPA ]

Looks like one of Morgan Pope’s robotic acrobats is suiting up for Disney:

[ Disney ] via [ Gizmodo ]

Here are some brief video highlights of the more unusual robots that were on display at IREX—including faceless robot baby Hiro-chan—from Japanese tech journalist Kazumichi Moriyama.

[ sbbit ]

The Oxford Dynamic Robot Systems Group has six papers at ICRA this year, and they’ve put together this teaser video.

[ DRS ]

Pepper and NAO had a busy 2019:

[ Softbank ]

Let’s talk about science! Watch the fourth episode of our #EZScience series to learn about NASA’s upcoming Mars 2020 rover mission by looking back at the Mars Pathfinder mission and Sojourner rover. Discover the innovative elements of Mars 2020 (including a small solar-powered helicopter!) and what we hope to learn about the Red Planet when our new rover arrives in February 2021.

[ NASA ]

Chen Li from JHU gave a talk about how snakes climb stairs, which is an important thing to know.

[ LCSR ]

This week’s CMU RI Seminar comes from Hadas Kress-Gazit at Cornell, on “Formal Synthesis for Robots.”

In this talk I will describe how formal methods such as synthesis – automatically creating a system from a formal specification – can be leveraged to design robots, explain and provide guarantees for their behavior, and even identify skills they might be missing. I will discuss the benefits and challenges of synthesis techniques and will give examples of different robotic systems including modular robots, swarms and robots interacting with people.

[ CMU RI ]

Drones of all sorts are getting smaller and cheaper, and that’s great—it makes them more accessible to everyone, and opens up new use cases for which big expensive drones would be, you know, too big and expensive. The problem with very small drones, particularly those with fixed-wing designs, is that they tend to be inefficient fliers, and are very susceptible to wind gusts as well as air turbulence caused by objects that they might be flying close to. Unfortunately, designing for resilience and designing for efficiency are two different things: Efficient wings are long and thin, and resilient wings are short and fat. You can’t really do both at the same time, but that’s okay, because if you tried to make long and thin wings for micro aerial vehicles (MAVs) they’d likely just snap off. So stubby wings it is!

In a paper published this week in Science Robotics, researchers from Brown University and EPFL are presenting a new wing design that’s able to deliver both highly efficient flight and robustness to turbulence at the same time. A prototype 100-gram MAV using this wing design can fly for nearly 3 hours, which is four times longer than similar drones with conventional wings. How did they come up with a wing design that offered such a massive improvement? Well, they didn’t— they stole it, from birds.

Conventional airfoils work best when you have airflow that “sticks” to the wing over as much of the wing surface as possible. When flow over an airfoil separates from the surface of the wing, it leads to a bunch of turbulence over the wing and a loss of lift. Aircraft wings employ all kinds of tricks to minimize flow separation, like leading edge extensions and vortex generators. Flow separation can lead to abrupt changes in lift, to loss of control, and to stalls. Flow separation is bad.

For many large insects and small birds, though, flow separation is just how they roll. In fact,  many small birds have wing features that have evolved specifically to cause flow separation right at the leading edge of the wing. Why would you want that if flow separation is bad? It turns out that flow separation is mostly bad for traditional airfoil designs, where it can be unpredictable and difficult to manage. But if you design a wing around flow separation, controlling where it happens and how the resulting turbulent flow over the wing is managed, things aren’t so bad. Actually, things can be pretty good. Since most of your wing is in turbulent airflow all the time, it’s highly resistant to any other turbulent air that your MAV might be flying through, which is a big problem for tiny outdoor fliers.

Image: Brown/EPFL/Science Robotics Photo of the MAV with the top surface of the wing removed to show how batteries and electronics are integrated inside. A diagram (bottom) shows the section of the bio-inspired airfoil, indicating how the flow separates at the sharp leading edge, transitions to turbulence, and reattaches over the flap.

In the MAV demonstrator created by the researchers, the wing (or SFA, for separated flow airfoil) is completely flat, like a piece of plywood, and the square front causes flow separation right at the leading edge of the wing. There’s an area of separated, turbulent flow over the front half of the wing, and then a rounded flap that hangs off the trailing edge of the wing pulls the flow back down again as air moving over the plate speeds up to pass over the flap. 

You may have noticed that there’s an area over the front 40 percent of the wing where the flow has separated (called a “separation bubble”), lowering lift efficiency over that section of the wing. This does mean that the maximum aerodynamic efficiency of the SFA is somewhat lower than you can get with a more conventional airfoil, where separation bubbles are avoided and more of the wing generates lift. However, the SFA design more than makes up for this with its wing aspect ratio—the ratio of wing length to wing width. Low aspect ratio wings are short and fat, while high aspect ratio wings are long and thin, and the higher the aspect ratio, the more efficient the wing is.

The SFA MAV has wings with an aspect ratio of 6, while similarly sized MAVs have wings with aspect ratios of between 1 and 2.5. Since lift-to-drag ratio increases with aspect ratio, that makes a huge difference to efficiency. In general, you tend to see those stubby low aspect ratio wings on MAVs because it’s difficult to structurally support long, thin, high aspect ratio wings on small platforms. But since the SFA MAV has no use for the conventional aerodynamics of traditional contoured wings, it just uses high aspect ratio wings that are thick enough to support themselves, and this comes with some other benefits. Thick wings can be stuffed full of batteries, and with batteries (and other payload) in the wings, you don’t need a fuselage anymore. With a MAV that’s basically all wing, the propeller in front sends high speed airflow directly over the center section of the wing itself, boosting lift by 20 to 30 percent, which is huge.

The challenge moving forward, say the researchers, is that current modeling tools can’t really handle the complex aerodynamics of the separated flow wing. They’ve been doing experiments in a wind tunnel, but it’s difficult to optimize the design that way. Still, it seems like the potential for consistent, predictable performance even under turbulence, increased efficiency, and being able to stuff a bunch of payload directly into a chunky wing could be very, very useful for the next generation of micro (and nano) air vehicles.

“A bioinspired Separated Flow wing provides turbulence resilience and aerodynamic efficiency for miniature drones,” by Matteo Di Luca, Stefano Mintchev, Yunxing Su, Eric Shaw, and Kenneth Breuer from Brown University and EPFL, appears in Science Robotics.

[ Science Robotics ]

It’s going to be a very, very long time before robots come anywhere close to matching the power-efficient mobility of animals, especially at small scales. Lots of folks are working on making tiny robots, but another option is to just hijack animals directly, by turning them into cyborgs. We’ve seen this sort of thing before with beetles, but there are many other animals out there that can be cyborgized. Researchers at Stanford and Caltech are giving sea jellies a try, and remarkably, it seems as though cyborg enhancements actually make the jellies more capable than they were before.

Usually, co-opting the mobility system of an animal with electronics doesn’t improve things for the animal, because we’re not nearly as good at controlling animals as they are at controlling themselves. But when you look at animals with very simple control systems, like sea jellies, it turns out that with some carefully targeted stimulation, they can move faster and more efficiently than they do naturally.

The researchers, Nicole W. Xu and John O. Dabiri, chose a friendly sort of sea jelly called Aurelia aurita, which is “an oblate species of jellyfish comprising a flexible mesogleal bell and monolayer of coronal and radial muscles that line the subumbrellar surface,” so there you go. To swim, jellies actuate the muscles in their bells, which squeeze water out and propel them forwards. These muscle contractions are controlled by a relatively simple stimulus of the jelly’s nervous system that can be replicated through external electrical impulses. 

To turn the sea jellies into cyborgs, the researchers developed an implant consisting of a battery, microelectronics, and bits of cork and stainless steel to make things neutrally buoyant, plus a wooden pin, which was used to gently impale each jelly through the bell to hold everything in place. While non-cyborg jellies tended to swim with a bell contraction frequency of 0.25 Hz, the implant allowed the researchers to crank the cyborg jellies up to a swimming frequency of 1 Hz.

While non-cyborg jellies tended to swim with a bell contraction frequency of 0.25 Hz, the implant allowed the researchers to crank the cyborg jellies up to a swimming frequency of 1 Hz

Peak speed was achieved at 0.62 Hz, resulting in the jellies traveling at nearly half a body diameter per second (4-6 centimeters per second), which is 2.8x their typical speed. More importantly, calculating the cost of transport for the jellies showed that the 2.8x increase in speed came with only a 2x increase in metabolic cost, meaning that the cyborg sea jelly is both faster and more efficient.

This is a little bit weird from an evolutionary standpoint—if a sea jelly has the ability to move faster, and moving faster is more efficient for it, then why doesn’t it just move faster all the time? The researchers think it may have something to do with feeding:

A possible explanation for the existence of more proficient and efficient swimming at nonnatural bell contraction frequencies stems from the multipurpose function of vortices shed during swimming. Vortex formation serves not only for locomotion but also to enable filter feeding and reproduction. There may therefore be no evolutionary pressure for A. aurita to use its full propulsive capabilities in nature, and there is apparently no significant cost associated with maintaining those capabilities in a dormant state, although higher speeds might limit the animals’ ability to feed as effectively.

Image: Science Advances

Sea jelly with a swim controller implant consisting of a battery, microelectronics, electrodes, and bits of cork and stainless steel to make things neutrally buoyant. The implant includes a wooden pin that is gently inserted through the jelly’s bell to hold everything in place, with electrodes embedded into the muscle and mesogleal tissue near the bell margin.

The really nice thing about relying on cyborgs instead of robots is that many of the advantages of a living organism are preserved. A cyborg sea jelly is perfectly capable of refueling itself as well as making any necessary repairs to its structure and function. And with an energy efficiency that’s anywhere from 10 to 1000 times more efficient than existing swimming robots, adding a control system and a couple of sensors could potentially lead to a useful biohybrid monitoring system.

Lastly, in case you’re concerned about the welfare of the sea jellies, which I definitely was, the researchers did try to keep them mostly healthy and happy (or at least as happy as an invertebrate with no central nervous system can be), despite stabbing them through the bell with a wooden pin. They were all allowed to take naps (or the sea jelly equivalent) in between experiments, and the bell piercing would heal up after just a couple of days. All animals recovered post-experiments, the researchers say, although a few had “bell deformities” from being cooped up in a rectangular fish tank for too long rather than being returned to their jelliquarium. Also, jelliquariums are a thing and I want one.

You may have noticed that over the course of this article, I have been passive-aggressively using the term “sea jelly” rather than “jellyfish.” This is because jellyfish are not fish at all—you are more closely related to a fish than a jellyfish is, which is why “sea jelly” is the more accurate term that will make marine biologists happy. And just as jellyfish should properly be called sea jellies, starfish should be called sea stars, and cuttlefish should be called sea cuttles. The last one is totally legit, don’t even question it.

“Low-power microelectronics embedded in live jellyfish enhance propulsion,” by Nicole W. Xu and John O. Dabiri from Stanford University and Caltech, is published in Science Advances.

[ Science Advances ]

When the going gets tough, future soft robots may break into a sweat to keep from overheating, much like marathon runners or ancient hunters chasing prey in the savannah, a new study finds.

Whereas conventional robots are made of rigid parts vulnerable to bumps, scrapes, twists, and falls, soft robots inspired by starfish, worms, and octopuses can resist many such kinds of damage and squirm past obstacles. Soft robots are also often cheaper and simpler to make, comparatively lightweight, and safer for people to be around.

However, the rubbery materials that make up soft robots often trap heat, exacerbating problems caused by overheating. Moreover, conventional devices used to control heat such as radiators and fans are typically made of rigid materials that are incompatible with soft robotics, says T.J. Wallin, a co-author and research scientist at Facebook Reality Labs.

To solve this problem, scientists decided to build robots that could sweat. "It turns out that the ability to perspire is one of the most remarkable features of humans," Wallin says. "We're not the fastest animals, but early humans found success as persistence hunters—the combination of sweating, relative hairlessness, and upright bipedal gait enabled us to physically exhaust our prey over prolonged chases."

"An elite marathon runner in the right conditions has been known to lose almost four liters of sweat an hour—this corresponds to roughly 2.5 kilowatts of cooling capacity," Wallin says. "To put that in perspective, refrigerators only use approximately 1 kilowatt-hour of energy. So as is often the case, biology provided an excellent guide for us engineers."

The researchers 3D-printed soft robot fingers that were hollow like balloons. These could bend or straighten to grip or drop objects, depending on the level of water pressure within each finger.

The robot fingers were each made of two different kinds of soft, flexible resin. The body of each finger was made of a resin that shrunk when heated above 40 degrees C, whereas the back of each finger was capped with a resin that expanded when heated above 30 degrees C.

The back of each finger was also dotted with microscopic pores. At temperatures cooler than 30 degrees C, these pores remained closed. However, at higher temperatures, the material on the back of each finger expanded, dilating the pores and letting the water in each finger sweat out. Moreover, as the heat rose, the material that made up the body of each finger shrank, helping squeeze out water.

"The best part of this synthetic strategy is that the thermoregulatory performance is baked into the material itself. We did not need to add sensors or other components to control the sweating rate—when the local temperature rose above the transition point, the pores would simply open and close on their own," Wallin says.

When exposed to wind from a fan, the sweaty fingers cooled off by about 39 degrees C per minute, or roughly six times faster than their dry counterparts. The amount by which the sweaty fingers cooled (about 107 watts per kilogram) also greatly exceeded the best cooling performance seen in animals (about 35 watts per kilogram, as seen in horses and humans), the scientists add.

"Much like in biology, where we have to manage internal heat through perspiring skin, we took inspiration and created sweat for high cooling power," says Robert Shepherd, a co-author and mechanical engineer at Cornell University.

"I think in order for the robot to operate with the sweating we have created, it would also have to be able to drink." —Robert Shepherd, Cornell University

Although sweat could make robot fingers more slippery, the researchers could design robot skin that wrinkles just like human fingers do when they get wet in order "to enhance gripping," Shepherd says.

Chemicals might also get added to robot sweat to remove contaminants from whatever they are touching, coat the surfaces of robots with a protective layer, or dissolve something they are touching. And the robot could then suck in whatever substance got dissolved to analyze it, Shepherd adds.

One problem these robot fingers experienced was how sweating reduced pressure within them, impairing their mobility. Future versions could separate the water networks behind sweating and mobility, at the expense of greater complexity, Wallin says.

There is also currently no way for sweating robots to replenish the water they lose. "The answer is right in front of me—I'm drinking some coffee right now," Shepherd says. "I think in order for the robot to operate with the sweating we have created, it would also have to be able to drink."

Another drawback that artificial perspiration might face is that it would likely not help much when the sweat cannot evaporate to cool robots, such as when the machines are underwater or when the air is very humid. "However, I would like to point out other heat transfer strategies, such as conduction, convection, and radiation, are ineffective at lowering the temperature of the body when it is below that of the environment, whereas sweating and evaporative water loss can do that," Wallin says. "In some ways it's a trade-off, but we feel it is an important benefit."

The scientists detailed their findings online on 29 January in the journal Science Robotics.

Two years ago, we wrote about an AI startup from UC Berkeley and OpenAI called Embodied Intelligence, founded by robot laundry-folding expert Pieter Abbeel. What exactly Embodied was going to do wasn’t entirely clear, and honestly, it seemed like Embodied itself didn’t really know—they talked about “building technology that enables existing robot hardware to handle a much wider range of tasks where existing solutions break down,” and gave some examples of how that might be applied (including in manufacturing and logistics), but nothing more concrete.

Since then, a few things have happened. Thing one is that Embodied is now Covariant.ai. Thing two is that Covariant.ai spent almost a year talking with literally hundreds of different companies about how smarter robots could potentially make a difference for them. These companies represent sectors that include electronics manufacturing, car manufacturing, textiles, bio labs, construction, farming, hotels, elder care—“pretty much anything you could think about where maybe a robot could be helpful,” Pieter Abbeel tells us. “Over time, it became clear to us that manufacturing and logistics are the two spaces where there’s most demand now, and logistics especially is just hurting really hard for more automation.” And the really hard part of logistics is what Covariant decided to tackle.

There’s already a huge amount of automation in logistics, but as Abbeel explains, in warehouses there are two separate categories that need automation: “The things that people do with their legs and the things that people do with their hands.” The leg automation has largely been taken care of over the last five or 10 years through a mixture of conveyor systems, mobile retrieval systems, Kiva-like mobile shelving, and other mobile robots. “The pressure now is on the hand part,” Abbeel says. “It’s about how to be more efficient with things that are done in warehouses with human hands.”

A huge chunk of human-hand tasks in warehouses comes down to picking. That is, taking products out of one box and putting them into another box. In the logistics industry, the boxes are usually called totes, and each individual kind of product is referred to by its stock keeping unit number, or SKU. Big warehouses can have anywhere from thousands to millions of SKUs, which poses an enormous challenge to automated systems. As a result, most existing automated picking systems in warehouses are fairly limited. Either they’re specifically designed to pick a particular class of things, or they have to be trained to recognize more or less every individual thing you want them to pick. Obviously, in warehouses with millions of different SKUs, traditional methods of recognizing or modeling specific objects is not only impractical in the short term, but would also be virtually impossible to scale.

This is why humans are still used in picking—we have the ability to generalize. We can look at an object and understand how to pick it up because we have a lifetime of experience with object recognition and manipulation. We’re incredibly good at it, and robots aren’t. “From the very beginning, our vision was to ultimately work on very general robotic manipulation tasks,” says Abbeel. “The way automation’s going to expand is going to be robots that are capable of seeing what’s around them, adapting to what’s around them, and learning things on the fly.”

Covariant is tackling this with relatively simple hardware, including an off-the-shelf industrial arm (which can be just about any arm), a suction gripper (more on that later), and a straightforward 2D camera system that doesn’t rely on lasers or pattern projection or anything like that. What couples the vision system to the suction gripper is one single (and very, very large) neural network, which is what helps Covariant to be cost effective for customers. “We can’t have specialized networks,” says Abbeel. “It has to be a single network able to handle any kind of SKU, any kind of picking station. In terms of being able to understand what’s happening and what’s the right thing to do, that’s all unified. We call it Covariant Brain, and it’s obviously not a human brain, but it’s the same notion that a single neural network can do it all.”

We can talk about the challenges of putting picking robots in warehouses all day, but the reason why Covariant is making this announcement now is because their system has been up and running reliably and cost effectively in a real warehouse in Germany for the last four months. 

This video is showing Covariant’s robotic picking system operating (for over an hour at 10x speed) in a warehouse that handles logistics for a company called Obeta, which overnights orders of electrical supplies to electricians in Germany. The robot’s job is to pick items from bulk storage totes, and add them to individual order boxes for shipping. The warehouse is managed by an automated logistics company called KNAPP, which is Covariant’s first partner. “We were searching a long time for the right partner,” says Peter Puchwein, vice president of innovation at KNAPP. “We looked at every solution out there. Covariant is the only one that’s ready for real production.” He explains that Covariant’s AI is able to detect glossy, shiny, and reflective products, including products in plastic bags. “The product range is nearly unlimited, and the robotic picking station has the same or better performance than humans.”

The key to being able to pick such a wide range of products so reliably, explains Abbeel, is being able to generalize. “Our system generalizes to items it’s never seen before. Being able to look at a scene and understand how to interact with individual items in a tote, including items it’s never seen before—humans can do this, and that’s essentially generalized intelligence,” he says. “This generalized understanding of what’s in a bin is really key to success. That’s the difference between a traditional system where you would catalog everything ahead of time and try to recognize everything in the catalog, versus fast-moving warehouses where you have many SKUs and they’re always changing. That’s the core of the intelligence that we’re building.”

To be sure, the details on how Covariant’s technology work are still vague, but we tried to extract some more specifics from Abbeel, particularly about the machine learning components. Here’s the rest of our conversation with him:

IEEE Spectrum: How was your system trained initially?

Pieter Abbeel: We would get a lot of data on what kind of SKUs our customer has, get similar SKUs in our headquarters, and just train, train, train on those SKUs. But it’s not just a matter of getting more data. Actually, often there’s a clear limit on a neural net where it’s saturating. Like, we give it more data and more data, but it’s not doing any better, so clearly the neural net doesn’t have the capacity to learn about these new missing pieces. And then the question is, what can we do to re-architect it to learn about this aspect or that aspect that it’s clearly missing out on?

You’ve done a lot of work on sim2real transfer—did you end up using a bajillion simulated arms in this training, or did you have to rely on real-world training?

We found that you need to use both. You need to work both in simulation and the real world to get things to work. And as you’re continually trying to improve your system, you need a whole different kind of testing: You need traditional software unit tests, but you also need to run things in simulation, you need to run it on a real robot, and you need to also be able to test it in the actual facility. It’s a lot more levels of testing when you’re dealing with real physical systems, and those tests require a lot of time and effort to put in place because you may think you’re improving something, but you have to make sure that it’s actually being improved.

What happens if you need to train your system for a totally new class of items?

The first thing we do is we just put new things in front of our robot and see what happens, and often it’ll just work. Our system has few-shot adaptation, meaning that on-the-fly, without us doing anything, when it doesn’t succeed it’ll update its understanding of the scene and try some new things. That makes it a lot more robust in many ways, because if anything noisy or weird happens, or there’s something a little bit new but not that new, you might do a second or third attempt and try some new things.

But of course, there are going to be scenarios where the SKU set is so different from anything it’s been trained on so far that some things are not going to work, and we’ll have to just collect a bunch of new data—what does the robot need to understand about these types of SKUs, how to approach them, how to pick them up. We can use imitation learning, or the robot can try on its own, because with suction, it’s actually not too hard to detect if a robot succeeds or fails. You can get a reward signal for reinforcement learning. But you don’t want to just use RL, because RL is notorious for taking a long time, so we bootstrap it off some imitation and then from there, RL can complete everything. 

Why did you choose a suction gripper?

What’s currently deployed is the suction gripper, because we knew it was going to do the job in this deployment, but if you think about it from a technological point of view, we also actually have a single neural net that uses different grippers. I can’t say exactly how it’s done, but at a high level, your robot is going to take an action based on visual input, but also based on the gripper that’s attached to it, and you can also represent a gripper visually in some way, like a pattern of where the suction cups are. And so, we can condition a single neural network on both what it sees and the end-effector it has available. This makes it possible to hot-swap grippers if you want to. You lose some time, so you don’t want to swap too often, but you could swap between a suction gripper and a parallel gripper, because the same neural network can use different gripping strategies.

And I would say this is a very common thread in everything we do. We really wanted to be a single, general system that can share all its learnings across different modalities, whether it’s SKUs, end of arm tools, different bins you pick from, or other things that might be different. The expertise should all be sharable.

“People often say neural networks are just black boxes and if you’re doing something new you have to start from scratch. That’s not really true . . . Their strength comes from the fact that you can train end-to-end, you can train from input to the desired output” —Pieter Abbeel, Covariant.ai

And one single neural net is versatile enough for this?

People often say neural networks are just black boxes and if you’re doing something new you have to start from scratch. That’s not really true. I don’t think what’s important about neural nets is that they’re black boxes—that’s not really where their strength comes from. Their strength comes from the fact that you can train end-to-end, you can train from input to the desired output. And you can put modular things in there, like neural nets that are an architecture that’s well suited to visual information, versus end-effector information, and then they can merge their information loads to come to a conclusion. And the beauty is that you can train it all together, no problem.

When your system fails at a pick, what are the consequences?

Here’s where things get very interesting. You think about bringing AI into the physical world—AI has been very successful already in the digital world, but the digital world is much more forgiving. There’s a long tail of scenarios that you could encounter in the real world and you haven’t trained against them, or you haven’t hardcoded against them. And that’s what makes it so hard and why you need really good generalization including few-shot adaptation and so forth. 

Now let’s say you want a system to create value. For a robot in a warehouse, does it need to be 100 percent successful? No, it doesn’t. If, say, it takes a few attempts to pick something, that’s just a slowdown. It’s really the overall successful picks per hour that matter, not how often you have to try to get those picks. And so if periodically it has to try twice, it’s really the picking rate that’s affected, not the success rate that’s affected. A true failure is one where human intervention is needed.

With true failures, where after repeated attempts the robot just can’t pick an item, we’ll get notified by that and we can then train on it, and the next day it might work, but at that moment it doesn’t work. And even if a robotic deployment works 90 percent of the time, that’s not good enough. A human picking station can range from 300 to 2000 picks per hour. 2000 is really rare and is peak pick for a very short amount of time, so if we look at the bottom of that range, 300 picks per hour, if we’re succeeding 90 percent, that means 30 failures per hour. Wow, that’s bad. At 30 fails per hour, fixing those up by a human probably takes more than an hour’s worth of work. So what you’ve done now is you’ve created more work than you save, so 90 percent is definitely a no go. 

At 99 percent that’s 3 failures per hour. If it usually takes a couple of minutes for a human to fix, at that point, a human could oversee 10 stations easily, and that’s where all of a sudden we’re creating value. Or a human could do another job, and just keep an eye on the station and jump in for a moment to make sure it keeps running. If you had a 1000 per hour station, you’d need closer to 99.9 percent to get there and so forth, but that’s essentially the calculus we’ve been doing. And that’s what you realize how any extra nine you want to get is so much more challenging than the previous nine you’ve already achieved.

Photo: Elena Zhukova Covariant co-founders (left to right): Tianhao Zhang, Rocky Duan, Peter Chen, and Pieter Abbeel.

There are other companies that are developing using similar approaches to picking—industrial arms, vision systems, suction grippers, neural networks. What makes Covariant’s system work better?

I think it’s a combination of things. First of all, we want to bring to bear any kind of learning—imitation learning, supervised learning, reinforcement learning, all the different kinds of learning you can. And you also want to be smart about how you collect data—what data you collect, what processes you have in place to get the data that you need to improve the system. Then related to that, sometimes it’s not just a matter of data anymore, it’s a matter of, you need to re-architect your neural net. A lot of deep learning progress is made that way, where you come up with new architectures and the new architecture allows you to learn something that otherwise would maybe not be possible to learn. I mean, it’s really all of those things brought together that are giving the results that we’re seeing. So it’s not really like any one that can be singled out as “this is the thing.”

Also, it’s just a really hard problem. If you look at the amount of AI research that was needed to make this work... We started with four people, and we have 40 people now. About half of us are AI researchers, we have some world-leading AI researchers, and I think that’s what’s made the difference. I mean, I know that’s what’s made the difference. 

So it’s not like you’ve developed some sort of crazy new technology or something?

There’s no hardware trick. And we’re not doing, I don’t know, fuzzy logic or something else out of left field all of a sudden. It’s really about the AI stuff that processes everything—underneath it all is a gigantic neural network. 

Okay, then how the heck are you actually making this work?

If you have an extremely uniquely qualified team and you’ve picked the right problem to work on, you can do something that is quite out there compared to what has otherwise been possible. In academic research, people write a paper, and everybody else catches up the moment the paper comes out. We’ve not been doing that—so far we haven’t shared the details of what we actually did to make our system work, because right now we have a technology advantage. I think there will be a day when we will be sharing some of these things, but it’s not going to be anytime soon. 

It probably won’t surprise you that Covariant has been able to lock down plenty of funding (US $27 million so far), but what’s more interesting is some of the individual investors who are now involved with Covariant, which include Geoff Hinton, Fei-Fei Li, Yann LeCun, Raquel Urtasun, Anca Dragan, Michael I. Jordan, Vlad Mnih, Daniela Rus, Dawn Song, and Jeff Dean

While we’re expecting to see more deployments of Covariant’s software in picking applications, it’s also worth mentioning that their press release is much more general about how their AI could be used:

The Covariant Brain [is] universal AI for robots that can be applied to any use case or customer environment. Covariant robots learn general abilities such as robust 3D perception, physical affordances of objects, few-shot learning and real-time motion planning, which enables them to quickly learn to manipulate objects without being told what to do. 

Today, [our] robots are all in logistics, but there is nothing in our architecture that limits it to logistics. In the future we look forward to further building out the Covariant Brain to power ever more robots in industrial-scale settings, including manufacturing, agriculture, hospitality, commercial kitchens and eventually, people’s homes.

Fundamentally, Covariant is attempting to connect sensing with manipulation using a neural network in a way that can potentially be applied to almost anything. Logistics is the obvious first application, since the value there is huge, and even though the ability to generalize is important, there are still plenty of robot-friendly constraints on the task and the environment as well as safe and low-impact ways to fail. As to whether this technology will effectively translate into the kinds of semi-structured and unstructured environments that have historically posed such a challenge for general purpose manipulation (notably, people’s homes)—as much as we love speculating, it’s probably too early even for that.

What we can say for certain is that Covariant’s approach looks promising both in its present implementation and its future potential, and we’re excited to see where they take it from here.

[ Covariant.ai ]

Japan has had a robust robot culture for decades, thanks (at least in part) to the success of the Gundam series, which are bipedal humanoid robots controlled by a human who rides inside of them. I would tell you how many different TV series and video games and manga there are about Gundam, but I’m certain I can’t count that high—there’s like seriously a lot of Gundam stuff out there. One of the most visible bits of Gundam stuff is a real life full-scale Gundam statue in Tokyo, but who really wants a statue, right? C’mon, Japan! Bring us the real thing!

Gundam Factory Yokohama, which is a Gundam Factory in Yokohama, is constructing an 18-meter-tall, 25-ton Gundam robot. The plan is for the robot to be fully actuated using a combination of electric and hydraulic actuators, achieving “Gundam-like movement” with its 24 degrees of freedom. This will include the ability to walk, which has already been simulated by the University of Tokyo JSK Lab:

Video: Kazumichi Moriyama/Impress

As we all know, simulation is pretty much just as good as reality, which is good because so far simulation is all we have of this robot, including these 1/30 scale models of the robot and the docking and maintenance facility that will be built for it:

Video: RobotStart

Apparently, the robot is coupled to a mobile support system (“Gundam Carrier”) that can move the robot in and out of the docking infrastructure, and perhaps provide power and support while the robot takes a step or two backwards and forwards, but it’s really not at all clear at this point how it’s all supposed to work. And it looks that when the robot does move, it’ll be remote controlled and spectators will be restricted to watching from a nearby building, which experience with watching large robots walk tells us is probably in the best interests of everyone.

Image: Sotsu/Sunrise/Gundam Factory Yokohama

The current schedule is for the robot to be open to the public by October, which seems like it’ll be a challenge—but if anyone can do it, it’s Gundam Factory Yokohama. Because no one else will.

[ Gundam Factory Yokohama ] via [ Impress ] and [ RobotStart ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here’s what we have so far (send us your events!):

Robotic Arena – January 25, 2020 – Wrocław, Poland DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, Wash., USA HRI 2020 – March 23-26, 2020 – Cambridge, U.K. ICARSC 2020 – April 15-17, 2020 – Ponta Delgada, Azores ICRA 2020 – May 31-4, 2020 – Paris, France

Let us know if you have suggestions for next week, and enjoy today’s videos.

I’ve got to hand it to Boston Dynamics—letting Adam Savage borrow a Spot for a year is a pretty savvy marketing move.

[ Tested ]

The Indian Space Research Organization (ISRO) plans to send a humanoid robot into space later this year. According to a Times of India story, the humanoid is called Vyommitra and will help ISRO prepare for its Gaganyaan manned space flight mission, expected for 2022. Before sending human astronauts, ISRO will send Vyommitra, which can speak but doesn’t move much (it currently has no legs). According to the Times of India, ISRO chief Kailasavadivoo Sivan said the “Gaganyaan mission is not just about sending a human to space, this mission provides us an opportunities to build a framework for long term national and international collaborations and cooperation. We all know that scientific discoveries, economic development, education, tech development and inspiring youth are coming goals for all nations. Human space flight provides perfect platform to meet all these objectives.”

[ Times of India ]

Soft robots have applications in safe human-robot interactions, manipulation of fragile objects, and locomotion in challenging and unstructured environments. In this paper, we present a computational method for augmenting soft robots with proprioceptive sensing capabilities. Our method automatically computes a minimal stretch-receptive sensor network to user-provided soft robotic designs, which is optimized to perform well under a set of user-specified deformation-force pairs. The sensorized robots are able to reconstruct their full deformation state, under interaction forces. We cast our sensor design as a sub-selection problem, selecting a minimal set of sensors from a large set of fabricable ones which minimizes the error when sensing specified deformation-force pairs. Unique to our approach is the use of an analytical gradient of our reconstruction performance measure with respect to selection variables. We demonstrate our technique on a bending bar and gripper example, illustrating more complex designs with a simulated tentacle.

Disney Research ]

Dragonfly is a rotorcraft lander that will explore Saturn’s large moon Titan. The sampling system called DrACO (Drill for Acquisition of Complex Organics) will extract material from Titan’s surface and deliver it to DraMS (Dragonfly Mass Spectrometer, provided by NASA Goddard Space Flight Center). Honeybee Robotics will build the end-to-end DrACO system (including hardware, avionics, and flight software) and will command its operation once Dragonfly lands on Titan in 2034.

Honeybee Robotics ]

DARPA’s Gremlins program has completed the first flight test of its X-61A vehicle. The test in late November at the U.S. Army’s Dugway Proving Ground in Utah included one captive-carry mission aboard a C-130A and an airborne launch and free flight lasting just over an hour-and-a-half.

The goal for this third phase of the Gremlins program is completion of a full-scale technology demonstration series featuring the air recovery of multiple, low-cost, reusable unmanned aerial systems (UASs), or “Gremlins.” Safety, reliability, and affordability are the key objectives for the system, which would launch groups of UASs from multiple types of military aircraft while out of range from adversary defenses. Once Gremlins complete their mission, the transport aircraft would retrieve them in the air and carry them home, where ground crews would prepare them for their next use within 24 hours.

[ DARPA ]

Thi is only sort of a robot, more of an automated system, but I like the idea: dog training!

[ CompanionPro ]

Free-falling paper shapes exhibit rich, complex and varied behaviours that are extremely challenging to model analytically. Physical experimentation aids in system understanding, but is time-consuming, sensitive to initial conditions and reliant on subjective visual behavioural classification. In this study, robotics, computer vision and machine learning are used to autonomously fabricate, drop, analyse and classify the behaviours of hundreds of shapes.

[ Nature ]

This paper introduces LiftTiles, modular inflatable actuators for prototyping room-scale shape-changing interfaces. Each inflatable actuator has a large footprint (e.g., 30 cm x 30 cm) and enables large-scale shape transformation. The ac- tuator is fabricated from a flexible plastic tube and constant force springs. It extends when inflated and retracts by the force of its spring when deflated. By controlling the internal air volume, the actuator can change its height from 15 cm to 150 cm.

We designed each module as low cost (e.g., 8 USD), lightweight (e.g., 1.8kg), and robust (e.g., with- stand more than 10 kg weight), so that it is suitable for rapid prototyping of room-sized interfaces. Our design utilizes constant force springs to provide greater scalability, simplified fabrication, and stronger retraction force, all essential for large-scale shape-change.

[ LiftTiles ]

Aibo may not be the most fearsome security pupper, but it does have what other dogs don’t: Wireless connectivity, remote control, and a camera.

[ Aibo ]

I missed this Toyota HSR demo at CES, which is really too bad because I really could have used a snack.

[ NEU ]

The HKUST Aerial Robotics Group has some impressive real-time drone planning that’ll be presented at ICRA 2020:

[ Paper ]

Gripping something tricky? When in doubt, just add more fingers.

[ Soft Robotics ]

Demo of the project of Nino Di Pasquale, Matthieu Le Cauchois, Alejandra Plaice and Joël Zbinden. The goal was to program in Python and combine in a project, elements of global path planning, local path planning, baysian filtering for pose estimation and computer vision. The video presents the visualisation interface in real time, assoicated with the real video of the setting with the Thymio robot controlled by wireless connection by the computer running the program.

[ EPFL ]

From public funding opportunities to the latest technologies in software and system integration, the combination of robotics and IT to hardware and application highlights plus updates on new platforms and open-source communities: ROS-Industrial Conference 2019 offered on 3 days in December a varied and top-class programme to more than 150 attendees.

[ ROS-I Consortium ]

Aaron Johnson and his students have been exploring whether hoof-inspired feet can help robots adapt to rough terrain without needing to exhaustively plan out every step.

There’s no paper or anything yet, but Aaron did give a talk at Dynamic Walking 2018.

[ Robomechanics Lab ]

YouTube has put some money into an original eight-episode series on robots and AI, featuring some well-known roboticists. Here are a couple of the more robot-y episodes:

You can watch the whole series at the link below.

[ Age of AI ]

On the AI Podcast, Lex Fridman speaks with Ayanna Howard from Georgia Tech.

[ Lex Fridman ]

NASA has decided that humans are going back to the Moon. That’s great! Before that actually happens, a whole bunch of other things have to happen, and excitingly, many of those things involve robots. As a sort of first-ish step, NASA is developing a new lunar rover called VIPER (Volatiles Investigating Polar Exploration Rover). VIPER’s job is to noodle around the permanently shaded craters at the Moon’s south pole looking for water ice, which can (eventually) be harvested and turned into breathable air and rocket fuel.

An engineering model of the Volatiles Investigating Polar Exploration Rover, or VIPER, is tested in the Simulated Lunar Operations Laboratory at NASA’s Glenn Research Center in Cleveland, Ohio. About the size of a golf cart, VIPER is a mobile robot that will roam around the Moon’s South Pole looking for water ice in the region and for the first time ever, actually sample the water ice at the same pole where the first woman and next man will land in 2024 under the Artemis program.

In the video, the VIPER engineering model is enjoying playtime in simulated lunar regolith (not stuff that you want to be breathing, hence the fancy hats) to help characterize the traction of the wheels on different slopes, and to help figure out how much power will be necessary. The final rover might look a bit more like this:

VIPER is more than somewhat similar to an earlier rover that NASA was working on called Resource Prospector, which was cancelled back in 2018. Resource Prospector was also scheduled to go to the Moon’s south pole to look for water ice, and VIPER will be carrying some of the instruments originally developed for Resource Prospector. If it seems a little weird that NASA cancelled Resource Prospector only to almost immediately start work on VIPER, well, yeah—the primary difference between the two rovers seems to be that VIPER is intended to spend several months operating, while Resource Prospector’s lifespan was only a couple of weeks.

The other big difference between VIPER and Resource Prospector is that NASA has been gradually shifting away from developing all of its own hardware in-house, and VIPER is no exception. One of the primary science instruments, a drilling system called TRIDENT (The Regolith and Ice Drill for Exploring New Terrain, obviously), comes from Honeybee Robotics, which has contributed a variety of tools that have been used to poke and prod at the surface of Mars on the Mars Exploration rovers, Phoenix, and Curiosity. There’s nothing wrong with this, except that for VIPER, it looks like NASA wants a commercial delivery system as well.

Finding water ice on the Moon is the first step towards the in-situ resource utilization (ISRU) robots necessary to practically sustain a long-term lunar mission

Last week, Space News reported that NASA is postponing procurement of a commercially-developed lander that would deliver VIPER to the lunar surface, meaning that not only does VIPER not have a ride to the Moon right now, but that it’s not very clear when it’ll actually happen—as recently as last November, the plan was to have a lander selected by early February, for a landing in late 2022. From the sound of things, the problem is that VIPER is a relatively chunky payload (weighing about 350 kg), meaning that only a few companies have the kind of hardware that would be required to get it safely to the lunar surface, and NASA has a limited budget that also has to cover a bunch of other stuff at the same time.

This delay is unfortunate, because VIPER plays an important part in NASA’s overall lunar strategy. Finding water ice on the Moon is the first step towards the in-situ resource utilization (ISRU) robots necessary to practically sustain a long-term lunar mission, and after that, it’ll take a bunch more work to actually deploy a system to harvest ice and turn it into usable hydrogen and oxygen with enough reliability and volume to make a difference. We have the technology—we’ve just got to get it there, and get it working. 

[ VIPER ]

Suction is a useful tool in many robotic applications, as long as those applications are grasping objects that are suction-friendly—that is, objects that are impermeable and generally smooth-ish and flat-ish. If you can’t form a seal on a surface, your suction gripper is going to have a bad time, which is why you don’t often see suction systems working outside of an environment that’s at least semi-constrained. Warehouses? Yes. Kitchens? Maybe. The outdoors? Almost certainly not.

In general, getting robotic grippers (and robots themselves) to adhere to smooth surfaces and rough surfaces requires completely different technology. But researchers from Zhejiang University in China have come up with a new kind of suction gripper that can very efficiently handle surfaces like widely-spaced tile and even rough concrete, by augmenting the sealing system with a spinning vortex of water.

Image: Zhejiang University To climb, the robot uses a micro-vacuum pump coupled to a rapidly rotating fan and a water source. Centripetal force causes the spinning water to form a ring around the outside of the vacuum chamber. Because water can get into all those surface irregularities that doom traditional vacuum grippers, the seal is much stronger.

The paper is a little bit dense, but from what I can make out, what’s going on is that you’ve got a traditional suction gripper with a vacuum pump, modified with a water injection system and a fan. The fan has nothing to do with creating or maintaining a vacuum—its job is to get the water spinning at up to 90 rotations per second. Centripetal force causes the spinning water to form a ring around the outside of the vacuum chamber, which keeps the water from being sucked out through the vacuum pump while also maintaining a liquid seal between the vacuum chamber and the surface. Because water can get into all of those annoying little nooks and crannies that can mean doom for traditional vacuum grippers, the seal is much better, resulting in far higher performance, especially on surfaces with high roughness.

Photo: Zhejiang University One of the potential applications for the water-vortex suction robot is as a “Spider-Man” wall-climbing device.

For example, a single suction unit weighing 0.8 kg was able to generate a suction force of over 245 N on a rough surface using less than 400 W, while a traditional suction unit of the same footprint would need several thousand watts (and weigh dozens of kilograms) to generate a comparable amount of suction, since the rough surface would cause a significant amount of leakage (although not a loss of suction). At very high power, the efficiency does decrease a bit— the “Spider-Man” system weighs 3 kg per unit, with a suction force of 2000 N using 650 W.

And as for the downsides? Er, well, it does kind of leak all over the place, especially when disengaging. The “Spider-Man” version leaks over 2 liters per minute. It’s only water, but still. And since it leaks, it needs to be provided with a constant water supply, which limits its versatility. The researchers are working on ways of significantly reducing water consumption to make the system more independent, but personally, I feel like the splooshyness is part of the appeal.

Vacuum suction unit based on the zero pressure difference method,” by Kaige Shi and Xin Li from Zhejiang University in China, is published in Physics of Fluids.

Pages