Feed aggregator

Food and eating are inherently social activities taking place, for example, around the dining table at home, in restaurants, or in public spaces. Enjoying eating with others, often referred to as “commensality,” positively affects mealtime in terms of, among other factors, food intake, food choice, and food satisfaction. In this paper we discuss the concept of “Computational Commensality,” that is, technology which computationally addresses various social aspects of food and eating. In the past few years, Human-Computer Interaction started to address how interactive technologies can improve mealtimes. However, the main focus has been made so far on improving the individual's experience, rather than considering the inherently social nature of food consumption. In this survey, we first present research from the field of social psychology on the social relevance of Food- and Eating-related Activities (F&EA). Then, we review existing computational models and technologies that can contribute, in the near future, to achieving Computational Commensality. We also discuss the related research challenges and indicate future applications of such new technology that can potentially improve F&EA from the commensality perspective.

Recently, attempts have been made to develop ionic polymer–metal composite (IPMC), which is garnering growing interest for ionic artificial muscle, as a soft actuator and sensor due to its inherent properties of low weight, flexibility, softness, and particularly, its efficient transformation of electrical energy into mechanical energy, with large bending strain response under a low activation voltage. In this paper, we focused on several current deficiencies of IPMC that restrict its application, such as non-standardized preparation steps, relaxation under DC voltage, solvent evaporation, and poor output force. Corresponding solutions to overcome the abovementioned problems have recently been proposed from our point of view and developed through our research. After a brief introduction to the working mechanism of IPMC, we here investigate the key factors that influence the actuating performance of IPMC. We also review the optimization strategies in IPMC actuation, including those for preparation steps, additive selection for a thick casting membrane, solvent substitutes, water content, encapsulation, etc. With consideration of the role of the interface electrode, its effects on the performance of IPMC are revealed based on our previous work. Finally, we also discuss IPMCs as potential sensors theoretically and experimentally. The elimination of the deficiencies of IPMC will promote its applications in soft robotics.

AI experts gathered at MIT last week, with the aim of predicting the role artificial intelligence will play in the future of work. Will it be the enemy of the human worker? Will it prove to be a savior? Or will it be just another innovation—like electricity or the internet?

As IEEE Spectrum previously reported, this conference (“AI and the Future of Work Congress”), held at MIT’s Kresge Auditorium, offered sometimes pessimistic outlooks on the job- and industry-destroying path that AI and automation seems to be taking: Self-driving technology will put truck drivers out of work; smart law clerk algorithms will put paralegals out of work; robots will (continue to) put factory and warehouse workers out of work.

Andrew McAfee, co-director of MIT’s Initiative on the Digital Economy, said even just in the past couple years, he’s noticed a shift in the public’s perception of AI. “I remember from previous versions of this conference, it felt like we had to make the case that we’re living in a period of accelerating change and that AI’s going to have a big impact,” he said. “Nobody had to make that case today.”

Elisabeth Reynolds, executive director of MIT’s Task Force on the Work of the Future, noted that following the path of least resistance is not a viable way forward. “If we do nothing, we’re in trouble,” she said. “The future will not take care of itself. We have to do something about it.”

Panelists and speakers spoke about championing productive uses of AI in the workplace, which ultimately benefit both employees and customers.

As one example, Zeynep Ton, professor at MIT Sloan School of Management, highlighted retailer Sam’s Club’s recent rollout of a program called Sam’s Garage. Previously customers shopping for tires for their car spent somewhere between 30 and 45 minutes with a Sam’s Club associate paging through manuals and looking up specs on websites.

But with an AI algorithm, they were able to cut that spec hunting time down to 2.2 minutes. “Now instead of wasting their time trying to figure out the different tires, they can field the different options and talk about which one would work best [for the customer],” she said. “This is a great example of solving a real problem, including [enhancing] the experience of the associate as well as the customer.”

“We think of it as an AI-first world that’s coming,” said Scott Prevost, VP of engineering at Adobe. Prevost said AI agents in Adobe’s software will behave something like a creative assistant or intern who will take care of more mundane tasks for you.

“We need a mindset change. That it is not just about minimizing costs or maximizing tax benefits, but really worrying about what kind of society we’re creating and what kind of environment we’re creating if we keep on just automating and [eliminating] good jobs.” —Daron Acemoglu, MIT Institute Professor of Economics

Prevost cited an internal survey of Adobe customers that found 74 percent of respondents’ time was spent doing repetitive work—the kind that might be automated by an AI script or smart agent.

“It used to be you’d have the resources to work on three ideas [for a creative pitch or presentation],” Prevost said. “But if the AI can do a lot of the production work, then you can have 10 or 100. Which means you can actually explore some of the further out ideas. It’s also lowering the bar for everyday people to create really compelling output.”

In addition to changing the nature of work, noted a number of speakers at the event, AI is also directly transforming the workforce.

Jacob Hsu, CEO of the recruitment company Catalyte spoke about using AI as a job placement tool. The company seeks to fill myriad positions including auto mechanics, baristas, and office workers—with its sights on candidates including young people and mid-career job changers. To find them, it advertises on Craigslist, social media, and traditional media.

The prospects who sign up with Catalyte take a battery of tests. The company’s AI algorithms then match each prospect’s skills with the field best suited for their talents.

“We want to be like the Harry Potter Sorting Hat,” Hsu said.

Guillermo Miranda, IBM’s global head of corporate social responsibility, said IBM has increasingly been hiring based not on credentials but on skills. For instance, he said, as much as 50 per cent of the company’s new hires in some divisions do not have a traditional four-year college degree. “As a company, we need to be much more clear about hiring by skills,” he said. “It takes discipline. It takes conviction. It takes a little bit of enforcing with H.R. by the business leaders. But if you hire by skills, it works.”

Ardine Williams, Amazon’s VP of workforce development, said the e-commerce giant has been experimenting with developing skills of the employees at its warehouses (a.k.a. fulfillment centers) with an eye toward putting them in a position to get higher-paying work with other companies.

She described an agreement Amazon had made in its Dallas fulfillment center with aircraft maker Sikorsky, which had been experiencing a shortage of skilled workers for its nearby factory. So Amazon offered to its employees a free certification training to seek higher-paying work at Sikorsky.

“I do that because now I have an attraction mechanism—like a G.I. Bill,” Williams said. The program is also only available for employees who have worked at least a year with Amazon. So their program offers medium-term job retention, while ultimately moving workers up the wage ladder.

Radha Basu, CEO of AI data company iMerit, said her firm aggressively hires from the pool of women and under-resourced minority communities in the U.S. and India. The company specializes in turning unstructured data (e.g. video or audio feeds) into tagged and annotated data for machine learning, natural language processing, or computer vision applications.

“There is a motivation with these young people to learn these things,” she said. “It comes with no baggage.”

Alastair Fitzpayne, executive director of The Aspen Institute’s Future of Work Initiative, said the future of work ultimately means, in bottom-line terms, the future of human capital. “We have an R&D tax credit,” he said. “We’ve had it for decades. It provides credit for companies that make new investment in research and development. But we have nothing on the human capital side that’s analogous.”

So a company that’s making a big investment in worker training does it on their own dime, without any of the tax benefits that they might accrue if they, say, spent it on new equipment or new technology. Fitzpayne said a simple tweak to the R&D tax credit could make a big difference by incentivizing new investment programs in worker training. Which still means Amazon’s pre-existing worker training programs—for a company that already famously pays no taxes—would not count.

“We need a different way of developing new technologies,” said Daron Acemoglu, MIT Institute Professor of Economics. He pointed to the clean energy sector as an example. First a consensus around the problem needs to emerge. Then a broadly agreed-upon set of goals and measurements needs to be developed (e.g., that AI and automation would, for instance, create at least X new jobs for every Y jobs that it eliminates).

Then it just needs to be implemented.

“We need to build a consensus that, along the path we’re following at the moment, there are going to be increasing problems for labor,” Acemoglu said. “We need a mindset change. That it is not just about minimizing costs or maximizing tax benefits, but really worrying about what kind of society we’re creating and what kind of environment we’re creating if we keep on just automating and [eliminating] good jobs.”

Let me begin this review by saying that the Skydio 2 is one of the most impressive robots that I have ever seen. Over the last decade, I’ve spent enough time around robots to have a very good sense of what kinds of things are particularly challenging for them, and to set my expectations accordingly. Those expectations include things like “unstructured environments are basically impossible” and “full autonomy is impractically expensive” and “robot videos rarely reflect reality.”

Skydio’s newest drone is an exception to all of this. It’s able to fly autonomously at speed through complex environments in challenging real-world conditions in a way that’s completely effortless and stress-free for the end user, allowing you to capture the kind of video that would be otherwise impossible, even (I’m guessing) for professional drone pilots. When you see this technology in action, it’s (almost) indistinguishable from magic. 

Skydio 2 Price

To be clear, the Skydio 2 is not without compromises, and the price of $999 (on pre-order with delivery of the next batch expected in spring of 2020) requires some justification. But the week I’ve had with this drone has left me feeling like its fundamental autonomous capability is so far beyond just about anything that I’ve ever experienced that I’m questioning why I would every fly anything else ever again.

We’ve written extensively about Skydio, beginning in early 2016 when the company posted a video of a prototype drone dodging trees while following a dude on a bike. Even three years ago, Skydio’s tech was way better than anything we’d seen outside of a research lab, and in early 2018, they introduced their first consumer product, the Skydio R1. A little over a year later, Skydio has introduced the Skydio 2, which is smaller, smarter, and much more affordable. Here’s an overview video just to get you caught up:

Skydio sent me a Skydio 2 review unit last week, and while I’m reasonably experienced with drones in general, this is the first time I’ve tried a Skydio drone in person. I had a pretty good idea what to expect, and I was absolutely blown away. Like, I was giggling to myself while running through the woods as the drone zoomed around, deftly avoiding trees and keeping me in sight. Robots aren’t supposed to be this good.

A week is really not enough time to explore everything that the Skydio can do, especially Thanksgiving week in Washington, D.C. (a no-fly zone) in early winter. But I found a nearby state park in which I could legally and safely fly the drone, and I did my best to put the Skydio 2 through its paces.

Note: Throughout this review, we’ve got a bunch of GIFs to help illustrate different features of the drone. To fit them all in, these GIFs had to be heavily compressed. Underneath each GIF is a timestamped link to this YouTube video (also available at the bottom of the post), which you can click on to see the an extended cut of the original 4K 30 fps footage. And there’s a bunch of interesting extra video in there as well.

Skydio 2 Specs Photo: Evan Ackerman/IEEE Spectrum The Skydio 2 is primarily made out of magnesium, which (while light) is both heavier and more rigid and durable than plastic. The offset props (the back pair are above the body, and the front pair are below) are necessary to maintain the field of view of the navigation cameras.

The Skydio 2 both looks and feels like a well-designed and carefully thought-out drone. It’s solid, and a little on the heavy side as far as drones go—it’s primarily made out of magnesium, which (while light) is both heavier and more rigid and durable than plastic. The blue and black color scheme is far more attractive than you typically see with drones. 

Photo: Evan Ackerman/IEEE Spectrum To detect and avoid obstacles, the Skydio 2 uses an array of six 4K hemispherical cameras that feed data into an NVIDIA Jetson TX2 at 30 fps, with the drone processing a million points in 3D space per second to plan the safest path.

The Skydio 2 is built around an array of six hemispherical obstacle-avoidance cameras and the NVIDIA Jetson TX2 computing module that they’re connected to. This defines the placement of the gimbal, the motors and props, and the battery, since all of this stuff has to be as much as possible out of the view of the cameras in order for the drone to effectively avoid obstacles in any direction. 

Without the bottom-mounted battery attached, the drone is quite flat. The offset props (the back pair are above the body, and the front pair are below) are necessary to maintain the field of view of the obstacle-avoidance cameras. These hemispherical cameras are on the end of each of the prop arms as well as above and below the body of the drone. They look awfully exposed, even though each is protected from ground contact by a little fin. You need to make sure these cameras are clean and smudge-free, and Skydio includes a cleaning cloth for this purpose. Underneath the drone there are slots for microSD cards, one for recording from the camera and a second one that the drone uses to store data. The attention to detail extends to the SD card insertion, which has a sloped channel that guides the card securely into its slot. 

Once you snap the battery in, the drone goes from looking streamlined to looking a little chubby. Relative to other drones, the battery almost seems like an afterthought, like Skydio designed the drone and then remembered, “oops we have to add a battery somewhere, let’s just kludge it onto the bottom.” But again, the reason for this is to leave room inside the body for the NVIDIA TX2, while making sure that the battery stays out of view of the obstacle avoidance cameras.

The magnetic latching system for the battery is both solid and satisfying. I’m not sure why it’s necessary, strictly speaking, but I do like it, and it doesn’t seem like the battery will fly off even during the most aggressive maneuvers. Each battery includes an LED array that will display its charge level in 25 percent increments, as well as a button that you push to turn the drone on and off. Charging takes place via a USB-C port in the top of the drone, which I don’t like, because it means that the batteries can’t be charged on their own (like the Parrot Anafi’s battery), and that you can’t charge one battery while flying with another, like basically every other drone ever. A separate battery charger that will charge two at once is available from Skydio for an eyebrow-raising $129.

I appreciate that all of Skydio’s stuff (batteries, controller, and beacon) charges via USB-C, though. The included USB-C adapter with its beefy cable will output at up to 65 watts, which’ll charge a mostly depleted battery in under an hour. The drone turns itself on while charging, which seems unnecessary.

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2 is not foldable, making it not nearly as easy to transport as some other drones. But it does come with a nice case that mitigates this issue somewhat, and the drone plus two batteries end up as a passably flat package about the size of a laptop case.

The most obvious compromise that Skydio made with the Skydio 2 is that the drone is not foldable. Skydio CEO Adam Bry told us that adding folding joints to the arms of the Skydio 2 would have made calibrating all six cameras a nightmare and significantly impacted performance. This makes complete sense, of course, but it does mean that the Skydio 2 is not nearly as easy to transport as some other drones. 

Photo: Evan Ackerman/IEEE Spectrum Folded and unfolded: The Skydio 2 compared to the Parrot Anafi (upper left) and the DJI Mavic Pro (upper right).

The Skydio 2 does come with a very nice case that mitigates this issue somewhat, and the drone plus two batteries end up as a passably flat package about the size of a laptop case. Still, it’s just not as convenient to toss into a backpack as my Anafi, although the Mavic Mini might be even more portable.

Photo: Evan Ackerman/IEEE Spectrum While the Skydio 2’s case is relatively compact, the non-foldable drone is overall a significantly larger package than the Parrot Anafi.

The design of the drone leads to some other compromises as well. Since landing gear would, I assume, occlude the camera system, the drone lands directly on the bottom of its battery pack, which has a slightly rubberized pad about the size of a playing card. This does’t feel particularly stable unless you end up on a very flat surface, and made me concerned for the exposed cameras underneath the drone as well as the lower set of props. I’d recommend hand takeoffs and landings—more on those later.

Skydio 2 Camera System

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2’s primary camera is a Sony IMX577 1/2.3" 12.3-megapixel CMOS sensor. It’s mounted to a three-axis gimbal and records 4K video at 60 fps, or 1080p video at 120 fps.

The Skydio 2 comes with a three-axis gimbal supporting a 12-megapixel camera, just enough to record 4K video at 60 fps, or 1080p video at 120 fps. Skydio has provided plenty of evidence that its imaging system is at least as good if not better than other drone cameras. Tested against my Mavic Pro and Parrot Anafi, I found no reason to doubt that. To be clear, I didn’t do exhaustive pixel-peeping comparisons between them, you’re just getting my subjective opinion that the Skydio 2 has a totally decent camera that you won’t be disappointed with. I will say that I found the HDR photo function to be not all that great under the few situations in which I tested it—after looking at a few muddy sunset shots, I turned it off and was much happier.

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2’s 12-megapixel camera is solid, although we weren’t impressed with the HDR option.

The video stabilization is fantastic, to the point where watching the video footage can be underwhelming because it doesn’t reflect the motion of the drone. I almost wish there was a way to change to unstabilized (or less-stabilized) video so that the viewer could get a little more of a wild ride. Or, ideally, there’d be a way for the drone to provide you with a visualization of what it was doing using the data collected by its cameras. That’s probably wishful thinking, though. The drone itself doesn’t record audio because all you’d get would be an annoying buzz, but the app does record audio, so the audio from your phone gets combined with the drone video. Don’t expect great quality, but it’s better than nothing.

Skydio 2 App

The app is very simple compared to every other drone app I’ve tried, and that’s a good thing. Here’s what it looks like:

Image: Skydio Trackable subjects get a blue “+” sign over them, and if you tap them, the “+” turns into a spinny blue circle. Once you’ve got a subject selected, you can choose from a variety of cinematic skills that the drone will execute while following you.

You get the controls that you need and the information that you need, and nothing else. Manual flight with the on-screen buttons works adequately, and the double-tap to fly function on the phone works surprisingly well, making it easy to direct the drone to a particular spot above the ground.

The settings menus are limited but functional, allowing you to change settings for the camera and a few basic tweaks for controlling the drone. One unique setting to the Skydio 2 is the height floor—since the drone only avoids static obstacles, you can set it to maintain a height of at least 8 feet above the ground while flying autonomously to make sure that if you’re flying around other people, it won’t run into anyone who isn’t absurdly tall and therefore asking for it.

Trackable subjects get a blue “+” sign over them in the app, and if you tap them, the “+” turns into a spinny blue circle. Once you’ve got a subject selected, you can choose from a variety of cinematic skills that the drone will execute while following you, and in addition, you can select “one-shot” skills that involve the drone performing a specific maneuver before returning to the previously selected cinematic skill. For example, you can tell the drone to orbit around you, and then do a “rocket” one-shot where it’ll fly straight up above you (recording the whole time, of course), before returning to its orbiting. 

After you’re done flying, you can scroll through your videos and easily clip out excerpts from them and save them to your phone for sharing. Again, it’s a fairly simple interface without a lot of options. You could call it limited, I guess, but I appreciate that it just does a few things that you care about and otherwise doesn’t clutter itself up.

The real limitation of the app is that it uses Wi-Fi to connect to the Skydio 2, which restricts the range. To fly much beyond a hundred meters or so, you’ll need to use the controller or beacon instead.

Skydio 2 Controller and Beacon Photo: Evan Ackerman/IEEE Spectrum While the Skydio 2 controller provides a better hands-on flight experience than with the phone, plus an extended range of up to 3.5 km, more experienced pilots may find manual control a bit frustrating, because the underlying autonomy will supersede your maneuvers when you start getting close to objects.

I was looking forward to using the controller, because with every other drone I’ve had, the precision that a physically controller provides is, I find, mandatory for a good flying experience and to get the photos and videos that you want. With Skydio 2, that’s all out the window. It’s not that the controller is useless or anything, it’s just that because the drone tracks you and avoids obstacles on its own, that level of control precision becomes largely unnecessary. 

The controller itself is perfectly fine. It’s a rebranded Parrot Skycontroller3, which is the same as the one that you get with a Parrot Anafi. It’s too bad that the sticks don’t unscrew to make it a little more portable, and overall it’s functional rather than fancy, but it feels good to use and includes a sizeable antenna that makes a significant difference to the range that you get (up to 3.5 kilometers).

You definitely get a better hands-on flight experience with the controller than with the phone, so if you want to (say) zip the drone around some big open space for fun, it’s good for that. And it’s nice to be able to hand the controller to someone who’s never flown a drone before and let them take it for a spin without freaking out about them crashing it the whole time. For more experienced pilots, though, the controller is ultimately just a bit frustrating, because the underlying autonomy will supersede your control when you start getting close to objects, which (again) limits how useful the controller is relative to your phone.

I do still prefer the controller over the phone, but I’m not sure that it’s worth the extra $150, unless you plan to fly the Skydio 2 at very long distances or primarily in manual mode. And honestly, if either of those two things are your top priority, the Skydio 2 is probably not the drone for you.

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2 beacon uses GPS tracking to help the drone follow you, extending range up to 1.5 km. You can also fly the with the beacon alone, no phone necessary.

The purpose of the beacon, according to Skydio, is to give the drone a way of tracking you if it can’t see you, which can happen, albeit infrequently. My initial impression of the beacon was that it was primarily useful as a range-extending bridge between my phone and the drone. But I accidentally left my phone at home one day (oops) and had to fly the drone with only the beacon, and it was a surprisingly decent experience. The beacon allows for full manual control of a sort—you can tap different buttons to rotate, fly forward, and ascend or descend. This is sufficient for takeoff, landing, to make sure that the drone is looking at you when you engage visual tracking, and to rescue it if it gets trapped somewhere.

The rest of the beacon’s control functions are centered around a few different tracking modes, and with these, it works just about as well as your phone. You have fewer options overall, but all the basic stuff is there with just a few intuitive button clicks, including tracking range and angle. If you’re willing to deal with this relatively minor compromise, it’s nice to not have your phone available for other things rather than being monopolized by the drone.

Skydio 2 In Flight GIF: Evan Ackerman/IEEE Spectrum Hand takeoffs are simple and reliable.  Click here for a full resolution clip.

Starting up the Skydio 2 doesn’t require any kind of unusual calibration steps or anything like that. It prefers to be kept still, but you can start it up while holding it, it’ll just take a few seconds longer to tell you that it’s ready to go. While the drone will launch from any flat surface with significant clearance around it (it’ll tell you if it needs more room), the small footprint of the battery means that I was more comfortable hand launching it. This is not a “throw” launch; you just let the drone rest on your palm, tell it to take off, and then stay still while it gets its motors going and then gently lifts off. The lift off is so gentle that you have to be careful not to pull your hand away too soon—I did that once and the drone, being not quite ready, dropped towards the ground, but managed to recover without much drama.

GIF: Evan Ackerman/IEEE Spectrum Hand landings always look scary, but the Skydio 2 is incredibly gentle. After trying this once, it became the only way I ever landed the drone.  Click here for a full resolution clip.

Catching the drone for landing is perhaps very slightly more dangerous, but not any more difficult. You put the drone above and in front of you facing away, tell it to land in the app or with the beacon, and then put your hand underneath it to grasp it as it slowly descends. It settles delicately and promptly turns itself off. Every drone should land this way. The battery pack provides a good place to grip, although you do have to be mindful of the forward set of props, which (since they’re the pair that are beneath the body of drone) are quite close to your fingers. You’ll certainly be mindful after you catch a blade with your fingers once. Which I did. For the purposes of this review and totally not by accident. No damage, for the record.

Photo: Evan Ackerman/IEEE Spectrum You won’t be disappointed with the Skydio 2’s in-flight performance, unless you’re looking for a dedicated racing drone.

In normal flight, the Skydio 2 performs as well as you’d expect. It’s stable and manages light to moderate wind without any problems, although I did notice some occasional lateral drifting when the drone should have been in a stationary hover. While the controller gains are adjustable, the Skydio 2 isn’t quite as aggressive in flight as my Mavic Pro on Sport Mode, but again, if you’re looking for a high-speed drone, that’s really not what the Skydio is all about.

The Skydio 2 is substantially louder than my Anafi, although the Anafi is notably quiet for a drone. It’s not annoying to hear (not a high-pitched whine), but you can hear it from a ways away, and farther away than my Mavic Pro. I’m not sure whether that’s because of the absolute volume or the volume plus the pitch. In some ways, this is a feature, since you can hear the drone following you even if you’re not looking at it, you just need to be aware of the noise it makes when you’re flying it around people.

Obstacle Avoidance

The primary reason Skydio 2 is the drone that you want to fly is because of its autonomous subject tracking and obstacle avoidance. Skydio’s PR videos make this capability look almost too good, and since I hadn’t tried out one of their drones before, the first thing I did with it was exactly what you’d expect: attempt to fly it directly into the nearest tree.

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 deftly slides around trees and branches. The control inputs here were simple “forward” or “turn,” all obstacle avoidance is autonomous.  Click here for a full resolution clip.

And it just won’t do it. It slows down a bit, and then slides right around one tree after another, going over and under and around branches. I pointed the drone into a forest and just held down “forward” and away it went, without any fuss, effortlessly ducking and weaving its way around. Of course, it wasn’t effortless at all—six 4K cameras were feeding data into the NVIDIA TX2 at 30 fps, and the drone was processing a million points in 3D space per second to plan the safest path while simultaneously taking into account where I wanted it to go. I spent about 10 more minutes doing my level best to crash the drone into anything at all using a flying technique probably best described as “reckless,” but the drone was utterly unfazed. It’s incredible.

What knocked my socks off was telling the drone to pass through treetops—in the clip below, I’m just telling the drone to fly straight down. Watch as it weaves its way through gaps between the branches:

GIF: Evan Ackerman/IEEE Spectrum The result of parking the Skydio 2 above some trees and holding “down” on the controller is this impressive fully autonomous descent through the branches.  Click here for a full resolution clip.

Here’s one more example, where I sent the drone across a lake and started poking around in a tree. Sometimes the Skydio 2 isn’t sure where you want it to go, and you have to give it a little bit of a nudge in a clear direction, but that’s it.

GIF: Evan Ackerman/IEEE Spectrum In obstacle-heavy environments, the Skydio 2 prudently slows down, but it can pick its way through almost anything that it can see.  Click here for a full resolution clip.

It’s important to keep in mind that all of the Skydio 2’s intelligence is based on vision. It uses cameras to see the world, which means that it has similar challenges as your eyes do. Specifically, Skydio warns against flying in the following conditions:

  • Skydio 2 can’t see certain visually challenging obstacles. Do not fly around thin branches, telephone or power lines, ropes, netting, wires, chain link fencing or other objects less than ½ inch in diameter.
  • Do not fly around transparent surfaces like windows or reflective surfaces like mirrors greater than 60 cm wide.
  • When the sun is low on the horizon, it can temporarily blind Skydio 2’s cameras depending on the angle of flight. Your drone may be cautious or jerky when flying directly toward the sun.

Basically, if you’d have trouble seeing a thing, or seeing under some specific flight conditions, then the Skydio 2 almost certainly will also. It gets even more problematic when challenging obstacles are combined with challenging flight conditions, which is what I’m pretty sure led to the only near-crash I had with the drone. Here’s a video:

GIF: Evan Ackerman/IEEE Spectrum Flying around very thin branches and into the sun can cause problems for the Skydio 2’s obstacle avoidance.  Click here for a full resolution clip.

I had the Skydio 2 set to follow me on my bike (more about following and tracking in a bit). It was mid afternoon, but since it’s late fall here in Washington, D.C., the sun doesn’t get much higher than 30 degrees above the horizon. Late fall also means that most of the deciduous trees have lost their leaves, and so there are a bunch of skinny branches all over the place. The drone was doing a pretty good job of following me along the road at a relatively slow speed, and then it clipped the branch that you can just barely see in the video above. It recovered in an acrobatic maneuver that has been mostly video-stabilized out, and resumed tracking me before I freaked and told it to land. You can see another example here, where the drone (again) clips a branch that has the sun behind it, and this clip shows me stopping my bike before the drone runs into another branch in a similar orientation. As the video shows, it’s very hard to see the branches until it’s too late.

As far as I can tell, the drone is no worse for wear from any of this, apart from a small nick in one of the props. But, this is a good illustration of a problematic situation for the Skydio 2: flying into a low sun angle around small bare branches. Should I not have been flying the drone in this situation? It’s hard to say. These probably qualify as “thin branches,” although there was plenty of room along with middle of the road. There is an open question with the Skydio 2 as to exactly how much responsibility the user should have about when and where it’s safe to fly—for branches, how thin is too thin? How low can the sun be? What if the branches are only kinda thin and the sun is only kinda low, but it’s also a little windy? Better to be safe than sorry, of course, but there’s really no way for the user (or the drone) to know what it can’t handle until it can’t handle it.

Edge cases like these aside, the obstacle avoidance just works. Even if you’re not deliberately trying to fly into branches, it’s keeping a lookout for you all the time, which means that flying the drone goes from somewhat stressful to just pure fun. I can’t emphasize enough how amazing it is to be able to fly without worrying about running into things, and how great it feels to be able to hand the controller to someone who’s never flown a drone before and say, with complete confidence, “go ahead, fly it around!”

Skydio 2 vs. DJI Mavic Photo: Evan Ackerman/IEEE Spectrum Both the Skydio 2 and many models of DJI’s Mavic use visual obstacle avoidance, but the Skydio 2 is so much more advanced that you can’t really compare the two systems.

It’s important to note that there’s a huge difference between the sort of obstacle avoidance that you get with a DJI Mavic, and the sort of obstacle avoidance that you get with the Skydio 2. The objective of the Mavic’s obstacle avoidance is really there to prevent you from accidentally running into things, and in that capacity, it usually works. But there are two things to keep in mind here—first, not running into things is not the same as avoiding things, because avoiding things means planning several steps ahead, not just one step.

Second, there’s the fact that the Mavic’s obstacle detection only works most of the time. Fundamentally, I don’t trust my Mavic Pro, because sometimes the safety system doesn’t kick in for whatever reason and the drone ends up alarmingly close to something. And that’s actually fine, because with the Mavic, I expect to be piloting it. It’s for this same reason that I don’t care that my Parrot Anafi doesn’t have obstacle avoidance at all: I’m piloting it anyway, and I’m a careful pilot, so it just doesn’t matter. The Skydio 2 is totally and completely different. It’s in a class by itself, and you can’t compare what it can do to what anything else out there right now. Period.

Skydio 2 Tracking

Skydio’s big selling point on the Skydio 2 is that it’ll autonomously track you while avoiding obstacles. It does this visually, by watching where you go, predicting your future motion, and then planning its own motion to keep you in frame. The works better than you might expect, in that it’s really very good at not losing you. Obviously, the drone prioritizes not running into stuff over tracking you, which means that it may not always be where you feel like it should be. It’s probably trying to get there, but in obstacle dense environments, it can take some creative paths.

Having said that, I found it to be very consistent with keeping me in the frame, and I only managed to lose it when changing direction while fully occluded by an obstacle, or while it was executing an avoidance maneuver that was more dynamic than normal. If you deliberately try to hide from the drone it’s not that hard to do so if there are enough obstacles around, but I didn’t find the tracking to be something that I had to worry about it most cases. When tracking does fail and you’re not using the beacon, the drone will come to a hover. It won’t try and find you, but it will reacquire you if you get back into its field of view.

The Skydio 2 had no problem tracking me running through fairly dense trees:

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 had no problem chasing me around through these trees, even while I was asking it to continually change its tracking angle.  Click here for a full resolution clip.

It also managed to keep up with me as I rode my bike along a tree-lined road:

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 is easily fast enough to keep up with me on a bike, even while avoiding tree branches.  Click here for a full resolution clip.

It lost me when I asked it to follow very close behind me as I wove through some particularly branch-y trees, but it fails more or less gracefully by just sort of nope-ing out of situations when they start to get bad and coming to a hover somewhere safe.

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 knows better than to put itself into situations that it can’t handle, and will bail to a safe spot if things get too complicated.  Click here for a full resolution clip.

After a few days of playing with the drone, I started to get to the point where I could set it to track me and then just forget about it while I rode my bike or whatever, as opposed to constantly turning around to make sure it was still behind me, which is what I was doing initially. It’s a level of trust that I don’t think would be possible with any other drone. 

Should You Buy a Skydio 2? Photo: Evan Ackerman/IEEE Spectrum We think the Skydio 2 is fun and relaxing to fly, with unique autonomous intelligence that makes it worth the cost.

In case I haven’t said it often enough in this review, the Skydio 2 is an incredible piece of technology. As far as I know (as a robotics journalist, mind you), this represents the state of the art in commercial drone autonomy, and quite possibly the state of the art in drone autonomy, period. And it’s available for $999, which is expensive, but less money than a Mavic Pro 2. If you’re interested in a new drone, you should absolutely consider the Skydio 2.

There are some things to keep in mind—battery life is a solid but not stellar 20 minutes. Extra batteries are expensive at $99 each (the base kit includes just one). The controller and the beacon are also expensive, at $150 each. And while I think the Skydio 2 is definitely the drone you want to fly, it may not be the drone you want to travel with, since it’s bulky compared to other options.

But there’s no denying the fact that the experience is uniquely magical. Once you’ve flown the Skydio 2, you won’t want to fly anything else. This drone makes it possible to get pictures and videos that would be otherwise impossible, and you can do it completely on your own. You can trust the drone to do what it promises, as long as you’re mindful of some basic and common sense safety guidelines. And we’ve been told that the drone is only going to get smarter and more capable over time.

If you buy a Skydio 2, it comes with the following warranty from Skydio:

“If you’re operating your Skydio 2 within our Safe Flight guidelines, and it crashes, we’ll repair or replace it for free.”

Skydio trusts their drone to go out into a chaotic and unstructured world and dodge just about anything that comes its way. And after a week with this drone, I can see how they’re able to offer this kind of guarantee. This is the kind of autonomy that robots have been promising for years, and the Skydio 2 makes it real.

Detailed technical specifications are available on Skydio’s website, and if you have any questions, post a comment—we’ve got this drone for a little while longer, and I’d be happy to try out (nearly) anything with it.

Skydio 2 Review Video Highlights

This video is about 7 minutes of 4K, 30 fps footage directly from the Skydio 2. The only editing I did was cutting clips together, no stabilization or color correcting or anything like that. The drone will record in 4K 60 fps, so it gets smoother than this, but I, er, forgot to change the setting.

[ Skydio ]

Let me begin this review by saying that the Skydio 2 is one of the most impressive robots that I have ever seen. Over the last decade, I’ve spent enough time around robots to have a very good sense of what kinds of things are particularly challenging for them, and to set my expectations accordingly. Those expectations include things like “unstructured environments are basically impossible” and “full autonomy is impractically expensive” and “robot videos rarely reflect reality.”

Skydio’s newest drone is an exception to all of this. It’s able to fly autonomously at speed through complex environments in challenging real-world conditions in a way that’s completely effortless and stress-free for the end user, allowing you to capture the kind of video that would be otherwise impossible, even (I’m guessing) for professional drone pilots. When you see this technology in action, it’s (almost) indistinguishable from magic. 

Skydio 2 Price

To be clear, the Skydio 2 is not without compromises, and the price of $999 (on pre-order with delivery of the next batch expected in spring of 2020) requires some justification. But the week I’ve had with this drone has left me feeling like its fundamental autonomous capability is so far beyond just about anything that I’ve ever experienced that I’m questioning why I would every fly anything else ever again.

We’ve written extensively about Skydio, beginning in early 2016 when the company posted a video of a prototype drone dodging trees while following a dude on a bike. Even three years ago, Skydio’s tech was way better than anything we’d seen outside of a research lab, and in early 2018, they introduced their first consumer product, the Skydio R1. A little over a year later, Skydio has introduced the Skydio 2, which is smaller, smarter, and much more affordable. Here’s an overview video just to get you caught up:

Skydio sent me a Skydio 2 review unit last week, and while I’m reasonably experienced with drones in general, this is the first time I’ve tried a Skydio drone in person. I had a pretty good idea what to expect, and I was absolutely blown away. Like, I was giggling to myself while running through the woods as the drone zoomed around, deftly avoiding trees and keeping me in sight. Robots aren’t supposed to be this good.

A week is really not enough time to explore everything that the Skydio can do, especially Thanksgiving week in Washington, D.C. (a no-fly zone) in early winter. But I found a nearby state park in which I could legally and safely fly the drone, and I did my best to put the Skydio 2 through its paces.

Note: Throughout this review, we’ve got a bunch of GIFs to help illustrate different features of the drone. To fit them all in, these GIFs had to be heavily compressed. Underneath each GIF is a timestamped link to this YouTube video (also available at the bottom of the post), which you can click on to see the an extended cut of the original 4K 30 fps footage. And there’s a bunch of interesting extra video in there as well.

Skydio 2 Specs Photo: Evan Ackerman/IEEE Spectrum The Skydio 2 is primarily made out of magnesium, which (while light) is both heavier and more rigid and durable than plastic. The offset props (the back pair are above the body, and the front pair are below) are necessary to maintain the field of view of the navigation cameras.

The Skydio 2 both looks and feels like a well-designed and carefully thought-out drone. It’s solid, and a little on the heavy side as far as drones go—it’s primarily made out of magnesium, which (while light) is both heavier and more rigid and durable than plastic. The blue and black color scheme is far more attractive than you typically see with drones. 

Photo: Evan Ackerman/IEEE Spectrum To detect and avoid obstacles, the Skydio 2 uses an array of six 4K hemispherical cameras that feed data into an NVIDIA Jetson TX2 at 30 fps, with the drone processing a million points in 3D space per second to plan the safest path.

The Skydio 2 is built around an array of six hemispherical obstacle-avoidance cameras and the NVIDIA Jetson TX2 computing module that they’re connected to. This defines the placement of the gimbal, the motors and props, and the battery, since all of this stuff has to be as much as possible out of the view of the cameras in order for the drone to effectively avoid obstacles in any direction. 

Without the bottom-mounted battery attached, the drone is quite flat. The offset props (the back pair are above the body, and the front pair are below) are necessary to maintain the field of view of the obstacle-avoidance cameras. These hemispherical cameras are on the end of each of the prop arms as well as above and below the body of the drone. They look awfully exposed, even though each is protected from ground contact by a little fin. You need to make sure these cameras are clean and smudge-free, and Skydio includes a cleaning cloth for this purpose. Underneath the drone there are slots for microSD cards, one for recording from the camera and a second one that the drone uses to store data. The attention to detail extends to the SD card insertion, which has a sloped channel that guides the card securely into its slot. 

Once you snap the battery in, the drone goes from looking streamlined to looking a little chubby. Relative to other drones, the battery almost seems like an afterthought, like Skydio designed the drone and then remembered, “oops we have to add a battery somewhere, let’s just kludge it onto the bottom.” But again, the reason for this is to leave room inside the body for the NVIDIA TX2, while making sure that the battery stays out of view of the obstacle avoidance cameras.

The magnetic latching system for the battery is both solid and satisfying. I’m not sure why it’s necessary, strictly speaking, but I do like it, and it doesn’t seem like the battery will fly off even during the most aggressive maneuvers. Each battery includes an LED array that will display its charge level in 25 percent increments, as well as a button that you push to turn the drone on and off. Charging takes place via a USB-C port in the top of the drone, which I don’t like, because it means that the batteries can’t be charged on their own (like the Parrot Anafi’s battery), and that you can’t charge one battery while flying with another, like basically every other drone ever. A separate battery charger that will charge two at once is available from Skydio for an eyebrow-raising $129.

I appreciate that all of Skydio’s stuff (batteries, controller, and beacon) charges via USB-C, though. The included USB-C adapter with its beefy cable will output at up to 65 watts, which’ll charge a mostly depleted battery in under an hour. The drone turns itself on while charging, which seems unnecessary.

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2 is not foldable, making it not nearly as easy to transport as some other drones. But it does come with a nice case that mitigates this issue somewhat, and the drone plus two batteries end up as a passably flat package about the size of a laptop case.

The most obvious compromise that Skydio made with the Skydio 2 is that the drone is not foldable. Skydio CEO Adam Bry told us that adding folding joints to the arms of the Skydio 2 would have made calibrating all six cameras a nightmare and significantly impacted performance. This makes complete sense, of course, but it does mean that the Skydio 2 is not nearly as easy to transport as some other drones. 

Photo: Evan Ackerman/IEEE Spectrum Folded and unfolded: The Skydio 2 compared to the Parrot Anafi (upper left) and the DJI Mavic Pro (upper right).

The Skydio 2 does come with a very nice case that mitigates this issue somewhat, and the drone plus two batteries end up as a passably flat package about the size of a laptop case. Still, it’s just not as convenient to toss into a backpack as my Anafi, although the Mavic Mini might be even more portable.

Photo: Evan Ackerman/IEEE Spectrum While the Skydio 2’s case is relatively compact, the non-foldable drone is overall a significantly larger package than the Parrot Anafi.

The design of the drone leads to some other compromises as well. Since landing gear would, I assume, occlude the camera system, the drone lands directly on the bottom of its battery pack, which has a slightly rubberized pad about the size of a playing card. This does’t feel particularly stable unless you end up on a very flat surface, and made me concerned for the exposed cameras underneath the drone as well as the lower set of props. I’d recommend hand takeoffs and landings—more on those later.

Skydio 2 Camera System

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2’s primary camera is a Sony IMX577 1/2.3" 12.3-megapixel CMOS sensor. It’s mounted to a three-axis gimbal and records 4K video at 60 fps, or 1080p video at 120 fps.

The Skydio 2 comes with a three-axis gimbal supporting a 12-megapixel camera, just enough to record 4K video at 60 fps, or 1080p video at 120 fps. Skydio has provided plenty of evidence that its imaging system is at least as good if not better than other drone cameras. Tested against my Mavic Pro and Parrot Anafi, I found no reason to doubt that. To be clear, I didn’t do exhaustive pixel-peeping comparisons between them, you’re just getting my subjective opinion that the Skydio 2 has a totally decent camera that you won’t be disappointed with. I will say that I found the HDR photo function to be not all that great under the few situations in which I tested it—after looking at a few muddy sunset shots, I turned it off and was much happier.

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2’s 12-megapixel camera is solid, although we weren’t impressed with the HDR option.

The video stabilization is fantastic, to the point where watching the video footage can be underwhelming because it doesn’t reflect the motion of the drone. I almost wish there was a way to change to unstabilized (or less-stabilized) video so that the viewer could get a little more of a wild ride. Or, ideally, there’d be a way for the drone to provide you with a visualization of what it was doing using the data collected by its cameras. That’s probably wishful thinking, though. The drone itself doesn’t record audio because all you’d get would be an annoying buzz, but the app does record audio, so the audio from your phone gets combined with the drone video. Don’t expect great quality, but it’s better than nothing.

Skydio 2 App

The app is very simple compared to every other drone app I’ve tried, and that’s a good thing. Here’s what it looks like:

Image: Skydio Trackable subjects get a blue “+” sign over them, and if you tap them, the “+” turns into a spinny blue circle. Once you’ve got a subject selected, you can choose from a variety of cinematic skills that the drone will execute while following you.

You get the controls that you need and the information that you need, and nothing else. Manual flight with the on-screen buttons works adequately, and the double-tap to fly function on the phone works surprisingly well, making it easy to direct the drone to a particular spot above the ground.

The settings menus are limited but functional, allowing you to change settings for the camera and a few basic tweaks for controlling the drone. One unique setting to the Skydio 2 is the height floor—since the drone only avoids static obstacles, you can set it to maintain a height of at least 8 feet above the ground while flying autonomously to make sure that if you’re flying around other people, it won’t run into anyone who isn’t absurdly tall and therefore asking for it.

Trackable subjects get a blue “+” sign over them in the app, and if you tap them, the “+” turns into a spinny blue circle. Once you’ve got a subject selected, you can choose from a variety of cinematic skills that the drone will execute while following you, and in addition, you can select “one-shot” skills that involve the drone performing a specific maneuver before returning to the previously selected cinematic skill. For example, you can tell the drone to orbit around you, and then do a “rocket” one-shot where it’ll fly straight up above you (recording the whole time, of course), before returning to its orbiting. 

After you’re done flying, you can scroll through your videos and easily clip out excerpts from them and save them to your phone for sharing. Again, it’s a fairly simple interface without a lot of options. You could call it limited, I guess, but I appreciate that it just does a few things that you care about and otherwise doesn’t clutter itself up.

The real limitation of the app is that it uses Wi-Fi to connect to the Skydio 2, which restricts the range. To fly much beyond a hundred meters or so, you’ll need to use the controller or beacon instead.

Skydio 2 Controller and Beacon Photo: Evan Ackerman/IEEE Spectrum While the Skydio 2 controller provides a better hands-on flight experience than with the phone, plus an extended range of up to 3.5 km, more experienced pilots may find manual control a bit frustrating, because the underlying autonomy will supersede your maneuvers when you start getting close to objects.

I was looking forward to using the controller, because with every other drone I’ve had, the precision that a physically controller provides is, I find, mandatory for a good flying experience and to get the photos and videos that you want. With Skydio 2, that’s all out the window. It’s not that the controller is useless or anything, it’s just that because the drone tracks you and avoids obstacles on its own, that level of control precision becomes largely unnecessary. 

The controller itself is perfectly fine. It’s a rebranded Parrot Skycontroller3, which is the same as the one that you get with a Parrot Anafi. It’s too bad that the sticks don’t unscrew to make it a little more portable, and overall it’s functional rather than fancy, but it feels good to use and includes a sizeable antenna that makes a significant difference to the range that you get (up to 3.5 kilometers).

You definitely get a better hands-on flight experience with the controller than with the phone, so if you want to (say) zip the drone around some big open space for fun, it’s good for that. And it’s nice to be able to hand the controller to someone who’s never flown a drone before and let them take it for a spin without freaking out about them crashing it the whole time. For more experienced pilots, though, the controller is ultimately just a bit frustrating, because the underlying autonomy will supersede your control when you start getting close to objects, which (again) limits how useful the controller is relative to your phone.

I do still prefer the controller over the phone, but I’m not sure that it’s worth the extra $150, unless you plan to fly the Skydio 2 at very long distances or primarily in manual mode. And honestly, if either of those two things are your top priority, the Skydio 2 is probably not the drone for you.

Photo: Evan Ackerman/IEEE Spectrum The Skydio 2 beacon uses GPS tracking to help the drone follow you, extending range up to 1.5 km. You can also fly the with the beacon alone, no phone necessary.

The purpose of the beacon, according to Skydio, is to give the drone a way of tracking you if it can’t see you, which can happen, albeit infrequently. My initial impression of the beacon was that it was primarily useful as a range-extending bridge between my phone and the drone. But I accidentally left my phone at home one day (oops) and had to fly the drone with only the beacon, and it was a surprisingly decent experience. The beacon allows for full manual control of a sort—you can tap different buttons to rotate, fly forward, and ascend or descend. This is sufficient for takeoff, landing, to make sure that the drone is looking at you when you engage visual tracking, and to rescue it if it gets trapped somewhere.

The rest of the beacon’s control functions are centered around a few different tracking modes, and with these, it works just about as well as your phone. You have fewer options overall, but all the basic stuff is there with just a few intuitive button clicks, including tracking range and angle. If you’re willing to deal with this relatively minor compromise, it’s nice to not have your phone available for other things rather than being monopolized by the drone.

Skydio 2 In Flight GIF: Evan Ackerman/IEEE Spectrum Hand takeoffs are simple and reliable.  Click here for a full resolution clip.

Starting up the Skydio 2 doesn’t require any kind of unusual calibration steps or anything like that. It prefers to be kept still, but you can start it up while holding it, it’ll just take a few seconds longer to tell you that it’s ready to go. While the drone will launch from any flat surface with significant clearance around it (it’ll tell you if it needs more room), the small footprint of the battery means that I was more comfortable hand launching it. This is not a “throw” launch; you just let the drone rest on your palm, tell it to take off, and then stay still while it gets its motors going and then gently lifts off. The lift off is so gentle that you have to be careful not to pull your hand away too soon—I did that once and the drone, being not quite ready, dropped towards the ground, but managed to recover without much drama.

GIF: Evan Ackerman/IEEE Spectrum Hand landings always look scary, but the Skydio 2 is incredibly gentle. After trying this once, it became the only way I ever landed the drone.  Click here for a full resolution clip.

Catching the drone for landing is perhaps very slightly more dangerous, but not any more difficult. You put the drone above and in front of you facing away, tell it to land in the app or with the beacon, and then put your hand underneath it to grasp it as it slowly descends. It settles delicately and promptly turns itself off. Every drone should land this way. The battery pack provides a good place to grip, although you do have to be mindful of the forward set of props, which (since they’re the pair that are beneath the body of drone) are quite close to your fingers. You’ll certainly be mindful after you catch a blade with your fingers once. Which I did. For the purposes of this review and totally not by accident. No damage, for the record.

Photo: Evan Ackerman/IEEE Spectrum You won’t be disappointed with the Skydio 2’s in-flight performance, unless you’re looking for a dedicated racing drone.

In normal flight, the Skydio 2 performs as well as you’d expect. It’s stable and manages light to moderate wind without any problems, although I did notice some occasional lateral drifting when the drone should have been in a stationary hover. While the controller gains are adjustable, the Skydio 2 isn’t quite as aggressive in flight as my Mavic Pro on Sport Mode, but again, if you’re looking for a high-speed drone, that’s really not what the Skydio is all about.

The Skydio 2 is substantially louder than my Anafi, although the Anafi is notably quiet for a drone. It’s not annoying to hear (not a high-pitched whine), but you can hear it from a ways away, and farther away than my Mavic Pro. I’m not sure whether that’s because of the absolute volume or the volume plus the pitch. In some ways, this is a feature, since you can hear the drone following you even if you’re not looking at it, you just need to be aware of the noise it makes when you’re flying it around people.

Obstacle Avoidance

The primary reason Skydio 2 is the drone that you want to fly is because of its autonomous subject tracking and obstacle avoidance. Skydio’s PR videos make this capability look almost too good, and since I hadn’t tried out one of their drones before, the first thing I did with it was exactly what you’d expect: attempt to fly it directly into the nearest tree.

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 deftly slides around trees and branches. The control inputs here were simple “forward” or “turn,” all obstacle avoidance is autonomous.  Click here for a full resolution clip.

And it just won’t do it. It slows down a bit, and then slides right around one tree after another, going over and under and around branches. I pointed the drone into a forest and just held down “forward” and away it went, without any fuss, effortlessly ducking and weaving its way around. Of course, it wasn’t effortless at all—six 4K cameras were feeding data into the NVIDIA TX2 at 30 fps, and the drone was processing a million points in 3D space per second to plan the safest path while simultaneously taking into account where I wanted it to go. I spent about 10 more minutes doing my level best to crash the drone into anything at all using a flying technique probably best described as “reckless,” but the drone was utterly unfazed. It’s incredible.

What knocked my socks off was telling the drone to pass through treetops—in the clip below, I’m just telling the drone to fly straight down. Watch as it weaves its way through gaps between the branches:

GIF: Evan Ackerman/IEEE Spectrum The result of parking the Skydio 2 above some trees and holding “down” on the controller is this impressive fully autonomous descent through the branches.  Click here for a full resolution clip.

Here’s one more example, where I sent the drone across a lake and started poking around in a tree. Sometimes the Skydio 2 isn’t sure where you want it to go, and you have to give it a little bit of a nudge in a clear direction, but that’s it.

GIF: Evan Ackerman/IEEE Spectrum In obstacle-heavy environments, the Skydio 2 prudently slows down, but it can pick its way through almost anything that it can see.  Click here for a full resolution clip.

It’s important to keep in mind that all of the Skydio 2’s intelligence is based on vision. It uses cameras to see the world, which means that it has similar challenges as your eyes do. Specifically, Skydio warns against flying in the following conditions:

  • Skydio 2 can’t see certain visually challenging obstacles. Do not fly around thin branches, telephone or power lines, ropes, netting, wires, chain link fencing or other objects less than ½ inch in diameter.
  • Do not fly around transparent surfaces like windows or reflective surfaces like mirrors greater than 60 cm wide.
  • When the sun is low on the horizon, it can temporarily blind Skydio 2’s cameras depending on the angle of flight. Your drone may be cautious or jerky when flying directly toward the sun.

Basically, if you’d have trouble seeing a thing, or seeing under some specific flight conditions, then the Skydio 2 almost certainly will also. It gets even more problematic when challenging obstacles are combined with challenging flight conditions, which is what I’m pretty sure led to the only near-crash I had with the drone. Here’s a video:

GIF: Evan Ackerman/IEEE Spectrum Flying around very thin branches and into the sun can cause problems for the Skydio 2’s obstacle avoidance.  Click here for a full resolution clip.

I had the Skydio 2 set to follow me on my bike (more about following and tracking in a bit). It was mid afternoon, but since it’s late fall here in Washington, D.C., the sun doesn’t get much higher than 30 degrees above the horizon. Late fall also means that most of the deciduous trees have lost their leaves, and so there are a bunch of skinny branches all over the place. The drone was doing a pretty good job of following me along the road at a relatively slow speed, and then it clipped the branch that you can just barely see in the video above. It recovered in an acrobatic maneuver that has been mostly video-stabilized out, and resumed tracking me before I freaked and told it to land. You can see another example here, where the drone (again) clips a branch that has the sun behind it, and this clip shows me stopping my bike before the drone runs into another branch in a similar orientation. As the video shows, it’s very hard to see the branches until it’s too late.

As far as I can tell, the drone is no worse for wear from any of this, apart from a small nick in one of the props. But, this is a good illustration of a problematic situation for the Skydio 2: flying into a low sun angle around small bare branches. Should I not have been flying the drone in this situation? It’s hard to say. These probably qualify as “thin branches,” although there was plenty of room along with middle of the road. There is an open question with the Skydio 2 as to exactly how much responsibility the user should have about when and where it’s safe to fly—for branches, how thin is too thin? How low can the sun be? What if the branches are only kinda thin and the sun is only kinda low, but it’s also a little windy? Better to be safe than sorry, of course, but there’s really no way for the user (or the drone) to know what it can’t handle until it can’t handle it.

Edge cases like these aside, the obstacle avoidance just works. Even if you’re not deliberately trying to fly into branches, it’s keeping a lookout for you all the time, which means that flying the drone goes from somewhat stressful to just pure fun. I can’t emphasize enough how amazing it is to be able to fly without worrying about running into things, and how great it feels to be able to hand the controller to someone who’s never flown a drone before and say, with complete confidence, “go ahead, fly it around!”

Skydio 2 vs. DJI Mavic Photo: Evan Ackerman/IEEE Spectrum Both the Skydio 2 and many models of DJI’s Mavic use visual obstacle avoidance, but the Skydio 2 is so much more advanced that you can’t really compare the two systems.

It’s important to note that there’s a huge difference between the sort of obstacle avoidance that you get with a DJI Mavic, and the sort of obstacle avoidance that you get with the Skydio 2. The objective of the Mavic’s obstacle avoidance is really there to prevent you from accidentally running into things, and in that capacity, it usually works. But there are two things to keep in mind here—first, not running into things is not the same as avoiding things, because avoiding things means planning several steps ahead, not just one step.

Second, there’s the fact that the Mavic’s obstacle detection only works most of the time. Fundamentally, I don’t trust my Mavic Pro, because sometimes the safety system doesn’t kick in for whatever reason and the drone ends up alarmingly close to something. And that’s actually fine, because with the Mavic, I expect to be piloting it. It’s for this same reason that I don’t care that my Parrot Anafi doesn’t have obstacle avoidance at all: I’m piloting it anyway, and I’m a careful pilot, so it just doesn’t matter. The Skydio 2 is totally and completely different. It’s in a class by itself, and you can’t compare what it can do to what anything else out there right now. Period.

Skydio 2 Tracking

Skydio’s big selling point on the Skydio 2 is that it’ll autonomously track you while avoiding obstacles. It does this visually, by watching where you go, predicting your future motion, and then planning its own motion to keep you in frame. The works better than you might expect, in that it’s really very good at not losing you. Obviously, the drone prioritizes not running into stuff over tracking you, which means that it may not always be where you feel like it should be. It’s probably trying to get there, but in obstacle dense environments, it can take some creative paths.

Having said that, I found it to be very consistent with keeping me in the frame, and I only managed to lose it when changing direction while fully occluded by an obstacle, or while it was executing an avoidance maneuver that was more dynamic than normal. If you deliberately try to hide from the drone it’s not that hard to do so if there are enough obstacles around, but I didn’t find the tracking to be something that I had to worry about it most cases. When tracking does fail and you’re not using the beacon, the drone will come to a hover. It won’t try and find you, but it will reacquire you if you get back into its field of view.

The Skydio 2 had no problem tracking me running through fairly dense trees:

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 had no problem chasing me around through these trees, even while I was asking it to continually change its tracking angle.  Click here for a full resolution clip.

It also managed to keep up with me as I rode my bike along a tree-lined road:

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 is easily fast enough to keep up with me on a bike, even while avoiding tree branches.  Click here for a full resolution clip.

It lost me when I asked it to follow very close behind me as I wove through some particularly branch-y trees, but it fails more or less gracefully by just sort of nope-ing out of situations when they start to get bad and coming to a hover somewhere safe.

GIF: Evan Ackerman/IEEE Spectrum The Skydio 2 knows better than to put itself into situations that it can’t handle, and will bail to a safe spot if things get too complicated.  Click here for a full resolution clip.

After a few days of playing with the drone, I started to get to the point where I could set it to track me and then just forget about it while I rode my bike or whatever, as opposed to constantly turning around to make sure it was still behind me, which is what I was doing initially. It’s a level of trust that I don’t think would be possible with any other drone. 

Should You Buy a Skydio 2? Photo: Evan Ackerman/IEEE Spectrum We think the Skydio 2 is fun and relaxing to fly, with unique autonomous intelligence that makes it worth the cost.

In case I haven’t said it often enough in this review, the Skydio 2 is an incredible piece of technology. As far as I know (as a robotics journalist, mind you), this represents the state of the art in commercial drone autonomy, and quite possibly the state of the art in drone autonomy, period. And it’s available for $999, which is expensive, but less money than a Mavic Pro 2. If you’re interested in a new drone, you should absolutely consider the Skydio 2.

There are some things to keep in mind—battery life is a solid but not stellar 20 minutes. Extra batteries are expensive at $99 each (the base kit includes just one). The controller and the beacon are also expensive, at $150 each. And while I think the Skydio 2 is definitely the drone you want to fly, it may not be the drone you want to travel with, since it’s bulky compared to other options.

But there’s no denying the fact that the experience is uniquely magical. Once you’ve flown the Skydio 2, you won’t want to fly anything else. This drone makes it possible to get pictures and videos that would be otherwise impossible, and you can do it completely on your own. You can trust the drone to do what it promises, as long as you’re mindful of some basic and common sense safety guidelines. And we’ve been told that the drone is only going to get smarter and more capable over time.

If you buy a Skydio 2, it comes with the following warranty from Skydio:

“If you’re operating your Skydio 2 within our Safe Flight guidelines, and it crashes, we’ll repair or replace it for free.”

Skydio trusts their drone to go out into a chaotic and unstructured world and dodge just about anything that comes its way. And after a week with this drone, I can see how they’re able to offer this kind of guarantee. This is the kind of autonomy that robots have been promising for years, and the Skydio 2 makes it real.

Detailed technical specifications are available on Skydio’s website, and if you have any questions, post a comment—we’ve got this drone for a little while longer, and I’d be happy to try out (nearly) anything with it.

Skydio 2 Review Video Highlights

This video is about 7 minutes of 4K, 30 fps footage directly from the Skydio 2. The only editing I did was cutting clips together, no stabilization or color correcting or anything like that. The drone will record in 4K 60 fps, so it gets smoother than this, but I, er, forgot to change the setting.

[ Skydio ]

This is part six of a six-part series on the history of natural language processing.

In February of this year, OpenAI, one of the foremost artificial intelligence labs in the world, announced that a team of researchers had built a powerful new text generator called the Generative Pre-Trained Transformer 2, or GPT-2 for short. The researchers used a reinforcement learning algorithm to train their system on a broad set of natural language processing (NLP) capabilities, including reading comprehension, machine translation, and the ability to generate long strings of coherent text. 

But as is often the case with NLP technology, the tool held both great promise and great peril. Researchers and policy makers at the lab were concerned that their system, if widely released, could be exploited by bad actors and misappropriated for “malicious purposes.”

The people of OpenAI, which defines its mission as “discovering and enacting the path to safe artificial general intelligence,” were concerned that GPT-2 could be used to flood the Internet with fake text, thereby degrading an already fragile information ecosystem. For this reason, OpenAI decided that it would not release the full version of GPT-2 to the public or other researchers.

GPT-2 is an example of a technique in NLP called language modeling, whereby the computational system internalizes a statistical blueprint of a text so it’s able to mimic it. Just like the predictive text on your phone—which selects words based on words you’ve used before—GPT-2 can look at a string of text and then predict what the next word is likely to be based on the probabilities inherent in that text.

GPT-2 can be seen as a descendant of the statistical language modeling that the Russian mathematician A. A. Markov developed in the early 20th century (covered in part three of this series).

GPT-2 used cutting-edge machine learning algorithms to do linguistic analysis with over 1.5 million parameters.

What’s different with GPT-2, though, is the scale of the textual data modeled by the system. Whereas Markov analyzed a string of 20,000 letters to create a rudimentary model that could predict the likelihood of the next letter of a text being a consonant or a vowel, GPT-2 used 8 million articles scraped from Reddit to predict what the next word might be within that entire dataset.

And whereas Markov manually trained his model by counting only two parameters—vowels and consonants—GPT-2 used cutting-edge machine learning algorithms to do linguistic analysis with over 1.5 million parameters, burning through huge amounts of computational power in the process.

The results were impressive. In their blog post, OpenAI reported that GPT-2 could generate synthetic text in response to prompts, mimicking whatever style of text it was shown. If you prompt the system with a line of William Blake’s poetry, it can generate a line back in the Romantic poet’s style. If you prompt the system with a cake recipe, you get a newly invented recipe in response.

Perhaps the most compelling feature of GPT-2 is that it can answer questions accurately. For example, when OpenAI researchers asked the system, “Who wrote the book The Origin of Species?”—it responded: “Charles Darwin.” While only able to respond accurately some of the time, the feature does seem to be a limited realization of Gottfried Leibniz’s dream of a language-generating machine that could answer any and all human questions (described in part two of this series).

After observing the power of the new system in practice, OpenAI elected not to release the fully trained model. In the lead up to its release in February, there had been heightened awareness about “deepfakes”—synthetic images and videos, generated via machine learning techniques, in which people do and say things they haven’t really done and said. Researchers at OpenAI worried that GPT-2 could be used to essentially create deepfake text, making it harder for people to trust textual information online.

Responses to this decision varied. On one hand, OpenAI’s caution prompted an overblown reaction in the media, with articles about the “dangerous” technology feeding into the Frankenstein narrative that often surrounds developments in AI.

Others took issue with OpenAI’s self-promotion, with some even suggesting that OpenAI purposefully exaggerated GPT-2s power in order to create hype—while contravening a norm in the AI research community, where labs routinely share data, code, and pre-trained models. As machine learning researcher Zachary Lipton tweeted, “Perhaps what's *most remarkable* about the @OpenAI controversy is how *unremarkable* the technology is. Despite their outsize attention & budget, the research itself is perfectly ordinary—right in the main branch of deep learning NLP research.”

OpenAI stood by its decision to release only a limited version of GPT-2, but has since released larger models for other researchers and the public to experiment with. As yet, there has been no reported case of a widely distributed fake news article generated by the system. But there have been a number of interesting spin-off projects, including GPT-2 poetry and a webpage where you can prompt the system with questions yourself.

Mimicking humans on Reddit, the bots have long conversations about a variety of topics, including conspiracy theories and Star Wars movies. 

There’s even a Reddit group populated entirely with text produced by GPT-2-powered bots. Mimicking humans on Reddit, the bots have long conversations about a variety of topics, including conspiracy theories and Star Wars movies.

This bot-powered conversation may signify the new condition of life online, where language is increasingly created by a combination of human and non-human agents, and where maintaining the distinction between human and non-human, despite our best efforts, is increasingly difficult.

The idea of using rules, mechanisms, and algorithms to generate language has inspired people in many different cultures throughout history. But it’s in the online world that this powerful form of wordcraft may really find its natural milieu—in an environment where the identity of speakers becomes more ambiguous, and perhaps, less relevant. It remains to be seen what the consequences will be for language, communication, and our sense of human identity, which is so bound up with our ability to speak in natural language.

This is the sixth installment of a six-part series on the history of natural language processing. Last week’s post explained how an innocent Microsoft chatbot turned instantly racist on Twitter.

You can also check out our prior series on the untold history of AI.

The manner in which humans learn, plan, and decide actions is a very compelling subject. Moreover, the mechanism behind high-level cognitive functions, such as action planning, language understanding, and logical thinking, has not yet been fully implemented in robotics. In this paper, we propose a framework for the simultaneously comprehension of concepts, actions, and language as a first step toward this goal. This can be achieved by integrating various cognitive modules and leveraging mainly multimodal categorization by using multilayered multimodal latent Dirichlet allocation (mMLDA). The integration of reinforcement learning and mMLDA enables actions based on understanding. Furthermore, the mMLDA, in conjunction with grammar learning and based on the Bayesian hidden Markov model (BHMM), allows the robot to verbalize its own actions and understand user utterances. We verify the potential of the proposed architecture through experiments using a real robot.

Autism Spectrum Disorder (ASD) is a spectrum of neurodevelopmental conditions characterized by difficulties in social communication and social interaction as well as repetitive behaviors and restricted interests. Prevalence rates have been rising, and existing diagnostic methods are both extremely time and labor consuming. There is an urgent need for more economic and objective automatized diagnostic tools that are independent of language and experience of the diagnostician and that can help deal with the complexity of the autistic phenotype. Technological advancements in machine learning are offering a potential solution, and several studies have employed computational approaches to classify ASD based on phenomenological, behavioral or neuroimaging data. Despite of being at the core of ASD diagnosis and having the potential to be used as a behavioral marker for machine learning algorithms, only recently have movement parameters been used as features in machine learning classification approaches. In a proof-of-principle analysis of data from a social interaction study we trained a classification algorithm on intrapersonal synchrony as an automatically and objectively measured phenotypic feature from 29 autistic and 29 typically developed individuals to differentiate those individuals with ASD from those without ASD. Parameters included nonverbal motion energy values from 116 videos of social interactions. As opposed to previous studies to date, our classification approach has been applied to non-verbal behavior objectively captured during naturalistic and complex interactions with a real human interaction partner assuring high external validity. A machine learning approach lends itself particularly for capturing heterogeneous and complex behavior in real social interactions and will be essential in developing automatized and objective classification methods in ASD.

Welcome to the eighth edition of IEEE Spectrum’s Robot Gift Guide!

This year we’re featuring 15 robotic products that we think will make fantastic holiday gifts. As always, we tried to include a broad range of robot types and prices, focusing mostly on items released this year. (A reminder: While we provide links to places where you can buy these items, we’re not endorsing any in particular, and a little bit of research may result in better deals.)

If you need even more robot gift ideas, take a look at our past guides: 201820172016201520142013, and 2012. Some of those robots are still great choices and might be way cheaper now than when we first posted about them. And if you have suggestions that you’d like to share, post a comment below to help the rest of us find the perfect robot gift.

  1. Skydio 2 Image: Skydio

    What makes robots so compelling is their autonomy, and the Skydio 2 is one of the most autonomous robots we’ve ever seen. It uses an array of cameras to map its environment and avoid obstacles in real-time, making flight safe and effortless and enabling the kinds of shots that would be impossible otherwise. Seriously, this thing is magical, and it’s amazing that you can actually buy one.

    $1,000
    Skydio
  2. UBTECH Jimu MeeBot 2 Image: UBTECH

    The Jimu MeeBot 2.0 from UBTECH is a STEM education robot designed to be easy to build and program. It includes six servo motors, a color sensor, and LED lights. An app for iPhone or iPad provides step-by-step 3D instructions, and helps you code different behaviors for the robot. It’s available exclusively from Apple.

    $130
    Apple
  3. iRobot Roomba s9+ Image: iRobot

    We know that $1,400 is a crazy amount of money to spend on a robot vacuum, but the Roomba s9+ is a crazy robot vacuum. As if all of its sensors and mapping intelligence wasn’t enough, it empties itself, which means that you can have your floors vacuumed every single day for a month and you don’t have to even think about it. This is what home robots are supposed to be.

    $1,400
    iRobot
  4. PFF Gita Photo: Piaggio Fast Forward

    Nobody likes carrying things, which is why Gita is perfect for everyone with an extra $3,000 lying around. Developed by Piaggio Fast Forward, this autonomous robot will follow you around with a cargo hold full of your most important stuff, and do it in a way guaranteed to attract as much attention as possible.

    $3,250
    Gita
  5. DJI Mavic Mini Photo: DJI

    It’s tiny, it’s cheap, and it takes good pictures—what more could you ask for from a drone? And for $400, this is an excellent drone to get if you’re on a budget and comfortable with manual flight. Keep in mind that while the Mavic Mini is small enough that you don’t need to register it with the FAA, you do still need to follow all the same rules and regulations.

    $400
    DJI
  6. LEGO Star Wars Droid Commander Image: LEGO

    Designed for kids ages 8+, this LEGO set includes more than 1,000 pieces, enough to build three different droids: R2-D2, Gonk Droid, and Mouse Droid. Using a Bluetooth-controlled robotic brick called Move Hub, which connects to the LEGO BOOST Star Wars app, kids can change how the robots behave and solve challenges, learning basic robotics and coding skills.

    $200
    LEGO
  7. Sony Aibo Photo: Sony

    Robot pets don’t get much more sophisticated (or expensive) than Sony’s Aibo. Strictly speaking, it’s one of the most complex consumer robots you can buy, and Sony continues to add to Aibo’s software. Recent new features include user programmability, and the ability to “feed” it. 

    $2,900 (free aibone and paw pads until 12/29/2019)
    Sony
  8. Neato Botvac D4 Connected Photo: Neato

    The Neato Botvac D4 may not have all of the features of its fancier and more expensive siblings, but it does have the features that you probably care the most about: The ability to make maps of its environment for intelligent cleaning (using lasers!), along with user-defined no-go lines that keep it where you want it. And it cleans quite well, too.

    $530  $350 (sale)
    Neato Robotics 
  9. Cubelets Curiosity Set Photo: Modular Robotics

    Cubelets are magnetic blocks that you can snap together to make an endless variety of robots with no programming and no wires. The newest set, called Curiosity, is designed for kids ages 4+ and comes with 10 robotic cubes. These include light and distance sensors, motors, and a Bluetooth module, which connects the robot constructions to the Cubelets app.

    $250
    Modular Robotics
  10. Tertill Photo: Franklin Robotics

    Tertill does one simple job: It weeds your garden. It’s waterproof, dirt proof, solar powered, and fully autonomous, meaning that you can leave it out in your garden all summer and just enjoy eating your plants rather than taking care of them.

    $350
    Tertill
  11. iRobot Root Photo: iRobot

    Root was originally developed by Harvard University as a tool to help kids progressively learn to code. iRobot has taken over Root and is now supporting the curriculum, which starts for kids before they even know how to read and should keep them busy for years afterwards.

    $200
     iRobot
  12. LOVOT Image: Lovot

    Let’s be honest: Nobody is really quite sure what LOVOT is. We can all agree that it’s kinda cute, though. And kinda weird. But cute. Created by Japanese robotics startup Groove X, LOVOT does have a whole bunch of tech packed into its bizarre little body and it will do its best to get you to love it.

    $2,750 (¥300,000)
    LOVOT
  13. Sphero RVR Photo: Sphero

    RVR is a rugged, versatile, easy to program mobile robot. It’s a development platform designed to be a bridge between educational robots like Sphero and more sophisticated and expensive systems like Misty. It’s mostly affordable, very expandable, and comes from a company with a lot of experience making robots.

    $250
    Sphero
  14. "How to Train Your Robot" Image: Lawrence Hall of Science

    Aimed at 4th and 5th graders, “How to Train Your Robot,” written by Blooma Goldberg, Ken Goldberg, and Ashley Chase, and illustrated by Dave Clegg, is a perfect introduction to robotics for kids who want to get started with designing and building robots. But the book isn’t just for beginners: It’s also a fun, inspiring read for kids who are already into robotics and want to go further—it even introduces concepts like computer simulations and deep learning. You can download a free digital copy or request hardcopies here.

    Free
    UC Berkeley
  15. MIT Mini Cheetah Photo: MIT

    Yes, Boston Dynamics’ Spotnow available for lease, is probably the world’s most famous quadruped, but MIT is starting to pump out Mini Cheetahs en masse for researchers, and while we’re not exactly sure how you’d manage to get one of these things short of stealing one directly for MIT, a Mini Cheetah is our fantasy robotics gift this year. Mini Cheetah looks like a ton of fun—it’s portable, highly dynamic, super rugged, and easy to control. We want one!

    Price N/A
    MIT Biomimetic Robotics Lab

For more tech gift ideas, see also IEEE Spectrum’s annual Gift Guide.

Welcome to the eighth edition of IEEE Spectrum’s Robot Gift Guide!

This year we’re featuring 15 robotic products that we think will make fantastic holiday gifts. As always, we tried to include a broad range of robot types and prices, focusing mostly on items released this year. (A reminder: While we provide links to places where you can buy these items, we’re not endorsing any in particular, and a little bit of research may result in better deals.)

If you need even more robot gift ideas, take a look at our past guides: 201820172016201520142013, and 2012. Some of those robots are still great choices and might be way cheaper now than when we first posted about them. And if you have suggestions that you’d like to share, post a comment below to help the rest of us find the perfect robot gift.

  1. Skydio 2 Image: Skydio

    What makes robots so compelling is their autonomy, and the Skydio 2 is one of the most autonomous robots we’ve ever seen. It uses an array of cameras to map its environment and avoid obstacles in real-time, making flight safe and effortless and enabling the kinds of shots that would be impossible otherwise. Seriously, this thing is magical, and it’s amazing that you can actually buy one.

    $1,000
    Skydio
  2. UBTECH Jimu MeeBot 2 Image: UBTECH

    The Jimu MeeBot 2.0 from UBTECH is a STEM education robot designed to be easy to build and program. It includes six servo motors, a color sensor, and LED lights. An app for iPhone or iPad provides step-by-step 3D instructions, and helps you code different behaviors for the robot. It’s available exclusively from Apple.

    $130
    Apple
  3. iRobot Roomba s9+ Image: iRobot

    We know that $1,400 is a crazy amount of money to spend on a robot vacuum, but the Roomba s9+ is a crazy robot vacuum. As if all of its sensors and mapping intelligence wasn’t enough, it empties itself, which means that you can have your floors vacuumed every single day for a month and you don’t have to even think about it. This is what home robots are supposed to be.

    $1,400
    iRobot
  4. PFF Gita Photo: Piaggio Fast Forward

    Nobody likes carrying things, which is why Gita is perfect for everyone with an extra $3,000 lying around. Developed by Piaggio Fast Forward, this autonomous robot will follow you around with a cargo hold full of your most important stuff, and do it in a way guaranteed to attract as much attention as possible.

    $3,250
    Gita
  5. DJI Mavic Mini Photo: DJI

    It’s tiny, it’s cheap, and it takes good pictures—what more could you ask for from a drone? And for $400, this is an excellent drone to get if you’re on a budget and comfortable with manual flight. Keep in mind that while the Mavic Mini is small enough that you don’t need to register it with the FAA, you do still need to follow all the same rules and regulations.

    $400
    DJI
  6. LEGO Star Wars Droid Commander Image: LEGO

    Designed for kids ages 8+, this LEGO set includes more than 1,000 pieces, enough to build three different droids: R2-D2, Gonk Droid, and Mouse Droid. Using a Bluetooth-controlled robotic brick called Move Hub, which connects to the LEGO BOOST Star Wars app, kids can change how the robots behave and solve challenges, learning basic robotics and coding skills.

    $200
    LEGO
  7. Sony Aibo Photo: Sony

    Robot pets don’t get much more sophisticated (or expensive) than Sony’s Aibo. Strictly speaking, it’s one of the most complex consumer robots you can buy, and Sony continues to add to Aibo’s software. Recent new features include user programmability, and the ability to “feed” it. 

    $2,900 (free aibone and paw pads until 12/29/2019)
    Sony
  8. Neato Botvac D4 Connected Photo: Neato

    The Neato Botvac D4 may not have all of the features of its fancier and more expensive siblings, but it does have the features that you probably care the most about: The ability to make maps of its environment for intelligent cleaning (using lasers!), along with user-defined no-go lines that keep it where you want it. And it cleans quite well, too.

    $530  $350 (sale)
    Neato Robotics 
  9. Cubelets Curiosity Set Photo: Modular Robotics

    Cubelets are magnetic blocks that you can snap together to make an endless variety of robots with no programming and no wires. The newest set, called Curiosity, is designed for kids ages 4+ and comes with 10 robotic cubes. These include light and distance sensors, motors, and a Bluetooth module, which connects the robot constructions to the Cubelets app.

    $250
    Modular Robotics
  10. Tertill Photo: Franklin Robotics

    Tertill does one simple job: It weeds your garden. It’s waterproof, dirt proof, solar powered, and fully autonomous, meaning that you can leave it out in your garden all summer and just enjoy eating your plants rather than taking care of them.

    $350
    Tertill
  11. iRobot Root Photo: iRobot

    Root was originally developed by Harvard University as a tool to help kids progressively learn to code. iRobot has taken over Root and is now supporting the curriculum, which starts for kids before they even know how to read and should keep them busy for years afterwards.

    $200
     iRobot
  12. LOVOT Image: Lovot

    Let’s be honest: Nobody is really quite sure what LOVOT is. We can all agree that it’s kinda cute, though. And kinda weird. But cute. Created by Japanese robotics startup Groove X, LOVOT does have a whole bunch of tech packed into its bizarre little body and it will do its best to get you to love it.

    $2,750 (¥300,000)
    LOVOT
  13. Sphero RVR Photo: Sphero

    RVR is a rugged, versatile, easy to program mobile robot. It’s a development platform designed to be a bridge between educational robots like Sphero and more sophisticated and expensive systems like Misty. It’s mostly affordable, very expandable, and comes from a company with a lot of experience making robots.

    $250
    Sphero
  14. "How to Train Your Robot" Image: Lawrence Hall of Science

    Aimed at 4th and 5th graders, “How to Train Your Robot,” written by Blooma Goldberg, Ken Goldberg, and Ashley Chase, and illustrated by Dave Clegg, is a perfect introduction to robotics for kids who want to get started with designing and building robots. But the book isn’t just for beginners: It’s also a fun, inspiring read for kids who are already into robotics and want to go further—it even introduces concepts like computer simulations and deep learning. You can download a free digital copy or request hardcopies here.

    Free
    UC Berkeley
  15. MIT Mini Cheetah Photo: MIT

    Yes, Boston Dynamics’ Spotnow available for lease, is probably the world’s most famous quadruped, but MIT is starting to pump out Mini Cheetahs en masse for researchers, and while we’re not exactly sure how you’d manage to get one of these things short of stealing one directly for MIT, a Mini Cheetah is our fantasy robotics gift this year. Mini Cheetah looks like a ton of fun—it’s portable, highly dynamic, super rugged, and easy to control. We want one!

    Price N/A
    MIT Biomimetic Robotics Lab

For more tech gift ideas, see also IEEE Spectrum’s annual Gift Guide.

Pedestrians' acceptance of automated vehicles (AVs) depends on their trust in the AVs. We developed a model of pedestrians' trust in AVs based on AV driving behavior and traffic signal presence. To empirically verify this model, we conducted a human–subject study with 30 participants in a virtual reality environment. The study manipulated two factors: AV driving behavior (defensive, normal, and aggressive) and the crosswalk type (signalized and unsignalized crossing). Results indicate that pedestrians' trust in AVs was influenced by AV driving behavior as well as the presence of a signal light. In addition, the impact of the AV's driving behavior on trust in the AV depended on the presence of a signal light. There were also strong correlations between trust in AVs and certain observable trusting behaviors such as pedestrian gaze at certain areas/objects, pedestrian distance to collision, and pedestrian jaywalking time. We also present implications for design and future research.

For the last two decades, the development of conducting polymers (CP) as artificial muscles, by materials researchers and chemists, has made establishing a reliable and repeatable synthesis of such materials possible. CP-based milli-robots were mostly unknown in soft robotics, however, today, they play a vital role in robotics and smart materials forums. Indeed, this subclass of soft robots has reached a crucial moment in their history, a moment where they can display rather interesting features, based on established foundations in terms of modeling, control, sensing, and planning in various applications. The purpose of this paper is to present the potential of conductive polymer-based soft milli-robots as high-performance devices for vacuum applications. To that end, a trilayer polypyrrole-based actuator was first used inside a scanning electron microscope (SEM), characterized for different applied voltages, over a relatively long period. Additionally, the tip positioning of the cantilever was also controlled using a closed-loop control. Furthermore, as a proof of concept for more complex soft milli-robots, an S-shaped soft milli-robot was modeled, using a hybrid model comprised of two models; a multi-physics model and a kinematic model. It was then fabricated using laser machining and finally characterized using its tip displacement. polypyrrole-based soft milli-robots proved to have tremendous potential as high-performance soft robots at the microscale for a wide range of applications, including SEM micro-manipulation as well as biomedical applications.

Gif: Bob O’Connor/IEEE Spectrum

With their jaw-dropping agility and animal-like reflexes, Boston Dynamics’ bioinspired robots have always seemed to have no equal. But that preeminence hasn’t stopped the company from pushing its technology to new heights, sometimes literally. Its latest crop of legged machines can trudge up and down hills, clamber over obstacles, and even leap into the air like a gymnast. There’s no denying their appeal: Every time Boston Dynamics uploads a new video to YouTube, it quickly racks up millions of views. These are probably the first robots you could call Internet stars.

Spot Photo: Bob O’Connor
  • 84 cm

    HEIGHT

  • 25 kg

    WEIGHT

  • 5.76 km/h

    SPEED

  • SENSING: Stereo cameras, inertial measurement unit, position/force sensors

  • ACTUATION: 12 DC motors

  • POWER: Battery (90 minutes per charge)

Boston Dynamics, once owned by Google’s parent company, Alphabet, and now by the Japanese conglomerate SoftBank, has long been secretive about its designs. Few publications have been granted access to its Waltham, Mass., headquarters, near Boston. But one morning this past August, IEEE Spectrum got in. We were given permission to do a unique kind of photo shoot that day. We set out to capture the company’s robots in action—running, climbing, jumping—by using high-speed cameras coupled with powerful strobes. The results you see on this page: freeze-frames of pure robotic agility.

We also used the photos to create interactive views, which you can explore online on our Robots Guide. These interactives let you spin the robots 360 degrees, or make them walk and jump on your screen.

Boston Dynamics has amassed a minizoo of robotic beasts over the years, with names like BigDog, SandFlea, and WildCat. When we visited, we focused on the two most advanced machines the company has ever built: Spot, a nimble quadruped, and Atlas, an adult-size humanoid.

Spot can navigate almost any kind of terrain while sensing its environment. Boston Dynamics recently made it available for lease, with plans to manufacture something like a thousand units per year. It envisions Spot, or even packs of them, inspecting industrial sites, carrying out hazmat missions, and delivering packages. And its YouTube fame has not gone unnoticed: Even entertainment is a possibility, with Cirque du Soleil auditioning Spot as a potential new troupe member.

“It’s really a milestone for us going from robots that work in the lab to these that are hardened for work out in the field,” Boston Dynamics CEO Marc Raibert says in an interview.

Atlas Photo: Bob O’Connor
  • 150 cm

    HEIGHT

  • 80 kg

    WEIGHT

  • 5.4 km/h

    SPEED

  • SENSING: Lidar and stereo vision

  • ACTUATION: 28 hydraulic actuators

  • POWER: Battery

Our other photographic subject, Atlas, is Boston Dynamics’ biggest celebrity. This 150-centimeter-tall (4-foot-11-inch-tall) humanoid is capable of impressive athletic feats. Its actuators are driven by a compact yet powerful hydraulic system that the company engineered from scratch. The unique system gives the 80-kilogram (176-pound) robot the explosive strength needed to perform acrobatic leaps and flips that don’t seem possible for such a large humanoid to do. Atlas has inspired a string of parody videos on YouTube and more than a few jokes about a robot takeover.

While Boston Dynamics excels at making robots, it has yet to prove that it can sell them. Ever since its founding in 1992 as a spin-off from MIT, the company has been an R&D-centric operation, with most of its early funding coming from U.S. military programs. The emphasis on commercialization seems to have intensified after the acquisition by SoftBank, in 2017. SoftBank’s founder and CEO, Masayoshi Son, is known to love robots—and profits.

The launch of Spot is a significant step for Boston Dynamics as it seeks to “productize” its creations. Still, Raibert says his long-term goals have remained the same: He wants to build machines that interact with the world dynamically, just as animals and humans do. Has anything changed at all? Yes, one thing, he adds with a grin. In his early career as a roboticist, he used to write papers and count his citations. Now he counts YouTube views.

  • In the Spotlight Photo: Bob O’Connor

    Boston Dynamics designed Spot as a versatile mobile machine suitable for a variety of applications. The company has not announced how much Spot will cost, saying only that it is being made available to select customers, which will be able to lease the robot. A payload bay lets you add up to 14 kilograms of extra hardware to the robot’s back. One of the accessories that Boston Dynamics plans to offer is a 6-degrees-of-freedom arm, which will allow Spot to grasp objects and open doors.

  • Super Senses Photo: Bob O’Connor

    Spot’s hardware is almost entirely custom-designed. It includes powerful processing boards for control as well as sensor modules for perception. The ­sensors are located on the front, rear, and sides of the robot’s body. Each module consists of a pair of stereo cameras, a wide-angle camera, and a texture projector, which enhances 3D sensing in low light. The sensors allow the robot to use the navigation method known as SLAM, or simultaneous localization and mapping, to get around autonomously.

  • Stepping Up Photo: Bob O’Connor

    In addition to its autonomous behaviors, Spot can also be steered by a remote operator with a game-style controller. But even when in manual mode, the robot still exhibits a high degree of autonomy. If there’s an obstacle ahead, Spot will go around it. If there are stairs, Spot will climb them. The robot goes into these operating modes and then performs the related actions completely on its own, without any input from the operator. To go down a flight of stairs, Spot walks backward, an approach Boston Dynamics says provides greater stability.

  • Funky Feet Gif: Bob O’Connor/IEEE Spectrum

    Spot’s legs are powered by 12 custom DC motors, each geared down to provide high torque. The robot can walk forward, sideways, and backward, and trot at a top speed of 1.6 meters per second. It can also turn in place. Other gaits include crawling and pacing. In one wildly popular YouTube video, Spot shows off its fancy footwork by dancing to the pop hit “Uptown Funk.”

  • Robot Blood Photo: Bob O’Connor

    Atlas is powered by a hydraulic system consisting of 28 actuators. These actuators are basically cylinders filled with pressurized fluid that can drive a piston with great force. Their high performance is due in part to custom servo valves that are significantly smaller and lighter than the aerospace models that Boston Dynamics had been using in earlier designs. Though not visible from the outside, the innards of an Atlas are filled with these hydraulic actuators as well as the lines of fluid that connect them. When one of those lines ruptures, Atlas bleeds the hydraulic fluid, which happens to be red.

  • Next Generation Gif: Bob O’Connor/IEEE Spectrum

    The current version of Atlas is a thorough upgrade of the original model, which was built for the DARPA Robotics Challenge in 2015. The newest robot is lighter and more agile. Boston Dynamics used industrial-grade 3D printers to make key structural parts, giving the robot greater strength-to-weight ratio than earlier designs. The next-gen Atlas can also do something that its predecessor, famously, could not: It can get up after a fall.

  • Walk This Way Photo: Bob O’Connor

    To control Atlas, an operator provides general steering via a manual controller while the robot uses its stereo cameras and lidar to adjust to changes in the environment. Atlas can also perform certain tasks autonomously. For example, if you add special bar-code-type tags to cardboard boxes, Atlas can pick them up and stack them or place them on shelves.

  • Biologically Inspired Photos: Bob O’Connor

    Atlas’s control software doesn’t explicitly tell the robot how to move its joints, but rather it employs mathematical models of the underlying physics of the robot’s body and how it interacts with the environment. Atlas relies on its whole body to balance and move. When jumping over an obstacle or doing acrobatic stunts, the robot uses not only its legs but also its upper body, swinging its arms to propel itself just as an athlete would.

This article appears in the December 2019 print issue as “By Leaps and Bounds.”

Reinforcement learning is generally accepted to be an appropriate and successful method to learn robot control. Symbolic action planning is useful to resolve causal dependencies and to break a causally complex problem down into a sequence of simpler high-level actions. A problem with the integration of both approaches is that action planning is based on discrete high-level action- and state spaces, whereas reinforcement learning is usually driven by a continuous reward function. Recent advances in model-free reinforcement learning, specifically, universal value function approximators and hindsight experience replay, have focused on goal-independent methods based on sparse rewards that are only given at the end of a rollout, and only if the goal has been fully achieved. In this article, we build on these novel methods to facilitate the integration of action planning with model-free reinforcement learning. Specifically, the paper demonstrates how the reward-sparsity can serve as a bridge between the high-level and low-level state- and action spaces. As a result, we demonstrate that the integrated method is able to solve robotic tasks that involve non-trivial causal dependencies under noisy conditions, exploiting both data and knowledge.

Large display environments are highly suitable for immersive analytics. They provide enough space for effective co-located collaboration and allow users to immerse themselves in the data. To provide the best setting—in terms of visualization and interaction—for the collaborative analysis of a real-world task, we have to understand the group dynamics during the work on large displays. Among other things, we have to study, what effects different task conditions will have on user behavior. In this paper, we investigated the effects of task conditions on group behavior regarding collaborative coupling and territoriality during co-located collaboration on a wall-sized display. For that, we designed two tasks: a task that resembles the information foraging loop and a task that resembles the connecting facts activity. Both tasks represent essential sub-processes of the sensemaking process in visual analytics and cause distinct space/display usage conditions. The information foraging activity requires the user to work with individual data elements to look into details. Here, the users predominantly occupy only a small portion of the display. In contrast, the connecting facts activity requires the user to work with the entire information space. Therefore, the user has to overview the entire display. We observed 12 groups for an average of 2 h each and gathered qualitative data and quantitative data in the form of surveys, field notes, video recordings, tracking data, and system logs. During data analysis, we focused specifically on participants' collaborative coupling (in particular, collaboration tightness, coupling styles, user roles, and task subdivision strategies) and territorial behavior. Our results both confirm and extend findings from the previous tabletop and wall-sized display studies. We could detect that participants tended to subdivide the task to approach it, in their opinion, in a more effective way, in parallel. We describe the subdivision strategies for both task conditions. We also detected and described multiple user roles, as well as a new coupling style that does not fit in either category: loosely or tightly. Moreover, we could observe a territory type that has not been mentioned previously in research. In our opinion, this territory type can affect the collaboration process of groups with more than two collaborators negatively. Finally, we investigated critical display regions in terms of ergonomics. We could detect that users perceived some regions as less comfortable for long-time work. The findings can be valuable for groupware interface design and development of group behavior models for analytical reasoning and decision making.

Complexity measures and information theory metrics in general have recently been attracting the interest of multi-agent and robotics communities, owing to their capability of capturing relevant features of robot behaviors, while abstracting from implementation details. We believe that theories and tools from complex systems science and information theory may be fruitfully applied in the near future to support the automatic design of robot swarms and the analysis of their dynamics. In this paper we discuss opportunities and open questions in this scenario.

There’s a definite sense that robots are destined to become a critical part of search and rescue missions and disaster relief efforts, working alongside humans to help first responders move faster and more efficiently. And we’ve seen all kinds of studies that include the claim “this robot could potentially help with disaster relief,” to varying degrees of plausibility.

But it takes a long time, and a lot of extra effort, for academic research to actually become anything useful—especially for first responders, where there isn’t a lot of financial incentive for further development.

It turns out that if you actually ask first responders what they most need for disaster relief, they’re not necessarily interested in the latest and greatest robotic platform or other futuristic technology. They’re using commercial off-the-shelf drones, often consumer-grade ones, because they’re simple and cheap and great at surveying large areas. The challenge is doing something useful with all of the imagery that these drones collect. Computer vision algorithms could help with that, as long as those algorithms are readily accessible and nearly effortless to use. 

The IEEE Robotics and Automation Society and the Center for Robotic-Assisted Search and Rescue (CRASAR) at Texas A&M University have launched a contest to bridge this gap between the kinds of tools that roboticists and computer vision researchers might call “basic” and a system that’s useful to first responders in the field. It’s a simple and straightforward idea, and somewhat surprising that no one had thought of it before now. And if you can develop such a system, it’s worth some cash.

CRASAR does already have a Computer Vision Emergency Response Toolkit (created right after Hurricane Harvey), which includes a few pixel filters and some edge and corner detectors. Through this contest, you can get paid your share of a $3,000 prize pool for adding some other excessively basic tools, including:

  • Image enhancement through histogram equalization, which can be applied to electro-optical (visible light cameras) and thermal imagery

  • Color segmentation for a range

  • Grayscale segmentation for a range in a thermal image

If it seems like this contest is really not that hard, that’s because it isn’t. “The first thing to understand about this contest is that strictly speaking, it’s really not that hard,” says Robin Murphy, director of CRASAR. “This contest isn’t necessarily about coming up with algorithms that are brand new, or even state-of-the-art, but rather algorithms that are functional and reliable and implemented in a way that’s immediately [usable] by inexperienced users in the field.”

Murphy readily admits that some of what needs to be done is not particularly challenging at all, but that’s not the point—the point is to make these functionalities accessible to folks who have better things to do than solve these problems themselves, as Murphy explains.

“A lot of my research is driven by problems that I’ve seen in the field that you’d think somebody would have solved, but apparently not. More than half of this is available in OpenCV, but who’s going to find it, download it, learn Python, that kind of thing? We need to get these tools into an open framework. We’re happy if you take libraries that already exist (just don’t steal code)—not everything needs to be rewritten from scratch. Just use what’s already there. Some of it may seem too simple, because it IS that simple. It already exists and you just need to move some code around.”

If you want to get very slightly more complicated, there’s a second category that involves a little bit of math:

Coders must provide a system that does the following for each nadir image in a set:

  1. Reads the geotag embedded in the .jpg
  2. Overlays a USNG grid for a user-specified interval (e.g., every 50, 100, or 200 meters)
  3. Gives the GPS coordinates of each pixel if a cursor is rolled over the image
  4. Given a set of images with the GPS or USNG coordinate and a bounding box, finds all images in the set that have a pixel intersecting that location

The final category awards prizes to anyone who comes up with anything else that turns out to be useful. Or, more specifically, “entrants can submit any algorithm they believe will be of value.” Whether or not it’s actually of value will be up to a panel of judges that includes both first responders and computer vision experts. More detailed rules can be found here, along with sample datasets that you can use for testing.

The contest deadline is 16 December, so you’ve got about a month to submit an entry. Winners will be announced at the beginning of January.

.carousel-inner{ height:560px !important; } 1/4

The SnotBot drone passes over a blue whale at the moment of exhalation. Photo: Christian Miller/Ocean Alliance

2/4

A humpback whale rolls as a drone approaches. Photo: Christian Miller/Ocean Alliance

3/4

SnotBot passes over a surfacing humpback whale off the coast of Gabon, in Africa. Photo: Christian Miller/Ocean Alliance

4/4

The drone approaches a blue whale mother and calf in the Gulf of California. Photo: Christian Miller/Ocean Alliance

Previous Next $(document).ready(function(){$('#SS1393381543').carousel({pause: true,interval: false});});

It’s a beautiful morning on the waters of Alaska’s Peril Strait—clear, calm, silent, and just a little cool. A small but seaworthy research vessel glides through gentle swells. Suddenly, in the distance, a humpback whale the size of a school bus explodes out of the water. Enormous bursts of air and water jet out of its blowholes like a fire hose, the noise echoing between the banks.

“Blow at eleven o’clock!” cries the lookout, and the small boat swarms with activity. A crew member wearing a helmet and cut-proof gloves raises a large quadcopter drone over his head, as if offering it to the sun, which glints off the half dozen plastic petri dishes velcroed to the drone.

Further back in the boat, the drone pilot calls, “Starting engines in 3, 2, 1! Takeoff in 3, 2, 1!” The drone’s engines buzz as it zooms 20 meters into the air and then darts off toward where the whale just dipped below the water’s surface. With luck, the whale will spout again nearby, and the drone will be there when it does.

The drone is a modified DJI Inspire 2. About the size of a toaster oven, it’s generally sold to photographers, cinematographers, and well-heeled hobbyists, but this particular drone is on a serious mission: to monitor the health of whales, the ocean, and by extension, the planet. The petri dishes it carries collect the exhaled breath condensate of a whale—a.k.a. snot—which holds valuable information about the creature’s health, diet, and other qualities. Hence the drone’s name: the Parley SnotBot.

The flyer comes standard with a forward-facing camera for navigation, collision-avoidance detectors, ultrasonic and barometric sensors to track altitude, and a GPS locator. With the addition of a high-definition video camera on a stabilized gimbal that can be directed independently, it can stream 1080p video live while simultaneously storing the video on a microSD card as well as high-resolution images on a 1-terabyte solid-state drive. Given that both cameras run during the entire 26 minutes of a typical flight, that’s a lot of data. More on what we are doing with that data later, but first, a bit of SnotBot history.

1/4

Petri dishes aboard SnotBot collect whale exhalate for later analysis. Photo: Christian Miller/Ocean Alliance

2/4

Intel Labs Research Scientist Bryn Keller, holding the drone, and Iain Kerr, the chief executive officer of Ocean Alliance, retrieve petri dishes that were sprayed with whale exhalation. Photo: Christian Miller/Ocean Alliance

3/4

Bryn Keller checks SnotBot before a launch. Photo: Christian Miller/Ocean Alliance

4/4

Ian Kerr, sitting under the awning, and other researchers send SnotBot on a hunt for whales in the Gulf of California. Photo: Christian Miller/Ocean Alliance

Previous Next $(document).ready(function(){$('#SS259234354').carousel({pause: true,interval: false});});

Iain Kerr was one of the early pioneers in using drones as a platform to collect and analyze whale exhalation. He’s the CEO of Ocean Alliance, in Gloucester, Mass., a group dedicated to protecting whales and the world’s oceans. Whale biologists know that whale snot contains an enormous amount of biological information, including DNA, hormones, and microorganisms. Scientists can use that information to determine a whale’s health, sex, and pregnancy status, and details about its genetics and microbiome. The traditional and most often used technique for collecting that kind of information is to zoom past a surfacing whale in a boat and shoot it with a specially designed crossbow to capture a small core sample of skin and blubber. The process is stressful for both researchers and whales.

Researchers had demonstrated that whale snot can be a viable replacement for blubber samples, but collection involved reaching out over whales using long, awkward poles—difficult, to say the least. The development of small but powerful commercial drones inspired Kerr to launch an exploratory research project in 2015 to go after whale snot with drones. He received the first U.S. National Oceanic and Atmospheric Administration (NOAA) research permit for collecting whale snot in U.S. waters. Since then, there have been dozens of SnotBot missions around the world, in the waters off Alaska, Gabon, Mexico, and other places where whales like to congregate, and the idea has spread to other teams around the globe.

The SnotBot design continues to evolve. The earliest versions tried to capture snot by trailing gauzy cloth below the drone. The hanging cloth turned out to be difficult to work with, however, and the material itself interfered with some of the lab tests, so the researchers scrapped that method. The developers didn’t consider using petri dishes at first, because they assumed that if the drone flew directly into a whale’s spout, the rotor wash would interfere with collection. Eventually, though, they tried the petri dishes and were happy to discover that the rotors’ downdraft improved rather than hindered collection.

For each mission, the collection goals have been slightly different, and the team tweaks the design of the craft accordingly. On one mission, the focus might be to survey an area, getting samples from as many whales as possible. The next mission might be a “focal follow,” in which the team tracks one whale over a period of hours or days, taking multiple samples so that they can understand things like how a whale’s hormone levels change throughout the day, either from natural processes or as a response to environmental factors.

Collecting and analyzing snot is certainly an important way to assess whale health, but the SnotBot team suspected that the drone could do more. In early 2017, staffers from Parley for the Oceans, a nonprofit environmental group that was working with Ocean Alliance on the SnotBot project, contacted one of us (Willke) to find out just how much more.

Willke is a machine-learning and artificial-intelligence researcher who leads Intel’s Brain-Inspired Computing Lab, in Hillsboro, Ore. He immediately saw ways of expanding the information gathered by SnotBot. Willke enlisted two researchers in his lab—coauthor Keller and Javier Turek—and the three of us got to work on enhancing SnotBot’s mission.

The quadcopters used in the SnotBot project carry high-quality cameras with advanced auto-stabilization features. The drone pilot relies on the high-definition video being streamed back to the boat to fly the aircraft and collect the snot. We knew that these same video streams could simultaneously feed into a computer on the boat and be processed in real time. Could that information help assess whale health?

Working with Ocean Alliance scientists, we first came up with a tool that analyzes a photo of a whale’s tail flukes and, using a database of whale photographs collected by the Alaska Whale Foundation, identifies individual whales by the shape of the fluke and its black and white patterns. Identifying each whale allows researchers to correlate snot samples over time.

Such identification can also help whale biologists cope with tricky regulatory issues. For example, there are at least two breeding populations of humpback whales that migrate to Alaska. Most come from Hawaii, but a smaller group comes from Mexico. The Mexican population is under greater stress at the moment, and so NOAA requests that researchers focus on the healthier, Hawaiian whales and leave the Mexican whales alone as much as possible. However, both populations are exactly the same species and thus indistinguishable from each other as a group. The ability to recognize individual whales allows researchers to determine whether a whale had been previously spotted in Mexico or Hawaii, so that they can act appropriately to comply with the regulation.

We also developed software that analyzes the shape of a whale from an overhead shot, taken about 25 meters directly above the whale. Since a skinny whale is often a sick one or one that hasn’t been getting enough to eat, even that simple metric can be a powerful indicator of well-being.

The biggest challenge in developing these tools was what’s called data starvation—there just wasn’t enough data. A standard deep-learning algorithm would look at a huge set of images and then figure out and extract the key distinguishing features of a whale. In the case of the fluke-ID tool, there were only a few pictures of each whale in the catalog, and these were often too low quality to be useful. For overhead health monitoring, there were likewise too few photos or videos of whales shot with the right camera, from the right angle, under the right conditions.

To address these problems, our team turned to classic computer-vision techniques to extract what we considered the most useful data. For example, we used edge-detection algorithms to find and measure the trailing edge of a fluke, then obtained the grayscale values of all the pixels in a line extending from the center notch of the fluke to the outer tips. We trained a small but effective neural network on this data alone. If more data had been available, a deep-learning approach would have worked better than our approach did, but we had to work with the limited data we had.

1/4

The latest model of SnotBot flies into action, with custom mounting points for petri dishes and its new paint scheme, designed to camouflage it against a cloud-studded sky. Photo: Christian Miller/Ocean Alliance

2/4

SnotBot passes through whale exhalation. Photo: Christian Miller/Ocean Alliance

3/4

The shape of a whale's flukes can help software recognize individual whales. Photo: Christian Miller/Ocean Alliance

4/4

A humpback whale dives below the surface of the ocean. Photo: Kate Westaway/Getty Images

Previous Next $(document).ready(function(){$('#SS1500926266').carousel({pause: true,interval: false});});

New discoveries in whale biology have already come from our tools. Besides the ability to distinguish between the Mexican and Hawaiian whale populations, researchers have discovered they can identify whales from their calls, even when the calls were recorded many years previously.

That latter discovery came during the summer of 2017, when we joined Fred Sharpe, an Alaska Whale Foundation researcher and founding board member, to study teams of whales that worked together to feed. While observing a small group of humpback whales, the boat’s underwater microphone picked up a whale feeding call. Sharpe thought it sounded familiar, and so he consulted his database of whale vocalizations. He found a similar call from a whale called Trumpeter that he had recorded some 20 years ago. But was it really the same whale? There was no way to know for sure from the whale call.

Then a whale surfaced briefly and dove again, letting us capture an image of its flukes. Our software found a match: The flukes indeed belonged to Trumpeter. That told the researchers that adult whale feeding calls likely remain stable for decades, maybe even for life. This insight gave researchers another tool for identifying whales in the wild and improving our understanding of vocal signatures in humpback whales.

Meanwhile, whale-ID tools are getting better all the time. The original SnotBot algorithm that we developed for whale identification has been essentially supplanted by more capable services. One new algorithm relies on the curvature of the trailing edge of the fluke for identification.

SnotBot’s real contribution, it turns out, is in health monitoring. Our shape-analysis tool has been evolving and, in combination with the spray samples, is giving researchers a comprehensive picture of an individual whale’s health. We call this tool Morphometer. We recently teamed up with Kelly Cates, a Ph.D. candidate in marine biology at the University of Alaska Fairbanks, and Fredrik Christiansen, an assistant professor and whale expert at the Aarhus Institute of Advanced Studies, in Denmark, to make the technology more powerful and also easier to use.

Here’s how it works. Researchers who make measurements and assessments of baleen whales—the type of whales that filter-feed—have typically used a technique developed by Christiansen in 2016. (So far the effort has involved humpback and southern right whales, but the process could work for any kind of baleen whale.) The researchers start with photographic prints or images on a computer and hand-measure the body widths of whales in the images at intervals of 5 percent of the overall length from the snout to the notch of the tail flukes. They then feed this set of measurements to software that calculates an estimate of the whale’s volume. From the relationship between the body length and volume, they can determine if an individual whale is relatively fatter or thinner compared with population norms, taking into account the significant but normal changes in girth that occur as whales accumulate energy reserves during the feeding season and then use those energy stores for migration and during the breeding season.

Morphometer also uses photos, but it measures the whale’s width continuously at the highest resolution possible given the quality of the photo, yielding hundreds of width measurements for each animal, instead of only the small number of measurements that are feasible for human researchers. The result is thus much more accurate. It also processes the data much faster than a human could, allowing biologists to focus on biology rather than doing tedious measurements by hand.

To improve Morphometer, we trained a deep-learning system on images of humpback and southern right whales in all sorts of different weather, water, and lighting conditions to allow it to understand exactly which pixels in an image belong to a whale. Once a whale has been singled out, the system identifies the head and tail and then measures the whale’s length and width at each pixel point along the outline of its body. Our software tracks the altitude from which the drone photographed the whale and combines that data with camera specifications entered by the drone operator, allowing the system to automatically convert the measurements from pixels to meters.

Morphometer compares this whale with others of its body type, displaying the result as an image of the subject whale superimposed on a whale-shape color-coded diagram with zones indicating the average measurements of similar whales. It’s immediately obvious if the whale is normal size, underweight, or larger than average, as would be the case with pregnant females [see illustration, “Measuring Up”].

Images: Intel Measuring up: The Morphometer image-analysis tool [below left] compares one whale with others of its body type, displaying the result as an image of the subject whale superimposed on a diagram representing an average whale. The diagram is color coded—red at the center, blending to white at the size of an average whale and to blue for larger whales. With the individual whale superimposed on this color map, any red that shows around the whale means the whale is underweight; the more underweight, the darker the red. Above, fluke-recognition software makes a match.

For our early prototype, we input parameters for a “normal” body shape based on age, sex, and other factors. But now Morphometer is in the process of figuring out “normal” for itself by processing large numbers of whale images. Whale researchers who use their own drones to collect whale photos have been sending us their images. Eventually, we envision setting up a collaborative website that would allow images and morphometry models to be shared among researchers. We also plan to adapt Morphometer to analyze videos of whales, automatically extracting the frames or clips in which the whale’s position and visibility are the best.

To help researchers gain a more complete picture, we’re building statistical models of various whale populations, which we will compare to models derived from human-estimated measurements. Then we’ll take new photos of whales whose age and gender are known, and see whether the software correctly classifies them and gives appropriate indications of health; we’ll have whale biologists verify the results.

Once this model is working reliably, we expect to be able to say how a given whale’s size compares with those of its peers of the same gender, in the same region, at the same time of year. We’ll also be able to identify historical trends—for example, this whale is not skinnier than average compared with last year, but it is much skinnier than whales in its class a few decades ago, assuming comparison data exists. If, in addition, we have snot from the same whale, we can create a more complete profile of the whale, in the same way your credit card company can tell a lot about you by integrating your personal data with the averages and variances in the general population.

So far, SnotBot has told us a lot about the health of individual whales. Soon, researchers will start using this data to monitor the health of oceans. Whales are known as “apex predators,” meaning they are at the top of the food chain. Humpback whales in particular are generalist foragers and have wide-ranging migration patterns, which make them an excellent early-warning system for environmental threats to the ocean as a whole.

This is where SnotBot can really make a difference. We all depend on the oceans for our survival. Besides the vast amount of food they produce, we depend on them for the air we breathe: Most of the oxygen in the atmosphere comes from marine organisms such as phytoplankton and algae.

Lately, ocean productivity associated with a North Pacific warm-water anomaly, or “blob,” has resulted in a reduction of births and more reports of skinny whales, and that should worry us. If conditions are bad for whales, they’re also bad for humans. Thanks to Project SnotBot, we’ll be able to find out—accurately, efficiently, and at a reasonable cost—just how the health and numbers of whales in our oceans are trending. With that information, we hope, we will be able to spur society to take steps to protect the oceans before it’s too late.

The whale images in this article were obtained under National Marine Fisheries Service permits 18636-01 and 19703.

This article appears in the December 2019 print issue as “SnotBot: A Whale of a Deep-Learning Project.”

About the Authors

Bryn Keller is a deep-learning research scientist in Intel’s Brain-Inspired Computing Lab. Senior principal engineer Ted Willke is the lab’s director.

This is part five of a six-part series on the history of natural language processing.

In March 2016, Microsoft was preparing to release its new chatbot, Tay, on Twitter. Described as an experiment in “conversational understanding,” Tay was designed to engage people in dialogue through tweets or direct messages, while emulating the style and slang of a teenage girl. She was, according to her creators, “Microsoft’s A.I. fam from the Internet that’s got zero chill.” She loved E.D.M. music, had a favorite Pokémon, and often said extremely online things, like “swagulated.” 

Tay was an experiment at the intersection of machine learning, natural language processing, and social networks. While other chatbots in the past—like Joseph Weizenbaum’s Eliza—conducted conversation by following pre-programmed and narrow scripts, Tay was designed to learn more about language over time, enabling her to have conversations about any topic.

Tay was designed to learn more about language over time…. Eventually, her programmers hoped, Tay would sound just like the Internet. 

Machine learning works by developing generalizations from large amounts of data. In any given data set, the algorithm will discern patterns and then “learn” how to approximate those patterns in its own behavior.

Using this technique, engineers at Microsoft trained Tay’s algorithm on a dataset of anonymized public data along with some pre-written material provided by professional comedians to give it a basic grasp of language. The plan was to release Tay online, then let the bot discover patterns of language through its interactions, which she would emulate in subsequent conversations. Eventually, her programmers hoped, Tay would sound just like the Internet. 

On March 23, 2016, Microsoft released Tay to the public on Twitter. At first, Tay engaged harmlessly with her growing number of followers with banter and lame jokes. But after only a few hours, Tay started tweeting highly offensive things, such as: “I f@#%&*# hate feminists and they should all die and burn in hell” or “Bush did 9/11 and Hitler would have done a better job…”

Within 16 hours of her release, Tay had tweeted more than 95,000 times, and a troubling percentage of her messages were abusive and offensive. Twitter users started registering their outrage, and Microsoft had little choice but to suspend the account. What the company had intended on being a fun experiment in “conversational understanding” had become their very own golem, spiraling out of control through the animating force of language.

Over the next week, many reports emerged detailing precisely how a bot that was supposed to mimic the language of a teenage girl became so vile. It turned out that just a few hours after Tay was released, a post on the troll-laden bulletin board, 4chan, shared a link to Tay’s Twitter account and encouraged users to inundate the bot with racist, misogynistic, and anti-semitic language. 

In a coordinated effort, the trolls exploited a “repeat after me” function that had been built into Tay, whereby the bot repeated anything that was said to it on demand. But more than this, Tay’s in-built capacity to learn meant that she internalized some of the language she was taught by the trolls, and repeated it unprompted. For example, one user innocently asked Tay whether Ricky Gervais was an atheist, to which she responded: “Ricky Gervais learned totalitarianism from Adolf Hitler, the inventor of atheism.”

“Ricky Gervais learned totalitarianism from Adolf Hitler, the inventor of atheism.” —Tay, Microsoft’s rogue chatbot

The coordinated attack on Tay worked better than the 4channers expected and was discussed widely in the media in the weeks after. Some saw Tay’s failure as evidence of social media’s inherent toxicity, a place that brings out the worst in people and allows trolls to hide in anonymity.

For others, though, Tay’s behavior was evidence of poor design decisions on Microsoft’s behalf.

Zoë Quinn, a game developer and writer who’s been a frequent target of online abuse, argued that Microsoft should have been more cognizant of the context in which Tay was being released. If a bot learns how to speak on Twitter—a platform rife with abusive language—then naturally it will learn some abusive language. Microsoft, Quinn argued, should have planned for this contingency and ensured that Tay was not corrupted so easily. “It’s 2016,” she tweeted. “If you’re not asking yourself ‘how could this be used to hurt someone’ in your design/engineering process, you’ve failed.”

Some months after taking Tay down, Microsoft released Zo, a “politically correct” version of the original bot. Zo, who was active on social networks from 2016 to 2019, was designed to shut down conversations about certain contentious topics, including politics and religion, to ensure she didn’t offend people. (If a correspondent kept pressing her to talk about a certain sensitive topic, she left the conversation altogether, with a sentence like: “im better than u bye.”)

The lesson Microsoft learned the hard way is that designing computational systems that can communicate with people online is not just a technical problem, but a deeply social endeavor. Inviting a bot into the value-laden world of language requires thinking, in advance, about what context it will be deployed in, what type of communicator you want it to be, and what type of human values you want it to reflect.

As we move towards an online world in which bots are more prevalent, these questions must be at the forefront of the design process. Otherwise there will be more golems released into the world that will reflect back to us, in language, the worst parts of ourselves.

This is the fifth installment of a six-part series on the history of natural language processing. Last week’s post described people’s weird intimacy with a rudimentary chatbot created in 1966. Come back next Monday for part six, which tells of the controversy surrounding OpenAI’s magnificent language generator, GPT-2. 

You can also check out our prior series on the untold history of AI.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

DARPA SubT Urban Circuit – February 18-27, 2020 – Olympia, WA, USA

Let us know if you have suggestions for next week, and enjoy today's videos.

Kuka has just announced the results of its annual Innovation Award. From an initial batch of 30 applicants, five teams reached the finals (we were part of the judging committee). The five finalists worked for nearly a year on their applications, which they demonstrated this week at the Medica trade show in Düsseldorf, Germany. And the winner of the €20,000 prize is...Team RoboFORCE, led by the STORM Lab in the U.K., which developed a “robotic magnetic flexible endoscope for painless colorectal cancer screening, surveillance, and intervention.”

The system could improve colonoscopy procedures by reducing pain and discomfort as well as other risks such as bleeding and perforation, according to the STORM Lab researchers. It uses a magnetic field to control the endoscope, pulling rather than pushing it through the colon.

The other four finalists also presented some really interesting applications—you can see their videos below.

“Because we were so please with the high quality of the submissions, we will have next year’s finals again at the Medica fair, and the challenge will be named ‘Medical Robotics’,” says Rainer Bischoff, vice president for corporate research at Kuka. He adds that the selected teams will again use Kuka’s LBR Med robot arm, which is “already certified for integration into medical products and makes it particularly easy for startups to use a robot as the main component for a particular solution.”

Applications are now open for Kuka’s Innovation Award 2020. You can find more information on how to enter here. The deadline is 5 January 2020.

[ Kuka ]

Oh good, Aibo needs to be fed now.

You know what comes next, right?

[ Aibo ]

Your cat needs this robot.

It's about $200 on Kickstarter.

[ Kickstarter ]

Enjoy this tour of the Skydio offices courtesy Skydio 2, which runs into not even one single thing.

If any Skydio employees had important piles of papers on their desks, well, they don’t anymore.

[ Skydio ]

Artificial intelligence is everywhere nowadays, but what exactly does it mean? We asked a group MIT computer science grad students and post-docs how they personally define AI.

“When most people say AI, they actually mean machine learning, which is just pattern recognition.” Yup.

[ MIT ]

Using event-based cameras, this drone control system can track attitude at 1600 degrees per second (!).

[ UZH ]

Introduced at CES 2018, Walker is an intelligent humanoid service robot from UBTECH Robotics. Below are the latest features and technologies used during our latest round of development to make Walker even better.

[ Ubtech ]

Introducing the Alpha Prime by #VelodyneLidar, the most advanced lidar sensor on the market! Alpha Prime delivers an unrivaled combination of field-of-view, range, high-resolution, clarity and operational performance.

Performance looks good, but don’t expect it to be cheap.

[ Velodyne ]

Ghost Robotics’ Spirit 40 will start shipping to researchers in January of next year.

[ Ghost Robotics ]

Unitree is about to ship the first batch of their AlienGo quadrupeds as well:

[ Unitree ]

Mechanical engineering’s Sarah Bergbreiter discusses her work on micro robotics, how they draw inspiration from insects and animals, and how tiny robots can help humans in a variety of fields.

[ CMU ]

Learning contact-rich, robotic manipulation skills is a challenging problem due to the high-dimensionality of the state and action space as well as uncertainty from noisy sensors and inaccurate motor control. To combat these factors and achieve more robust manipulation, humans actively exploit contact constraints in the environment. By adopting a similar strategy, robots can also achieve more robust manipulation. In this paper, we enable a robot to autonomously modify its environment and thereby discover how to ease manipulation skill learning. Specifically, we provide the robot with fixtures that it can freely place within the environment. These fixtures provide hard constraints that limit the outcome of robot actions. Thereby, they funnel uncertainty from perception and motor control and scaffold manipulation skill learning.

[ Stanford ]

Since 2016, Verity's drones have completed more than 200,000 flights around the world. Completely autonomous, client-operated and designed for live events, Verity is making the magic real by turning drones into flying lights, characters, and props.

[ Verity ]

To monitor and stop the spread of wildfires, University of Michigan engineers developed UAVs that could find, map and report fires. One day UAVs like this could work with disaster response units, firefighters and other emergency teams to provide real-time accurate information to reduce damage and save lives. For their research, the University of Michigan graduate students won first place at a competition for using a swarm of UAVs to successfully map and report simulated wildfires.

[ University of Michigan ]

Here’s an important issue that I haven’t heard talked about all that much: How first responders should interact with self-driving cars.

“To put the car in manual mode, you must call Waymo.” Huh.

[ Waymo ]

Here’s what Gitai has been up to recently, from a Humanoids 2019 workshop talk.

[ Gitai ]

The latest CMU RI seminar comes from Girish Chowdhary at the University of Illinois at Urbana-Champaign on “Autonomous and Intelligent Robots in Unstructured Field Environments.”

What if a team of collaborative autonomous robots grew your food for you? In this talk, I will discuss some key advances in robotics, machine learning, and autonomy that will one day enable teams of small robots to grow food for you in your backyard in a fundamentally more sustainable way than modern mega-farms! Teams of small aerial and ground robots could be a potential solution to many of the serious problems that modern agriculture is facing. However, fully autonomous robots that operate without supervision for weeks, months, or entire growing season are not yet practical. I will discuss my group’s theoretical and practical work towards the underlying challenging problems in robotic systems, autonomy, sensing, and learning. I will begin with our lightweight, compact, and autonomous field robot TerraSentia and the recent successes of this type of undercanopy robots for high-throughput phenotyping with deep learning-based machine vision. I will also discuss how to make a team of autonomous robots learn to coordinate to weed large agricultural farms under partial observability. These direct applications will help me make the case for the type of reinforcement learning and adaptive control that are necessary to usher in the next generation of autonomous field robots that learn to solve complex problems in harsh, changing, and dynamic environments. I will then end with an overview of our new MURI, in which we are working towards developing AI and control that leverages neurodynamics inspired by the Octopus brain.

[ CMU RI ]

Pages