Feed aggregator

I’ll admit to having been somewhat skeptical about the strategy of dangling payloads on long tethers for drone delivery. I mean, I get why Wing does it— it keeps the drone and all of its spinny bits well away from untrained users while preserving the capability of making deliveries to very specific areas that may have nearby obstacles. But it also seems like you’re adding some risk as well, because once your payload is out on that long tether, it’s more or less out of your control in at least two axes. And you can forget about your drone doing anything while this is going on, because who the heck knows what’s going to happen to your payload if the drone starts moving around?

NYU roboticists, that’s who.

This research is by Guanrui Li, Alex Tunchez, and Giuseppe Loianno at the Agile Robotics and Perception Lab (ARPL) at NYU. As you can see from the video, the drone makes keeping rock-solid control over that suspended payload look easy, but it’s very much not, especially considering that everything you see is running onboard the drone itself at 500Hz— all it takes is an IMU and a downward-facing monocular camera, along with the drone’s Snapdragon processor.

To get this to work, the drone has to be thinking about two things. First, there’s state estimation, which is the behavior of the drone itself along with its payload at the end of the tether. The drone figures this out by watching how the payload moves using its camera and tracking its own movement with its IMU. Second, there’s predicting what the payload is going to do next, and how that jibes (or not) with what the drone wants to do next. The researchers developed a model predictive control (MPC) system for this, with some added perception constraints to make sure that the behavior of the drone keeps the payload in view of the camera. 

At the moment, the top speed of the system is 4 m/s, but it sounds like rather than increasing the speed of a single payload-swinging drone, the next steps will be to make the overall system more complicated by somehow using multiple drones to cooperatively manage tethered payloads that are too big or heavy for one drone to handle alone.

For more on this, we spoke with Giuseppe Loianno, head of the ARPL.

IEEE Spectrum: We've seen some examples of delivery drones delivering suspended loads. How will this work improve their capabilities?

Giuseppe Loianno: For the first time, we jointly design a perception-constrained model predictive control and state estimation approaches to enable the autonomy of a quadrotor with a cable suspended payload using onboard sensing and computation. The proposed control method guarantees the visibility of the payload in the robot camera as well as the respect of the system dynamics and actuator constraints. These are critical design aspects to guarantee safety and resilience for such a complex and delicate task involving transportation of objects.

The additional challenge involves the fact that we aim to solve the aforementioned problem using a minimal sensor suite for autonomous navigation made by a single camera and IMU. This is an ambitious goal since it concurrently involves estimating the load and the vehicle states. Previous approaches leverage GPS or motion capture systems for state estimation and do not consider the perception and physical constraints when solving the problem. We are confident that our solution will contribute to making a reality the autonomous delivery process in warehouses or in dense urban areas where the GPS signal is currently absent or shadowed.

Will it make a difference to delivery systems that use an actuated cable and only leave the load suspended for the delivery itself?

This is certainly an interesting question. We believe that adding an actuated cable will introduce more disadvantages than benefits. Certainly, an actuated cable can be leveraged to compensate for cable's swinging motions in windy conditions and/or increase the delivery precision. However, the introduction of additional actuated mechanisms and components come at the price of an increased system mass and inertia. This will reduce the overall flight time and the vehicle’s agility as well as the system resilience with respect to the transportation task. Finally, active mechanisms are also more difficult to design compared to passive ones.

What's challenging about doing all of this on-vehicle?

There are several challenges to solve on-board this problem. First, it is very difficult to concurrently run perception and action on such computationally constrained platforms in real-time. Second, the first aspect becomes even more challenging if we consider as in our case a perception-based constrained receding horizon control problem that aims to guarantee the visibility of the payload during the motion, while concurrently respecting all the system physical and sensing limitations. Finally, it has been challenging to run the entire system at a high rate to fully unleash the system’s agility. We are currently able to reach rates of 500 Hz.

Can your method adapt to loads of varying shapes, sizes, and masses? What about aerodynamics or flying in wind?

Technically, our approach can easily be adapted to varying objects sizes and masses. Our previous contributions have already shown the ability to estimate online changes in the vehicle/load configuration and can potentially be used to operate the proposed system in dynamic conditions, where the load’s characteristics are unknown and/or may vary across consecutive flights. This can be useful for both package delivery or warehouse operations, where different types of objects need to be transported or manipulated.

The aerodynamics problem is a great point. Overall, our past work has investigated the aerodynamics of wind disturbances for a single robot without a load. Formulating these problems for the proposed system is challenging and is still an open research question. We have some ideas to approach this problem combining Bayesian estimation techniques with more recent machine learning approaches and we will tackle it in the near future.

What are the limitations on the performance of the system? How fast and agile can it be with a suspended payload? 

The limits of the performances are established by the actuating and sensing system. Our approach intrinsically considers both physical and sensing limitations of our system. From a sensing and computation perspective, we believe to be close to the limits with speeds of up to 4 m/s. Faster speeds can potentially introduce motion blur while decreasing the load tracking precision. Moreover, faster motions will increase as well aerodynamic disturbances that we have just mentioned. In the future, modeling these phenomena and their incorporation in the proposed solution can further push the agility.

Your paper talks about extending this approach to multiple vehicles cooperatively transporting a payload, can you tell us more about that?

We are currently working on a distributed perception and control approach for cooperative transportation. We already have some very exciting results that we will share with you very soon! Overall, we can employ a team of aerial robots to cooperatively transport a payload to increase the payload capacity and endow the system with additional resilience in case of vehicles’ failures. A cooperative cable suspended payload cooperative transportation system allows as well to concurrently and independently control the load’s position and orientation. This is not possible just using rigid connections. We believe that our approach will have a strong impact in real-world settings for delivery and constructions in warehouses and GPS-denied environments such as dense urban areas. Moreover, in post disaster scenarios, a team of physically interconnected aerial robots can deliver supplies and establish communication in areas where GPS signal is intermittent or unavailable.

PCMPC: Perception-Constrained Model Predictive Control for Quadrotors with Suspended Loads using a Single Camera and IMU, by Guanrui Li, Alex Tunchez, and Giuseppe Loianno from NYU, will be presented (virtually) at ICRA 2021.

<Back to IEEE Journal Watch

We introduce a soft robot actuator composed of a pre-stressed elastomer film embedded with shape memory alloy (SMA) and a liquid metal (LM) curvature sensor. SMA-based actuators are commonly used as electrically-powered limbs to enable walking, crawling, and swimming of soft robots. However, they are susceptible to overheating and long-term degradation if they are electrically stimulated before they have time to mechanically recover from their previous activation cycle. Here, we address this by embedding the soft actuator with a capacitive LM sensor capable of measuring bending curvature. The soft sensor is thin and elastic and can track curvature changes without significantly altering the natural mechanical properties of the soft actuator. We show that the sensor can be incorporated into a closed-loop “bang-bang” controller to ensure that the actuator fully relaxes to its natural curvature before the next activation cycle. In this way, the activation frequency of the actuator can be dynamically adapted for continuous, cyclic actuation. Moreover, in the special case of slower, low power actuation, we can use the embedded curvature sensor as feedback for achieving partial actuation and limiting the amount of curvature change.

Model-Based Reinforcement Learning (MBRL) algorithms have been shown to have an advantage on data-efficiency, but often overshadowed by state-of-the-art model-free methods in performance, especially when facing high-dimensional and complex problems. In this work, a novel MBRL method is proposed, called Risk-Aware Model-Based Control (RAMCO). It combines uncertainty-aware deep dynamics models and the risk assessment technique Conditional Value at Risk (CVaR). This mechanism is appropriate for real-world application since it takes epistemic risk into consideration. In addition, we use a model-free solver to produce warm-up training data, and this setting improves the performance in low-dimensional environments and covers the shortage of MBRL’s nature in the high-dimensional scenarios. In comparison with other state-of-the-art reinforcement learning algorithms, we show that it produces superior results on a walking robot model. We also evaluate the method with an Eidos environment, which is a novel experimental method with multi-dimensional randomly initialized deep neural networks to measure the performance of any reinforcement learning algorithm, and the advantages of RAMCO are highlighted.

Over the past two decades, scholars developed various unmanned sailboat platforms, but most of them have specialized designs and controllers. Whereas these robotic sailboats have good performance with open-source designs, it is actually hard for interested researchers or fans to follow and make their own sailboats with these open-source designs. Thus, in this paper, a generic and flexible unmanned sailboat platform with easy access to the hardware and software architectures is designed and tested. The commonly used 1-m class RC racing sailboat was employed to install Pixhawk V2.4.8, Arduino Mega 2,560, GPS module M8N, custom-designed wind direction sensor, and wireless 433 Mhz telegram. The widely used open-source hardware modules were selected to keep reliable and low-cost hardware setup to emphasize the generality and feasibility of the unmanned sailboat platform. In software architecture, the Pixhawk V2.4.8 provided reliable states’ feedback. The Arduino Mega 2,560 received estimated states from Pixhawk V2.4.8 and the wind vane sensor, and then controlled servo actuators of rudder and sail using simplified algorithms. Due to the complexity of introducing robot operating system and its packages, we designed a generic but real-time software architecture just using Arduino Mega 2,560. A suitable line-of-sight guidance strategy and PID-based controllers were used to let the autonomous sailboat sail at user-defined waypoints. Field tests validated the sailing performance in facing WRSC challenges. Results of fleet race, station keeping, and area scanning proved that our design and algorithms could control the 1-m class RC sailboat with acceptable accuracy. The proposed design and algorithms contributed to developing educational, low-cost, micro class autonomous sailboats with accessible, generic, and flexible hardware and software. Besides, our sailboat platform also facilitates readers to develop similar sailboats with more focus on their missions.

Soft robots are ideal for underwater manipulation in sampling and other servicing applications. Their unique features of compliance, adaptability, and being naturally waterproof enable robotic designs to be compact and lightweight, while achieving uncompromized dexterity and flexibility. However, the inherent flexibility and high nonlinearity of soft materials also results in combined complex motions, which creates both soft actuator and sensor challenges for force output, modeling, and sensory feedback, especially under highly dynamic underwater environments. To tackle these limitations, a novel Soft Origami Optical-Sensing Actuator (SOSA) with actuation and sensing integration is proposed in this paper. Inspired by origami art, the proposed sensorized actuator enables a large force output, contraction/elongation/passive bending actuation by fluid, and hybrid motion sensing with optical waveguides. The SOSA design brings two major novelties over current designs. First, it involves a new actuation-sensing mode which enables a superior large payload output and a robust and accurate sensing performance by introducing the origami design, significantly facilitating the integration of sensing and actuating technology for wider applications. Secondly, it simplifies the fabrication process for harsh environment application by investigating the boundary features between optical waveguides and ambient water, meaning the external cladding layer of traditional sensors is unnecessary. With these merits, the proposed actuator could be applied to harsh environments for complex interaction/operation tasks. To showcase the performance of the proposed SOSA actuator, a hybrid underwater 3-DOFs manipulator has been developed. The entire workflow on concept design, fabrication, modeling, experimental validation, and application are presented in detail as reference for wider effective robot-environment applications.

When NASA first sent humans to the moon, astronauts often made risky blind landings on the lunar surface because of billowing dust clouds churned up during their descent. Astronauts could avoid repeating those harrowing experiences during future missions to the moon with the help of a 3D-printed lunar landing pad designed by a NASA-backed student team.

The landing pad developed by students from 10 U.S. universities and colleges is shaped to minimize the lunar dust clouds stirred up by rocket landing burns and could eventually be made from lunar regolith material found on the moon. A prototype of the pad is scheduled to undergo a rocket hot fire test under the watchful eye of both students and NASA engineers at Camp Swift, Texas in early March.

“We showed that you can 3D print the structure with our existing prototype,” says Helen Carson, a material science and engineering student at the University of Washington in Seattle and a principal investigator for the Lunar PAD team. “For now, we have a lot of flexibility with different directions we can take depending on how the materials develop.”

Such a Lunar PAD concept could prove especially helpful with NASA’s current roadmap aimed at returning humans to the moon through the Artemis Program; the U.S. space agency has already issued contracts to companies such as SpaceX, Blue Origin, and Dynetics to start developing ideas for a human lunar lander. Any future moon landings could benefit from reducing the risk of possible catastrophe that comes from flying blind in a dust cloud. Furthermore, dust and rocks accelerated to high speeds by engine exhaust could pose a serious danger to astronauts, robots, or other equipment already on the surface of the moon.

The Lunar PAD team first came together during NASA’s L’SPACE (Lucy Student Pipeline Accelerator and Competency Enabler) Virtual Academy held in the summer of 2019. Carson and her colleagues won funding from the NASA Proposal Writing and Evaluation process to move forward on the project and to make a presentation at NASA Marshall Space Flight Center in June 2020. At that event, additional funding was awarded so that the team could print and test their pad prototype. The students also presented a paper on Lunar PAD at the AIAA SciTech Forum and Exposition that was held 19-21 January 2021.

Image: Lunar PAD Team The multidisciplinary, multiuniversity team has come up with a solution to a problem that astronauts would most certainly face when humans return to the moon.

The team’s early idea included creating an inflatable deflector that would be inflated by the rocket engine exhaust and block any debris blasted outward from the landing (or launch) zone of the pad. But that would have required transporting flexible yet durable materials manufactured on Earth to the moon.

“That got pretty complicated with material choice and design, and the actual transportation of it,” says Luke Martin, a mechanical engineering student at Arizona State University. “So we tried coming up with other more in-situ resource ideas.”

Lunar PAD currently has a top surface layer where rockets and lunar landers could both land and launch. But the key to mitigating the worst of any dust clouds or small particles accelerated to high velocities is the open interior space of the pad that sits below the top layer. Slanted grates in the top layer would channel the rocket exhaust into the interior space.

The pad’s interior includes vent dividers—some shaped like teardrops or leaflets—that help channel the rocket exhaust and any accompanying dust or rock particles outward from the center of the pad. The cosmetically appealing layout of the vent dividers—which some liken to flower petals—proved to be the most efficient pattern that came out of numerous iterations tested through flow simulations.

“It's very practical, very efficient, and just so happens to also be very beautiful,” says Vincent Murai, a mechanical engineering student at Kapiolani Community College in Honolulu.

The exhaust and any accompanying particles leave the pad’s interior space through specific exits, called Kinetic Energy Diffusers, embedded in the outside walls of the circular pad. Such diffusers consist of hollow rectangular blocks that could also include fans to convert some of the rocket exhaust’s excess energy into the circular fan motion and block some particles with the turning fan blades. 

Any high-velocity particles that get through the fans would also encounter deflectors placed right outside the exits in the full-scale version of the pad. And an “apron” surrounding the landing pad would also include a perimeter deflector wall to direct any remaining exhaust-propelled particles up and away from any nearby spacecraft, people, or structures.

The subscale prototype of the pad was manufactured by a gantry-style 3D printer developed by the Austin-based company ICON. The company is already working with NASA to adapt its 3D printing technology for space-based construction on the moon and Mars.

3D printing the main layers of the subscale pad prototype took up just one day. The team also spent three additional days on tasks such as using the printer to fill various components with concrete and patching or smoothing certain parts of the pad. People also had to manually install fiber optic sensors to detect changes in strain and temperature.

But the most labor-intensive and hands-on part of the construction involved trimming and placing pre-cut blocks of water-soluble foam to provide temporary structural support for overhanging areas of the pad. Full-scale construction of such a pad on the moon or Mars would require a different and ideally more efficient solution for providing such removable supports.

“It became especially apparent after a few days of of cutting and wrapping and inserting foam that it's probably not the best use of an astronaut’s time,” says Andres Campbell, an integrated engineering student with an emphasis on aerospace engineering at Minnesota State University in Mankato and a principal investigator for the team. “This would also be something that would be robotically complex to do.”

In any case, a full-scale and operational Lunar PAD would not have to handle the dust mitigation work on its own. For example, Carson originally proposed an electrodynamic dust shielding technology that would passively push dust off the landing pad by taking advantage of the charged nature of lunar dust. Automated cleaning tools such what Campbell described as a “space Roomba” robot could also help keep the launch and landing zone dust free.

“The idea that you can combine the pad with not just electrodynamic dust shielding but any sort of passive dust mitigation system is still worth consideration,” Carson says. “Because in addition to that pad, you would still have dust that could be kicked up from other activities on the surface.”

The 3D-printed pad concept could eventually prove useful for future missions to Mars and other destinations. Such pad designs would have to account for some differences in atmosphere and gravity on rocket plumes and dust clouds, not to mention factors such as the moon’s electrostatically charged dust particles and Martian dust storms. Still, the team designed the pad to potentially work beyond lunar landing scenarios.

“Our goal was to build a reusable pad for all extraterrestrial environments,” Murai says.

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

HRI 2021 – March 8-11, 2021 – [Online Conference] RoboSoft 2021 – April 12-16, 2021 – [Online Conference] ICRA 2021 – May 30-5, 2021 – Xi'an, China

Let us know if you have suggestions for next week, and enjoy today's videos.

If you’ve ever swatted a mosquito away from your face, only to have it return again (and again and again), you know that insects can be remarkably acrobatic and resilient in flight. Those traits help them navigate the aerial world, with all of its wind gusts, obstacles, and general uncertainty. Such traits are also hard to build into flying robots, but MIT Assistant Professor Kevin Yufeng Chen has built a system that approaches insects’ agility.

Chen’s actuators can flap nearly 500 times per second, giving the drone insect-like resilience. “You can hit it when it’s flying, and it can recover,” says Chen. “It can also do aggressive maneuvers like somersaults in the air.” And it weighs in at just 0.6 grams, approximately the mass of a large bumble bee. The drone looks a bit like a tiny cassette tape with wings, though Chen is working on a new prototype shaped like a dragonfly.

[ MIT ]

National Robotics Week is April 3-11, 2021!

[ NRW ]

This is in a motion capture environment, but still, super impressive!

[ Paper ]

Thanks Fan!

Why wait for Boston Dynamics to add an arm to your Spot if you can just do it yourself?

[ ETHZ ]

This video shows the deep-sea free swimming of soft robot in the South China Sea. The soft robot was grasped by a robotic arm on ‘HAIMA’ ROV and reached the bottom of the South China Sea (depth of 3,224 m). After the releasing, the soft robot was actuated with an on-board AC voltage of 8 kV at 1 Hz and demonstrated free swimming locomotion with its flapping fins.

Um, did they bring it back?

[ Nature ]

Quadruped Yuki Mini is 12 DOF robot equipped with a Raspberry Pi that runs ROS. Also, BUNNIES!

[ Lingkang Zhang ]

Thanks Lingkang!

Deployment of drone swarms usually relies on inter-agent communication or visual markers that are mounted on the vehicles to simplify their mutual detection. The vswarm package enables decentralized vision-based control of drone swarms without relying on inter-agent communication or visual fiducial markers. The results show that the drones can safely navigate in an outdoor environment despite substantial background clutter and difficult lighting conditions.

[ Vswarm ]

A conventional adopted method for operating a waiter robot is based on the static position control, where pre-defined goal positions are marked on a map. However, this solution is not optimal in a dynamic setting, such as in a coffee shop or an outdoor catering event, because the customers often change their positions. We explore an alternative human-robot interface design where a human operator communicates the identity of the customer to the robot instead. Inspired by how [a] human communicates, we propose a framework for communicating a visual goal to the robot, through interactive two-way communications.

[ Paper ]

Thanks Poramate!

In this video, LOLA reacts to undetected ground height changes, including a drop and leg-in-hole experiment. Further tests show the robustness to vertical disturbances using a seesaw. The robot is technically blind, not using any camera-based or prior information on the terrain.

[ TUM ]

RaiSim is a cross-platform multi-body physics engine for robotics and AI. It fully supports Linux, Mac OS, and Windows.

[ RaiSim ]

Thanks Fan!

The next generation of LoCoBot is here. The LoCoBot is an ROS research rover for mapping, navigation and manipulation (optional) that enables researchers, educators and students alike to focus on high level code development instead of hardware and building out lower level code. Development on the LoCoBot is simplified with open source software, full ROS-mapping and navigation packages and modular opensource Python API that allows users to move the platform as well as (optional) manipulator in as few as 10 lines of code.

[ Trossen ]

MIT Media Lab Research Specialist Dr. Kate Darling looks at how robots are portrayed in popular film and TV shows.

Kate's book, The New Breed: What Our History with Animals Reveals about Our Future with Robots can be pre-ordered now and comes out next month.

[ Kate Darling ]

The current autonomous mobility systems for planetary exploration are wheeled rovers, limited to flat, gently-sloping terrains and agglomerate regolith. These vehicles cannot tolerate instability and operate within a low-risk envelope (i.e., low-incline driving to avoid toppling). Here, we present ‘Mars Dogs’ (MD), four-legged robotic dogs, the next evolution of extreme planetary exploration.

[ Team CoSTAR ]

In 2020, first-year PhD students at the MIT Media Lab were tasked with a special project—to reimagine the Lab and write sci-fi stories about the MIT Media Lab in the year 2050. “But, we are researchers. We don't only write fiction, we also do science! So, we did what scientists do! We used a secret time machine under the MIT dome to go to the year 2050 and see what’s going on there! Luckily, the Media Lab still exists and we met someone…really cool!” Enjoy this interview of Cyber Joe, AI Mentor for MIT Media Lab Students of 2050.

[ MIT ]

In this talk, we will give an overview of the diverse research we do at CSIRO’s Robotics and Autonomous Systems Group and delve into some specific technologies we have developed including SLAM and Legged robotics. We will also give insights into CSIRO’s participation in the current DARPA Subterranean Challenge where we are deploying a fleet of heterogeneous robots into GPS-denied unknown underground environments.

[ GRASP Seminar ]

Marco Hutter (ETH) and Hae-Won Park (KAIST) talk about “Robotics Inspired by Nature.”

[ Swiss-Korean Science Club ]

Thanks Fan!

In this keynote, Guy Hoffman Assistant Professor and the Mills Family Faculty Fellow in the Sibley School of Mechanical and Aerospace Engineering at Cornell University, discusses “The Social Uncanny of Robotic Companions.”

[ Designerly HRI ]

Video Friday is your weekly selection of awesome robotics videos, collected by your Automaton bloggers. We’ll also be posting a weekly calendar of upcoming robotics events for the next few months; here's what we have so far (send us your events!):

HRI 2021 – March 8-11, 2021 – [Online Conference] RoboSoft 2021 – April 12-16, 2021 – [Online Conference] ICRA 2021 – May 30-5, 2021 – Xi'an, China

Let us know if you have suggestions for next week, and enjoy today's videos.

If you’ve ever swatted a mosquito away from your face, only to have it return again (and again and again), you know that insects can be remarkably acrobatic and resilient in flight. Those traits help them navigate the aerial world, with all of its wind gusts, obstacles, and general uncertainty. Such traits are also hard to build into flying robots, but MIT Assistant Professor Kevin Yufeng Chen has built a system that approaches insects’ agility.

Chen’s actuators can flap nearly 500 times per second, giving the drone insect-like resilience. “You can hit it when it’s flying, and it can recover,” says Chen. “It can also do aggressive maneuvers like somersaults in the air.” And it weighs in at just 0.6 grams, approximately the mass of a large bumble bee. The drone looks a bit like a tiny cassette tape with wings, though Chen is working on a new prototype shaped like a dragonfly.

[ MIT ]

National Robotics Week is April 3-11, 2021!

[ NRW ]

This is in a motion capture environment, but still, super impressive!

[ Paper ]

Thanks Fan!

Why wait for Boston Dynamics to add an arm to your Spot if you can just do it yourself?

[ ETHZ ]

This video shows the deep-sea free swimming of soft robot in the South China Sea. The soft robot was grasped by a robotic arm on ‘HAIMA’ ROV and reached the bottom of the South China Sea (depth of 3,224 m). After the releasing, the soft robot was actuated with an on-board AC voltage of 8 kV at 1 Hz and demonstrated free swimming locomotion with its flapping fins.

Um, did they bring it back?

[ Nature ]

Quadruped Yuki Mini is 12 DOF robot equipped with a Raspberry Pi that runs ROS. Also, BUNNIES!

[ Lingkang Zhang ]

Thanks Lingkang!

Deployment of drone swarms usually relies on inter-agent communication or visual markers that are mounted on the vehicles to simplify their mutual detection. The vswarm package enables decentralized vision-based control of drone swarms without relying on inter-agent communication or visual fiducial markers. The results show that the drones can safely navigate in an outdoor environment despite substantial background clutter and difficult lighting conditions.

[ Vswarm ]

A conventional adopted method for operating a waiter robot is based on the static position control, where pre-defined goal positions are marked on a map. However, this solution is not optimal in a dynamic setting, such as in a coffee shop or an outdoor catering event, because the customers often change their positions. We explore an alternative human-robot interface design where a human operator communicates the identity of the customer to the robot instead. Inspired by how [a] human communicates, we propose a framework for communicating a visual goal to the robot, through interactive two-way communications.

[ Paper ]

Thanks Poramate!

In this video, LOLA reacts to undetected ground height changes, including a drop and leg-in-hole experiment. Further tests show the robustness to vertical disturbances using a seesaw. The robot is technically blind, not using any camera-based or prior information on the terrain.

[ TUM ]

RaiSim is a cross-platform multi-body physics engine for robotics and AI. It fully supports Linux, Mac OS, and Windows.

[ RaiSim ]

Thanks Fan!

The next generation of LoCoBot is here. The LoCoBot is an ROS research rover for mapping, navigation and manipulation (optional) that enables researchers, educators and students alike to focus on high level code development instead of hardware and building out lower level code. Development on the LoCoBot is simplified with open source software, full ROS-mapping and navigation packages and modular opensource Python API that allows users to move the platform as well as (optional) manipulator in as few as 10 lines of code.

[ Trossen ]

MIT Media Lab Research Specialist Dr. Kate Darling looks at how robots are portrayed in popular film and TV shows.

Kate's book, The New Breed: What Our History with Animals Reveals about Our Future with Robots can be pre-ordered now and comes out next month.

[ Kate Darling ]

The current autonomous mobility systems for planetary exploration are wheeled rovers, limited to flat, gently-sloping terrains and agglomerate regolith. These vehicles cannot tolerate instability and operate within a low-risk envelope (i.e., low-incline driving to avoid toppling). Here, we present ‘Mars Dogs’ (MD), four-legged robotic dogs, the next evolution of extreme planetary exploration.

[ Team CoSTAR ]

In 2020, first-year PhD students at the MIT Media Lab were tasked with a special project—to reimagine the Lab and write sci-fi stories about the MIT Media Lab in the year 2050. “But, we are researchers. We don't only write fiction, we also do science! So, we did what scientists do! We used a secret time machine under the MIT dome to go to the year 2050 and see what’s going on there! Luckily, the Media Lab still exists and we met someone…really cool!” Enjoy this interview of Cyber Joe, AI Mentor for MIT Media Lab Students of 2050.

[ MIT ]

In this talk, we will give an overview of the diverse research we do at CSIRO’s Robotics and Autonomous Systems Group and delve into some specific technologies we have developed including SLAM and Legged robotics. We will also give insights into CSIRO’s participation in the current DARPA Subterranean Challenge where we are deploying a fleet of heterogeneous robots into GPS-denied unknown underground environments.

[ GRASP Seminar ]

Marco Hutter (ETH) and Hae-Won Park (KAIST) talk about “Robotics Inspired by Nature.”

[ Swiss-Korean Science Club ]

Thanks Fan!

In this keynote, Guy Hoffman Assistant Professor and the Mills Family Faculty Fellow in the Sibley School of Mechanical and Aerospace Engineering at Cornell University, discusses “The Social Uncanny of Robotic Companions.”

[ Designerly HRI ]

Swabbing tests have proved to be an effective method of diagnosis for a wide range of diseases. Potential occupational health hazards and reliance on healthcare workers during traditional swabbing procedures can be mitigated by self-administered swabs. Hence, we report possible methods to apply closed kinematic chain theory to develop a self-administered viral swab to collect respiratory specimens. The proposed sensorized swab models utilizing hollow polypropylene tubes possess mechanical compliance, simple construction, and inexpensive components. In detail, the adaptation of the slider-crank mechanism combined with concepts of a deployable telescopic tubular mechanical system is explored through four different oral swab designs. A closed kinematic chain on suitable material to create a developable surface allows the translation of simple two-dimensional motion into more complex multi-dimensional motion. These foldable telescopic straws with multiple kirigami cuts minimize components involved in the system as the characteristics are built directly into the material. Further, it offers a possibility to include soft stretchable sensors for realtime performance monitoring. A variety of features were constructed and tested using the concepts above, including 1) tongue depressor and cough/gag reflex deflector; 2) changing the position and orientation of the oral swab when sample collection is in the process; 3) protective cover for the swabbing bud; 4) a combination of the features mentioned above.

The dawn of the robot revolution is already here, and it is not the dystopian nightmare we imagined. Instead, it comes in the form of social robots: Autonomous robots in homes and schools, offices and public spaces, able to interact with humans and other robots in a socially acceptable, human-perceptible way to resolve tasks related to core human needs. 

To design social robots that “understand” humans, robotics scientists are delving into the psychology of human communication. Researchers from Cornell University posit that embedding the sense of touch in social robots could teach them to detect physical interactions and gestures. They describe a way of doing so by relying not on touch but on vision.

A USB camera inside the robot captures shadows of hand gestures on the robot’s surface and classifies them with machine-learning software. They call this method ShadowSense, which they define as a modality between vision and touch, bringing “the high resolution and low cost of vision-sensing to the close-up sensory experience of touch.” 

Touch-sensing in social or interactive robots is usually achieved with force sensors or capacitive sensors, says study co-author Guy Hoffman of the Sibley School of Mechanical and Aerospace Engineering at Cornell University. The drawback to his group’s approach has been that, even to achieve coarse spatial resolution, many sensors are needed in a small area.

However, working with non-rigid, inflatable robots, Hoffman and his co-researchers installed a consumer-grade USB camera to which they attached a fisheye lens for a wider field of vision. 

“Given that the robot is already hollow, and has a soft and translucent skin, we could do touch interaction by looking at the shadows created by people touching the robot,” says Hoffman. They used deep neural networks to interpret the shadows. “And we were able to do it with very high accuracy,” he says. The robot was able to interpret six different gestures, including one- or two-handed touch, pointing, hugging and punching, with an accuracy of 87.5 to 96 percent, depending on the lighting.

This is not the first time that computer vision has been used for tactile sensing, though the scale and application of ShadowSense is unique. “Photography has been used for touch mainly in robotic grasping,” says Hoffman. By contrast, Hoffman and collaborators wanted to develop a sense that could be “felt” across the whole of the device. 

The potential applications for ShadowSense include mobile robot guidance using touch, and interactive screens on soft robots. A third concerns privacy, especially in home-based social robots. “We have another paper currently under review that looks specifically at the ability to detect gestures that are further away [from the robot’s skin],” says Hoffman. This way, users would be able to cover their robot’s camera with a translucent material and still allow it to interpret actions and gestures from shadows. Thus, even though it’s prevented from capturing a high-resolution image of the user or their surrounding environment, using the right kind of training datasets, the robot can continue to monitor some kinds of non-tactile activities. 

In its current iteration, Hoffman says, ShadowSense doesn’t do well in low-light conditions. Environmental noise, or shadows from surrounding objects, also interfere with image classification. Relying on one camera also means a single point of failure. “I think if this were to become a commercial product, we would probably [have to] work a little bit better on image detection,” says Hoffman.

As it was, the researchers used transfer learning—reusing a pre-trained deep-learning model in a new problem—for image analysis. “One of the problems with multi-layered neural networks is that you need a lot of training data to make accurate predictions,” says Hoffman. “Obviously, we don’t have millions of examples of people touching a hollow, inflatable robot. But we can use pre-trained networks trained on general images, which we have billions of, and we only retrain the last layers of the network using our own dataset.”

Demand for battery-making metals is projected to soar as more of the world’s cars, buses, and ships run on electricity. The coming mining boom is raising concerns of environmental damage and labor abuses—and it’s driving a search for more sustainable ways of making batteries and cutting-edge electronics.

Artificial intelligence could help improve the way battery metals are mined, or replace them altogether. KoBold Metals is developing an AI agent to find the most desirable ore deposits in the least problematic locations. IBM Research, meanwhile, is harnessing AI techniques to identify alternative materials that already exist and also develop new chemistries.

KoBold, a mining exploration startup, says its technology could reduce the need for costly and invasive exploration missions, which often involve scouring the Earth many times over to find rare, high-quality reserves. 

“All the stuff poking out of the ground has already been found,” said Kurt House, co-founder and CEO of the San Francisco Bay area company. “At the same time, we’ve realized we need to massively change the energy system, which requires all these new minerals.”

KoBold is partnering with Stanford University’s Center for Earth Resource Forecasting to develop an AI agent that can make decisions about how and where explorers should focus their work. The startup is mainly looking for copper, cobalt, nickel, and lithium—metals key to making electric vehicle batteries as well as solar panels, smartphones, and many other devices.

Jef Caers, a professor of geological sciences at Stanford, said the idea is to accelerate the decision-making process and enable explorers to evaluate multiple sites at once. He likened the AI agent to a self-driving car: The vehicle not only gathers and processes data about its surrounding environment but also acts upon that information to, say, navigate traffic or change speeds.

“We can’t wait another 10 or 20 years to make more discoveries,” Caers said. “We need to make them in the next few years if we want to have an impact on [climate change] and go away from fossil fuels.”

Light-duty cars alone will have a significant need for metals. The global fleet of battery-powered cars could expand from 7.5 million in 2019 to potentially 2 billion cars by 2050 as countries work to reduce greenhouse gas emissions, according to a December paper in the journal Nature. Powering those vehicles would require 12 terawatt-hours of annual battery capacity—roughly 10 times the current U.S. electricity generating capacity—and mean a “drastic expansion” of metal supply chains, the paper’s authors said. 

Photo: KoBold Metals Patrick Redmond of KoBold Metals evaluates a prospective cobalt-copper mining site in Zambia.

Almost all lithium-ion batteries use cobalt, a material that is primarily supplied by the Democratic Republic of Congo, where young children and adults often work in dangerous conditions. Copper, another important EV material, requires huge volumes of water to mine, yet much of the global supply comes from water-scarce regions near Chile’s Atacama desert. 

For mining companies, the challenge is to expand operations without wreaking havoc in the name of sustainable transportation. 

KoBold’s AI-driven approach begins with its data platform, which stores all available forms of information about a particular area, including soil samples, satellite-based hyperspectral imaging, and century-old handwritten drilling reports. The company then applies machine learning methods to make predictions about the location of compositional anomalies—that is, unusually high concentrations of ore bodies in the Earth’s subsurface.

Working with Stanford, KoBold is refining sequential decision-making algorithms to determine how explorers should next proceed to gather data. Perhaps they should fly a plane over a site or collect drilling samples; maybe companies should walk away from what is likely to be a dud. Such steps are currently risky and expensive, and companies move slowly to avoid wasting resources. 

The AI agent could make such decisions roughly 20 times faster than humans can, while also reducing the rate of false positives in mining exploration, Caers said. “This is completely new within the Earth sciences,” he added.

Image: KoBold Metals An AI visualization by KoBold Metals depicts a plot of predictions from the borehole electromagnetic model, with true values on the left and predictions on the right.

KoBold, which is backed by the Bill Gates-led Breakthrough Energy Ventures, is already exploring three sites in Australia, North America, and Sub-Saharan Africa. Field data collected this year will provide the first validations of the company’s predictions, House said.

As the startup searches for metals, IBM researchers are searching for solvents and other materials to reduce the use of battery ingredients such as cobalt and lithium.

Research teams are using AI techniques to identify and test solvents that offer higher safety and performance potential than current lithium-ion battery options. The project focuses on existing and commercially available materials that can be tested immediately. A related research effort, however, aims to create brand-new molecules entirely.

Using “generative models,” experts train AI to learn the molecular structure of known materials, as well as characteristics such as viscosity, melting point, or electronic conductivity.  

“For example, if we want a generative model to design new electrolyte materials for batteries— such as electrolyte solvents or appropriate monomers to form ion-conducting polymers—we should train the AI with known electrolyte material data,” Seiji Takeda and Young-hye Na of IBM Research said in an email. 

Once the AI training is completed, researchers can input a query such as “design a new molecular electrolyte material that meets the characteristics of X, Y, and Z,” they said. “And then the model designs a material candidate by referring to the structure-characteristics relation.”

IBM has already used this AI-boosted approach to create new molecules named called photoacid generators that could eventually help produce more environmentally friendly computing devices. Researchers also designed polymer membranes that apparently absorbs carbon dioxide better than membranes currently used in carbon capture technologies.

Designing a more sustainable battery “is our next challenge,” Takeda and Na said.

The development of biodegradable soft robotics requires an appropriate eco-friendly source of energy. The use of Microbial Fuel Cells (MFCs) is suggested as they can be designed completely from soft materials with little or no negative effects to the environment. Nonetheless, their responsiveness and functionality is not strictly defined as in other conventional technologies, i.e. lithium batteries. Consequently, the use of artificial intelligence methods in their control techniques is highly recommended. The use of neural networks, namely a nonlinear autoregressive network with exogenous inputs was employed to predict the electrical output of an MFC, given its previous outputs and feeding volumes. Thus, predicting MFC outputs as a time series, enables accurate determination of feeding intervals and quantities required for sustenance that can be incorporated in the behavioural repertoire of a soft robot.

In this paper, we present a generalized modeling tool for predicting the output force profile of vacuum-powered soft actuators using a simplified geometrical approach and the principle of virtual work. Previous work has derived analytical formulas to model the force-contraction profile of specific actuators. To enhance the versatility and the efficiency of the modelling process we propose a generalized numerical algorithm based purely on geometrical inputs, which can be tailored to the desired actuator, to estimate its force-contraction profile quickly and for any combination of varying geometrical parameters. We identify a class of linearly contracting vacuum actuators that consists of a polymeric skin guided by a rigid skeleton and apply our model to two such actuators-vacuum bellows and Fluid-driven Origami-inspired Artificial Muscles-to demonstrate the versatility of our model. We perform experiments to validate that our model can predict the force profile of the actuators using its geometric principles, modularly combined with design-specific external adjustment factors. Our framework can be used as a versatile design tool that allows users to perform parametric studies and rapidly and efficiently tune actuator dimensions to produce a force-contraction profile to meet their needs, and as a pre-screening tool to obviate the need for multiple rounds of time-intensive actuator fabrication and testing.

In my experience, there are three different types of consumer drone pilots. You’ve got people for whom drones are a tool for taking pictures and video, where flying the drone is more or less just a necessary component of that. You’ve also got people who want a drone that can be used to take pictures or video of themselves, where they don’t want to be bothered flying the drone at all. Then you have people for whom flying the drone itself is the appealing part; people who like flying fast and creatively because it’s challenging, exciting, and fun. And that typically means flying in First Person View, or FPV, where it feels like you’re a tiny little human sitting inside of a virtual cockpit in your drone.

For that last group of folks, the barrier to entry is high. Or rather, the barriers are high, because there are several. Not only is the equipment expensive, you often have to build your own system comprising the drone, FPV goggles, and accompanying transmitter and receiver. And on top of that, it takes a lot of skill to fly an FPV drone well; all of the inevitable crashes just add to the expense.

Today, DJI is announcing a new consumer first-person view drone system that includes everything you need to get started. You get an expertly designed and fully integrated high-speed FPV drone, a pair of FPV goggles with exceptional image quality and latency that’s some of the best we’ve ever seen, plus a physical controller to make it all work. Most importantly, though, there’s on-board obstacle avoidance plus piloting assistance that means even a complete novice can be zipping around with safety and confidence on day one.

Because the point of an FPV drone is to let you fly from a first-person viewpoint. The drone has a forward-facing camera that streams video to a pair of goggles in real time. This experience is a unique one, and there’s only so much that I can do to describe it, but it turns flying a drone into a much more personal, visceral, immersive thing. With an FPV drone, it feels much more like you are the drone. 

The Drone Photos: DJI DJI’s FPV drone is basically a battery with a camera and some motors attached.

DJI’s FPV drone itself is a bit of a chonker, as far as drones go. It’s optimized for going very fast while giving you a good first-person view, and no concessions are given to portability. It weighs 800g (of which 300 is the battery), and doesn’t fold up even a little bit, although the props are easy to remove.

Photo: Evan Ackerman/IEEE Spectrum Efficient design, but not small or portable.

Top speed is a terrifying 140 km/h, albeit in a mode  that you have to unlock (more on that later), and it’ll accelerate from a hover to 100 km/h in two seconds. Battery life maxes out at 20 minutes, but in practice you’ll get more like 10-15 minutes depending on how you fly. The camera on the front records in 4K at 60 FPS on an electronically stabilized tilt-only gimbal, and there’s a microSD card slot for local recording.

We’re delighted to report that the DJI FPV drone also includes some useful sensors that will make you significantly less likely to embed it in the nearest tree. These sensors include ground detection to keep the drone at a safe altitude, as well as forward-looking stereo-based obstacle detection that works well enough for the kinds of obstacles that cameras are able to see. 

The Goggles Photos: DJI You’ll look weird with these on, but they work very, very well.

What really makes this drone work are the FPV goggles along with the radio system that connects to the drone. The goggles have two tiny screens in them, right in front of your eyeballs. Each screen can display 1440 x 810p at up to 120 fps (which looks glorious), and it’ll do so while the drone is moving at 140 km/h hundreds of meters away. It’s extremely impressive, with quality that’s easily good enough to let you spot (and avoid) skinny little tree branches. But even more important than quality is latency— the amount of time it takes for the video to be captured by the drone, compressed, sent to the goggles, decompressed, and displayed. The longer this takes, the less you’re able to trust what you’re seeing, because you know that you’re looking at where the drone used to be rather than where it actually is. DJI’s FPV system has a latency that’s 28ms or better, which is near enough to real-time that it feels just like real-time, and you can fly with absolute confidence that the control inputs you’re giving and what you’re seeing through the goggles are matched up. 

Photo: Evan Ackerman/IEEE Spectrum The goggles are adjustable to fit most head sizes.

The goggles are also how you control all of the drone options, no phone necessary. You can attach your phone to the goggles with a USB-C cable for firmware updates, but otherwise, a little joystick and some buttons on the top of the goggles lead you to an intuitive interface for things like camera options, drone control options, and so on. A microSD card slot on the goggles lets you record the downlinked video, although it’s not going to be the same quality as what’s recorded on-board the drone. 

Don’t get your hopes up on comfort where the goggles are concerned. They’re fine, but that’s it. My favorite thing about them is that they don’t weigh much, because the battery that powers them has a cable so that you can keep it in your pocket rather than hanging it off the goggles themselves. Adjustable straps mean the goggles will fit most people, and there’s inter-eye distance adjustments for differently shaped faces. My partner, who statistically is smaller than 90% of adult women, found that the google inter-eye distance adjustment was almost, but not quite, adequate for her. Fortunately, the goggles not being super comfortable isn’t a huge deal because you won’t be wearing them for that long, unless you invest in more (quite expensive) battery packs.

The Controller Photos: DJI All the controls you want, none that you don’t. Except one.

The last piece of the kit is the controller, which is fairly standard as far as drone controllers go. The sticks unscrew and stow in the handles, and you can also open up the back panels to adjust the stick tension, which experienced FPV pilots will probably want to do.

Before we get to how the drone flies, a quick word on safe operation— according to the FAA, if you’re flying an FPV drone, you’ll need a spotter who keeps the drone in view at all times. And while the range of the DJI FPV drone is (DJI claims) up to 10km, here in the United States you’re not allowed to fly it higher than 400ft AGL, or farther away than your spotter can see, without an FAA exemption. Also, as with all drones, you’ll need to find places that are both safe and legal to fly. The drone will prevent you from taking off in restricted areas that DJI knows about, but it’s on you to keep the drone from flying over people, or otherwise being dangerous and/or annoying.

In Flight— Normal Mode Photos: Evan Ackerman/IEEE Spectrum It's not the most graceful looking drone, but the way it flies makes you not care.

DJI has helpfully equipped the FPV drone with three flight modes: Normal, Sport, and Manual. These three modes are the primary reason why this drone is not a terrible idea for almost everyone— Normal mode is a lot of fun, and both safe and accessible for FPV novices. Specifically, Normal mode brings the top speed of the drone down to a still rather quick 50 km/h, and will significantly slow the drone if the front sensors think you’re likely to hit something. As with DJI’s other drones, if you start getting into trouble you can simply let go of the control sticks, and the drone will bring itself to a halt and hover. This makes it very beginner friendly, and (as an FPV beginner myself) I didn’t find it at all stressful to fly. With most drones (and especially drones that cost as much as this one does) fear of crashing is a tangible thing always sitting at the back of your mind. That feeling is not gone with the DJI FPV drone, but it’s reduced so much that the experience can really just be fun.

To be clear, not crashing into stuff is not enough to make FPV flying an enjoyable experience. Arguably the best feature of DJI’s FPV drone is how much help it gives you behind the scenes to make flying effortless.

When flying a conventional drone, you’ve got four axes of control to work with, allowing the drone to pitch, roll, yaw, move vertically up and down, or do any combination of those things. Generally, a drone won’t couple these axes for you in an intelligent way. For example, if you want to go left with a drone, you can either roll left, which will move the drone left without looking left, or yaw left, which will cause the drone to look left without actually moving. To gracefully fly a drone around obstacles at speed, you need to fuse both of these inputs together, which is a skill that can take a long time to master. I have certainly not mastered it.

The drone does exactly what you think it should do, and it works beautifully.

For most people, especially beginners, it’s much more intuitive for the drone to behave more like an airplane when you want it to turn. That is, when you push the left stick (traditionally the yaw control), you want the drone to begin to roll while also yawing in the same direction and increasing throttle to execute a lovely, sweeping turn. And this is exactly what DJI FPV does—  thanks to a software option called coordinated turns that’s on by default, the drone does exactly what you think it should do, and it works beautifully. 

I could tell you how well this works for me, someone who has flown non-FPV drones for years, but my partner has flown a drone only one single time before, when she spent five minutes with the controller of my Parrot Anafi a few years ago and got it to go up and down and occasionally sideways a little bit. But within literally two minutes, she was doing graceful figure 8s with the DJI FPV drone. The combination of the FPV view and the built-in coordinated turns makes it just that intuitive.

In Flight— Sport Mode

Once you’re comfortable with Normal mode, Sport mode (which you can select at any time with a toggle switch on the controller) bumps up the speed of the drone from 50 km/h to 97 km/h. More importantly, the obstacle avoidance no longer slows the drone down for you, although it does give you escalating warnings when it thinks you’re going to hit something. As you get more comfortable with the drone, you’ll find that the obstacle avoidance tends to be on the paranoid side, which is as it should be. Once you’ve practiced a bit and you want to (say) fly between two trees that are close together, Sport mode will let you do that without slowing down.

Along with a higher top speed, Sport mode will also make the drone literally scream. When in flight it makes a loud screaming noise, especially when you ask a lot from the motors, like with rapid direction changes or gaining altitude at speed. This happens in Normal mode, but gets markedly louder in Sport mode. This doesn’t matter, really, except that if you’re flying anywhere near other people, they’re likely to find it obnoxious. 

I was surprised by how not-puking I was during high-speed FPV flight in Sport mode. I tend to suffer from motion sickness, but I had no trouble with the drone, as long as I kept my head still. Even a small head movement while the drone was in flight could lead to an immediate (although minor) wave of nausea, which passed as soon as I stopped moving. My head sometimes subconsciously moved along with the motion of the drone, to the point where after a few minutes of flying I’d realize that I’d ended up staring sideways at the sky like an idiot, so if you can just manage to keep your head mostly still and relaxed in a comfortable position, you’ll be fine. 

A word on speed— even though Sport mode has a max speed of only (only?) 97 km/h, coming from flying a Mavic Pro, it feels very, very fast. In tight turns, the video coming through the goggles sometimes looked like it was playing back at double speed. You could always ask for more speed, I suppose, and Manual mode gives it to you. But I could see myself being perfectly happy to fly in Sport mode for a long, long time, since it offers both speed and some additional safety.

The following video includes clips of Normal and Sport mode, to give you an idea of how smooth the drone moves and how fast it can go, along with a comparison between the standard 1080p recorded by the drone and what gets shown in the goggles. As you watch the video, remember that I’m not an FPV pilot. I’ve never flown an FPV drone before, and what you’re seeing is the result of less than an hour of total flight time.

In Flight— Manual Mode

There is one final mode that the DJI FPV drone comes with: Manual mode. Manual mode is for pilots who don’t need or want any hand-holding. You get full control over all axes as well as an unrestricted top speed of 140 km/h. Manual mode must be deliberately enabled in menus in the goggles (it’s not an available option by default), and DJI suggests spending some time in their included simulator before doing so. I want to stress that Manual mode doesn’t just disable the coordinated turns function, making control of the drone more like a traditional camera drone— if that’s something you want, there’s an option to do that in Normal and Sport mode. Manual mode is designed for people with drone racing experience, and enabling it turns the FPV drone into a much different thing, as I found out.

My test of Manual mode ended approximately 15 seconds after it began due to high speed contact with the ground. I wouldn’t call what happened a “crash,” in the sense that I didn’t fly the drone into an obstacle— there was a misunderstanding or a lack of information or an accidental input or some combination of those things that led to the drone shutting all of its motors of at about 150 feet up and then falling to the ground. 

Photo: Evan Ackerman/IEEE Spectrum The crash broke the drone’s two rear arms, but the expensive parts all seem perfectly fine.

I’d planned to just fly the drone in Manual mode a little bit, with plenty of altitude and over a big open space, primarily to get a sense of how much faster it is in Manual mode than in normal mode. After taking off in Normal mode and giving the drone a lot of room, I switched over to Manual mode. Immediately, the drone began to move in a much less predictable way, and after about five seconds of some tentative control inputs to see if I could get a handle on it, I felt uncomfortable enough to want the drone to stop itself.

The first thing I did was stop giving any control inputs. In Normal and Sport mode, the drone will respond to no input (centered sticks) by bringing itself to a hover. This doesn’t happen in Manual mode, and the drone kept moving. The second thing I did was push what I thought was the emergency stop button, which would switch the drone back to Normal mode and engage a thoughtfully included emergency stop mode to bring it to a stable hover as quickly as possible. I hadn’t yet needed an emergency stop in Normal or Sport mode, since just taking my fingers off of the sticks worked just fine before. What I learned post-crash was that in Manual mode, the button that says “Stop” on it (which is one of the easiest to press buttons on the controller since your right index finger naturally rests there) gains a new emergency shut-off functionality that causes the drone to disable all of its motors, whereupon it will then follow a ballistic trajectory until the inevitable happens no matter how much additional button pushing or frantic pleading you do. 

I certainly take some responsibility for this. When the DJI FPV drone showed up, it included a quick start guide and a more detailed reviewer’s guide, but neither of those documents had detailed information about what all the buttons on the controller and headset did. This sometimes happens with review units— they can be missing manuals and stuff if they get sent to us before the consumer packaging is complete. Anyway, having previous experience with DJI’s drones, I just assumed that which buttons did what would just be obvious, which was 100% my bad, and it’s what led to the crash.

Also, I recognize why it’s important to have an emergency shut-off, and as far as I know, most (if not all) of DJI’s other consumer drones include some way of remotely disabling the motors. It should only be possible to do this deliberately, though, which is why their other drones require you to use a combination of inputs that you’re very unlikely to do by accident. Having what is basically a self-destruct button on the controller where you naturally rest a finger just seems like a bad idea— I pushed it on purpose thinking it did something different, but there are all kinds of reasons why a pilot might push it accidentally. And if you do, that’s it, your drone is going down. 

DJI, to their credit, was very understanding about the whole thing, but more importantly, they pointed out that accidental damage like this would be covered under DJI Care Refresh, which will completely replace the drone if necessary. This should give new pilots some piece of mind, if you’re willing to pay for the premium. Even if you’re not, the drone is designed to be at least somewhat end-user repairable.

Fundamentally, I’m glad Manual mode is there. DJI made the right choice by including it so that your skill won’t outgrow the capabilities of the drone. I just wish that the transition to Manual mode was more gradual, like if there was a Sport Plus mode that unlocked top speed while maintaining other flight assistance features. Even without that, FPV beginners really shouldn’t feel like Manual mode needs to be a goal— Normal mode is fun, and Sport mode is even more fun, with the added piece of mind that you’ve got options if things start to get out of your control. And if things do get out of your control in Manual mode, for heaven’s sake, push the right button.

I mean, the left button.

Is This The Right Drone for You? Photo: Evan Ackerman/IEEE Spectrum My partner, who is totally not interested in drones, actually had fun flying this one.

DJI’s FPV drone kit costs $1,299, which includes the drone, goggles, one battery, and all necessary chargers and cabling. Two more batteries and a charging hub, which you’ll almost certainly want, adds $299. This is a lot of money, even for a drone, so the thing to ask yourself is whether an FPV drone is really what you’re looking for. Yes, it’ll take good quality pictures and video, but if that’s what you’re after, DJI has lots of other drones that are cheaper and more portable and have some smarts to make them better camera platforms. And of course there’s the Skydio 2, which has some crazy obstacle avoidance and autonomy if you don’t want to have to worry about flying at all. I have a fantasy that one day, all of this will be combined into one single drone, but we’re not there yet. 

If you’re sure you want to get into FPV flying, DJI’s kit seems like a great option, with the recognition that this is an expensive, equipment-intensive sport. There are definitely ways of doing it for cheaper, but you’ll need to more or less build up the system yourself, and it seems unlikely that you’d end up with the same kind of reliable performance and software features that DJI’s system comes with. The big advantage of DJI’s FPV kit is that you can immediately get started with a system that works brilliantly out of the box, in a way that’s highly integrated, highly functional, high performing while being fantastic for beginners and leaving plenty of room to grow.

In my experience, there are three different types of consumer drone pilots. You’ve got people for whom drones are a tool for taking pictures and video, where flying the drone is more or less just a necessary component of that. You’ve also got people who want a drone that can be used to take pictures or video of themselves, where they don’t want to be bothered flying the drone at all. Then you have people for whom flying the drone itself is the appealing part— people who like flying fast and creatively because it’s challenging and exciting and fun. And that typically means flying in First Person View, or FPV, where it feels like you’re a tiny little human sitting inside of a virtual cockpit in your drone.

For that last group of folks, the barrier to entry is high. Or rather, the barriers are high, because there are several. Not only is the equipment expensive, you often have to build your own system of drone, FPV goggles, and transmitter and receiver. And on top of that, it takes a lot of skill to fly an FPV drone well, and all of the inevitable crashes just add to the expense.

Today, DJI is announcing a new consumer first-person view drone system that includes everything you need to get started. You get an expertly designed and fully integrated high-speed FPV drone, a pair of FPV goggles with exceptional image quality and latency that’s some of the best we’ve ever seen, plus a physical controller to make it all work. Most importantly, though, there’s on-board obstacle avoidance plus piloting assistance that means even a complete novice can be zipping around with safety and confidence on day one.

An FPV drone is a drone that you fly from a first-person viewpoint. The drone has a forward-facing camera that streams video to a pair of goggles in real time. This experience is a unique one, and there’s only so much that I can do to describe it, but it turns flying a drone into a much more personal, visceral, immersive thing. With an FPV drone, it feels much more like you are the drone. 

The Drone Photos: DJI DJI’s FPV drone is basically a battery with a camera and some motors attached.

DJI’s FPV drone itself is a bit of a chonker, as far as drones go. It’s optimized for going very fast while giving you a good first-person view, and no concessions are given to portability. It weighs 800g (of which 300 is the battery), and doesn’t fold up even a little bit, although the props are easy to remove.

Photo: Evan Ackerman/IEEE Spectrum Efficient design, but not small or portable.

Top speed is a terrifying 140 km/h, albeit in a mode  that you have to unlock (more on that later), and it’ll accelerate from a hover to 100 km/h in two seconds. Battery life maxes out at 20 minutes, but in practice you’ll get more like 10-15 minutes depending on how you fly. The camera on the front records in 4K at 60 FPS on an electronically stabilized tilt-only gimbal, and there’s a microSD card slot for local recording.

We’re delighted to report that the DJI FPV drone also includes some useful sensors that will make you significantly less likely to embed it in the nearest tree. These sensors include ground detection to keep the drone at a safe altitude, as well as forward-looking stereo-based obstacle detection that works well enough for the kinds of obstacles that cameras are able to see. 

The Goggles Photos: DJI You’ll look weird with these on, but they work very, very well.

What really makes this drone work are the FPV goggles along with the radio system that connects to the drone. The goggles have two tiny screens in them, right in front of your eyeballs. Each screen can display 1440 x 810p at up to 120 fps (which looks glorious), and it’ll do so while the drone is moving at 140 km/h hundreds of meters away. It’s extremely impressive, with quality that’s easily good enough to let you spot (and avoid) skinny little tree branches. But even more important than quality is latency— the amount of time it takes for the video to be captured by the drone, compressed, sent to the goggles, decompressed, and displayed. The longer this takes, the less you’re able to trust what you’re seeing, because you know that you’re looking at where the drone used to be rather than where it actually is. DJI’s FPV system has a latency that’s 28ms or better, which is near enough to real-time that it feels just like real-time, and you can fly with absolute confidence that the control inputs you’re giving and what you’re seeing through the goggles are matched up. 

Photo: Evan Ackerman/IEEE Spectrum The goggles are adjustable to fit most head sizes.

The goggles are also how you control all of the drone options, no phone necessary. You can attach your phone to the goggles with a USB-C cable for firmware updates, but otherwise, a little joystick and some buttons on the top of the goggles lead you to an intuitive interface for things like camera options, drone control options, and so on. A microSD card slot on the goggles lets you record the downlinked video, although it’s not going to be the same quality as what’s recorded on-board the drone. 

Don’t get your hopes up on comfort where the goggles are concerned. They’re fine, but that’s it. My favorite thing about them is that they don’t weigh much, because the battery that powers them has a cable so that you can keep it in your pocket rather than hanging it off the goggles themselves. Adjustable straps mean the goggles will fit most people, and there’s inter-eye distance adjustments for differently shaped faces. My partner, who statistically is smaller than 90% of adult women, found that the google inter-eye distance adjustment was almost, but not quite, adequate for her. Fortunately, the goggles not being super comfortable isn’t a huge deal because you won’t be wearing them for that long, unless you invest in more (quite expensive) battery packs.

The Controller Photos: DJI All the controls you want, none that you don’t. Except one.

The last piece of the kit is the controller, which is fairly standard as far as drone controllers go. The sticks unscrew and stow in the handles, and you can also open up the back panels to adjust the stick tension, which experienced FPV pilots will probably want to do.

Before we get to how the drone flies, a quick word on safe operation— according to the FAA, if you’re flying an FPV drone, you’ll need a spotter who keeps the drone in view at all times. And while the range of the DJI FPV drone is (DJI claims) up to 10km, here in the United States you’re not allowed to fly it higher than 400ft AGL, or farther away than your spotter can see, without an FAA exemption. Also, as with all drones, you’ll need to find places that are both safe and legal to fly. The drone will prevent you from taking off in restricted areas that DJI knows about, but it’s on you to keep the drone from flying over people, or otherwise being dangerous and/or annoying.

In Flight— Normal Mode Photos: Evan Ackerman/IEEE Spectrum It's not the most graceful looking drone, but the way it flies makes you not care.

DJI has helpfully equipped the FPV drone with three flight modes: Normal, Sport, and Manual. These three modes are the primary reason why this drone is not a terrible idea for almost everyone— Normal mode is a lot of fun, and both safe and accessible for FPV novices. Specifically, Normal mode brings the top speed of the drone down to a still rather quick 50 km/h, and will significantly slow the drone if the front sensors think you’re likely to hit something. As with DJI’s other drones, if you start getting into trouble you can simply let go of the control sticks, and the drone will bring itself to a halt and hover. This makes it very beginner friendly, and (as an FPV beginner myself) I didn’t find it at all stressful to fly. With most drones (and especially drones that cost as much as this one does) fear of crashing is a tangible thing always sitting at the back of your mind. That feeling is not gone with the DJI FPV drone, but it’s reduced so much that the experience can really just be fun.

To be clear, not crashing into stuff is not enough to make FPV flying an enjoyable experience. Arguably the best feature of DJI’s FPV drone is how much help it gives you behind the scenes to make flying effortless.

When flying a conventional drone, you’ve got four axes of control to work with, allowing the drone to pitch, roll, yaw, move vertically up and down, or do any combination of those things. Generally, a drone won’t couple these axes for you in an intelligent way. For example, if you want to go left with a drone, you can either roll left, which will move the drone left without looking left, or yaw left, which will cause the drone to look left without actually moving. To gracefully fly a drone around obstacles at speed, you need to fuse both of these inputs together, which is a skill that can take a long time to master. I have certainly not mastered it.

The drone does exactly what you think it should do, and it works beautifully.

For most people, especially beginners, it’s much more intuitive for the drone to behave more like an airplane when you want it to turn. That is, when you push the left stick (traditionally the yaw control), you want the drone to begin to roll while also yawing in the same direction and increasing throttle to execute a lovely, sweeping turn. And this is exactly what DJI FPV does—  thanks to a software option called coordinated turns that’s on by default, the drone does exactly what you think it should do, and it works beautifully. 

I could tell you how well this works for me, someone who has flown non-FPV drones for years, but my partner has flown a drone only one single time before, when she spent five minutes with the controller of my Parrot Anafi a few years ago and got it to go up and down and occasionally sideways a little bit. But within literally two minutes, she was doing graceful figure 8s with the DJI FPV drone. The combination of the FPV view and the built-in coordinated turns makes it just that intuitive.

In Flight— Sport Mode

Once you’re comfortable with Normal mode, Sport mode (which you can select at any time with a toggle switch on the controller) bumps up the speed of the drone from 50 km/h to 97 km/h. More importantly, the obstacle avoidance no longer slows the drone down for you, although it does give you escalating warnings when it thinks you’re going to hit something. As you get more comfortable with the drone, you’ll find that the obstacle avoidance tends to be on the paranoid side, which is as it should be. Once you’ve practiced a bit and you want to (say) fly between two trees that are close together, Sport mode will let you do that without slowing down.

Along with a higher top speed, Sport mode will also make the drone literally scream. When in flight it makes a loud screaming noise, especially when you ask a lot from the motors, like with rapid direction changes or gaining altitude at speed. This happens in Normal mode, but gets markedly louder in Sport mode. This doesn’t matter, really, except that if you’re flying anywhere near other people, they’re likely to find it obnoxious. 

I was surprised by how not-puking I was during high-speed FPV flight in Sport mode. I tend to suffer from motion sickness, but I had no trouble with the drone, as long as I kept my head still. Even a small head movement while the drone was in flight could lead to an immediate (although minor) wave of nausea, which passed as soon as I stopped moving. My head sometimes subconsciously moved along with the motion of the drone, to the point where after a few minutes of flying I’d realize that I’d ended up staring sideways at the sky like an idiot, so if you can just manage to keep your head mostly still and relaxed in a comfortable position, you’ll be fine. 

A word on speed— even though Sport mode has a max speed of only (only?) 97 km/h, coming from flying a Mavic Pro, it feels very, very fast. In tight turns, the video coming through the goggles sometimes looked like it was playing back at double speed. You could always ask for more speed, I suppose, and Manual mode gives it to you. But I could see myself being perfectly happy to fly in Sport mode for a long, long time, since it offers both speed and some additional safety.

The following video includes clips of Normal and Sport mode, to give you an idea of how smooth the drone moves and how fast it can go, along with a comparison between the standard 1080p recorded by the drone and what gets shown in the goggles. As you watch the video, remember that I’m not an FPV pilot. I’ve never flown an FPV drone before, and what you’re seeing is the result of less than an hour of total flight time.

In Flight— Manual Mode

There is one final mode that the DJI FPV drone comes with: Manual mode. Manual mode is for pilots who don’t need or want any hand-holding. You get full control over all axes as well as an unrestricted top speed of 140 km/h. Manual mode must be deliberately enabled in menus in the goggles (it’s not an available option by default), and DJI suggests spending some time in their included simulator before doing so. I want to stress that Manual mode doesn’t just disable the coordinated turns function, making control of the drone more like a traditional camera drone— if that’s something you want, there’s an option to do that in Normal and Sport mode. Manual mode is designed for people with drone racing experience, and enabling it turns the FPV drone into a much different thing, as I found out.

My test of Manual mode ended approximately 15 seconds after it began due to high speed contact with the ground. I wouldn’t call what happened a “crash,” in the sense that I didn’t fly the drone into an obstacle— there was a misunderstanding or a lack of information or an accidental input or some combination of those things that led to the drone shutting all of its motors of at about 150 feet up and then falling to the ground. 

Photo: Evan Ackerman/IEEE Spectrum The crash broke the drone’s two rear arms, but the expensive parts all seem perfectly fine.

I’d planned to just fly the drone in Manual mode a little bit, with plenty of altitude and over a big open space, primarily to get a sense of how much faster it is in Manual mode than in normal mode. After taking off in Normal mode and giving the drone a lot of room, I switched over to Manual mode. Immediately, the drone began to move in a much less predictable way, and after about five seconds of some tentative control inputs to see if I could get a handle on it, I felt uncomfortable enough to want the drone to stop itself.

The first thing I did was stop giving any control inputs. In Normal and Sport mode, the drone will respond to no input (centered sticks) by bringing itself to a hover. This doesn’t happen in Manual mode, and the drone kept moving. The second thing I did was push what I thought was the emergency stop button, which would switch the drone back to Normal mode and engage a thoughtfully included emergency stop mode to bring it to a stable hover as quickly as possible. I hadn’t yet needed an emergency stop in Normal or Sport mode, since just taking my fingers off of the sticks worked just fine before. What I learned post-crash was that in Manual mode, the button that says “Stop” on it (which is one of the easiest to press buttons on the controller since your right index finger naturally rests there) gains a new emergency shut-off functionality that causes the drone to disable all of its motors, whereupon it will then follow a ballistic trajectory until the inevitable happens no matter how much additional button pushing or frantic pleading you do. 

I certainly take some responsibility for this. When the DJI FPV drone showed up, it included a quick start guide and a more detailed reviewer’s guide, but neither of those documents had detailed information about what all the buttons on the controller and headset did. This sometimes happens with review units— they can be missing manuals and stuff if they get sent to us before the consumer packaging is complete. Anyway, having previous experience with DJI’s drones, I just assumed that which buttons did what would just be obvious, which was 100% my bad, and it’s what led to the crash.

Also, I recognize why it’s important to have an emergency shut-off, and as far as I know, most (if not all) of DJI’s other consumer drones include some way of remotely disabling the motors. It should only be possible to do this deliberately, though, which is why their other drones require you to use a combination of inputs that you’re very unlikely to do by accident. Having what is basically a self-destruct button on the controller where you naturally rest a finger just seems like a bad idea— I pushed it on purpose thinking it did something different, but there are all kinds of reasons why a pilot might push it accidentally. And if you do, that’s it, your drone is going down. 

DJI, to their credit, was very understanding about the whole thing, but more importantly, they pointed out that accidental damage like this would be covered under DJI Care Refresh, which will completely replace the drone if necessary. This should give new pilots some piece of mind, if you’re willing to pay for the premium. Even if you’re not, the drone is designed to be at least somewhat end-user repairable.

Fundamentally, I’m glad Manual mode is there. DJI made the right choice by including it so that your skill won’t outgrow the capabilities of the drone. I just wish that the transition to Manual mode was more gradual, like if there was a Sport Plus mode that unlocked top speed while maintaining other flight assistance features. Even without that, FPV beginners really shouldn’t feel like Manual mode needs to be a goal— Normal mode is fun, and Sport mode is even more fun, with the added piece of mind that you’ve got options if things start to get out of your control. And if things do get out of your control in Manual mode, for heaven’s sake, push the right button.

I mean, the left button.

Is This The Right Drone for You? Photo: Evan Ackerman/IEEE Spectrum My partner, who is totally not interested in drones, actually had fun flying this one.

DJI’s FPV drone kit costs $1,299, which includes the drone, goggles, one battery, and all necessary chargers and cabling. Two more batteries and a charging hub, which you’ll almost certainly want, adds $299. This is a lot of money, even for a drone, so the thing to ask yourself is whether an FPV drone is really what you’re looking for. Yes, it’ll take good quality pictures and video, but if that’s what you’re after, DJI has lots of other drones that are cheaper and more portable and have some smarts to make them better camera platforms. And of course there’s the Skydio 2, which has some crazy obstacle avoidance and autonomy if you don’t want to have to worry about flying at all. I have a fantasy that one day, all of this will be combined into one single drone, but we’re not there yet. 

If you’re sure you want to get into FPV flying, DJI’s kit seems like a great option, with the recognition that this is an expensive, equipment-intensive sport. There are definitely ways of doing it for cheaper, but you’ll need to more or less build up the system yourself, and it seems unlikely that you’d end up with the same kind of reliable performance and software features that DJI’s system comes with. The big advantage of DJI’s FPV kit is that you can immediately get started with a system that works brilliantly out of the box, in a way that’s highly integrated, highly functional, high performing while being fantastic for beginners and leaving plenty of room to grow.

Haru is a social, affective robot designed to support a wide range of research into human–robot communication. This article analyses the design process for Haru beta, identifying how both visual and performing arts were an essential part of that process, contributing to ideas of Haru’s communication as a science and as an art. Initially, the article examines how a modified form of Design Thinking shaped the work of the interdisciplinary development team—including animators, performers and sketch artists working alongside roboticists—to frame Haru’s interaction style in line with sociopsychological and cybernetic–semiotic communication theory. From these perspectives on communication, the focus is on creating a robot that is persuasive and able to transmit precise information clearly. The article moves on to highlight two alternative perspectives on communication, based on phenomenological and sociocultural theories, from which such a robot can be further developed as a more flexible and dynamic communicative agent. The various theoretical perspectives introduced are brought together by considering communication across three elements: encounter, story and dance. Finally, the article explores the potential of Haru as a research platform for human–robot communication across various scenarios designed to investigate how to support long-term interactions between humans and robots in different contexts. In particular, it gives an overview of plans for humanities-based, qualitative research with Haru.

Pages