IBM, Microsoft, and Nvidia Pit AI Against Wildfires

In 2018, California experienced its worst wildfire season on record, with uncontrolled fires causing unprecedented death and devastation. 2019 saw another difficult season, with millions of residents experiencing precautionary power shutoffs. With much of the world self-quarantining and the 2020 fire season looming, Nvidia’s virtual GTC 2020 hosted a panel highlighting how AI practitioners are working with forest stewards to bring fire management into a new, data-enabled era.

Firewatch duty was traditionally served by fire lookouts on towers overlooking forests, but moving that role into the 21st century is easier said than done. In order to be truly useful to first responders in a wildfire, an AI model needs to be able to detect smoke – not fire – very, very quickly. After all, by the time a fire is large enough to be easily spotted, it’s typically less containable, and if the smoke is reported too late to act on it, the fire will have had substantial time to grow. 

“Every piece of information that can give you a better picture of what’s going on at that time can make crucial differences in the decision that you actually use to move forward,” said Kristen M. Allison, a fire planning management specialist for the U.S. Forest Service. Still, for many practitioners, analog devices are still frequently used to communicate and collect data due to large dead zones of cell services that spread over the forestland that hosts wildfires. 

“When I heard about the wildland fires in California, it really struck me personally,” said Jacci Cenci-McGrody, senior technical marketing engineer for Nvidia. “I thought to myself, ‘How can I use Nvidia’s technologies to help the firefighters?’”

The answer: AI-enabled drones and a collaborative AI pipeline.

Anusua Trivedi, senior data science lead at Microsoft’s AI for Good research lab, had been working with Cenci-McGrody for several years. Cenci-McGrody approached Trivedi with the idea of using AI models to enable smoke detection. Trivedi, who thought the idea “fit very well” with AI for Good’s objectives, began scouting AI models for smoke detection.

This turned out to be a difficult task. From an AI perspective, detecting smoke is harder than detecting fire: smoke is not a discrete object (like an animal), but is, instead, amorphous and translucent, necessitating boundaries for when smoke stops being smoke. Smoke can also be easily confused with clouds – and Trivedi knew that with lives on the line, it was essential to have a model that predicted smoke and nothing else. Finally, the model had to be extremely lightweight. This was a requirement because the model would be run on unmanned aerial vehicles (UAVs), an increasingly common tool for forest management. 

Cenci-McGrody supplied Trivedi and the AI for Good team with a wealth of labeled smoke data. The AI for Good team tested a variety of models on the data, eventually settling on a two-pronged approach combining object detection with image segmentation and based on the MobileNet-V2 neural network vision model. IBM Cloud Object Storage (running on an on-premises data lake) ingested UAV-produced wildfire videos and images, feeding the images to the model on Nvidia DGX-2 systems using IBM Spectrum Scale for training. The result: a lightweight model for smoke detection.

Meanwhile, a little over a year ago, another partner entered the mix: AeroVironment, a UAV manufacturer. Working with Nvidia and Microsoft, AeroVironment delivered a prototype implementation. “One of the things that we developed was a system using a Puma AE [UAV], explained Bogdan Pugach, a computer vision and deep learning engineer at AeroVironment. “The idea behind this is you can launch it at a wildfire and it uses AI to autonomously detect the smoke, find the fire, and route to it, and it creates a real-time map of the fire that gets sent to smartphones on the ground for real-time information.” 

Even with that actionable result for first responders, the data flow doesn’t stop there. Once the drones return, the data is loaded onto large systems for storage, after which teams of data labelers filter and label the immense amounts of data in order to make it useful for conservation researchers. These researchers, like Eric Rowell (a wildland fire scientist at Tall Timbers Research & Land Conservancy) can put the data to use for longer-term projects.

“We’ve been developing fast-solving fluid dynamics-based fire behavior models that require and then use these inputs to put out new ways to look at fire behavior,” Rowell explained. “Specifically, to look at fire behavior at really fine scales and give us a space [where] we can help managers expand what their decision space is, in terms of when they choose to do a prescribed fire or even just decisions that they make as they’re managing a wildland fire.”

The team sang each other’s praises, highlighting how collaboration made the ambitious project possible. “Having everybody – like Kristen, Eric, Bogdan, Jacci – all together has made this model work so well,” Trivedi said. “All their efforts have been invaluable in bringing everything into the model and making the inference work to help the field work.”

Still, Allison – who is on the front lines of fire response – cautioned the audience that the project was not a silver bullet for wildfires.

“There’s no magic solution in disaster management,” Allison said. “What this can do is it can improve validation, it can improve collaboration, it can put people on the same page to make decisions that are right in that time and that space.”