PulseSatellite: A tool using human-AI feedback loops for satellite image analysis in humanitarian contexts

PulseSatellite: A tool using human-AI feedback loops f or satellite image analysis in humanitarian contexts T omaz Logar , 1 Joseph Bullock, 1 , 2 Edoardo Nemni, 3 Lars Bromley , 3 John A. Quinn, 1 Miguel Luengo-Oroz 1 1 United Nations Global Pulse 2 Institute for Data Science, Durham Univ ersity 3 UNOSA T tomaz@unglobalpulse.org, miguel@unglobalpulse.or g Abstract Humanitarian response to natural disasters and conflicts can be assisted by satellite image analysis. In a humanitarian con- text, v ery specific satellite image analysis tasks must be done accurately and in a timely manner to provide operational sup- port. W e present PulseSatellite, a collaborative satellite im- age analysis tool which lev erages neural network models that can be retrained on-the fly and adapted to specific humani- tarian conte xts and geographies. W e present two case studies, in mapping shelters and floods respecti v ely , that illustrate the capabilities of PulseSatellite. Introduction When dealing with conflict and humanitarian crises, accu- rate and timely satellite image analysis is key to supporting critical operations on the ground. Use cases include moni- toring population displacement, settlement mapping, dam- age assessment, fire detection associated with human rights violations, damage to transportation networks, floods assess- ment or identifying direct impact of earthquakes, v olcanoes, cyclones and landslides (Lang et al. 2015). In these contexts, automated processes pro viding vital information in decision making must be carefully v alidated and tuned to maximal performance since e.g. a false positi v e may be detrimental to human life. While there exist se veral general purpose satel- lite image analysis tools, few are designed and optimized for humanitarian use cases. PulseSatellite is a tool to ana- lyze satellite imagery assisted by neural networks that seeks to incorporate humans-in-the-loop at different stages in the model inference process to enable optimal results and expert validation in humanitarian contexts. The conceptual frame- work used to implement and deploy PulseSatellite has been previously presented in (Quinn et al. 2018). PulseSatellite Functionalities The tool enables the performance of such specific humani- tarian tasks as identifying and counting structures in refugee settlements and rapid mapping flooded areas after a natural disaster . These tasks can be repetitiv e in nature, and hence lend themselves to support with AI tools; howe ver , each spe- cific analysis scenario can ha ve distinct features (such as a certain type of shelter construction) which are not all fea- sible to handle with a single pre-trained system. Therefore, the PulseSatellite system allows users to fine-tune a hierar- chy of models as shown in Figure 1, thus including e xpert knowledge through a human-in-the-loop system. F or exam- ple, the generic shelter detection model may be fine tuned progressiv ely to a model specialized for desert terrain, then to another which is specific to a different scenario (such as the camp in Figure 2). An y of these models can then be used as starting points for new scenarios. On-the-fly e valuation helps the analyst to determine whether the assessment qual- ity is good enough for release, in which case detection re- sults can be exported. These steps can all be carried out via the web front end and with multiple remote collaborators. PulseSatellite Machine Learning Framew ork PulseSatellite is designed to host a variety of specialised models, fine-tuned for dif ferent functionalities. Given that each scenario may require different outputs, the tool must be dynamic such that new models can be trained in real- time (as users provide feedback on detection results), and easily adaptable to new situations. In this section we e xplore a series of case studies demonstrating a cross-section of hu- manitarian contexts PulseSatellite is being used to tackle. Figure 1: Detail of PulseSatellite showing a panel that allo ws the selection of a trained network from a hierarchy of models adapted to different use cases. System Architecture PulseSatellite is a distributed cloud application, currently running on Amazon W eb Services (A WS). The web-based user interface (UI) is written in Jav aScript+HTML, and the web server , which communicates with the UI via HTTP and WS protocols, is written in NodeJS. The database is man- aged using PostgreSQL with PostGIS extension and is cur - rently also used as a message streaming service for Pub/Sub messaging between the web, GPU and tiling serv ers. All backend files are generated and stored on a Amazon Elas- tic File System (EFS). Multiple GPU servers can run ML processes across a range of framew orks including PyT orch, T ensorFlow and Keras. GIS processes, including the tiling servers, use OSGeo libraries and programs, most notably GD AL. All used software is open source. Case Study: Mapping Refugee Settlements Counting and classifying structures in a refugee settlement is a common analysis task for humanitarian agencies. In prac- tice, this is currently manually done by human expert ana- lysts using satellite imagery . A single settlement may ha ve tens of thousands of structures, and identifying each of them can take se veral days. For camp mapping, we hav e a Mask R-CNN model (He et al. 2017) trained on images from 12 settlements, where the image of each has been split into 300x300 pixel tiles and annotated by human e xperts. Once the model has been run on an unseen camp, the analyst can inspect the result in the tool and correct the outputs on a subset of tiles. An adaptation stage can then be performed to fine-tune the model to the unseen image and increase per- formance. In recent independent tests camp completion rates increased from 77.3% to 94.7% after adaptation, with a final user accuracy of 94.4% -in line with humanitarian perfor - mance requirements described in (Quinn et al. 2018). Figure 2: PulseSatellite mapping a refugee camp in Jordan. Model detected buildings are highlighted in blue, with ana- lyst ev aluation structures in green. Case Study: Rapid Flood Mapping Rapid analysis of regions af fected by natural disasters is essential for response planning. Current methods for rapid flood mapping require an analyst to perform a series of man- ual pixel intensity thresholding and image processing op- erations, taking sev eral hours to be completed. The Puls- eSatellite tool uses a U-Net model (Ronneberger , Fischer , and Brox 2015) to produce a flood map in the order of min- utes. The model has been trained and tested on images from Bangladesh and Somalia achieving ov erall accuracy scores of ov er 90%, in line with current manual methods. Impact and Discussion The PulseSatellite camp mapping function has been pi- loted in operational conditions in multiple settlements across Africa and the Middle East. Results rev ealed that while high lev els of accuracy are possible with machine learning meth- ods, there is considerable variation in the characteristics of new imagery in real life scenarios requiring analysis work- flows that allo w the incorporation of automatic methods in a progressiv e manner . Using AI assistants as an augmentation of human analysts is a reasonable strategy to transition from current fully manual operational pipelines to ones which are both more efficient and hav e the necessary lev els of quality control. PulseSatellite is e xpected to allow human analysts to drastically reduce the time required to hav e high accuracy detection in particular use cases. Future specific AI modules and front-end functionalities to be added in PulseSatellite include change detection for cultural heritage preservation and damage assessment. PulseSatellite is a vailable to inter- national and humanitarian organizations. Acknowledgments United Nations Global Pulse is supported by the Gov- ernments of Netherlands, Sweden and Germany and the W illiam and Flora Hewlett Foundation. The United Nations Institute for T raining and Researchs (UNIT AR) Operational Satellite Applications Programme (UNOSA T) is supported by the Norwegian Agency for Dev elopment Cooperation. JB also is supported by UK Science and T echnology Facilities Council (STFC) grant number ST/P006744/1. References [He et al. 2017] He, K.; Gkioxari, G.; Doll ´ ar , P .; and Gir- shick, R. B. 2017. Mask r-cnn. 2017 IEEE International Confer ence on Computer V ision (ICCV) 2980–2988. [Lang et al. 2015] Lang, S.; F ¨ ureder , P .; Kranz, O.; Card, B.; Roberts, S.; and Papp, A. 2015. Humanitarian emer gencies: causes, traits and impacts as observed by remote sensing. In Remote Sensing of W ater Resources, Disasters, and Urban Studies . CRC Press. 483–512. [Quinn et al. 2018] Quinn, J. A.; Nyhan, M. M.; Nav arro, C.; Coluccia, D.; Bromley , L.; and Luengo-Oroz, M. 2018. Hu- manitarian applications of machine learning with remote- sensing data: revie w and case study in refugee settlement mapping. Philosophical T ransactions of the Royal Society A: Mathematical, Physical and Engineering Sciences. 376. [Ronneberger , Fischer, and Brox 2015] Ronneberger , O.; Fischer , P .; and Brox, T . 2015. U-net: Con volutional networks for biomedical image segmentation. Medical Image Computing and Computer-Assisted Intervention (MICCAI) 9351:234–241.