Developing a Portable Solution for Post-Event Analysis Pipelines

In recent years, the monitoring and study of natural hazards have gained significant attention, particularly due to climate change, which exacerbates incidents like floods, droughts, storm surges, and landslides. Together with the constant risk of earthquakes, these climate-induced events highlight the critical necessity for enhanced risk assessment and mitigation strategies in susceptible areas such as Italy. In this work, we present a Science Gateway framework for the development of portable and fully automated post-event analysis pipelines integrating Photogrammetry techniques, Data Visualization and Artificial Intelligence technologies, applied on aerial images, to assess extreme natural events and evaluate their impact on risk-exposed assets.

💡 Research Summary

The paper presents a Science Gateway framework that integrates photogrammetry, artificial intelligence, and data visualization into a portable, fully automated post‑event analysis pipeline for natural hazards. Motivated by the increasing frequency and severity of climate‑induced disasters (floods, droughts, storm surges, landslides, earthquakes) and the need for rapid damage assessment, the authors develop a system that can be deployed on the INAF PLEIADI high‑performance computing (HPC) infrastructure in Italy.

The related‑work section reviews existing gateways such as CyberGISX, MyGEOHub, Quakeworx, and OneSciencePlace, highlighting their domain‑specific focus. In contrast, the proposed solution adopts a multi‑paradigm approach: Apache Airflow for programmatic workflow orchestration, Common Workflow Language (CWL) for declarative, portable workflow definitions, and Docker for reproducible execution environments.

The underlying infrastructure consists of three PLEIADI sites (Bologna, Catania, Trieste). The Catania site provides 56 CPU nodes (up to 256 GB RAM) and, since March 2025, ten GPU nodes equipped with four NVIDIA Tesla V100 GPUs each. Storage is handled by the parallel file system BEEGFS (174 TB) and inter‑node communication uses a 100 Gbps Omni‑Path network. Job scheduling is performed by SLURM, while Airflow’s components (metadata database, scheduler, web server, executor, workers) are deployed in a distributed fashion. User authentication and authorization rely on Keycloak (OIDC), the front‑end is built with React, the back‑end with FastAPI, and all services run inside Docker containers for easy deployment and maintenance.

The post‑event analysis pipeline comprises four main steps: (1) UAV image acquisition with predefined flight plans and high overlap; (2) photogrammetric reconstruction using Agisoft Metashape’s Python API to generate Structure‑from‑Motion (SfM) models, depth maps, point clouds, and georeferenced 3D meshes; (3) AI‑driven semantic segmentation of the aerial imagery, performed by a Docker‑packaged deep‑learning model invoked through Airflow’s PythonOperator; and (4) web‑based visualization and data delivery via CesiumJS, allowing stakeholders to explore 3D digital twins, download results, and integrate them into GIS workflows.

Two distinct Airflow DAGs are implemented: one for the photogrammetric chain and another for the machine‑learning chain. Both DAGs use a combination of DockerOperator, BashOperator, and PythonOperator, and they read a user‑generated configuration file (dagrun.cfg) that contains all processing parameters. The photogrammetric DAG orchestrates image import, quality filtering, camera alignment, depth‑map generation, and optional point‑cloud‑to‑mesh conversion, leveraging GPU acceleration on the V100 nodes. The machine‑learning DAG consumes the resulting 3D models, runs the segmentation model, and produces damage masks and attribute tables. CWL descriptors are also provided, enabling the same workflow to be executed on alternative platforms such as Kubernetes.

Preliminary experiments on thousands of high‑resolution UAV images demonstrate that a complete 3D reconstruction can be achieved in under ten minutes, while the segmentation model attains an F1‑score of 0.87 for damage detection. The CesiumJS web client loads the visualizations in less than five seconds, and concurrent user sessions do not degrade performance, thanks to the scalable Airflow‑Celery executor and the high‑throughput storage system.

In conclusion, the authors deliver a reproducible, scalable, and user‑friendly Science Gateway that automates the entire post‑disaster analysis workflow, from raw UAV data to interactive 3D damage maps. Future work will extend the pipeline to additional hazard types, incorporate real‑time streaming data, and adopt FAIR principles for broader data sharing and international collaboration.