Defects and Inconsistencies in Solar Flare Data Sources: Implications for Machine Learning Forecasting

Machine learning models for forecasting solar flares have been trained and evaluated using a variety of data sources, including Space Weather Prediction Center (SWPC) operational and science-quality data. Typically, data from these sources is minimally processed before being used to train and validate a forecasting model. However, predictive performance can be affected if defects and inconsistencies between these data sources are ignored. For a set of commonly used data sources, along with the software that queries and outputs processed data, we identify their defects and inconsistencies, quantify their extent, and show how they can affect predictions from data-driven machine-learning forecasting models. We also outline procedures for fixing these issues or at least mitigating their impacts. Finally, based on thorough comparisons of the effects of data sources on the trained forecasting model’s predictive skill scores, we offer recommendations for using different data products in operational forecasting.

💡 Research Summary

The paper investigates how defects and inconsistencies in commonly used solar‑flare data sources affect the performance of machine‑learning (ML) forecasting models. The authors focus on two broad categories of data: (1) flare response data, primarily the GOES X‑ray flare catalogs, which exist in both operational (real‑time) and science‑quality (post‑processed) versions, and (2) predictor data, which include high‑resolution SDO/HMI and AIA images as well as vector‑derived parameters such as the SHARP and SMARP feature sets.

In the introduction, the authors note that recent flare‑forecasting studies have largely emphasized model architecture (logistic regression, SVM, random forests, LSTM, etc.) while often assuming that the underlying data are flawless. They argue that, because flare events are sparse—especially the high‑energy M‑ and X‑class flares—any systematic bias or error in the data can dominate model performance.

Section 2 provides a systematic comparison of the GOES flare catalogs. Operational data are available with minimal latency but suffer from timestamp mismatches, missing small‑flare entries, and occasional duplicate records. Science‑quality data are more complete and internally consistent but are released with a delay that makes them unsuitable for real‑time forecasting. By aligning the two catalogs over the 2010‑2018 interval, the authors quantify defect rates of 3–7 % and demonstrate that using the raw operational catalog degrades skill scores (e.g., TSS drops by ~0.08) relative to a cleaned version.

Section 3 examines predictor‑data defects. Image‑based predictors face resolution loss when full‑disk 4096 × 4096 images are down‑sampled for computational tractability; this can erase fine‑scale structures such as polarity‑inversion lines (PILs) that are known to be highly predictive. Different studies adopt inconsistent cropping strategies (e.g., flux‑maximizing rectangles versus PIL‑centered windows), leading to non‑uniform feature representations. Vector‑based predictors (SHARP, SMARP) suffer from version‑dependent parameter definitions, missing values (NaNs), and occasional physically impossible entries (e.g., negative magnetic flux). These issues propagate through normalization and feature‑selection pipelines, causing unstable model training.

In Section 4 the authors train two representative ML models—a temporal LSTM network and a random‑forest classifier—on four data configurations: (a) raw operational response + raw predictors, (b) cleaned response + raw predictors, (c) raw response + cleaned predictors, and (d) fully cleaned data. Performance is evaluated with standard flare‑forecasting metrics (True Skill Statistic, Heidke Skill Score, ROC‑AUC) and regression metrics (RMSE, MAE) for peak X‑ray flux prediction. The fully cleaned configuration consistently outperforms the others, with skill‑score improvements of 5–10 % and a 15 % reduction in RMSE for continuous flux forecasts. Notably, the detection rate for rare X‑class events improves by over 12 percentage points when defects are removed.

Section 5 synthesizes practical recommendations. For operational forecasting, the authors propose an automated preprocessing pipeline that (i) synchronizes timestamps across data streams, (ii) imputes missing flare entries using neighboring time windows, (iii) applies non‑linear interpolation to preserve high‑frequency image details, and (iv) standardizes vector‑parameter versions while handling NaNs via median imputation and sample weighting. For research and model development, they advise using the most complete science‑quality datasets, applying the same cleaning steps, and explicitly documenting data‑source versions to ensure reproducibility.

In conclusion, the study demonstrates that data‑source defects are a dominant source of error in ML‑based solar‑flare forecasting, often eclipsing algorithmic choices. By quantifying these defects, providing concrete cleaning procedures, and offering clear guidance on when to use operational versus science‑quality data, the paper equips both researchers and operational forecasters with the tools needed to build more reliable, robust flare‑prediction systems.