DIFET: Distributed Feature Extraction Tool For High Spatial Resolution Remote Sensing Images

In this paper, we propose distributed feature extraction tool from high spatial resolution remote sensing images. Tool is based on Apache Hadoop framework and Hadoop Image Processing Interface. Two corner detection (Harris and Shi-Tomasi) algorithms and five feature descriptors (SIFT, SURF, FAST, BRIEF, and ORB) are considered. Robustness of the tool in the task of feature extraction from LandSat-8 imageries are evaluated in terms of horizontal scalability.

💡 Research Summary

The paper presents DIFET (Distributed Feature Extraction Tool), a system designed to extract local image features from high‑resolution remote sensing imagery in a distributed fashion using the Apache Hadoop ecosystem and the Hadoop Image Processing Interface (HIPI). The authors motivate the work by noting that modern satellite sensors such as Landsat‑8 generate images on the order of several gigapixels (approximately 7 000 × 7 000 pixels, 32‑bit RGBA), which translates into hundreds of megabytes per scene. Processing such data on a single workstation quickly becomes a bottleneck in terms of CPU time, memory consumption, and I/O throughput, especially when many images must be processed for applications like object detection, target tracking, image matching, or stitching.

DIFET’s architecture places Hadoop at the core, leveraging its distributed file system (HDFS) for storage and the MapReduce programming model for parallel execution. HIPI is used to bundle many images into a single HipiImageBundle (HIB) file, which reduces the number of HDFS metadata entries and allows each mapper to receive one image as a key‑value pair (the key being a HipiImageHeader, the value a FloatImage). Inside the mapper, the FloatImage is converted to an OpenCV Mat object, transformed to grayscale, and then processed by one of several feature‑detection or descriptor algorithms. The paper implements two corner detectors (Harris and Shi‑Tomasi) and five descriptor pipelines (SIFT, SURF, FAST, BRIEF, ORB). After processing, the resulting data (key points, descriptors, or visualizations) are converted back to FloatImage and written to HDFS, completing the Map phase. No Reduce phase is required because each image is processed independently.

The authors provide pseudo‑code for the Harris corner detector and the SURF descriptor to illustrate the mapper logic. They also discuss the choice of algorithms: Harris and Shi‑Tomasi are classic, accurate corner detectors but are computationally heavier; FAST is extremely lightweight and suitable for real‑time scenarios; SIFT and SURF provide scale‑ and rotation‑invariant descriptors at the cost of higher CPU usage (especially SIFT, which is known to be slow without GPU acceleration); BRIEF and ORB are binary descriptors that dramatically reduce memory and computation while still offering reasonable discriminative power.

Experimental evaluation was performed on a modest cluster of four commodity machines (each with a quad‑core Intel i7‑950, 8 GB RAM, two 1 TB 7200 RPM disks) running Ubuntu 10.10 and Hadoop 1.0.2, as well as a single‑machine MATLAB baseline. Two data sets were used: three Landsat‑8 images and twenty images, each roughly 230 MB in memory. Running times for each algorithm were measured on one node (MATLAB), two nodes (MapReduce), and four nodes (MapReduce). For the small data set (N = 3), the single‑node baseline was faster, reflecting the overhead of Hadoop’s job setup and data shuffling. However, for the larger data set (N = 20), the four‑node cluster consistently outperformed the two‑node configuration, achieving speed‑ups of roughly 30 %–45 % depending on the algorithm. This demonstrates that the MapReduce approach scales favorably when the workload per node is sufficiently large to amortize the framework overhead.

The number of extracted features per algorithm was also reported. FAST and ORB produced thousands of key points, while SIFT and SURF yielded a few hundred, reflecting their differing design goals. These counts are important for downstream tasks such as image matching or stitching, where a balance between descriptor richness and computational load must be struck.

The paper acknowledges several limitations. HIPI’s native support is limited to common image formats (JPEG, PNG); remote sensing formats such as GeoTIFF or HDF5 require pre‑conversion, adding an extra I/O step. The current implementation relies on the Java bindings of OpenCV, which incurs additional data copying between Hadoop’s FloatImage representation and OpenCV’s Mat structure. Moreover, the system is built on Hadoop 1.x, lacking the more flexible resource management and containerization features of YARN (Hadoop 2.x) or modern in‑memory frameworks like Apache Spark. Consequently, real‑time or low‑latency processing scenarios are not well served.

Future work suggested by the authors includes extending HIPI to natively handle GeoTIFF/HDF5, migrating the pipeline to Hadoop YARN or Spark to benefit from dynamic resource allocation, and integrating GPU‑accelerated OpenCV or deep‑learning based feature extractors (e.g., SuperPoint, D2‑Net) to improve both speed and robustness. They also propose building a full end‑to‑end remote‑sensing workflow that couples DIFET’s feature extraction with large‑scale image matching, mosaicking, and change detection, and evaluating the cost‑effectiveness of running such pipelines on cloud platforms.

In summary, DIFET demonstrates that a Hadoop‑based, MapReduce‑driven approach can successfully distribute the computationally intensive task of local feature extraction across multiple commodity nodes, achieving measurable speed‑ups for realistic remote‑sensing workloads. By integrating well‑known computer‑vision algorithms through HIPI and OpenCV, the system provides a practical foundation for big‑data remote‑sensing analytics, while also highlighting areas where further engineering (format support, newer execution engines, hardware acceleration) can push performance toward the needs of operational, large‑scale Earth observation programs.