Targeted change detection in remote sensing images

T argeted c hange detection in remote sensing images Vladimir Ignatiev 1 , Alexey T rekin 1 , Viktor Lobac hev 1 , Georgy P otap o v 1 , and Evgen y Burnaev 1 Sk olko vo institute of Science and technology , Moscow, Nobel st. 1, Russia, aeronetlab@skoltech.ru , WWW home page: http://crei.skoltech.ru/cdise/aeronet-lab/ Abstract. Recen t developmen ts in the remote sensing systems and im- age pro cessing made it p ossible to prop ose a new method of the ob ject classification and detection of the sp ecific c hanges in the series of satel- lite Earth images (so called targeted c hange detection). In this pap er w e prop ose a formal problem statement that allows to use effectiv ely the deep learning approach to analyze time-dep enden t series of remote sensing images. W e also introduce a new framework for the developmen t of deep learning mo dels for targeted change detection and demonstrate some cases of business applications it can b e used for. Key w ords: remote sensing, change detection, satellite imagery , com- puter vision, deep learning 1 Problem o verview Earth observ ation systems ha ve b een developing during the last decades in terms of qualit y of the imagery and frequency of the imaging [27]. How ev er, un til no w the daily basis imagery wasnt av ailable for any lo cation or area of interest. Eac h satellites and UA V systems hav e their own limitations - viz. n umber of satellites, weather and dayligh t for optical systems, vegetation for SAR systems, not to sa y the commercial ones. That lead the remote sensing industry to the situation where the amount of the data and the pro claimed frequency of the rep eating imaging could hav e given us full daily cov erage of the earth surface by the high-resolution imagery [23], but in the reality it turns out to be composed of the div erse, inconsistent and often incomplete data. And yet that business and technology trend raise the issues how all this daily imagery can b e targeted to the sp ecific needs and used in a v arious t yp es of applications. The tec hnical issues that can b e raised following these business needs are: – data fusion of different types of remote sensing data coming from differen t sensors (optical, SAR, different resolution etc.) - to provide the maxim um capacit y of op erational monitoring in an y time or weather conditions [18, 19] – detection of c hanges in the time series of imagery and in a v ery short time spans to pro vide near real time v aluable information related to the ob jects 2 Vladimir Ignatiev et al. of interest. As the traditional approac hes fail to mak e this massiv e dataflo w a reliable source of the useful answers w e should prop ose the new machine learning approaches that can handle it and provide the automated analysis that can comp ete to the human one b y accuracy The business needs lead to the concept of the targeted change detection (TCD) that is consolidated detection and classification of the changes in order to obtain only the in teresting for the problem (target) changes rather than all the c hanges in general [25, 26]. Our motiv ation in this pap er is to build a general description of change detection (CD) problem taking into accoun t characteristics of input data - series of remote sensing (RS) images - and information ab out “targets” obtained from classification of sp ecific business applications of the resp ective territories suc h as “transp ortation & logistic”, “residen tial areas”, “building & construction” etc. Ha ving formalized problem statement it is p ossible to develop o v erall framework that will reduce costs and simplify stages of collecting data, mo del construction and in tegration in workflo w of business solutions. 1.1 Business applications of CD Commercial services pro ducing analytics based on series of multitemporal re- mote sensing images b egin o ccupying significant share in suc h market areas as smart farming [2, 3, 4], monitoring of securit y zones of linear ob jects that hav e enormous transport netw ork [9, 10, 7], monitoring of protected areas [5, 6, 8], as- sessmen t of damage caused by natural and man-made disasters [ ? , ? ], analytical solutions for consulting agencies in geo-mark eting [12, 13]. Our surv ey of the commercial solutions based on analysis of RS images rev eals the follo wing problems: – only a n umber of applications hav e algorithmic solutions; – most of solutions do not ha ve enough accuracy; – the plen ty of problems do not ha ve any reliable solutions; – companies are in search of new framew orks that will anticipate their risks or will mak e precise and timely estimation of losses. It is ob vious that eac h of use cases has its own demands and requiremen ts to the c hanges that should b e detected. F or example, if one is going to monitor securit y zones of pip elines, he should find out the list of targets, c hanges of whic h will b e detected. Each target means a specific t yp e of ob jects that has the c haracteristic linear size and this v alue defines the spatial resolution of RS images to be used. Most of real business applications requires to pro cess with RS images of high and v ery high spatial resolution from 5 cm/pixel to 10 m/pixel. 1.2 RS datasets av ailable for CD algorithms The idea of making general framework pro viding developmen t of robust algo- rithms for detecting c hanges for the sp ecific applied problem requires building T argeted change detection 3 represen tative datasets. There are scarce datasets for change detection on RS im- ages no wada ys [14, 15, 16, 17]. Only few of existing datasets can b e implemen ted within the framework of targeted c hange detection [14, 17]. Most of them hav e not enough cov erage or time spread to learn and test algorithms using mo dern tec hniques of mac hine learning and deep learning. So the one of the challenges of our w ork is to collect and build a num b er of datasets dedicated to the task of targeted CD. 1.3 Curren t metho ds and algorithms Solutions prop osed in the latest pap ers could b e divided into several groups according to the used mo dels and algorithms, to market domains and to RS images in use. T raditional metho ds of change detection by series of satellite images are t ypically limited in several aspects: – using just t wo images for justification with ground truth; – do not tak e into accoun t usability of changes in specific use case; – only empirical approaches for selection of the appropriate RS images for real applications; – no automated end-to-end solutions for business cases. One of the first applied research in this area was conducted by W.Rop er and S.Dutta [29]. They made an attempt to understand which kind of sensor system aerial or satellite is b etter in a sense of probability of detection unauthorized in trusion on to a pipeline right-of-w ay . The main conclusion of this work is that one should use aerial imagery when the imaging frequency is lo w er than once per mon th, but starting from once per da y imaging frequency - probabilit y of c hange detection with satellite imagery is higher. There were use d standard algorithms of classification built-in suc h softw are pack ages as ENVI [28]. The future works were fo cused on impro ving c hange detection algorithms themselv es, but the idea of targeted CD were pushed into the background. So, the b est classical approaches are methods based on anomaly detection frame- w ork on time series of multispectral low-resolution satellite images and so called sp ectral indices [34], metho ds based on Mark ov Random Fields and global op- timization on graphs [35, 36, 37], approac hes using ob ject-based segmen tation with post-classification of c hanges [38, 39, 40] and metho ds based on Multiv ari- ate Alteration Detection [41, 30]. All the men tioned approac hes shows quiet goo d results in quality , but either use m ulti framed series of lo w resolution satellite images or use high resolution images, but the same tw o pairs of images for b oth training and testing. Latest researc h achiev emen ts in computer vision made it possible to use deep neural netw orks for accurate semantic segmentation ob jects on daily images. The next step was to transfer this technology on remote sensing data. Actually , its ongoing research. This results are the basement for developmen t new algorithms of CD. It can b e emphasised the most successful studies in this direction. A.M. Amin et al. [42] prop osed a metho d for binary classification of changes based on CNN with input tw o high resolution satellite images (0.65 - 2.62 m/pixel) gives 4 Vladimir Ignatiev et al. the accuracy by Kappa co efficien t - 0.876. The second approach b y Y. Chu et al. [43], based on clustering analysis of difference of tw o images (0.65 - 2.62 m/pixel) uses deep b elief netw ork model with restricted Boltzmann mac hine arc hitecture sho ws the accuracy by Kappa co efficien t from 0.905 to 0.987. The main adv an- tage of this method that it allo ws to a void radiometric correction of images, they outp erform the standard approac hes and could be adopted for targeted change detection in differen t use cases. Deep learning techniques need a lot of compu- tational resources for training pro cess and could b e unstable for analysis of new images, but the last problem may b e eliminated ha ving a represen tative dataset for mo del training. F urther researc hes describ ed in [30, 37, 45, 32, 31] giv e promising results, but most of them share the same drawbac k: low amoun t of data used for the metho d v alidation. The of creation a large and reliable dataset is one of the keys to the dev elopment of robust and reliable algorithms for the targeted change detection. One of the imp ortant issues is that many real problems do not need any c hanges to b e detected, but only the interesting, problem-sp ecific ones. This implies a c hallenging task to develop a metho d that could extract only the sp ecified classes of c hanges. The deep neural netw orks app ear to be the most promising approach to the problem, but the curren t solutions are highly limited b y the amount of the data a v ailable for the training of the mo dels. 2 Remote sensing data for c hange detection In order to understand the capabilit y of the mo dern space imagery we need to examine its characteristics. As stated in the previous section, according to the target sizes w e need the images in high (1-10 meters) and very high (less than 1 meter) spatial resolution. A t ypical high resolution satellite in 2018 has 4 or more spectral bands and a panc hromatic image of higher spatial resolution (see table 2). A completely new approach to the data acquisition is demonstrated b y the PlanetScope satellite group, that consists of a large group of v ery small satellites (3-unit Cub eSat), that can acquire far more imagery than traditional satellites of comparable mass and cost, ho wev er at the cost of less data quality: they ha ve less spectral bands and the noise level is higher [20]. SPOT-6 PlanetScop e W orldview-3 Visible channels 4 3 6 Near infrared c hannels 1 0/1 2 SW infrared c hannels 0 0 8 Multisp ectral resolution, m 6.0 3.0 / 3.7 1.24 / 3.72 P anchromatic resolution, m 1.5 - 0.31 Daily cov erage, millions sq.km/ day 3 150 0.68 T able 1. Ov erview of the technical sp ecifications of the modern RS satellites T argeted change detection 5 Let us call a set of images taken by one space or aerial instrument at one flight and cov ering the same territory a “multi-la y er image”. W e need this term do differen tiate from the multispectral image where each lay er represents a sp ectral band and the panchromatic la yer is not included in to the m ultispectral image. In our case we add it as well as any other information represented in a raster form as a lay er to our image representation, as we wan t to collect all the information within a single instance of a m ulti-lay er image. I = I 1 , ..., I C is a m ulti-lay er image with C bands (c hannels). Every band I i is a single 2-dimensional matrix with the size ( X i , Y i ). Differen t bands can hav e differen t spatial resolution so, under the condition that all the bands must cov er the same territory , the sizes of the channels are different, but prop ortional b y an in teger ratio. If we order the bands b y the decrease of the size, their dimensions will b e X i /X i + 1 = Y i / Y i +1 ∈ N F or the task of change detection the images are organized in time series: S = I 1 , ...I T , where each image I t , t = 1 ...T co vers the same area, but differen t images can b e obtained by differen t instruments and ha ve differen t characteris- tics. W e in tentionally put this restriction on the area, b ecause w e need not to describ e all the data a v ailable, but to construct the datasets that can b e used by the data analysis methods within the prop osed framework. If the given data are not fully spatially consistent, we can crop the exten ts and, if needed, divide the time series in to several separate ones of differen t spatial and temp oral exten t. 3 Prop osed framew ork This section discusses details of prop osed framework and shows the example of metho ds for targeted change detection in classical situation when tw o input satellite images are used for classification of one type of changes. 3.1 Problem statement Giv en the series of images S = I 1 , ...I T where each image I t is a m ulti-lay er image tak en at the moment of time t , a parameterized description of changes o ccurred in the series can b e introduced as C = F ( S ). The exact definition of the op erator F is done at the p ost-processing stage and is b eyond the scop e of this paper. It is assumed that the calculation of the vector C will use only the set of ob jects of pair-wise changes { r ij } , so the problem goal can b e reduced to the searc h of r ij - the change b et ween the images I i and I j . The pair c hange ob ject definition includes: – Category of detected c hanges; – The set of p olygonal domains (in ph ysical co ordinates) where the changes are detected. Ev aluation of the detected c hange is done with one of the measures: – In tersection ov er Union; 6 Vladimir Ignatiev et al. – The pair Recall-Precision; – F1-measure. All three measures assume a v ailabilit y of the ground truth changes for the in- v estigated pair of images. Eac h measure can be calculated either per unit area or p er ob ject; the latter is sp ecified heuristically . 3.2 Description of the framew ork for TCD applications Our framew ork is restricted neither b y the t ype of remote sensing data, nor meth- o ds, algorithms, but it do es tak e into account information ab out the “targets” of the applied area. The question of determining the context of using the notion “change” is remo ved by establishing a one-to-one correspondence of the model or algorithm for detecting changes and the business case to b e solved. Robustness of the created algorithms is guaran teed by creating reference sets of markup of c hanges for a series of different images and applying modern methods of in-depth training on data generated from reference sets. As one can see on the sc heme depicted on the figure 1 there are three stages in prop osed framew ork: preparing dataset, training a mo del and analyzing series of images for pair-wise targeted c hange detection. Dataset for sp ecific business application should b e constructed in accordance with the formal mathematical problem statement describ ed ab ov e. It means that one should load series of images concerning the size of targets, the characteristic time perio d of o ccurrence of c hanges, type of images and so on. This data are combined with external mark- ups of changes of ob jects (targets) under consumption. The mo del for pair-wise targeted c hange detection is c hosen from the set of neural netw orks arc hitectures prepared in adv ance and is trained and v alidated on the dataset built of the first stage. The process of analyzing new series of images starts from building map of changes for each pair of images inside the series. After that there is a p ost-processing and verification pro cedures that resulted in recommendation, forecasts or alerting rep orts with detected changes, that can b e visualized in in teractive mode. 3.3 Metho d of TCD based on deep learning techniques Our metho d is fundamentally different from all the metho ds listed in section 1.3 because it takes into account information ab out targets of the application under consumption, is based on deep neural netw ork trained and v alidated on represen tative dataset, can be automatically adopted to a new territory . The approach we are dev eloping is based on a sp ecific arc hitecture of the con volutional net work whic h has had a great success in solving the semantic segmen tation problems. In the mathematical problem statement, a general for- mal description of the detected changes is introduced for a pair of images; the description being applied to all datasets analyzed. The neural netw ork archi- tecture makes it possible to use the pre-trained parameters of the net work and adapt it to new datasets with an additional training at a very little exp ense. T argeted change detection 7 Fig. 1. Scheme of the prop osed framework 8 Vladimir Ignatiev et al. The tec hnique can b e applied to all pairs of the images in the series; the p ost-processing is then used to build a more common ob ject that describes the image evolution within the series. The p ost-processing can be specific for eac h business application and do es not depend on the deep learning tec hnique used at the early stages of analysis. 4 Conclusion In this paper w e hav e summarized the kno wledge about the business applications of change detection, kno wn methods of the image pro cessing and deep learning and the p ossibilities and limitations of the RS data to prop ose a generalized framew ork that can embed the solutions for different problems in the area of the targeted change detection in the remote sensing data. The prop osed frame- w ork to be implemented in our future research will facilitate the integration of the current and future data sources and the adv anced algorithms of the data pro cessing. References 1. PlanetScop e Flock, http://space.skyrock et.de/do c sdat/flo c k-1.htm 2. Gama ya, https://gama y a.com/ 3. DigitalGlob e Services, https://www.digitalglobe.com/industries/agriculture 4. Exact F arming, www.exactfarming.com 5. So vzond, h ttps://sovzond.ru/ 6. Unmanned Systems, h ttp://unmanned.ru/en/ 7. Orbital Eye - PIMSyS, h ttps://orbitaleye.nl 8. Zala Aero Group, http://zala.aero/ 9. I-Cub ed, h ttp://www.i3.com/ 10. Satellite Imaging Corporation, www.satimagingcorp.com 11. Europ ean Space Agency , h ttp://www.esa.int/ESA 12. OOO Rilos, h ttps://geomatrix-retail.com/ 13. Geomark eting & Location In telligence, h ttps://www.gim.b e/en/products- services/gim-exp ertise/geomark eting-location-intelligence 14. Bourdis, N., Marraud, D., & Sahbi, H. (2011, July). Constrained optical flow for aerial image c hange detection. In Geoscience and Remote Sensing Symp osium (IGARSS), 2011 IEEE International (pp. 4176-4179). IEEE. 15. SAR Change Detection Dataset, http://fenggao.sinaapp.com/san dataset.h tml 16. Go ogle Earth Engine, https://earthengine.google.com/datasets/ 17. CDnet, http://www.c hangedetection.net/ 18. Dong, J., Zhuang, D., Huang, Y., & F u, J. (2009). Adv ances in multi-sensor data fusion: Algorithms and applications. Sensors, 9(10), 7771-7784. 19. Zhang, J. (2010). Multi-source remote sensing data fusion: status and trends. In- ternational Journal of Image and Data F usion, 1(1), 5-24. 20. San tilli, G., V endittozzi, C., Capp elletti, C., Battistini, S., & Gessini, P . (2018). Cub eSat constellations for disaster managemen t in remote areas. Acta Astronautica. T argeted change detection 9 21. h ttps://www.intelligence-airbusds.com/files/pmedia/public/r2928 9 in t 012 sp ot6- 7 en lo w-2.p df 22. h ttp://worldview3.digitalglobe.com/ 23. h ttps://www.planet.com/pro ducts/satellite- imagery/files/Planet Com bined Imagery Pro duct Specs December2017.p df 24. h ttps://qz.com/1126301/the-company-photographing-ev ery-sp ot-of-land-on- earth-ev ery-single-day/ 25. F ernndez-Prieto, D., & Marconcini, M. (2011). A nov el partially supervised ap- proac h to targeted c hange detection. IEEE T ransactions on Geoscience and Remote Sensing, 49(12), 5016-5038. 26. Y e, S., Chen, D., & Y u, J. (2016). A targeted c hange-detection pro cedure by com- bining change vector analysis and post-classification approac h. ISPRS Journal of Photogrammetry and Remote Sensing, 114, 115-124. 27. T oth, C., & Jkw, G. (2016). Remote sensing platforms and sensors: A survey . ISPRS Journal of Photogrammetry and Remote Sensing, 115, 22-36. 28. ENVI softw are, https://www.harris.com/solution/en vi 29. Rop er, W., & Dutta, S. (2005). Oil spill and pipeline condition assessmen t using remote sensing and data visualization managemen t systems. In: 2005 ESRI User Conference Pro ceedings, San Diego, California, pp. 1-25. 30. W ang, B., Choi, J., Choi, S., Lee, S., W u, P ., & Gao, Y. (2017). Image F usion-Based Land Cov er Change Detection Using Multi-T emp oral High-Resolution Satellite Im- ages. Remote Sensing, 9(8), 804. 31. Huang, S., Ramirez, C., Kennedy , K., Mallory , J., W ang, J., & Chu, C. (2017). Up dating land co ver automatically based on change detection using satellite images: case study of national forests in Southern California. GIScience & Remote Sensing, 54(4), 495-514. 32. Mcdermid, G. J., Pape, A., Chubey , M. S., & F ranklin, S. E. (2003, Octob er). Ob ject-oriented analysis for change detection. In Proceedings of the 25th Canadian symp osium on remote sensing, Mon tral, Canada (pp. 14-17). 33. Alm utairi, A., & W arner, T. A. (2010). Change detection accuracy and image prop erties: a study using sim ulated data. Remote Sensing, 2(6), 1508-1529. 34. Cai S., & Liu, D. (2015). Detecting Change Dates from Dense Satellite Time Series Using a Sub-Annual Change Detection Algorithm. Remote Sensing, 7, 8705-8727. doi:10.3390/rs70708705. 35. V ak alopoulou, M., Karantzalos, K., Komo dakis, N., & Paragios, N. (2015). Simul- taneous registration and change detection in multitemporal, very high resolution remote sensing data. In Pro ceedings of the IEEE Conference on Computer Vision and Pattern Recognition W orkshops (pp. 61-69). 36. Y u, H., Y ang, W., Hua, G., Ru, H., & Huang, P . (2017). Change Detection Using High Resolution Remote Sensing Images Based on Active Learning and Mark ov Random Fields. Remote Sensing, 9(12), 1233. 37. Gu, W., Lv, Z., & Hao, M. (2017). Change detection method for remote sensing images based on an impro ved Marko v random field. Multimedia T o ols and Appli- cations, 76(17), 17719-17734. 38. Jian ya, G., Haigang, S., Guorui, M., & Qiming, Z. (2008). A review of multi- temp oral remote sensing data change detection algorithms. The In ternational Arc hives of the Photogrammetry , Remote Sensing and Spatial Information Sciences, 37(B7), 757-762. 39. Huang, L., F ang, Y., Zuo, X., & Y u, X. (2015). Automatic change detection metho d of m ultitemp oral remote sensing images based on 2D-Otsu algorithm improv ed by firefly algorithm. Journal of Sensors, 2015. 10 Vladimir Ignatiev et al. 40. Vittek, M., Brink, A., Donna y , F., Simonetti, D., & Descle, B. Land cov er change monitoring using Landsat MSS/TM satellite image data o ver west africa betw een 1975 and 1990. Remote Sensing 2014; 6 (1): 658-676. 41. Jabari, S., & Zhang, Y. (2016, April). Building c hange detection using m ulti-sensor and m ulti-view-angle imagery . In IOP Conference Series: Earth and Environmen tal Science (V ol. 34, No. 1, p. 12-18). IOP Publishing. 42. El Amin, A. M., Liu, Q., & W ang, Y. (2016, July). Con volutional neural netw ork features based change detection in satellite images. In First In ternational W orkshop on Pattern Recognition (V ol. 10011, p. 100110W). In ternational Society for Optics and Photonics. 43. Ch u, Y., Cao, G., & Hay at, H. (2016). Change Detection of Remote Sensing Image Based on Deep Neural Netw orks. 44. Lyu, H., Lu, H., & Mou, L. (2016). Learning a transferable change rule from a recurren t neural netw ork for land cov er change detection. Remote Sensing, 8(6), 506. 45. Sakurada, K., Ok atani, T., & Deguchi, K. (2013, June). Detecting changes in 3D structure of a scene from multi-view images captured b y a vehicle-moun ted camera. In Computer Vision and Pattern Recognition (CVPR), 2013 IEEE Conference on (pp. 137-144). IEEE.