Semi-supervised Complex-valued GAN for Polarimetric SAR Image Classification

SEMI-SUPER VISED COMPLEX-V ALUED GAN FOR POLARIMETRIC SAR IMA GE CLASSIFICA TION Qigong Sun, Xiufang Li, Lingling Li, Xu Liu, F ang Liu, Licheng Jiao K ey Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, International Research Center for Intelligent Perception and Computation, Joint International Research Laboratory of Intelligent Perception and Computation, School of Artificial Intelligence, Xidian Uni versity , Xian, Shaanxi Province 710071, China ABSTRA CT Polarimetric synthetic aperture radar (PolSAR) images are widely used in disaster detection and military reconnaissance and so on. Howe ver , their interpretation faces some chal- lenges, e.g., deficienc y of labeled data, inadequate utilization of data information and so on. In this paper , a comple x-v alued generativ e adv ersarial network (GAN) is proposed for the first time to address these issues. The complex number form of model complies with the physical mechanism of PolSAR data and in fa v or of utilizing and retaining amplitude and phase information of PolSAR data. GAN architecture and semi- supervised learning are combined to handle deficiency of la- beled data. GAN expands training data and semi-supervised learning is used to train network with generated, labeled and unlabeled data. Experimental results on two benchmark data sets show that our model outperforms existing state-of-the- art models, especially for conditions with fewer labeled data. Index T erms — PolSAR image classification, complex- valued operations, semi-supervised learning, generative ad- versarial network 1. INTR ODUCTION Many researches ha ve been done on PolSAR image classifica- tion, and breakthrough benefits from the development and ap- plication of deep conv olutional neural networks(DCNN) [1]. As we all kno w , PolSAR data are usually e xpressed by coher- ent matrices or cov ariance matrices which contain amplitude and phase information in complex number form. Howe ver , a general real-valued DNN loses significant phase informa- tion when it is applied to interpret PolSAR data directly . [2] con verts a complex-valued coherent or cov ariance matrix into a normalized 6-D real-valued vector for PolSAR data clas- sification, while ignoring important phase information. Dif- ferent from direct conv ersion of complex number into a real This work was supported in part by the State Key Program of National Natural Science of China (No. 61836009, No. 91438201 and No. 91438103), the National Natural Science Foundation of China (No. 61871310, No. 61876220). number , some other strategies are introduced. Besides the coherency matrix extended to the rotation domain, Chen et al . [3] also take the null angle and roll-in variant polarimetric features as input to e xtract ample polarimetric features. Liu et al . [4] propose a novel polarimetric scattering coding method for gaining more polarimetric features in classification. Ho w- ev er , their operations are all in the real number domain. Instead, in order to make full use of PolSAR data in- formation, some complex-valued DNN models are proposed. Inspired by the application of complex-v alued con volutional neural network (CV -CNN) [5], Zhang et al [6] proposed the application of CV -CNN on PolSAR data classification and obtained a great success. This is the beginning of CV -CNN to classify PolSAR data. Besides retaining information, CV - CNN has the strengths of faster learning and conv erenge [7]. In addition, deep learning is a data-dri ven approach. How- ev er , the labeled samples are extremely deficient in PolSAR data. Thus, unsupervised or semi-supervised networks are used for the classification of PolSAR data, for example, deep con v olutional autoencoder [8]. Meanwhile, GAN [9] is able to e xpand data. It can learn the potential distribution of ac- tual data and generate fake data that has the same distrib u- tion with actual data. W ith the successful application in man y fields (the generation of natural images [10] and Neural Di- alogue [11] and so on), the GAN architecture has receiv ed increasing attention in recent years. In order to further solve the deficiency of labeled data, it is advisable to combine GAN architecture and semi-supervised learning. Therefore, in this paper , we propose a complex-v alued GAN framework. Our nov el model has three advantages: 1) The comple x- valued neural network complies with the physical mechanism of the complex numbers, and it can retain amplitude and phase information of PolSAR data; 2) GAN extended to complex number field can expand PolSAR samples, which hav e similar distribution with actual samples. Increased sam- ples can improv e the classification performance of PolSAR data. 3) Besides labeled data, unlabeled data are also used to update model parameters by semi-supervised learning and improv e network performance to a certain extent. 2. SEMI-SUPER VISED COMPLEX-V ALUED GAN 2.1. Network Architectur e The data generated by general real-v alued GAN is differ - ent from PolSAR data in feature and distrib ution. There- fore, we extend real-valued GAN to the complex number domain and propose a complex-valued GAN. Figure 1 il- lustrates the framew ork of our model, and it is composed by Complex-v alued Generator and Complex-v alued Dis- criminator . This framework consists of complex-v alued full connection, complex-v alued deconv olution, complex- valued con volution, complex-v alued activ ation function and complex-v alued batch normalization, which are represented by ”CFC”, ”CDeCon v”, ”CCon v”, ”CA” and ”CBN”, respec- tiv ely . In addition, a comple x-valued network also makes full use of the amplitude and phase features of PolSAR data. C A C B N C A C A - - . . . + + . . . - + C B N C B N C A C A - + C = 1 C = 2 C = 3 C = 4 C = K - 1 C = K . . . F a k e R e a l C A . . . . . . C F C C D e C o n v C C o n v C C o n v C o m p l e x - v a l u e d D i s c r i m i n a t o r C o m p l e x - v a l u e d G e n e r a t o r 1 3 2 R e a l P a r t I m a g i n a r y P a r t R e s h a p e R e s h a p e Fig. 1 : The framework of semi-supervised complex-v alued GAN for image classification.  denotes minus arguments in element-wise and ⊕ denotes adds arguments in element-wise. In the Complex-v alued Generator , after a serious of complex-v alued operations, two randomly generated v ec- tors sho wn as the green block and blue block are translated into a complex-v alued matrix, which has the same shape and distribution with PolSAR data. In the Complex-v alued Discriminator , we use complex-v alued operations to extract complete complex-v alued features, which are in the form of a pair . Then we concatenate the real part and imaginary part of the last feature to the real domain for final classifica- tion. In the training processing, generated fake data, labeled and unlabeled actual data are used to alternately train this complex-v alued GAN by semi-supervised learning, and until the network can effecti vely identify the authenticity of input data and achiev e correct classification. 2.2. Complex-V alued Operation Mask For simplifying the calculation, we choose the algebraic form to e xpress a complex number . In the algebraic form, the num- bers in real part and imaginary part are real numbers with one dimension. W e use z 1 = a + ib and z 2 = c + id to denote two comple x numbers, the multiplication and addition are re- defined as follows: z 1 ∗ z 2 = ( a + ib ) ∗ ( c + id ) (1) = ( a ∗ c − b ∗ d ) + i ( a ∗ d + b ∗ d ) z 1 ± z 2 = ( a ± c ) + i ( b ± d ) (2) T o indicate the complex-v alued operation mentioned in detail, a complex-v alued operation mask is proposed, as shown in Figure 2. The green and the blue block represents the real and imaginary part, respecti vely . This mask can make some complex number calculations, whose input data ( I N r , I N i ), the weight ( W r , W i ) and output data ( OU T r , O U T i ) are consisted of a real part and an imaginary part. Therefore, this type of operation can be decomposed to four traditional real operations, one addition operation and one subtraction operation. Each complex-v alued operation in our network complies with this mask. The same expression and physical mechanism of data and network parameters in fav or of obtaining full data features used for classification. + op op op op OUT_r OUT_i IN_r IN_i W_r W_i Complex-Valued Operation Mask Fig. 2 : Comple x-V alued Operation Mask. The circular block denotes real- valued operations, the red circles are undetermined operations and the violet are explicit operations. ”op” can be full connection, con volution or deconv o- lution. 2.3. Complex-V alued Batch Normalization Batch normalization has been widely used in deep neural net- works for unifying data and accelerate con vergence rate. In addition, complex-valued batch normalization can stabilize the performance of GANs. Ho we ver , scanty training samples and less batch sizes restrict the effect of batch normalization. In order to address this issue, a novel batch normalization is proposed in this paper . The e xpectation and cov ariance ma- trices are replaced by constantly updated av erage expectation and cov ariance matrices, so that they hold all sample informa- tion in training proceeding. The follo wing formulation shows the normalization of the t th batch x t : ˆ x t = ( ¯ V t ) − 1 2 ( x t − ¯ σ t ) (3) where ¯ σ t and ¯ V t represent the a verage e xpectation and co- variance matrix from t − m to t batches, which is computed as follows: ¯ σ t = 1 m t X t − m E [ x t ] (4) ¯ V t =  ¯ V t rr ¯ V t ri ¯ V t ir ¯ V t ii  (5) =  1 m P t t − m V t rr 1 m P t t − m V t ri 1 m P t t − m V t ir 1 m P t t − m V t ii  where m denotes the length of state remembered, and ¯ V ri is equal to ¯ V ir . The square root of a Matrix of 2 times 2 ¯ V t is computed: S t = ( ¯ V t rr × ¯ V t ii − ¯ V t ri × ¯ V t ri ) 1 2 (6) T t = ( ¯ V t rr + ¯ V t ii + 2 S t ) 1 2 (7) ¯ V − 1 2 t = " ( ¯ V t ii + S t ) S t T t − ¯ V t ri S t T t − ¯ V t ri S t T t ( ¯ V t rr + S t ) S t T t # (8) This operation can translate the data mean to 0 and vari - ance to 1. Ultimately , we use the following computing to de- note complex-v alued batch normalization: B N ( ˆ x t ) = γ ˆ x t + β (9) where γ and β are defined as two parameters to reconstruct the distribution. 2.4. Semi-Supervised Learning In this complex-valued GAN, for further utilizing features of unlabeled data, we use semi-supervised learning to optimize network with a classifier of softmax. The output of genera- tor (G) is a K + 1 dimensional vector { p 1 , p 2 , ..., p K , p K +1 } , where from p 1 to p K are the probability of first K classes and p K +1 is the probability of input image being fake. In order to optimize the generator (G) and discriminator (D), we define the loss function as follows: L = L labeled + L unlabeled + L gener ated (10) L labeled = − E [ log P ( C | X real , C < K + 1)] (11) L unlabeled = − E [ log [1 − P ( C = K + 1 | X real )]] (12) L gener ated = − E [ log P ( C = K + 1 | X f ak e )] (13) where L labeled , L unlabeled and L g ener ated represent classifi- cation loss of labeled samples, unlabeled samples, and gener - ated samples, respectively . Therefore, classification losses of labeled and generated samples are easily acquired. Howe ver , the classification loss of unlabeled samples is not easy to ex- press because of inexplicit ground truth. W ith this inevitable problem, the output probability of softmax is operated as fol- lows: p sum = log K X i =1 e ( p − p max ) + p max (14) where p max denotes the max value in p i ( i < K + 1) , and logistic regression as a binary classification is utilized. When the output approaches 1, the probability p K +1 << p sum ac- cordingly , the f acticity of data is discriminated. By this de- duction, unlabeled data can also be used to update our net- work model. 3. EXPERIMENTS In our experiments, two benchmarks data sets of Flev oland and San Francisco are used. In order to verify the effecti ve- ness of our method, our model is compared with complex- valued con volutional neural network (CV -CNN) and real- valued con v olutional neural network (R V -CNN), they have similar configurations with our Complex-v alued Discrimina- tor . The ov erall accuracy (OA), average accuracy (AA), and Kappa coefficient are used to measure the performance of all the methods. 3.1. Experiments on Standard Data Set W e use a coherent matrix T , which is a 3 × 3 conjugate sym- metrical complex v alue matrix and follows complex W ishart distribution, to express all information of the corresponding pixel on PolSAR images. In Fle voland data, 0.2%, 0.5%, 0.8%, 1.0%, 1.2%, 1.5%, 1.8%, 2.0%, 3.0%, 5.0% labeled data in each of 15 categories are randomly selected as training data, and the remained labeled data for testing. In addition, 10% unlabeled samples are used to train our semi-supervised complex-v alued GANs. In San Francisco data, we randomly chose 10, 20, 30, 50, 80, 100, 120,150, 200, 300 labeled data in each of the 5 cate gories for training and 10% data, no mat- ter whether labeled, as actual samples. 0.75 0.8 0.85 0.9 0.95 1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Overall Accuracy (% ) Sample Rate (% ) 0.75 0.8 0.85 0.9 0.95 1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Average Accuracy (% ) Sample Rate (% ) 0.75 0.8 0.85 0.9 0.95 1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Kappa Sample Rate (% ) RC CC Ours RC CC Ours RC CC Ours Fig. 3 : Flev oland O A, AA, and Kappa in different sample ratios. 0.75 0.8 0.85 0.9 0.95 1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Over all Ac c u r ac y (% ) S amp le Rate (% ) RC CC Our s 0.75 0.8 0.85 0.9 0.95 1 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 A ve r age Ac c u r ac y (% ) S amp le Rate (% ) RC CC Our s 0.75 0.8 0.85 0.9 0.95 1 0 0.5 1 1.5 2 2.5 3 3.5 4 4. 5 5 K ap p a Sampl e Rat e (% ) RC CC O u r s 55 60 Ov 6 5 a 7 0 Acc 7 5 ura 8 0 cy 8 5 %) 9 0 95 1 00 0 50 1 00 1 50 2 00 2 50 3 00 er ll ( Sa mple Number RC CC O urs 5 5 6 0 6 5 7 0 7 5 8 0 8 5 9 0 9 5 1 00 0 50 1 00 1 50 2 00 2 50 3 00 Av er a g e Acc ura cy ( %) Sa mple Number RC CC O urs 6 0 6 5 7 0 7 5 8 0 8 5 9 0 9 5 1 00 0 50 1 00 1 50 2 00 2 50 3 00 K a pp a Sa mple Number RC CC O urs 55 60 65 70 75 80 85 90 95 100 0 50 100 150 200 250 300 Overall Accuracy (% ) Sample Number RC CC Ours 55 60 65 70 75 80 85 90 95 100 0 50 100 150 200 250 300 Average Accuracy (% ) Sample Number RC CC Ours 60 65 70 75 80 85 90 95 100 0 50 100 150 200 250 3 00 Kappa Sample Number RC CC Ou rs Fig. 4 : San Francisco OA, AA, and Kappa in dif ferent sample numbers. T able 1 : Classification accuracy(%), O A(%), AA(%) and Kappa Flevoland methods 1 2 3 4 5 6 7 8 9 RC 87.18 97.85 95.56 94.58 86.72 93.96 98.17 98.89 96.70 CC 90.79 98.39 95.95 89.71 93.00 93.21 97.46 99.24 97.54 ours 98.22 99.25 99.29 86.71 95.40 95.27 99.85 99.85 98.59 methods 10 11 12 13 14 15 OA AA Kappa RC 94.88 97.70 83.45 95.56 99.00 52.95 95.12 91.54 94.68 CC 98.02 97.01 91.18 90.48 98.91 65.57 95.12 93.10 94.68 ours 97.56 97.76 96.07 99.06 100.0 87.38 97.21 96.68 96.97 San Francisco methods 1 2 3 4 5 OA AA Kappa RC 99.16 86.86 59.93 19.29 31.52 74.36 59.35 63.37 CC 99.07 84.05 53.81 65.51 50.14 80.83 70.51 72.41 ours 99.45 88.33 86.72 61.91 90.61 89.23 85.41 84.48 The parameters of all experiments in this paper are set as follows: the patch size is 32 × 32 , the learning rate is 0.0005, and the optimization method is Adam with β 1 = 0 . 5 and β 2 = 0 . 999 . Figure 3 and Figure 4 show the change of OA, AA, and Kappa with the sample ratio in two data sets. In Flev oland data, the results verified the superiority of our new network with less labeled samples, and this la w especially obvious when training samples less than 3.0%. This same advantage also is shown in San Francisco data, especially if numbers of training data less than 50. In order to exhibit the contributions of our model on each category , we list all test accuracy of Flev oland data with 0.8% sampling ratio and of San Francisco data with 10 labeled training samples in T able 1. In Flevoland data, we can find that accuracies of different categories have generally improved especially for the fifteenth category , which has the least training samples and achiev es in- crease of 65.1% and 33.17% compare to the real-valued and complex-v alued neural networks in accuracy , respecti vely . In San Francisco data, comparing to the complex-v alued neural network, complex-valued GAN further improves classifica- tion accuracy than the real-valued neural network, especially for Developed, Low-Density Urban and High-Density Urban with the increase of 44.7%, 220.9%, 187.4%. 3.2. Generated Data Analysis In order to analyze the effecti veness of our complex-v alued GAN, we discuss the similarity of actual and generated data in appearance and distribution. T ake Fle voland data for ex- ample, we randomly select 100 pcolors of the real part in di- agonal elements of T , as shown in Figure 5. W e can clearly find that generated data hav e high similarity with actual data. Based on the known data distrib ution of T matrix [12], we further count the distribution of actual and generated data in Figure 6. For actual data, the real and imaginary part statis- tic histograms of T 11 shown in (a1) and (a2) and of T 12 in (a3) and (a4). (b1) - (b4) represent the corresponding statistic histograms of generated T 11 and T 12 . W e can find the high similarity of generated data with actual data. ( a 1 ) ( a 2 ) ( a 3 ) ( a 4 ) ( b 1 ) ( b 2 ) ( b 3 ) ( b 4 ) Fig. 5 : Pcolor comprised by real parts of T 11 , T 22 , T 33 . (a1 - a4) show the actual data image patches. (b1 - b4) show the generated data image patches. (a1) (a2) (a3) (a4) (b1) (b2) (b3) (b4) Fig. 6 : Histograms of representativ e v ariables. (a1 - a4) are the statistics of actual data, and (b1 - b4) are the statistics of generated data. 4. CONCLUSION In this paper , a complex-v alued GAN is proposed to classify PolSAR data. Nearly all operations are extended to the com- plex number field, and this model obeys the physical mean- ing of PolSAR data and holds complete phase and amplitude feature. T o the best of our kno wledge, this is the first time that complex-v alued data is generated by a netw ork, and the generated data is similar to actual complex-v alued data in ap- pearance and distribution The complex-v alued GAN is alter- nately trained with generated data, labeled data and unlabeled data by semi-supervised learning. With the utilization of un- labeled and generated samples features, our complex-v alued semi-supervised GAN obtains ob viously precede ov er other models especially when labeled samples are insuf ficient. It opens up a ne w w ay for our researches on solving the prob- lem of lacking complex-v alued samples. 5. REFERENCES [1] Alex Krizhevsk y , Ilya Sutske ver , and Geoffre y E Hinton, “Im- agenet classification with deep conv olutional neural networks, ” in NIPS , 2012, pp. 1097–1105. [2] Y u Zhou, Haipeng W ang, Feng Xu, and Y a-Qiu Jin, “Polari- metric sar image classification using deep con volutional neural networks, ” IEEE Geoscience and Remote Sensing Lett. , vol. 13, no. 12, pp. 1935–1939, 2016. [3] Si-W ei Chen and Chen-Song T ao, “Polsar image classifica- tion using polarimetric-feature-driven deep con v olutional neu- ral network, ” IEEE Geoscience and Remote Sensing Lett. , v ol. 15, no. 4, pp. 627–631, 2018. [4] Xu Liu, Licheng Jiao, Xu T ang, Qigong Sun, and Dan Zhang, “Polarimetric conv olutional network for polsar image classifi- cation, ” IEEE T rans. Geosci. Remote Sens. , 2018. [5] Nitzan Guberman, “On complex valued con volutional neural networks, ” arXiv pr eprint arXiv:1602.09046 , 2016. [6] Zhimian Zhang, Haipeng W ang, Feng Xu, and Y a Qiu Jin, “Complex-v alued conv olutional neural network and its appli- cation in polarimetric sar image classification, ” IEEE T rans. Geosci. Remote Sens. , vol. PP , no. 99, pp. 1–12, 2017. [7] T Nitta, “On the critical points of the complex-v alued neural network, ” in Neur al Information Pr ocessing, 2002. ICONIP’02. Pr oceedings of the 9th International Confer ence on . IEEE, 2002, vol. 3, pp. 1099–1103. [8] Jie Geng, Jianchao Fan, Hongyu W ang, Xiaorui Ma, Baom- ing Li, and Fuliang Chen, “High-resolution sar image clas- sification via deep conv olutional autoencoders, ” IEEE Geo- science and Remote Sensing Lett. , v ol. 12, no. 11, pp. 2351– 2355, 2015. [9] Ian Goodfello w , Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David W arde-Farley , Sherjil Ozair , Aaron Courville, and Y oshua Bengio, “Generati ve adversarial nets, ” in NIPS , 2014, pp. 2672–2680. [10] Alec Radford, Luke Metz, and Soumith Chintala, “Unsuper- vised representation learning with deep con volutional genera- tiv e adversarial netw orks, ” arXiv pr eprint:1511.06434 , 2015. [11] Jiwei Li, W ill Monroe, T ianlin Shi, S ´ ebastien Jean, Alan Ritter , and Dan Jurafsky , “ Adversarial learning for neural dialogue generation, ” arXiv pr eprint:1701.06547 , 2017. [12] Nathaniel R Goodman, “Statistical analysis based on a certain multiv ariate complex gaussian distribution (an introduction), ” The Annals of mathematical statistics , vol. 34, no. 1, pp. 152– 177, 1963.