Typical Performance of Irregular Low-Density Generator-Matrix Codes for Lossy Compression

T ypical p erformance of irreg ular lo w-den sit y generator -matrix co de s for lossy compress ion Kazushi Mim ura F aculty of Information Sciences, Hiroshima Cit y Universit y , 3-4 -1 Ohtsuk a-Higa shi, Asaminami-Ku, Hiroshima, 7 31-31 94, Japan E-mail: mimura @hiro shima-cu.ac.jp Abstract. W e ev aluate t ypica l p erfor ma nce of irregular lo w- density generator -matrix (LDGM) co des, whic h is defined b y spar se matrices with arbitrar y irregular bit degree distribution and ar bitrary c he ck degree distribution, for lossy compr e s sion. W e apply the replica metho d under o ne-step replica sy mmetry break ing (1RSB) ans a tz to this problem. P ACS n umber s: 89.90.+n, 0 2.50,-r , 0 5.50.+q , 7 5 .10.Hk T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 2 1. In tr o duction The c hannel co ding can b e considered as the dual problem of lossy source coding in rate-distortion the o r y [1, 2]. Matsunaga and Y amamoto sho w ed that it is p o ssible to approac h t he bina r y rate-distortion b ound using LDPC co des [3]. In recen t y ears, lossy source co ding problem based on lo w-densit y g enerator-matrix (LDGM) co des is widely in v estigated. This sc heme can a ttain high p erformance very close to t he Shannon b ound, ho we ver it needs solin ving a com binatorial o pt imizatio n problem to obtain optimal source co ding. Some practical encoding a lgorithms are prop osed for this sc heme, e.g., a b elief- propagation-ba sed encoder prop osed b y Mura y ama [4] and a surv ey-propaga tion-based enco der prop osed b y W ain wrigh t and Manev a [5]. P erformance of this sc heme is a lso explored by v arious approa c hes. Muray ama and Ok ada applied replica metho ds to ev aluate p erfor ma nce of LD GM co des defined b y regular sparse matrices for lossy compression [6]. Ciliberti et al. ha v e used the ca vit y metho d to ev aluate c hec k-regular LDGM p erformance [7, 8]. On the other hand, Martinian and W ain wright deriv ed rigorous upp er b ounds on the effectiv e rate-distortio n function of LD GM co des for the binary symmetric source [9]. D imakis et al. deriv ed lo we r bounds for c hec k-regular LDG M co des [10, 11]. With respect to irregular LDGM co des analyzed so f a r, elemen ts of a repro duced message are giv en b y exactly K elemen ts c hosen at r a ndom from a co dew o rd. This implies that previous analyses treat only the case where a bit degree distribution is P oissonian. An irregular bit and c heck degree distributions of a generator matrix are not optimized for lossy source co ding. The goal of this pap er is to ev aluate how typical p erformance of ir r egular LDGM co des f or lossy compression dep ends on a bit degree distribution and a c heck degree distribution. 2. Bac kground Let us first provide the concepts of the rate- distort ion theory [1 ]. Let x b e a binary i.i.d. source discrete whic h tak es in a source alphab et X = { 0 , 1 } with P [ x = 0 ] = P [ x = 1] = 1 / 2, where P represen ts the probabilit y of its argument. An source mes sage of M random v ariables, x = t ( x 1 , · · · , x M ) ∈ X M , is compressed into a shorter expression, where the op erator t denotes the transp ose. The enco der desc rib es the source sequence x ∈ X M b y a co deword z = F ( x ) ∈ X N . The dec o der represen ts x b y a repro duced message ˆ x = G ( z ) ∈ X M . No t e that M represe nts the length o f a source sequence, while N ( < M ) represen ts the length of a co dew ord. The co de rate is R = N / M . The distortion b et w een single letters is measure d b y the Hamming distortion defined by d ( x, ˆ x ) = n 0 , if x = ˆ x, 1 , if x 6 = ˆ x, (1) and t he distortion b etw een M -bit sequences x ∈ X M and ˆ x ∈ X M is measured by the av eraged single-letter distortion as d ( x , ˆ x ) = 1 M P M µ =1 d ( x µ , ˆ x µ ). This results in T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 3 the pro babilit y of error distortion, since E [ d ( x, ˆ x )] = P [ x 6 = ˆ x ], where E represe n ts the exp ectatio n. The distortion asso ciated with the co de is defined as D = E [ d ( x , ˆ x )], where the exp ectation is ov er the probability distribution on X M × X M . A rate distortion pair ( R, D ) is said to b e achievable if there exists a sequen ce of rate distortion co des ( F , G ) with E [ d ( x , ˆ x )] ≤ D in the limit M → ∞ . The rate distortion function R ( D ) is the infim um of rates R suc h that ( R, D ) is in the rate distortion region of the source for a g iv en distortion D . The rate-distortion function of a Bernoulli(1 / 2 ) i.i.d. source is giv en b y R ( D ) = 1 − h 2 ( D ) , (2) where h 2 ( x ) = − x log 2 ( x ) − (1 − x ) log 2 (1 − x ) is the binary entrop y function. 3. Lossy Compression Sc heme An source message o f M random v ar ia bles, x ∈ X M , is compressed in to a shorter expression, where the op erator t denotes the transp ose. The encoder describes the source sequence x ∈ X M b y a co dew ord z = F ( x ) ∈ X N . The decoder represen ts x b y a repro duced message ˆ x = G ( z ) ∈ X M . The co de rate is R = N / M ≤ 1 . Using a given M × N sparse matrix A = ( a µi ) ∈ { 0 , 1 } M × N , the deco der is defined as G ( z ) = A z (mo d 2) . (3) The encoding is represen ted b y F ( x ) = a rgmin ˆ z ∈X N d ( x , G ( ˆ z )) , (4) where d is the distortion measure. In this pap er, w e use the Hamming distortion. Although the definition means that a computationa l cost of the enco ding is of O ( e N ), w e can utilize some sub optimal algorithms based o n mess a g e passing to enco de [4, 5]. 4. Analysis T o simplify the calculations, w e first in tro duce a simple isomorphism b et w een the additiv e Bo olean group ( { 0 , 1 } , ⊕ ) and the m ultiplicativ e Ising gro up ( { +1 , − 1 } , × ) defined b y J × ˆ J = ( − 1) x ⊕ ˆ x , where J, ˆ J ∈ { +1 , − 1 } = J and x, ˆ x ∈ { 0 , 1 } = X . Hereafter, w e use the following Ising (bip o lar) represen tations : the Ising source message J ∈ J M , the Ising reproduced message ˆ J ∈ J M and the Ising codeword ξ ∈ J N . The source bit can b e described as a ra ndo m v ariable with the probabilit y: P J ( J ) = 1 2 δ ( J − 1) + 1 2 δ ( J + 1) , (5) where δ ( x ) denotes D irac’s delta function. The µ - t h elemen t of the Ising repro duced message ˆ J µ is giv en b y pro ducts of the elemen ts of the t en tativ e Ising co dew ord s ∈ J N : ˆ J µ = Y i ∈L ( µ ) s i , (6) T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 4 where L ( µ ) = { i | a µi = 1 , A = ( a µi ) } . The matrix A has K µ nonzero elemen ts in the µ -th row and C i nonzero elemen ts in the i -th column. W e consider the source length and the co dew or d length to b e infinite, while co de rate R is k ept finite. The parameter K 1 · · · K M and C 1 , · · · , C N are usually of O ( N 0 ), therefore the mat rix A b ecomes very sparse. In densely constructed cases, w e also assume t hat these par a meters are not of O ( N 0 ) but K , C 1 , · · · , C N ≪ N holds. Coun ting the n um b er of nonzero elemen ts in the mat rices leads t o K 1 + · · · + K M = C 1 + · · · + C N . The co de rat e is therefore R = ˜ K / ˜ C , where ˜ K = 1 N P M µ =1 K µ and ˜ C = 1 N P N i =1 C i . Code constructions are describ ed b y the conne ctivit y para meter D µ i 1 , ··· ,i K µ ∈ { 0 , 1 } whic h sp ecifies a set o f indices i 1 , · · · , i K µ corresp onding to nonzero elemen ts in the µ -th ro w of the sparse matrix A . The connectivit y parameter is defined b y D µ i 1 , ··· ,i K µ = ( 1 , if { i 1 , · · · , i K µ } = L ( µ ) 0 , otherwise . (7) An ensem ble of co des is generated as follow s. (i) Sets of { K 1 , · · · K M } and { C 1 , · · · C N } are sampled indep enden tly fro m an iden tical distributions P K ( K ) and P C ( C ), resp ective ly . ( ii) The connectivit y parameters D µ i 1 , ··· ,i K µ are generated suc h that M X µ =1 X h i 1 = i,i 2 , ··· ,i K µ i D µ i,i 2 , ··· ,i K µ = C i , (8) where P h i 1 = i,i 2 , ··· ,i K µ i denote the summation ov er { ( i 2 , · · · , i K µ ) ∈ { 1 , · · · , N } K µ − 1 | i 2 < · · · < i K µ ,i 2 6 = i, · · · , i K µ 6 = i } . T o analyz e typic a l perfo r ma nce of rate-compatible LDGM co des f o r lossy compression, w e apply a analytical metho d similar to references [6, 12, 13, 14, 15]. The Ha mming distortion d ( J , ˆ J ) b ecomes d ( J , ˆ J ) = 1 2 − 1 2 M P M µ =1 J µ { Q i ∈L ( µ ) s i } , since J , ˆ J ∈ J M . Using the connectivit y parameter D µ i 1 , ··· ,i K µ , w e can rewrite this Hamming distortion in the form: d ( J , ˆ J ) = 1 2 − 1 2 M M X µ =1 X h i 1 , ··· ,i K µ i D µ i 1 , ··· ,i K µ J µ s i 1 · · · s i K µ , (9) where P h i 1 , ··· ,i K µ i denote the summation o ver { ( i 1 , · · · , i K µ ) ∈ { 1 , · · · , N } K µ | i 1 < · · · < i K µ } . W e here define the Hamiltonian H ( s , J ) = M d ( J , ˆ J ( s )) , (10) to explore t ypical p erformance. The free energy is calculated from t he pa rtition function Z ( β ) = P s ∈J N exp[ − β H ( s , J )]. F rom the free energy , w e can obtain a distortion b et we en a n o r ig inal message and a repro duction message D for a fixed co de r a te R . W e follo w the calculation of references [6, 1 2, 16 , 17, 18, 19]. T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 5 4.1. R eplic a symmetric solution W e first assume the replica sym metry (RS). Using the replica symmetric pa rtition function Z RS ( β ), w e find the replica symmetric free energy as f RS ( β ) = − 1 β nM ln E A , J [ Z RS ( β ) n ] (11) = 1 2 − 1 β extr π , ˆ π h ln cosh β 2 − ¯ K Z 1 − 1 dxπ ( x ) Z 1 − 1 d ˆ x ˆ π ( ˆ x ) ln(1 + x ˆ x ) + X K P K ( K )  K Y k =1 Z 1 − 1 dx k π ( x k )  E J h ln  1 +  tanh β J 2  K Y k =1 x k i + ¯ K ¯ C X C P C ( C )  C Y c =1 Z 1 − 1 d ˆ x c ˆ π ( ˆ x c )  ln  X σ = ± 1 C Y c =1 [1 + σ ˆ x c ] i , (12) where the parameters are determined by the saddle-p oint equations obtained b y calculating functional v ariations: π ( x ) = X C C ¯ C P C ( C )  C − 1 Y c =1 Z 1 − 1 d ˆ x c ˆ π ( ˆ x c )  δ  x − ta nh h C − 1 X c =1 tanh − 1 ˆ x c i , (13) ˆ π ( ˆ x ) = X K K ¯ K P K ( K )  K − 1 Y k =1 Z 1 − 1 dx k π ( x k )  E J h δ  ˆ x −  tanh β J 2  K − 1 Y k =1 x k i , (14) with ¯ K = P K K P K ( K ) and ¯ C = P C C P C ( C ) (See the outline o f the deriv a tion in App endix A). W e can obtain the distortio n, whic h is r epro duction errors, u RS ( β ) = ∂ [ β f RS ( β )] / ∂ β and the replica symmetric (RS) en trop y s RS ( β ) = β [ u RS ( β ) − f RS ( β )]. F or arbitra ry P K ( K ), P C ( C ) a nd β , π ( x ) = δ ( x ) and ˆ π ( ˆ x ) = δ ( ˆ x ) are alwa ys solutions of the saddle-p o in t equations (13) and (14). These a re correspo nd the paramagnetic solution. The paramag netic free-energy , internal energy and entrop y are giv en b y f P ARA ( β ) = 1 2 − 1 β ln cosh β 2 − R β ln 2, u P ARA ( β ) = 1 2 − 1 2 tanh β 2 and s P ARA ( β ) = ln cosh β 2 − β 2 tanh β 2 + R ln 2, resp ective ly . Ho wev er, this R S solution tak es negativ e en trop y while R ln 2 < β 2 tanh β 2 − ln cosh β 2 . Esp ecially , when the inv erse temp erature β → ∞ , the RS en tropy b ecomes s RS ( β ) = ( R − 1) ln 2. This means w e ha ve to lo ok f or the true solution b ey ond the RS a nsatz f o r R ≤ 1 . 4.2. One-step r eplic a symmetry br e aking solution The rep lica symmetric breaking (RSB) theory for sparse s ystems is still under dev elopmen t [20 , 21, 22, 23, 24, 25]. Therefore, as a first approach we intro duce the frozen RSB to pro duce a solution with non-negat ive en t rop y [6, 12, 13]. The frozen RSB metho d is a limited v ersion o f full one-step RSB (1RSB) and includes the RS metho d a s a sp ecial case. In this 1RSB sc heme, n replicas ar e divided in to n/m groups whic h con tain m replicas eac h. The symmetry breaking parameter m w as found to be m = β g /β , where β g is a inv erse temp erature at whic h t he replica symmetric entrop y T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 6 0 0.2 0.4 0.6 0.8 1 0 0.1 0.2 0.3 0.4 0.5 R D (a) 0 1 2 3 4 5 0 0.2 0.4 0.6 0.8 1 β g R (b) Figure 1 . Example of n umer ical solutions for finite connectivity systems with P K ( K ) = δ K, 2 and P C ( C ) = ˜ P C ( C ). (a) Rate distortion p erformance for r = 0 . 3 , 0 . 4 , · · · , 0 . 9 (squares). The solid line denotes the rate distortion perfo rmance in large ¯ K and ¯ C limit, which coincides with the Shannon b ound. (b) Inv erse temp er a ture β g for for r = 0 . 3 , 0 . 4 , · · · , 0 . 9 (squares). T he so lid line denotes the inverse temp erature β c , which is defined by s P ARA ( β c ) = 0 . v anishes, i.e., s RS ( β g ) = 0 (See App endix B ). This 1 RSB sche me g ives the exact solution for the random energy mo del (REM) [6, 26]. F or β > β g , the 1RSB free energy b ecomes f 1 RS B ( β ) = f RS ( β g ). It can b e ragarded as a constant with resp ect to the in v erse temp erature β . W e assume that the 1RSB sc heme is enough go o d to appro ximate the solution ev en if ¯ K and ¯ C are finite. Under this assumption, the distortion D is simply giv en b y D = lim β →∞ u 1 RS B ( β ) = u RS ( β g ). 5. Results and discussion 5.1. Basic r esults In large ¯ K and ¯ C limit, there are no other solutions except π ( x ) = δ ( x ) a nd ˆ π ( ˆ x ) = δ ( ˆ x ) for t he saddle-p oin t equations. W e then found the relationship R = 1 − h 2 ( D ) , (15) from s RS ( β g ) = ln cosh β g 2 − β g 2 tanh β g 2 + R ln 2 = 0 and D = u RS ( β g ) = 1 2 − 1 2 tanh β g 2 . In finite ¯ K a nd ¯ C case, the solutions π ( x ) = δ ( x ) and ˆ π ( ˆ x ) = δ ( ˆ x ) a lso exist, but these are no longer stable [6]. W e hav e to solve the equations (13) and (14) numerically . W e c ho ose the prop er v alue of the in vers e temp erature β g whic h giv es s RS ( β g ) = 0 b y using the nu merical results of the saddle-p o in t equations. Since the distortion D can b e ev a lua ted from D = u RS ( β g ), w e can also obtain the relation b et w een the co de rate R = ¯ K / ¯ C and the distortion D in the finite connectiv ity systems . As one of the simplest examples to treat the arbitrary co de rate, we here in tro duce degree distributions P K ( K ) = δ K, 2 and P C ( C ) = 7 r − 2 5 r δ C, 2 + 2(1 − r ) 5 r δ C, 7 ( ≡ ˜ P C ( C )), whic h are v alid for 2 7 ≤ r ≤ 1. Here, δ m,n denotes Kro nec k er’s delta taking 1 if m = n and 0 otherwise. In this case, we can adjust the co de ra t e R (= r ) via the para meter r . T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 7 0 2 4 6 8 10 -1 -0.5 0 0.5 1 π (x) 0 5 10 15 20 -1 -0.5 0 0.5 1 π (x) ^ ^ (a) P K ( K ) = δ K, 2 , P C ( C ) = δ C, 4 0 2 4 6 8 10 -1 -0.5 0 0.5 1 π (x) 0 5 10 15 20 -1 -0.5 0 0.5 1 π (x) ^ ^ (b) P K ( K ) = δ K, 2 , P C ( C ) = P C ( C ) 0 50 100 150 200 -1 -0.5 0 0.5 1 π (x) 0 50 100 150 200 -1 -0.5 0 0.5 1 π (x) ^ ^ (c) P K ( K ) = P K ( K ), P C ( C ) = δ C, 4 Figure 2. Snapsho ts o f the o rder functions π ( x ) and ˆ π ( ˆ x ). (b) a regula r case, (a) a chec k-reg ular a nd bit-irregular ca se, a nd (c) a chec k - irregula r and bit-regular case. W e apply the Mon te-Carlo inte g ration to solv e the saddle-p oint equations. Fig ure 1 (a) sho ws the ra te-distortion p erformance of this system. Figure 1 (b) show s the in verse temp erature β g , whic h is a ro ot of the replica symmetric en trop y s RS ( β g ) = 0. 5.2. Some typic al irr e gular c onstructions W e next apply some degree distributions as typical examples. It should b e noted that these distributions discuss ed here are not optimized but heuristically c hosen. All three examples ha v e the co de rate R = 1 / 2 . Firstly , w e consider a regular case characterize d as P K ( K ) = δ K, 2 and P C ( C ) = δ C, 4 . Figure 2 (a) show s stable solutions π ( x ) and ˆ π ( ˆ x ) of the saddle-p oin t equations for this case. It can b e confirmed that the functions π ( x ) and ˆ π ( ˆ x ) ar e bro ad in shap e. In this case, the distortion b ecomes D = 0 . 1 1 6. The Shannon b ound is D S B = 0 . 1100. Secondly , W e tr eat a c hec k-regular and bit-irregular case whose degree distributions are defined as P K ( K ) = δ K, 2 and P C ( C ) = P C ( C ), where P C ( C ) = 0 . 0 4 δ C, 1 + 0 . 15 δ C, 2 + 0 . 22 δ C, 3 + 0 . 22 δ C, 4 + 0 . 18 δ C, 5 + 0 . 11 δ C, 6 + 0 . 08 δ C, 7 . (16) This P C ( C ) is a ro ugh appro ximatio n of the P o issonian distribution e − λ λ C − 1 / ( C − 1)! with λ = 3. The distort io n is D = 0 . 11 5 for this case. It represen ts an ensem ble whic h T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 8 ha ve at least one non-zero elemen t in eac h r ow. In t he c hec k-regular case, when we c ho ose the non-zero elemen ts randomly , there exists some columns whos e elemen ts are all zero. In suc h a situation, the co de rate esse ntially b ecomes small. Lastly , for a c hec k- irregular and bit-regular case, w e apply P K ( K ) = P K ( K ) and P C ( C ) = δ C, 4 , where P K ( K ) = 0 . 36 δ K, 1 + 0 . 36 δ K, 2 + 0 . 20 δ K, 3 + 0 . 08 δ K, 4 . (17) This P K ( K ) is a ro ugh approx imat io n of the Poiss onia n distribution e − λ λ K − 1 / ( K − 1)! with λ = 1. The reason wh y w e consider this distribution is same to P C ( C ). In this case, the distortion b ecomes D = 0 . 115. Thes e three kinds of distributions giv e almost same distortio n. Figure 2 (b) and (c) show stable solutions for these irregular cases. It can b e confirmed that the distribution π ( x ) and ˆ π ( ˆ x ) b ecome a little bit narrow than the regular case. It is considered that the distortion can b ecome small due to this. 6. Conclusions W e ev alua te t ypical p erformance of LDGM co des with irregular bit a nd che ck degree distributions b y applying the replica metho d under 1RSB ansatz. Our result shows that w e can use an arbitrary co de rate. It might b e p ossible to in v estigate suboptimal irregular degree distributions b y using the hill- climbing approac h similar to the case of the dens it y ev o lution [27, 28]. In the practical p oin t of view, it m ust b e imp ortant to ev aluate some po lynomial time encoding algorithms with arbitrary degree distributions. It should b e noted that the analysis addressed here is based on an exact calculation o f the enco der’s definition. Therefore it can b e considered that the distortion obtained b y this analysis pro vides the theoretical limit for giv en c heck and bit degree distributions. Recen tly , the cav ity metho d w as in tro duced to ev aluate the typic a l p erformance [7]. Since the cavit y method do es not need the replica t r ic k, it might b e able to a v oid some assumptions. Applying the ca vit y metho d to this problem is also imp ortant and is a part of o ur future w ork. Ac knowledgm ents The author w ould like to thank Ka zutak a Nak am ura, T atsuto Mura y a ma and Y oshiyuki Kabashima for their helpful commen ts. W e also t ha nk T or u Y ano for giving us v aluable preprin ts. This w ork w as partially supp orted b y a Gran t-in-Aid for Encouragemen t of Y oung Scien tists (B) No. 187002 3 0 from the Ministry of Education, Culture, Sp or ts, Science a nd T ec hnology of Japan. T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 9 App endix A. Deriv ation of replica symmetric free energy W e assume that the ev en t D µ i 1 , ··· ,i K µ = 1 o ccurs indep enden tly for ev ery row µ . W e then ha ve P ( D µ i 1 , ··· ,i K µ = 1) = p µ , (A.1) P ( D µ i 1 , ··· ,i K µ = 0) = 1 − p µ , (A.2) where P ( · · · ) denotes the probabilit y of the ev en t ( · · · ) and p µ = ( N K µ ) − 1 ≃ K µ ! / N K µ . In tro ducing the constraint concerning the column ( 8) by using Dirac’s delta function, the ense mble a v erage ov er the co des is represen ted as E A [( · · · )] =  X { K µ } M Y µ =1 P K ( K µ )  X { C i } N Y i =1 P C ( C i )  × 1 N D E D hn N Y i =1 δ  M X µ =1 X h i 1 = i,i 2 , ··· ,i K µ i D µ i 1 = i,i 2 , ··· ,i K µ ; C i o ( · · · ) i , =  X { K µ } M Y µ =1 P K ( K µ )  X { C i } N Y i =1 P C ( C i )  × 1 N D E D hn N Y i =1 I d Z i 2 π i 1 Z C i +1 i N Y µ =1 Y h i 1 = i,i 2 , ··· ,i K µ i Z D µ i 1 = i,i 2 , ··· ,i K µ i o ( · · · ) i , (A.3) where E D denotes t he a ve ra ge ov er the connectivit y parameter. Observing t hat P h i 1 , ··· ,i K µ i ( · · · ) = 1 K µ ! ( P i ( · · · )) K µ for large N , the normalization constan t N D is giv en b y N D = E D h N Y i =1 δ  M X µ =1 X h i 1 = i,i 2 , ··· ,i K µ i D µ i 1 = i,i 2 , ··· ,i K µ ; C i i = ( N ¯ C )! N N ¯ C N Y i =1 C i ! . (A.4) T o ev aluate t he f ree energy , w e calculate t he r eplicated partitio n function: E A , J [ Z ( β ) n ] = e − nM β 2 E A , J  X s 1 , ··· , s n exp h β 2 M X µ =1 X h i 1 , ··· ,i K µ i D µ i 1 , ··· ,i K µ J µ n X α =1 s α i 1 · · · s α i K µ oi  = e − nM β 2  X { K µ } M Y µ =1 P K ( K µ )  X { C i } N Y i =1 P C ( C i )  × 1 N D  N Y i =1 I d Z i 2 π i 1 Z C i +1 i  M Y µ =1  p µ X h i 1 , ··· ,i K µ i (cosh β 2 ) n Z i 1 · · · Z i K µ + p µ n X m =1 X h α 1 , ··· ,α m i (cosh β 2 ) n E J h (tanh β J 2 ) m i T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 10 × X h i 1 , ··· ,i K µ i ( Z i 1 s α 1 i 1 · · · s α m i 1 ) · · · ( Z i K µ s α 1 i K µ · · · s α m i K µ )  . (A.5) W e next in tro duce order parameters q α 1 , ··· ,α m and q 0 , defined by q α 1 , ··· ,α m = 1 N N X i =1 Z i s α 1 i · · · s α m i , (A.6) q 0 = 1 N N X i =1 Z i . (A.7) Using t he F ourier expression of the Dirac delta function, we find E A , J [ Z ( β ) n ] = e − nM β 2  Z dq 0 d ˆ q 0 2 π  Y h α 1 i Z dq α 1 d ˆ q α 1 2 π  · · ·  Y h α 1 , ··· ,α n i Z dq α 1 , ··· ,α n d ˆ q α 1 , ··· ,α n 2 π  ×  X { K µ } M Y µ =1 P K ( K µ )  X { C i } N Y i =1 P C ( C i )  1 N D  N Y i =1 I d Z i 2 π i 1 Z C i +1 i  × exp h − N n q 0 ˆ q 0 + · · · + X h α 1 , ··· ,α n i q α 1 , ··· ,α n ˆ q α 1 , ··· ,α n o + ˆ q 0 N X i =1 Z i + · · · + X h α 1 , ··· ,α n i ˆ q α 1 , ··· ,α n N X i =1 Z i s α 1 i · · · s α n i i × M Y µ =1  T 0 q K µ 0 + n X m =1 X h α 1 , ··· ,α m i T m ( q α 1 , ··· ,α m ) K µ  , (A.8) with T m = (cosh β 2 ) n E J [(tanh β J 2 ) m ]. T o pro ceed further, w e in tro duce the replica- symmetric (RS) assumption: q α 1 , ··· ,α m = q Z 1 − 1 dxπ ( x ) x m , (A.9) ˆ q α 1 , ··· ,α m = ˆ q Z 1 − 1 d ˆ x ˆ π ( ˆ x ) ˆ x m , (A.10) where π ( x ) ≥ 0, ˆ π ( ˆ x ) ≥ 0 and R 1 − 1 dxπ ( x ) = R 1 − 1 d ˆ x ˆ π ( ˆ x ) = 1. This assumption means that the order parameters dep end only on the n um b er of indices . W e write the replica symmetric partition function as Z RS ( β ). Using the integral f orm o f the Dira c’s delta function, w e obtain E A , J [ Z RS ( β ) n ] = extr π , ˆ π, q , ˆ q e − nM β 2 N D  X { K µ } M Y µ =1 P K ( K µ )  X { C i } N Y i =1 P C ( C i )  ×  N Y i =1 n ˆ q C i C i !  C i Y c =1 Z 1 − 1 d ˆ x c ˆ π ( ˆ x c )  X σ = ± 1 C i Y c =1 (1 + σ ˆ x c )  n o T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 11 × exp h − N q ˆ q Z 1 − 1 dxπ ( x ) Z 1 − 1 d ˆ x ˆ π ( ˆ x )(1 + x ˆ x ) n i × M Y µ =1  q K µ (cosh β 2 ) n  K µ Y k =1 Z 1 − 1 dx k π ( x k )  E J h 1 + (tanh β J 2 ) K µ Y k =1 x k  n i . (A.11) Finally , substituting this into (11) a nd t a king the limit n → 0, w e arriv e at (12). The saddle-p oin t equations (13) and (14) are simply o btained as the extremization condition of (1 2). App endix B. One-step replica symmetry breaking solution W e follow the calculation of the reference [13]. W e assume that t he space o f configuration is divided in n/m groups with m iden tical configurations in eac h. 1 N s α · s β = ( 1 , if α and β are in the same group q , o therwise . (B.1) Using this ergo dicity breaking assumption, the 1R SB replicated partition f unction b ecomes E A , J [ Z 1 RS B ( β ) n ] | ( B.1 ) = E A , J  X s e − β H ( s , J )  n      ( B.1 ) = E A , J  X s e − β m H ( s , J )  n/m  = E A , J [ Z RS ( β m ) n/m ] . (B.2) Then w e o btain the 1R SB free energy as f 1 RS B ( β ) = − 1 β E A , J [ln Z 1 RS B ( β )] = − 1 β  ∂ ∂ n E A , J [ Z 1 RS B ( β ) n ]     n =0 = − 1 β m  ∂ ∂ ( n/m ) E A , J [ Z RS ( β m ) n/m ]     n/m =0 = − 1 β m E A , J [ln Z RS ( β m )] = f RS ( β m ) (B.3) The symmetry br eaking parameter m should b e determined to extremize the 1RSB free energy as ∂ ∂ m f 1 RS B ( β ) = 0 . (B.4) The left hand side of this condition b ecomes ∂ ∂ m f 1 RS B ( β ) = − ∂ ∂ m 1 β m E A , J [ln Z RS ( β m )] T ypic al p erformanc e of irr e gular LDGM c o des for lossy c om pr ession 12 = − 1 m  ∂ [ E A , J [ln Z RS ( β m )]] ∂ ( β m ) − 1 β m E A , J [ln Z RS ( β m )]  = 1 m  ∂ [( β m ) f RS ( β m )] ∂ ( β m ) − f RS ( β m )  = 1 β m 2 s RS ( β m ) . (B.5) Namely , t he condition (B.4) is equiv alent to s RS ( β m ) = 0 . Therefore, the symmetry breaking parameter is giv en b y m = β g /β with s RS ( β g ) = 0. References [1] T. M. Cover a nd J. A. Thomas, Elements of Information The ory, 2nd e d. , John Wiley a nd So ns Inc., 20 06. [2] I. Csisz´ ar a nd J. K¨ orner, Information The ory: Co ding The or ems for Discr ete Memoryless Systems , Academic P ress, 1 981. [3] Y. Matsunag a and H. Y amamoto, IEEE T r ans. Inform. The ory , vol. 49, 2225, 2 003. [4] T. Muray a ma, Phys. R ev. E , vol. 69 , 0351 05(R), 200 4. [5] M. J. W ainwrigh t and E. Ma nev a, Pr o c. Int’l. Symp o. Info. The ory , 14 93, 2005 . [6] T. Muray a ma and M. Ok ada, J . Phys. A: Math. Gen. , vol. 36 , 1112 3, 200 3. [7] S. Cilib erti, M. M´ ezard, J Stat. Me ch. , vol. 3, 58, 2006 . [8] S. Cilib erti, M. M´ ezard and R. Z e cchina, Complex s ystems metho ds , vol. 3, 58, 200 6 . [9] E . Martinian and M. J. W ainwrigh t, Workshop on Info. The ory and its A pplic ations , 2006 . [10] A. G. Dimakis, M. J. W ainwrigh t and K. Ra mchandran, In fo. The ory Workshop , 650, 2005 . [11] M. J. W ainwright, IEEE Signal Pr o c essing Magazine , 47, Sept. 2007 . [12] Y. K a bashima a nd D. Saad, Eur ophys. L ett. , vol. 4 5, 9 7, 19 99. [13] R. Vicente, D. Saa d and Y. Kabashima , J. Phys. A: Math. Gen. , vol. 33, 6 527, 200 0. [14] K . Nak amura, Do ctor al thesis , T okyo Institute of T echnology , 200 3. [15] K . Mimura a nd M. Ok ada, Phys. R ev. E , vol. 74 , 0261 08, 200 6 . [16] W. K r auth a nd M. M´ ezard, J. Phys. F r anc e , vol. 50, 3057, 19 8 9. [17] T . T anak a and D. Saad, T e ch. Re p ort , unpublished 20 03. [18] T . Y ano, T. T anak a and D. Saa d, Pr o c. of SIT A2007 , 41 .4, 200 7. [19] T . Y ano, Do ctor al thesis , Keio Univ., (preprint), 2008 . [20] K . Y. W ong and D. Sherringto n, J. Phys. A: Math. Gen. , vol. 21, L4 59, 1 9 88. [21] C . de Dominicis and Y. Y. Go ldschmidt , J . Phys. A: Math. Gen. , vol. 22 , L775 , 1989 . [22] P . Lai and Y. Y. Goldschmidt, J . Phys. A: Math. Gen. , vol. 23 , 332 9, 199 0. [23] Y. Y. Goldschmidt and C. de Deminicis, Phys. R ev. B , vol. 41, 2184, 1990. [24] R. Mo nasson, J . Phys. A: Math. Gen. , vol. 31, 513, 1998 . [25] G. Parisi and F. T ria , Eur o. Phys. J. B , vol. 30, 4 , 1434, 20 02. [26] B . Derr ida, Phys. R ev. B , vol. 24, 2613, 1981. [27] T . J. Richardson and R. L. Urbanke, IEEE T ra n s. Info. The ory , vol. 47, 2, 599, 2 0 01. [28] T . J . Richardson, M. A. Sho krollahi and R. L . Urba nke, IEEE T r ans. Info. The ory , vol. 47, 2, 619 , 2001.