Constructing Linear Codes with Good Joint Spectra

Constructing Linear Codes with Good Joint S pectra Shengtian Y ang, Y an Chen, Thomas Honold, Zhaoyan g Zhang, and Peiliang Qiu Departmen t of I nform ation Science & E lectronic En gineerin g Zhejiang Un iv ersity Hangzho u, Zhejiang 3 1002 7, China { yang shengtian, qiupl41 8, honold , ning ming, q iupl } @zju.ed u.cn Abstract — The problem of findin g good lin ear codes f or joint source-channel coding (JSCC) is inv estigated in this paper . By the code-spectrum approach, it has b een prov ed in the authors’ prev ious pap er that a good li near code f or the authors’ JSCC scheme is a code with a goo d joint spectrum, so the main task in t his p aper is to construct linear codes with goo d joint spectra. First, the code-spectru m approach is developed further to facilitate the calculation of spectra. Second, some general principles fo r constructing good linear codes are p resented. Finally , we propose an explicit construction of linear codes with good joint sp ectra based on low density parity check (LDPC) codes and low density generator matrix (LDGM) codes. I . I N T R O D U C T I O N A lot of resear ch on pr actical design s of lossless joint source-ch annel coding (JSCC) b ased o n line ar co des h av e been done for specific co rrelated sources and multiple-ac cess channels (MACs ), e.g., correlated sources over separated noisy channels ( e.g., [1 ]), cor related sou rces over add itiv e white Gaussian noise (A WGN) MA Cs (e.g., [2] ), correlated sources over Rayleigh fading MACs (e. g., [ 3]). Howe ver, for tr ans- mission of a rbitrary correlated sou rces over arbitra ry MACs , it is still not clear how to co nstruct an optimal lo ssless JSCC scheme. In [ 4], we pr oposed a lossless JSCC sch eme based o n linear cod es fo r MA Cs, which proved to be op timal if good linear codes an d good conditio nal pro babilities are chosen. Figure 1 illustrates the mechanism of ou r scheme in detail. Using the code-spectr um approach established in [4], we found that a go od linear code f or ou r JSCC scheme is a cod e with a good joint spectrum . Hence, to design a lossless JSCC scheme in practice, a big problem is how to constru ct linear codes with good joint spectra. In this pape r , we will inv estigate the problem in depth and give an explicit construction of lin ear codes w ith good jo int spectr a based on sparse matrices. In the sequ el, symbo ls, real v ariables and determ inistic mapping s are denoted by lowercase letters. Sets a nd rando m elements are denoted by cap ital letters, and the e mpty set is denoted by ∅ . Alph abets are den oted by script capital letters. All logar ithms are taken to the natur al b ase e an d deno ted by ln . The compo sition o f th e fu nctions f an d g is denoted by f ◦ g , wh ere ( f ◦ g )( x ) △ = f ( g ( x )) . Th e indicato r f unction is denoted by 1 { ·} . The card inality of a set A is denoted by | A | . 1 This work was supported by Zhejiang Provinci al Natural Science Foun- dation of China (No. Y106068), by the National Natural Science Founda tion of China (No. 60602023, 6077209 3), and by the National High T echnol- ogy Research and Dev elopmen t Program of China (No. 2006AA01Z273, 2007AA01Z257). For any r andom eleme nts F and G in a common measurable space, the e quality F d = G m eans that F and G ha ve th e same probab ility distribution. I I . B A S I C S O F T H E C O D E - S P E C T RU M A P P R O AC H Before in vestigating the problem of constru cting good linear codes, we first need to briefly intro duce ou r “cod e-spectrum” approa ch established in [4], which may be regarded as a generalizatio n of the weig ht-distribution approa ch (e.g., [5 ]). Let X and Y be two finite ( additive) abelian gr oups. W e define a lin ear cod e as a homo morph ism f : X n → Y m , i.e., a map satisfyin g f ( x n 1 + x n 2 ) = f ( x n 1 ) + f ( x n 2 ) ∀ x n 1 , x n 2 ∈ X n where X n and Y m denote the n -fold dire ct p roduct o f X and the m - fold direct pro duct of Y , respectively , and x n denotes any seq uence x 1 x 2 · · · x n in X n . W e also d efine the rate of a linear c ode f to b e the ratio n/m , and denote it b y R ( f ) . Note that any p ermutation (or interleav er) σ n on n letters can be regarded as an au tomorp hism on X n . W e de note by Σ n a u niformly distributed rand om perm utation on n letters. W e tacitly assum e that different r andom per mutations o ccurring in the same expre ssion are independ ent. Next, we introdu ce the concept of types [6]. The typ e of a sequenc e x n in X n is the empirical distribution P x n on X defined b y P x n ( a ) △ = 1 n n X i =1 1 { x i = a } ∀ a ∈ X For a (prob ability) distribution P on X , the set of sequen ces of typ e P in X n is denoted by T n P ( X ) . A distribution P on X is called a ty pe of sequ ences in X n if T n P ( X ) 6 = ∅ . W e denote by P ( X ) th e set o f all distributions on X , an d d enote by P n ( X ) the set of all possible typ es of sequ ences in X n . Now , we introdu ce the spectrum , the most important c oncept in the code -spectrum a pproac h. The spectru m of a non empty set A ⊆ X n is the empir ical distribution S X ( A ) on P ( X ) defined b y S X ( A )( P ) △ = |{ x n ∈ A | P x n = P } | | A | ∀ P ∈ P ( X ) . Analogou sly , the joint spectrum of a no nempty set B ⊆ X n × Y m is the emp irical d istribution S X Y ( B ) on P ( X ) × P ( Y ) M u l t i p l e - A c c e ss C h a n n e l I n p u t S o u r c e n V R a n d o m I n t e r l e a v e r n Σ L i n e a r C o d e n F R a n d o m I n t e r l e a v e r l Σ R a n d o m V e c t o r l U + Q u a n t i z a t i o n ( , ) n n q V ⋅ Fig. 1. The proposed linear codes base d lossless joint source-channe l encodi ng scheme of eac h enco der for multiple -acce ss channels defined b y S X Y ( B )( P , Q ) △ = |{ ( x n , y m ) ∈ B | P x n = P, P y m = Q } | | B | for all P ∈ P ( X ) , Q ∈ P ( Y ) . Fu rthermo re, we define the mar ginal spectra S X ( B ) , S Y ( B ) as the marginal distrib utions of S X Y ( B ) , that is, S X ( B )( P ) △ = X Q ∈P ( Y ) S X Y ( B )( P , Q ) S Y ( B )( Q ) △ = X P ∈P ( X ) S X Y ( B )( P , Q ) . Please note that the su mmation in the d efinition of S X ( B )( P ) (or S Y ( B )( Q ) ) is taken over an infin ite set P ( Y ) (or P ( X ) ), but is actually over a finite set because S X Y ( B )( P , Q ) = 0 for any ( P , Q ) satisfying P ∈ P ( X ) \P n ( X ) or Q ∈ P ( Y ) \P m ( Y ) . W e d efine the con ditiona l spectr a S Y |X ( B ) , S X |Y ( B ) as the con ditional distributions of S X Y ( B ) , that is, S Y |X ( B )( Q | P ) △ = S X Y ( B )( P , Q ) S X ( B )( P ) ∀ P satisfying S X ( B )( P ) 6 = 0 S X |Y ( B )( P | Q ) △ = S X Y ( B )( P , Q ) S Y ( B )( Q ) ∀ Q satisfying S Y ( B )( Q ) 6 = 0 . Then natur ally , for any given fu nction f : X n → Y m , we can define its jo int spectrum S X Y ( f ) , fo rwar d condition al spectrum S Y |X ( f ) , backwar d cond itional spectrum S X |Y ( f ) , and image spectrum S Y ( f ) as S X Y (rl( f )) , S Y |X (rl( f )) , S X |Y (rl( f )) , and S Y (rl( f )) , respectively , wher e rl( f ) is th e r ela tion defined by { ( x n , f ( x n )) | x n ∈ X n } . In this case, th e forward con ditional spec trum is given by S Y |X ( f )( Q | P ) = S X Y ( f )( P, Q ) S X ( X n )( P ) . If f is a linear code, we further define its kernel spectrum as S X (ker f ) , where ker f △ = { x n | f ( x n ) = 0 m } . In this case, we have S Y ( f ) = S Y ( f ( X n )) since f is a ho momor phism acco rding to the de finition of linear c odes. The above definitions can be e asily extended to m ore gen- eral cases. For example, we ma y consid er the jo int spectru m S X Y Z ( C ) of a set C ⊆ X n × Y m × Z l , or consider the joint spectru m S X 1 X 2 Y 1 Y 2 ( g ) o f a fun ction g : X n 1 1 × X n 2 2 → Y m 1 1 × Y m 2 2 . A series of properties r egarding th e spectrum of c odes were proved in [4]. Reader s m ay refer to [4] for the details. Some results are listed below for easy reference. Pr opo sition 2.1: For all P ∈ P n ( X ) an d P i ∈ P n i ( X i ) ( 1 ≤ i ≤ m ), we have S X ( X n )( P ) =  n nP  |X | n , S X 1 ···X m ( m Y i =1 A i )( P 1 , · · · , P m ) = m Y i =1 S X i ( A i )( P i ) , where  n nP  △ = n ! Q a ∈X ( nP ( a ))! and A i ⊆ X n i i ( 1 ≤ i ≤ m ). Pr opo sition 2.2: For any g i ven ran dom f unction F : X n → Y m , we h ave Pr { ˜ F ( x n ) = y m } = |Y | − m α ( F )( P x n , P y m ) (1) for any x n ∈ X n , y m ∈ Y m , where ˜ F △ = Σ m ◦ F ◦ Σ n (2) and α ( F )( P, Q ) △ = E [ S X Y ( F )( P, Q )] S X Y ( X n × Y m )( P, Q ) = E [ S Y |X ( F )( Q | P )] S Y ( Y m )( Q ) . (3) Pr opo sition 2.3: For any giv en linear code f : X n → Y m , we hav e α ( f )( P 0 n , P y m ) = |Y | m 1 { y m = 0 m } . (4) If both X and Y are the Galo is field F q , we define a particu lar random linear code F RLC q,n, m : F n q → F m q by x n 7→ A m × n · x n , where x n represents an n -dimensiona l column vector , and A m × n denotes a ran dom matrix with m rows and n columns, each entr y indep endently taking v alues in F q accordin g to a unifor m distribution on F q . (Note that for eac h realization of A m × n , we then ob tain a corr espondin g realization o f F RLC q,n, m . Such a ra ndom con struction h as alrea dy been ad opted in [ 7, Section 2 .1], [ 8], etc.) Th en we have Pr { ˜ F RLC q,n, m ( x n ) = y m } = Pr { F RLC q,n, m ( x n ) = y m } = q − m (5) for all x n ∈ F n q \{ 0 n } an d y m ∈ F m q , or eq uiv a lently α ( F RLC q,n, m )( P, Q ) = 1 (6) for all P ∈ P n ( F q ) \{ P 0 n } and Q ∈ P m ( F q ) . I I I . S O M E N E W R E S U LT S A B O U T C O D E S P E C T R A In order to e valuate the perf ormance of a linear code, we need to calculate or estimate its spectrum. Howe ver, the results established in [4 ] are still n ot enoug h fo r this purp ose. So in this section, we will presen t some new results to facilitate the c alculation of spectra. All the pr oofs ar e easy and he nce omitted here. First, we proved the f ollowing two propo sitions, wh ich imply th at any codes (o r fun ctions) may be regard ed as condition al probab ility distributions. Such a viewpoint is very helpful w hen calculating the spectru m of a complex code consisting of m any simple codes. Pr opo sition 3.1: For any rando m fun ction F : X n → Y m and any x n ∈ X n , we h ave Pr { ( F ◦ Σ n )( x n ) ∈ T m Q ( Y ) } = E [ S Y |X ( F )( Q | P x n )] . (7) Pr opo sition 3.2: For any two r andom fun ctions F : X n → Y m and G : Y m → Z l , an d any O ∈ P n ( X ) , Q ∈ P l ( Z ) , we have E [ S Z |X ( G ◦ Σ m ◦ F )( Q | O )] = X P ∈P m ( Y ) E [ S Y |X ( F )( P | O )] E [ S Z |Y ( G )( Q | P )] . Second, let us de velop a ge nerating f unction method for the calculations of spe ctra. For any set A ⊆ X n , we define the generating fu nction G ( A ) of its sp ectrum to be G ( A )( u ) △ = X P ∈P n ( X ) S X ( A )( P )( u nP ) ⊗ where u is a map fr om X to C (the set of com plex n umbers) or u ∈ C X , and f or any u, v ∈ C X , we d efine ( ru )( a ) △ = r u ( a ) ∀ r ∈ C , a ∈ X , ( u v )( a ) △ = u ( a ) v ( a ) ∀ a ∈ X , ( u ) ⊗ △ = Y a ∈X u ( a ) . Also no te that P ∈ P n ( X ) ⊆ C X . An alogously , for a ny set B ⊆ X n × Y m , we define the g enerating fu nction of its joint spectrum as G ( B )( u , v ) △ = X P ∈P n ( X ) ,Q ∈P m ( Y ) S X Y ( B )( P , Q )( u nP ) ⊗ ( v mQ ) ⊗ , where u ∈ C X , v ∈ C Y . This in particular defines G ( f ) △ = G (rl( f )) for any fun ction f : X n → Y m . Based on th e above defin itions, we proved the following proper ties. Pr opo sition 3.3: For any two sets A 1 ⊆ X n 1 and A 2 ⊆ X n 2 , we have G ( A 1 × A 2 )( u ) = G ( A 1 )( u ) · G ( A 2 )( u ) . (8) For any two sets A 1 ⊆ X n and A 2 ⊆ Y m , we h ave G ( A 1 × A 2 )( u, v ) = G ( A 1 )( u ) · G ( A 2 )( v ) . (9) For any two sets B 1 ⊆ X n 1 × Y m 1 and B 2 ⊆ X n 2 × Y m 2 , we hav e G ( B 1 × B 2 )( u, v ) = G ( B 1 )( u, v ) · G ( B 2 )( u, v ) . (10) Cor ollary 3.1 : G ( X n )( u ) =  ( u ) ⊕ |X |  n , where ( u ) ⊕ △ = X a ∈X u ( a ) . Cor ollary 3.2 : For any two functions f 1 : X n 1 → Y m 1 and f 2 : X n 2 → Y m 2 , we have G ( f 1 ⊙ f 2 )( u, v ) = G ( f 1 )( u, v ) · G ( f 2 )( u, v ) , (11) where f 1 ⊙ f 2 is the m ap from X n 1 + n 2 to Y m 1 + m 2 defined by ( f 1 ⊙ f 2 )( x n 1 + n 2 ) △ = f 1 ( x 1 ··· n 1 ) f 2 ( x ( n 1 +1) ··· ( n 1 + n 2 ) ) for all x n 1 + n 2 ∈ X n 1 + n 2 . I V . G E N E R A L P R I N C I P L E S F O R C O N S T RU C T I N G L I N E A R C O D E S W I T H G O O D J O I N T S P E C T R A In this section, we will investigate som e g eneral p roblems for constructing linear codes with good joint spectra . At first, we n eed to intr oduce some co ncepts of goo d linear codes. Acc ording to [4, T able I ], a sequ ence of δ - asymptotically go od (ran dom) linear co des F n : X n → Y m n for JSCC is a seq uence of linear co des whose joint spectra satisfy lim sup n →∞ max P ∈P n ( X ) \{ P 0 n } , Q ∈P m n ( Y ) 1 n ln E [ S X Y ( F n )( P, Q )] S X Y ( X n × Y m n )( P, Q ) ≤ δ. And fo r compa rison, a seq uence o f δ -asympto tically good (rando m) linear codes F n : X n → Y m n for channe l co ding is a sequence of linear codes whose image spectra satisfy lim sup n →∞ max Q ∈P m n ( Y ) \{ P 0 m n } 1 m n ln E [ S Y ( F n ( X n ))( Q )] S Y ( Y m n )( Q ) ≤ δ. When δ equals zero, the above cod es are then c alled asymp - totically good linear codes for JSCC and channel coding, respectively . From the linearity o f codes, it follows easily that δ - asymptotically good line ar codes f or JSCC are subsets o f δ R ( F ) -asymptotically go od linea r co des f or ch annel co ding, where R ( F ) △ = lim sup n →∞ R ( F n ) and F △ = { F n } ∞ n =1 . T hen naturally , o ur first pr oblem is: if a seq uence of asymptotically good line ar codes f n for chan nel coding is given, can we find a sequ ence of asy mptotically go od linear code s g n for JSCC such that g n ( X n ) = f n ( X n ) ? In oth er words ( assuming th at X = Y = F q ), if a seque nce of asym ptotically good cha nnel codes is given, can we cho ose a good sequen ce of g enerator matrices so that the linear co des are a symptotically go od for JSCC? When X = F q , the a nswer is p ositi ve, as a consequ ence of the following th eorem. Theor em 4.1 : For any linear code f : F n q → Y m , there exists a linear code g : F n q → Y m such th at g ( F n q ) = f ( F n q ) and S Y | F q ( g )( Q | P ) < S Y ( f ( F n q ))( Q ) 1 − q − 1 − q − 2 (12) for all P ∈ P n ( F q ) \{ P 0 n } , Q ∈ P m ( Y ) . Pr oof: [Sketch of Proo f] The main idea of th e proo f is to co nstruct a r andom linear code G △ = f ◦ F RLC q,n, n d = f ◦ Σ n ◦ F RLC q,n, n , where F RLC q,n, n is defined in Proposition 2.3. Then b y Prop osition 3.2, we have E [ S Y | F q ( G )( Q | P )] = S Y ( f ( F n q ))( Q ) for all P 6 = P 0 n and Q . Th is tog ether with Proposition 4.1 ( see below) the n co ncludes the the orem. Pr opo sition 4.1: L et rank( F ) be the r ank of the g enerator matrix of the line ar co de F : F n q → F m q . Then we h av e Pr { rank( F RLC q,n, m ) = m } = m Y i =1  1 − q i − 1 q n  (13) where F RLC q,n, m is define d in Proposition 2.3 an d m ≤ n . Furthermo re, we have m Y i =1  1 − q i − 1 q n  >  1 − q m − n − k q − 1  k Y i =1 (1 − q m − n − i ) , (14) where 1 ≤ k ≤ m . Let m = n and k = 1 , th en we have Pr { rank( F RLC q,n, n ) = n } > 1 − q − 1 − q − 2 . (15) Pr oof: The identity (13) is a well k nown result in probab ility theory . T o o btain a lower bound of the right h and side of (1 3), we have m Y i =1  1 − q i − 1 q n  = m − k Y i =1  1 − q i − 1 q n  k Y i =1  1 − q m − i q n  ≥  1 − m − k X i =1 q i − 1 q n  k Y i =1  1 − q m − i q n  =  1 − q m − k − 1 q n ( q − 1)  k Y i =1  1 − q m − i q n  >  1 − q m − n − k q − 1  k Y i =1  1 − q m − i q n  . This conclude s (14), an d ( 15) fo llows c learly . The above result do es give a possible way for con structing good linear codes for JSCC based on g ood channel codes. Howe ver, such a construction is somewhat difficult to im - plement in pr actice, because the ran dom generato r matrix of F RLC q,n, m is d ense. T hus, o ur seco nd pr oblem is h ow to construct linear cod es with go od joint spectra based on spa rse matric es so that known iterative enco ding and deco ding pro cedures have low complexity . The follo wing theorem gi ves one feasible solution. Theor em 4.2 : For a giv en sequenc e of sets { A n ⊆ P m n ( X ) \{ P 0 m n }} ∞ n =1 , if there exist two sequences of random linear codes F n : X n → X m n and G n : X m n → X l n satisfying F n ( X n \{ 0 n } ) ⊆ [ P ∈ A n T m n P ( X ) (16) and lim sup n →∞ max P ∈ A n ,Q ∈P l n ( X ) 1 n ln E [ S X |X ( G n )( Q | P )] S X ( X l n )( Q ) ≤ δ (17) respectively , where δ ≥ 0 , then we hav e lim sup n →∞ max O ∈P n ( X ) \{ P 0 n } ,Q ∈P l n ( X ) 1 n ln E [ S X |X ( G n ◦ Σ m n ◦ F n )( Q | O )] S X ( X l n )( Q ) ≤ δ. Pr oof: For all O ∈ P n ( X ) \{ P 0 n } and Q ∈ P l n ( X ) , and for any ǫ > 0 and sufficiently large n , we have S X |X ( G n ◦ Σ m n ◦ F n )( Q | O ) (a) = X P ∈P m n ( X ) E [ S X |X ( G n )( Q | P )] E [ S X |X ( F n )( P | O )] (b) = X P ∈ A n E [ S X |X ( G n )( Q | P )] E [ S X |X ( F n )( P | O )] (c) ≤ X P ∈ A n e n ( δ + ǫ ) S X ( X l n )( Q ) E [ S X |X ( F n )( P | O )] ≤ e n ( δ + ǫ ) S X ( X l n )( Q ) , where (a) follows fro m Pro position 3 .2, (b ) f rom the cond ition (16), an d (c) from th e condition (1 7). Therefore, for any ǫ > 0 and sufficiently large n , max O ∈P n ( X ) \{ P 0 n } ,Q ∈P l n ( X ) 1 n ln E [ S X |X ( G n ◦ Σ m n ◦ F n )( Q | O )] S X ( X l n )( Q ) ≤ δ + ǫ, which establishes th e theo rem. Using Theorem 4.2, we can no w construct good linear codes by a serial concaten ation scheme, where the inner co de is approx imately δ - asymptotically goo d (satisfying (17)) and the outer co de is a linear cod e having good distan ce prop erties if we set A n = { P ∈ P m n ( X ) | 1 − P (0) > γ } in the cond ition (16). Accord ing to [9 , Section IV], there exists a goo d low density parity check (LDPC) co de F n satisfying (1 6) for an approp riate γ . Then our final pr oblem is how to find a sequence of approximately δ -asymp totically good linear codes satis fying (17) with A n = { P ∈ P m n ( X ) | 1 − P (0) > γ } . In the next section, we will find such codes in a family o f code s called low density g enerator matrix ( LDGM) cod es. V . T H E S P E C T R A O F L D G M C O D E S In this section, we will in vestigate the joint spectra of LDGM codes. W e assume that the alp habet of codes is F q , and we denote a r egular LDGM code by the m ap F LD n,c,d : F n q → F m q defined b y F LD n,c,d △ = ( ⊙ m i =1 F CHK d ) ◦ Σ cn ◦ ( ⊙ n i =1 f REP c ) where nc = md , and f REP c is a single sy mbol r epetition cod e f REP c : F q → F c q defined by f REP c ( x ) △ = xx · · · x ∀ x ∈ F q , and ⊙ m i =1 F CHK d denotes a parallel co ncatenatio n of m in- depend ent copies of the random sing le symbol check cod e F CHK d : F d q → F q defined by F CHK d ( x d ) △ = d X i =1 C i x i ∀ x d ∈ F d q where C i ( i = 1 , 2 , · · · , d ) deno tes an independ ent unifo rm random variable on the set F q \{ 0 } . T o ev aluate th e joint spectrum o r conditional spectrum o f F LD n,c,d , we first need to calcu late the joint spectrum or cond i- tional spectru m of f REP c and F CHK d . W e have the following results. Pr opo sition 5.1: G ( f REP c )( u, v ) = 1 q X a ∈ F q u ( a )[ v ( a )] c , (18) G ( ⊙ n i =1 f REP c )( u, v ) = 1 q n X P ∈P n ( F q )  n nP  ( u nP ) ⊗ ( v ncP ) ⊗ , (19) S F q F q ( ⊙ n i =1 f REP c )( P, Q ) = S F q ( F n q )( P )1 { P = Q } , (20) S F q | F q ( ⊙ n i =1 f REP c )( Q | P ) = 1 { P = Q } . (21) Pr oof: The id entity ( 18) ho lds clearly . From ( 18) and Corollary 3 .2, we th en h av e G ( ⊙ n i =1 f REP c )( u, v ) =  1 q X a ∈ F q u ( a )[ v ( a )] c  n = 1 q n X P ∈P n ( F q )  n nP  Y a ∈ F q [ u ( a )] nP ( a ) [ v ( a )] ncP ( a ) = 1 q n X P ∈P n ( F q )  n nP  ( u nP ) ⊗ ( v ncP ) ⊗ . This proves (19). The identities (2 0) an d (21) are easy conse- quences o f (19). In order to obtain the joint spectrum o f F CHK d , we need th e following p roposition (also well k nown), which c an b e easily proved by ma thematical induction. Pr opo sition 5.2: L et Y d = d X i =1 X i , (22) where X i ( i = 1 , 2 , · · · , d ) is an in depend ent unifor m rando m variable on th e set F q \{ 0 } . Then we h av e Pr { Y d = a } = 1 { a = 0 } 1 q  1 −  − 1 q − 1  d − 1  + 1 { a 6 = 0 } 1 q  1 −  − 1 q − 1  d  (23) for any a ∈ GF( q ) . Now let us calculate the jo int spectrum of F CHK d . By Propo - sition 2.2, 5.2 and Coro llary 3.2, we obtain ed th e following propo sition. Its proof is long and he nce omitted here, and readers may r efer to [10 ] for the details. Pr opo sition 5.3: E [ G ( F CHK d )( u, v )] = 1 q d +1  (( u ) ⊕ ) d ( v ) ⊕ +  q u (0) − ( u ) ⊕ q − 1  d ( q v (0) − ( v ) ⊕ )  , (24) E [ S F q F q ( ⊙ m i =1 F CHK d )( P, Q )] = coef ( g 1 ( u, Q ) , ( u mdP ) ⊗ ) , (25) E [ S F q F q ( ⊙ m i =1 F CHK d )( P, Q )] ≤ g 2 ( O, P , Q ) , ∀ O ∈ P md ( F q ) ( O ( a ) > 0 , ∀ a ∈ { a | P ( a ) > 0 } ) , (26) 1 m ln α ( ⊙ m i =1 F CHK d )( P, Q ) ≤ δ d ( P (0) , Q (0)) + O  ln m m  , (27) where co ef ( f ( u ) , ( u v ) ⊗ ) denotes the coefficient of ( u v ) ⊗ in the p olynom ial f ( u ) , a nd g 1 ( u, Q ) △ =  m mQ  q m ( d +1)  (( u ) ⊕ ) d + ( q − 1)  q u (0) − ( u ) ⊕ q − 1  d  mQ (0)  (( u ) ⊕ ) d −  q u (0) − ( u ) ⊕ q − 1  d  m (1 − Q (0)) , g 2 ( O, P , Q ) △ =  m mQ  q m ( d +1) ( O mdP ) ⊗  1 + ( q − 1)  q O (0) − 1 q − 1  d  mQ (0)  1 −  q O (0) − 1 q − 1  d  m (1 − Q (0)) , δ d ( x, y ) △ = inf 0 < ˆ x< 1  dD ( x k ˆ x ) + y ln  1 + ( q − 1)  q ˆ x − 1 q − 1  d  + (1 − y ) ln  1 −  q ˆ x − 1 q − 1  d  . (28) where D ( x k ˆ x ) is the info rmation divergence d efined by D ( x k ˆ x ) △ = x ln x ˆ x + (1 − x ) ln 1 − x 1 − ˆ x . Based on the above pr eparation s, we now start to analyze the joint spe ctrum of the regular LD GM cod e F LD n,c,d . Theor em 5.1 : 1 n ln α ( F LD n,c,d )( P, Q ) ≤ c d δ d ( P (0) , Q (0)) + O  ln n n  . (2 9) where δ d is d efined by (2 8). Let A n ( γ ) △ = { P ∈ P n ( F q ) | 1 − P (0) > γ } , (30) where 0 < γ < 1 . Then, when q > 2 , for any 0 < γ < 1 and any δ > 0 , there exits a positive integer d 0 = d 0 ( γ , δ ) such that lim sup n →∞ max P ∈ A n ( γ ) ,Q ∈P m ( F q ) 1 n ln α ( F LD n,c,d )( P, Q ) ≤ δ (31) for all in tegers d ≥ d 0 . Pr oof: At first, accor ding to the definition of regular LDGM co des, we have E [ S F q | F q ( F LD n,c,d )( Q | P )] (a) = X O ∈P nc ( F q ) E [ S F q | F q ( ⊙ n i =1 f REP c )( O | P )] · E [ S F q | F q ( ⊙ m i =1 F CHK d )( Q | O )] (b) = X O ∈P nc ( F q ) 1 { P = O } E [ S F q | F q ( ⊙ m i =1 F CHK d )( Q | O )] = E [ S F q | F q ( ⊙ m i =1 F CHK d )( Q | P )] (c) ≤ e m ( δ d ( P (0) ,Q (0))+O( ln m m )) S F q ( F m q )( Q ) , where (a) fo llows from Pro position 3.2 an d th e definition of F LD n,c,d , (b) from Prop osition 5.1, and (c) fro m Proposition 5.3. This then c onclude s (29). By the defin ition o f δ d , we have δ d ( x, y ) ≤ dD ( x k x ) + y ln  1 + ( q − 1)  q x − 1 q − 1  d  + (1 − y ) ln  1 −  q x − 1 q − 1  d  ≤ ln  1 + ( q y − 1 )  q x − 1 q − 1  d  ≤ ( qy − 1)  q x − 1 q − 1  d ≤ ( q − 1)     q x − 1 q − 1     d . Furthermo re, when 0 ≤ x < 1 − γ and q > 2 , we hav e − 1 < − 1 q − 1 ≤ q x − 1 q − 1 ≤ 1 − q γ q − 1 < 1 or     q x − 1 q − 1     < max { 1 q − 1 , 1 − q γ q − 1 } < 1 . Then there exists a positiv e integer d 0 = d 0 ( γ , δ ) such that sup 0 ≤ x< 1 − γ , 0 ≤ y ≤ 1 δ d ( x, y ) ≤ dδ c ∀ d ≥ d 0 . Note here that the ratio d/c is the rate of the cod e and h ence should b e a co nstant or at least boun ded. Therefo re, for any d ≥ d 0 , we have max P ∈ A n ( γ ) ,Q ∈P m ( F q ) 1 n ln α ( F LD n,c,d )( P, Q ) (a) ≤ c d sup 0 ≤ x< 1 − γ , 0 ≤ y ≤ 1 δ d ( x, y ) + O  ln n n  ≤ δ + O  ln n n  , where (a) fo llows f rom (2 9). Th is co ncludes ( 31). Theorem 5. 1 ac tually exhibits a family of co des wh ose joint spectra are approximately δ -asympto tically g ood. Then together with the conclusion at the e nd o f Sectio n I V, we have completed the co nstruction of line ar codes with go od joint spectra, i.e., a serial concatenatio n schem e with one LDPC code as an outer code and one LDGM cod e as an inn er co de. An analog ous co nstruction has been proposed by Hsu in his thesis [11], but his purpose was only to fin d goo d channel codes and o nly a rate-1 LDGM cod e was employed a s an inner co de in his con struction. R E F E R E N C E S [1] Y . Zhao and J. Garcia-Frias, “T urbo compression/ joint source-cha nnel coding of correlated binary sources with hidden Mark ov correla tion, ” Signal Pr ocessing , vol . 86, no. 11, pp. 3115–3122, 2006. [2] J. Garcia -Frias, Y . Zhao, and W . Zhong, “T urbo-like codes for transmis- sion of correlated sources ov er noisy channe ls, ” IEEE Signal Proce ss. Mag . , v ol. 24, no. 5, pp. 58–66, Sep. 2007. [3] Y . Z hao, W . Zhong, and J. G arcia -Frias, “Transmissi on of correla ted senders over a Rayleigh fading multiple access channe l, ” Signal Pro- cessing , vol. 86, no. 11, pp. 3150–3 159, 2006. [4] S. Y ang, Y . Chen, and P . Qiu, “Linear codes based lossless joint source- channe l coding for multiple- access chan nels, ” submitted to IEEE T rans. Inf. Theory , draft a v aila ble at http:/ /arxi v . org/a bs/cs.IT/0611146. [5] D. Di vsalar , H. Jin, and R. J. McEliece, “Codi ng theorems for “T urbo- lik e” codes, ” in 36th Allerton Conf. on Communicati on, Contr ol, and Computing , Sep. 1998, pp. 201–210. [6] I. Csisz ´ ar and J. K ¨ orne r , Informati on Theory: Coding Theorems for Discr ete Memo ryless Systems . New Y ork: Academic Press, 1981. [7] R. G. Gallager , Low-Density P arity-Chec k Codes . Cambridge , MA: MIT Press, 1963. [8] I. Csisz ´ ar , “Linear codes for sources and source networks: Error expo- nents, univ ersal coding, ” IEEE T rans. Inf. Theory , vol. 28, no. 4, pp. 585–592, J ul. 1982. [9] A. Bennatan and D. Burshtein, “On the appli cation of LDPC codes to arbitrary disc rete-memoryl ess channels, ” IEEE T rans. Inf. Theory , vol. 50, no. 3, pp. 417–437, Mar . 2004. [10] S. Y ang, et al, “Construc ting linear codes with good spectra, ” in prepara tion, to be submitt ed to IE EE Tr ans. Inf . Theory . [11] C.-H. Hsu, “Design and analysis of capaci ty-achi e ving codes and optimal recei vers with lo w comple xity , ” Ph.D. dissertation, Uni versi ty of Michi gan, 2006.