A New Sphere-Packing Bound for Maximal Error Exponent for Multiple-Access Channels

A New Sphere-P ac k ing Bo und for Maximal Error Exp onen t for Multiple-Access Cha nnels Ali Nazari, Sandeep Pradhan and Ac hilleas Anastasop oulos Electrical Engineering and Computer Science Dept. Univ ersit y of Mic hig an, Ann Arb or, MI 48109-2 122, USA E-mail: { anazari,pradhanv,anastas } @umic h.edu No v em b er 19, 201 8 Abstract In this w ork, a new lo w er b ound for the maximal error probability of a tw o-user discrete memoryless (DM) multiple-access channel ( MAC) is derived. This is the first b oun d of this type that explicitly imp oses inde- p endence of the users’ input distributions (conditioned on the time-sharing auxiliary v ariable) and t hus results in a tigh ter sphere-packing exp onent when compared to th e tigh test known exponent deriv ed by Haroutunian. 1 in tro duction An in teresting problem in net work information theory is to deter mine the mini- m um probability of error which can b e achiev ed on a dis c rete memory less (DM), m ultiple-access channel (MA C). Mor e sp ecifically , a t wo-user DM-MAC is de- fined b y a sto c hastic matr ix 1 W : X × Y → Z , where the input alphab ets, X , Y , and the o utput a lpha bet, Z , ar e finite sets. The channel tr a nsition probability for sequences of length n is given by W n ( z | x , y ) , n Y i =1 W ( z i | x i , y i ) (1) where x , ( x 1 , ..., x n ) ∈ X n , y , ( y 1 , ..., y n ) ∈ Y n and z , ( z 1 , ..., z n ) ∈ Z n . 1 W e use the f ollowing notation throughout this work. Script capitals U , X , Y , Z , . . . denote finite, nonempt y sets. T o show the cardinality of a s et X , we use |X | . W e also use the letters P , Q , . . . for probability distributions on finite sets, and U , X , Y , . . . for random v ariables. 1 It is known [1], that for a n y ( R X , R Y ) in the interior of a certain se t C , and for all sufficiently la rge n , there exis ts a multiuser co de with an arbitra r y small average probability of er ror. Conv ersely , for an y ( R X , R Y ) outside of C , the av erage probability of erro r is bounded awa y from 0. The set C , which is called c ap acity r e gion for W , is the closure of the set of all rate pairs ( R X , R Y ) satisfying [2] 0 ≤ R X ≤ I ( X ∧ Z | Y , Q ) (2a) 0 ≤ R Y ≤ I ( Y ∧ Z | X , Q ) (2b) 0 ≤ R X + R Y ≤ I ( X Y ∧ Z | Q ) , (2c) for all choices of joint distributions ov er the ra ndo m v ar iables Q, X , Y , Z of the form P ( q ) P ( x | q ) P ( y | q ) W ( z | x, y ) with Q ∈ Q and |Q| ≤ 4. Haroutunian [3] der ived a lower b ound o n the o ptimal av erage erro r proba - bilit y for W . This re s ult asserts that the average probability o f error is b ounded below b y exp {− nE sp ( R X , R Y , W ) } , where E sp ( R X , R Y , W ) , max P X Y min V Z | X Y D ( V Z | X Y || W | P X Y ) . (3) Here, the maximum is taken ov er all p ossible jo int distributions ov er the ra ndo m v ar iables X , Y , a nd the minimu m over a ll tes t channels V Z | X Y which satisfy at least one of the following conditions I V ( X ∧ Z | Y ) ≤ R X (4a) I V ( Y ∧ Z | X ) ≤ R Y (4b) I V ( X Y ∧ Z ) ≤ R X + R Y , (4c) where V , V Z | X Y × P X Y . This b ound tends to b e somewhat lo ose be cause it do es not take into account the sepa ration of the tw o e nc o der s in the MAC. In this pap er, we derive a new lower b ound that explicitly ca ptures the separatio n o f the enco ders in the MA C and th us is tig h ter than the one provided by Haro utunian. Howev er, this b ound is o nly v a lid for the ma ximal and not the av erage error pr obability . Nev ertheless, w e be lie v e that the techniques used in this deriv ation can b e extended to provide low er bo unds for the av erage error probability as well. The pap er is o rganized as follows. Firs t, some pr eliminaries ar e int ro duced in section 2. Then in sectio n 3, we state a nd prov e the main res ult. The pro of hinges up on a str ong conv erse theor em w hich is also stated in the same Section and prov ed in the App e ndix. 2 Preliminaries F or an y alphab et X , P ( X ) denotes the set o f all probability distributions on X . The typ e of a s equence x = ( x 1 , ..., x n ) ∈ X n is the distributions P x on X defined by P x ( x ) , 1 n N ( x | x ) , x ∈ X , (5) 2 where N ( x | x ) denotes the num ber of o ccurr ences of x in x . Let P n ( X ) denote the set of all types in X n , and define the set of all s equences in X n of t yp e P as T P , { x ∈ X n : P x = P } . (6) The joint t yp e of a pair ( x , y ) ∈ X n × Y n is the pr obability distribution P x , y on X × Y defined by P x , y ( x, y ) , 1 n N ( x, y | x , y ) , ( x, y ) ∈ X × Y , (7) where N ( x, y | x , y ) is the n umber o f o ccur rences of ( x , y ) in ( x , y ). The rela - tive entropy or Kul lb ack L eibler distance b etw e en t wo pro babilit y distribution P, Q ∈ P ( X ) is defined a s D ( P || Q ) , X x ∈X P ( x ) lo g P ( x ) Q ( x ) . (8) Let W ( Y |X ) denote the set o f a ll sto chastic matrices with input alphabe t X and output alphab et Y . Then, given s tochastic matrices V , W ∈ W ( Y |X ), the conditional I-diver genc e is defined by D ( V || W | P ) , X x ∈X P ( x ) D ( V ( ·| x ) || W ( ·| x )) . (9) An ( n, M , λ ) co de for W : X → Z , is a s ystem { ( u i , D i ) : 1 ≤ i ≤ M } with • u i ∈ X n , D i ⊂ Z n • D i ∩ D i ′ = ∅ for i 6 = i ′ • W n ( D i | u i ) ≥ 1 − λ , for 1 ≤ i ≤ M . An ( n, M , N ) multi-user co de is a set { ( u i , v j , D ij ) : 1 ≤ i ≤ M , 1 ≤ j ≤ N } with • u i ∈ X n , v j ∈ Y n , D ij ⊂ Z n • D ij ∩ D i ′ j ′ = ∅ for ( i, j ) 6 = ( i ′ , j ′ ). Finally , an ( n, M , N , λ ) co de for the MAC, W , is an ( n, M , N ) co de with 1 M N M X i =1 N X j =1 W n ( D i,j | u i , v j ) ≥ 1 − λ. (10) 3 main result The main res ult of this pap er is a lower ( spher e p acki ng ) b ound for the maximal error probability for a MAC. T o state the new b ound we need an intermediate result that has the form of a strong co n verse for the MA C. W e state this r esult here and relegate the pro of to the app endix. 3 Definition 1. F or any D M-MA C, W , for any joint distribution P ∈ P ( X × Y ) , any 0 ≤ λ < 1 , and any ( n, M , N ) c o de, C , define E W ( P, λ ) , { ( u i , v j ) ∈ C : W ( D ij | u i , v j ) ≥ 1 − λ 2 , ( u i , v j ) ∈ T P } . (11) Theorem 1. Consider any ( n, M , N ) c o de C . F or every P n X Y ∈ P n ( X × Y ) , such that |E W ( P n X Y , λ ) | ≥ 1 ( n +1) |X ||Y | (1 − 2 λ 1+ λ ) M N , then ( 1 n log M , 1 n log N ) ∈ C n W ( P n X Y ) (12) wher e C n W ( P ) is define d as the closur e of the set of al l ( R 1 , R 2 ) p airs satisfying R 1 ≤ I ( ¯ X ∧ ¯ Z | ¯ Y , Q ) + ǫ n (13a) R 2 ≤ I ( ¯ Y ∧ ¯ Z | ¯ X , Q ) + ǫ n (13b) R 1 + R 2 ≤ I ( ¯ X ¯ Y ∧ ¯ Z | Q ) + ǫ n (13c) for some choic e of r andom variab les Q define d on { 1 , 2 , 3 , 4 } , and joint distri- bution p ( q ) p ( x | q ) p ( y | q ) w ( z | x, y ) , with mar ginal distribution p ( x, y ) = P n ( x, y ) . Her e, ǫ n → 0 an n → ∞ . W e further define C W ( P ) ( the li miting v ersion of the sets C n W ( P )) as the closure of the set of all ( R 1 , R 2 ) pairs satisfying R 1 ≤ I ( ¯ X ∧ ¯ Z | ¯ Y , Q ) (1 4 a) R 2 ≤ I ( ¯ Y ∧ ¯ Z | ¯ X , Q ) (14b) R 1 + R 2 ≤ I ( ¯ X ¯ Y ∧ ¯ Z | Q ) , (14c) for some choice of random v a riables Q defined o n { 1 , 2 , 3 , 4 } , and joint distribu- tion p ( q ) p ( x | q ) p ( y | q ) w ( z | x, y ), with marg inal distribution p ( x, y ) = P ( x, y ). Theorem 2. (Spher e Pack ing B ound). F or any R X , R Y > 0 , δ > 0 and a ny DM-MAC , W : X × Y → Z , every ( n, M , N , λ ) c o de, C with 1 n log M ≥ R X + δ (15a) 1 n log N ≥ R Y + δ, (15b) has maximu m pr ob ability of err or P m e ≥ 1 2 exp  − nE sp ( R X , R Y , W )(1 + δ )  , (16) wher e E sp ( R X , R Y , W ) , max P X Y ∈P ( X ×Y ) min V :( R X ,R Y ) / ∈ C V ( P X Y ) D ( V || W | P X Y ) . (17) 4 Pr o of. If λ = 1, the result is trivial. Assume λ < 1. Let’s cho ose λ ′ such that max { 1 − δ, λ } < λ ′ < 1 . Since λ ′ > λ , every ( n, M , N , λ ) co de is als o an ( n, M , N , λ ′ ) c o de. Using the same argument as [4, pp. 189], we co nclude that there e x ist at le a st one dominant type P n ∈ P n ( X ×Y ), such that |E W ( P n , λ ′ ) | ≥ 1 ( n +1) |X ||Y | (1 − 2 λ ′ 1+ λ ′ ) M N . Consider an arbitrary DM-MAC V : X × Y → Z , such that ( R X , R Y ) / ∈ C n V ( P n ). By Theo rem 1 , there exist at least one pair ( u i , v j ) with joint type P n X Y such that V n ( D c ij | u i , v j ) > 1 + λ ′ 2 > 1 − δ 2 . (18) Using the same method as Cs iszar in [5, pp. 167], w e hav e W n ( D c ij | u i , v j ) ≥ exp n − D ( V || W | P n ) + h (1 − δ 2 ) 1 − δ 2 o ≥ 1 2 exp {− nD ( V || W | P n )(1 + δ ) } , (19) for small eno ugh δ satisfying h (1 − δ 2 ) < 1 − δ 2 . By ma x imizing the re s ult ov er the arbitrar y channel V , we g et P m e ≥ max V :( R X ,R Y ) / ∈ C n V ( P n ) 1 2 exp  − nD ( V || W | P n )(1 + δ )  = 1 2 exp  − n min V :( R X ,R Y ) / ∈ C n V ( P n ) D ( V || W | P n )(1 + δ )  ≥ min P n ∈ P n ( X ×Y ) 1 2 exp  − n min V :( R X ,R Y ) / ∈ C n V ( P n ) D ( V || W | P n )(1 + δ )  ≥ min P ∈ P ( X ×Y ) 1 2 exp  − n min V :( R X ,R Y ) / ∈ C n V ( P ) D ( V || W | P )(1 + δ )  (20) Using Lemma 6 , we conclude that for sufficiently lar ge n , P m e ≥ min P ∈ P ( X ×Y ) 1 2 exp  − n min V :( R X ,R Y ) / ∈ C V ( P ) D ( V || W | P )(1 + δ )  . (21) which completes the pro of. 4 app endix The basic idea of the pro of is wr ing ing technique whic h was used for the fir st time, by Ahlswede [6]. 5 Consider any P n X Y ∈ P n ( X × Y ), such that |E W ( P n X Y , λ ) | ≥ 1 ( n +1) |X ||Y | (1 − 2 λ 1+ λ ) M N . let’s define A , { ( i, j ) : W ( D ij | u i , v j ) ≥ 1 − λ 2 , ( u i , v j ) ∈ T P n X Y } . Since |A| = |E W ( P n X Y , λ ) | , we c o nclude tha t |A| ≥ 1 ( n + 1) |X || Y | (1 − 2 λ 1 + λ ) M N . (22) Define, C ( i ) = { ( i, j ) : ( i, j ) ∈ A , 1 ≤ j ≤ N } (23 a) B ( j ) = { ( i, j ) : ( i, j ) ∈ A , 1 ≤ i ≤ M } . (23b) Consider the subco de { ( u i , v j , D ij ) : ( i, j ) ∈ A } a nd define rando m v ariables X n , Y n P r (( X n , Y n ) = ( u i , v j )) = 1 |A| if ( i, j ) ∈ A . (24) Lemma 1. F or r andom variables X n , Y n define d in (24) , the mut ual informa- tion satisfies the fol lowing ine quality: I ( X n ∧ Y n ) ≤ − log(1 − 2 λ 1 + λ ) + |X ||Y | log ( n + 1 ) . (25) Pr o of. This is a ge neralization of the pr o of by Dueck in [4]. Observe that H ( Y n | X n ) = X u i P r ( X n = u i ) H ( Y n | u i ) . (26) How ev er, by the definition o f the v ariables X n , Y n we hav e H ( Y n | u i ) = log |{ j : ( i, j ) ∈ A}| (27) and P r ( X n = u i ) = |A| − 1 . |{ j : ( i, j ) ∈ A}| . (28) Hence, H ( Y n | X n ) = |A| − 1 M X i =1 |{ j : ( i, j ) ∈ A}| log |{ j : ( i, j ) ∈ A}| . (29) In the right hand side of (29), the summands are of the form m log m . This function of m is increasing and conv ex in m . Thus, H ( Y n | X n ) ≥ |A| − 1 ( M X i =1 |{ j : ( i, j ) ∈ A}| ) log( M − 1 M X i =1 |{ j : ( i, j ) ∈ A}| ) , (30) 6 and since M X i =1 |{ j : ( i, j ) ∈ A}| = |A| , (31) we hav e H ( Y n | X n ) ≥ log ( M − 1 |A| ) . (32) By (22), we co nclude that H ( Y n | X n ) ≥ log N + log(1 − 2 λ 1 + λ ) − |X ||Y | log ( n + 1 ) . (33) Finally , I ( X n ∧ Y n ) = H ( Y n ) − H ( Y n | X n ) ≤ log N − H ( Y n | X n ) , (34) which concludes the pro of. Lemma 2. [7] L et X n , Y n b e R V’s with values in X n , Y n r esp. and assume that I ( X n ∧ Y n ) ≤ σ (35) Then, for any 0 < δ < σ ther e exist t 1 , t 2 , ..., t k ∈ { 1 , ..., n } wher e 0 ≤ k < 2 σ δ such that for some ¯ x t 1 , ¯ y t 1 , ¯ x t 2 , ¯ y t 2 , ..., ¯ x t k , ¯ y t k I ( X t ∧ Y t | X t 1 = ¯ x t 1 , Y t 1 = ¯ y t 1 , ..., X t k = ¯ x t k , Y t k = ¯ y t k ) ≤ δ for t = 1 , 2 , ..., n (36) and P r ( X t 1 = ¯ x t 1 , Y t 1 = ¯ y t 1 , ..., X t k = ¯ x t k , Y t k = ¯ y t k ) ≥ ( δ |X ||Y | (2 σ − δ ) ) k . (37) Consider the sub co de { ( u i , v j , D ij ) : ( i, j ) ∈ ¯ A} , where ¯ A , { ( i, j ) ∈ A : u it l = ¯ x t l , v j t l = ¯ y t l 1 ≤ l ≤ k } (38) and define ¯ C ( i ) = { ( i, j ) : ( i, j ) ∈ ¯ A , 1 ≤ j ≤ N } (39a) ¯ B ( j ) = { ( i, j ) : ( i, j ) ∈ ¯ A , 1 ≤ i ≤ M } . (39b) 7 Lemma 3. The sub c o de { ( u i , v j , D ij ) : ( i, j ) ∈ ¯ A} , is a sub c o de with maximal err or pr ob ability 1+ λ 2 , and | ¯ A| ≥ ( δ |X ||Y | (2 σ − δ ) ) k |A| . (40) Mor e over, X x,y | P r ( X t = x, Y t = y ) − P r ( X t = x ) P r ( Y t = y ) | ≤ 2 δ 1 / 2 , (41) wher e X n = ( X 1 , ..., X n ) , Y n = ( Y 1 , ..., Y n ) ar e distribute d ac c or ding to t he F ano-distribution of the sub c o de { ( u i , v j , D ij ) : ( i, j ) ∈ ¯ A} . Pr o of. Since ¯ A ⊂ A , the maximal probability of err or fo r this subco de is at most 1+ λ 2 . The sec o nd par t o f Le mma 2, yields immediately (40). On the other hand, P A ( X t = x, Y t = y | ¯ x t 1 , ¯ y t 1 , ¯ x t 2 , ¯ y t 2 , ..., ¯ x t k , ¯ y t k ) = P A ( X t = x, Y t = y , ¯ x t 1 , ¯ y t 1 , ¯ x t 2 , ¯ y t 2 , ..., ¯ x t k , ¯ y t k ) P A ( ¯ x t 1 , ¯ y t 1 , ¯ x t 2 , ¯ y t 2 , ..., ¯ x t k , ¯ y t k ) = N A ( X t = x, Y t = y , ¯ x t 1 , ¯ y t 1 , ¯ x t 2 , ¯ y t 2 , ..., ¯ x t k , ¯ y t k ) N A ( ¯ x t 1 , ¯ y t 1 , ¯ x t 2 , ¯ y t 2 , ..., ¯ x t k , ¯ y t k ) = N ¯ A ( X t = x, Y t = y ) | ¯ A| = P ¯ A ( X t = x, Y t = y ) . (42) Therefore, b y the first part of Lemma 2, we conclude that I ( X t ∧ Y t ) ≤ δ, for 1 ≤ t ≤ n. (43) Since I ( X t ∧ Y t ) is an I- diver genc e , Pinsker’s inequality implies [8] X x,y | P r ( X t = x, Y t = y ) − P r ( X t = x ) P r ( Y t = y ) | ≤ 2 δ 1 / 2 . (44) Lemma 4. [9]: F or a ( n, M , λ ) c o de { ( u i , D i ) : 1 ≤ i ≤ M } for the non- stationary DMC ( W t ) ∞ t =1 log M < n X t =1 I ( X t ∧ Z t ) + 3 1 − λ |X | n 1 / 2 , (45) wher e the distribution of the R V’s ar e determine d by the F ano-distribution on the c o dewor ds. 8 Define random v a riables ¯ X n , ¯ Y n on X n resp. Y n by P r (( ¯ X n , ¯ Y n ) = ( u i , v j )) = 1 | ¯ A| if ( i, j ) ∈ ¯ A . (46) Lemma 5. F or any 0 ≤ λ < 1 , any ( n, M , N ) c o de C , { ( u i , v j , D ij ) : 1 ≤ i ≤ M , 1 ≤ j ≤ N } for the any MAC , W , and for any P n X Y ∈ P n ( X × Y ) , such that |E W ( P n X Y , λ ) | ≥ 1 ( n +1) |X ||Y | (1 − 2 λ 1+ λ ) M N log M ≤ n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t ) + c 1 ( λ ) n 1 / 2 + c 1 k log ( 2 σ δ ) log N ≤ n X t =1 I ( ¯ Y t ∧ ¯ Z t | ¯ X t ) + c 2 ( λ ) n 1 / 2 + c 2 k log ( 2 σ δ ) log M N ≤ n X t =1 I ( ¯ X t ¯ Y t ∧ ¯ Z t ) + c 3 ( λ ) n 1 / 2 + c 3 k log ( 2 σ δ ) , wher e the distributions of the R V’s ar e determine d by the F ano-distribution on the c o dewor ds { ( u i , v j ) : ( i, j ) ∈ ¯ A} . Her e, c i ( λ ) and c i ar e su itable functions of λ . Pr o of. F or any fixed j , cons ider ( n, | ¯ B ( j ) | ) co de { ( u i , D ij ) : ( i, j ) ∈ ¯ B ( j ) } . Any pair of co dewords in this c ode has pro babilit y of err or at most eq ua l to 1+ λ 2 . Let’s define λ ′ , 1+ λ 2 . It follows from Lemma 4 that log | ¯ B ( j ) | ≤ n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t = v j t ) + 3 1 − λ ′ |X | n 1 / 2 . (47) Similarly , log | ¯ C ( i ) | ≤ n X t =1 I ( ¯ Y t ∧ ¯ Z t | ¯ X t = u it ) + 3 1 − λ ′ |Y | n 1 / 2 (48) log | ¯ A| ≤ n X t =1 I ( ¯ X t ¯ Y t ∧ ¯ Z t ) + 3 1 − λ ′ |X ||Y | n 1 / 2 . (49) Since P r ( ¯ Y t = y ) = | ¯ A| − 1 P ( i,j ) ∈ ¯ A 1 { v jt ,y } , | ¯ A| − 1 X ( i,j ) ∈ ¯ A log | ¯ B ( j ) | ≤ X ( i,j ) ∈ ¯ A n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t = v j t ) P y 1 { v jt ,y } | ¯ A| + 3 1 − λ ′ |X | n 1 / 2 = n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t ) + 3 1 − λ ′ |X | n 1 / 2 . (50) 9 Define λ ∗ , 2 λ 1+ λ , a nd B ∗ , 1 − λ ∗ n M ( n + 1) |X || Y | ( δ |X ||Y | (2 σ − δ ) ) k . (51) Therefore, | ¯ A| − 1 X ( i,j ) ∈ ¯ A log | ¯ B ( j ) | = | ¯ A| − 1 X j | ¯ B ( j ) | log | ¯ B ( j ) | ≥ | ¯ A| − 1 X j : | ¯ B ( j ) |≥ B ∗ | ¯ B ( j ) | log | ¯ B ( j ) | ≥ | ¯ A| − 1 log( B ∗ ) X j : | ¯ B ( j ) |≥ B ∗ | ¯ B ( j ) | ≥ | ¯ A| − 1 log( B ∗ )( | ¯ A| − N B ∗ ) . (52) By lemma 3, (22), and definition of B ∗ , N B ∗ ≤ 1 n | ¯ A| . (53) Therefore, | ¯ A| − 1 X ( i,j ) ∈ ¯ A log | ¯ B ( j ) | ≥ | ¯ A| − 1 log( B ∗ )( | ¯ A| − 1 n | ¯ A| ) = (1 − 1 n ) log( 1 − λ ∗ n M ( n + 1) |X || Y | ( δ |X ||Y | (2 σ − δ ) ) k ) . (54) By (50), (54) log M ≤ (1 + 2 n )( n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t ) + 3 1 − λ ′ |X | n 1 / 2 ) − log(1 − λ ∗ ) + log n + |X ||Y | log( n + 1 ) + k log( |X ||Y | 2 σ δ ) ≤ n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t ) + c 1 ( λ ′ ) n 1 / 2 + c 1 k log ( 2 σ δ ) + 2 |Z | (55) Analogously , log N ≤ n X t =1 I ( ¯ Y t ∧ ¯ Z t | ¯ X t ) + c 2 ( λ ′ ) n 1 / 2 + c 2 k log ( 2 σ δ ) + 2 |Z | . (56) 10 T o find an upp er b ound for lo g M N , we first try to find a lower bound on the log | ¯ A| . By Lemma 3 log | ¯ A| ≥ log |A| + k log ( δ |X ||Y | (2 σ − δ ) ) ≥ log |A| + k log ( δ |X ||Y | 2 σ ) = log |A| − k log ( 2 σ δ ) − k log( |X ||Y | ) ≥ log( M N ) − |X ||Y | log( n + 1 ) + log(1 − 2 λ 1 + λ ) − k log( 2 σ δ ) − k log ( |X ||Y | ) . (57) Therefore, log( M N ) ≤ log | ¯ A| + c 3 k log( 2 σ δ ) . (58) Using (49), log M N ≤ n X t =1 I ( ¯ X t ¯ Y t ∧ ¯ Z t ) + c 3 ( λ ′ ) n 1 / 2 + c 3 k log ( 2 σ δ ) . (59) Note that, in general ¯ X t and ¯ Y t are not indep endent. In the following, we prov e that they are nearly indep endent. Now, w e co mbine (25) and lemma 3. F or an ( n, M , N ) co de { ( u i , v j , D ij ) : 1 ≤ i ≤ M , 1 ≤ j ≤ N } which has the par ticula r prop erty men tioned in theorem 1, define A , ¯ A as defined b efore . Apply lemma 3 with parameter δ = n − 1 / 2 . Using σ = − log(1 − 2 λ 1+ λ ) + |X ||Y | log ( n + 1), we conclude that k ≤ 2 σ δ = 2 √ n ( − lo g(1 − 2 λ 1 + λ ) + |X ||Y | log( n + 1 )) ∼ O ( √ n log n ) (60) and | P r ( ¯ X t = x, ¯ Y t = y ) − P r ( ¯ X t = x ) P r ( ¯ Y t = y ) | ≤ 2 n − 1 / 4 , (61) for any x ∈ X , y ∈ Y , and t = 1 , ..., n . W e can write the ab ov e equations as follows 1 n log M ≤ 1 n n X t =1 I ( ¯ X t ∧ ¯ Z t | ¯ Y t ) + C ( λ ) o ( n ) n (62a) 1 n log N ≤ 1 n n X t =1 I ( ¯ Y t ∧ ¯ Z t | ¯ X t ) + C ( λ ) o ( n ) n (62b) 1 n log M N ≤ 1 n n X t =1 I ( ¯ X t ¯ Y t ∧ ¯ Z t ) + C ( λ ) o ( n ) n . (62c) 11 The expressions in (62a)-(62c) are the averages of the mutual informatio ns calculated at the empirical distributions in the column t of the mentioned sub c ode . W e can rewr ite these equatio ns with the new v a riable Q , where Q = q ∈ { 1 , 2 , ..., n } with pro ba bilit y 1 n . Using the same metho d a s Cov er [1, pp. 402 ], we obtain the result. The only thing remained to b e found is the distribution under which we calculate the mutual infor mations. How ever, by (61) | P ( ¯ X = x, ¯ Y = y | Q = q ) − P ( ¯ X = x | Q = q ) P ( ¯ Y = y | Q = q ) | = | P r ( ¯ X q = x, ¯ Y q = y ) − P r ( ¯ X q = x ) P r ( ¯ Y q = y ) | ≤ 2 n − 1 / 4 . (63) Using the contin uit y of conditional m utual information with resp ect to distribu- tions, using the same idea of [10, pp. 722], we conclude that, if t wo distributions are clos e, the conditional mutual informatio ns, calcula ted based o n them, can- not b e to o far. More prec is ely , we can say that there ex ists a sequence { δ n } ∞ n =1 , δ n → 0 as n → ∞ , such that, 1 n log M ≤ I ( ¯ X t ∧ ¯ Z t | ¯ Y t , Q ) + C ( λ ) o ( n ) n + δ n 1 n log N ≤ I ( ¯ Y t ∧ ¯ Z t | ¯ X t , Q ) + C ( λ ) o ( n ) n + δ n 1 n log M N ≤ I ( ¯ X t ¯ Y t ∧ ¯ Z t , Q ) + C ( λ ) o ( n ) n + δ n . (6 4) Here, the m utual informations calculated based o n p ( q ) p ( x | q ) p ( y | q ) w ( z | x, y ), with mar ginal distribution P n X Y ( x, y ). On the other hand, the joint pr o bability distribution of ¯ X and ¯ Y is P ( ¯ X = x, ¯ Y = y ) = X ( i,j ) ∈ ¯ A P ( ¯ X ( W 1 ) = x, ¯ Y ( W 2 ) = y , W 1 = i, W 2 = j ) = X ( i,j ) ∈ ¯ A P ( ¯ X ( i ) = x, ¯ Y ( j ) = y ) P ( i, j ) = 1 | ¯ A| X ( i,j ) ∈ ¯ A P ( ¯ X ( i ) = x, ¯ Y ( j ) = y ) = 1 | ¯ A| X ( i,j ) ∈ ¯ A 1 n n X q =1 1 { ¯ X q ( i ) = x, ¯ Y q ( j ) = y } (65) How ev er, all co dewords have the sa me joint t yp e P n X Y , therefore, n X q =1 1 { ¯ X q ( i ) = x, ¯ Y q ( j ) = y } = nP n X Y ( x, y ) . (66) 12 (65) and (6 6) result in P ( ¯ X = x, ¯ Y = y ) = P n X Y ( x, y ) . (67) Finally , we can conclude tha t P ( q , x, y , z ) = p ( q ) p ( x | q ) p ( y | q ) W ( z | x, y ) , (68) in which the marginal distribution of ¯ X and ¯ Y is P n X Y ( x, y ). The cardinality b ound on the time-sharing random v ar iable, Q , is the con- sequence of Carath´ eo dor y’s theor em on the conv ex set [11], [12], [1]. Lemma 6. F or any fixe d P ∈ P ( X × Y ) , r ate p air ( R X , R Y ) , lim n →∞ min V ∈ D n ( P ) D ( V || W | P ) = min V ∈ D ( P ) D ( V || W | P ) , (69) wher e D ( P ) , { V : ( R X , R Y ) / ∈ C V ( P ) } D n ( P ) , { V : ( R X , R Y ) / ∈ C n V ( P ) } . (70) Pr o of. Define α n , min V ∈ D n ( P ) D ( V || W | P ), and α ∗ , min V ∈ D ( P ) D ( V || W | P ). Moreov er, supp ose α ∗ is achieved by V ∗ . S ince { α n } ∞ n =1 is a decr easing sequence and it is b ounded from below ( α n ≥ α ∗ ), ther efore it has a limit. Supp ose the limit is not equal to α ∗ . Therefore, there exist a δ > 0 , suc h that for all sufficiently large n , | α n − α ∗ | ≥ δ. (71) Hence, for all V ∈ D n ( P ), for all sufficiently la rge n , D ( V || W | P ) − α ∗ ≥ δ (72) which concludes that V ∗ cannot b elong to D n ( P ), i.e., for all sufficiently large n , ( R X , R Y ) ∈ C n V ∗ ( P ) . (73) Since V ∗ ∈ D ( P ), ( R X , R Y ) / ∈ C V ∗ ( P ) . (74) Therefore C n V ∗ ( P ) canno t converge to C V ∗ ( P ), whic h is a c ont radiction. References [1] T. M. Co ver a nd J. A. Tho ma s, Elements of Information T he ory . New Y or k: John Wiley & Sons, 1991. 13 [2] D. Slepian and J. K. W olf, “A co ding theorem for multiple access channels with correla ted sources ,” b el l Syst. te ch. J. , vol. 52, pp. 10 37–10 7 6, 197 3. [3] E . A. Haro utunian, “L ower b ound fo r the erro r probability o f m ultiple- access channels,” Pr oblemy Per e dachi I n formatsii , vol. 11, pp. 23 –36, June 1975. [4] G. Dueck, “The stro ng conv erse of the co ding theo r em for the multi-user channels,” Journal of Combinatorics, Information and System scienc es , pp. 187–1 96, 198 1. [5] I. Csiszar a nd J. Kor ner, Information the ory: Co ding the or ems for D iscr ete memoryless Systems. , 1981 . [6] R. Ahlswede, “On tw o-wa y comm unication channels a nd a problem by zarankiewic s ,” in Pr obl. of Contr ol and Inform. The ory . [7] ——, “An elemen tary pr o o f of the strong con v erse theorem for the mul- tiple acce ss c hannel,” J ou r n al of Combinatorics, Information and System scienc es , pp. 216–2 30, 1982. [8] P . H. A. F edotov and F. T opsøe, “Refinemen ts o f pinsk er’s inequality ,” IEEE T r ans. Information The ory , vol. 49, no . 6, pp. 1491– 1498, J une 2003. [9] U. Augustin, “Gedach tnisfreie k annale for diskrete z e it,” Z. Wahrschein- lichkelts the ory verw , pp. 10–6 1 , 1966. [10] R. Ahlswede, “ The rate-distortion for m ultiple description without excess rates,” IEEE T r ans. Information The ory , vol. 31 , no. 6, pp. 721 –726, Nov. 1985. [11] H. G. Eggleston, Convexity . Cambridge, UK : Cambridge Universit y P r ess, 1969. [12] B. Gr ¨ un baum, Convex Polytop es . New Y ork: McGraw-Hill, 1 9 76. 14