Universal Coding for Lossless and Lossy Complementary Delivery Problems

1 Uni v ersal Coding for Lossless and Lossy Complementary Deli v ery P roblems Shigeaki Kuzuoka, Member , IEEE, Akisa to Kimura, Senior Me mber , IEEE, and T omohiko Uyematsu, S enior Membe r , IEEE Abstract This paper deals wit h a coding problem called complementary delivery , where messages from two correlated sources are jointly encoded and each decoder reproduces one of two messages using the other messag e as the side information. Both lossless and lossy uni versal complementary deliv ery coding schemes are in vestigated. In the lossless case, it is demonstrated that a unive rsal complementary deliv ery code can be constructed by only combining two Slepian-W olf codes. Especially , it is shown that a univ ersal lossless complementary deliv ery code, for which error probability is expo nentially tight, can be constructed from two linear Slepian-W olf codes. In t he lossy case, a unive rsal complementary deli very coding scheme based on W yner-Zi v co des is proposed. While the proposed scheme cannot attain the optimal rate-distortion trade-of f in general, the rate-loss is upper bound ed by a univ ersal constant under some mild conditions. The proposed sche mes allo ws us to apply an y Slepian-W olf and W yner-Zi v code s to complementary deliv ery coding. Index T erms complementary delivery , multiterminal source coding, network coding, univ ersal coding, S lepian-W olf coding, W yner-Zi v codin g. I . I N T R O D U C T I O N The sou rce co ding p roblem fo r co rrelated in formatio n sources was initiated by Slepian and W olf [1]. T hey treated the case where two information sources are encoded separately and then reproduced at the single destination. Subsequen tly , v ario us coding problem s deriv ed from Slepian-W o lf coding have been consider ed (e.g. [2]–[ 5]). Correspon ding lo ssy codin g pro blem was studied by W yner and Z iv [6] , where they in vestigated the lossy codin g problem when the decoder can fully ob serve the side inform ation. While the messages are enco ded separately in S. Kuzuoka is with the Depa rtment of C omputer and Communication Sciences, W akayama Univ ersity , 930 Sakaedani, W akayama, 640-8510 Japan (e-mail: kuzuok a@ieee .org) A. Ki mura i s wi th NTT Communicati on Science Labor atories, NTT Corporatio n, 3-1 Mo rinosato W akamiya, Atsugi-shi, Kanagaw a, 243-0198 Japan (e-mail: resear ch@akisat o.org) T . Uyematsu is with the Department of Communicatio ns and Integrat ed Systems, T okyo Instit ute of T echnology , 2-12-1 Ookayama, Meguro-ku, T okyo, 152-8550 Ja pan (e-mail: uyematsu@ie ee.org) 2 Encoder Decoder 1 Decoder 2 P S f r a g r e p l a c e m e n t s X n 1 Y n 1 ˆ X n 1 ˆ Y n 1 Fig. 1. Complementary del iv ery problem Slepian-W olf and W yner-Ziv coding pr oblems, the cod ing problem s inv olv ing jo int encodin g pr ocesses h as been also explor ed (e.g . [7]– [10]). This p aper de als w ith a specific co ding p roblem inv olving joint encoding, which is called complementary delivery coding . The blo ck diagram of the comple mentary delivery co ding is dep icted in Fig. 1. T he encod er o bserves messages emitted fr om two correla ted sou rces, and delivers these messages to two de stinations ( decoder 1 and 2). Each deco der repro duces on e of two m essages using the o ther message as the sid e info rmation. Both lossless and lossy co nfiguratio ns hav e b een co nsidered. Th e lossless complementar y deliv ery co ding can be r egarded as a special case of the co ding prob lem in vestigated by Csisz ´ ar and K ¨ orner [1 1] and W yner, W olf and W illems [10]. Kimur a et al. [12], [13] p roposed a u niversal coding scheme fo r lossless comp lementary delivery based on graph coloring . The lossy complem entary delivery pro blem was in vestigated b y Kimur a an d Uyematsu [ 14], [15 ]. In this pap er , we propo se un iv ersal co ding schemes for lossless an d lossy complem entary d eliv ery coding problem s. At first, we propose a simple construc tion o f the l ossless complementar y delivery c ode based o n Sl epian- W olf cod es. T he key idea of our co ding schem e is as fo llows. W e prep are two Slepian -W olf codes. On e of two codes is a Slepian-W olf code fo r t he source X with side in formatio n Y , which is used as the cod e fr om the enco der to the de coder 1. T he other co de is a Slepian-W olf code f or the sou rce Y with side in formatio n X , which is used as the co de from th e encod er to the decoder 2. Each source is en coded separately b y th e co rrespon ding Slepian-W olf code, and then, the encod er sends the summ ation of two cod ew o rds. Notice that, by using the side informa tion Y , the dec oder 1 can c alculate the c odeword fr om the encoder to deco der 2. Th erefore, fro m the sum mation of two codewords, decoder 1 can extract the codeword to repr oduce X . The d ecoder 2 c an repr oduce Y an alogou sly . The above men tioned scheme allows us to apply any Slepian- W olf code to lossless complem entary delivery co ding. This drastically enriches the variety o f complemen tary d eliv ery coding. F o r example, we can u se universal Slepian - W olf co des (e.g . [16]–[ 18]). W e can also apply Slepian -W olf co des based on the low-density parity-ch eck matrices (e.g. [19], [20]) . I n this paper , we d emonstrate that a univ ersal lossless complementary deli very code, for which the error pr obability is exponentially tight in som e rate region, can b e constru cted by combin ing linear Slepian-W o lf codes [16 ]. Next, we propo se a universal lossy co mplemen tary delivery co ding schem e based on W y ner-Zi v code s [ 6]. Our scheme is uni versal in the sense th at it d oes n ot depen d on the joint probability distribution o f the co rrelated sources. While ou r coding scheme can not attain the o ptimal rate-d istortion trade-off in general, the rate -loss is up per boun ded by a u niversal co nstant under some m ild cond itions. Moreover , our scheme allows us to con struct a un iv ersal lossy 3 1 2 3 4 5 6 0 P S f r a g r e p l a c e m e n t s X X X X Y Y Y Y Fig. 2. The source node 0 observes the correlated sources ( X , Y ) , and sends the message to the sink nodes 5 and 6 over the network. The node 3 (resp. 5 , 6 ) corresponds to the encoder (resp. decoder 1,2) in Fig. 1. compleme ntary d eliv ery cod e by u sing (non -universal) W yner-Zi v codes (e.g . [21]– [23]) . The c omplemen tary delivery co ding can b e regard ed as a special case of the network c oding [24] , [25] . Let us consider the n etwork dep icted in Fig. 2 . Th e source no de 0 observes the messages em itted from th e correlated sources ( X , Y ) , and sends th e m essage to the sink nod es 5 an d 6 over the network. Assume that the all ed ges except the edg e b etween the n odes 3 a nd 4 h av e su fficiently large capacity , and thus, the outpu t from X (resp. Y ) can be delivered to th e nodes 3 an d 6 ( resp. 3 and 5 ). The pro blem is to find the m inimum cap acity b etween the nodes 3 an d 4 satisfying that the co deword neede d to r eprod uce X an d Y can b e delivered to the nodes 5 and 6 . Then, this problem can be r egarded a s the complementary d eliv ery problem depicted in Fig. 1. The node 3 (resp. 5 , 6 ) co rrespon ds to the encod er (resp. d ecoder 1,2) . The co ding prob lem o f cor related source s over a n etwork was studied by Han [ 26]. In the rec ent y ears, conside rable attentio ns h av e been devoted to Slepian- W olf co ding over a network (e.g . [2 7]–[3 1]). T his pap er shows that, f or th e specific ne twork depicted in Fig. 2, the optimal code can be c onstructed by only comb ining two Slepian -W olf codes. Further, the lossy c omplemen tary deliv ery investigated in this paper can be seen as a special case of lossy co ding of cor related sources over a network, which is no t so studied well as the lossless case. This paper is o rganized as fo llows. I n Section I I, we introdu ce definitions and notations used in th is pap er . I n Section III, lossless complemen tary delivery co ding is considered. W e propo se a simp le construction of the lossless compleme ntary d eliv ery c ode ba sed on Slep ian-W olf co des. Furth er , we pr opose another simple coding sch eme which can work in a specific case. In Section IV, lossy com plementary d eliv ery codin g is co nsidered. W e propo se a universal lossy comp lementary delivery coding schem e based on W yner-Ziv codes. In Section V, we present o ur conclusion s and some directio ns for fu rther work. I I . P R E L I M I N A RY W e denote by N a set of positi ve integers { 1 , 2 , . . . } . For a finite set S , | S | de notes t he cardinality of S . Througho ut this p aper, we take all log a nd exp to the base 2. W e d enote r andom variables by u pper case letters such as X . Their sample values (r esp. alphabets) are denoted by the c orrespon ding lower case letters s uch as x (resp. callig raph letters such as X ). For a ra ndom variable X , P X denotes the proba bility distribution of X . Similarly , for a pair of rando m variables ( X , Y ) , the jo int distribution is denoted by P X Y and the cond itional d istribution of Y give X is written by P Y | X . For each n ∈ N , X n denotes a rand om n -vecto r ( X 1 , X 2 , . . . , X n ) , and x n = ( x 1 , . . . , x n ) 4 denotes a specific sample value in X n which is the n -th Car tesian prod uct of X . A su bstring of x n is written as x j i = ( x i , x i +1 , . . . , x j ) fo r i ≤ j . When the dimension is clear from the co ntext, vectors will b e d enoted by boldface letters such as x ∈ X n . A d iscrete m emoryle ss sour ce (DMS) is a seq uence X △ = { X i } ∞ i =1 of indepen dent a nd ide ntically distributed (i.i.d.) copies of a r andom variable X . For simp licity , we call a DMS X = { X i } ∞ i =1 as a sour ce X . In th is paper, informa tion th eoretic quantities will be denoted following the usual conventions of the inform ation theory liter ature (see, e.g. [32], [3 3]). The entr opy rate of a sou rce X is deno ted by H ( X ) . For a pair ( X, Y ) of co rrelated sour ces X an d Y , th e condition al e ntr opy of Y given X is denoted by H ( Y | X ) , and the mutual inform ation between X and Y is denoted by I ( X ; Y ) . The r elative e ntr opy or dive r gence between two prob ability distributions P and Q is deno ted b y D ( P k Q ) . For a given pa ir ( x , y ) ∈ ( X × Y ) n of seque nces, the joint type of ( x , y ) is d efined as the empirical distribution Q xy of ( x , y ) , that is, Q xy ( a, b ) = |{ 1 ≤ i ≤ n : x i = a, y i = b }| n for all ( a, b ) ∈ X × Y [33]. Let P n ( X × Y ) b e the set o f all join t ty pes of sequen ces in ( X × Y ) n . By th e type counting lemm a [33, Lemm a 1.2.2 ], we have |P n ( X × Y ) | ≤ ( n + 1) |X ||Y | . (1) Hence, we can d efine an injection ι n : P n → { 1 , 2 , . . . , ( n + 1) |X ||Y | } . ι n ( P ˆ X ˆ Y ) is called the in dex assign ed to P ˆ X ˆ Y ∈ P n ( X × Y ) . I I I . L O S S L E S S C O M P L E M E N TA RY D E L I V E RY A. P r evious results In this sub section, we for mulate the lossless c omplemen tary deli very prob lem and show a funda mental bound of the co ding rate. Definition 1: A lossless complemen tary delivery code o f b lock len gth n is de fined by a triple o f m apping s ( f n , φ (1) n , φ (2) n ) where f n : X n × Y n → M n , φ (1) n : M n × Y n → X n , φ (2) n : M n × X n → Y n , where M n = { 1 , 2 , . . . , k f n k} and k f n k < ∞ . Definition 2: For a gi ven pair ( X, Y ) of correlated sources X and Y , a rate R is said to be losslessl y-achievable 5 if ther e exists a sequ ence { ( f n , φ (1) n , φ (2) n ) } ∞ n =1 of code s satisfy ing lim sup n →∞ 1 n log k f n k ≤ R, lim sup n →∞ Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o = 0 , lim sup n →∞ Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o = 0 . As a special case of results of [ 11] and [10] , it can be shown that the in fimum of the losslessly-ach iev ab le rate is given by max { H ( X | Y ) , H ( Y | X ) } . Kimura et a l. [12 ] pr oposed the universal co ding schem e based on graph coloring which can achieve any rate R > max { H ( X | Y ) , H ( Y | X ) } . Theor em 1 (Lossless coding the or e m; dir ect pa rt [12]): For a gi ven rate R , there exists a sequence { ( f n , φ (1) n , φ (2) n ) } ∞ n =1 of a code such that fo r any ( X , Y ) , lim sup n →∞ 1 n log k f n k ≤ R and Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o + Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o ≤ e x p ( − n " min P ˆ X ˆ Y ∈ S n ( R ) D ( P ˆ X ˆ Y k P X Y ) − ζ n #) where S n ( R ) △ =  P ˆ X ˆ Y ∈ P n ( X × Y ) : max { H ( ˆ X | ˆ Y ) , H ( ˆ Y | ˆ X ) } > R  and ζ n △ = 1 n {|X × Y | log ( n + 1) + 1 } → 0 ( n → ∞ ) . On the other ha nd, the next theor em shows tha t the er ror exponent of th e code ap peared in Th eorem 1 is tight. Theor em 2 (Lossless coding the or e m; conver se part [12]): For any co de ( f n , φ (1) n , φ (2) n ) satisfying (1 /n ) log k f n k = R , we have Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o + Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o ≥ exp ( − n " min P ˆ X ˆ Y ∈ S n ( R + ζ n ) D ( P ˆ X ˆ Y k P X Y ) + ζ n #) . (2) 6 B. Un iversal coding ba sed o n S lepian-W olf co des As shown in Section III-A, th e cod ing scheme prop osed in [12 ] is universal and optimal. Howev er , it r equires the exponentially lar ge coding table. In this subsection, we propose a simple coding scheme based on Slepian-W olf codes. At first, we consider Slepian-W olf cod ing problem of a sou rce X with side inf ormation Y . A Slepian-W o lf code of block length n for a source X with side informa tion Y is defined by a pair o f m appings ( g (1) n , ψ (1) n ) where g (1) n : X n → ¯ M n , ψ (1) n : ¯ M n × Y n → X n , and ¯ M n = { 1 , 2 , . . . , k g (1) n k} . Similar ly , we can defin e a Slepian-W olf code ( g (2) n , ψ (2) n ) fo r a source Y with side inform ation X . Th e next lemma gives a simple constru ction of a lo ssless com plementary delivery co de from Slepian-W olf co des. Lemma 1: For g iv en Slepian-W olf co des ( g (1) n , ψ (1) n ) and ( g (2) n , ψ (2) n ) , there exists a lo ssless com plementar y delivery code ( f n , φ (1) n , φ (2) n ) such th at k f n k ≤ max {k g (1) n k , k g (2) n k} and Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o ≤ Pr n X n 6 = ψ (1) n  g (1) n ( X n ) , Y n o , Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o ≤ Pr n Y n 6 = ψ (2) n  g (2) n ( Y n ) , X n o . Pr oof: Let M n = max {k g (1) n k , k g (2) n k} and define f n , φ (1) n , and φ (2) n by f n ( x , y ) △ = g (1) n ( x ) ⊕ g (2) n ( y ) , φ (1) n ( m, y ) △ = ψ (1) n  m ⊖ g (2) n ( y ) , y  , φ (2) n ( m, x ) △ = ψ (2) n  m ⊖ g (1) n ( x ) , x  , where ⊕ (resp. ⊖ ) den otes the add ition (re sp. sub traction) in m odulo M n arithmetic. The lemma follows fro m th e construction o f the cod e ( f n , φ (1) n , φ (2) n ) . Lemma 1 allo ws us to apply an y Slepian-W olf code to lo ssless complementary deli very problem. This d rastically enriches the variety of co mplemen tary delivery c oding. In the rem aining part of this subsection , we d emonstrate that a u niv ersal code which achieves the optimal rate max { H ( X | Y ) , H ( Y | X ) } can b e co nstructed by apply ing universal liner Slepian- W olf codes [1 6] to Le mma 1. 7 Theor em 3: Assume that X = Y and X is a Galois field. Fix k ( k ≤ n ) and let R = ( k/ n ) log |X | . There exists a seque nce { ( f n , φ (1) n , φ 2 n ) } ∞ n =1 of lossless compleme ntary d eliv ery co des such that for any ( X , Y ) , 1 n log k f n k = R and Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o ≤ exp  − n  e 1 r ( R, P X Y ) − ε n  Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o ≤ exp  − n  e 2 r ( R, P X Y ) − ε n  where ε n △ = 2 log ( n + 1) n |X | 2 |Y | 2 and e 1 r ( R, P X Y ) △ = min P ˜ X ˜ Y  D ( P ˜ X ˜ Y k P X Y ) +    R − H ( ˜ X | ˜ Y )    +  e 2 r ( R, P X Y ) △ = min P ˜ X ˜ Y  D ( P ˜ X ˜ Y k P X Y ) +    R − H ( ˜ Y | ˜ X )    +  where th e m inimization is over all dummy r andom variables ˜ X an d ˜ Y with joint distribution P ˜ X ˜ Y , an d | t | + △ = max( t, 0) . Pr oof: Csisz ´ ar [ 16] showed that there exists a linear Slepian-W olf c ode ( g (1) n , ψ (1) n ) for a source X with side informa tion Y such that g (1) n : X n → X k and for any ( X , Y ) , Pr n X n 6 = ψ (1) n  g (1) n ( X n ) , Y n o ≤ e x p  − n  e 1 r ( R, P X Y ) − ε n  . Similarly , there exists a linear Slep ian-W olf code ( g (2) n , ψ (2) n ) fo r a source Y with side info rmation X such that g (2) n : Y n → Y k and for any ( X , Y ) , Pr n Y n 6 = ψ (2) n  g (2) n ( Y n ) , X n o ≤ e x p  − n  e 2 r ( R, P X Y ) − ε n  . By app lying two codes ( g (1) n , ψ (1) n ) and ( g (2) n , ψ (2) n ) to Lemma 1, we have th e theorem . Remark 1: As mentioned in Section I, T heorem 3 can be seen as a result of Slepian -W olf coding over a specific network (see Fig. 2). In [31 , Theorem 6], Ho e t al. also applied the linear coding ap proach in [16] to Slepian-W olf coding over a network. Howe ver , ther e are some d ifferences between ou r results an d th e result of [3 1]. While Ho 8 et al. consider ed more gen eral network s than the n etwork dep icted in Fig. 2, T heorem 6 of [31] dealt with the special case where th ere exists o nly one recei ver . Hence, o ur result, Th eorem 3, cannot be derived only by applying Theorem 6 of [3 1] to th e network depicted in Fig. 2. Moreover , Le mma 1 allows us to ap ply n ot only liner but also various Slepian-W olf code s to n etwork co ding. Fur ther, the techniq ue used in the proo f of Lemm a 1 can b e applied to the lossy ca se which was not co ncerned in [31] (see Rem ark 4). Remark 2: In [12], th e co ding scheme ba sed o n graph co loring is ap plied to variable-rate cod ing f or lossless compleme ntary delivery pro blem. In the sam e w ay as the appro ach in [12], our codin g sche me c an be also modified and applied to variable-rate coding. For a given pair of sequ ences ( x , y ) to b e encode d, let ˆ X and ˆ Y be random variables such that P ˆ X ˆ Y is identical to th e join type of ( x , y ) . Let R = max { H ( ˆ X | ˆ Y ) , H ( ˆ Y | ˆ X ) } and ( f n , φ (1) n , φ (2) n ) be a fixed-r ate lossless complem entary delivery code such that (1 /n ) lo g k f n k = R . If x = φ (1) n ( f n ( x , y ) , y ) and y = φ (2) n ( f n ( x , y ) , x ) , then , the e ncoder sends the codeword consisting of the flag b it “0”, the ind ex ι n ( P ˆ X ˆ Y ) o f P ˆ X ˆ Y , and f n ( x , y ) . This c odeword can be r epresented by using at mo st 1 + |P n ( X × Y ) | + R b its. On the other hand, if x 6 = φ (1) n ( f n ( x , y ) , y ) or y 6 = φ (2) n ( f n ( x , y ) , x ) , then, th e enco der sends the co dew ord consisting o f the flag bit “1” an d ( x , y ) , which can be represen ted by using 1 + ⌈ n log |X × Y |⌉ bits. T he o verflow prob ability o f the coding rate of this scheme can be bounded in the s ame way as an error pr obability of fixed-rate co ding (see [1 2] fo r more details). Hence, it can be shown that, by usin g Slepian-W olf code s, we can constru ct a univ ersal variable-rate lossless complemen tary deli very code for which the coding rate is smaller than or equal to max { H ( X | Y ) , H ( Y | X ) } asymptotically almo st su rely . Now , we in vestigate the tig htness of the erro r expo nent of the p roposed schem e. I t is known th at e 1 r ( R, P X Y ) = min H ( ˜ X | ˜ Y ) ≥ R D ( P ˜ X ˜ Y k P X Y ) e 2 r ( R, P X Y ) = min H ( ˜ Y | ˜ X ) ≥ R D ( P ˜ X ˜ Y k P X Y ) if R ≤ R i cr ( i = 1 , 2 ), wher e R i cr = R i cr ( P X Y ) is the largest R for which the cu rve e i r ( R, P X Y ) meets its sup porting line of slope one [16]. Hence, as a corollary of Theorem 3, we have Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o + Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o ≤ 2 exp  − n min P ˜ X ˜ Y D ( P ˜ X ˜ Y k P X Y ) − ε n  (3) for R such that max { H ( X | Y ) , H ( Y | X ) } ≤ R ≤ min i =1 , 2 R i cr . By c omparin g (3 ) with (2), it can be seen that th e error bound (3) is e xpon entially tigh t for R s uch that max { H ( X | Y ) , H ( Y | X ) } ≤ R ≤ min i =1 , 2 R i cr . On the other hand, th e err or expone nt b ound for large rates can be imp roved in the same way as impr oving the er ror expon ent of Slepian- W olf codin g. Theor em 4: Assume that X = Y and X is a Galois field. Fix k ( k ≤ n ) and let R = ( k/ n ) log |X | . There exists 9 a seque nce { ( f n , φ (1) n , φ 2 n ) } ∞ n =1 of lossless compleme ntary d eliv ery co des such that for any ( X , Y ) , 1 n log k f n k = R and Pr n X n 6 = φ (1) n ( f n ( X n , Y n ) , Y n ) o ≤ e x p  − n  e 1 x ( R, P X Y ) − ε n  Pr n Y n 6 = φ (2) n ( f n ( X n , Y n ) , X n ) o ≤ e x p  − n  e 2 x ( R, P X Y ) − ε n  where e 1 x ( R, P X Y ) △ = min ˜ X : H ( ˜ X ) ≥ R ( E ˜ X " − log X x,y q P X Y ( x, y ) P X Y ( x ⊖ ˜ X , y ) # + R − H ( ˜ X ) ) e 2 x ( R, P X Y ) △ = min ˜ Y : H ( ˜ Y ) ≥ R ( E ˜ Y " − log X x,y q P X Y ( x, y ) P X Y ( x, y ⊖ ˜ Y ) # + R − H ( ˜ Y ) ) where ⊖ denotes the subtrac tion in the field X (= Y ) and E ˜ X (resp. E ˜ Y ) denotes the exp ectation with resp ect to P ˜ X (resp. P ˜ Y ). Pr oof: By using Slepian- W olf codes wh ich attain the expu rgated b ound [16, Theo rem 3 ], we can prove the theorem in the same way as T heorem 3. C. Binary symmetric case In this subsection, we pro pose anothe r simple coding scheme for lo ssless com plementar y delivery p roblem which can work in a specific case. L et X = Y = { 0 , 1 } , and consider a b inary symme tric source with p arameter p ( 0 ≤ p ≤ 1 / 2 ), that is, P X Y ( xy ) =      1 − p 2 , if x = y , p 2 , if x 6 = y . In this case, a simple uni versal lossless code g iv es an optimal lossless complementary deli very scheme. For a g iv en x ∈ X n and y ∈ Y n , let w △ = x ⊕ y , where ⊕ deno tes the add ition in mo dulo 2 arithmetic. Then, w can be regarded as an ou tput from the so urce W △ = X ⊕ Y , which satisfies that P W (0) = 1 − p and P W (1) = p . It is well known that (see e.g. [33]) there exists a un iv ersal lossless code ( ¯ f n , ¯ φ n ) with rate R such that lim n →∞ Pr  W n 6 = ¯ φ n  ¯ f n ( W n )  = 0 10 provided that R ≥ h ( p ) , wh ere h is the binary entropy fu nction d efined as h ( t ) △ = − t log t − (1 − t ) lo g(1 − t ) . By using ( ¯ f n , ¯ φ n ) , we can define the code ( f n , φ (1) n , φ (2) n ) as f n ( x , y ) △ = ¯ f n ( x ⊕ y ) , φ (1) n ( m, y ) △ = ¯ φ n ( m ) ⊕ y , φ (2) n ( m, x ) △ = ¯ φ n ( m ) ⊕ x . By the constru ction of the co de, this simple co de ( f n , φ (1) n , φ (2) n ) is universal. Further, it ach iev es the o ptimal rate h ( p ) = max { H ( X | Y ) , H ( Y | X ) } since h ( p ) = H ( X | Y ) = H ( Y | X ) . Especially , if ( ¯ f n , ¯ φ n ) is a lo ssless code f or which the error expo nent is tigh t (e.g. a code appear ed in [ 33]), then, th e error e xpon ent of the code ( f n , φ (1) n , φ (2) n ) based o n ( ¯ f n , ¯ φ n ) is also tight, that is, ( f n , φ (1) n , φ (2) n ) attains the erro r expo nent appeared in (2). Furthe rmore, in the same way , we can also ap ply lo ssy cod es to con struct a lossy complemen tary delivery co de (See the proo f of Theorem 8 in App endix B ). I V . L O S S Y C O M P L E M E N TA RY D E L I V E RY A. P r evious results In this sub section, we formulate th e lo ssy com plementar y delivery problem an d s how a funda mental boun d o f th e coding rate. Let ˆ X and ˆ Y b e re construction alp habets, and d (1) : X × ˆ X → [0 , d (1) max ] an d d (2) : Y × ˆ Y → [0 , d (2) max ] be single-letter distortion function s ( d ( i ) max < ∞ ( i = 1 , 2) ). Then, for each n ∈ N , the norm alized distortion d (1) n ( x , ˆ x ) between x ∈ X n and ˆ x ∈ ˆ X n is d efined as d (1) n ( x , ˆ x ) = 1 n n X i =1 d (1) ( x i , ˆ x i ) . For y ∈ Y n and ˆ y ∈ ˆ Y n , d (2) n ( y , ˆ y ) is defined similarly . Now , w e define c odes f or lo ssy compleme ntary d eliv ery problem . Definition 3: A lossy co mplementary delivery code of block length n is defined by a triple of mapping s ( f n , φ (1) n , φ (2) n ) where f n : X n × Y n → M n , φ (1) n : M n × Y n → ˆ X n , φ (2) n : M n × X n → ˆ Y n , where M n = { 1 , 2 , . . . , k f n k} and k f n k < ∞ . Next, we defin e the achievability of rate and th e optimal ra te attained by lossy co mplementar y delivery cod ing. Definition 4: For a gi ven ( X, Y ) and a distortion pair (∆ (1) , ∆ (2) ) , a rate R is s aid to be ( ∆ (1) , ∆ (2) ) -achievable 11 if ther e exists a sequ ence { ( f n , φ (1) n , φ (2) n ) } ∞ n =1 of code s satisfy ing lim sup n →∞ 1 n log k f n k ≤ R, lim sup n →∞ E X Y h d (1) n  X n , φ (1) n ( f n ( X n , Y n ) , Y n ) i ≤ ∆ (1) , lim sup n →∞ E X Y h d (2) n  Y n , φ (2) n ( f n ( X n , Y n ) , X n ) i ≤ ∆ (2) , where E X Y denotes the expectation with respect to P X Y . Definition 5: For a pair o f sources ( X , Y ) and a pair of distortio ns (∆ (1) , ∆ (2) ) , let R ∗ ( X, Y | ∆ (1) , ∆ (2) ) △ = inf n R : R is (∆ (1) , ∆ (2) ) -achiev able o . Kimura and Uyematsu [14 ], [ 15] revealed th e optimal ach iev ab le rate f or the lossy co mplemen tary delivery . Theor em 5 (Lossy cod ing theor em [14], [15 ]): For a given ( X , Y ) , R ∗ ( X, Y | ∆ (1) , ∆ (2) ) = min P U | X Y [max { I ( X ; U | Y ) , I ( Y ; U | X ) } ] where the m inimization is over all the auxiliary rando m variable U satisfying th e fo llowing prop erties: 1) P X Y U ( x, y , u ) = P X Y ( x, y ) P U | X Y ( u | x, y ) , 2) U takes a value over an alp habet U satisfying |U | ≤ |X × Y | + 2 , and 3) there ar e function s ϕ (1) : U × Y → ˆ X and ϕ (2) : U × X → ˆ Y satisfying ∆ (1) ≥ E X Y U h d (1) ( X, ϕ (1) ( U, Y )) i , ∆ (2) ≥ E X Y U h d (2) ( Y , ϕ (2) ( U, X )) i . B. Un iversal coding ba sed o n W yner-Ziv codes The coding scheme appe ared in the direct par t of th e proof of Theo rem 5 d epends on the joint distribution P X Y of ( X , Y ) . W e prop ose a lo ssy co mplemen tary delivery coding scheme wh ich does no t depend on the joint distribution. At first, we con sider W yne r-Zi v codin g pro blem o f a source X with side info rmation Y und er the distortion constraint ∆ (1) associated with the d istortion measure d (1) : X × ˆ X → [0 , d (1) max ] . A W yner-Zi v code o f block le ngth n fo r a sou rce X with side info rmation Y is defin ed by a pair o f mapp ings ( g (1) n , ψ (1) n ) wh ere g (1) n : X n → ¯ M n , ψ (1) n : ¯ M n × Y n → ˆ X n , and ¯ M n = { 1 , 2 , . . . , k g (1) n k} . Define R W Z ( X, Y | d (1) , ∆ (1) ) by R W Z ( X, Y | d (1) , ∆ (1) ) △ = min P U | X { I ( X ; U ) − I ( Y ; U ) } 12 where the m inimization is over all the rand om variables U satisfying the following pr operties: 1) P X Y U ( x, y , u ) = P U | X ( u, x ) P X Y ( x, y ) , 2) |U | ≤ |X | + 1 , an d 3) there exists a functio n ϕ : U × Y → ˆ X satisfying that E X Y U [ d (1) ( X, ϕ ( U, Y ))] ≤ ∆ (1) . (4) T o simplify th e notation, we den ote R W Z ( X, Y | d (1) , ∆ (1) ) by R (1) W Z (∆ (1) , P X Y ) . It is k nown that the optimal coding rate which can be achieved b y a W yne r-Zi v code fo r a sou rce X w ith side inf ormation Y under the distortion constrain t ∆ (1) is g iv en b y R (1) W Z (∆ (1) , P X Y ) . Theor em 6 ( [6], [33 ]): For any δ > 0 and any ( X , Y ) , there exists l 0 = l 0 ( δ, d (1) max , |X | , |Y | ) such that fo r any l ≥ l 0 there exists a code ( g (1) l , ψ (1) l ) satisfying 1 l log k g (1) l k ≤ R (1) W Z (∆ (1) , P X Y ) + δ and Pr n d (1) l  X l , ψ (1) l  g (1) l ( X l ) , Y l  > ∆ (1) o ≤ δ. Remark 3: While ( g (1) l , ψ (1) l ) depend s on ( X, Y ) , the virtue of the meth od of types [33] allows us to choose the block size l which d epends only o n δ , d (1) max , |X | , and |Y | . In a similar manner to th e above discussion, we can consider W yner-Zi v coding pr oblem of a sou rce Y with side informa tion X u nder the distortion constraint ∆ (2) associated with the distortion measure d (2) : Y × ˆ Y → [0 , d (2) max ] . W e d enote R W Z ( Y , X | d (2) , ∆ (2) ) by R (2) W Z (∆ (2) , P X Y ) . Now , we describ e a universal lossy com plementar y delivery coding schem e b ased on W yn er-Zi v codes. Fix γ > 0 and R > 0 . Choose δ > 0 such tha t δ < γ / 4 and 4 δ d ( i ) max < γ ( i = 1 , 2 ). By Theo rem 6, we can ch oose l = l ( δ, d (1) max , d (2) max , |X | , |Y | ) suffi ciently large so that, for any correlated sou rces ( ˆ X , ˆ Y ) , there are ( g (1) l , ψ (1) l ) and ( g (2) l , ψ (2) l ) satisfying 1 l log k g ( i ) l k ≤ R ( i ) W Z (∆ ( i ) , P ˆ X ˆ Y ) + δ, i = 1 , 2 (5) and Pr { ˆ X l ˆ Y l / ∈ Γ l ( P ˆ X ˆ Y ) } ≤ 2 δ (6) where Γ l ( P ˆ X ˆ Y ) △ = ( ( x l , y l ) : d (1) l  x l , ψ (1) l ( g (1) l ( x l ) , y l )  ≤ ∆ (1) , d (2) l  y l , ψ (2) l ( g (2) l ( y l ) , x l )  ≤ ∆ (2) ) . 13 Note th at ( g ( i ) l , ψ ( i ) l ) ( i = 1 , 2 ) may depend on P ˆ X ˆ Y . Esp ecially , for each joint type P ˆ X ˆ Y , we can choose th e pair of code s { ( g ( i ) l , ψ ( i ) l ) } i =1 , 2 satisfying (5) and (6). For each n ∈ N , fix a corresp onden ce between P n ( X × Y ) and th e set of pair s of W yner-Ziv cod es so th at the pair { ( g ( i ) l , ψ ( i ) l ) } i =1 , 2 correspo nding to P ˆ X ˆ Y ∈ P n ( X × Y ) satisfies (5) and (6) 1 . Let ¯ M l △ = 2 l ( R +3 γ / 4) . Let n be so large that n > l and ( |X | |Y | / n ) log( n + 1 ) < γ / 4 . In the fo llowings, we assume that n = T l ( T ∈ N ) for simplicity . At first, we describ e the enco ding scheme. For a g iv en ( x n , y n ) , find P ˆ X ˆ Y ∈ P n such that 2 max i =1 , 2 R ( i ) W Z (∆ ( i ) , P ˆ X ˆ Y ) ≤ R + γ / 2 (7) and    n t : ( x ( t +1) l tl +1 , y ( t +1) l tl +1 ) / ∈ Γ l ( P ˆ X ˆ Y ) o    ≤ 4 δ T . (8) Note that P ˆ X ˆ Y is no t necessarily the joint type o f ( x n , y n ) . If there is n o P ˆ X ˆ Y ∈ P n satisfying (7) an d ( 8), then err or is declared. If there exists P ˆ X ˆ Y ∈ P n satisfying (7) and ( 8), then find the pair o f W yner-Ziv codes correspo nding to P ˆ X ˆ Y , that is, { ( g ( i ) l , ψ ( i ) l ) } i =1 , 2 satisfying (5) and (6). Parse ( x n , y n ) into T blo cks of size l , and then, enc ode each bloc k as m t = g (1) l ( x ( t +1) l tl +1 ) ⊕ g (2) l ( y ( t +1) l tl +1 ) , t = 0 , . . . , T − 1 where ⊕ denotes the add ition in mo dulo ¯ M l arithmetic (N ote that ¯ M l ≥ max i =1 , 2 k g ( i ) l k ). Then, the codeword assigned to ( x n , y n ) is ( ι ( P ˆ X ˆ Y ) , m 0 , . . . , m T − 1 ) . Since (1) holds, the code word can b e described b y using n ( R + γ ) bits becau se log( n + 1) |X ||Y | + T log ¯ M l ≤ n ( R + γ ) . Next, we descr ibe the dec oding scheme. W e only describe the decod er φ (1) n which o utputs the r eprodu ction sequence ˆ x n ∈ ˆ X n by using th e co deword ( ι ( P ˆ X ˆ Y ) , m 0 , . . . , m T − 1 ) and the side information y n . The d ecoder φ (2) n can be defined analo gously . φ (1) n decodes the ind ex ι ( P ˆ X ˆ Y ) at first, an d then, com putes ˆ x n as ˆ x ( t +1) l tl +1 △ = ψ (1) n  m t ⊖ g (2) l ( y ( t +1) l tl +1 ) , y ( t +1) l tl +1  , t = 0 , . . . , T − 1 where ψ (1) n and g (2) l are the mapping s c orrespon ding to P ˆ X ˆ Y , and ⊖ deno tes the su btraction in modulo ¯ M l arithmetic. The next theo rem shows th e perfor mance o f the cod ing scheme descr ibed above. Theor em 7: Fix γ > 0 and R > 0 . There exists a sequen ce { ( f n , φ (1) n , φ (2) n ) } ∞ n =1 of lossy comp lementary deli very codes which satisfies the following pro perty: If ( X , Y ) satisfies R ≥ max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) then, fo r sufficiently large n , 1 n log k f n k ≤ R + γ 1 If there are two or more pairs of codes s atisfyin g (5) and (6) for a joint type P ˆ X ˆ Y , then choose one of them arbitrarily and assign it to P ˆ X ˆ Y 2 If t here a re t wo or more joint types satisfying the conditi ons, choose one of them arbi trarily . 14 and Pr n d (1) n  X n , φ (1) n ( f n ( X n , Y n ) , Y n )  > ∆ (1) + γ o ≤ γ , Pr n d (2) n  Y n , φ (2) n ( f n ( X n , Y n ) , X n )  > ∆ (2) + γ o ≤ γ . Remark 4: The p roposed scheme is universal in the sense that th e sch eme d oes no t depen d on the prob ability distribution P X Y of ( X , Y ) . T o deal with some tech nical difficulties in ev a luating the perform ance o f the code, we ad opt the co ding scheme which parses the seq uence in to b locks of fixed length l and then encod es each block. On the other hand, if we know the the joint distribution P X Y , we can av oid techn ical difficulties. I n fact, a (n on- universal) lossy complementary deliv ery code can be con structed by co mbining tw o W yner-Zi v codes ( g (1) n , ψ (1) n ) and ( g (2) n , ψ (2) n ) in the same way as a lo ssless complementar y delivery cod e is constructed b y Slepian-W olf codes (Lemma 1). In the lo ssless case, we can u se u niversal Slep ian-W olf cod es to construct a universal lossless com plementar y delivery code. Howe ver , as long as the au thors know , no universal W y ner-Zi v c ode has been prop osed (While universal W y ner-Zi v cod ing was rec ently studied in [23], [3 4], it is assumed that the cond itional distribution P Y | X of the source is known). The pro of of the theorem will be g iv en in Append ix A . Our co ding sch eme allows us to constru ct a universal lossy complem entary delivery cod ing scheme based on (non- universal) W yner-Ziv codes. Especially , we can ap ply practical W y ner-Zi v codes (e.g. [21] –[23] ) to universal lossy complemen tary deli very . Ho wever , our scheme cannot attain the optimal rate appeared in Theorem 5 in general. Theor em 8: There exists ( X, Y ) such that max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) > R ∗ ( X, Y | ∆ (1) , ∆ (2) ) . Remark 5: In the proof of Theorem 8, we give an examp le where a simp le cod ing scheme app eared in Section III-C can a ttain the o ptimal rate R ∗ ( X, Y | ∆ (1) , ∆ (2) ) . See Append ix B for more d etails. On the o ther hand, we c an show that the loss of the codin g rate can be bou nded by a universal co nstant u nder some co nditions. Let X = ˆ X = { 1 , 2 , . . . , M (1) } an d Y = ˆ Y = { 1 , 2 , . . . , M (2) } . Su ppose that d ( i ) ( i = 1 , 2 ) is a balance d distor tion measure [3 5], that is, d ( i ) ( a, ˆ a ) = ¯ d ( i ) ( a ⊖ ˆ a ) , ∀ a, ˆ a ∈ { 1 , . . . , M ( i ) } for some ¯ d ( i ) : { 1 , . . . , M ( i ) } → [0 , d ( i ) max ] , wher e ⊖ den otes modu lo- M ( i ) subtraction. Let C ( i ) ( i = 1 , 2 ) be the minimax cap acity [ 36], defined as C ( i ) (∆ ( i ) ) = inf N : E N [ ¯ d ( i ) ( N )] ≤ ∆ ( i ) sup W : W ⊥ N , E W [ ¯ d ( i ) ( W )] ≤ ∆ ( i ) I ( W ; W ⊕ N ) where N and W a re rando m variables o n { 1 , 2 , . . . , M ( i ) } , ⊥ denotes statistical indepen dence, and ⊕ d enotes modulo - M ( i ) addition. The next theo rem gives the b ound on the rate-loss of ou r scheme. 15 Theor em 9: Suppose that X = ˆ X and Y = ˆ Y . I f d (1) and d (2) are balance d d istortion measures, th en max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) ≤ R ∗ ( X, Y | ∆ (1) , ∆ (2) ) + C ( i ∗ ) (∆ ( i ∗ ) ) where i ∗ = arg max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) . Proofs of Theorem 8 an d Theo rem 9 will be given in Append ix B. V . C O N C L U S I O N W e pro posed a u niversal lossless (resp. lossy) co mplementar y delivery cod ing schem e b ased on Slepian-W olf (resp. W yner-Ziv) codes. I t was demonstrated tha t a uni versal lossless complementary delivery code, f or wh ich e rror probab ility is expon entially tight, can b e constructed by o nly co mbining two linear Slepian- W olf codes. On the other ha nd, pro posed lo ssy comp lementary delivery co ding sche me can not attain the optim al rate g enerally , wh ile it d oes not depend on the distribution of the sou rce. The r ate-loss of our lossy co ding scheme was ev aluated. W e also pr opose another simple coding scheme which can work for a binary symm etric source. Further work inclu des extensions to gen eralized comp lementary delivery network [13] . Ano ther importan t work is to construct a u niversal lossy com plementary delivery code which attains the optima l rate. A P P E N D I X P R O O F S O F T H E O R E M S A. P r oof o f Theorem 7 Before proving The orem 7, we in troduce some lemmas. In this append ix, the variational distance between P X Y and P ˆ X ˆ Y is deno ted b y ρ  P X Y , P ˆ X ˆ Y  . Lemma 2: For any δ > 0 and l ∈ N , there exists ǫ 1 = ǫ 1 ( l, δ, |X | , |Y | ) > 0 su ch that if ρ  P X Y , P ˆ X ˆ Y  ≤ ǫ 1 then, ρ  P X l Y l , P ˆ X l ˆ Y l  ≤ δ. Pr oof: If ρ  P X Y , P ˆ X ˆ Y  ≤ ǫ 1 , then for any ( x l , y l ) ∈ ( X × Y ) l , P X l Y l ( x l , y l ) = l Y i =1 P X Y ( x i , y i ) ≤ l Y i =1  P ˆ X ˆ Y ( x i , y i ) + ǫ 1  ≤ l Y i =1 P ˆ X ˆ Y ( x i , y i ) + δ ( ǫ 1 , l ) = P ˆ X l ˆ Y l ( x l , y l ) + δ ( ǫ 1 , l ) 16 where δ ( ǫ 1 , l ) → 0 as ǫ 1 → 0 . Similarly , we h av e P X l Y l ( x l , y l ) ≥ P ˆ X l ˆ Y l ( x l , y l ) − δ ( ǫ 1 , l ) . Hence, we have the lemma. Lemma 3: 1) For any γ > 0 and any P ˆ X ˆ Y , ther e exists ζ > 0 satisfying R ( i ) W Z (∆ ( i ) , P ˆ X ˆ Y ) ≤ R ( i ) W Z (∆ ( i ) + ζ , P ˆ X ˆ Y ) + γ / 4 , i = 1 , 2 . 2) For any γ > 0 , ζ > 0 and a ny ( X, Y ) , there exists ǫ 2 > 0 such that if ρ  P X Y , P ˆ X ˆ Y  < ǫ 2 then R ( i ) W Z (∆ ( i ) + ζ , P ˆ X ˆ Y ) ≤ R ( i ) W Z (∆ ( i ) , P X Y ) + γ / 4 , i = 1 , 2 . Pr oof: The first part of the lemma f ollows fro m the fact th at R ( i ) W Z (∆ ( i ) , P ˆ X ˆ Y ) is continu ous in ∆ ( i ) [6]. By the d efinition of R (1) W Z (∆ (1) , P X Y ) , we can cho ose P U | X and ϕ satisfy ing R (1) W Z (∆ (1) , P X Y ) = I ( X ; U ) − I ( Y ; U ) and (4). If ρ  P X Y , P ˆ X ˆ Y  is sufficiently small, the n P ˆ X ˆ Y satisfies that I ( ˆ X ; U ) − I ( ˆ Y ; U ) ≤ I ( X ; U ) − I ( Y ; U ) + γ / 4 and E ˆ X ˆ Y U [ d (1) ( ˆ X , ϕ ( U, ˆ Y ))] ≤ ∆ (1) + ζ . Hence, we have R (1) W Z (∆ (1) + ζ , P ˆ X ˆ Y ) ≤ R (1) W Z (∆ (1) , P X Y ) + γ / 4 . Similarly , we can prove that R (2) W Z (∆ (2) + ζ , P ˆ X ˆ Y ) ≤ R (2) W Z (∆ (2) , P X Y ) + γ / 4 provided that ρ  P X Y , P ˆ X ˆ Y  is sufficiently small. Pr oof of Theor em 7: W e prove the theo rem b y showing that the cod e defined in Section I V -B satisfies the proper ty appe ared in th e theorem. Let δ > 0 an d l ∈ N b e num bers satisfying the condition s ap peared in the d escription of codin g schem e. Let n = T l ∈ N be sufficiently large. Suppose that there exists P ˆ X ˆ Y ∈ P n satisfying (8). Then, d (1) n  x n , φ (1) n ( f n ( x n , y n ) , y n )  ≤ 1 n n 4 T δ l d (1) max + T l ∆ (1) o = ∆ (1) + 4 δ d (1) max ≤ ∆ (1) + γ . Similarly , d (2) n  y n , φ (2) n ( f n ( x n , y n ) , x n )  ≤ ∆ (2) + γ . 17 Hence, to prove th e theorem, it is sufficient to show that, for s ufficiently lar ge n , we can find P ˆ X ˆ Y ∈ P n satisfying (7) and (8) with p robab ility g reater than 1 − γ . Choose ǫ 1 (resp. ζ , ǫ 2 ) satisfying Lemma 2 (resp. Lem ma 3) . Fix ǫ > 0 such that ǫ < ǫ i ( i = 1 , 2 ). I f n is sufficiently large, there exists a joint ty pe P ˆ X ˆ Y ∈ P n ( X × Y ) satisfying ρ  P X Y , P ˆ X ˆ Y  ≤ ǫ . (9) By Lemm a 3, we have R ( i ) W Z (∆ ( i ) , P ˆ X ˆ Y ) ≤ R ( i ) W Z (∆ ( i ) , P X Y ) + γ / 2 , i = 1 , 2 . Since R ≥ max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) , P ˆ X ˆ Y satisfies (7). In the f ollowings, we prove that P ˆ X ˆ Y also satisfies (8) with pr obability greater tha n 1 − γ . By Lemma 2, (6), an d (9), P X l Y l (Γ ∁ l ) ≤ P ˆ X l ˆ Y l (Γ ∁ l ) + δ ≤ 3 δ (10) where Γ ∁ l denotes the complemen t of Γ l ( P ˆ X ˆ Y ) . Let B t ( t = 0 , 1 , . . . , T − 1 ) b e rando m variables defin ed b y B t △ =      1 ,  X ( t +1) l tl +1 , Y ( t +1) l tl +1  ∈ Γ ∁ l , 0 , otherwise . Since (1 0) implies E X l Y l [ B i ] ≤ 3 δ , by the law of large n umbers, we have lim T →∞ Pr ( 1 T T − 1 X t =0 B t > 3 δ + δ ) = 0 . Hence, if n = T l is sufficiently large, th en we can find P ˆ X ˆ Y ∈ P n satisfying (7) and (8) with proba bility greater than 1 − γ . B. P r oofs of Theorem 8 and The or e m 9 Before provin g Theo rem 8 and Theo rem 9, we introdu ce some notation s. Let R (1) both (∆ (1) , P X Y ) be the optimal rate o f the lossy source co ding prob lem with side inf ormation at the enco der and the d ecoder [6] , tha t is, R (1) both (∆ (1) , P X Y ) △ = min P Z (1) | X Y I ( X ; Z (1) | Y ) where the min imization is with respect to a ll random variables Z (1) such that P X Y Z (1) ( x, y , z ) = P X Y ( x, y ) P Z (1) | X Y ( z | x, y ) is the probab ility distribution on X × Y × ˆ X satisfying P x,y ,z d (1) ( x, z ) P X Y Z (1) ( x, y , z ) ≤ ∆ (1) . Similarly , let R (2) both (∆ (2) , P X Y ) △ = min P Z (2) | X Y I ( Y ; Z (2) | X ) . Note that max i =1 , 2 R ( i ) both (∆ ( i ) , P X Y ) ≤ min P U | X Y [max { I ( X ; U | Y ) , I ( Y ; U | X ) } ] = R ∗ ( X, Y | ∆ (1) , ∆ (2) ) (11) 18 where min P U | X Y is taken over U satisfying th e pr operties appeared in Theorem 5. Pr oof of Theo r e m 8: Let X = Y = ˆ X = ˆ Y = { 0 , 1 } , and consid er a bin ary symm etric source with par ameter p ( 0 < p < 1 / 2 ). L et d (1) and d (2) be th e Ham ming d istortion m easure, that is d ( i ) ( x, ˆ x ) = 0 if x = ˆ x and d ( i ) ( x, ˆ x ) = 1 o therwise. Let ∆ (1) = ∆ (2) = ∆ ( ∆ < p ). For a g iv en x ∈ X n and y ∈ Y n , let w △ = x ⊕ y , wh ere ⊕ den otes the additio n in m odulo 2 arith metic. Then , w can be regard ed as an output from the source W △ = X ⊕ Y , which satisfies that P W (0) = 1 − p and P W (1) = p . It is k nown that [35] ther e exists a lossy code ( ¯ g n , ¯ ψ n ) with rate h ( p ) − h (∆) satisfying that lim n →∞ Pr n d (1) n  W n , ¯ ψ n ( ¯ g n ( W n )) > ∆  o = 0 . Based on ( ¯ g n , ¯ ψ n ) , define the co de ( f n , φ (1) n , φ (2) n ) as f n ( x , y ) △ = ¯ g n ( x ⊕ y ) , φ (1) n ( m, y ) △ = ¯ ψ n ( m ) ⊕ y , φ (2) n ( m, x ) △ = ¯ ψ n ( m ) ⊕ x . Since d (1) n ( x , φ (1) n ( m, y )) = d (1) n ( w , ¯ ψ n ( ¯ g n ( w ))) and d (2) n ( y , φ (2) n ( m, x )) = d (2) n ( w , ¯ ψ n ( ¯ g n ( w ))) , the co de ( f n , φ (1) n , φ (2) n ) satisfies the distor tion con straints. Th is fact ind icates that h ( p ) − h (∆) ≥ R ∗ ( X, Y | ∆ (1) , ∆ (2) ) . (12) Further, by the r esult of lossy sour ce codin g with side inf ormation at the encod er and deco der [6 ], we have R ( i ) both (∆ ( i ) , P X Y ) = h ( p ) − h (∆) , i = 1 , 2 . Since (1 1) holds, we h av e h ( p ) − h (∆) ≤ R ∗ ( X, Y | ∆ (1) , ∆ (2) ) . (13) By com bining (12) an d ( 13), we have h ( p ) − h (∆) = R ∗ ( X, Y | ∆ (1) , ∆ (2) ) . (14) On the other ha nd, by the r esult of W yner-Ziv codin g p roblem [6] , we h av e for each i = 1 , 2 , R ( i ) W Z (∆ ( i ) , P X Y ) = inf θ ,β { θ [ h ( p ∗ β ) − h ( β )] } (15) where the infimum is with resp ect to all θ , β such that 0 ≤ θ ≤ 1 , 0 ≤ β < p , and ∆ = θβ + (1 − θ ) p . ( 15) and (14) indicate th at max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) > R ∗ ( X, Y | ∆ (1) , ∆ (2) ) f or all 0 < p < 1 / 2 . 19 Pr oof o f Theo r em 9 : Since ( 11) holds, we can b ound the ra te loss a s max i =1 , 2 R ( i ) W Z (∆ ( i ) , P X Y ) − R ∗ ( X, Y | ∆ (1) , ∆ (2) ) ≤ R ( i ∗ ) W Z (∆ ( i ∗ ) , P X Y ) − max i =1 , 2 R ( i ) both (∆ ( i ) , P X Y ) ≤ R ( i ∗ ) W Z (∆ ( i ∗ ) , P X Y ) − R ( i ∗ ) both (∆ ( i ∗ ) , P X Y ) . It is kn own that the difference R ( i ∗ ) W Z (∆ ( i ∗ ) , P X Y ) − R ( i ∗ ) both (∆ ( i ∗ ) , P X Y ) is bounded by a universal constan t C ( i ∗ ) (∆ ( i ∗ ) ) [36]. R E F E R E N C E S [1] D. Slep ian and J. K. W olf, “Noisele ss codi ng of c orrelate d informat ion sourc es, ” IEE E T rans. Inf. Theory , vol. IT -19, no. 4, pp. 471–480, Jul. 19 73. [2] A. D. W yner , “On source coding with side information at the decoder , ” IEEE T rans. Inf. Theory , vol. IT -21, no. 3, pp. 294–300, May 1975. [3] R. F . Ahlswede and J. K ¨ orner , “Source coding wit h side informati on and a con verse for de graded broadcast channel s, ” IEE E T rans. Inf. Theory , vol. IT -21, no. 6, pp. 629–637, Nov . 1975 . [4] A. Sgarro, “Source coding with side information at sev eral de coders, ” IEEE T rans. Inf. Theory , vol. IT -23, no. 2, pp. 179–182, Mar . 197 7. [5] J . K ¨ orner and K. Marton, “Imag es of a set via two channe ls and their role in mult i-user communicati on, ” IE EE T rans. Inf . Theory , vol. IT -23, no. 6, pp. 751–761, Nov . 1977. [6] A. D. W yner and J. Ziv , “The rat e-distort ion function for sourc e coding wit h side informati on at the dec oder , ” IEEE T rans. Inf . Theory , vol. IT -22, no. 1, pp. 1 –10, Jan. 1976. [7] R. M. Gray and A. D. W yner , “Source coding for a simple network, ” Bell Syst. T ech. J. , vol. 53, pp. 1681–1721, No v . 1974. [8] H. Y amamoto, “Source coding theory for cascade and branching communication systems, ” IEEE T rans. Inf. Theory , vol. 27, no. 3, pp. 299–308, may 1981. [9] ——, “Source coding theo ry for a t riangula r communication system, ” IEEE T rans. Inf. Theory , vol . 4 2, n o. 3, pp. 8 48–853, m ay 1996. [10] A. D. W yner , J. K. W olf, an d F . M. J. Wil lems, “Communic ating via a p rocessing broa dcast satel lite. ” IEEE T rans. Inf . Theory , v ol. 48, no. 6, pp. 1243–1249, 200 2. [11] I. Csisz ´ ar and J. K ¨ orner , “T ow ards a general theory of source ne tworks, ” IEEE T rans. Inf. Theory , vol. IT -26, no. 2, pp. 155–1 65, Mar . 1980. [12] A. Kimura, T . Uyematsu, and S. Kuzuo ka, “Univ ersal coding for correlat ed sources with complement ary deli very , ” IEICE T rans. Fundamental s , vol. E90-A, n o. 9, pp. 1 840–1847, Sep. 2007. [13] A. Kimura, T . Uyematsu, S. Kuzuoka, and S. W atanabe, “Univ ersal source coding ove r generalize d complementary deli very networks, ” IEEE T rans. Inf. The ory , submit ted for publica tion. [Online]. A vai lable: htt p://arxi v .org/abs/0710.4987 [14] A. Kimura and T . Uyematsu, “Multitermin al source coding with complementary deli very , ” in P r oc. of Internati onal Symposium on Informatio n Theory and its Applications (ISIT A2006) , Seou l, K orea, Oct. 2006, p p. 1 89–194. [15] ——, “Multi terminal source coding with complementary de li very , ” IEICE T rans. F undamenta ls , submitt ed for publicati on. [16] I. Csi sz ´ ar, “Linear cod es for sources and source ne tworks: Error exponents, univ ersal codi ng, ” IEEE T rans. Inf. Theory , v ol. I T -28, no. 4, pp. 585–592, Jul. 1982. [17] Y . Oohama and T . S. Han, “Univ ersal coding for the Slepian-Wolf data compression system and the strong con verse theorem. ” IE EE T rans. Inf. Theory , v ol. 40, no. 6, pp. 1908–1919, No v . 1 994. [18] T . Uyematsu, “ An a lgebrai c construction of codes for Slepia n-Wolf source networks, ” IEEE T rans. I nf. Theory , v ol. 47, no. 7, pp. 3082– 3088, N ov . 2001. [19] A. D. Live ris, Z. Xiong, and C. N. Geor ghiades, “Compre ssion of binary s ources with s ide information at the decoder using LDPC codes, ” IEEE Commun. Lett. , vol. 6, no. 10, pp. 440–442, Oct. 2002. 20 [20] J. Muramatsu, T . Uyematsu, and T . W adayama, “Lo w-density parity-check matrices for coding of correlated sources, ” IEEE T rans. Inf. Theory , vol. 51, no. 10, pp. 3645–3654, Oc t. 2005. [21] R. Zamir , S . Shamai, and U. Erez, “Nested linear/la ttice codes for structured multite rminal binning, ” IEEE T rans. Inf. Theory , vol. 48, no. 6, pp. 1250–1276, Jun. 2002. [22] S. S. Pradhan and K. Ra mchandran, “Distrib uted s ource cod ing using syndr omes (DISCUS): De sign and construction, ” IEEE T rans. Inf. Theory , vol. 49, no. 3, pp. 626–643, Mar . 2003. [23] S. Jalali, S. V erd ´ u, and T . W eissman, “ A uni versal Wyner-Zi v scheme for discrete sources, ” in Proc . of 2007 IEEE International Symposium on I nformation Theory (ISIT2007) , Nice, Franc e, Jun. 2007. [24] R. Ahl swede, N. Cai, S.-Y . R. Li, and R. W . Y eung, “Ne twork information flow , ” IEEE T rans. Inf . Theory , vol. 46, no. 4, pp. 1204–1216, Jul. 20 00. [25] S. -Y . R. Li, R. W . Y eung, and N. Cai, “Linear netw ork cod ing, ” IEEE T rans. Inf. Th eory , v ol. 49, no. 2, pp. 371–381, Feb . 2003. [26] T . S. Han, “Slepian-Wol f-Cov er theorem for networks of chann els, ” Informatio n and Co ntr ol , vol. 47, no . 1, pp. 67 –83, Oct. 1980. [27] R. Cristescu, B. Beferull-Lozan o, and M. V etterl i, “Network ed Slepian-Wol f: theory , algorith ms, and scaling laws, ” IEEE T rans. Inf. Theory , vol. 51, no. 12, pp. 4057–4073, De c. 2005. [28] J. Barros and S. D. Serv etto, “Net work informat ion flow wi th corre lated sources, ” IEE E Tr ans. Inf. The ory , vol . 52, no. 1, pp. 1 55–170, Jan. 20 06. [29] Y . Wu, V . Stanko vi ´ c, Z . Xiong, and S.-Y . Kung, “On practical design for joint distrib uted source and network coding, ” in Proc . 1st W orkshop on Network Cod ing, Theory , and Appli cations (Ne tCod2005) , Ri va del Garda, Italy , Apr . 20 05. [30] A. Ramamoorthy , K. Jain, P . A. Chou, and M. Effros, “Separati ng distrib uted source coding from network coding, ” IEE E T rans. Inf. Theory , vol. 52, no. 6, pp. 2785–2795, Jun. 2006. [31] T . Ho, M. M ´ edard, R. Koette r , D. R. Karger , M. Eff ros, J. Shi, and B. Leong, “ A random linear netwo rk coding appro ach to mult icast, ” IEEE T rans. Inf. The ory , v ol. 52, no. 10, pp. 4413– 4430, Oc t. 2006 . [32] T . M. Cov er and J. A. Thomas, Eleme nts of Informatio n Theory . New Y ork: W iley , 1991. [33] I. Csisz ´ ar and J. K ¨ orner , Information The ory: C oding The ore ms for Discr ete M emoryless S ystems . New Y ork: Acade mic, 1981. [34] N. Merh av and J. Zi v , “On the Wyner-Zi v problem for i ndi vidual seq uences, ” IEEE T rans. Inf. Theory , v ol. 52, no. 3, pp . 86 7–873, Mar . 2006. [35] T . Berger , Rate Distortion Theory: A Mat hematica l Basis for Data Co mpressi on . New Jersey: Pre ntice-Ha ll, Inc., 197 1. [36] R. Zamir , “The rate loss in t he Wyner -Z i v pro blem, ” IEEE T rans. In f. Theory , vol . 42, no. 6, pp. 2073–2084, No v . 1996 .