Achieving the Quadratic Gaussian Rate-Distortion Function for Source Uncorrelated Distortions

Achie ving the Quadrati c Gaussian Rate-Di stortion Function for Source Uncorrelated Distortions Milan S. Derpich, Jan Østergaard, and Daniel E. Quev edo School of Electrical Engineering and Computer Science, The Un iv ersity of Newcastle, NSW 2308 , Australia milan.der pich@studentmail. newcastle.edu.au, jan.ostergaard@newcastle.edu.au, d quevedo@ieee.org Abstract — W e pr ov e a chieva bility o f the recently characterized quadratic Gaussian rate-distortion function (RDF) subject to th e constraint that the distortion is uncorr elated to the so urce. This result is based on shaped dit hered lattice quantization in the limit as the lattice d imension tends to infinity and holds for all positive distortions. It turns out that this uncorrelated distortion RDF can be r ealized ca usally . This feature, which stands in contrast to Shannon’ s RDF , is illustrated by causal transf orm coding. M oreo ve r , we pr ov e that b y using feed back noise shapi ng the uncorrelated distortion RDF can be achiev ed causally and with memoryless entropy codin g. Wh ilst achievability relies upon infinite dimensional quantizers, we pr ov e that the rate loss incurred in the finite dimensional case can be upper-bounded by the space filling l oss of the quantizer and, thus, is at most 0.254 bit/d imension. I . I N T RO D U C T I O N Shannon ’ s rate- distortion func tion R ( D ) for a stationary zero-mea n Gau ssian source X with m emory and under the MSE fid elity criterion ca n b e wr itten in a parametric form (the r ev erse water-filling solution) [ 1] R ( D ) = 1 2 π Z ω : S X ( ω ) >θ 1 2 log  S X ( ω ) θ  (1a) D = 1 2 π Z π − π S Z ( ω ) dω , (1b) where S X ( ω ) denotes the power spectr al density (PSD) of X and th e distortio n PSD S Z ( ω ) is given by S Z ( ω ) = ( θ, if S X ( ω ) > θ S X ( ω ) , otherwise . (1c) The water level θ is chosen such that the distor tion con- straint (1b) is satisfied. It is well kn own that in ord er to achieve Sha nnon’ s RDF in the q uadratic Gaussian case, th e distortion m ust be ind e- penden t of the ou tput. This clearly im plies that the distor tion must be co rr elated to the so urce. Interestingly , m any well known source c oding schemes ac- tually lea d, by c onstruction , to sour ce-uncor related distortions. In particular, th is is the case wh en the source code r satisfies the following two conditio ns: a) The linear pro cessing stages (if any) achieve perfect r econstruction (PR) in the a bsence of quantization ; b) the q uantization error is uncor related to the source. The first con dition is typica lly satisfied by PR filter- banks [2 ], transfo rm coders [3 ] and fe edback quan tizers [4]. The secon d cond ition is m et when sub tractiv e (and of ten when non-sub tractive) dither quantizers are employed [5]. Thu s, any PR scheme using, for example, s ubtractively dithered quantization , leads to source- uncorr elated distortions. An im portant fu ndamental qu estion, which was raised by the authors in a recen t pap er [6 ], is: “What is the impact on Shannon ’ s rate-distortio n function, when we further impose the constraint that the en d-to-en d distortion m ust b e uncorr elated to the inp ut?” In [6], we form alized the notio n o f R ⊥ ( D ) , which is th e quadra tic rate-d istortion fun ction s ubject to the co nstraint t hat the d istortion is un correlated to the in put. F or a Gau ssian source X ∈ R N , we d efined R ⊥ ( D ) as [6 ] R ⊥ ( D ) , min Y : E [ X ( Y − X ) T ]= 0 , 1 N tr ( K Y − X ) ≤ D, 1 N | K Y − X | 1 N > 0 1 N I ( X ; Y ) , (2) where the nota tion K X denotes the cov ariance matrix of X and |·| refer s to the determinant. For zero mean Gaussian stationary sources, we sho wed in [6] th at the abov e minimum (in the limit when N → ∞ ) satisfies the f ollowing equations: R ⊥ ( D ) = 1 2 π Z π − π log p S X ( ω ) + α + p S X ( ω ) √ α ! dω (3a) D = 1 2 π Z π − π S Z ( ω ) dω , where S Z ( ω ) = 1 2  p S X ( ω ) + α − p S X ( ω )  p S X ( ω ) , ∀ ω , (3b) is the PSD of the optimal distortion, which n eeds to be Gaussian. Notice th at here the param eter α ( akin to θ in ( 1)) does not represent a “w ater le vel”. In deed, unless X is white, the PSD of the op timal distor tion f or R ⊥ ( D ) is not white, for all D > 0 . 1 In the p resent paper we prove achievability of R ⊥ ( D ) by constructin g coding schemes based on dithered lattice quantization , wh ich, in the limit as the quantizer d imension approa ches infinity , are able to achie ve R ⊥ ( D ) for any positive D . W e also show that R ⊥ ( D ) can be realized causally , i.e., that for all Gaussian sou rces and for all po siti ve distortions one can build forward test channels that realize R ⊥ ( D ) without using non-ca usal filters. Th is is con trary to the case of Shann on’ s rate distortion fun ction R ( D ) , where at least one of the filters 1 Other similarities and differen ces be tween R ⊥ ( D ) and Shannon’ s R ( D ) are discussed i n [6]. of the forward test chann el that r ealizes R ( D ) needs to be non-ca usal [1]. T o further illustrate th e cau sality of R ⊥ ( D ) , we pre sent a cau sal tr ansform codin g ar chitecture that r ealizes it. W e also sh ow that the use of feed back noise-shap ing allows one to achieve R ⊥ ( D ) with memoryless entro py coding. This para llels a recent result by Za mir , K ochman and Erez for R ( D ) [7]. W e con clude the paper by showing that, in all the discussed architectu res, the rate-lo ss (with r espect to R ⊥ ( D ) ) wh en u sing a finite-d imensional q uantizer can b e upper bo unded by the space- filling loss of th e quantizer . Thus, f or any Gaussian source with m emory , by using noise- shaping and scalar d ithered q uantization, the scalar entropy (conditio ned to the dith er) of the quantized o utput e xceeds R ⊥ ( D ) by at most 0.2 54 bit/dimen sion. I I . B AC K G RO U N D O N D I T H E R E D L AT T I C E Q U A N T I Z A T I O N A randomiz ed lattice qu antizer is a lattice q uantizer with subtractive dither ν , followed by e ntropy encodin g. The d ither ν ∼ U ( V 0 ) is u niform ly distrib uted over a V oronoi cell V 0 of the lattice quantizer .Due to th e dither , the quan tization erro r is truly indepen dent of the input. Furth ermore, it was shown in [8] that the cod ing rate of th e q uantizer, i. e. R Q N , 1 N H ( Q N ( X + ν ) | ν ) (4) can be written as the mutual information between the input and the output of an additi ve noise channel Y ′ = X + E ′ , where E ′ denotes the chann el’ s additive noise and is distributed as − ν . More precisely , R Q N = 1 N I ( X ; Y ′ ) = 1 N I ( X ; X + E ′ ) and the qu adratic d istortion per dim ension is given by 1 N E k Y ′ − X k 2 = 1 N E k E ′ k 2 . It has fur thermore b een shown that whe n ν is white there exists a sequenc e of lattice qu antizers {Q N } where th e quan ti- zation err or (and therefo re also the dither ) tends to b e app roxi- mately Gaussian distributed ( in the d iv ergence sense) for large N . Specifically , let E ′ have a pr obability distribution (PDF) f E ′ , and let E ′ G be Gaussian distributed with the same mea n and cov ariance as E ′ . Then lim N →∞ 1 N D ( f E ′ ( e ) k f E ′ G ( e )) = 0 with a conv ergence rate of log( N ) N if the sequ ence {Q N } is chosen approp riately [9]. In the n ext section we will be interested in the case where the dither is not necessarily wh ite. By shaping the V orono i cells of a lattice quantizer wh ose dither ν is white, we also shape ν , obtain ing a colored dither ν ′ . This situation was considered in d etail in [9] fro m where we obtain the following lemma ( which was proven in [ 9] but no t put into a lemma). Lemma 1: Let E ∼ U ( V 0 ) be white, i.e . E is uniform ly distributed over the V oronoi cell V 0 of the lattice quantizer Q N and K E = ǫ I . Furtherm ore, let E ′ ∼ U ( V ′ 0 ) , where V ′ 0 denotes the sha ped V oronoi cell V ′ 0 = { x ∈ R : M − 1 x ∈ V 0 } and M is some inv ertible lin ear transform ation. Deno te the covariance of E ′ by K E ′ = M M T ǫ . Sim ilarly , let E G ∼ N ( 0 , K E G ) having cov ariance m atrix K E G = K E and let E ′ G ∼ N ( 0 , K E ′ G ) whe re K E ′ G = K E ′ . Then there exists a sequence of sh aped lattice quan tizers such that 1 N D ( f E ′ ( e ) k f E ′ G ( e )) = O (lo g( N ) / N ) . (5) Pr o of: The divergence is in variant to inv ertible transform ations since h ( E ′ ) = h ( E ) + log 2 ( | M | ) . Thus, D ( f E ′ ( e ) k f E ′ G ( e )) = D ( f M E ( e ) k f M E G ( e )) = D ( f E ( e ) k f E G ( e )) for any N . I I I . A C H I E V A B I L I T Y O F R ⊥ ( D ) The simplest fo rward channel tha t realizes R ⊥ ( D ) is shown in Fig . 1. Accord ing to (3), all that is need ed fo r the m utual informa tion per d imension between X and Y to equal R ⊥ ( D ) is that Z be Gaussian with PSD equal to the rig ht han d side (RHS) of (3 b). Z X Y Fig. 1: Forward test chan nel In view o f the asympto tic p roperties of ran domized lattice quantizers discussed in Section I I, the achie v ability of R ⊥ ( D ) can be shown by replacing the test channel of Fig.1 by an ad- equately shaped N -dimen sional rando mized lattice quan tizer Q ′ N and t hen letting N → ∞ . In or der t o establish this result, the fo llowing lemma is need ed. Lemma 2: Let X , X ′ , Z and Z ′ be mutu ally indepe ndent random ve ctors. Let X ′ and Z ′ be arbitrarily d istrib uted, and let X and Z b e Gaussian having the same mean an d covariance as X ′ and Z ′ , r espectively . The n I ( X ′ ; X ′ + Z ′ ) ≤ I ( X ; X + Z ) + D ( Z ′ k Z ) . (6) Pr o of: I ( X ′ ; X ′ + Z ′ ) = h ( X ′ + Z ′ ) − h ( Z ′ ) ( a ) = h ( X + Z ) − h ( Z ) + D ( Z ′ k Z ) − D ( X ′ + Z ′ k X + Z ) ≤ I ( X ; X + Z ) + D ( Z ′ k Z ) , where ( a ) stems from the well known re sult D ( X ′ k X ) = h ( X ) − h ( X ′ ) , see, e. g., [10, p. 254]. W e can n ow prove the achiev ability of R ⊥ ( D ) . Theor em 1: F or a sour ce X bein g an infinite length Gaus- sian rando m vector with zer o me an, R ⊥ ( D ) is a chievable. Pr o of: Let X ( N ) be th e sub- vector containing the first N elements of X . For a fixed distortion D = tr ( K Z ( N ) ) / N , the av erage mutual infor mation p er dim ension 1 N I ( X ( N ) ; X ( N ) + Z ( N ) ) is minimized when X ( N ) and Z ( N ) are jointly Gaussian and K Z ( N ) = 1 2 q K 2 X ( N ) + α K X ( N ) − 1 2 K X ( N ) , (7) see [6 ]. Let th e N -dimen sional shaped ran domized lattice quantizer Q ′ N be suc h that th e dither is distributed as − E ′ ( N ) ∼ U ( V ′ 0 ) , with K E ′ ( N ) = K Z ( N ) . I t f ollows that the coding r ate of the quantizer is gi ven by R Q N = 1 N I ( X ( N ) ; X ( N ) + E ′ ( N ) ) . The rate loss du e to u sing Q N to quantize X ( N ) is given by R Q N ( D ) − R ⊥ ( D ) = 1 N h I ( X ( N ) ; X ( N ) + E ′ ( N ) ) − I ( X ( N ) ; X ( N ) + E ′ ( N ) G ) i ( a ) ≤ 1 N D ( f E ′ ( N ) ( e ) k f E ′ G ( N ) ( e )) , (8) where f E ′ G ( N ) is the PDF of the Gaussian random vector E ′ G ( N ) , indepen dent of E ′ ( N ) and X ( N ) , and having the sam e first a nd seco nd order statistics as E ′ ( N ) . In (8), inequ ality ( a ) follows dir ectly f rom Lemma 2, since the use of subtractive dither yields the er ror E ′ ( N ) indepen dent of X ( N ) . T o co mplete th e pr oof, we in v oke Lem ma 1, which guaran- tees th at th e RHS of (8) vanishes as N → ∞ . Remark 1 : 1) For ze ro mean stationar y Gaussian ran- dom so urces, R ⊥ ( D ) is ach iev ed by taking X in The- orem 1 to be th e comp lete inpu t process. For this case, as shown in [6], the Fourier transform of the autocorre- lation f unction of Z ( N ) tends to the RHS of (3b). 2) For vector pr ocesses, the achievability o f R ⊥ ( D ) fol- lows by building X in T heorem 1 from the concatenation of in finitely many co nsecutive v ectors. 3) Note that if on e has an infinite num ber o f parallel scalar random pro cesses, R ⊥ ( D ) can be achieved causally by forming X in Theo rem 1 fro m the k -th sample of each of th e p rocesses and using entropy co ding after Q . The fact that R ⊥ ( D ) can be realized ca usally is further illustrated in th e following section. I V . R E A L I Z AT I O N O F R ⊥ ( D ) B Y C AU S A L T R A N S F O R M C O D I N G W e will next sho w that for a Gaussian ran dom v ector X ∈ R N with positiv e definite cov ariance matrix K X , R ⊥ ( D ) can be realized by causal transform cod ing [11], [12 ]. A typical tran sform co ding architec ture is shown in Fig. 2. In this figu re, T is an N × N matrix, and W is a Gau ssian vector, indepen dent of X , with covariance matrix K W = σ 2 W I . T he system clear ly satisfies th e perfect r econstructio n con dition Y = X + T − 1 W . The recon struction er ror is the Gaussian random vector Z , Y − X , and the MSE is D = 1 N tr { K Z } , where K Z = σ 2 W T − 1 T − T . W Y X T T − 1 U ˆ U Fig. 2: T ransform cod er . By restricting T to be lower tr iangular, the transform coder in Fig. 2 becomes causal, in the sense th at ∀ k ∈ { 1 , .., N } , the k -th elem ents of U and ˆ U can be de termined u sing just the first k elem ents of X and the k - th elemen t of W . T o h av e 1 N I ( X ; Y ) = R ⊥ ( D ) , it is necessary a nd su fficient that T − 1 T − T = K Z ⋆ /σ 2 W , (9) where th e covariance matrix of th e optim al distortio n is [6] K Z ⋆ , 1 2 q K 2 X + α K X − 1 2 K X . (10) Since T − 1 is lower triangular, (9) is th e Cholesky decompo- sition of K Z ⋆ /σ 2 W , wh ich always exists. 2 Thus, R ⊥ ( D ) c an be rea lized by causal transfo rm coding. In practice, transform coders are implemented by replacing the (vector) A WGN channel ˆ U = V + W by a qu antizer (or se veral quantizers) followed b y entropy coding. The latter process is simplified if the quantized outp uts are indep endent. When using quantizers with subtractiv e dither , this c an b e shown to b e e quiv alent to ha ving 1 N P N k =1 I ( ˆ U k − W k ; ˆ U k ) = 1 N I ( U ; ˆ U ) in the transfor m coder when using the A WGN channel. Notice that, sinc e T in (9) is invertible, the mu tual informa tion per d imension 1 N I ( U ; ˆ U ) is also equal to R ⊥ ( D ) . By th e chain rule of mutual informatio n we have 1 N X N k =1 I ( ˆ U k − W k ; ˆ U k ) ≥ 1 N I ( U ; ˆ U ) = R ⊥ ( D ) , (11) with equality iff the elements of ˆ U ar e mutually independ ent. If ˆ U is G aussian, this is equiv alent to K ˆ U being diagon al. Clearly , th is can not be ob tained with the ar chitecture shown in Fig. 2 using causal matrices (while at the sam e time satisfying ( 9)). Howe ver , it can be achieved by using err or feedback , as we show next. Consider the scheme sh own in Fig. 3, where A ∈ R N × N is lower triang ular and F ∈ R N × N is strictly lower trian- gular . Again, a sufficient a nd nece ssary condition to have A − 1 A F Y W U V ˆ U X W Fig. 3 : A causal tr ansform cod ing scheme with er ror feed back. 1 N I ( X ; Y ) = R ⊥ ( D ) is that K Z = K Z ⋆ , see (1 0), i.e., σ 2 W A − 1 ( I − F )  A − 1 ( I − F )  T = K Z ⋆ ⇐ ⇒ ( I − F )( I − F ) T = AK Z ⋆ A T /σ 2 W . (12) On the oth er hand , equa lity in (11) is ac hiev ed only if K ˆ U = AK X A T + σ 2 W ( I − F )( I − F ) T = D , (13) for some d iagonal matrix D with positiv e elem ents. If we substitute the Cholesky factorization K Z ⋆ = LL T into (1 2), we o btain ( I − F )( I − F ) T = ALL T A T /σ 2 W , and th us A = σ W ( I − F ) L − 1 . (14) 2 Furthermore, sinc e K Z ⋆ > 0 , there exist s a unique T having only positi ve element s on its main diagonal that s atisfies (9 ), see [13]. Substituting the ab ove into (13) we o btain D = σ 2 W ( I − F ) h L − 1 K X L − T + I i ( I − F ) T (15) Thus, th ere exist 3 A and F satisfying ( 12) and (13). Substi- tution of (14) into (15 ) yields D = A ( K X + K Z ⋆ ) A T , and log | D | = 2 log | A | + log | K x + K Z ⋆ | . From (12) and the fact that | I − F | = 1 it follows that | A | 2 = σ 2 W / | K Z ⋆ | , and therefor e 4 1 N X N k =1 I ( V k ; ˆ U k ) = 1 N X N k =1 log  σ 2 ˆ U k σ 2 W  = 1 2 N log | D | σ 2 W = 1 2 N log | K x + K Z ⋆ | − 1 2 N log | K Z ⋆ | = 1 2 N X N k =1 log  √ λ 2 k + λ k α + λ k √ λ 2 k + λ k α − λ k  = R ⊥ ( D ) , (16) thus ach ieving equality in (11). W e have seen th at the use o f erro r feed back allows on e to make the average scalar mutual in formation between the inp ut and outpu t of each A WGN ch annel in the tran sform domain equal to R ⊥ ( D ) . In the following section we show how this result can b e extended to stationar y Gaussian proce sses. V . A C H I E V I N G R ⊥ ( D ) B Y N O I S E S H A P I N G In this section we show that, for any colored stationary Gaussian stationary source and for any positive distortion , R ⊥ ( D ) can be realized by noise shaping, and that R ⊥ ( D ) is ach iev able using memo ry-less entr opy coding . A. Realization of R ⊥ ( D ) by Noise-S haping The fact th at R ⊥ ( D ) ca n be realized b y the additive colored Gaussian noise test channel of Fig. 1 sug gests that R ⊥ ( D ) could also be ach iev ed by an a dditive white Gaussian noise (A WGN) channel em bedded in a no ise-shaping feedb ack loop, see Fig. 4. In this figure, { X k } is a Gaussian stationary process with PSD S x ( e j ω ) . The filters A ( z ) a nd F ( z ) are L TI. Th e A WGN chan nel is situated between V and ˆ U , where white Gaussian noise { W k } , independe nt of { X k } , is added. T he reconstruc ted signal Y is obta ined by p assing ˆ U through th e filter A ( z ) − 1 , yielding the recon struction error Z k = Y k − X k . W F ( z ) ˆ U A ( z ) − 1 X A ( z ) U W V Y Fig. 4: T est channe l built b y embeddin g the A WGN ch annel ˆ U k = V k + W k in a n oise feed back loo p. The fo llowing theorem states th at, f or this scheme, the scalar m utual inform ation across the A WGN chan nel can actually e qual R ⊥ ( D = σ 2 Z ) . 3 For any positi v e definite matrices K X and K Z ⋆ = L L T , there exists a unique matrix F ha ving zeros on its main diagonal that s atisfies (15), see [14]. 4 The last equality in (16) follo ws from the expression for R ⊥ ( D ) for Gaussian vect or s ources deriv ed in [6 ]. Theor em 2: Consider the sc heme in Fig . 4. Let { X k } , { W k } be in depend ent stationary Gaussian random p r oce sses. Suppo se tha t th e differ ential entr opy rate of { X k } is bounded, and tha t { W k } is white. Then, fo r every D > 0 , th er e exist causal and stable filters A ( z ) , A ( z ) − 1 and F ( z ) such that I ( V k ; ˆ U k ) = R ⊥ ( D ) , wher e D , σ 2 Z . (17 ) Pr o of: Conside r all possible choices o f the filters A ( z ) and F ( z ) such that the obtained sequence { ˆ U k } is white, i.e., such that S ˆ U ( e j ω ) = σ 2 ˆ U , ∀ ω ∈ [ − π , π ] . Fro m Fig. 4, th is is achieved if f the filters A ( z ) and F ( z ) satisfy σ 2 ˆ U =   A ( e j ω )   2 S X ( e j ω ) +   1 − F ( e j ω )   2 σ 2 W . (18) On the oth er hand, since { W k } is Gaussian, a nece ssary a nd sufficient condition in ord er to achieve R ⊥ ( D ) is that S Z ( e j ω ) =   1 − F ( e j ω )   2   A ( e j ω )   − 2 σ 2 W (19) = 1 2  p S X ( ω ) + α − p S X ( ω )  p S X ( ω ) (20 ) , S Z ⋆ ( e j ω ) , ∀ ω ∈ [ − π , π ] . (21) This h olds iff   A ( e j ω )   2 = σ 2 W   1 − F ( e j ω )   2 /S Z ⋆ ( e j ω ) . Substituting th e latter and ( 21) into (18), and after some algebra, we o btain   1 − F ( e j ω )   2 = σ 2 ˆ U σ 2 W " p S X ( e j ω ) + α − p S X ( e j ω ) √ α # 2 , (22a)   A ( e j ω )   2 = 2 σ 2 ˆ U p S X ( e j ω ) + α − p S X ( e j ω ) α p S X ( e j ω ) . (22b) Notice that the f unctions on the righ t hand sides of (22) are bou nded a nd positive for all ω ∈ [ − π , π ] , and th at a boun ded differential entro py rate of { X k } implies that | R π − π S X ( e j ω ) dω | < ∞ . From the Paley-W iener criterio n [15] (see also, e.g. , [ 16]), this implies th at (1 − F ( z )) , A ( z ) and A ( z ) − 1 can be chosen to be stable and causal. Furthermo re, recall that f or any fixed D > 0 , th e corr espondin g value of α is un ique (see [6] ), and thus fixed. Since the variance σ 2 W is also fixed, it fo llows that each frequen cy response magnitud e   1 − F ( e j ω )   that satis fies ( 22a) can be associated to a unique value of σ 2 ˆ U . Since F ( z ) is strictly causal an d stab le, the minimum value o f the variance σ 2 ˆ U is ach iev ed when 1 2 π Z π − π log   1 − F ( e j ω )   dω = 0 , (23) i.e., if 1 − F ( z ) h as no zero s outside the unit circle (equiva- lently , if 1 − F ( z ) is minimum phase), see, e.g., [1 7]. If we choose in (22a) a filter F ( z ) that satisfies (23), and th en we take th e logarithm and integrate b oth sides of ( 22a), we obtain 1 2 log σ 2 ˆ U σ 2 W ! = 1 2 π π Z − π log " √ α p S X ( e j ω ) + α − p S X ( e j ω ) # dω = 1 2 π π Z − π log " p S X ( e j ω ) + α + p S X ( e j ω ) √ α # dω = R ⊥ ( D ) . where ( 3a) h as been used. W e then h av e that R ⊥ ( D ) = 1 2 log  σ 2 ˆ U σ 2 W  = 1 2 log(2 π e σ 2 ˆ U ) − 1 2 log(2 π e σ 2 W ) ( a ) = h ( ˆ U k ) − h ( W k ) ( b ) = h ( ˆ U k ) − h ( V K + W k | V k ) = I ( V k ; ˆ U k ) , where ( a ) follows from the Gaussianity of W k and ˆ U k , and ( b ) from the fact that W k is inde pendent of V k (since F is strictly causal). T his comp letes the proof . Alternatively , R ⊥ ( D ) ( a ) ≤ ¯ I ( { X k } ; { Y k } ) = ¯ h ( A − 1 { ˆ U k } ) − ¯ h ( { X k } + A − 1 (1 − F ) { W k }|{ X k } ) = ¯ h ( A − 1 { ˆ U k } ) − ¯ h ( A − 1 (1 − F ) { W k } ) ( b ) = ¯ h ( { ˆ U k } ) − ¯ h ((1 − F ) { W k } ) ( c ) ≤ h ( ˆ U k | U − k ) − h ( W k ) ( d ) ≤ h ( ˆ U k ) − h ( W k ) ( e ) = h ( ˆ U k ) − h ( V K + W k | V k ) = I ( V k ; ˆ U k ) , In ( a ) , equality is achieved iff the right hand side o f (19) equals ( 22a), i. e., if Z has the optimal PSD. Equality ( b ) holds because | R π − π log   A ( e j ω )   | dω < ∞ , which follows from ( 22b). The fact that { ˆ U k } is stationary has been used in ( c ) , wher ein eq uality is achieved iff | 1 − F | is minimum phase, i.e., if (23) holds. Eq uality in ( d ) holds if an o nly if the elements of { ˆ U k } ar e indepen dent, which, fro m the Gaussianity of { ˆ U k } , is equivalent to (18). Finally , ( e ) stems from the fact that W k is ind ependen t of V k . Notice that the key to the pro of of Theor em 2 relies on knowing a prio ri the PSD of the end to end d istortion requir ed to re alize R ⊥ ( D ) . Indee d, one c ould also use th is fact to realize R ⊥ ( D ) by embedd ing th e A WGN in a DPCM feedback loop, an d then following a reasonin g similar to tha t in [7 ]. B. Achieving R ⊥ ( D ) Thr o ugh F eed back Quantization In orde r to achieve R ⊥ ( D ) by using a q uantizer instead of an A WGN chan nel, one w ould require the quantiza tion errors to be Gaussian. This cannot be achieved with scalar quantizer s. Howe ver , as we have seen in II, dithered lattice quantizers are able to yield q uantization errors appr oximately Gaussian as the lattice dimension tends to infinity . T he sequential (causal) nature of th e feed back ar chitecture does n ot immedia tely allow for the p ossibility of using vector quan tizers. Howe ver , if several sourc es are to be pro cessed simultaneo usly , we can overcome this difficulty b y using an idea su ggested in [7 ] where the sou rces are proce ssed in parallel by sepa rate feedback quan tizers. The f eedback quantizers are operating indepen dently of each other except that th eir scalar quan tizers are replaced b y a single vector quantizer . If the number of parallel sources is large, then the vector quan tizer gu arantees that th e marginal distributions of the individual compo nents of th e quantized vector s beco mes app roximately Gaussian dis- tributed. Thus, due to the dithe ring within the vector quantizer, each feed back quantize r observes a sequ ence of i.i.d. Gau ssian quantization noises. Further more, the effectiv e coding rate (per source) is th at o f a high dimen sional entro py constrained dithered q uantizer (p er dim ension). The fact that th e s calar mutual info rmation between V k and ˆ U k equals th e mu tual in formation rate between { V K } an d { ˆ U k } in each of the parallel coders implies that R ⊥ ( D ) can be ach iev ed by using a m emoryless entro py coder . V I . R AT E L O S S W I T H D I T H E R E D F E E D BA C K Q UA N T I Z AT I O N The results presen ted in sections IV and V suggest that if a test chan nel embed ding an A WGN chann el r ealizes R ⊥ ( D ) , then a source cod er obtain ed b y rep lacing the A WGN channel by a dithered, finite dimen sional lattice q uantizer, would exhibit a rate close to R ⊥ ( D ) . The ne xt theore m, whose proof follows th e line of the results giv en in [7, sec. VII] , provides an upp er bound on the rate- loss incurred in this case. Theor em 3: Consider a source coder with a finite di- mensional subtractively dithered lattice quantizer Q . I f when r eplacing the quantizer by an A WGN c hannel the scalar mutual information acr oss th e chan nel equals R ⊥ ( D ) , t hen the scalar entr o py of the quan tized output exceeds R ⊥ ( D ) by at most 0 . 254 b it/dimension. Pr o of: Let W be the n oise of the A WGN channe l, an d V an d ˆ U de note the channel input and output sig nals. From the con ditions of the theore m, we have that I ( V k ; ˆ U k ) = R ⊥ ( D ) . (24 ) If we now rep lace the A WGN b y a dithered quan tizer with subtractive d ither ν , such that the qu antization noise W ′ is obtained with the same fir st and second o rder statistics as W , then the end to e nd MSE remains the same . The corresp onding signals in the quantized case, namely V ′ and ˆ U ′ , will also have the same seco nd ord er statistics as th eir Gaussian counterparts V and ˆ U . Th us, by using Lemma 2 we o btain I ( V ′ k ; ˆ U ′ k ) ≤ R ⊥ ( D ) + D ( ˆ U ′ k k ˆ U k ) . (25) Finally , fr om [8, Th eorem 1], we have that H ( Q ( V k + ν k ) | ν k ) = I ( V ′ k ; ˆ U ′ k ) . Substitution of ( 25) into this last equation y ields the result. V I I . C O N C L U S I O N S W e have proved the achie vability o f R ⊥ ( D ) by using lattice quantization with su btractive dither . W e hav e shown that R ⊥ ( D ) can be realized causally , and that the use of feedback allows one to achieve R ⊥ ( D ) by using memo ryless entropy cod ing. W e also showed that the scalar en tropy of the quantized outpu t when using o ptimal finite-dimensional dithered lattice quantization exceeds R ⊥ ( D ) by at most 0 . 254 bits/dimension. R E F E R E N C E S [1] R. Gallager , Information Theory and Reliable Communication . Ne w Y ork: J ohn W ile y & S ons, 1968. [2] P . P . V aidyanathan, Multirat e Systems and Filt er B anks . E ngle woo d Clif fs, New Jersey: Prentice-Ha ll, 1993. [3] V . K. Goyal, “Theoret ical foundations of transform coding, ” IEEE Signal Pr ocessing Mag . , vol. 18, pp. 9–21, Sept. 2001. [4] N. Jayant and P . Noll, Digital coding of waveforms. Principle s and appr oac hes to speec h and video. Engle wood Cliffs, NJ: Prentice Hall, 1984. [5] R. M. Gray and T . G. Stockham, “Dithered quantize rs, ” IEEE T rans. Inform. Theory , vol. 39, no. 3, pp. 805–812, May 1993. [6] M. S. Derpich , J. Østerga ard, and G. C. Goodwin, “The quadra tic Gaussian rate-disto rtion function for source uncorrel ated distorti ons, ” in Proc. of the Data Compr ession Confer ence , DCC , 2008, to appear (av ailable from http://arXi v .org ). [7] R. Z amir , Y . Kochman, and U. Erez, “ Achie ving the Gaussian rate- distorti on functio n by prediction. ” IEEE T rans. Inform. Theory , (sub- mitted). [8] R. Zamir and M. Feder , “On univ ersal quant izati on by randomize d uniform/la tice quantizers, ” IEE E T rans. Inform. Theory , vol . 38, pp. 428–436, 1992. [9] ——, “On latti ce quantizati on noise, ” IEEE T rans. Inform. Theory , vol. 42, no. 4, pp. 1152–1159, Jul y 1996. [10] T . M. Cove r and J. A. Thomas, Elements of informati on theory , 2nd ed. Hoboke n, N.J: W ile y-Intersci ence, 2006. [11] A. Habibi and R. Hershel, “ A unifie d representati on of diff erentia l pulse- coded m odulat ion (DPCM) and transform coding s ystems, ” IE EE T rans. Commun. , vol. 22, no. 5, pp. 692–696, May 1974. [12] A.-M. Phoong and Y .-P . Lin, “Prediction- based lowe r triangula r trans- form, ” IEE E T rans. Signal P r ocessing , vol. 48, pp. 1947–1955, 2000. [13] R. A. Horn and C. R. Johnson, Matrix Analysis . Cambrid ge, UK: Cambridge Unive rsity Press, 1985. [14] M. Derpich, D. Quev edo, and G. Goodwin, “Conditi ons for optimality of scalar feedback quantiz ation, ” Proc . IEEE Int. Conf. Acoust. Speech Signal Pr ocess. , 2008, to appear . [15] M. Wi ener and R. Pale y , F ourier transforms in the Comple x Domain . Provid ence, R. I.: Am erica n Mathematics Society , 1934. [16] J. G. Proakis and D. G. Manolakis, Digital Signal Proce ssing . Upper Saddle Ri ver , NJ: Prentice-Hall , 1996. [17] M. M. Serón, J. H. Brasla vsky , and G. C. Goodwin, Fundamental Limitati ons in Fil tering and Contr ol . Spri nger-V erlag, London, 1997.