Performance of LDPC Codes Under Faulty Iterative Decoding

1 Performance of LDPC Codes Under F aulty Iter ati v e Decoding La v R. V arshney Abstract Departing from traditional communicatio n theory where de coding algorithm s are assumed to p erform without error, a system whe re noise p erturbs b oth co mputation al devices and commun ication channels is considered here. This pa per studies limits in processing n oisy signals with no isy circu its by in vestigating the effect of noise on standard iterativ e decod ers for low-density parity-ch eck codes. Conce ntration of d ecoding perfo rmance aro und its av erage is shown to hold when n oise is in troduc ed into me ssage-passing a nd loca l com putation. De nsity ev olution equations for simple faulty iterati ve decoders are deri ved. In one model, computing nonlinear estimation thresholds shows that perfo rmance degrades smooth ly a s deco der noise increases, but arb itrarily small prob ability of error is not ach iev a ble. Probability of err or may be driven to zero in another system mo del; the decod ing th reshold ag ain decreases smoothly with decoder noise. As an application of the method s de veloped, an achievability result fo r reliable me mory systems c onstructed fr om u nreliable comp onents is pr ovided. Index T erms Low-density parity-ch eck codes, communicatio n system fault tolerance, density ev olution , deco ding, memories I . I N T RO D U C T I O N The ba sic g oal in channel co ding is to des ign en coder-decoder pairs tha t a llo w reliable communication over noisy channe ls at information rates c lose to capa city [1]. The p rimary obstac le in the que st for practical ca pacity-achieving codes has been de coding complexity [2]–[4]. Low-density parity-check (LDPC) c odes have, howev er , emerged as a clas s of codes that h av e performance at or near the Shannon limit [5], [6] a nd y et are suf ﬁciently structured as to have dec oders with circuit implementations [7]–[9]. In addition to decoder complexity , decod er reliability may also limit practical channel coding. 1 In Shannon’ s schema tic d iagram of a general commu nication syste m [1, Fig. 1] and in the traditional informati on and commu- nication theories that have developed within the c onﬁnes o f that diagram, noise is loc alized in the commun ication channe l. Th e d ecoder is ass umed to operate without error . Given the pos sibility of unreliable computation on faulty hardware, there is value in studying error-prone dec oding. In fact Ha mming’ s original development o f parity-check codes was motiv ated by applications in c omputing rather than in communication [11 ]. The goal of this paper is to in vestigate limits of communication systems with n oisy decode rs and has dual moti vations. The ﬁrst is the eminen tly practical motiv a tion of determining how well error co ntrol codes work when decode rs are faulty . Th e s econd is the deeper motiv ation o f de termining funda mental limits for proce ssing un reliable signals with unreliable computational devices, illustrated schematically in F ig. 1. T he moti vations are intertwined. As no ted by Pierce, “The down-to- earth problem of making a computer work, in fact, beco mes tangled with this dif ﬁcult philosoph ical problem: ‘What is possible and what is impo ssible when unreliable circuits are use d to process unreliable information?’” [12]. A ﬁrst step in understanding t hese issues is to analyze a particular class of codes and decoding techniques: i terativ e messag e-passing de coding a lgorithms for LDPC codes . Whe n the co de is repres ented a s a factor g raph, algorithm Manuscript prepared May 20 08; re vised April 2009, May 2010. This work was supported in part by a National Science Foundation Graduate Resea rch Fello wship and was performed in part when t he author was with l’ ´ Ecole P olytechnique F ´ ed ´ erale d e Lausanne. The material in this paper was presented in part at the Information Theory W orkshop, Lake T ahoe, CA, September 2007. L. R. V arshney is with t he Department of Electri cal Engineering and Computer Science, the L aboratory for Information and Decision Systems, and the Research Laboratory of Electronics, Massachu setts Institute of T echnology , Cambridge, MA, 02139 US A (e-mail: lrv@mit.edu). 1 One may also consider the ef fect of encoder complexity [10], howe ver encoder noise need not be e xplicitly considere d, since it may be incorporated i nto channel noise, using the noise combining argum ent suggested by Fig. 3. 2 INFORMATION SOURCE TRANSMITTER MESSAGE SIGNAL RECEIVED SIGNAL MESSAGE RECEIVER DESTINATION NOISE SOURCE NOISE SOURCE Fig. 1. Schematic diagram of an information system that processes unreliable signals with unreliable circuits. computations occur a t nodes and algo rithm communica tion is ca rried out over edg es. Co rresponden ce between the factor graph an d the algorithm is not only a tool for exposition but also the way d ecoders are implemented [7 ]–[9]. In traditional performance an alysis, the deco ders are assumed to work without error . In this paper , there will be transient loc al computation and message -passing errors, whether the deco der is analog or digital. When the decod er itself is no isy , one might b eliev e that ach ieving a rbitrarily small prob ability of error ( Sh annon r eliability ) is not pos sible, but this is inde ed pos sible for ce rtain se ts of noisy chan nels an d noisy deco ders. This is shown by example. For other sets of noisy channels a nd noisy decode rs, Sha nnon reliability is not achievable, but error probab ility tending to extremely small values is achiev able. Small probability of error , η , is o ften satisfactory in practice, and so η -reliable p erformance is also in vestigated. De coding thres holds at η -reliability decrea se smo othly with increa sing decod er noise. Communica tion systems may display graceful d egradation with respect to noise lev els in the de coder . The remainder of the pap er is organized a s follows. Sec tion II reviews moti vations and related work. Se ction III formalizes notation and Se ction IV giv es conc entration results that a llo w the de nsity ev olution metho d o f analysis, generalizing results in [13]. A n oisy version of the Gallager A decoder for processing the output of a binary symmetric chan nel is analyze d in Section V, where it is shown that Shannon reliability is unattainable. In Section VI, a noisy decoder for A WGN cha nnels i s analyze d. For this model, the proba bility of error may be dri ven to zero and the decoding threshold degrades smoo thly as a function o f de coder noise. As an app lication of the results o f Section V, Se ction VII prec isely cha racterizes the information storage capacity of a memory built from unreliable compone nts. Section VIII provides some con clusions. I I . B AC K G RO U N D A. Practical Motivations Although always presen t [11], [14], rece nt technologica l trends in digital c ircuit design bring practical motiv ations to the fore [15 ]–[17]. The 2008 up date of the International T echn ology Roadmap for Semico nductors (ITRS) 2 points out tha t for complemen tary metal-oxide-silicon (CMOS) technology , inc reasing p ower d ensities, d ecreasing s upply volt ages , and decreasing sizes have increase d sensiti vity to cosmic radiation, electromagn etic interference, and thermal ﬂuc tuations. The ITRS further say s that an ong oing shift in the manufacturing pa radigm will dramatically reduce costs b ut will lea d to more transient failures of signals, logic values, de vices, a nd interconnects. De vice technologies beyond CMOS, s uch as single-electron tunnelling technology [18], c arbon-bas ed n anoelectronics [19], and chemica lly ass embled elec tronic nano computers [20], are a lso p rojected to enter prod uction, but they a ll display erratic, random device behavior [21], [22]. Analog comp utations are a lw ays s ubject to n oise [23], [24]. Similar issue s arise whe n performing real-valued computations on d igital computers since quantization, whether ﬁxed-point or ﬂoa ting-point, is often we ll-modeled as bounded , a dditi ve s tochastic n oise [25]. B. Coding and Computing Information and commu nication theory hav e provided limi ts for proce ssing unreliable signals with reliable circuits [1], [13], [26], whe reas fault-tolerant computing theory h as provided limits for process ing re liable sign als (inpu ts) with unreliable c ircuits [12], [27]–[31]. This work brings the two together . 2 The ov erall objectiv e of the ITRS is to present the consensus of the semicondu ctor industry on the best current estimate of research and de velopme nt needs for the next ﬁf teen years. 3 A brief overview of terms and c oncepts fr om fault-tolerant c omputing, based on [32], [33], is now provided. A fault is a phys ical d efect, imperfection, or ﬂaw that occu rs within s ome hardware o r software component. An err or is the informational man ifestation o f a fault. A permanent fault exists indeﬁ nitely until correcti ve action is taken, whereas a transient f ault a ppears and disap pears in a sho rt period of time. Noisy c ircuits in which the interconnection pattern of compone nts are trees are called formulas [34], [35]. In a n err or mod e l , the ef fects o f faults are giv en directly in the informational un i verse. For example, the basic von Neuma nn mod el of noisy circuits [27] models transient f aults in logic gates and wires as message and node computation noise that is both s patially and temporally indep endent; this has more recently also bee n called the Hegde-Shanbh ag model [36], a fter [37]. This error mo del is use d here. Error models of p ermanent faults [38], [39] or of miswired circuit interconnection [28], [40] hav e been considered e lsewhere. Su ch permanent errors in decoding circuits may be interpreted as either changing the factor graph use d for dec oding or as introdu cing new potentials into the factor grap h; the c ode u sed by the encod er and the cod e use d by the dec oder are different. There a re several design p hilosophies to combat faults. F ault avo idance seeks to ma ke p hysical co mponents mo re reliable. F ault mask ing see ks to prevent faults from introduc ing e rrors. F a ult tolerance is the ability of a system to continue pe rforming its fun ction in the prese nce of f aults. This paper is primari ly concerned with fault tolerance, but Section VII considers fault masking. C. Related W ork Empirical ch aracterizations of message -passing dec oders have demon strated tha t proba bility of error performance does not change much when messa ges a re quantized at high resolution [26]. Even algo rithms that are coarsely quantized versions of o ptimal b elief p ropagation show little degradation in performance [13], [41 ]–[46]. It s hould be emphasize d, ho wever , that fault-fr ee, qu antized decoders dif fer signiﬁcantly f rom decod ers that mak e random errors. 3 The difference is similar to that be tween control s ystems with ﬁnite-capacity n oiseless ch annels a nd c ontrol s ystems with noisy c hanne ls of equal c apacity [50]. Se emingly the only previous work on me ssage -passing algo rithms with random errors is [51], which deals with problems in distributed inferen ce. 4 The information the oretic problem of mismatch capa city [52] and its analog for iterati ve decoding [53] deal with scenarios whe re an incorrect decoding metric is us ed. Th is may arise, e.g., d ue to incorrect estimation of the channe l noise po wer . For messa ge-pass ing d ecoding algorithms, mismatch leads to inc orrect parameters for local computations. These are p ermanent faults rathe r than the kind of transient faults co nsidered in this paper . Noisy LD PC dec oders were pre viously ana lyzed in the context of designing reliable me mories from unreliable compone nts [54], [55] (re visited in Section VII), using Gallager’ s original methods [ 26]. Several LPDC code analysis tools h av e s ince be en developed, including simulation [56], expander graph arguments [57] , [58], EXIT charts [59], [60], a nd de nsity ev olution [13], [61], [62]. This work gene ralizes as ymptotic charac terizations developed by Richardson and Urba nke for no iseless deco ders [13], showing that den sity evoluti on is a pplicable to f aulty de coders. Expande r g raph ar guments have a lso been extended to the case of noisy decoding in a paper [63] that appeare d concurrently wit h the ﬁrst presentation of this w ork [64]. Note t hat pre vious works hav e not e ven cons idered the possibility that Shanno n reliability is achiev able with noisy decod ing. I I I . C O D E S , D E C O D E R S , A N D P E R F O R M A N C E This sec tion e stablishes the bas ic n otation o f LDPC cha nnel codes and mes sage-pa ssing decode rs for c ommuni- cation systems d epicted in Fig. 1. It primarily follows establishe d notation in the ﬁeld [13], [65], an d will therefore be brief. Ma ny of the n otational co n ventions are depicted s chematically in Fig. 2 using a factor graph-ba sed de coder implementation. Consider the s tandard ensemble of ( d v , d c ) -regular LDP C c odes of len gth n , C n ( d v , d c ) , deﬁned by a uniform measure on the s et of labeled bipartite factor grap hs with vari able nod e degree d v and c heck nod e degree d c . 5 There 3 Randomized algorithms [47] and stochastic computation [ 48] (used for decoding in [49]) make use of randomness to increase functionality , but the randomn ess is deplo yed in a controlled manner . 4 If the graphical model of the code and the graph of noisy communication links in a distributed system coincide, then the distributed inference problem and the message-passing decoding problem can be made to coincide. 5 A factor graph determines an “ordered code, ” but the opposite is not true [66]. Moreov er , since codes are unordered objects, se veral “ordered codes” are i n fact the same code. 4 Ψ x 1 y 1 p y | x u v Ψ x 2 y 2 p y | x u v Ψ x i y i p y | x u v Ψ x n y n p y | x u v Φ u c Φ u c Φ u c Fig. 2. Schematic diagram of a f actor graph-based implementation of a noisy decoder circuit. Only one v ariable-to-check message and on e check-to-v ari able message are highlighted. Other wires, sho wn in gray , will also carry noisy messages. are n variable n odes corresp onding t o the c odeword letters and nd v /d c check nodes correspon ding to the parity check constraints. Th e design rate of the code is 1 − d v /d c , t hough the actual rate might be higher s ince not all checks may be ind epende nt; the true rate co n verges to the de sign rate for large n [65, Le mma 3.22]. One may also consider irregular codes , C n ( λ, ρ ) ch aracterized b y the d egree distribution pair ( λ, ρ ) . Generating fun ctions of the variable nod e a nd che ck node degree distributions, λ ( ζ ) and ρ ( ζ ) , a re functions of the form λ ( ζ ) = P ∞ i =2 λ i ζ i − 1 and ρ ( ζ ) = P ∞ i =2 ρ i ζ i − 1 , where λ i and ρ i specify the fraction o f ed ges that connect to nod es with degree i . The design ra te is 1 − R 1 0 ρ ( ζ ) dζ / R 1 0 λ ( ζ ) dζ . In the communica tion system of Fig. 1, a codeword is selected by the transmitter and is sen t through the noisy channe l. Channel input a nd output letters are denoted X ∈ X and Y ∈ Y . Since binary linea r c odes are u sed, X can be taken as {± 1 } . Th e rece i ver con tains a noisy me ssage -passing decoder , which is us ed to process the channe l output co dewor d to produce an estimate of X that is denoted ˆ X . The goal of the re ceiv er is to recover the channe l input codeword with low probability o f error . Throu ghout this work, prob ability of bit e rror P e is us ed as the performance criterion; 6 P e = Pr[ X 6 = ˆ X ] . The messag e-passing decod er works in iterati ve stages and the iteration time is indexed by ℓ = 0 , 1 , . . . . W ithin the decoder , at time ℓ = 0 , each v ariable node has a realization of Y , y i . A message-pass ing d ecoder exchanges messag es between nodes alon g wires. First each variable node sends a message to a neighboring chec k node over a noisy me ssaging wire. Generically , sent me ssage s are denoted a s ν v → c , mess age wire noise rea lizations as w v → c , and received mes sages as µ v → c : as sume without loss of generality that ν v → c , w v → c , and µ v → c are drawn from a common messa ging alphabet M . Each c heck no de process es receiv ed message s and sen ds back a message to e ach neigh boring variable node over a noisy me ssage wire. The no isiness of the check node proc essing is gene rically den oted by an inp ut random variable U c ∈ U . The ch eck node computation is de noted Φ ( ℓ ) : M d c − 1 × U 7→ M . The no tations ν c → v , µ c → v , and w c → v are used for signaling from ch eck nod e to variable no de; ag ain without loss of gen erality a ssume tha t ν c → v , w c → v , µ c → v ∈ M . Each variable node now processes its y i and the mess ages it receives to produce new messages. The new messages are produc ed through pos sibly noisy proces sing, where the no ise input is generica lly denoted U v ∈ U . The variable node computation is denoted Ψ ( ℓ ) : Y × M d v − 1 × U 7→ M . Lo cal c omputations an d mes sage-pa ssing continue iterati vely . 6 An alternativ e would be to conside r block error pro bability , ho we ver an exac t e valuation of this quantity is difﬁcult due to the de pendence between different symbo ls of a codew ord, e ven if the bit error probab ility is the same for all symbols in the codew ord [67]. 5 Message passing indu ces dec oding neighbo rhoods , which in volve no des/wires that have communicated with one another . For a given node ˙ n , its ne igh borhoo d o f depth d is the indu ced subgraph consis ting of all nodes reac hed and edges traversed by paths of len gth at mos t d starting from ˙ n (including ˙ n ) and is de noted N d ˙ n . The directed neighbor hood of depth d of a wire v → c , denoted by N d v → c , is de ﬁned as the induced sub graph containing all wires and nod es on paths sta rting from the same place as v → c but different from v → c . E quiv alently for a wire c → v , N d c → v is the indu ced sub graph co ntaining all wires and nodes on paths starting fr om the same place as c → v but dif ferent fr om c → v . If the induce d s ubgraph (correspon ding to a neigh borhood) is a tree then the neighborhoo d is tr ee-like , otherwise it is not tree-lik e. The neighborhoo d is tree-like if a nd o nly if all in volved nodes are d istinct. Note that only extri nsic information is used in node c omputations. Also note that in the sequel, all decode r noises ( U c , U v , W v → c , and W c → v ) will be assume d to be indepen dent of e ach o ther , as in the von Ne umann e rror model of faulty co mputing. A communication sys tem is judged by information rate, error proba bility , and b locklength. For ﬁxed ch annels, information theory spec iﬁes the limits of the se three pa rameters when optimizing over the unc onstrained cho ice of codes and decoders; Sh annon reliability is ac hiev able for rates below capacity in the limit of increasing blocklen gth. When decode rs a re restricted to be n oisy , tighter information theoretic limits are no t known. The refore comparing performance of sys tems with noisy deco ders to s ystems using identical cod es but noiseless decode rs is more appropriate than comparing to Shanno n limits. Coding theory follows from information theory by res tricting decoding c omplexity; ana lysis of noisy decoders follo ws from coding theory by restricting deco ding reliability . I V . D E N S I T Y E VO L U T I O N C O N C E N T R A T I O N R E S U LT S Considering the great succe sses achieved b y ana lyzing the no iseless decode r performance of ensemble s o f co des [13], [61], [65] rather than of p articular c odes [26], the s ame approach is pursued for noisy decoders. The ﬁrst mathematical contribution of this work is to extend the method of ana lysis promulgated in [13] to the case of decode rs with random noise. Several facts that simplify p erformance a nalysis are proven. First, under c ertain sy mmetry conditions with wide applicability , the probability of e rror doe s not dep end on wh ich co dewor d is transmitted. Second, the indi vidual performances of code s in an en semble a re, with high probability , the s ame as the a verage pe rformance of the ensemble. Fina lly , this average behavior con ver ges to the behavior o f a cod e deﬁ ned on a cycle-free graph. Performance analys is then reduc es to determining a verage performance on an inﬁnite tree: a noisy formula is analyze d in place of general noisy circuits. For brevity , only regular LDPC codes a re con sidered in this se ction, however the results c an b e ge neralized to irregular LDPC codes . In particular , replacing node d egrees by maximum node degrees, t he proofs stand mutatis mutandis . Similarly , o nly binary L DPC codes are considered; generalizations to non-binary a lphabets also follow , as in [68]. A. Restriction to All-One Codeword If c ertain symmetry c onditions a re satisﬁed b y the sys tem, then the proba bility of error is cond itionally indepen- dent of the codeword that is trans mitted. It is ass umed throughou t this s ection that messag es in the decode r are in belief format . Deﬁnition 1: A mess age in an iterativ e me ssage-pa ssing decoder for a bina ry cod e is said to b e in belief format if the sign of the messag e indica tes the bit estimate a nd the magnitude of the message is a n increasing function of the conﬁde nce level. In particular , a positiv e-valued mes sage indicates belief that a bit is +1 wherea s a negative-v alued messag e indica tes belief that a bit is − 1 . A me ssage of magnitude 0 indica tes complete uncertainty whereas a messag e o f inﬁnite magnitude indicates complete conﬁden ce in a bit value. Note, howev er , that it is not obvious that this is the best format for n oisy mess age-pas sing [65, Appendix B.1]. The symmetry conditions can b e restated for messag es in other formats. The several symme try con ditions a re: Deﬁnition 2 (Channel Symmetry): A memoryless chan nel is binary-input output-symmetric if it satisﬁes p ( Y t = y | X t = 1) = p ( Y t = − y | X t = − 1) 6 for a ll chann el usag e times t = 1 , . . . , n . Deﬁnition 3 (Check Node Symme try): A che ck nod e me ssage ma p is s ymmetric if it satisﬁes Φ ( ℓ ) ( b 1 µ 1 , . . . , b d c − 1 µ d c − 1 , b d c u ) = Φ ( ℓ ) ( µ 1 , . . . , µ d c − 1 , u ) d c Y i =1 b i ! for a ny ± 1 se quence ( b 1 , . . . , b d c ) . That is to say , the signs of the messa ges and the noise factor out of the ma p. Deﬁnition 4 (V a riable Node Symmetry): A vari able node mess age map is symmetric if it satisﬁes Ψ (0) ( − µ 0 , − u ) = − Ψ (0) ( µ 0 , u ) and Ψ ( ℓ ) ( − µ 0 , − µ 1 , . . . , − µ d v − 1 , − u ) = − Ψ ( ℓ ) ( µ 0 , µ 1 , . . . , µ d v − 1 , u ) , for ℓ ≥ 1 . Tha t is to s ay , the initial me ssage from the variable node on ly depends o n the rec eiv ed value and internal noise and there is sign in version in variance for all mess ages. Deﬁnition 5 (Message W ire Symmetry): Conside r any messa ge wire to be a mapping Ξ : M × M → M . The n a messa ge wire is symmetric if µ = Ξ ( ν, w ) = − Ξ( − ν , − w ) , where µ is any mes sage rece i ved at a n ode wh en the messa ge sent from the oppos ite node is ν an d w is mes sage wire noise with d istrib ution sy mmetric abo ut 0 . An example wh ere the message wire symmetry condition holds is if the mes sage wire noise w is additive and symmetric about 0 . Then µ = ν + w = − ( − ν − w ) an d w is symmetric about 0 . Theorem 1 (Conditional Inde penden ce of Er ror): For a given binary linear c ode and a given noisy message- passing a lgorithm, le t P ( ℓ ) e ( x ) deno te the c onditional probability o f error after the ℓ th dec oding iteration, ass uming that codeword x was se nt. If the chan nel and the decode r sa tisfy the symmetry co nditions gi ven in Deﬁnitions 2 – 5, then P ( ℓ ) e ( x ) does not depen d on x . Pr oof: Modiﬁcation of [13, Lemma 1] or [65, Lemma 4.92]. Appen dix A gives de tails. Suppose a s ystem meets the se symmetry c onditions. Sinc e proba bility of error is independe nt of the trans mitted codeword and since all LDPC codes have the all-one codeword in the code book, one may assu me without loss of g enerality that this codeword is sent. Doing so remov es the randomnes s asso ciated with transmitted c odeword selection. B. Concentration a round Ens emble A verage The next simpliﬁca tion follows by se eing tha t the average performanc e of the ensemb le of codes rather than the pe rformance of a particular code may be stud ied, since a ll codes in the en semble perform similarly . T he performances of almost a ll LD PC codes c losely match the a verage p erformance o f the ensemble from which they are drawn. The av erage is over the instance o f the code, the realization of the channel noise, a nd the realizations of the two forms of decode r noise. T o simplify things, assu me that the number of decoder iterations is ﬁxed at some ﬁn ite ℓ . Let Z be the numb er of incorrect values h eld among a ll d v n variable node-incide nt edges at the end of the ℓ th iteration (for a pa rticular code , cha nnel noise realization, and dec oder noise realization) and let E [ Z ] be the expected value of Z . By co nstructing a martingale through sequ entially revealing all of the random elements and then using the Ho eff ding-Azuma inequa lity , it can be shown that: Theorem 2 (Conc e ntration Ar ound Ex pected V alue): There exi sts a positi ve constan t β = β ( d v , d c , ℓ ) such that for a ny ǫ > 0 , Pr [ | Z − E [ Z ] | > nd v ǫ/ 2] ≤ 2 e − β ǫ 2 n . Pr oof: Follo ws the ba sic ideas of the proofs of [13, The orem 2] or [65, Theorem 4.94 ]. Appendix B gi ves details. A primary communication s ystem performance criteri on is proba bility of error P e ; if the numbe r of inc orrect values Z concen trates, the n s o d oes P e . 7 C. Con ver gence to the Cycle-F ree Ca se The previous theorem showed that the noisy de coding a lgorithm beh av es es sentially deterministically for large n . As now shown, this ensemble average pe rformance con verges to the pe rformance of an associated tree ens emble, which will allo w the assumption of independe nt messa ges. For a gi ven edge whose d irected neighborho od of d epth 2 ℓ is tree-like, let p be the expected number of incorrect messag es receiv ed along this edge (after message noise ) a t the ℓ th it eration, averaged over all graphs, inputs and decode r noise realizations of both types. Theorem 3 (Con vergence to Cycle-F ree Case ): There exists a pos iti ve constant γ = γ ( d v , d c , ℓ ) su ch that for any ǫ > 0 and n > 2 γ /ǫ , | E [ Z ] − nd v p | < n d v ǫ/ 2 . The proof is identical t o the proof of [13, Theorem 2 ]. The basic idea is that the computation tree created by unwrapping the code graph to a particular depth [69 ] a lmost surely has no repeated nodes. The concentration and c on ver gence resu lts directly imply co ncentration a round the average pe rformance of a tree ensemble: Theorem 4 (Conc e ntration Ar ound Cy cle-F ree Case ): The re exist po siti ve con stants β = β ( d v , d c , ℓ ) and γ = γ ( d v , d c , ℓ ) suc h that for any ǫ > 0 an d n > 2 γ /ǫ , Pr [ | Z − nd v p | > nd v ǫ ] ≤ 2 e − β ǫ 2 n . Pr oof: Follows directly from T heorems 2 and 3. D. Density Evolution W ith the conditional ind epende nce a nd con centration results, a ll rand omness is removed from explicit cons ider- ation an d all mess ages are indep endent. The problem reduces to density ev olution, the analysis of a discrete-time dynamical s ystem [62]. The dynamical system state vari able of most interest is the probability of bit error , P e . Denote the proba bility of b it e rror of a code g ∈ C n after ℓ iterations o f decoding by P ( ℓ ) e ( g , ε, α ) , whe re ε is a channe l noise pa rameter (such as noise p ower or cross over proba bility) and α is a decoder noise parameter (suc h as logic gate e rror probability). Then de nsity ev o lution co mputes lim n →∞ E h P ( ℓ ) e ( g , ε, α ) i , where the expec tation is ov er the choice of the code and the various noise realizations. Th e main interest is in the long-term behavior o f the probability of error after performing many iterations. T he long-term behavior of a generic dyna mical s ystem may be a limit cycle or a ch aotic attractor , howe ver density evolution usua lly conv erges to a stab le ﬁxed p oint. Monotonicity (either increasing o r d ecreasing ) with res pect to iteration number ℓ need no t hold, but it often does. If there is a stable ﬁxed p oint, the limiting pe rformance co rresponds to η ∗ = lim ℓ →∞ lim n →∞ E h P ( ℓ ) e ( g , ε, α ) i . In cha nnel coding, certain se ts of parame ters ( g , ε, α ) lead to “ good” performance , in the sens e of sma ll η ∗ , whereas other se ts of paramete rs lead to “bad ” p erformance with large η ∗ . The goa l of dens ity evolution an alysis is to determine the bounda ry between these good and bad sets. Though it is natural to expe ct the pe rformance o f an algorithm to improv e as the quality of its input improves and a s mo re res ources are alloca ted to it, this may not be so. For many decode rs, ho wever , there is a monotonicity property that limit ing behavior η ∗ improves as channel n oise ε de creases and as decoder noise α decrease s. Moreover , just as i n other nonlinear estimation systems f or dimension ality-expanding signals [70]–[72], there is a threshold phenome non such that the limiting probability of error may change precipitously with the values of ε and α . In traditional c oding theo ry , there is n o p arameter α , a nd the goal is often to de termine the range of ε for wh ich η ∗ is z ero. The bo undary is o ften called the decoding threshold and may be den oted ε ∗ ( η ∗ = 0) . A decoding threshold for optimal co des unde r optimal deco ding may be compu ted from the rate of the co de g and the c apacity of the ch annel as a function of ε , C ( ε ) . Since this Shannon limi t threshold is for optimal codes an d de coders, it 8 COMMUNICATION NOISE NOISY COMPUTATION COMPUTATION NOISE NOISELESS COMPUTATION ≈ COMMUNICATION NOISE NOISY COMMUNICATION COMPUTATION NOISE NOISELESS COMPUTATION Fig. 3. Local computation noise may be incorporated into message-passing noise without essential l oss of generality . is clearly an upper bound to ε ∗ (0) for any g i ven code and dec oder . If the tar ge t e rror probability η ∗ is non-zero, then the Shanno n limit threshold is derived from the s o-called η ∗ -capacity , C ( ε ) 1 − h 2 ( η ∗ ) , rathe r tha n C ( ε ) . 7 In the ca se o f f aulty decoders , the Shannon limits also provide uppe r b ounds on the ε -boundary for the set of ( ε, α ) that ac hiev e go od performance. One might hope for a S hannon theoretic characterization of the entire ( ε, α ) - bounda ry , but as noted previously , such resu lts are n ot extant. Alternately , in the next sections, s ets of ( ε , α ) that can ac hiev e η ∗ -reliability for p articular LDPC co des g ∈ C n are ch aracterized using the d ensity ev olution method developed in this se ction. V . E X A M P L E : N O I S Y G A L L AG E R A D E C O D E R Section IV showed that density ev o lution equations determine the performance o f almost a ll codes in the lar ge blocklength regime. Here the den sity ev olution equation for a simple no isy messa ge-pass ing dec oder , a noisy version of Ga llager’ s d ecoding algorithm A [26], [74] , is derived. Th e algorithm has mess age alphabe t M = {± 1 } , wit h messag es in b elief format simply indica ting the estimated sign of a bit. Although this simple decoding algorithm cannot match the performance of belief propa gation du e to its restricted messaging alphabe t M , it is of interest since it is of extremely low complexity and can be analyz ed an alytically [74]. Consider decoding the LDPC-code d o utput of a binary symmetric channel (BSC) with cross over probability ε . At a check no de, the outgo ing mes sage along e dge ~ e is the produc t of all incoming messages exclud ing the one incoming on ~ e , i.e. the che ck no de map Φ is the XOR ope ration. At a variable n ode, the outgoing messag e is the original receiv ed c ode s ymbol u nless a ll incoming messa ges g i ve the o pposite c onclusion. That is , Ψ = ( − y , if µ 1 = · · · = µ d v − 1 = − y , y , otherwise. There is no esse ntial loss of ge nerality by c ombining co mputation noise and mess age-pass ing no ise into a s ingle form of noise, as demo nstrated s chematically in Fig. 3 and proven in [75, Le mma 3.1]. T his noise c ombining is performed in the sequel to reduc e the number o f decoder noise parameters and allow a clean examina tion of the central p henomen on. Thus, each mes sage in the Ga llager algorithm A is passe d over an indepen dent and identical BSC wire with crossover proba bility α . The dens ity evolut ion equa tion lead s to an analytic charac terization of the se t of ( ε, α ) pa irs, which parameterize the noisiness o f the communication system. A. Density Evolution Equation The de nsity evolut ion eq uation is developed for g eneral irregular LDPC e nsembles . Th e state variable of de nsity ev olution, s ℓ , is taken to b e the expected proba bility of bit error at the vari able node s in the large blocklength limit, denoted here as P ( ℓ ) e ( ε, α ) . The original receiv ed me ssage is in e rror with probability ε , thus P (0) e ( ε, α ) = s 0 = ε . The initial variable-to-check mess age is in error with probability (1 − ε ) α + ε (1 − α ) , since it is pa ssed through a BSC( α ). For further iterations, ℓ , the proba bility o f error , P ( ℓ ) e ( ε, α ) , is found by induction. Assume P ( i ) e ( ε, α ) = s i 7 The function h 2 ( · ) is the binary entropy function. The η ∗ -capacity expression is obtained by adjusting capacity by t he rate-distortion function of an equipro bable binary source under frequency of error constraint η ∗ , R ( η ∗ ) = 1 − h 2 ( η ∗ ) [73 ]. 9 for 0 ≤ i ≤ ℓ . No w consider the e rror probability o f a check-to-variable messa ge in the ( ℓ + 1) th iteration. A check-to-variable mes sage e mitted by a check n ode of degree d c along a particular e dge is the product of all the ( d c − 1) incoming messa ges along all other edges. By ass umption, ea ch su ch mes sage is in error with p robability s ℓ and all messa ges are independent. These messag es are pa ssed through BSC( α ) before be ing rece i ved, so the probability of being received in error is s ℓ (1 − α ) + (1 − s ℓ ) α = α + s ℓ − 2 αs ℓ . Due to the XOR o peration, the outgoing mess age will be in error if a n o dd number of these rece i ved messag es are in e rror . The probab ility of this event, averaged over the degree d istrib ution, yields the probab ility 1 − ρ [ 1 − 2( α + s ℓ − 2 αs ℓ )] 2 . Now conside r P ( ℓ +1) e ( ε, α ) , the e rror prob ability at the variable nod e in the ( ℓ + 1) th iteration. Consider an ed ge which is co nnected to a variable node of degree d v . Th e o utgoing variable-to-check mess age along this edge is i n error in the ( ℓ + 1) th iteration if the original receiv ed value is in error an d n ot all inco ming mess ages are receiv ed correctly or if the originally rece i ved value is c orrect b ut all incoming messages are in error . The ﬁrst ev ent h as probability ε 1 −  1 − (1 − α )  1 − ρ [1 − 2( α + s ℓ − 2 αs ℓ )] 2  − α  1 + ρ [ 1 − 2( α + s ℓ − 2 αs ℓ )] 2  d v − 1 ! . The second event has probability (1 − ε )  (1 − α )  1 − ρ [1 − 2( α + s ℓ − 2 αs ℓ )] 2  + α  1 + ρ [1 − 2( α + s ℓ − 2 αs ℓ )] 2  d v − 1 ! . A veraging over the d egree distribut ion and adding the two terms together yields the density ev olution equation in recursiv e form: s ℓ +1 = ε − εq + α ( s ℓ ) + (1 − ε ) q − α ( s ℓ ) . (1) The expressions q + α ( ˇ s ) = λ  1 + ρ ( ω α ( ˇ s )) − 2 αρ ( ω α ( ˇ s )) 2  , q − α ( ˇ s ) = λ  1 − ρ ( ω α ( ˇ s )) + 2 αρ ( ω α ( ˇ s )) 2  , and ω α ( ˇ s ) = (2 α − 1)(2 ˇ s − 1) are used to deﬁn e the de nsity ev olution rec ursion. B. P er formance Ev aluation W ith the density ev olution eq uation esta blished, the performanc e of the cod ing-decoding system with particular values of quality parameters ε and α may be determined. T aking the bit error probability as the state v ariable, stable ﬁxed p oints of the de terministic, discrete-time, dynamical system a re to be found. Usua lly o ne would want the probability of error to con ver ge to ze ro, but sinc e this might not b e possible, a weaker pe rformance criterion may be n eeded . T o start, con sider partially no iseless cas es. 1) Noisy Channe l, Noise less Decode r: For the no iseless d ecode r case, i.e. α = 0 , it has b een kn own that there are thres holds on ε , be low which the probability of error goes to ze ro as ℓ increases, and above which the prob ability of error g oes to some large value. The se ca n b e fou nd analytically for the G allager A a lgorithm [74 ]. 10 2) Noiseless Channe l, Noisy Decode r: For the noisy Gallager A system un der c onsideration, the prob ability o f error do es not go to zero as ℓ goes to inﬁnity for any α > 0 . This ca n be seen by considering the case of the perfect o riginal chan nel, ε = 0 , an d a ny α > 0 . The density ev olution equ ation red uces to s ℓ +1 = q − α ( s ℓ ) , (2) with s 0 = 0 . The recursion d oes not h av e a ﬁxed point at zero, and since err or probability is bou nded belo w by zero, it must increase. The deri vati ve is ∂ ∂ s q − α ( s ) = λ ′  1 − ρ ( ω α ( s )) + 2 αρ ( ω α ( s )) 2  ρ ′ ( ω α ( s ))(2 α − 1) 2 , which is greater than zero for 0 ≤ s ≤ 1 2 and 0 ≤ α ≤ 1 2 ; thus the error ev olution forms a monotonically increas ing sequen ce. Since the sequence is mon otone increasing starting from zero, and there is no ﬁxed point at zero, it follo ws tha t this con verges to the smallest rea l solution of s = q − α ( s ) since the ﬁxed point ca nnot be jumped due to monotonicity . 3) Noisy Channel, Noisy Deco der: The same phenome non must also happen if the starting s 0 is positiv e, ho wever the value to which the density ev olution conv erges is a non-zero ﬁxed point s olution of the original equation (1), not o f (2), a nd is a function of both α and ε . Intuiti vely , for somewhat lar ge initial values of ε , the n oisy decode r decreas es the proba bility of e rror in the ﬁrst few iterations, just like the noiseles s one, but when the e rror probability becomes close to the internal deco der e rror , the p robability of e rror settles a t that level. T his is summarized in the follo wing proposition. Pr oposition 1: Fina l error prob ability η ∗ > 0 for any LDPC ensemble decoded using the noisy Gallager A system deﬁne d in Se ction V , for every de coder no ise level α > 0 a nd every chan nel noise level ε .  The fact tha t probability of error ca nnot asymptotically be driv en to zero with the n oisy Gallager d ecode r is expected yet is s eemingly displeas ing. In a practical scenario, howe ver , the ability to dri ve P e to a very s mall number is also desirable. As such, a p erformance objective of achieving P e less than η is deﬁned and the wor st channe l (ordered by ε ) for which a decode r with noise level α can a chieve that objec ti ve is determined. T he ch annel parameter ε ∗ ( η , α ) = sup { ε ∈ [0 , 1 2 ] | lim ℓ →∞ P ( ℓ ) e ( g , ε, α ) < η } is called the thres hold. For a large interv al of η values, there is a single threshold value belo w which η -reliable communication is possible a nd ab ove wh ich it is no t. Alternativ ely , on e ca n de termine the probability of error to which a system with p articular α a nd ε ca n be driven, η ∗ ( α, ε ) = lim ℓ →∞ P ( ℓ ) e , and see whether this value is sma ll. In order to ﬁnd the thresh old in the case of α > 0 and ε > 0 , the real ﬁxed point solutions of d ensity evoluti on recursion (1) need to be found. The real s olutions of the polynomial equation in s , ε − εq + α ( s ) + (1 − ε ) q − α ( s ) − s = 0 are denoted 0 < r 1 ( α, ε ) ≤ r 2 ( α, ε ) ≤ r 3 ( α, ε ) ≤ · · · . 8 The ﬁ nal probability of error η ∗ is determined by the r i , since these are ﬁxed points of the recursion (1). The real solutions of the polynomial equa tion in s , s − q − α ( s ) 1 − q + α ( s ) − q − α ( s ) − s = 0 , (3) are denoted 0 < τ 1 ( α ) ≤ τ 2 ( α ) ≤ · · · . 8 The threshold ε ∗ as well as the region in the α − ε plan e where the d ecoder improves performance over no decoding are determined by the τ i , since (3) is ob tained by solving recursion (1) for ε and setting equal to zero. For particular ensembles o f LDPC codes , these values c an be compute d ana lytically . For these particular e nsembles , it can be determined whether the ﬁxed points are s table or unstable. Moreov er , various monotonicity results can be established to show that ﬁxed points ca nnot be jumped . Analytical express ions for the r i ( α, ε ) an d τ i ( α ) are d etermined for the (3,6 ) regular LDPC code by solving the appropriate polynomial equ ations an d n umerical ev aluations of the r i expressions are shown as thin lines in Fig. 4 as func tions of ε for ﬁxed α . The point w here r 1 ( α, ε ) = ε is τ 1 ( α ) a nd the point where r 2 ( α, ε ) = ε is τ 2 ( α ) . In Fig. 4, these are points wh ere the thin lines cros s. 8 The number of real solutions can be determined through Descartes’ rule of signs or a similar tool [76]. 11 0 0.005 0.01 0.015 0.02 0.025 0.03 0 0.05 0.1 0.15 0.2 0.25 ε η * r 3 ( α , ε ) r 2 ( α , ε ) ε r 1 ( α , ε ) Fig. 4. Thick line sho ws ﬁnal error probability , η ∗ , after decoding a C ∞ (3 , 6) code with the noisy Gall ager A algorithm, α = 0 . 005 . This is determined by the ﬁxed points of density ev olution, r i ( α, ε ) , sho wn wi th thin lines. By analyzing the dynamica l system equ ation (1) for the (3,6 ) c ode in detail, it can be s hown that r 1 ( α, ε ) and r 3 ( α, ε ) a re stable ﬁxed points of de nsity evolution. Contrarily , r 2 ( α, ε ) is an unstable ﬁxed point, which dete rmines the b oundary between the regions of attraction for the tw o stable ﬁxed points. Since r 1 ( α, ε ) and r 3 ( α, ε ) a re stable ﬁxed points, the ﬁn al error probability η ∗ will take on one of these two values, d epending on the starting point of the recursion, ε . The thick line in Fig. 4 shows the ﬁna l error prob ability η ∗ as a function o f initial error p robability ε . One ma y no te that η ∗ = r 1 is the desirable small e rror probability , whe reas η ∗ = r 3 is the undesirable large error p robability and that τ 2 delimits these two regimes. The τ ( α ) points determine when it is beneﬁ cial to use the dec oder , in the sense that η ∗ < ε . By v arying α (as if in a s equenc e of plots like Fig. 4), an α − ε region whe re the de coder is beneﬁc ial is d emarcated; this is sh own in Fig. 5. The function τ 2 ( α ) is the η -reliability dec oding thresh old for large ranges o f η . Notice that the previously known s pecial case , the dec oding threshold of the noise less decoder , can be recovered from these results. The deco ding threshold for the noiseless dec oder is den oted ε ∗ B RU and is eq ual to the followi ng expression [74]. ε ∗ B RU = 1 − √ σ 2 , where σ = − 1 4 + q − 5 12 − b 2 + q − 5 6 + 11 4 √ − 5 / 12 − b 2 and b = 8 3  2 83 + 3 √ 993  1 3 − 1 3  83 + 3 √ 993 2  1 3 . This v alue is recovered from noisy deco der resu lts by noting tha t η ∗ ( α = 0 , ε ) = 0 for ε ∈ [0 , ε ∗ B RU ] , which are the ordinate intercepts of the region in Fig. 5. T o provide a better se nse of the pe rformance of the nois y Gallager A a lgorithm, T able I lists some values of α , ε , and η ∗ (numerical ev aluations are listed and an example of an analytical expression is gi ven i n Appe ndix C). As can be seen from these results, particularly from the τ 2 curve i n Fig. 5, the error probability performance of the system degrades gracefully as nois e is adde d to the de coder . Returning to thresh old charac terization, an an alytical expression for the thresh old within the region to u se dec oder is: ε ∗ ( η , α ) = η − q − α ( η ) 1 − q + α ( η ) − q − α ( η ) , which is the solution to the p olynomial e quation in ˇ ǫ , ˇ ǫ − ˇ ǫq + α ( η ) + (1 − ˇ ǫ ) q − α ( η ) − η = 0 . 12 0 0.002 0.004 0.006 0.008 0 0.01 0.02 0.03 0.04 α ε τ 2 ( α ) τ 1 ( α ) Fig. 5. Decoding a C ∞ (3 , 6) code with the noisy Gallager A algorithm. Region where it is beneﬁcial to use decoder is belo w τ 2 and abo ve τ 1 . T ABL E I P E R F O R M A N C E O F N O I S Y G A L L AG E R A A L G O R I T H M F O R ( 3 , 6 ) C O D E α ε ∗ (0 . 1 , α ) η ∗ ( α, ε ∗ ) η ∗ ( α, 0 . 01) 0 0 . 03946 36562 0 0 1 × 10 − 10 0 . 03946 36560 7 . 8228 × 10 − 11 1 . 3333 × 10 − 11 1 × 10 − 8 0 . 03946 36335 7 . 8228 × 10 − 9 1 . 3333 × 10 − 9 1 × 10 − 6 0 . 03946 13836 7 . 8234 × 10 − 7 1 . 3338 × 10 − 7 1 × 10 − 4 0 . 03923 59948 7 . 8866 × 10 − 5 1 . 3812 × 10 − 5 3 × 10 − 4 0 . 03877 81564 2 . 4050 × 10 − 4 4 . 4357 × 10 − 5 1 × 10 − 3 0 . 03714 77336 8 . 4989 × 10 − 4 1 . 8392 × 10 − 4 3 × 10 − 3 0 . 03219 84070 3 . 0536 × 10 − 3 9 . 2572 × 10 − 4 5 × 10 − 3 0 . 02660 99758 6 . 3032 × 10 − 3 2 . 4230 × 10 − 3 The threshold is drawn for s ev eral v alues of η in Fig. 6. A thresho ld line determines the equi valence of channe l noise and decoder noise with res pect to ﬁnal probability o f error . If for example, the binary s ymmetric channels in the s ystem are a resu lt of hard-detec ted A WGN channe ls, such a line may be used to derive the e quiv alent channe l noise power for deco der noise p ower o r vice versa. Th reshold lines therefore provide gu idelines for power allocation in communication s ystems. 0 0.002 0.004 0.006 0.008 0 0.01 0.02 0.03 0.04 α ε η = 0.005 η = 0.002 η = 0.001 η = 0.0005 η = 0.0001 η = 0.00005 η = 0.00001 Fig. 6. η -thresholds (gray lines) for decoding a C ∞ (3 , 6) code with the noisy Gallager A algorithm within the re gion to use decoder (delimited with red line). 13 0 0.002 0.004 0.006 0.008 0.01 0.012 0.014 0.016 0 0.01 0.02 0.03 0.04 0.05 α ε Fig. 7. Region to use decoder for Bazzi et al. ’ s optimized rate 1 / 2 LDP C code with noisy Gallager A decoding (black) is contained within the regio n to use dec oder for a rate 1 / 2 LDPC code in Bazzi et al. ’ s optimal family of codes with a = 1 / 10 (green) and contains the region to use decoder for the C ∞ (3 , 6) code (gray). C. Code Optimization At this p oint, the bit error performance of a sys tem has simply bee n measured; no attempt has be en made to optimize a code for a particular decoder and s et of p arameters. For f ault-free de coding, it has bee n demons trated that irre gular code ensembles can p erform much b etter than regular code e nsembles like the (3,6) LDPC considered above [74], [77]. One might ho pe for similar improvements when LDPC code d esign takes dec oder noise into accoun t. The space o f s ystem p arameters to be conside red for noisy d ecoders is much lar ger than for noise less decode rs. As a ﬁrst step, con sider the ensemble of rate 1 / 2 LDP C co des that were optimized by Bazzi et al. for the fault-f ree G allager A d ecoding algorithm [74]. The left degree distribution is λ ( ζ ) = aζ 2 + (1 − a ) ζ 3 and the right degree d istrib ution is ρ ( ζ ) = 7 a 3 ζ 6 + 3 − 7 a 3 ζ 7 , where the op timal a is speciﬁed analytically . Numerically , a opt = 0 . 1115 . . . . Measuring the pe rformance of this code with the noisy Gallager A dec oder yields the region to use d ecoder shown in Fig. 7; the region to use decode r for the (3,6) code is shown for comp arison. By essentially any c riterion of performance, this optimized code is better than the (3,6) code. Are there othe r codes that c an perform better on the faulty decod er than the co de optimized for the fault- free decode r? T o see whether this is p ossible, arbitraril y restrict to the family of ens embles that were found to contain the optimal degree distrib ution for the fault-free d ecoder and take a = 1 / 10 . Also let α = 1 / 500 be ﬁxed. The numerical v alue of the threshold ε ∗ 1 / 10 (1 / 10 , α ) = 0 . 048239 , whereas the nu merical v alue o f the thres hold ε ∗ a opt (1 / 10 , α ) = 0 . 047857 . In this sense , the a = 1 / 10 code is better than the a = a opt code. In fact, as seen in Fig. 7, the region to us e dec oder for this a = 1 / 10 code contains the region to us e d ecoder for the a opt code. On the o ther h and, the ﬁnal error p robability when ope rating a t threshold for the a = 1 / 10 code η ∗ 1 / 10 ( α, ε ∗ 1 / 10 (1 / 10 , α )) = 0 . 0186 9 , where as the ﬁnal err or probability whe n operating at thresho ld for the a = a opt code is η ∗ a opt ( α, ε ∗ a opt (1 / 10 , α )) = 0 . 0176 6 . So in this s ense, the a = a opt code is better than the a = 1 / 10 cod e. The f act that highly optimized ensembles u sually lead to more simultaneous critical p oints is the main complication. If both thresh old an d ﬁn al bit error p robability a re pe rformance c riteria, there is no total order on code s and therefore there may be n o notion of an optimal c ode. V I . E X A M P L E : N O I S Y G AU S S I A N D E C O D E R It is also of interest to analyze a noisy version of the belief propagation dec oder applied to the o utput of a continuous -alphabet channe l. Density e volution for belief prop agation is dif ﬁcult to ana lyze even in the noiseles s 14 decode r ca se, and so a Gaus sian approximation method [78] is us ed. The state variables are one-dimensiona l rather than inﬁnite-dimensiona l as for full analysis o f b elief propag ation. T he spec iﬁc node computations c arried out by the dec oder are as in be lief p ropagation [13]; the se can be app roximated by the fun ctions Φ a nd Ψ deﬁned below . The messag es and noise model are speciﬁed in terms of the approximation. Section V had considered decoding the output of a BSC with a d ecoder that was constructed wi th BSC components and Proposition 1 had s hown that probability of bit error c ould ne ver b e dri ven to zero. Here, the probability of bit e rror does in fact go to ze ro. Consider a binary input A WGN cha nnel with variance ε 2 . Th e output is de coded using a noisy Gau ssian dec oder . For s implicity , only regular LDPC c odes are considered . T he m essa ges that are passe d in this d ecoder are real-v a lued, M = R ∪ {±∞} , and are in belief format. The variable-to-check me ssage s in the zeroth iteration are the log-lik elihood ratios c omputed fr om the cha nnel output symbols, ν ( y ) , ν v → c = ν ( y ) = log p ( y | x = 1) p ( y | x = − 1) . The chec k node takes the received version s of thes e messag es, µ v → c , as input. The no de implements a map ping Φ whose output, ν c → v , satisﬁes: etanh( ν c → v ) = d c − 1 Y i =1 etanh( µ v → c i ) , where the produc t is taken over me ssage s on all incoming edges except the one o n which the messag e will be outgoing, and etanh( ˇ v ) = 1 √ 4 π ˇ v Z R tanh v 2 e − ( v − ˇ v ) 2 4 v dv . The c heck n ode mapping is motiv ated by Gaussian likelihood computations. For the sequel, it is u seful to deﬁne a slightly diff erent function φ ( ˇ v ) = ( 1 − etanh( ˇ v ) , ˇ v > 0 1 , ˇ v = 0 which can be approximated as φ ( ˇ v ) ≈ e a ˇ v c + b , with a = − 0 . 452 7 , b = 0 . 0218 , c = 0 . 86 [78]. For iterations ℓ ≥ 1 , the variable node takes the receiv ed versions of the c → v mes sages , µ c → v , a s inputs. The mapping Ψ yields o utput ν v → c giv en by ν v → c = ν ( y ) + d v − 1 X i =1 µ c → v i , where the sum is taken over receiv ed messag es from the neigh boring check node s except the one to which this messag e is o utgoing. Ag ain, the operation of the vari able n ode is motiv ated by Gau ssian likelihood computations . As in Sec tion V, local compu tation noise is c ombined into message-pa ssing noise (Fig. 3). T o mo del qua ntization [25] or random phenome na, consider e ach message passed in the decoder to be corrupted by signal-independ ent additiv e noise which is bounded as − α/ 2 ≤ w ≤ α/ 2 . This c lass of noise mode ls includes uniform noise, a nd truncated Gaussian noise, among others. If the noise is symmetric, then The orem 1 applies. Foll owing the v on Neumann error mo del, each noise realization w is assumed to be indep endent. A. Density Evolution Equation The deﬁnition of the c omputation rules and the nois e model may be used to derive the a pproximate density ev olution equation. The one-dimensional state variable ch osen to be tracked is s , the mean b elief at a variable node. The symmetry condition relating me an belief to belief v ariance [13] , [78] is enforced. Thus, if the all- one codeword was transmitted, then the value s going to + ∞ implies tha t the density of ν v → c tends to a “mass point at inﬁnity , ” which in turn implies that P e goes to 0 . 15 T o bound decoding performanc e und er any noise model in the class of additiv e b ounded noise, cons ider (non- stochas tic) worst-case noise. Assuming that the all-one codew ord was se nt, all messag es sh ould be as po siti ve as possible to move tow ards the correct decode d codeword (mean b eliefs o f + ∞ indicate perfect conﬁde nce in a bit being 1 ). Consequently , the worst bounded nois e that may be imposed is to sub tract α/ 2 from all messag es that are pas sed; this requires k nowledge of the trans mitted cod ew ord being all-one. If another codeword is transmitted, then certain messag es would h av e α/ 2 ad ded instead o f s ubtracted. Such a worst-case noise model do es no t meet the conditions of Theo rem 1, but transmiss ion o f the all-one codeword is ass umed non etheless. If there were a n adversa ry with knowledge of the transmitted co dewor d imposing worst-case noise on the decod er , then probability of bit error would be conditionally indepe ndent o f the transmitted codeword, a s g i ven in Appe ndix A-1. Note that the adversary is res tricted to se lecting each noise realization indep endently . More complicated and devious error p atterns in s pace or in time are no t possible in the von Ne umann error mode l. More over , the performance criterion is p robability of bit error rather than probability of block e rror , so c omplicated e rror patterns would provide no great beneﬁ t to the adversary . Since the no ise is c onditionally de terministic given the transmitted codeword, d eri vation of the density evolution equation is mu ch simpliﬁed. An induction argument is used , and the ba se ca se is s 0 = 2 ε 2 , where ε 2 is the channel noise power . This follo ws from the log-likelihood computation for an A WGN communication channe l with input alphabe t X = {± 1 } . The induc ti ve assumption in the induc tion ar gument is s ℓ − 1 . This messa ge is communica ted over message -passing noise to get s ℓ − 1 − α 2 . Next the ch eck no de co mputation is made to y ield φ − 1  1 − [1 − φ ( s ℓ − 1 − α 2 )] d c − 1  . By the inductive assumption, all messa ges will be eq uiv a lent; that is why the product is a ( d c − 1) -f old product of the same quantity . Th is value is c ommunicated over mes sage-pas sing no ise to g et φ − 1  1 − [1 − φ ( s ℓ − 1 − α 2 )] d c − 1  − α 2 . Finally the vari able-node computation yields s 0 + ( d v − 1) n φ − 1  1 − [1 − φ ( s ℓ − 1 − α 2 )] d c − 1  − α 2 o . Again, a ll messages will be eq ui valent so the sum is a ( d v − 1) -fold sum of the same quantity . Thus t he density ev olution equ ation is s ℓ = 2 ε 2 − ( d v − 1) α 2 + ( d v − 1) n φ − 1  1 − [1 − φ ( s ℓ − 1 − α 2 )] d c − 1 o . (4) B. P er formance Ev aluation One might wonde r whether there are sets of noise parame ters α > 0 and ε > 0 such that s ℓ → + ∞ . Indeed there are, and there is a threshold p henomen on just like Chu ng et al. s howed for α = 0 [78]. Pr oposition 2: Fina l error probability η ∗ = 0 for LDPC ens embles decod ed using the noisy Gau ssian system deﬁned in Section VI, for b inary-input A WGN chan nels with noise lev el ε < ε ∗ ( α ) . Pr oof: S ubstituting s = + ∞ into (4) de monstrates that it is a stable ﬁxed p oint. It may further b e veriﬁed that the dynamica l system procee ds tow ard that ﬁ xed po int if ε < ε ∗ ( α ) . Unlike Section V whe re the ε ∗ ( η , α ) thresholds could be evaluated analytically , only numerical ev a luations o f these ε ∗ ( α ) thres holds a re poss ible. Th ese are s hown in Fig. 8 for three regular L DPC ens embles with rate 1 / 2 , namely the (3,6) e nsemble, the (4,8) ens emble, and the (5,10) en semble. As can be ob served, thresh olds decrea se smo othly as the deco der n oise level increas es. Moreover , the o rdering of the codes remains the same for a ll le vels of decode r noise depicted. Code optimization rema ins to b e done. 16 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 0.4 0.45 0.5 0.55 0.6 0.65 0.7 0.75 0.8 0.85 0.9 α ε * Fig. 8. Thresholds for decoding the C ∞ (3 , 6) code (triangle), the C ∞ (4 , 8) code (quadrangle), and the C ∞ (5 , 10) (pentangle), each with the noisy Gaussian approximation algorithm. Notice that the ordinate intercepts are ε ∗ C RU (3 , 6) = 0 . 8747 , ε ∗ C RU (4 , 8) = 0 . 8323 , and ε ∗ C RU (5 , 10) = 0 . 7910 , [78, T able I]. The b asic reaso n for the dispa rity between Prop ositions 2 and 1 is that here, the noise is bound ed whereas the messag es a re unbounded . Thus once the messag es g ro w lar ge, the noise has esse ntially n o ef fect. T o use a term from [67], once the de coder reaches the br eakout value , noise cann ot stop the decoder from ach ie ving Shann on reliability . Perhaps a pea k amp litude con straint o n mes sages would provide a more realistic computation model, but the equiv alent of Proposition 2 ma y no t hold. Quantiﬁed data processing inequalities may pro vide insigh t into what forms of noise and messag e constraints are truly limiting [34], [35]. V I I . A P P L I C A T I O N : R E L I A B L E M E M O R I E S C O N S T R U C T E D F RO M U N R E L I A B L E C O M P O N E N T S In Se ction I, co mplexity an d reliability were cas t as the primary limit ations on practica l de coding. By considering the d esign o f f ault masking techn iques for memory s ystems, a commu nication problem beyond Fig. 1, b oth complexity and r eliability ma y be explicitly c onstrained. Inde ed, the problem of con structing reliable informati on storage d evices from unreliable compo nents is ce ntral to fault-tolerant c omputing, an d determining the information storage cap acity of such devices is a long-standing op en problem [79]. This problem is related to p roblems in distrib uted information storage [80] and is intimately tied to the performance of c odes unde r faulty de coding. The analysis techniques developed thus far may be u sed directly . In particular , one may co nstruct a memory a rchitecture with noisy registers and a noisy LDPC correcting network. At ea ch time step, the correcting network deco des the register c ontents and restores them. The correcting network prev ents the codeword stored in the registers from wandering too far away . T aylor and others have shown that there exist non-zero lev els of comp onent noisiness su ch that the LDPC-bas ed construction achieves no n-zero storage capac ity [54], [55], [63]. Results as in Section V may be used to precisely c haracterize storage capacity . Before procee ding with an achievabili ty result, requisite deﬁnitions and the problem stateme nt are given [54]. Deﬁnition 6: An elementar y ope ration is any Boolean function of two binary o perands. Deﬁnition 7: A s ystem is co nsidered to be co nstructed from compone nts , which are devices that either perform one elementary operation o r store one bit. Deﬁnition 8: The complexity χ of a system is the nu mber of c omponents within the sy stem. Deﬁnition 9: A memory sys tem that stores k information b its is said to have a n information storage capa bility of k . Deﬁnition 10: Conside r a seque nce of memories { M i } , ordered according to their information storage capa bility i (bits). The seque nce { M i } is stable if it satisﬁes the follo wing: 1) F or any k , M k must h av e 2 k allowed inputs denoted { I k i } , 1 ≤ i ≤ 2 k . 2) A class of states, C ( I k i ) , is associated with each input I k i of M k . The clas ses C ( I k i ) and C ( I k j ) must be disjoint for all i 6 = j and all k . 3) The complexity o f M k , χ ( M k ) , must be bounde d by θ k , where redundancy θ is ﬁxed for all k . 17 4) At ℓ = 0 , let one of the inputs from { I k i } be stored in eac h memory M k in the s equenc e of memories { M i } , with no furt her inputs in times ℓ > 0 . Let I k i denote the particular input stored in memory M k . Let λ k i ( T ) d enote the probability tha t the state of M k does not belong to C ( I k i ) at ℓ = T and further let P max k ( T ) = max i λ k i ( T ) . T hen for a ny T > 0 and δ > 0 , there must exist a k such that P max k ( T ) < δ . The demarcation of c lasses of states is equiv alent to dema rcating dec oding regions. Deﬁnition 11: The storage capacity , C , of memory is a number such that there exist stable memory sequences for a ll memory redundan cy values θ greater than 1 / C . Note that un like cha nnel capa city for the communication problem, there is no informational deﬁnition of storage capac ity that is known to go with the operationa l deﬁnition. The bas ic prob lem then is to determine storage capacity , which is a mea sure of the circuit c omplexity required to achieve arbitraril y reliable information storage. Th e c ircuit c omplexity must be linea r in blockleng th, a prop erty satisﬁed by systems with message -passing correcting networks for LDPC c odes. Although P roposition 1 shows that Sh annon reliability is n ot a chiev able for any noisy Gallager A deco der , the deﬁnition of sta ble information s torage d oes not require this. By only req uiring ma intenance within a deco ding region, the deﬁnition implies tha t either the conten ts o f the memory may be rea d-out in c oded form or e quiv a lently that the re is a noiseless output de vice that yields decod ed information; call this noiseles s o utput device the silver decode r . Consider the cons truction o f a memory with n oisy registers as s torage elements. Thes e registers are co nnected to a noisy Ga llager A LDPC decod er (as desc ribed in Section V), wh ich takes the register values as inputs and stores its computational results back into the registers. T o ﬁn d the storage ca pacity of this cons truction, ﬁrst compute the complexity (presu pposing that the cons truction will yield a stab le se quence o f memories). The Gallager A check node operation is a ( d c − 1) -input XOR gate, which ma y be constructed from d c − 2 two-input XOR gates . A variable node determines whether its d v − 1 inpu ts are a ll the same and then compares to the original receiv ed v alue. Let D d v denote the complexity of this logic. The outpu t of the comparison to the original recei ved value is the v alue of the consensu s v iew . One c onstruction to implement the consensu s log ic is to OR together the outputs of a ( d v − 1) -input AND gate and a ( d v − 1) -input AND gate with in verted inpu ts. This is then XO Red with the stored value. Such a circuit can be implemented with 2( d v − 2) + 2 compone nts, so D d v = 2 d v − 2 . The storage is ca rried out in n registers. T he total complexity of the memory M k , χ ( M k ) C n ( d v ,d c ) , is χ ( M k ) C n ( d v ,d c ) = n (1 + 2 d v − 2 + d v ( d c − 2)) = n ( d v d c − 1) . The information storage capa bility is n times the rate of the co de, R . The c omplexity of a n irredundan t memory with the same storage capability is χ irr n = Rn . Hence , the redundan cy is χ ( M k ) C n ( d v ,d c ) χ irr n = n ( d v d c − 1) Rn ≤ ( d v d c − 1) 1 − d v /d c which is a con stant. By [65, Lemma 3.22 ], the inequality almost holds with e quality with high probability for large n . For the (3 , 6) regular LDPC code, the redundancy v alue is 34 , s o C = 1 / 34 , if the cons truction does in fact yield stable memories. The cond itions u nder which the memory is s table depen ds o n the s ilver de coder . Since silver dec oder co mplexity does not enter , ma ximum likelihood should b e used. The Gallager lower bou nd to the ML de coding threshold for the (3 , 6) regular LDPC code is ε ∗ GLB = 0 . 091475 5 [81, T able II]. Re call from F ig. 5 tha t the decoding threshold for G allager A decod ing is ε ∗ B RU = 0 . 039463 6562 . If the probab ility of bit error for the correcting network in the memory stays wit hin the decoding threshold of the silver deco der , the n stability follows. Thus the que stion reduces to determining the sets of compon ent n oisiness lev els ( α, ε ) for which the deco ding circuit a chieves ( η = ε ∗ M L ) -reliabilit y . Consider a memory system where b its are stored in registers with probability α r of ﬂ ipping at each time step. An LDPC codeword is stored in these registers; the probability of incorrect storage at the ﬁrst time s tep is ε . At each iteration, the v ariable node value from the correcting network is p laced in the register . This stored value is used in the su bseque nt Gallager A variable node co mputation rathe r than a rece i ved value from the input pins. Suppose that the compone nt noise values in the c orrecting network may be parameterized as in Section V. Then a 18 0 0.002 0.004 0.006 0.008 0 0.01 0.02 0.03 0.04 α ε Fig. 9. For a memory system constructed with noisy registers and a (3 , 6) LDPC Gallager A correcting netwo rk, the region R (delimited by black line) comprises the “region to use decoder” and its hypograph . slight modiﬁcation of the analysis in Se ction V yields a density evoluti on equation s ℓ +1 = ε 2 − ε 2 q + α ( s ℓ ) + (1 − ε 2 ) q − α ( s ℓ ) , where ε 2 = s ℓ (1 − α r ) + α r (1 − s ℓ ) . T here is a “region to us e de coder” for this system, just as in Section V. If α r = α , this region is s hown in Fig. 9, and is slightly smaller tha n the region in Fig. 5. Denote this region and its hypograph as R . It follo ws that ( η = ε ∗ B RU ) -reliabilit y is a chieved for R . Since ε ∗ B RU -reliability is a chiev able, ε ∗ GLB -reliability is a chiev able by mono tonicity . Thus the cons truction yields stable memories. Pr oposition 3: Le t R be the set of memory compo nent noise pa rameters ( α, ε ) within the region to use decod er or its hy pograph corresponding to a s ystem with a Ga llager A correcting n etwork for the (3 , 6) LDP C code, d epicted in Fig. 9. Then a s equen ce of memories constructed from R -compon ents have a storage capacity lower boun ded as C ≥ 1 / 34 . This may be directly generalized for any choice of cod e ens emble as follows. Theorem 5 : Le t R b e the (computab le) set of memory co mponent noise parameters ( α, ε ) within the r egion to use d ecoder or its hypo graph corresponding to a system with a Gallager A correc ting network for the ( λ, ρ ) LDP C code. Then a s equen ce of memories constructed from R -compon ents have a storage capac ity lower bound ed as C ≥ 1 − λ ′ (1) /ρ ′ (1) λ ′ (1) ρ ′ (1) − 1 . The bound reduces to (1 − d v /d c ) / ( d v d c − 1) for regular codes . This theorem gi ves a precise a chiev ability res ult that bo unds storage capacity . It also implies a code ens emble optimization problem similar to the one in Section V -C. The que stion of an optimal a rchitecture for memory systems howe ver rema ins o pen. V I I I . C O N C L U S I O N S Loeliger et al. [7] had o bserved tha t dec oders a re robust to nonidealities and noise in physica l implementations, howe ver they had no ted that “the quantitati ve analysis of these e f fects is a challeng ing the oretical problem. ” Th is work h as taken steps to addres s this cha llenge by cha racterizing robustness to decoder noise. The extension of the density ev o lution method to the c ase of faulty decoders allo ws a simpliﬁed means of asymptotic performance cha racterization. Res ults from this method show that in certain case s Shannon reliabilit y is not a chiev able (Propos ition 1), whereas in other cases it is ac hiev able (Proposition 2). In either case , howev er , the degradation o f a suitably deﬁned decoding threshold is smooth with increa sing decoder noise, whether in circuit nodes o r circuit wires. Due to this smoothness, cod es optimized for f ault-free decoders do w ork well with faulty decode rs, h owe ver op timization of codes for s ystems with faulty de coders remains to be studied. No attempt was ma de to app ly fault masking methods to develop decoding algorithms with improved performance in the prese nce of noise . One ap proach might be to us e coding within the dec oder so as to reduc e the values of α . Of 19 course, the within-decoder code would need to be decoded. There are also more direct circuit-oriented techniqu es that ma y be applied [82], [83]. Follo wing the conc ept o f co ncatenate d c odes, c oncatena ted dec oders may also be promising. The basic idea of using a ﬁrst (noiseless) d ecoder to correct ma ny errors a nd then a s econd (noiseless ) decode r to clean things up was alread y p resent in [61], but it ma y b e extended to the faulty deco der setting. Reducing power cons umption in d ecoder circuits has been a n acti ve area of rese arch [37], [ 84]–[90], howe ver power reduction o ften h as the ef fect of increasing noise in the dec oder [91]. The tradeoff developed b etween the quality of the communication cha nnel and the q uality of the de coder may p rovide guidelines for a llocating res ources in communication system design. Analysis of other decoding algorithms with o ther error models will p resumably yield results s imilar to those obtained he re. For g reater generality , one might move beyond simple LDPC codes a nd c onsider arbitrary codes decode d with very gene ral iterati ve dec oding circuits [90] with suitable error models. An even more gene ral model of computation such as a T uring mac hine or beyond [92] does not seem to have an obvious, appropriate error model. Even just a bit of imagination provides numerous mo dels of cha nnel noise and c ircuit faults that may be in vestigated in the future to provide further insights into the funda mental limits of n oisy c ommunication and computing. A C K N O W L E D G M E N T I thank R ¨ udiger L. Urban ke and ˙ I. Emre T elatar for several e nlightening discus sions, for encou ragement, a nd for hosting my visit at EPF L. I a lso thank the anonymous revie wers, Sanjoy K. Mitter , G. David Forney , and V ivek K Goyal for ass istance in improving the pap er . Tha nks a lso to Shas hi Kiran Chilappag ari for telling me about h is work. A P P E N D I X A P R O O F O F T H E O R E M 1 Let x ∈ C n be a codeword a nd let Y denote the c orresponding chann el output Y = xZ (where the notation means pointwise multiplication on length n vec tors). Note that Z is equal to the ch annel output observation wh en x is all- one. The goal is to show that messages sent during the decoding proces s for case s when the received codeword is e ither xZ o r x correspond. Let ˙ n i be an arbitrary v ariable node and let ˙ n j be one o f its ne ighboring check node s. L et ν ( ℓ ) ij ( y ) and µ ( ℓ ) ij ( y ) denote the v ariable-to-check message from ˙ n i to ˙ n j at the respectiv e terminals in iteration ℓ , assuming receiv ed value y . Similarly , let ν ( ℓ ) j i ( y ) and µ ( ℓ ) j i ( y ) be the check-to-variable messa ge from ˙ n j to ˙ n i at the respe cti ve terminal in iteration ℓ assuming received value y . By De ﬁnition 2, the cha nnel is memoryless binary-input ou tput-symmetric and it may b e modeled multiplicativ ely as Y t = x t Z t , (5) where { Z t } is a seque nce of i.i.d. random variables and t is the chann el usage time. The validity of the multiplicati ve model is shown in [13 , p. 605] and [65 , p. 1 84]. The proof proce eds by induction and so the base cas e is established ﬁrst. By the multiplicati ve model (5), ν (0) ij ( y ) = ν (0) ij ( xz ) . Re calling that x i ∈ {± 1 } , by the v ariable nod e symme try condition (Deﬁ nition 3 ) which includes computation noise u (0) ˙ n i , it follo ws that ν (0) ij ( y ) = ν (0) ij ( xz ) = x i ν (0) ij ( z ) . Now take the wire noise w (0) ij on the mes sage from ˙ n i to ˙ n j into ac count. It is symmetric (De ﬁnition 5) an d s o ν (0) ij ( y ) = x i ν (0) ij ( z ) implies a s imilar property for µ (0) ij . In particular , µ (0) ij ( y ) = Ξ( ν (0) ij ( y ) , w (0) ij ) (6) = Ξ( x i ν (0) ij ( z ) , w (0) ij ) = x i Ξ( ν (0) ij ( z ) , x i w (0) ij ) 20 where the last step follows bec ause x i ∈ {± 1 } and s o it can b e taken outside of Ξ by Deﬁn ition 5, when it is put back in for the wire noise. Now s ince x i ∈ {± 1 } and s ince the wir e noise is sy mmetric about 0 by Deﬁn ition 5 , x i Ξ( ν (0) ij ( z ) , x i w (0) ij ) will correspond to x i µ (0) ij ( z ) , in the sens e that error event proba bilities will be ide ntical. Assume that µ ( ℓ ) ij ( y ) co rresponds to x i µ ( ℓ ) ij ( z ) for all ( i, j ) p airs and some ℓ ≥ 0 as the inducti ve as sumption. Let N ˙ n j be the se t of all v ariable nodes that are connec ted to check no de ˙ n j . Since x is a code word, it satisﬁes the parity checks , and so Q k ∈N ˙ n j = 1 . Then from the check node symmetry co ndition (Deﬁnition 3), ν ( ℓ +1) j i ( y ) correspond s to x i ν ( ℓ +1) j i ( z ) . Further , by the wire noise symmetry condition (Deﬁnition 5) a nd the same ar gument as for the base case , µ ( ℓ +1) j i ( y ) corresponds to x i µ ( ℓ +1) j i ( z ) . By in voking the variable node symmetry co ndition (Deﬁnition 4) again, it fol lows that ν ( ℓ +1) ij ( y ) co rresponds to x i ν ( ℓ +1) ij ( z ) for all ( i, j ) pairs. Thus by induc tion, all message s to and from v ariable n ode ˙ n i when y is rec eiv ed co rrespond to the produc t of x i and the c orresponding messa ge when z is receiv ed. Both decod ers proceed in correspond ence and commit exactly the sa me n umber of errors. 1) W orst-Case Noise: The same result with the s ame basic proof also ho lds when the wire noise operation Ξ is symmetric but w is n ot sy mmetric stochas tic, but is instead worst-case. The only essential mo diﬁcation is in (6 ) and the re lated part of the induction step. Since wire n oise is depend ent o n x i , it c an be written as x i w . Thus, µ (0) ij ( y ) = Ξ( ν (0) ij ( y ) , x i w (0) ij ) = Ξ( x i ν (0) ij ( z ) , x i w (0) ij ) ( a ) = x i Ξ( ν (0) ij ( z ) , w (0) ij ) = x i µ (0) ij ( z ) where step (a) follows because x i ∈ {± 1 } an d so it can be taken o utside of Ξ by the s ymmetry property of Ξ . Thus the two dec oders will p roceed in exa ct one-to-one corresp ondenc e, not just in probabilistic corresp ondenc e. A P P E N D I X B P R O O F O F T H E O R E M 2 Prior to giving the proof of Theorem 2, a re view of some deﬁnitions from probability theory [93] and the Hoeffding-Azuma inequality are provided. Consider a measu rable spac e (Ω , F ) consisting o f a sample space Ω and a σ -algebra F of s ubsets o f Ω that contains the whole s pace and is closed unde r c omplementation and coun table unions. A random vari able is a n F -measurable function on Ω . If there is a collection ( Z γ | γ ∈ C ) of random variables Z γ : Ω → R , the n Z = σ ( Z γ | γ ∈ C ) is deﬁned to be the s mallest σ -algebra Z on Ω such that each map ( Z γ | γ ∈ C ) is Z -mea surable. Deﬁnition 12 (F iltration): Let {F i } be a sequen ce of σ -algebras with res pect to the s ame sample space Ω . Thes e F i are sa id to form a ﬁltrati on if F 0 ⊆ F 1 ⊆ · · · are ordered by reﬁn ement in the sen se that eac h s ubset of Ω in F i is also in F j for i ≤ j . Also F 0 = {∅ , Ω } . Usually , {F i } is the natural ﬁ ltration F i = σ ( Z 0 , Z 1 , . . . , Z i ) of s ome seque nce of rando m variables ( Z 0 , Z 1 , . . . ) , and then the knowledge about ω known a t step i c onsists of the values Z 0 ( ω ) , Z 1 ( ω ) , . . . , Z i ( ω ) . For a probability triple (Ω , F , P ) , a version of the co nditional expectation of a random variable Z gi ven a σ - algebra F is a random variable den oted E [ Z |F ] . T wo versions of co nditional expectation agree almost s urely , but measure zero d epartures a re n ot considered su bseque ntly; one version is ﬁxed as c anonical. Con ditional expe ctation giv en a meas urable event E is denoted E [ Z | σ ( E )] and c onditional expectation giv en a rando m variable W is denoted E [ Z | σ ( W )] . Deﬁnition 13 (Martingale): Let F 0 ⊆ F 1 ⊆ · · · be a ﬁltration o n Ω and let Z 0 , Z 1 , . . . be a sequen ce of random variables on Ω suc h that Z i is F i -measurable. Then Z 0 , Z 1 , . . . is a martinga le with respect to the ﬁltration F 0 ⊆ F 1 ⊆ · · · if E [ Z i |F i − 1 ] = Z i − 1 . A generic way to co nstruct a martingale is Doob ’ s c onstruction. Deﬁnition 14 (Doob Martingale): Let F 0 ⊆ F 1 ⊆ · · · be a ﬁltr ation on Ω and let Z be a random v ariable on Ω . T hen the sequ ence of random variables Z 0 , Z 1 , . . . suc h that Z i = E [ Z |F i ] is a Doob martingale. 21 Lemma 1 (Hoeffding-Azuma Inequ ality [13], [94], [95]): Let Z 0 , Z 1 , . . . be a martingale with respect to the ﬁltration F 0 ⊆ F 1 ⊆ · · · such that for e ach i > 0 , the followi ng bou nded difference co ndition is s atisﬁed | Z i − Z i − 1 | ≤ α i , α i ∈ [0 , ∞ ) . Then for all n > 0 a nd any ξ > 0 , Pr [ | Z n − Z 0 | ≥ ξ ] ≤ 2 exp  − ξ 2 2 P n k =1 α 2 k  . Now to the proof of Th eorem 2 ; a s no ted before, it is an extens ion of [13, Theorem 2] o r [65, T heorem 4.94]. The basic ide a is to con struct a Doo b martingale abo ut the ob ject of intere st by revealing v arious ran domly determined aspec ts in a ﬁltration-reﬁning manner . The ﬁrst set of steps is used to rev eal wh ich code was chosen from the ensemble of c odes; the n d v edges in the bipartite grap h are ordered in some arbitrary manner an d expo sed one by one. Th en the n channel noise realizations are re vealed. At this point the exact graph and the exact ch annel no ise realizations encountered ha ve been revealed. Now the de coder noise rea lizations must be re vealed. The re are n variable no des, s o the computation nois e in each of them is re vealed o ne by one. There are n d v edges over wh ich variable-to-check communication n oise is manifested. Then there are nd v /d c check no des with computation no ise, and ﬁnally there are nd v check-to-variable communication n oises for one iteration of the algorithm. The decod er noise realizations a re revealed for e ach iteration. At the beginning of the revelation process, the average (over choice of code, c hannel noise r ealization, and decoder noise realization) is known; after t he m = ( d v +2 ℓd v +1+ ℓ + ℓd v /d c ) n rev elation steps, the exa ct system used is known. Recall that Z denotes the number of incorrect values held at the end of the ℓ th iteration for a p articular ( g , y , w , u ) ∈ Ω . S ince g is a graph in the s et of labeled bipa rtite factor graphs with variable nod e degree d v and check node degree d c , G n ( d v , d c ) ; y is a particular input to the dec oder , y ∈ Y n ; w is a p articular realization of the me ssage -passing noise, w ∈ M 2 ℓd v n ; an d u is a particular rea lization of the local compu tation noise, u ∈ U ( ℓ + ℓd v /d c ) n , the sample space is Ω = G n ( d v , d c ) × Y n × M 2 ℓd v n × U ( ℓ + ℓd v /d c ) n . In order to deﬁne rand om variables, ﬁrst deﬁn e the following exposu re procedure. Su ppose realizations of rando m quantities are expos ed se quentially . First expose the d v n edg es of the grap h o ne at a time. At s tep i ≤ d v n expose the particular c heck nod e socket which is conne cted to the i th variable n ode socket. Next, in the following n steps, expose the receiv ed v alues y i one at a time. Finally in the remaining (2 d v + 1 + d v /d c ) ℓn steps, expose the decoder noise values u i and w i that were encountered in all iterations up to iteration ℓ . Let ≡ i , 0 ≤ i ≤ m , be a sequenc e o f eq uiv a lence relations on the sample sp ace Ω ordered b y reﬁne ment. Re - ﬁnement means that ( g ′ , y ′ , w ′ , u ′ ) ≡ i ( g ′′ , y ′′ , w ′′ , u ′′ ) implies ( g ′ , y ′ , w ′ , u ′ ) ≡ i − 1 ( g ′′ , y ′′ , w ′′ , u ′′ ) . The equi valence relations deﬁne equ i valence classes suc h that ( g ′ , y ′ , w ′ , u ′ ) ≡ i ( g ′′ , y ′′ , w ′′ , u ′′ ) if a nd on ly if the realizations of random quantities re vealed in the ﬁrst i s teps for b oth p airs is the s ame. Now , de ﬁne a seque nce of random variables Z 0 , Z 1 , . . . , Z m . Let the random variable Z 0 be Z 0 = E [ Z ] , where the exp ectation is over the code choice, c hannel noise , an d decode r noise. The rema ining random variables Z i are constructed as conditional expectations given the measurable equiv alence events ( g ′ , y ′ , w ′ , u ′ ) ≡ i ( g , y , w , u ) : Z i ( g , y , w , u ) = E  Z ( g ′ , y ′ , w ′ , u ′ ) | σ (( g ′ , y ′ , w ′ , u ′ ) ≡ i ( g , y , w , u ))  . Note that Z m = Z and that by c onstruction Z 0 , Z 1 , . . . , Z m is a Doob martingale. The ﬁltr ation is u nderstood to be the natural ﬁltration of the ran dom vari ables Z 0 , Z 1 , . . . , Z m . T o use the Hoe f fding-Azuma inequality to g i ve bo unds o n Pr [ | Z − E [ Z ] | > nd v ǫ/ 2] = Pr [ | Z m − Z 0 | > nd v ǫ/ 2] , bounde d d if ference conditions | Z i +1 ( g , y , w , u ) − Z i ( g , y , w , u ) | ≤ α i , i = 0 , . . . , m − 1 need to b e proved for suitable cons tants α i that may depend on d v , d c , a nd ℓ . For the steps whe re bipa rtite grap h e dges a re exposed, it was shown in [13, p. 614] that | Z i +1 ( g , y , w , u ) − Z i ( g , y , w , u ) | ≤ 8( d v d c ) ℓ , 0 ≤ i < nd v . 22 It was further shown in [13, p. 615 ] tha t for the steps when the chan nel outputs are revealed tha t | Z i +1 ( g , y , w , u ) − Z i ( g , y , w , u ) | ≤ 2( d v d c ) ℓ , nd v ≤ i < n (1 + d v ) . (7) It remains to show that the inequality is also f ulﬁlled for steps when dec oder noise realizations a re rev ealed. The bounding procedure is n early identical to tha t which yie lds (7). When a n ode n oise realization u is revealed, c learly only s omething whose directed neighborhood include s the node at which the no ise u causes pe rturbations ca n be aff ected. Similarly , wh en an edge nois e realiza tion w is revealed, only something whose d irected neighborhood includes the e dge on which the no ise w caus es perturbations can be affected. In [13, p. 603], it is s hown that the s ize of the directed neigh borhood of de pth 2 ℓ of the no de ˙ n ( u ) assoc iated with noise u is bou nded as |N 2 ℓ ˙ n ( u ) | ≤ 2( d v d c ) ℓ and similarly the size of the directed neighborho od of length 2 ℓ o f the edg e ~ e ( w ) a ssociated with noise w is b ounded as |N 2 ℓ ~ e ( w ) | ≤ 2( d v d c ) ℓ . Sinc e the max imum d epth that can be af fected by a noise perturbation is 2 ℓ , a weak un iform bound for the remaining exposure steps is | Z i +1 ( g , y , w , u ) − Z i ( g , y , w , u ) | ≤ 2( d v d c ) ℓ , n (1 + d v ) d v ≤ i < m . Since bounded dif feren ce constan ts α i have been provided for all i , the theorem follo ws from application of the Hoeffding-Azuma inequality to the martingale. One may compute a particular value of β to use as follows. The boun ded difference s um is m X k =1 α 2 k = 64 nd v ( d v d c ) 2 ℓ + 4 n ( d v d c ) 2 ℓ + 4[2 ℓd v n + nℓ + n ℓd v /d c ]( d v d c ) 2 ℓ = n n 64 d v + 4 + 8 d v ℓ + ℓ + d v ℓ d c o d v 2 ℓ d c 2 ℓ Setting constan ts in the theorem and in the Hoeffding-Azuma inequality equal yields 1 β = 512 d v 2 ℓ − 1 d c 2 ℓ + 32 d v 2 ℓ − 2 d c 2 ℓ + 64 ℓd v 2 ℓ − 1 d c 2 ℓ + 8 ℓd v 2 ℓ − 1 d c 2 ℓ − 1 + 8 ℓd v 2 ℓ − 2 d c 2 ℓ ≤ (544 + 80 ℓ ) d v 2 ℓ − 1 d c 2 ℓ Thus 1 β can b e taken as (544 + 80 ℓ ) d v 2 ℓ − 1 d c 2 ℓ . A P P E N D I X C A N A NA L Y T I C A L E X P R E S S I O N An analytical expression for ε ∗ ( η = 1 / 10 , α = 5 × 10 − 3 ) is 1 2  1 − √ 1 + 4 c 7  , where c 7 is the s econd root of the p olynomial in ˇ ε c 1 + c 2 ˇ ε + c 3 ˇ ε 2 + c 4 ˇ ε 3 + c 5 ˇ ε 4 + c 6 ˇ ε 5 , and constan ts ( c 1 , . . . , c 6 ) are de ﬁned as follows. c 1 = 36 α 2 − 360 α 3 + 1860 α 4 − 6240 α 5 + 14752 α 6 − 25344 α 7 + 31680 α 8 − 28160 α 9 + 16896 α 10 − 6144 α 11 + 1024 α 12 = 34245 729141 29280658801 40000 000000 00000000000000 c 2 = 1 − 72 α + 1080 α 2 − 8160 α 3 + 38640 α 4 − 125952 α 5 + 29542 4 α 6 − 50688 0 α 7 + 63360 0 α 8 − 56320 0 α 9 + 33792 0 α 10 − 122880 α 11 + 20480 α 12 = 13320 075219 5329280658801 20000 000000 0000000000000 c 3 = 32 − 864 α + 10080 α 2 − 69120 α 3 + 31488 0 α 4 − 10122 24 α 5 + 23649 28 α 6 − 40550 40 α 7 + 50688 00 α 8 − 45056 00 α 9 + 2703360 α 10 − 98304 0 α 11 + 16384 0 α 12 = 69808 884183 5929280658801 25000 000000 000000000000 23 c 4 = 160 − 3840 α + 42240 α 2 − 28160 0 α 3 + 12672 00 α 4 − 40550 40 α 5 + 94617 60 α 6 − 16220160 α 7 + 20275 200 α 8 − 18022 400 α 9 + 10813 440 α 10 − 39321 60 α 11 + 65536 0 α 12 = 88638 487171 6129280658801 62500 000000 00000000000 c 5 = 320 − 7680 α + 84480 α 2 − 56320 0 α 3 + 25344 00 α 4 − 81100 80 α 5 + 18923 520 α 6 − 32440 320 α 7 + 40550 400 α 8 − 36044 800 α 9 + 21626 880 α 10 − 78643 20 α 11 + 1310720 α 12 = 88638 487171 6129280658801 31250 000000 00000000000 c 6 = 256 − 6144 α + 67584 α 2 − 45056 0 α 3 + 20275 20 α 4 − 64880 64 α 5 + 15138 816 α 6 − 25952 256 α 7 + 32440 320 α 8 − 28835 840 α 9 + 17301 504 α 10 − 62914 56 α 11 + 1048576 α 12 = 88638 487171 6129280658801 39062 500000 00000000000 As giv en in T able I, the numerical value of ε ∗ ( η = 1 / 10 , α = 5 × 10 − 3 ) is 0 . 026 609975 8 . Similarly co mplicated analytical expres sions are available for the other entries of T able I and the values used to create Figs. 4, 5, and 6. R E F E R E N C E S [1] C. E. Shannon, “ A mathematical theory of communication, ” Bell Syst. T ech. J. , vo l. 27, pp. 379–423 , 623–656 , July/Oct. 1948. [2] R. E. Blahut, Theory and Practice of Err or Contro l Codes . Reading, MA: Addison-W esley Publishing Company , 1983. [3] A. R. Calderbank, “The art of signaling: Fif ty years of coding theory , ” IEEE T rans. I nf. T heory , vol. 44, no. 6, pp. 2561–2595, Oct. 1998. [4] D. J. Costello, Jr . and G. D. Forne y , Jr ., “Channel coding: The r oad to channel capacity , ” Proc. IEEE , v ol. 95, no. 6, pp. 1150–1 177, Jun. 2007 . [5] H. D. Pﬁster, I. S ason, and R. Urbanke, “Capacity-achieving ens embles for t he binary erasure channel with bounded complexity , ” IE EE T rans. Inf. Theory , vol. 51, no. 7, pp. 2352–23 79, Jul. 2005. [6] S.-Y . Chung, G. D. Forney , Jr . , T . J. R ichardson, and R. Urbanke, “On the design of low-den sity parity-check codes within 0.0045 dB of the Shannon limit, ” I EEE Commun. Lett. , vol. 5, no. 2, pp. 58–60, Feb . 2001. [7] H.-A. Loeliger , F . Lustenberger , M. Helfenstein, and F . T ark ¨ oy, “Probability propagation and decoding in analog VL SI, ” IE EE T rans. Inf. Theory , v ol. 47, no. 2, pp. 837–843, Feb . 2001. [8] A. J. Blanksby and C. J. Howlan d, “ A 690-mW 1-Gb/s 1024-b, rate-1/2 low-density parity-check code decoder , ” IEEE J. Solid-State Cir cuits , vol. 37, no. 3, pp. 404–412, Mar . 2002 . [9] T . Zhang and K. K. Parhi, “ An FPGA i mplementation of (3 , 6) -regular lo w-density parity-check code decoder , ” EURASIP J. Appl. Signal Process. , vol. 2003, no. 6, pp. 530–542, 2003. [10] L. M. J. B azzi and S. K. Mitter , “Endcodin g complexity versus minimum distance, ” IEE E Tr ans. Inf. Theory , v ol. 51, no. 6, pp. 2103–2 112, Jun. 2005. [11] R. W . H amming, “E rror detecting and error correcting codes, ” Bell Syst. T ech. J. , vol. 26, no. 2, pp. 147–160 , Apr . 1950 . [12] W . H. P ierce, F ailure-T olerant Computer Design . New Y ork: Academic P ress, 1965. [13] T . J. Ri chardson and R. L. Urbanke , “The capacity of l o w-density parity-check codes under message-passing decoding, ” IEEE T rans. Inf. Theory , v ol. 47, no. 2, pp. 599–618, Feb . 2001. [14] P . Larsson and C. Svensson , “Noise in digital dynamic CMOS circuits, ” IEE E J . Solid-State Cir cuits , vo l. 29, no. 6, pp. 655–662 , Jun. 1994. [15] R. Ho, K. W . Mai, and M. A. Horowitz, “The future of wires, ” Pr oc. IEE E , vol. 89, no. 4, pp. 490– 504, Apr . 2001. [16] C. Zhao, X. Bai, and S. Dey , “Eva luating transient error effects in digital nanometer circuits, ” IEEE T rans. Rel. , vol. 56, no. 3, pp. 381–39 1, Sep. 2007. [17] T . Reji mon, K. Lingasubramanian, and S. Bhanja, “Probabilistic error modeling for nano-domain logic circuits, ” IEE E T ran s. VL SI Syst. , vol. 17, no. 1, pp. 55–65, Jan. 2009. [18] K. K. Likharev , “Si ngle-electron dev ices and their applications, ” Pr oc. I E EE , vol. 87, no. 4, pp. 606–6 32, Apr . 1999. [19] A. Bachtold, P . Hadley , T . Nakanishi, and C. Dekker , “Logic circuits with carbon nanotube t ransistors, ” Science , vol. 294, no. 5545, pp. 1317 –1320, Nov . 2001. [20] C. P . Collier , E. W . W ong, M. Belohradsk ´ y, F . M. Raymo, J. F . Stoddart, P . J. Kuekes, R. S . W i lliams, and J. R. Heath, “Electronically conﬁgurable molecular -based logic gates, ” Science , vol. 285, no. 5426 , pp. 391–39 4, Jul. 1999. [21] J. Han and P . Jonk er , “ A defect- and fault-tolerant architecture for nanoco mputers, ” Nanotec hnology , vol. 14, no. 2, pp. 224–230, Feb. 2003. [22] L. Anghel and M. Ni colaidis, “Defects tolerant logic gates for u nreliable future nanotechnologies, ” in Compu tational and A mbient Intelligence , ser . Lecture Notes in Compu ter Science, F . Sando v al, A. P rieto, J. Cabestan y , and M. Gra ˜ na, Eds. Berlin: Springer , 2007, vol. 4507, pp. 422–429. 24 [23] V . Bush, “The dif ferential analyzer . A ne w machine for solving differential equations, ” J . Fr anklin Inst. , vol. 212, no. 4, pp. 447–4 88, Oct. 1931 . [24] R. Sarpeshkar , “ Analog versus digital: Extrapolating from electron ics to neu robiology , ” Neural Comput. , vol. 10, no. 7, pp. 1601–1 638, Oct. 1998 . [25] B. Widro w and I. K oll ´ ar , Quantization Noise: Roundoff Error in Digital Computation, Signal Proce ssing, Contr ol, and Communications . Cambridge: Cambridge Univ ersity Press, 2008. [26] R. G. Gallager, Low-Density P arity-Check Codes . Cambridge , MA: MIT Press, 1963. [27] J. von Neumann, “Probabilistic logics and the synthesis of reliable org anisms from unreliable components, ” i n Automa ta Studies , C . E. Shannon and J. McCarthy , Eds. Princeton: Princeton Univ ersity Press, 1956, pp. 43–98. [28] S. W inograd and J. D. Co wan, R eliable Computation in the Pr esence of Noise . Cambridge, MA: MIT P ress, 1963. [29] C. N. Hadjicostis, Coding A ppr oaches to F ault T olerance in Combinational and Dynamic Systems . Boston: Kluwer Academic Publishers, 2002. [30] P . G ´ acs and A. G ´ al, “Lo wer bounds for the complexity of reliable Boolean circuits wi th no isy gates, ” IEEE T rans. Inf. Theory , v ol. 40, no. 2, pp. 579– 583, Mar . 1994. [31] P . E lias, “Computation in t he presence of noise, ” IBM J. Res. Dev elop. , v ol. 2, no. 4, pp. 346–353, Oct. 1958. [32] B. W . Johnson, Design and Analysis of F ault-T olerant Di gital Systems . Reading, MA: Addison-W esley Publishing Company , 1989. [33] D. K. Pradhan, F ault-T olerant Computer System Design . Upper Saddle Riv er , N J: P rentice Hall, 1996. [34] N. Pippenger , “Reliable computation by formulas in the presence of noise, ” IEE E T rans. Inf. Theory , vo l. 34, no. 2, pp. 194–197, Mar . 1988. [35] W . S. Ev ans and L . J. Schulman, “Signal propag ation and no isy circuits, ” IEEE Tr ans. Inf . Theory , v ol. 45, no. 7, pp. 2367–237 3, Nov . 1999. [36] L. Benini and G. De Micheli, Networks on Chips: T echnolo gy and T ools . San Francisco, CA: Morgan Kaufmann Publishers, 2006. [37] R. Hegde and N. R. Shanbhag, “T o ward achieving energy efﬁciency in presence of deep submicron noise, ” IEE E T rans. VLSI Syst. , vol. 8, no. 4, pp. 379–391, Aug. 2000. [38] C. Cri ck and A. Pfeffer , “Loopy b elief propagation as a basis for communication in sensor networks, ” in Pro c. 19th Annu. Conf. Uncertainty in Artiﬁcial Intelligen ce (UAI’03 ) , Aug. 2003, pp. 151–158. [39] O. W . Y eung and K. M. Chugg, “On the error tolerance of iterativ e decod er cir cuitry , ” in P r oc. 2008 Inf. Theory Appl. W orkshop , Jan. 2008. [40] L. W ei, “Rob ustness of LDPC codes and internal noisy systems, ” in Pr oc. 41st A nnu. Allerton Conf. Commun. Contro l Comput. , Oct. 2003, pp. 1665–1674 . [41] L. Ping and W . K. Leung, “Decoding lo w density parity check codes with ﬁnite quantization bits, ” IEEE Commun. Lett. , vol. 4, no. 2, pp. 62–6 4, Feb . 2000. [42] R. Singhal, G. S. Choi, and R. N. Mahapatra, “Quantized LDPC decoder design for binary symmetric channels, ” in P roc. IEEE Int. Symp. Cir cuits Syst. (ISCAS 2005) , vol. 6, May 2005, pp. 5782–5785 . [43] N. Miladinov ic and M. P . C. Fossorier , “Impro ved bit-ﬂipping decod ing of low-density parity-check codes, ” IEEE T ran s. Inf. Theory , vol. 51, no. 4, pp. 1594–1606, Apr . 2005. [44] J. Chen, A. Dholakia, E. El eftheriou, M. P . C. Fossorier , and X.-Y . Hu, “Reduce d-complexity decoding of LDPC codes, ” IEEE T rans. Commun. , vol. 53, no. 8, pp. 1288–12 99, Aug. 2005. [45] J. Zhao, F . Zarkeshv ari, and A. H. Banihashemi, “On implementation of min-sum algorithm and i ts modiﬁcations for decoding low- density parity-check (LDPC ) codes, ” IEEE Tr ans. Commun. , vo l. 53, no. 4, pp. 549–554, Apr . 2005. [46] T . Y u, R.-S. Lin, B. Super , and B. T ang, “Efﬁcient message represen tations for belief propagation, ” in Proc. 11th IEE E Int. C onf. Computer V isi on , Oct. 2007 , pp. 1–8. [47] R. Motw ani and P . Raghav an, Randomized Algorithms . Cambridge: Cambridge Uni versity Press, 1995. [48] S. T . Ribeiro, “Random -pulse machines, ” IEEE T rans. El ectr on. Comput. , vo l. E C-16, no. 3, pp. 261–276, Jun. 1967 . [49] S. S. T ehrani, W . J. Gross, and S. Mannor , “S tochastic decoding of LDP C codes, ” IEEE Commun. L ett. , vol. 10, no. 10, pp. 716– 718, Oct. 2006 . [50] A. Sahai and S. Mitter , “The necessity and suf ﬁ ciency of anytime capacity for stabilization of a li near system o ver a noisy communication link—Part I: Scalar systems, ” IE EE Tr ans. Inf. Theory , vol. 52, no. 8, pp. 3369–3395, Aug. 2006 . [51] A. T . Ihler , J. W . Fisher, III, and A. S. W illsky , “Loopy belief propagation: Con ver gence and effe cts of message errors, ” J. Mach. Learn. Res. , vol. 6, pp. 905–936, May 2005. [52] A. Ganti, A. Lapidoth, and ˙ I. E . T elatar, “Mismatched decodin g re visited: General alphabe ts, channels wit h memory , and t he wi de-band limit, ” IEEE T rans. Inf. Theory , vo l. 46, no. 7, pp. 2315–2328, Nov . 2000. [53] H. Saeedi and A. H. Banihashemi, “Performance of belief propag ation for decodin g LDPC codes i n t he presenc e of cha nnel estimation error , ” I E EE Tr ans. Commun. , vo l. 55, no. 1, pp. 83–89, Jan. 2007. [54] M. G. T aylor , “Reliable i nformation storage in memories designed from unreliable components, ” Bell Syst. T ech. J. , v ol. 47, no. 10, pp. 2299 –2337, Dec. 1968. [55] A. V . Kuznetsov , “Information storage in a memory assembled fr om unreliable components, ” Pr obl. Inf. T ran sm. , vol. 9, no. 3, pp. 100–11 4, July-Sept. 1973. [56] D. J. C . MacKay and R. M. Neal, “Near Shannon limit performance of lo w density parity check codes, ” IEE Electro n. L ett. , vol. 33, no. 6, pp. 457– 458, Mar . 1997. [57] M. S ipser and D. A. Spielman, “Expander codes, ” IEEE T rans. Inf. Theory , vol. 42, no. 6, pp. 1710–17 22, Nov . 1996. [58] D. Burshtein and G. Mill er , “Expander graph argu ments for message-passing algorithms, ” IEEE T ran s. Inf. Theory , vo l. 47, no. 2, pp. 782–79 0, Feb. 2001. [59] S. ten Brink, “Con vergence of iterative decoding , ” IEE Electr on. Lett. , vol. 35, no. 10, pp. 806–808, May 1999. [60] M. Ardakani and F . R. Kschischang, “ A more accurate one-dimensional analysis and design of irregular LDPC codes, ” IEEE T rans. Commun. , vol. 52, no. 12, pp. 2106–2114 , Dec. 2004. 25 [61] M. G. L uby , M. Mitzenmacher , M. A. Shokrollahi, and D. A. Spielman, “Improv ed low-density parity-check codes using irregular graphs, ” IEEE T rans. Inf. Theory , vo l. 47, no. 2, pp. 585–598, Feb . 2001. [62] T . R ichardson and R. Urbanke, “Fi xed points and stability of density e volution, ” Commun. Inf. Syst. , vol. 4, no. 1, pp. 103–116 , Sep. 2004. [63] S. K. Chilappagari and B. V asic, “Fault tolerant memories based on expan der graphs, ” in Pr oc. IEEE Inf. Theory W orkshop (ITW’07) , Sep. 2007, pp. 126–1 31. [64] L. R. V arshney , “P erformance of LDPC codes unde r noisy message-p assing decoding, ” i n Pr oc. IEE E Inf. T heory W orkshop (ITW’07) , Sep. 2007, pp. 178–1 83. [65] T . R ichardson and R. Ur bank e, Modern Coding T heory . Cambridge: Cambridge Univ ersity P ress, 2008. [66] T . R. Halford and K. M. Chugg, “The extraction and complexity limits of graphical models for linear codes, ” IEEE T rans. Inf . Theory , vol. 54, no. 9, pp. 3884–3906, Sep. 2008. [67] M. Lentmaier , D. V . T ruhache v , K. S. Zi gangiro v , and D. J. Costello, Jr ., “ An analysis of the block error probability performance of iterativ e decoding, ” IEEE T rans. I nf. Theory , vol. 51, no. 11, pp. 3834– 3855, Nov . 2005. [68] V . Rathi and R. Urbank e, “Density ev olution, thresholds and the stability condition for non-binary LDPC codes, ” IEE Pro c. Commun. , vol. 152, no. 6, pp. 1069–1074, Dec. 2005. [69] Y . W eiss, “Correctness of local probability propag ation in graphical models with loops, ” Neural Comput. , vol. 12, no. 1, pp. 1–41 , Jan. 2000. [70] P . M. W oodward, P r obability and Information Theory , W ith Applications to Radar . New Y ork: McGra w-Hill, 1953. [71] D. Slepian, “The threshold ef fect in modulation systems that expand bandwidth, ” IRE T rans. Inf. Theory , vol. IT -8, no. 5, pp. 122– 127, Sep. 1962. [72] J. M. W ozencraft and I. M. Jacobs, Principles of Communication Engineering . New Y ork: John Wiley & Sons, 1965. [73] V . Erokhin, “ ε -entropy of a discrete random variable, ” T heory Proba b . Appl. , vol. 3, no. 1, pp. 97–100, 1958. [74] L. Bazzi, T . J. Richardson, and R. L. Urbank e, “Exact thresholds and optimal codes for the binary-symmetric channel and Gallager’ s decoding algorithm A, ” IEEE Tr ans. Inf. Theory , vol. 50, no. 9, pp. 2010– 2021, Sep. 2004. [75] R. L. Dobrushin and S. I. Ortyukov , “Lower bound for the redundanc y of self-correcting arrangements of unreliable functional elements, ” Pr obl. Inf. T ransm. , vol. 13, no. 1, pp. 82–89, Jan.-Mar . 1977. [76] L. Y ang, “Recent advan ces on determining the number of real roots of parametric polynomials, ” J. Symbol. Comput. , vol. 28, no. 1-2, pp. 225– 242, Jul. 1999 . [77] T . J. Richardson, M. A. Shokrollahi, and R. L. Urbanke, “Design of capacity-approaching i rregular lo w-density parity-check codes, ” IEEE Tr ans. Inf. Theory , vol. 47, no. 2, pp. 619– 637, Feb . 2001 . [78] S.-Y . Chung, T . J. Richard son, and R. L. Urbanke, “ Analysis of sum-product deco ding of low-density parity-check codes using a Gaussian approximation, ” IEEE T rans. Inf. Theory , vol. 47, no. 2, pp. 657–670 , Feb. 2001. [79] S. K. Chilappag ari, B. V asic, and M. Marce llin, “Can the storage capacity of memories built from unreliable comp onents be determined?” in Proc. 2008 Inf. Theory Appl. W orkshop , Jan.-Feb . 2008 , pp. 41–43 . [80] A. G. Dimakis and K. Ramchandran, “Netwo rk coding for distributed storage in wireless networks, ” in N et w orked Sensing Information and Contro l , V . S aligrama, Ed. Ne w Y ork: Springer , 2008, pp. 115–136. [81] A. Montanari, “T ight bounds for LDPC and LDGM code s under MAP decoding, ” IEE E T ran s. Inf. Theory , vol. 51, n o. 9, pp. 3221–3246 , Sep. 2005. [82] J. G . Tryon, “Quadded logic, ” i n Redundancy T echniques for Computing Systems , R. H. Wilcox and W . C. Mann , Eds. W ashington: Spartan Books, 1962, pp. 205–228. [83] J. B. Gao, Y . Qi, and J. A. B. Fortes, “Bifurcations an d fundamental error bounds for fault-tolerant computation s, ” IEEE Tr ans. Nanotech nol. , vol. 4, no. 4, pp. 395–4 02, Jul. 2005. [84] S. H. Nawab, A. V . Oppenheim, A. P . Chandrakasan, J. M. W inograd, and J. T . Ludwig, “ Approximate signal processing, ” J. VL SI Signal Process. , vol. 15, no. 1-2, pp. 177– 200, Jan. 1997. [85] A. Sinha, A. W ang, and A. Chandrakasan , “Energ y scalable system design, ” IEEE T rans. VLSI Syst. , vo l. 10, no. 2, pp. 135–145, Apr . 2002. [86] M. Bhardwaj, R. Min, and A. P . Chandrakasan, “Quantifying and enhancing powe r awareness of VLSI systems, ” IEEE T rans. VLSI Syst. , vol. 9, no. 6, pp. 757–772 , Dec. 2001. [87] L. W ang and N. R. Shanbhag, “Energy-ef ﬁ ciency bounds for deep submicron VLSI systems in the presence of noise, ” IE EE Tr ans. VLSI Syst. , vol. 11, no. 2, pp. 254–269, Apr . 2003. [88] G. Masera, M. Mazza, G. Piccinini, F . V iglione, and M. Z amboni, “ Architectural strategies for low-po wer VLSI turbo decoders, ” IEEE T rans. VLSI Syst. , vol. 10, no. 3, pp. 279–285, Jun. 2002. [89] M. M. Mansour and N. R. S hanbhag , “ A 640 -Mb/s 2048 -bit programmable LDPC deco der chip, ” IEEE J . Solid-State Cir cuits , vo l. 41, no. 3, pp. 684– 698, Mar . 2006. [90] A. Sahai and P . Grov er , “The price of certainty: “waterslide curves” and the gap to capac ity , ” EECS Department, Univ ersity of California, Berkele y , T ech. Rep. UCB/E ECS-2008-1, Jan. 2008. [91] A. Maheshwari, W . Burleson, and R. T essier, “T rading off transient fault tolerance and po wer consumption in deep submicron (DSM) VLSI circuits, ” IEE E T rans. VLSI Syst. , vol. 12, no. 3, pp. 299–311, Mar . 2004. [92] B. J. Copeland, “Hypercomp utation, ” Minds Mach. , vol. 12, no. 4, pp. 461– 502, Nov . 2002 . [93] D. W illiams, Pro bability w i th Martingales . Cambridge: Cambridge Uni versity Press, 1991. [94] K. Azuma, “W eighted sums of certain dependent random v ariables, ” T ohoku Math. J. , vol. 19, no. 3, pp. 357–367 , 1967. [95] W . Hoef fding, “Probability inequalities f or sums of bounde d random v ariables, ” J. Am. Stat. Assoc. , vol. 58, no. 301, pp. 13–30, Mar . 1963.

Performance of LDPC Codes Under Faulty Iterative Decoding

Original Paper

Comments & Academic Discussion

Leave a Comment

Original Paper

Related Papers

Comments & Academic Discussion

Leave a Comment