Capacity of General Discrete Noiseless Channels

Capacity of General Discrete Noiseless Channels G. Bocherer Institute for Theor etical Information T echnolo gy R WTH Aachen Univ ersity 52056 Aachen, Germany Email: boech erer@ti.rwth- aachen.d e V .C. da Ro cha Jr . , C. P imentel Communica tions Research Grou p - CODEC Departmen t of Electron ics and Systems, P .O. Box 78 00 Federal University of Pern ambuco 50711 -970 Recife PE, Brazil E-mail: { vcr,cecilio } @ufpe.br Abstract —This paper concerns the capacity of the discrete noiseless chann el in troduced by S hannon. A sufficient condition is giv en for the capacity to be well-d efined. For a genera l discrete noiseless channel allowing non-integer va lued symbol weights, it is shown t h at the capacity—if well-defined —can b e determined from th e radiu s of con v ergence of its generating fu nction, from the smallest positive pole of its generating function, or from the rightmost real singularity of its complex generating function. A generalisation i s giv en for Pringsheim’s Th eorem and for the E xp on ential Growth Formula to generating functions of combinatorial structures with n on-integer valued symbol weights. I . I N T RO D U C T I O N When mo delling digital comm unication systems, ther e are situations where we do not e xplicitly model physical noise. W e r ather introd uce constrain ts on the allowed system config - urations that minimise the influ ence of u ndesired e ffects. An example is the ru nlength -limited constraint in magnetic recor d- ing [1]. W e consider in this paper the discrete noiseless channel (DNC) as in troduc ed by Shanno n [2]. A DNC is specified by a set of co nstraints imp osed on strings over a certain alph abet, and only those strings that fu lfil the constraints are allowed for transmission or stor age. A DNC allows the specification of two types of co nstraints. The first co nstraint is on symbo l constellations (for example, only b inary strings with not more than tw o consecuti ve 0 s are allowed), an d the second constraint is on symbol weig hts (for example, the symbo l a has to be of duration 5 . 53 secon ds). Dep ending on the sy stem we want to model, the symbol weig hts repr esent the cr itical r esource over which we want to optimise. This can for example b e d uration , length or en ergy . W e then ask the follo wing q uestion. What is the maxim um r ate of data per string weight that can be transmitted over a DNC? This question was first answer ed by Shan non in [2]. In [3], the authors extend Sh annon ’ s results to DNCs with non -integer valued symb ol weig hts. In b oth [2] and [3], the autho rs use the fo llowing ap proach to derive the capacity o f a DNC. Th ey restrict the class of consider ed DNCs to tho se that allow the transmission of a set of strings fo rming a regular lan guage . The regularity allo ws the n to represent the DNC by a finite state machin e and results from matrix th eory are applied to derive the c apacity o f the DNC. Our approach is different in the following sense. W e c on- sider gen eral DNCs with the o nly restriction that the capacity as de fined in [2] and generalised in [3] has to be well-d efined, which will turn o ut to b e a r estriction on the set of possible string weights. This allows us then to represent th e com binatorial comp lex- ity of a DNC by a gen erating fu nction with a well-defined radius o f co nv ergence a nd we u se ana lytical m ethods to d erive the capacity . In this sense, our work is a gen eralisation of [3]. Perhaps more important, in many cases tha t could be treated by the techn iques p ropo sed in [3], it is much simp ler to construct the generating functio n of the co nsidered DNC and to u se o ur results to der iv e th e c apacity . W e give two simple examples that m ay ser ve as illu strations. I n th is sense, our work can also be considered as an interesting alternative to [3]. I I . D E FI N I T I O N S W e formally defin e a DNC and its gen erating fu nction as follows. Definition 1. A DNC A = ( A, w ) consists o f a c ountable set A of strings accepted by the ch annel and an associated weight function w : A 7→ R ⊕ ( R ⊕ denotes the nonnegativ e real numb ers) with the following prop erty . If s 1 , s 2 ∈ A and s 1 s 2 ∈ A ( s 1 s 2 denotes the co ncatenatio n of s 1 and s 2 ), then w ( s 1 s 2 ) = w ( s 1 ) + w ( s 2 ) . (1) By con ventio n, th e empty string ε is always of weight zer o, i.e., w ( ε ) = 0 . Definition 2. Let A = ( A, w ) represent a DNC. W e define the gen e rating function of A by G A ( y ) = X s ∈ A y w ( s ) y ∈ R . (2) W e or der and index the set of possible string weig hts w ( A ) such that w ( A ) = { w k } ∞ k =1 with w 1 < w 2 < · · · . W e can then write G A ( y ) = ∞ X k =1 N [ w k ] y w k (3) where for each k ∈ N , the coefficient N [ w k ] is equal to the number of distinct strings o f weight w k . Since the coefficients N [ w k ] result from an enumeration, they are all nonnegativ e. Note that for any DNC A , we h av e G A (0) = N [0] = 1 since ev ery DNC allo ws the transmission of the empty strin g and since th ere is only on e em pty string . The m aximum r ate of data per string weight that can be transmitted over a DNC is g iv en by its cap acity . W e define capacity in accordan ce with [2] and [3] as fo llows. Definition 3 . The cap acity C of a DNC A = ( A, w ) is g iv en by C = lim sup k →∞ ln N [ w k ] w k (4) in nats p er symbol weigh t. This is equ iv alen t to the following. For all ǫ with C > ǫ > 0 , the following two pro perties hold. 1) The numb er N [ w k ] is greater than or equ al to e w k ( C − ǫ ) infinitely often ( i . o . ) with respect to k . 2) The numb er N [ w k ] is less than or equal to e w k ( C + ǫ ) almost everywhere ( a . e . ) with respect to k . W e assume in the f ollowing that th e nu mber sequen ce { w k } ∞ k =1 is not too dense in the sense that for any integer n ≥ 0 max w k 1 . According to Definition 3 , the ca pacity o f the DNC is then equal to zero bec ause of ln N [ w k ] = 0 for all k ∈ N . Howe ver, the channel ac cepts R n distinct strings of weigh t smaller than n . Th e a verage amount of data per string weig ht tha t we can tr ansmit over the channel is thus lower -b ound ed b y ln R n /n = ln R , which is accord ing to the assumption greater than zero. ⊳ Whenever we say that the capac ity o f a DNC is well-d efined, we mean that (5) is fu lfilled. I I I . C A PAC I T Y B Y R A D I U S O F C O N V E R G E N C E One way to c alculate the ca pacity of a DNC is b y determ in- ing the radiu s of convergence of its gen erating f unction . Lemma 1. Let A be a DNC with the generating function G A ( y ) . If the capa c ity C of A is well-defin ed, then it is given by C = − ln R where R denotes the radius of con ver gence of G A ( y ) . In the pro of of this lemma, we will ne ed the following result from [ 3]. Lemma 2. If (5) is fulfi lled and if ρ is a positive real number , then P ∞ k =1 ρ w k conver ges iff ρ < 1 . Pr oof of Lemma 1: W e define M [ k ] = N 1 /w k [ w k ] and write G A ( y ) as G A ( y ) = ∞ X k =1  M [ k ] y  w k . (7) W e define the two sets D ( y ) an d E ( y ) as D ( y ) =  k ∈ N   M [ k ] y < 1  (8) E ( y ) = N \ D ( y ) =  k ∈ N   M [ k ] y ≥ 1  (9) and write G A ( y ) = X k ∈ D ( y )  M [ k ] y  w k + X l ∈ E ( y )  M [ l ] y  w l . (10) It follows from Lemma 2 that G A ( y ) con verges iff the set E ( y ) is finite. The number R is th e radius of co n vergence of G A ( y ) , theref ore, fo r any δ > 1 , the set E ( R/δ ) is finite. Since D ( y ) = N \ E ( y ) , the finiteness of E ( R /δ ) is equ iv alent to k ∈ D ( R/δ ) a . e . (11) W e define ǫ = ln δ . E quation (11) is th en equiv alent to N [ w k ] < e w k ( − l n R + ǫ ) a . e . (12) which implies N [ w k ] ≤ e w k ( − l n R + ǫ ) a . e . (13) Again sinc e R is the rad ius of conver gence of G A ( y ) , for any δ > 1 , the set E ( R δ ) is infinite. For ǫ = ln δ , th is is eq uiv ale nt to N [ w k ] ≥ e w k ( − ln R − ǫ ) i . o . (14) It fo llows from (13) and (14) an d Definition 3 that − ln R is equal to the capacity o f A . W e theref ore have C = − ln R . In th e fo llowing example, we show how Lemma 1 a pplies in practice. W e den ote by A ∪ B the union of the two sets A an d B , we d enote by AB th e set o f all co ncatenation s ab with a ∈ A an d b ∈ B , an d we denote by S ⋆ the Kleene star operation o n S , which is define d a s S ⋆ = ǫ ∪ S ∪ S S ∪ · · · . Example 2 . W e co nsider a DNC A = ( A, w ) with the alp habet { 0 , 1 } and symb ol weights w (0) = 1 and w (1) = π . T he DNC A does n ot allow strin gs that co ntain two or mo re consecu ti ve 1 s. W e represent A by a regular expre ssion and write A = { ε ∪ 1 } { 0 ∪ 01 } ⋆ . For the generating f unction of A we get G A ( y ) = (1 + y π ) ∞ X n =0 ( y + y 1+ π ) n . (15) The r adius of convergence is gi ven b y the sma llest positive solution R of the eq uation y + y 1+ π = 1 . W e find R = 0 . 729 37 . According to Le mma 1, the c apacity of A is thus given by C = − ln R = 0 . 31 558 . ⊳ I V . C A PAC I T Y B Y R I G H T M O S T R E A L S I N G U L A R I T Y There are cases where we de riv e th e closed-fo rm rep resen- tation of the gener ating func tion of a DNC witho ut explicitly using its series rep resentation. The techniq ues introduced in [4] and [5] may serve as tw o examp les. In this section, we show how th e capacity of a DNC A can be determined from the clo sed-form rep resentation of its generatin g function . W e do this in tw o steps. W e first identif y th e region o f conv ergence (r .o.c.) of the comp lex generating fu nction F A ( e − s ) with its righ tmost real singularity . The complex generating func- tion F A ( e − s ) r esults fro m ev aluating the generating functio n G A ( y ) in y = e − s , s ∈ C . Secon d, we show that the r ightmost real singular ity of F A ( e − s ) determines the capacity of A . Theorem 1. If the r . o .c. of F A ( e − s ) is determined by ℜ { s } > Q , then F A ( e − s ) has a singu larity in s = Q . Pr oof: Suppo se in contrary that F A ( e − s ) is analytic in s = Q implying th at it is analytic in a disc of r adius r cen tred at Q . W e choo se a n umber h such that 0 < h < r / 3 , and we consider the T ay lor expansion of F A ( e − s ) aro und s 0 = Q + h as follows. F A ( e − s ) = ∞ X n =0  F A ( e − s 0 )  ( n ) n ! ( s − s 0 ) n (16) = ∞ X n =0 ∞ P k =1 N [ w k ]( − w k ) n e − w k s 0 n ! ( s − s 0 ) n . (17) For s = Q − h , this is a ccordin g to our supposition a conver g- ing d ouble sum with positive terms an d we can reorganise it in any way we want. W e thus have conver gence in F A ( e − Q + h ) = ∞ X n =0 ∞ P k =1 N [ w k ]( − w k ) n e − w k s 0 n ! ( − 2 h ) n (18) = ∞ X k =1 N [ w k ] e − w k s 0 ∞ X n =0 w n k (2 h ) n n ! (19) = ∞ X k =1 N [ w k ] e − w k s 0 e w k 2 h (20) = ∞ X k =1 N [ w k ] e − w k ( Q − h ) . (21) But co n vergence in the last line con tradicts that the r .o.c. of F A ( e − s ) is strictly given by ℜ { s } > Q . W e now r elate the rightmost real singularity of F A ( e − s ) to the capacity of A . Theorem 2. Assume that F A ( e − s ) has its righ tmost r eal singularity in s = Q . The ca pacity of A is then given by C = Q . Pr oof: Sinc e F A ( e − s ) has its rightmo st real singular ity in s = Q , it follows fr om The orem 1 that the r .o.c. of F A ( e − s ) is determined by ℜ { s } > Q . For F A ( e − s ) , we have F A ( e − s ) = ∞ X k =1 N [ w k ] e − w k s (22) ≤ ∞ X k =1 | N [ w k ] e − w k s | (23) = ∞ X k =1 N [ w k ] | e − w k s | (24) where equality in (24) holds because the c oefficients N [ w k ] are all non negativ e and where we have eq uality in ( 23) if s is r eal. It f ollows that if the r .o.c. of F A ( e − s ) is given b y ℜ { s } > Q , then the radiu s of conver gence o f G A ( y ) is given by R = e − Q . Using Lemma 1, we have for the capacity C = − ln R = Q . Note 1 . W ith Theorem 1 an d Theor em 2 , we g eneralised Pringsheim’ s The orem and the Exponential Growth Formula, see [6], to generatin g fun ctions of DNCs with non-in teger valued symbol weights. V . C A PAC I T Y B Y S M A L L E S T P O S I T I V E P O L E W e formu late the most impo rtant application of Theorem 2 in th e following coro llary: Corollary 1 . Let A repr esent a DNC with a well-defin ed capacity C . Supp o se th a t th e generating function G A ( y ) c a n be written as G A ( y ) = n 1 y τ 2 + n 2 y τ 2 + · · · + n p y τ p d 1 y ν 1 + d 2 y ν 2 + · · · + d q y ν q (25) for some finite positive inte gers p a nd q . The capac ity C is then given by − ln P whe re P is the smallest positive po le of G A ( y ) . Note 2 . The c orollary was already stated in [4, Theor em 1 ]. Howe ver, th e proof g iv en by the au thors does not ap ply for the g eneral case, which we conside r in this p aper . Pr oof of Cor o llary 1: If G A ( y ) is of the form (25), the com plex generatin g func tion F A ( e − s ) as defined in the previous sectio n is m eromo rphic, wh ich implies tha t all its singularities are poles. The substitution y = e − s , for s rea l, is a one-to- one mappin g from the r eal axis to th e p ositiv e real axis. Therefo re, if Q is the rightmo st r eal singu larity o f F A ( e − s ) , then e − Q is the smallest positiv e pole of G A ( y ) . Applying Theorem 2, we get fo r th e ca pacity C = Q = − ln P . Example 3 . W e consider the DNC A = ( A, w ) where A is the set of all bin ary strings that do not co ntain the substring 111 and where th e sy mbol weights are gi ven b y w (0) = w (1) . W e use a result from [5] in the for m of [6, Proposition 1.4]. It states that the set of binary strings that do n ot co ntain a certain pattern p has the genera ting fu nction f ( y ) = c ( y ) y k + (1 − 2 y ) c ( y ) (26) where k is the leng th (in bits) of p and where c ( y ) is the auto- correlation polyno mial of p . It is defined as c ( y ) = P k − 1 i =0 c i y i with c i giv en by c i = δ [ p 1+ i p 2+ i · · · p k , p 1 p 2 · · · p k − i ] (27) where p i denotes the i th bit (fro m the left) of p and where δ [ a, b ] = 1 if a = b and δ [ a , b ] = 0 if a 6 = b . F or p = 111 , we have c ( y ) = 1 + y + y 2 and k = 3 . This yields for the generating f unction of A G A ( y ) = 1 + y + y 2 y 3 + (1 − 2 y )(1 + y + y 2 ) . (28) Note th at the ap plication o f the techniqu e from [4] would have led to the same for mula. For the smallest positive pole P o f G A ( y ) we find P = 0 . 5436 9 . Accor ding to Coro llary 1, the capacity of A is thus given by C = − ln P = 0 . 6093 8 . ⊳ V I . C O N C L U S I O N S For a general DNC, we ide ntified the capac ity with the characteristics of its genera ting function, namely the radius of con vergence o f its gene rating fun ction, the righ tmost real singularity of its com plex g enerating fu nction, and the small- est positive p ole of its gen erating function . W e g eneralised Pringsheim’ s Theorem and the Expone ntial Growth Formula as given in [6] to g enerating func tions th at allow non-in teger valued symbol weights. Representing a DNC by its g enerating functio n and not by a finite state machin e has an additional advantage. Althou gh the finite state machine allows th e deri vation of th e correct capacity of the DNC, it says no thing ab out the exact number of v alid strings of weight w . T he generating f unction of a DNC provides th is inf ormation . The coefficients N [ w k ] ar e equ al to the n umber of d istinct string s of len gth w k that are a ccepted by the D NC. The coe fficients can either b e calcu lated by an algebraic expansion of the gen erating fun ction or they can be approx imated by mean s of ana lytic asymptotics as d iscussed for integer valued symbol weights in [6]. In [7], the a nalytic approa ch is extended to generating functions of DNCs with non-in teger valued symbo l weights. For a regular DNC fulfilling some further restrictions, the authors in [3] define a Markov process th at gen erates valid strings at an en tropy rate equal to the capacity o f the chan nel. Based on g enerating fun ctions as introduced in this paper , it is shown in [7] that for a g eneral DNC, any entr opy rate C ′ smaller than the capacity C is ach iev ab le in the sense that there exists a ran dom process that generates strings th at are transmitted over the channel at an entro py rate C ′ . A C K N O W L E D G E M E N T V . C. da Rocha Jr . and C. Pime ntel acknowledge partial support of this r esearch by th e Brazilian Natio nal Council for Scien tific and T echnolog ical Development (CNPq) under Grants No. 3 0522 6/200 3-7 and 30 1253 /2004 -8, respectively . R E F E R E N C E S [1] B. H. Marcus, R. M. Roth, and P . H. Siegel, “ An introducti on to coding for constrained s ystems, ” O ct. 2001. [2] C. E. Shannon, “ A mathematical theory of communication , ” Bell Syst. T ech. J. , vol. 27, pp. 379–423 and 623–656, Jul. and Oct. 1948. [3] A. Khandekar , R. McEliece, and E. Rodemich, “The discrete noiseless channe l re visited, ” in Coding , Communications, and Br oadcasting . Re- search Studies Press L td., 2000, pp. 115–1 37. [4] C. Pimentel and B. Uchoa-Filho, “ A combinatorial approac h to finding the capacit y of the discret e noisele ss channe l, ” IEE E T rans. Inf . Theory , vol. 49, no. 8, pp. 2024–2028, 2003. [5] L. J. Guibas and A. M. Odlyzk o, “String overl aps, pattern matchin g, and nontransi ti ve games, ” Journal of Combinatorial Theory , vol . A 30, pp. 183–208, 1981. [6] P . Flajolet and R. Sedge wick, Analyti c Combinatoric s . Cambridge Uni versity Press, 2008. [7] G. Bochere r , “ Analytic asymptotics of discrete noiseless channe ls, ” Master’ s thesis, ETH Zurich, 2007. [Online ]. A vai lable : http:/ /www .georg- boechere r .de/repository/analyticAsymptotics.pdf