Weighted Superimposed Codes and Constrained Integer Compressed Sensing

1 W eighted Superimposed Codes and Cons trained Inte ger Compress ed Sensing W ei Dai and Olgica Milenkovic Dept. of Electrical and Computer Engine ering University of Illinois, Urbana- Champaign Abstract W e introd uce a new family o f cod es, term ed weig hted su perimpo sed codes (WSCs). T his family generalizes the class of Euclidean superim posed codes (ESCs), used in multiuser iden tification systems. WSCs allo w for discriminating all bound ed, integer-v alued linear com binations of rea l-valued codew or ds that satisfy prescrib ed nor m and non -negativity constraints. By d esign, WSCs are in herently noise tolera nt. Theref ore, these codes can be seen as special instances of robust com pressed sensing schemes. The main results of the paper are lo wer and uppe r b ound s on the largest achiev able code rates of several classes of WSCs. These bounds su ggest that with the c odeword and weighting vector con straints at h and, one can improve th e code rates achiev able b y standard compressive sensing . I . I N T RO D U C T I O N Superimpose d c odes (SCs) and designs were introduced by Kautz and Singleton [1], for the purpose of studying databas e retrie val and group testing problems. In their original formulation, supe rimposed designs were defin ed as arrays of bina ry c odewords with the property that bitwise OR functions of all suf fic iently small collections of codewords are distinguisha ble. Su perimposed des igns c an there fore b e viewed as binary “p arity-check” matrices for which syndromes represent bitwise OR, rather than XOR, fun ctions of selected sets of columns. The no tion of binary superimposed code s was further generalize d by prescribing a distan ce cons traint on the OR ev a luations of subs ets of columns, a nd by extending the fields in which the codeword sy mbols lie [2]. In the latter cas e, Ericso n and Gy örfi introduced Eu clidean superimpose d co des (ESCs), for wh ich the symbol field is R , for which the OR function is replace d by real addition, and for which all sums of less than K codewords are required to have pairwise Eu clidean distance at least d . Th e best known uppe r boun d on the size o f Euclidean superimpose d code s was d eri ved by Füred i a nd Ruszinko [3], who used a comb ination of sphere pa cking a r gu ments and probabilistic concentration formulas to prove their result. On the other ha nd, compressed sensing (CS) is a new sampling method usually applied to K - sparse signals , i.e. signals emb edded in an N -dimensional space that can be rep resented by only K ≪ N sign ificant coe f fic ients [4]– [6]. Alternatively , wh en the signal is projected onto a prop erly cho sen basis of the transform s pace, its ac curate representation relies only on a small n umber of coe f fic ients. Enc oding of a K -sp arse discrete-time signal x of dimension N is a ccomplishe d by computing a measurement vector y that consists of m ≪ N linear projections, i.e. y = Φ x . He re, Φ represen ts an m × N matrix, u sually over the fie ld of rea l numbe rs. Conse quently , the measu red vector represents a linear co mbination of c olumns of the matrix Φ , with we ights prescribed by the no nzero entries of the vector x . Although the reconstruction of the sign al x ∈ R N from the (pos sibly n oisy) projections is an ill-posed problem, the prior knowledge of signal sparsity allo ws for ac curate re covery of x . ∗ This work is supported by the NSF Grant CCF 0644427 , t he NSF Career A ward, and the DARP A Y oung Fac ulty A ward of the second author . Parts of the results were presented at the CCIS’2008 ant ITW’2008 conferences. 2 The conne ction betwe en error-correcting coding theory and compres sed sen sing was in vestigated by Can dès and T ao in [7], an d remarked upon in [8]. In the former work, the authors s tudied random c odes over the real numbers, the noisy observations of which can be dec oded us ing linear programming techniques. As with the case of compress ed se nsing, the performance g uarantees of this c oding sche me are prob abilistic, and the K - sp arse signal is assume d to lie in R N . W e propo se to study a new clas s of codes, termed weighted su perimpos ed codes (WSCs), which provide a link between SCs and CS matrices. As with the case of the former two entities, WSCs are defined over the field of real numbers. But unlike ESCs , for which the spa rse signa l x c onsists of z eros and o nes only , and u nlike C S, for which x is assumed to belong to R N , in WSCs the vector x is drawn from B N , where B deno tes a bou nded, symmetric set of inte gers . The moti vation for studying WSCs comes from the fact that in many applications, the alphabe t o f the sparse signal can be modeled as a finite set of integers. Codewords from the famil y of WSCs ca n be d esigned to obey pres cribed norm a nd n on-negativit y cons traints. The res triction of the weighting coefficients to a bounded s et o f integers ensures reco nstruction robustness in the presenc e of noise - i.e., all weighted sums o f at mos t K c odewords can be cho sen at “minimum distance” d from each other . This minimum distanc e property provides deterministic performance gua rantees, which CS tech niques usually lack. A nother be nefit of the input a lphabet restriction is the po tential to red uce the de coding complexity compared to that of CS reconstruction techniques. T his res earch problem was address ed by the authors in [9]–[11], but is beyon d the sco pe o f this paper . The central problem of this paper is to cha racterize t he rate region for which a WSC with certain parameters exists. The main results o f this work include gen eralized sphere pa cking upper bo unds and random c oding lo we r b ounds on the rates of several WSC families. The up per and lo wer bou nds dif fer only by a constant, and therefore imply that the su perposition constraints are en sured whenever m = O ( K log N/ log K ) . In the lang uage of CS the ory , this result sugges ts that the nu mber of required signal measurements is less than the standard O ( K log ( N /K ) ) , required for discriminating real-valued linear combinations of co dewor ds . This reduc tion in the re quired nu mber of measuremen ts (codeleng th) c an be seen as a result of restricting the input alph abet of the sparse s ignal. The p aper is organized as follows. Section III introduce s the relev ant terminology an d defi nitions. Se ction IV contains the main results of the paper – upper a nd lo wer b ounds on the size o f WSCs. The proofs of the rate bounds are presented in Sections V and V III. Concluding remarks are gi ven in Section IX. I I . M OT I V A T I N G A P P L I C A T I O N S W e describe next t wo applications - one arising in wireless c ommunication, the other in bioengineering - moti vating the study of WSCs. The adder channe l and sign atur e c odes : One common ap plication of E SCs is for signa ling over multi-acce ss channe ls. For a giv en set o f k ≤ K active us ers in the chann el, the input to the receiver y eq uals the sum of the signals (signatures) x i j , j = 1 , . . . , k , of the k active users , i.e . y = P k j =1 x i j . The signa tures are only used for identification purpo ses, and in order to minimize energy consumption, all users a re assigne d unit energy [2], [3]. Now , conside r the cas e that in addition to identifying their pres ence, active us ers also have to conv ey some limited information to the rec eiv er by adapting their transmiss ion po we r . The received signal can in this c ase be represented by a weighted sum o f the signatures of ac ti ve u sers, i.e., y = P k j =1 √ p i j x i j . The code book us ed in this sch eme re presents a s pecial form of WSC, termed W eighted Eu clidean Superimpose d Co des (WESCs); these codes are formally defined in Section III. Compressive sensing micr oarrays : A microarray is a bioen gineering device used for measuring the level of certain molecules, such as RN A (ribonucleic acid) sequen ces, represe nting the j oint expres sion profi le of t ho usand s of genes. 3 A microarray c onsist of thou sands of microsc opic spo ts of DN A sequen ces, called probes. The c omplementary DNA (cDN A) s equenc es o f RN A molecules being me asured are labe led with fluo rescent tags, and such units are termed tar ge ts. If a target seque nce has a significant homolog y with a probe seq uence on the microarray , the tar get cDN A and probe DNA molecules will bind or “hybridize” so as to form a stable structure. As a res ult, upon exposure to laser light of the appropriate wa velength, the microarray spots with lar ge hyb ridization acti vity will be illuminated. The specific illumination pattern an d intensities of microarray spots ca n be used to infer the concentration of RN A molecules. In traditional microarray d esign, ea ch spo t of probes is a unique iden tifier of only one target molecule. In our recen t work [12], [13], we proposed the conce pt of compressive se nsing micr oar rays (CSM), for which each probe has the poten tial to h ybridize with several diff e rent tar ge ts. It use s the observation that, although the numbe r of potential target RNA type s is large, not all o f them are expected to be present in a significant conce ntration at all observed times . Mathematically , a microarray is repres ented by a measureme nt matrix, with an entry in the i th row and the j th column correspon ding to the hybridization probability between the i th probe and the j th tar ge t. In this case, a ll the entries in the measu rement matrix are no nnegativ e real numbe rs, an d a ll the columns of the mea surement matrix are expected to h ave l 1 -norms e qual to one . In microarray experiments, the inpu t vec tor x has e ntries tha t correspond to integer multiples of the smallest detectable concen tration of tar get cDN A molecu les. Since the numbe r of different tar ge t cDNA typ es in a typ ical test s ample is small co mpared to the n umber of a ll potential types , one can as sume that the vector x is sparse. Fu rthermore, the numbe r of RNA molecu les in a c ell at a ny point in time is upper bounde d due to ene r gy constraints, and du e to intracellular spa ce limitations. Hen ce, the integer-v alued entries of x are assumed to have bounde d magnitudes and to be relativ ely small compa red to the nu mber of dif ferent RNA types. W ith the ab ove conside rations, the me asuremen t matrix of a CSM can b e described by n onnegativ e l 1 -WSCs, formally defined in Section III. I I I . D E F I N I T I O N S A N D T E R M I N O L O G Y Throughou t the pap er , we use the following n otation and definitions. A c ode C is a fin ite set of N codewords (vectors) v i ∈ R m × 1 , i = 1 , 2 , · · · , N . The co de C is s pecified by its codeword matrix (co debook ) C ∈ R m × N , obtained by arranging the codewords in columns of the matrix. For tw o g i ven positi ve integers, t an d K , let B t = [ − t, t ] = {− t, − t + 1 , · · · , t − 1 , t } ⊂ Z be a symmetric, bounded set of integers, an d let B K =  b ∈ B N t : k b k 0 ≤ K  denote the l 0 ball of rad ius K , with k b k 0 representing the numbe r of nonze ro components in the vec tor b (i.e., the suppo rt size of the vector). W e formally define WESCs as follows. Definition 1: A code C is said to be a WES C with parameters ( N , m, K , d, η , B t ) for some d ∈ (0 , η ) , if 1) C ∈ R m × N , 2) k v i k 2 = η , for all i = 1 , · · · , N , and, 3) if the followi ng minimum dis tance property h olds: d E ( C , K , B t ) := min b 1 6 = b 2 k Cb 1 − Cb 2 k 2 ≥ d for all b 1 , b 2 ∈ B K . 4 Henceforth, we focu s our attention on WESCs with η = 1 , a nd denote the s et of pa rameters of interest b y ( N , m, K , d, B t ) . The definition above can be extended to h old for other normed spaces. Definition 2: A code C is said to be an l p -WSC with parameters ( N , m, K, d, B t ) if 1) C ∈ R m × N , 2) k v i k l p = 1 , for all i = 1 , · · · , N , and , 3) if the followi ng minimum dis tance property h olds: d p ( C , K, B t ) := min b 1 6 = b 2 k Cb 1 − C b 2 k l p ≥ d for all b 1 , b 2 ∈ B K . Note that specia lizing p = 2 reproduces the defin ition of a WESC. Moti vated b y the practical applications des cribed in the previous sec tion, we also define the class o f nonne gative l p -WSC. Definition 3: A cod e C is said to b e a nonnegati ve l p -WSC with parameters ( N , m , K, d, B t ) if it is an l p -WSC such that all entries of C are nonnegativ e. Gi ven the parameters m, K, d an d B t , let N ( m, K, d, B t ) denote the maximum size o f a WSC, N ( m, K, d, B t ) := max { N : C ( N , m, K, d, B t ) 6 = φ } . The asymptotic code exponent is defined as R ( K, d, B t ) := lim sup m →∞ log N ( m, K , d, B t ) m . W e are interested in qua ntifying the asymptotic code expon ent of WSCs, and in particular , WESCs and nonneg- ati ve WSCs with p = 1 . Results pertaining to these classes o f codes are summarized in the n ext se ction. I V . O N T H E C A R D I N A L I T Y O F W S C F A M I L I E S The cen tral problem of this paper is to determine the existence of a sup erimposed code with certain parameters. In [2], [3], it was shown that for ES Cs, for wh ich the co dew ord a lphabet B t is rep laced by the a symmetric se t { 0 , 1 } , one ha s log K 4 K (1 + o d (1)) ≤ R ( K , d, { 1 } ) ≤ log K 2 K (1 + o d (1)) , where o d (1) con verges to zero as K → ∞ . The main result of the paper is the uppe r and lo wer bounds on the asymptotic code expone nts o f sev eral WSC famili e s. For WESC s, introduc ing weighting c oefficients larger tha n one does not c hange the asymptotic order o f the code exponent. Theorem 1: Let t be a fixed pa rameter . For suf ficie ntly lar ge K , the asymptotic co de expone nt o f WESCs can be bounded as log K 4 K (1 + o (1)) ≤ R ( K, d, B t ) ≤ log K 2 K (1 + o t,d (1)) (1) where o (1) → 0 an d o t,d (1) → 0 as K → ∞ . The exact expressions of the o (1) and o t,d (1) terms a re given in Equations (19) and (7), respectiv ely . Remark 1: The deri vations leading to the exp ressions in Theo rem 1 s how that one can also bound the code exponent in a non-asymptotic regime. Unfortunately , thos e expressions are too complicated for prac tical use. 5 Nev e rtheless, this obs ervation implies that the results pertaining to WESC are ap plicable for the same parameter regions as those arising in the context of CS theory . Remark 2: The parameter t can also be allo wed to inc rease with K . For WESCs, the value of t do es not af fect the lower bound on the asymptotic code exponent, while the up per bound is v alid as long as t = o ( K ) . For clarity of exp osition, the proof of the lower boun d is postponed to Section VI, while the proo f o f the upper bound, along with the p roofs of the up per b ounds for other WSC families, are presented in S ection V. W e briefly sketch the main steps o f the proofs in the discussion that follo ws. The proof of the upper bound is b ased on the s phere packing ar gume nt. The classical s phere packing ar g ument is v alid for all WSC families discussed in this p aper . The leading term of the resulting upper bou nd is (log K ) /K . This result can be improved when restricting one’ s attention to the Eu clidean n orm. T he key idea is to show that mo st points of the form Cb lie in a b all o f radius significa ntly smaller than the one de ri ved by the c lassic sphere pack ing ar gume nt. The leading term of the upper bound can in this case be improved from (log K ) /K to (log K ) / ( 2 K ) . The lo we r b ound in Theorem 1 is proved by random coding a r gu ments. W e first rando mly g enerate a family of WESCs from the Gaussian ensemble, with the code rates s atisfying lim ( m,N ) →∞ log N m < log K 4 K (1 + o (1)) . Then we prove that these randomly generated codebo oks satisfy d E ( C , K ) ≥ d with high probability . This fact implies tha t the asymptotic code exponent R ( K, d, B t ) = lim s u p m →∞ log N ( m, K , d, B t ) m ≥ log K 4 K (1 + o ( 1)) . W e also an alyze two more classe s of WSCs: the class of general l 1 -WSCs and the family of nonn egati ve l 1 -WSCs. The characterization of the asymptotic code rates of the se codes is giv en in Theorems 2 and 3, respectiv ely . Theorem 2: For a fixed value of the parame ter t an d su f fic iently lar ge K , the asymptotic c ode expone nt o f l 1 -WSCs is bounded as log K 4 K (1 + o (1)) ≤ R ( K, d, B t ) ≤ log K K (1 + o t,d (1)) , (2) where the expressions for o (1) and o t,d (1) are gi ven in Equations (31) and (6), respectiv ely . Pr oof: The lo we r bound is prov ed in Section VII, while the u pper bound is proved in Section V. Theorem 3: For a fixed value of the parame ter t an d su f fic iently lar ge K , the asymptotic c ode expone nt o f nonne ga tive l 1 -WSCs is bounde d as log K 4 K (1 + o t (1)) ≤ R ( K , d, B t ) ≤ log K K (1 + o t,d (1)) , (3) where the expressions for o t (1) and o t,d (1) are giv en by Equations (40) and (6), respectively . Pr oof: The lo we r and u pper bounds are proved in Sections VIII and V, respectively . Remark 3: The upper bound s in Equations (2) and (3) also hold if one allo ws t to grow with K , so that t = o ( K ) . The lower bou nd in (2) for general l 1 -WSCs doe s not depend on the value of t . Howe ver , the lo wer b ound (3) for nonnegative l 1 -WSCs requires that t = o  K 1 / 3  (see Equation (40) for de tails). This d if ference in the con ver ge nce 6 regime of the two l 1 -WSCs is a con seque nce o f the use of different proof techniques . F or the proof of the rate regime of general l 1 -WSCs, Gaussian codebook s were use d. On the other hand, for nonnegati ve l 1 -WSCs, the analys is is complicated by the fact tha t one has to analyze linear combinations of nonnegati ve rando m v ariables . T o overcome this difficulty , we used the Cen tral Limit Theorem an d Berry-Essen type of distributi on approximations [14]. Th e obtained results depend on the v alue of t . Remark 4: The upper bound for WESCs is roughly o ne half of the correspond ing bound for l 1 -WSCs. This improvement in the cod e exponent of WESCs rests on the fact that the l 2 -norm of a vector can be expres sed as an inne r p roduct, i.e. k v k 2 2 = v † v (in other words, l 2 is a Hilbert space). Other normed spaces c onsidered in the paper lack this property , and at the present, we are not able to improve the upp er bounds for l p -WSCs with p 6 = 2 . V . P RO O F O F T H E U P P E R B O U N D S B Y S P H E R E PAC K I N G A R G U M E N T S It is straightforward to apply the sphere pac king argument to upper bound the c ode exponen ts of WSCs . Regard an l p -WSC with arbitrary p ∈ Z + . The superpos ition Cb sa tisfies k Cb k p ≤ k b k 0 X j =1   v i j b i j   p ≤ K t for all b suc h that k b k 0 ≤ K , where the b i j s, 1 ≤ j ≤ k b k 0 ≤ K , denote the nonzero entries of b . N ote that the l p distance of any two superpositions is required to be at least d . The size of the l p -WSC codeboo k, N , satisfies the sphere packing bound K X k =1  N k  (2 t ) k ≤ tK + d 2 d 2 ! m . (4) A simple algebraic manipulation of the above equation shows K X k =1  N k  (2 t ) k ≥  N K  (2 t ) K ≥  N − K K  K (2 t ) K , so that one has log N m ≤ 1 K log  1 + 2 tK d  − log (2 t ) m − log  1 K − 1 N  m = log K K + 1 K log  2 t d + 1 K  − log (2 t ) m − log  1 K − 1 N  m . The asymptotic code exponent is therefore upp er bounded by log K K (1 + o t,d (1)) , (5) where o t,d (1) = log  2 t d + 1 K  log K K →∞ − → 0 (6) if t = o ( K ) . This sphere packing b ound can be significantly improved whe n considering the Euc lidean norm. The resu lt is a n upper boun d with the leading term (log K ) / (2 K ) . The p roof is a gen eralization of the ideas u sed by Füredi an d 7 Ruszinko in [3]: mo st points of the form Cb lie in a ball with radius smaller than q K 3 ( t + 1) , an d therefore the right hand side of the classic sphere p acking bound (5) can be reduced by a factor of two. T o procee d, we a ssign to e very b ∈ B K the probability 1 |B K | = 1 P K k =1  N k  (2 t + 1) k . For a given codeword matrix C , defi ne a rando m v ariable ξ = k Cb k 2 . W e shall uppe r bo und the probability Pr { ξ ≥ λµ } , for arbitrary λ, µ ∈ R + , via Markov’ s inequality Pr ( ξ ≥ λµ ) ≤ E [ ξ ] λµ ≤ p E [ ξ 2 ] λµ . W e calculate E  ξ 2  as follo ws. For a giv en vector b , let I ⊂ [1 , N ] be its support se t - i.e., the set of indices for which the entries of b are nonzero. Let b I be the vec tor composed of the nonzero entries of b . Furthermore, define B t,k = ( B t \ { 0 } ) k . Then, E  ξ 2  = 1 |B K | K X k =1 X | I | = k X b I ∈ B t,k       k X j =1 b i j v i j       2 2 , where i j ∈ I , j = 1 , · · · , k . N ote that X | I | = k X b I ∈ B t,k       k X j =1 b i j v i j       2 2 = X | I | = k X b I ∈ B t,k   k X j =1 b 2 i j + X 1 ≤ l 6 = j ≤ k b i j b i l v † i j v i l   = X | I | = k X b I ∈ B t,k k X j =1 b 2 i j | {z } ( ∗ ) + X | I | = k X b I ∈ B t,k X 1 ≤ l 6 = j ≤ k b i j b i l v † i j v i l | {z } ( ∗∗ ) . It is straightforward to e valuate the two sums in the above expression in c losed form: ( ∗ ) =  N k  X b I ∈ B t,k k X j =1 b 2 i j =  N k  k X b I ∈ B t,k b 2 i 1 =  N k  k (2 t ) k − 1 X b i 1 ∈ B t, 1 b 2 i 1 =  N k  k (2 t ) k − 1 t ( t + 1) (2 t + 1) 3 ; 8 and ( ∗∗ ) =  N k  X 1 ≤ l 6 = j ≤ k X b I ∈ B t,k b i l b i j v † i l v i j =  N k  (2 t ) k − 2 X 1 ≤ i 6 = j ≤ k X b i l ,b i j ∈ B t, 1 b i l b i j v † i l v i j = 0 , where the last equality follo ws from the observation that X b i l ∈ B t, 1 ,b i j ∈ B t, 1 b i l b i j v † i l v i j = X b i l > 0 ,b i j ∈ B t, 1 b i l b i j v † i l v i j + X b i l > 0 ,b i j ∈ B t, 1 ( − b i l ) b i j v † i l v i j = 0 . Consequ ently , one has X | I | = k X b I ∈ B t,k       k X j =1 b i j v i j       2 2 =  N k  (2 t ) k k ( t + 1) (2 t + 1) 6 , so that E  ξ 2  = P K k =1  N k  (2 t ) k k ( t +1)(2 t +1) 6 P K k =1  N k  (2 t ) k . Next, s ubstitute E  ξ 2  into Markov’ s inequality , with µ = p E [ ξ 2 ] , so that for any λ > 1 , it holds that Pr ( ξ ≥ λµ ) ≤ 1 λ . This result implies that at least a ( 1 − 1 /λ ) -fraction of all possible C b vectors lie within an m -dimens ional ba ll of radius λµ around the origin. As a result, one obtains a sphere packing b ound of the form  1 − 1 λ  |B K | ≤ λµ + d 2 d 2 ! m . Note that µ 2 = E  ξ 2  ≤ K 3 ( t + 1) 2 , and that |B K | ≥  N − K K  K (2 t ) K . 9 Consequ ently , one has  1 − 1 λ   N − K K  K (2 t ) K ≤ 1 + λ √ k ( t + 1) d ! m , or , e quiv alently , log N m ≤ log K 2 K + 1 K log  λ ( t + 1) d + 1 √ K  − log  1 − 1 λ  mK − 1 m log  1 K − 1 N  − log (2 t ) m . W ithout loss of generality , we choos e λ = 2 . The asy mptotic c ode exponent is the refore upper bounded by log K 2 K (1 + o t,d (1)) , where o t,d (1) = 2 log K log  2 ( t + 1) d + 1 √ K  K →∞ − → 0 (7) if t = o ( K ) . Th is proves the upper bound of Theorem 1. V I . P R O O F O F T H E L O W E R B O U N D F O R W E S C S Similarly as for the cas e o f c ompressive sensing matrix de sign, we s how that standa rd Gaus sian rando m ma trices, with appropriate scaling, can be use d as cod ebooks of WESCs. L et H ∈ R m × N be a standard Gauss ian rand om matrix, and let h j denote the j th column of H . Let v j = h j / k h j k 2 and C = [ v 1 · · · v N ] . Then C is a codebo ok with unit l 2 -norm codewords. No w c hoose a δ > 0 such that d (1 + δ ) < 1 . Let E 1 = N [ j =1  H : 1 √ m k h j k 2 ∈ (1 − δ, 1 + δ )  (8) be the ev e nt that the normalized l 2 -norms of all the columns of H concentrate arou nd one. Let E 2 = [ B K ∋ b 1 6 = b 2 ∈B K { H : k C ( b 1 − b 2 ) k 2 ≥ d } . (9) In other words, E 2 denotes the event that any two different superpo sitions of co dew ords lie at Euclidea n distance at least d from each other . In the follo wing, we show that for any R < log K 4 K (1 + o (1)) , for which o (1) is given by Equation (19), if lim ( m,N ) →∞ log N m ≤ R , (10) then lim ( m,N ) →∞ Pr ( E 2 ) = 1 . (11) This will establish the lower bound of Theorem 1. Note that Pr ( E 2 ) ≥ Pr  E 2 \ E 1  = P r ( E 1 ) − Pr  E 1 \ E c 2  . 10 According to Theorem 4, stated and prov ed in the next subsection, on e has lim ( m,N ) →∞ Pr ( E 1 ) = 1 . Thus, the desired relation (11) holds if lim ( m,N ) →∞ Pr  E 1 \ E c 2  = 0 . Observe tha t C ( b 1 − b 2 ) = 1 √ m Hb ′ , where b ′ := Λ H ( b 1 − b 2 ) , (12) and Λ H =     √ m/ k h 1 k 2 . . . √ m/ k h N k 2     . (13) By Theorem 19 in Section VI-B, in the a symptotic domain of (10), Pr  E 1 \  H : 1 √ m H  (1 + δ ) b ′  ≤ d (1 + δ )  = P r  E 1 \  H : 1 √ m Hb ′ ≤ d  = P r  E 1 \ { H : C ( b 1 − b 2 ) ≤ d }  → 0 . This establishes the lower bound of Theorem 1. A. Column Norms of H In this s ubsec tion, we quantify the rate regime in which the Euclidean n orms of all columns of H , whe n prope rly normalized, are concentrated around the value one with high probability . Theorem 4: Let H ∈ R m × N be a standard Gaussian random matrix, and h j be the j th column of H . 1) For a given δ ∈ (0 , 1) , Pr      1 m k h j k 2 2 − 1     > δ  ≤ 2 exp  − m 4 δ 2  for all 1 ≤ j ≤ N . 2) If m, N → ∞ simultaneo usly , s o that lim ( m,N ) →∞ 1 m log N < δ 2 4 , then it holds that lim ( m,N ) →∞ Pr   N [ j =1      1 m k h j k 2 2 − 1     > δ    = 0 . 11 Pr oof: The first p art of this theorem is proved by in voking large deviati on s techniques. No te that k h j k 2 2 = P m i =1 | H i,j | 2 is chi-square distrib uted. W e have Pr ( 1 m m X i =1 | H i,j | 2 > 1 + δ ) ( a ) ≤ exp n − m  α (1 + δ ) − log E h e α | H i,j | 2 io = exp  − m  α (1 + δ ) + 1 2 log (1 − 2 α )  ( b ) = exp n − m 2 ( δ − log (1 + δ )) o , (14 ) and Pr ( 1 m m X i =1 | H i,j | 2 < 1 − δ ) ( c ) ≤ exp n m  α (1 − δ ) + log E h e − α | H i,j | 2 io = exp  m  α (1 − δ ) − 1 2 log (1 + 2 α )  ( d ) = exp n − m 2 ( − log (1 − δ ) − δ ) o , (15) where ( a ) and ( c ) hold for arbitrary α > 0 , and ( b ) and ( d ) are obtained by specializing α to 1 2 δ 1+ δ and 1 2 δ 1 − δ , respectively . By ob serving that − log ( 1 − δ ) − δ > δ − log (1 + δ ) > 0 , we arri ve at Pr      1 m k h j k 2 2 − 1     > δ  ≤ 2 exp n − m 2 ( δ − log (1 + δ )) o ≤ 2 exp  − m 4 δ 2  . The second part of the claimed result is prov ed by applying the union bound, i.e. Pr    N [ j =1      1 m k h j k 2 − 1     > δ     ≤ 2 N exp n − m 4 δ 2 o = exp  − m  δ 2 4 −  1 m log N + log 2 m  . This completes the proof of Theorem 4. B. The Distance Between T wo Diff e r ent Super positions This s ection is d ev oted to ide ntifying the rate regime in wh ich any pair of dif ferent superpos itions is at s ufficiently lar ge Euc lidean d istance. Th e ma in result is presented in The orem 6 at the end of this subse ction. Since the proof of this theorem is rather tech nical, a nd sinc e it in volves complicated no tation, we first prove a s implified version of the result, stated in Theorem 5. 12 Theorem 5: Let H ∈ R m × N be a stan dard Gaus sian rando m matrix, and let δ ∈ (0 , 1) b e fixed. For sufficiently lar ge K , if lim ( m,N ) →∞ log N m < log K 4 K (1 + o δ (1)) where the exact expres sion for o δ (1) gi ven by Equation (18), then lim ( m,N ) →∞ Pr  1 m k Hb k 2 2 ≤ δ 2  = 0 , (16) for all b ∈ B N 2 t such that k b k 0 ≤ 2 K . Pr oof: By the union bound, we hav e Pr   [ k b k 0 ≤ 2 K  H : 1 m k Hb k 2 2 ≤ δ 2    = P r   2 K [ k =1 [ k b k 0 = k  H : 1 m k Hb k 2 2 ≤ δ 2    ≤ 2 K X k =1  N k  (4 t ) k Pr  1 m k Hb k 2 2 ≤ δ 2 , k b k 0 = k  . (17) W e sha ll up per bound the probability Pr  1 m k Hb k 2 2 ≤ δ 2 , k b k 0 = k  for each k = 1 , · · · , 2 K . F rom Chernoff ’ s inequa lity , for all α > 0 , it holds that Pr  1 m k Hb k 2 2 ≤ δ 2 , k b k 0 = k  ≤ exp n m  αδ 2 + log E h e − α ( H i, · b ) 2 io , where H i, · is the i th row of the H matrix. Furthermore, E h e − α ( H i, · b ) 2 i = E h exp n − α k b k 2 2 ( H i, · ( b / k b k 2 )) 2 oi ( a ) ≤ E h exp n − αk ( H i, · ( b / k b k 2 )) 2 oi ( b ) ≤ − 1 2 log (1 + 2 αk ) , where ( a ) follo ws from the fact that k b k 2 2 ≥ k for all b ∈ B N 2 t such that k b k 0 = k , and where ( b ) holds becau se H i, · ( b / k b k 2 ) is a standard Gaussian random vari ab le. Le t α = 1 2 k k − δ 2 δ 2 = 1 2 δ 2 − 1 2 k . 13 Then Pr  1 m k Hb k 2 2 ≤ δ 2 , k b k 0 = k  ≤ exp  m  1 2 − δ 2 2 k  − 1 2 log k δ 2  = exp  − m 2  log k − log δ 2 + δ 2 k − 1  . Substituting the above expression into the union bound g i ves Pr   [ k b k 0 ≤ 2 K  H : 1 m k Hb k 2 2 ≤ δ 2    ≤ 2 K X k =1 exp  − m 2  log k − log δ 2 + δ 2 k − 1 − 2 k m log N − 2 k m log (4 t )  ≤ 2 K X k =1 exp  − mk  log k 2 k + δ 2 /k − log δ 2 − 1 2 k − 1 m log N − 1 m log (4 t )  . Now , let K be sufficiently large so that log 2 K 4 K + δ 2 / (2 K ) − log δ 2 − 1 4 K = min 1 ≤ k ≤ 2 K  log k 2 k + δ 2 /k − log δ 2 − 1 2 k  . If lim ( m,N ) →∞ log N m < log K 4 K (1 + o δ (1)) , where o δ (1) = log 2 + δ 2 / (2 K ) − log δ 2 − 1 log K , (18) then lim ( m,N ) →∞ Pr  1 m k Hb k 2 2 ≤ δ 2  = 0 for all b ∈ B N 2 t such that k b k 0 ≤ 2 K . This completes the proof of the claimed result. Based on Theorem 5, the asymptotic region in which Pr ( E c 2 T E 1 ) → 0 is ch aracterized in below . Theorem 6: Let H ∈ R m × N be a standa rd Gaussian random matrix, and for a given H , let Λ H be as defin ed in (13). For a giv en d ∈ (0 , 1) , ch oose a δ > 0 such that d (1 + δ ) < 1 . De fine the set E 1 as in (8). For su f fic iently lar ge K , if lim ( m,N ) →∞ log N m < log K 4 K (1 + o (1)) where o (1) = log 2 − 1 log K , (19) 14 then lim ( m,N ) →∞ Pr  1 m k HΛ H ( b 1 − b 2 ) k 2 2 ≤ d 2 , E 1  = 0 , (20) for all pairs of b 1 , b 2 ∈ B K such that b 1 6 = b 2 . Pr oof: The proof is analogous to that of Theorem 4 with minor change s. Let b ′ = Λ H ( b 1 − b 2 ) . On the set E 1 , since 1 √ m k h j k 2 ≤ 1 + δ for all 1 ≤ j ≤ N , the nonzero entries of (1 + δ ) b ′ satisfy   (1 + δ ) b ′ i   ≥ 1 . Replace b in Theorem 5 with ( 1 + δ ) b ′ . All the arguments in the proof of Theorem 5 are still valid, exce pt that the higher order term is changed to o d (1+ δ ) (1) = log 2 + d 2 (1 + δ ) 2 / (2 K ) − 1 − log  d 2 (1 + δ ) 2  log K ≥ log 2 − 1 log K . This completes the proof of the theorem. V I I . P RO O F O F T H E L O W E R B O U N D F O R l 1 - W S C S The p roof is similar to that of the lower boun d for WESCs. Let A ∈ R m × N be a standa rd Gaussian random matrix, and let H be the matrix with entries H i,j = √ 2 π 2 A i,j , 1 ≤ i ≤ m, 1 ≤ j ≤ N . Once more, let h j be the j th column of H . Le t v j = h j / k h j k 1 and C = [ v 1 · · · v N ] . The n C is a c odeboo k with unit l 1 -norm codewords. No w choo se a δ > 0 su ch that d (1 + δ ) < 1 . Le t E 1 = N [ j =1  H : 1 m k h j k 1 ∈ (1 − δ, 1 + δ )  , (21 ) and E 2 = [ B K ∋ b 1 6 = b 2 ∈B K { H : k C ( b 1 − b 2 ) k 1 ≥ d } . (22) W e conside r the asy mptotic regime where lim ( m,N ) →∞ log N m ≤ R , R < log K 4 K (1 + o (1)) , and o d (1) is gi ven in Equation (31 ). Theorem 7 in Section VII-A sugges ts that lim ( m,N ) →∞ Pr ( E 1 ) = 1 , 15 while Theorem 8 in Section VII-B shows that lim ( m,N ) →∞ Pr  E 1 \ E c 2  = 0 . Therefore, lim ( m,N ) →∞ Pr ( E 2 ) ≥ lim ( m,N ) →∞ Pr ( E 1 ) − Pr  E 1 \ E c 2  = 1 . This result implies the lo we r bo und of Theorem 2. A. Column Norms of H The follo wing theorem quantifies the rate regime in which the l 1 -norms of all c olumns o f H , with prop er normalization, are conce ntrated around one with high probability . Theorem 7: Let A ∈ R m × N be a standard Gaussian random matrix. Let H be the matrix with e ntries H i,j = √ 2 π 2 A i,j , 1 ≤ i ≤ m, 1 ≤ j ≤ N . Let h j be the j th column of H . 1) For a given δ ∈ (0 , 1) , Pr      1 m k h j k 1 − 1     > δ  ≤ c 1 e − mc 2 δ 2 for some positi ve constan t c 1 and c 2 ; 2) Let m, N → ∞ simultaneo usly , w ith lim ( m,N ) →∞ 1 m log N < c 2 δ 2 . The it holds that lim ( m,N ) →∞ Pr   N [ j =1      1 m k h j k 1 − 1     > δ    = 0 . Pr oof: 1) Since A i,j is a standard Gaus sian random variable, | A i,j | is a Subg aussian distrib uted random variable, and E [ | A i,j | ] = 2 √ 2 π . Acc ording to Proposition 1 in Appendix A, | A i,j | − 2 √ 2 π is a Subga ussian random v ariable with zero mean. A direct application of Theorem 12 stated in Appendix A g i ves Pr      1 m k h j k 1 − 1     > δ  = P r      m X i =1  | A i,j | − 2 √ 2 π       > 2 mδ √ 2 π ! ≤ c 1 exp  − c 2 mδ 2  , which proves claim 1). 16 2) This part is proved by using the union bound: first, note that Pr   N [ j =1      1 m k h j k 1 − 1     > δ    ≤ exp  − mc 2 δ 2 + log c 1 + log N  = exp  − m  c 2 δ 2 − 1 m log c 1 − 1 m log N  . If lim ( m,N ) →∞ 1 m log N < c 2 δ 2 , then one has lim ( m,N ) →∞ Pr   N [ j =1      1 m k h j k 1 − 1     > δ    = 0 . This completes the proof of claim 2). B. The Distance Between T wo Diff e r ent Super positions Similarly to the an alysis performed for WESCs, we start with a proo f of a simplified version o f the res ult ne eded in order to simplify tedious n otation. W e then explain h ow to establish the proof of Theorem 9 b y modifying some of the steps of the simplified theorem. Theorem 8: Let A ∈ R m × N be a standard Gaussian random matrix. Let H be the matrix with e ntries H i,j = √ 2 π 2 A i,j , 1 ≤ i ≤ m, 1 ≤ j ≤ N . Let δ ∈ (0 , 1) be given. For s ufficiently large K , if lim ( m,N ) →∞ log N m < log K 4 K (1 + o δ (1)) , where o δ (1) = 2 log K  log π 2 δ − 1  , (23) then lim ( m,N ) →∞ Pr  1 m k Hb k 1 ≤ δ  = 0 (24) for all b ∈ B N 2 t such that k b k 0 ≤ 2 K . Pr oof: The proof starts by using the union bound, a s Pr   [ k b k 0 ≤ 2 K  H : 1 m k Hb k 1 ≤ δ    ≤ 2 K X k =1  N k  (4 t ) k Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  . (25) T o estimate the above upper bound , we have to upp er bound the probability Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  , 17 for each k = 1 , · · · , 2 K . Le t us deri ve next a n expression for such an upper bound that holds for arbitrary values of k ≥ 1 . Note that E h e − α | P k j =1 b j A i,j | i = Z ∞ 0 2 √ 2 π k b k 2 e − x 2 2 k b k 2 2 e − αx · dx ( a ) = Z ∞ 0 2 √ 2 π e − x 2 2 e − α k b k 2 x · dx = e α 2 k b k 2 2 2 Z ∞ 0 2 √ 2 π exp − ( x + α k b k 2 ) 2 2 ! · dx ( b ) = e α 2 k b k 2 2 2 Z ∞ α k b k 2 2 √ 2 π exp  − x 2 2  · dx ≤ e α 2 k b k 2 2 2 Z ∞ α k b k 2 x α k b k 2 · 2 √ 2 π exp  − x 2 2  · dx = 1 α k b k 2 · 2 √ 2 π e α 2 k b k 2 2 2 e − α 2 k b k 2 2 2 ( c ) ≤ 1 α 2 √ 2 π k , (26) where ( a ) and ( b ) follo w from the chan ge of variables x ′ = x / k b k 2 and x ′ = x + α k b k 2 , resp ectiv ely . Inequa lity ( c ) holds based on the assumption that k b k 2 ≥ k . As a result, Pr   1 m m X i =1       X j b j H i,j       ≤ δ   = P r   1 m m X i =1       k X j =1 b j A i,j       ≤ 2 √ 2 π δ   ≤ exp  m  α 2 δ √ 2 π + log E h e − α | P j b j H j | i  ≤ exp  m  α 2 δ √ 2 π + log  2 √ 2 π k 1 α  = exp ( m α 2 δ √ 2 π − log α r π k 2 !!) = exp n m  1 − log  √ k π 2 δ o , (27) where the last equality is obtained by specializing α = √ 2 π / 2 δ . The upper bound in (27) is useful only when it is les s than one, or equiv alently , log  √ k π 2 δ  > 1 . (28) For any δ ∈ (0 , 1) , if k ≥ 4 , inequality (28) holds. Thus, for any k ≥ 4 ,  N k  (4 t ) k Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  ≤ exp  − mk  log k 2 k (1 + o δ (1)) − log (4 t ) m − log N m  (29) → 0 , 18 as ( m, N ) → ∞ with lim ( m,N ) →∞ log N m < log k 2 k (1 + o δ (1)) , where o δ (1) = 2 log k  log π 2 δ − 1  . Another uppe r bound is ne eded for k = 1 , 2 , 3 . For a fixed k taking one of these values, Pr  1 m k Hb k 1 ≤ δ  = P r   1 m X i       X j A i,j b j       < 2 √ 2 π δ   = P r   X i         X j A i,j b j       − 2 k b k 2 √ 2 π   < 2 m √ 2 π ( δ − k b k 2 )   . It is straightforward to veri fy that P j A i,j b j is Gaussian and that E         X j A i,j b j         = 2 k b k 2 √ 2 π . Thus X i         X j A i,j b j       − 2 k b k 2 √ 2 π   is a sum of independen t z ero-mean s ubgaus sian rando m variables. Fu rthermore, k b k 2 ∈ h √ k , √ 2 k t i and therefore, δ − k b k 2 < 0 . Hence, we can apply T heorem 12 of Appe ndix A: as a res ult, there exist positi ve constants c 3 ,k and c 4 ,k such that Pr  1 m k Hb k 1 ≤ δ  ≤ c 3 ,k exp  − c 4 ,k m ( δ − k b k 2 ) 2  ≤ c 3 ,k exp  − c 4 ,k m  √ k − δ  2  . Note that the values of c 3 ,k and c 4 ,k depend on k . Conse quently ,  N k  (4 t ) k Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  ≤ c 3 ,k exp ( − mk c 4 ,k  1 − δ √ k  2 − log (4 t ) m − log N m !) (30) → 0 as m, N → ∞ with lim ( m,N ) →∞ log N m < c 4 ,k  1 − δ √ k  2 . 19 Finally , s ubstitute the upper bounds of (29 ) and (30) into the u nion b ound of Equation (25). If K is lar ge enough so that log K 4 K (1 + o δ (1)) < c 4 ,k  1 − δ √ k  2 for all k = 1 , 2 , 3 , and if log K 4 K (1 + o δ (1)) ≤ min 4 ≤ k ≤ 2 K log k 2 k (1 + o δ (1)) , where o δ (1) is as gi ven in (23), then the desired result (24) holds. Based on Theorem 8, we are ready to characterize the asymptotic region in which Pr ( E c 2 T E 1 ) → 0 . Theorem 9: Define A and H as in Theorem 9. For a giv en H , defin e the diago nal matrix Λ H =     m/ k h 1 k 1 0 . . . 0 m/ k h N k 1     . For a given d ∈ (0 , 1) , c hoose a δ > 0 such that d (1 + δ ) < 1 . Define the set E 1 as in (21). For sufficiently large K , if lim ( m,N ) →∞ log N m < log K 4 K (1 + o (1)) , where o (1) = 2 log K (log π − 1 − log 2) , (31) then lim ( m,N ) →∞ Pr  1 m k HΛ H ( b 1 − b 2 ) k 1 ≤ d, E 1  = 0 for all pairs of b 1 , b 2 ∈ B K such that b 1 6 = b 2 . Pr oof: Let b ′ = Λ H ( b 1 − b 2 ) . On the set E 1 , since 1 √ m k h j k 1 ≤ 1 + δ, for all 1 ≤ j ≤ N , a ll the nonzero entries of ( 1 + δ ) b ′ satisfy   (1 + δ ) b ′ i   ≥ 1 . Replace b in T heorem 8 with (1 + δ ) b ′ . All arguments used in the proof of Theorem 8 are s till valid, except that now , the higher order term (23) in the a symptotic expression reads as o d (1+ δ ) (1) = 2 log K (log π − 1 − log 2 − log ( d (1 + δ ) )) ≥ 2 log K (log π − 1 − log 2) . This completes the proof. V I I I . P RO O F O F T H E L O W E R B O U N D F O R N O N N E G AT I V E l 1 - W S C S The proof follows alon g the same lines as the one described for l 1 -WSCs. Ho wever , there is a se rious technic al dif ficu lty ass ociated with the a nalysis of nonnegative l 1 -WSCs. Let A ∈ R m × N be a standa rd Gaussian rando m 20 matrix. For g eneral l 1 -WSCs, we let H i,j = √ 2 π 2 A i,j , 1 ≤ i ≤ m, 1 ≤ j ≤ N , and therefore, N X j =1 H i,j b j is a Gauss ian random variable, whos e parameters a re easy to determine. Howe ver , for nonnegativ e l 1 -WSCs, one has to set H i,j = √ 2 π 2 | A i,j | , 1 ≤ i ≤ m, 1 ≤ j ≤ N . (32) Since the random v ariables H i,j s are not Gaussian , but rather o ne-sided Gaussian, N X j =1 H i,j b j is not Gaussia n distributed, a nd it is complicated to exactly cha racterize its prope rties. Nev e rtheless, w e can still de fine E 1 and E 2 as in Equations (21) and (22). The res ults of Theorem 7 are s till valid un der the non-negativity ass umption: the norms of all H columns co ncentrate around one in the asymptotic regime de scribed in The orem 7 . The ke y step in the proof of the lower bound is to identify the as ymptotic region in which any two dif ferent sup erpositions a re sufficiently se parated in terms of the l 1 -distance. W e there fore use an a pproach similar to the one we in voked twice before: we first prove a simplified version o f the claim, a nd the n proceed with proving the n eeded result by introduc ing some auxiliary vari a bles a nd notation. Theorem 10: Let A ∈ R m × N be a standard Gaussian random matrix. Let H b e the matrix with entries H i,j = √ 2 π 2 | A i,j | , 1 ≤ i ≤ m, 1 ≤ j ≤ N . Let δ ∈ (0 , 1) be given. For a gi ven suf ficie ntly la r ge K , if lim ( m,N ) →∞ log N m < log K 4 K (1 + o t (1)) where o t (1) is gi ven in (39), then lim ( m,N ) →∞ Pr  1 m k Hb k 1 ≤ δ  = 0 (33) for all b ∈ B N 2 t and k b k 0 ≤ 2 K . Pr oof: Similarly a s for the corresp onding proof for ge neral l 1 -WSCs, we need a tight upper b ound on the moment generation function of the random vari ab le       k X j =1 b j | A i,j |       . For this purpose , we reso rt to the use of the Cen tral Limit Theorem. W e first approximate the distribution of P k j =1 b j | A i,j | by a Gaussian d istrib ution. Then, we uniformly uppe r boun d the approximation error according to the Berry-Esseen The orem (se e [14] and Append ix B for a n overview o f this theo ry). Bas ed on this approximation, we ob tain a n up per bou nd on the mome nt g enerating function, with leading term (log k ) / √ k (se e Equ ation (38) for details). 21 T o simplify the notation, for a b ∈ B N 2 t with k b 0 k = k , let Y b ,k = N X j =1 √ 2 π 2 | A j | b j , where A j s are standard Gaussian random v ariables . Th en, Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  ≤ exp n m  αδ + log E h e − α | Y b ,k | io , where the ineq uality h olds for all α > 0 . Now , we fix α an d u pper bound the momen t gen erating function as follo ws. Note that E h e − α | Y b ,k | i = E  e − α | Y b ,k | , | Y b ,k | ≥ 1 α log √ k  (34) + E  e − α | Y b ,k | , | Y b ,k | < 1 α log √ k  . (35) The first term (34) is upper bounded by E  e − α 1 α log √ k , | Y b ,k | ≥ 1 α log √ k  ≤ E  1 √ k , | Y b ,k | ≥ 1 α log √ k  ≤ 1 √ k Pr  | Y b ,k | ≥ 1 α log √ k  ≤ 1 √ k . (36) In o rder to uppe r bound the seco nd term in Equa tion (35), we ap ply Lemma 2 from the Appendix, proved using the Central Limit Theorem and the Berry-Esseen result: E  1 , | Y b ,k | < 1 α log √ k  = Pr  | Y b ,k | < 1 α log √ k  = Pr         k X j =1 b j | A j |       < 2 √ 2 π 1 α log √ k   ≤ 2 √ 2 π 1 απ log √ k √ k + 12 t 3 √ k E h | A | 3 i = 1 √ k 1 √ 2 π  log k απ + 48 t 3  . (37) Combining the upper bounds in (36) and (37) s hows that E h e − α | Y b ,k | i ≤ 1 √ k  1 + 1 √ 2 π  log k απ + 48 t 3  ≤ 1 √ k  1 + log k 4 α + 24 t 3  . (38) 22 Next, s et α = 1 /δ . Then Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  ≤ exp  − m  1 2 log k  (1 + o t,k (1))  , where o t,k (1) = − 2 + 2 log  1 + log k 4 + 24 t 3  log k . Now we cho ose a k 0 ∈ Z + such that for all k ≥ k 0 , log k 2 k (1 + o t,k (1)) > 0 . It is straightforward to veri fy that k 0 is well defined. Consider the case when 1 ≤ k ≤ k 0 . It can be verified that k X j =1 b j H i,j = √ 2 π 2 k X j =1 b j | A i,j | is Subgaus sian and that E         k X j =1 b j H i,j         ≥ 1 for all b ∈ B N 2 t such that k b k 0 = k . By applying the large deviati on s result for Subg aussian rando m variables, a s stated in Theorem 12, and the union bound, it can be prov ed that there exists a c k > 0 su ch that Pr   [ k b k 0 = k  H : 1 m k Hb k 1 ≤ δ    ≤ exp  − mk  c k − log (4 t ) m − log N m  → 0 . The above result holds whenever m, N → ∞ simultaneo usly , w ith lim ( m,N ) →∞ log N m < c k . Finally , let K be sufficiently lar ge so that log K 4 K (1 + o t, 2 K (1)) ≤ min k 0 ≤ k ≤ 2 K log k 2 k (1 + o t,k (1)) , and log K 4 K (1 + o t, 2 K (1)) ≤ min 1 ≤ k ≤ k 0 c k . 23 Then Pr   [ k b k 0 ≤ 2 K  H : 1 m k Hb k 1 ≤ δ    ≤ 2 K X k =1  N k  (4 t ) k Pr  1 m k Hb k 1 ≤ δ, k b k 0 = k  ≤ k 0 X k =1 exp  − mk  c k − log (4 t ) m − log N m  + 2 K X k = k 0 +1 exp  − mk  log k 2 k (1 + o t,k (1)) − log (4 t ) m − log N m  → 0 , as m, N → ∞ with lim ( m,N ) →∞ log N m < log K 4 K (1 + o t (1)) , where o t (1) = − 2 + 2 log  1 + log 2 K 4 + 24 t 3  log (2 K ) . (39) Based on Th eorem 10, we c an cha racterize the rate region in wh ich any two distinct superpo sitions are suf fic iently separated in the l 1 space . Theorem 11: Define A and H as in Theorem 10. For a given H , define the diagonal matrix Λ H =     m/ k h 1 k 1 0 . . . 0 m/ k h N k 1     . Also, for d ∈ (0 , 1) , choose a δ ∈  0 , 1 2  such that d (1 + δ ) < 1 . Defin e the set E 1 as in (21 ). Provided that K is sufficiently large, if lim ( m,N ) →∞ log N m < log K 4 K (1 + o t (1)) where o t (1) = − 2 + 2 log  1 + log 2 K 4 + 648 t 3  log (2 K ) , (40) then it holds lim ( m,N ) →∞ Pr  1 m k HΛ H ( b 1 − b 2 ) k 1 ≤ d, E 1  = 0 , for all pairs of b 1 , b 2 ∈ B K such that b 1 6 = b 2 . Pr oof: The proof is very similar to that o f The orem 10. The only diff ere nce is the following. Let b ′ = Λ H ( b 1 − b 2 ) . Since 1 2 ≤ 1 − δ ≤ 1 m k h j k 1 ≤ 1 + δ ≤ 3 2 , 24 all the nonzero entries of (1 + δ ) b ′ on the set E 1 satisfy the follo wing ineq uality 1 ≤   (1 + δ ) b ′ i   ≤ 3 t. As a result, we hav e the higher order term o t (1) as gi ven in Equation (40). I X . C O N C L U S I O N S W e introduced a new f amily of codes ov e r the reals, termed we ighted superimpose d cod es. W eighted superimpose d codes c an be a pplied to all problems in which one s eeks to robustly distinguish betwe en bo unded integer valued linear combinations of codewords that obey predefin ed norm and sign c onstraints. As such , they can be see n as a special instan t of c ompressed sens ing schemes in wh ich the s parse s ensing vectors contain en tries from a sy mmetric, bounde d set of integers. W e charac terized the achievable rate regions of three clas ses of weighte d superimpos ed codes, for which the codewords obey l 2 , l 1 , and non-negativity constraints. A P P E N D I X A. Subgaus sian Rando m V ariables Definition 4 (The Subga ussian and Subexponen tial distribut ions ): A ran dom variable X is said to be Subgau s- sian if there exist positive cons tants c 1 and c 2 such that Pr ( | X | > x ) ≤ c 1 e − c 2 x 2 ∀ x > 0 . It is Subexpone ntial if there exist pos iti ve c onstants c 1 and c 2 such that Pr ( | X | > x ) ≤ c 1 e − c 2 x ∀ x > 0 . Lemma 1 (Moment Generating Fu nction): Let X be a zero-mean ran dom variable. The n, the following two statements are equiv alent. 1) X is Su bgauss ian. 2) ∃ c such that E  e αX  ≤ e cα 2 , ∀ α ≥ 0 . Theorem 12: Let X 1 , · · · , X n be inde penden t Subgauss ian rando m variables with zero mean. For any given a 1 , · · · , a n ∈ R , P k a k X k is a Subgaus sian rando m variable. Furthermore, there exist positi ve con stants c 1 and c 2 such that Pr      X k a k X k      > x ! ≤ c 1 e − c 2 x 2 / k a k 2 2 , ∀ x > 0 , where k a k 2 2 = P k a 2 k . Pr oof: See [15, Lecture 5, Theorem 5 and Corollary 6]. W e prove n ext a result that ass erts that translating a Subgaus sian random variable produces another Subgau ssian random vari a ble. Pr oposition 1: Let X be a S ubgaus sian rando m vari a ble. For any given a ∈ R , Y = X + a is a S ubgaus sian random vari a ble as well. Pr oof: It can be verified that for any y ∈ R , ( y − a ) 2 ≤ 1 2 y 2 − a 2 , and ( y + a ) 2 ≤ 1 2 y 2 − a 2 . 25 Now for y > | a | , Pr ( | Y | > y ) = Pr ( X + a > y ) + Pr ( X + a < − y ) ≤ P r ( X > y − a ) + Pr ( X < − y − a ) . (41) When a > 0 , ( 41 ) ≤ Pr ( | X | > y − a ) ≤ c 1 e − c 2 ( y − a ) 2 ≤ c 1 c c 2 a 2 e − c 2 y 2 / 2 . (42) When a ≤ 0 , ( 41 ) ≤ Pr ( | X | > y + a ) ≤ c 1 e − c 2 ( y + a ) 2 ≤ c 1 c c 2 a 2 e − c 2 y 2 / 2 . (43) Combining Equations (42) and (43), one can show that Pr ( | Y | > y ) ≤ c 1 c c 2 a 2 e − c 2 y 2 / 2 , ∀ y > | a | . On the other hand, Pr ( | Y | ≤ y ) ≤ 1 ≤ e c 2 a 2 / 2 e − c 2 y 2 / 2 , ∀ y ≤ | a | . Let c 3 = m ax  c 1 e c 2 a 2 , e c 2 a 2 / 2  and c 4 = c 2 / 2 . Then Pr ( | Y | > y ) ≤ c 3 e − c 4 y 2 . This proves the claimed result. B. The Berry-Ess een Theorem and Its Conse quence The Cen tral Limit The orem (CL T) states that unde r ce rtain con ditions, an ap propriately n ormalized su m of independ ent random variables con ver ge s weakly to the stand ard Gauss ian distrib ution. The Berry-Esse en theorem quantifies the rate at which this con vergence takes p lace. Theorem 13 (The Berry-Ess een Theorem): Let X 1 , X 2 , . . . , X k be i nd epende nt random variables suc h that E [ X i ] = 0 , E  X 2 i  = σ 2 i , E    X 3 i    = ρ i . Also, let s 2 k = σ 2 1 + · · · + σ 2 k , and r k = ρ 1 + · · · + ρ k . Denote b y F k the c umulativ e distrib ution function of the normalized sum ( X 1 + · · · + X k ) /s k , a nd by N the standard Gaussian distrib ution. The n for all x and k , | F k ( x ) − N ( x ) | ≤ 6 r k s 3 k . The Berry-Esse en theorem is used in the proof of the lo wer b ound for the achiev able rate region of nonnegati ve l 1 -WSCs. In the proof, o ne n eed to ide ntify a tight bound on the prob ability of a weighted sum of nonn egati ve 26 random variables. The proba bility of this sum lying in a given interval can be estimated by the Berry-Esseen, a s summarized in the follo wing lemma. Lemma 2: Assume that b ∈ B k t is such that k b k 0 = k , let X 1 , X 2 , · · · , X k be inde pende nt standard Gaussian random vari a bles. For a given positi ve consta nt c > 0 , one has Pr         k X j =1 b j | X j |       < c log √ k   ≤ c π log √ k √ k + 12 ρ t 3 √ k , where ρ := E h | X | 3 i . Pr oof: This lemma is proved by a pplying the Berry-Esse n theo rem. Note that the b j | X j | ’ s are independ ent random v ariables . Their sum P k j =1 b j | X j | can be approximated by a Gaussian random v ariable with properly chosen mean and variance, according to the Cen tral Limit Theorem. In the p roof, we first use the Ga ussian approximation to estimate the probability Pr         k X j =1 b j | X j |       < c log √ k   . Then we subseq uently employ the Berry-Essen theorem to upper bound the approximation error . T o simplify notation, let Y b ,k = k X j =1 b j | X j | , and let N ( x ) de note, a s before, the standard Gaussian distrib ution. The n, Pr  | Y b ,k | < c log √ k  ≤ P r   Y b ,k q P j b 2 j ∈   − c log √ k q P j b 2 j , c log √ k q P j b 2 j     ≤ P r   Y b ,k q P j b 2 j ∈ − c log √ k √ k , c log √ k √ k !   ≤ P r Y b ,k k b k 2 ≤ c log √ k √ k ! − P r Y b ,k k b k 2 ≤ c log √ k √ k ! , where in the second inequality we used the f a ct that b j ≥ 1 , s o that P k j =1 b 2 j ≥ k . According to Theorem 13, for all x ∈ R an d all k ,     Pr  Y b ,k k b k 2 ≤ x  − N ( x )     ≤ 6 ρ P k j =1 | b j | 3  P k j =1 | b j | 2  3 / 2 ≤ 6 k ρt 3 k 3 / 2 = 6 ρt 3 √ k , since P k j =1 | b j | 3 ≤ k t 3 , and P k j =1 | b j | 2 ≥ k . 27 Thus, Pr  | Y b ,k | < c log √ k  ≤ N c log √ k √ k ! + 6 ρt 3 √ k − N − c log √ k √ k ! + 6 ρt 3 √ k ≤ 2 2 π c log √ k √ k + 12 ρt 3 √ k , which completes the proof of the claimed res ult. R E F E R E N C E S [1] W . Kautz and R. Singleton, “Nonrandom binary superimposed codes, ” IEE E T rans. Inform. Theory , vol. 10, no. 4, pp. 363–377, 1964. [2] T . Ericson and L . Györfi, “Superimposed codes in R n , ” IEE E T rans. Inform. T heory , vol. 34, no. 4, pp. 877–880, 198 8. [3] Z. Füredi and M. Ruszinkó, “ An improved upper bound of the rate of Euclidean superimpo sed codes, ” IEEE T rans. Inform. Theory , vol. 45, no. 2, pp. 799–80 2, 1999. [4] D. Donoho, “Compressed sensing, ” IEE E Tr ans. Inform. T heory , vol. 52, no. 4, pp. 1289– 1306, 2006. [5] E. Candès, J. Romberg, and T . T ao, “Robu st uncertainty principles: exact signal reconstruction f rom highly incomplete frequency information, ” IE EE T rans. Inform. Theory , vol. 52, no. 2, pp. 489–50 9, 2006. [6] E. J. Candès, J. K. Romberg, and T . T ao, “Stable signal recov ery from incomplete and inaccurate measuremen ts, ” Comm. Pure A ppl. Math. , vol. 59, no. 8, pp. 1207–12 23, 2006. [7] E. J. Candès and T . T ao, “Near-optimal signal recovery from random projections: Univ ersal encoding strategies?” IEE E T rans. Inform. Theory , vol. 52, no. 12, pp. 5406–5425, 2006 . [8] G. Cormode and S. Muthukrishnan, “What’ s hot and what’ s not: T racking most frequent items dynamically , ” IEEE T rans. Inform. Theory , vol. 50, no. 10, pp. 2231–2242, 2004 . [9] W . Dai and O. Milenkovic, “W eighted euclidean superimposed codes for integ er compressed sensing, ” in IEEE Information T heory W orkshop (IT W) , 2008, submitted. [10] ——, “Constrained compressed sensing via superimposed coding, ” in Information Theory and Applications W orkshop , San Di ego, CA, in vited talk, Jan. 2008. [11] ——, “Sparse weighted euclidean superimposed coding for integ er compressed sensing, ” in Confer ence on Infomation Sciences and Systems (CISS) , 2008, submitted. [12] M. Sheikh, O. Milenko vic, and R. Baraniuk, “Designin g compressi ve sensing DN A microarrays, ” Procee dings of the IEE E W orkshop on Computational A dvances in Multi-Sensor Adaptive Processin g (CAMSAP), St. Thomas, U.S. V irgin Islands , Dec. 200 7. [13] W . Dai, M. Sheikh, O. Milenkovic, and R. Baraniuk, “Probe designs for compressed sensing microarrays, ” in IEEE I nternational Confer ence on Bioinformatics and Biomedicine , Philadelphia, P A , submitted, 2008. [14] W . Feller , An Intr oduction to Pr obability Theory and Its Applications, V olume 2 , 2nd ed. Wiley , 1971. [15] R. V ershynin, Non-asymptotic Random Matrix Theory (L ectur e N otes) , 2007.