A Posteriori Equivalence: A New Perspective for Design of Optimal Channel Shortening Equalizers

VERSION NO VEMBER 6, 2018 1 A P osteriori Equi v alence: A Ne w Perspecti ve for Design of Optimal Ch annel Shortening Equalizers Raman V enkatarama ni Member , IEEE, and M. Fatih Erde n Mem ber , IEEE Abstract — The problem of chann el sh ortening equalization for optimal detection in ISI channels is considered. The problem is to choose a lin ear equalizer and a partial response target filter such th at the combination p roduces t he best detection perfo rmance. In stead of using th e traditi onal approach of M MSE equalization, we directly seek all equalizer and target p airs that yield optimal detection perfo rmance in terms of the sequence or symbol error rate. This leads to a new notion of a posteriori equivalence between the equalized and target channels with a simple characterization in terms of their und erlying probability distributions. Using this characterization we show the su rprising existence an infinite family of equalizer and tar get pairs for which any maximum a posteriori (MAP) b ased detector designed for the target channel is simultaneously MAP optimal for the equ alized channel. Fo r chann els whose in put symbols h a ve equal energy , such as q -PSK, the MMSE equalizer designed with a monic targ et constraint yields a solution b elonging to this optimal family of designs. Although, th ese designs produce IIR target filters, th e ideas are extended to d esign good FIR targets. For an arbitrary choice of target and equ alizer , we derive an expression fo r the probability of sequence d etection error . This expression is used to design optimal FIR targets and IIR equ alizers and to q uantify the FIR app roxima tion penalty . Index T erms — Intersymbol interference, linear equalization, channel sh ortening, partial response, target design, MAP d etec- tion, decision feedback. I . I N T RO D U C T I O N The prob lem of designing chann el shortening equalizer s for max imum-likelihoo d sequ ence detection in inter-symbol interferen ce (ISI) ch annels has been w idely studied [1 –5]. T he function of the equalizer is to mo dify the chan nel response to reduce the leng th of th e ISI in th e system th ereby red ucing the comp lexity of the sequen ce detector . Traditionally , the equalizer is designed so th at the equ alized chann el response approx imates a p re-specified short FIR sequence called the partial response (PR) target. T wo commo nly studied classes of eq ualizers ar e the zero -forcin g equalizer (ZFE) and the minimum m ean-squ ared erro r (M MSE) equalize r . The ZFE forces the equ alized cha nnel resp onse to match the target response exactly . The undesired effect of zero f orcing is that it co lors the noise sp ectrum and may amplify th e noise significantly . In contrast, the MMSE equalizer minim izes the variance of the equalizatio n er ror, but th e error is signal depend ent. In bo th cases the go al is to m ake the equalized channel resp onse close to the target respon se. However , the ultimate goal of the channel shortening eq ualization ought to be a detection per forman ce mea sure such as the seq uence or symbol error rate. The authors are with Seagate Research, Pittsbur gh, P A 15222. Email: raman v@ieee.org, fat ih.erden@sea gate.com In this work we take revisit the p roblem eq ualizer d esign in the co ntext of optim al (MAP) detection o f th e inp ut. Th e main contribution of th is pap er is a new perspective for the problem of ch annel shortenin g equaliza tion in terms o f the under lying a posteriori p robab ilities (APPs) rather than the tradition al approa ch of using the MMSE equalization er ror as the criterion [1–6] . W e p ose the q uestion: In what sense should th e tar get channel be equivalen t to the equalized chann el to achieve best detection performanc e? Th e answer to th is question naturally leads us to a new notion o f a po steriori equ ivalence (APE) between the equalized ch annel and the target ch annel. W e show that this form of equ iv alence, which is e xpressed in ter ms of their und erlying a po steriori probab ilities, guaran tees n o perfor mance loss d ue to equ alization compa red to th e optimal detector for the origina l chan nel. T his result thu s p rovides a new re cipe for e qualizer and target design, which is different from the heuristic app roach o f matching the respo nses of the target an d the equalized chan nel. W e also p rove th at there is a family of IIR eq ualizers and targets which guarantee APE . This paper is o rganized a s follows. In Sections II and III we revie w th e backg roun d ma terial on o ptimal sequ ence detection and linear eq ualization. In Section IV w e co nsider the problem of sequence detection for the equ alized ch annel. W e pr esent our main theor etical resu lts inc luding a po steriori equiv alence a nd its algebr aic character ization. I n Sectio n V we co nsider p ractical imp lications of ou r results. In p articular, we show th at the MM SE equalizer d esigned with a m onic target constrain t y ields an o ptimal solutio n fo r ISI channels when the input symb ols hav e equal energy . Un fortun ately , the equ iv alence co nditions usua lly hold on ly for IIR targets, making the results somewhat useless f or chann el shor tening. Howe ver , in Section VI we extend the results to FIR target design wh ere we seek the best FIR target and IIR equalizer with a small but acceptable “FIR a pprox imation pe nalty . ” W e derive an expression for the sequen ce detection error rate, and use this as a per forman ce me asure for the filter design. For simplicity of the analysis we con sider only IIR eq ualizers. The problem of FIR equa lizer design would entail the ad ditional task of optimizing the pro cessing delay . W e refer the rea der to [6–8] for the problem on op timizing the processing d elay f or systems using MMSE eq ualization. A similar a nalysis related to FIR equ alizer design would be equally imp ortant in ou r problem , but is b eyond the scope o f this p aper . Finally , in Section VII we apply our theor y to an examp le ISI channe l with binary and non- binary inputs to confirm our pred ictions throug h co mputer simu lation. 2 VERSION NO VEMBER 6, 2018 A. Definition s a nd Notation Let a d enote a discrete-time sequ ence { a n : n ∈ Z } . If a has finite energy its discrete -time Fourier tr ansform is defined as F { a } = A ( ω ) = X n a n e − j nω . The co nv olution o f two sequen ces a and b is den oted by c = a ⋆ b : c n = X m a m b n − m . Let δ d enote the discrete delta function : δ n = 0 for n 6 = 0 and δ 0 = 1 . Define the inner product between two seq uences a and b as h a , b i = X n a ∗ n b n = 1 2 π Z π − π A ∗ ( ω ) B ( ω ) dω where ∗ denote s complex conjug ation for scalars or conju gate- transposition f or m atrices. T hus, th e no rm o f a is k a k = h a , a i 1 / 2 . Giv en a seque nce a , let ¨ a be ob tained by time-r ev ersal an d conjuga tion o f a , i.e., ¨ a n = a ∗ − n . The Fourier transfor m of ¨ a is A ∗ ( ω ) . Thus, we rea dily obtain the following identity: h a ⋆ b , c i = h b , ¨ a ⋆ c i (1) i.e., the adjoint of the co nv olution ope ration with a is conv o- lution with ¨ a . Let x an d y denote real or c omplex station ary random processes. T he cr oss-correlatio n f unction is defined by r xy n = ζ − 1 E ( x m + n y ∗ m ) where E ( · ) den otes expectation and ζ is the nu mber of real dimensions per sample, i.e., ζ = 1 for real pro cesses and ζ = 2 for co mplex ones. The au tocorre lation of x is obtained by setting y = x . The power spectral d ensity of x is S x ( ω ) = F { r xx } . W e write x ⊥ y if r xy = 0 . B. ISI Channel Mode l Consider the fo llowing discrete-tim e mo del fo r a r eal or complex-valued linear time inv ariant system y = h ⋆ x + w (2) where x = { x m } is the input to the channe l, h = { h m } is the chan nel impulse r esponse an d w = { w n } is additive white Ga ussian noise with S w ( ω ) = σ 2 w . Assume that h has finite ene rgy but is po ssibly non- causal and in finite. Th e channel mod el (2) is usually the base-band representa tion after whitene d matched filter ing [9] and describes a variety of co mmun ication sy stems. In the case of co mplex chann els, the n oise is assumed to be circu larly symmetric. Thu s, the r eal and imag inary com - ponen ts of the no ise samples are ind epende nt with variance σ 2 w . Let the input power sp ectral den sity b e S x ( ω ) . As a special case w e also sh all consider ind ependen t and identically distributed ( IID) inputs with S x ( ω ) = 1 . An examp le for the input symbol set is the Q -ph ase PSK constellation, C = { √ 2 e j 2 π q /Q : q = 0 , . . . , Q − 1 } in the com plex case or the BPSK (b ipolar binary) con stellation C = {− 1 , + 1 } in the real case. I I . O P T I M A L S E Q U E N C E D E T E C T I O N Suppose that a m essage x = { x m : m = 0 , . . . , M − 1 } of finite leng th M symbo ls is tr ansmitted thr ough the chann el (2). Th e receiv ed sig nal is given by y n = M − 1 X m =0 h n − m x m + w n . (3) Since the add iti ve noise is wh ite Gaussian, we ha ve P ( y | x ) ∝ exp  − D ( y , x ) 2 σ 2 w  (4) where D ( y , x ) = X n    y n − M − 1 X m =0 h n − m x m    2 (5) with the summ ation over n carried over th e finite r egion of interest whe re the samples y n are a vailable. Giv en the ou tput sequ ence y , the m aximum a po steriori (MAP) estimate of x is given by ˆ x def = arg max x P ( x | y ) = arg max x P ( y | x ) P ( x ) = arg min x  D ( y , x ) 2 σ 2 w − lo g P ( x )  (6) where P ( x ) is the prior pr obability distribution on x . If this distribution is unifo rm, then ( 6) redu ces to maximu m- likelihood (ML) detection o f th e input sequ ence: ˆ x = arg min x D ( y , x ) = k y − h ⋆ x k 2 . (7) Unfortu nately , the direct use o f the above expression is limited due to its compu tational com plexity which gr ows exponentially with the length of the ISI. Howev er, when h n is a sho rt FIR sequen ce, the above cost fun ction can be minimized exactly an d computatio nally efficiently using the V iterbi algorithm which was originally devised to decod e conv olutional cod es [9–11]. I I I . R E V I E W O F L I N E A R E Q U A L IZ A T I O N In o rder to im plement th e V iterbi algo rithm to solve the ML seque nce d etection (7) w ith man ageable complexity , we need to redu ce the len gth of the ISI in th e system. This is usually accom plished by using a line ar equalizer to co ndition the channel respon se to match a pre -specified ta r get response . When the target is a short FIR filter, it is called a partial r esponse (PR) target. The V iterbi detector operates on the equalized channel to p erform seq uence detection p retendin g that the samples we re the ou tput of a hypoth etical target channel. VERSION NO VEMBER 6, 2018 3 Let f = { f n } and g = { g n } denote th e eq ualizer an d target filters r espectively . For the moment assum e that the target is fixed. Fig. 1 illustrates the system with an equalizer whose output is z = f ⋆ y = f ⋆ h ⋆ x + f ⋆ w = l ⋆ x + u (8) where l = f ⋆ h is the the respo nse of the equalized ch annel and u = f ⋆ w is the outp ut noise who se p ower spectr al density is S u ( ω ) = | F ( ω ) | 2 S w ( ω ) = σ 2 w | F ( ω ) | 2 . Definition 1 : The tar get channel is a hy pothetical chann el defined by ˜ z = g ⋆ x + v (9) where x is th e input, v is additive wh ite Gaussian no ise with S v ( ω ) = σ 2 v , and ˜ z is the outpu t. The or iginal channe l with the equalizer is illustrated in Fig. 1 an d the target ch annel that appro ximates it is shown in Fig. 2. T radition ally , the equalizer and target are designe d to make th e eq ualized chan nel respon se l clo se to target g , while keeping the noise white. H F x n w n z n y n Fig. 1. The equali zed channel x n G ˜ z n v n Fig. 2. The targ et channe l A. Zer o F or cing Equa lizer (ZFE) The Z FE mo difies the chann el respon se to match the target filter exactly , i.e., l = g . Thus, in the freq uency d omain, th e equalizer is given b y F ( ω ) = G ( ω ) H ( ω ) . (10) The spe ctral de nsity of the noise u is S u ( ω ) = | F ( ω ) | 2 S w ( ω ) = | G ( ω ) | 2 | H ( ω ) | 2 σ 2 w . (11) An un desirable p roblem with zero -forcin g equ alization is that when the channel response | H ( ω ) | has a spectral null o r attains very small values, the equalized no ise is high ly colore d an d has large variance. Th e ZFE is rarely u sed fo r this reason. B. Minimum Mea n S quared Err or ( MMSE) E qualizer A widely u sed equalizer in practical systems is the MMSE equalizer which is designed to m inimize the variance o f the equalization error e defined as e def = g ⋆ x − f ⋆ y . (12) The MM SE equ alizer en sures tha t e ⊥ y , which yields F ( ω ) = S x ( ω ) H ∗ ( ω ) G ( ω ) | H ( ω ) | 2 S x ( ω ) + σ 2 w (13) where S x ( ω ) is the power spectr al d ensities o f the inp ut x . The spe ctral de nsity of the estimation e rror is given b y S e ( ω ) = | G ( ω ) | 2 S x ( ω ) σ 2 w | H ( ω ) | 2 S x ( ω ) + σ 2 w . (14) The ad vantage of th e MMSE design over the Z FE is th at the spectrum of the M MSE no ise (14) is less color ed an d always smaller than th e ZFE no ise (1 1) an d spectr al nulls in H ( ω ) cause n o prob lems. Howe ver , e is signal depen dent, which may c ause the V iter bi de tection to be suboptima l. C. T ar get Design Instead of ch oosing a fixed target, we seek the best target of a fixed leng th. I n pra ctice, the target is usually designed for an MMSE eq ualizer . T hus, we minimize the variance of th e MMSE equ alization err or ( 14): min g 1 2 π Z π − π S e ( ω ) dω (15) where the target g is assumed to have length L : g = { g 0 , g 1 , . . . , g L − 1 } . The r esulting cost fun ction is a simple quad ratic func tion o f the target filter taps. Clearly , with no further constraints on g we obtain the tri vial solution g = 0 . Th erefore , an additional constraint is imposed o n g such as the unit-energy constraint X n g 2 n = 1 (16) or th e monic constraint g 0 = 1 . (17) or som etimes the unit-tap constraint g k = 1 fo r some k . In e ach of these cases, the o ptimal target, kn own as the generalized partial r esponse (GPR) target, is foun d easily by solving (1 5) su bject to the approp riate co nstraints. For i llustrative purposes, we derive th e solution to the mo nic design in the I IR limit ( L → ∞ ), where the prob lem can b e expressed in the fr equency domain as min g = 1 2 π Z π − π | G ( ω ) | 2 S x ( ω ) σ 2 w | H ( ω ) | 2 S x ( ω ) + σ 2 w dω (18) over all causal targets g with g 0 = 1 . The causal an d mon ic constraint on g is cumb ersome to express d irectly in the frequency do main. Howe ver , we know that among all the causal an d stable spectral factors of Q ( ω ) = 4 VERSION NO VEMBER 6, 2018 | G ( ω ) | 2 the value of g 0 is maximiz ed for th e min imum-p hase factor [1 2]. This maximu m value is given by log g 0 = 1 2 π Z π − π log Q ( ω ) dω . Therefo re, we r ewrite the op timization (18) in terms of Q ( ω ) as min g 1 2 π Z π − π Q ( ω ) S x ( ω ) σ 2 w | H ( ω ) | 2 S x ( ω ) + σ 2 w dω such th at 1 2 π Z π − π log Q ( ω ) dω = 0 . (19) The L agrang ian L ( q , λ ) = Z π − π Q ( ω ) S x ( ω ) σ 2 w dω | H ( ω ) | 2 S x ( ω ) + σ 2 w − λ Z π − π log Q ( ω ) dω is stationary at the solution . Using calculu s o f variations, we obtain | G ( ω ) | 2 = Q ( ω ) = λ | H ( ω ) | 2 S x ( ω ) + σ 2 w S x ( ω ) σ 2 w = λ σ 2 w | H ( ω ) | 2 + λ S x ( ω ) (20) where the Lagra nge multip lier λ is ch osen to satisfy (19): λ = exp  − 1 2 π Z π − π log  | H ( ω ) | 2 S x ( ω ) + σ 2 w S x ( ω ) σ 2 w  dω  . The optimal G ( ω ) is th e causal minimum -phase spectral factor of Q ( ω ) , and the MMSE equalizer (13) r educes to F ( ω ) = λ σ 2 w H ∗ ( ω ) G ∗ ( ω ) (21) The sp ectrum o f the e stimation error ( 14) is white for th is solution: S e ( ω ) = λ. (22) Hencefor th, we refer to th is solutio n as the monic design o r monic solu tion im plicitly associatin g the op timal target with the MMSE equalizer . For the spec ial case of zero -mean IID inputs with S x ( ω ) = 1 , the above solution reduce s to F ( ω ) = H ∗ ( ω ) G ( ω ) | H ( ω ) | 2 + σ 2 w (23) | G ( ω ) | 2 = λ σ 2 w ( | H ( ω ) | 2 + σ 2 w ) (24) Coincidentally , this solutio n is related to the lin ear MMSE decision feedback equalizer (DFE) for the given ISI chan nel [13–1 7]. T he MMSE-DFE structur e is op timal in achieving the capacity f or an I SI cha nnel with additive white Gaussian noise [1 6–19] . Howe ver , it is n ot imm ediately obvio us or e ven always true th at the above eq ualizer and target filters would be optimal for sequ ence detectio n of (no n-Gaussian) input symbols. As a caveat we reiterate that the seq uence detection is not meant to be imp lemented with decision feedback . W e still use the V iterbi algorithm or a MAP based algor ithm su ch as the forward-b ackward algor ithm to compu te the symb ol a posteriori prob abilities (APPs). It has be en observed that the monic design perf orms b etter in detection than other d esign criteria such as the energy con straint (16) or the unit-ta p constraint on the target. In the following sectio n, we sh all formally p rove this conjecture . In practice, we need to design FIR eq ualizers and targets with un known channel and noise ch aracteristics. I n th is case the secon d ord er statistics of th e cha nnel in put and ou tput are estimated usin g training and subsequen tly used to design FIR filters. The so lutions to th ese problem s fo r the various target constraints is d escribed in [1, 6 , 20]. W e point out that this method is also applicable if the noise is co lored because the design e nsures that the no ise wh itening is automatically absorbed into the equalizer f . I V . S E Q U E N C E D E T E C T I O N F O R T H E E Q UA L I Z E D C H A N N E L T raditionally , the seq uence d etection is perfor med in two steps. The first step is to equalize the ch annel ou tput. The next step is to p erform the detectio n pretend ing that the equalizer ou tput z (Fig. 1) were the outp ut of the hypoth etical target channel ( Fig. 2). In oth er words, althoug h th e sequ ence detector is optimally designed for the target ch annel it is, in reality , app lied to the equalized chan nel. In this section we consider the per forman ce of such a detecto r . For simplicity of analysis we assume that the target an d e qualizer are IIR and the target is causal. W e consider the design of FIR targets in Section VI. Consider the system described b y ( 8), restated below: z = l ⋆ x + f ⋆ w . By design , the above ch annel appro ximates the target chann el (9). Th e con ditional prob ability o f the outp ut of the target channel is P ( ˜ z | x ) ∝ exp  − ˜ D ( z , x ) 2 σ 2 v  (25) where ˜ D ( ˜ z , x ) = X n    ˜ z n − M − 1 X m =0 g n − m x m    2 . (26) with the summatio n over n ca rried over a finite region of interest wh ere th e samples of ˜ z are av ailable. The f ollowing result provides an alternate expression for ˜ D ( ˜ z , x ) wh ich will be u seful in proving a for m of equiv alence between the target channel an d th e equ alized ch annel. Lemma 1: Suppose th e equa lizer f an d target g are chosen such th at ¨ g ⋆ f = α ¨ h for som e α > 0 , then ˜ D ( z , x ) − k z k 2 = h x , s ⋆ x i + α ( D ( y , x ) − k y k ) 2 where s = ¨ g ⋆ g − α ¨ h ⋆ h . Pr o of: W e begin b y expan ding ˜ D ( z , x ) as follows ˜ D ( z , x ) = k z − g ⋆ x k 2 = k z k 2 − 2 ℜh g ⋆ x , z i + h g ⋆ x , g ⋆ x i = k z k 2 − 2 ℜh x , ¨ g ⋆ f ⋆ y i + h x , ¨ g ⋆ g ⋆ x i VERSION NO VEMBER 6, 2018 5 where ℜ de notes the re al part. The la st step follows by applying (1) an d using z = f ⋆ y . Using the hypo thesis that ¨ g ⋆ f = α ¨ h we o btain ˜ D ( z , x ) = k z k 2 − 2 α ℜh ¨ h ⋆ y , x i + h x , ¨ g ⋆ g ⋆ x i . (27) Meanwhile, a similar argument shows that D ( y , x ) = k y − h ⋆ x k 2 = k y k 2 − 2 ℜh ¨ h ⋆ y , x i + h ¨ h ⋆ h ⋆ x , x i . (28) From ( 27) an d (2 8), we obtain the de sired re sult ˜ D ( z , x ) − k z k 2 = h x , s ⋆ x i + α ( D ( y , x ) − k y k ) 2 where s = ¨ g ⋆ g − α ¨ h ⋆ h . A. Equiva lence o f Equalized Chan nel a nd T ar get Channel W e now in terpret Lemma 1 in terms of the un derlyin g probab ility d istributions. L et u pper-case letters de note ra ndom variables an d lower-case letters deno te realizations of these random variables. Supp ose that F ( ω ) is a stable filter , i. e., it has no spectral nu lls or sing ularities. Then , z = f ⋆ y is in vertible. He nce, f or th e equalized cha nnel P ( x | z ) = P ( x | y ) ∝ P ( x ) P ( y | x ) ∝ P ( x ) exp  − D ( y , x ) 2 σ 2 w  . where the constants of prop ortionality ab ove (and hen ceforth ) are always inde penden t of x . Using Lem ma 1 and noting that y a nd z are constants, we obtain P ( x | z ) ∝ P ( x ) exp  − ˜ D ( z , x ) 2 ασ 2 w + h x , s ⋆ x i 2 ασ 2 w  . (29) Suppose tha t th e hy pothetical target ch annel is assigned an input pr ior d istribution ˜ P ( x ) which is possibly different fro m P ( x ) . The a posteriori pro bability of x is P ( x | ˜ z ) ∝ ˜ P ( x ) P ( ˜ z | x ) ∝ ˜ P ( x ) exp  − ˜ D ( ˜ z , x ) 2 σ 2 v  . (30) Comparing (29) and (30), we see that by setting th e noise variance σ 2 v and the input prior distribution ˜ P ( x ) of the target channel ( 9) to σ 2 v def = ασ 2 w (31) ˜ P ( x ) ∝ P ( x ) exp  h x , s ⋆ x i 2 σ 2 v  (32) we en sure tha t the a posteriori PDFs fo r the equalize d and target chan nels are equal: P ( X = x | Z = z ) = P T ( X = x | ˜ Z = z ) with th e unde rstanding th at the left-han d side is the APP correspo nding to the equalized ISI ch annel ( 2) with a prior P ( x ) o n x , while th e right-han d side is the APP cor respond ing to the targ et channel ( 9) with inp ut PDF ˜ P ( x ) . Remark 1 : W e reiterate tha t the target chann el is a hy - pothetical chan nel and we are fr ee to treat its p arameters g and σ 2 v as well as its input PDF ˜ P ( x ) as design parameters . W e assume neither that f is the MMSE eq ualizer d esigned for the target g no r that σ 2 v is th e variance of e qualization error . Altho ugh this a pproac h is rad ically different from the traditional ap proach in the literature on chann el shortenin g equalization [1, 3, 6], it is essen tial to d erive the correct form of equivalen ce between the target an d equa lized chan nels defined below . Definition 2 : The eq ualized channel is equivalent to the target chann el in the a po steriori sense if they pr oduce the same a posteriori prob ability f or the in put g iv en the output. This fo rm of e quiv alence is called a posteriori eq uiv alence (APE). Evidently , this defin ition o f equivalence is the m ost natural one from the per spective of MAP detection. As a c av eat, we point out that P T ( ˜ Z = z ) and P ( Z = z ) need not be equa l, i.e., the equ alizer output z would not be a ty pical ou tput of the target channel. Th e above observation may be stated succin ctly as f ollows: Theor em 1 : The eq ualized cha nnel ( 8) with the p rior dis- tribution P ( x ) and th e target cha nnel (9) with the pr ior distribution ˜ P ( x ) are a p osteriori e quiv alent. In g eneral, the MMSE or ZFE equalizers do no t gu arantee this form of equ iv alence e ven thoug h they attemp t to make the equalized ch annel resp onse clo se to the target response. Cor o llary 1: Supp ose that the target and equalizer are cho- sen to be the mon ic solutio n (20) a nd (21). Furth ermor e, let σ 2 v = λ and let ˜ P ( x ) ∝ P ( x ) exp  h x , s ⋆ x i 2 λ  be the inp ut p rior distribution for the target c hannel (9) where S ( ω ) = λ/S x ( ω ) . Th en, the equalized ch annel is eq uiv alent to the target ch annel in the a p osteriori sense. Pr o of: O bserve that th e monic target (2 0) and e qualizer (21) satisfy the h ypotheses in Lemm a 1 if we set α = λ/σ 2 w and S ( ω ) = λ/S x ( ω ) . Th erefor e, by (31), σ 2 v = ασ 2 w = λ . The claim ed r esult follows fro m Th eorem 1. The a bove result shows that we ca n use the mon ic design for optimal MAP detectio n provided that we use the pr ior distribution ˜ P ( x ) fo r the target c hannel. In many cases, th e input is II D with a flat spectrum ( S x ( ω ) = 1 ) im plying that ˜ P ( x ) = P ( x ) , i.e., w e do not n eed a different pr ior PDF fo r the target channel. Remark 2 : If we pr etend that the equalizer outpu t z came from the ou tput of the target chan nel with a carefully ch osen input prior distribution, th en all MAP-based detection algo- rithms de signed for the target chann el work optimally whe n applied to th e equ alized chan nel. These algorithm s includ e hard-d ecision d ecoding suc h as th e V iterbi alg orithm, and soft-decision d ecoding suc h as soft-outpu t V iterbi algorithm (SO V A) and the BCJR algorithm . Soft-decisio n alg orithms, unlike the V iterbi algorithm , use an extra parameter, viz. the variance of the additive n oise in the channel. When applying soft decodin g to the target chann el, we must u se σ 2 v as this variance par ameter . Our calculation above show that σ 2 v simply equals λ , the equalizatio n err or variance (see (2 2)). This fact is rou tinely assumed in m any system designs with no rigorou s justification but it is fortun ately th e correct value to use. 6 VERSION NO VEMBER 6, 2018 V . P R AC T I C A L C O N S I D E R A T I O N S W e n ow consider some practical imp lications o f our main result in Section IV. He nceforth , we assume that P ( x ) is a unifor m distribution over the set o f allowed code sequ ences. In this c ase, the MAP sequence estimate (6) co incides with the ML estimate (7). Theor em 2 : Suppo se th at all th e input sequences in the message codeboo k have equa l en ergy an d that th e equalizer f and ta rget g are c hosen such that G ∗ ( ω ) F ( ω ) = αH ∗ ( ω ) (33) | G ( ω ) | 2 = α ( | H ( ω ) | 2 + β ) (34) for some α > 0 and β ∈ R that pro duces a valid G ( ω ) , the n we can set ˜ P ( x ) = P ( x ) . Furthermo re, if P ( x ) is unifo rm, the optimal estimate of th e input is ˆ x = arg min x D ( y , x ) = arg min x ˜ D ( z , x ) . (35) Pr o of: In the time dom ain, the hypoth eses imply th at s = ¨ g ⋆ g − α ¨ h ⋆ h = αβ δ and ¨ g ⋆ f = α ¨ h . Ther efore, ˜ P ( x ) = P ( x ) . The pr oof n ow readily follows b y apply ing Theorem 1. Theorem 2 is applicable , fo r example, if the input symbols are elemen ts of the Q -phase PSK constellation, i.e., x n ∈ C = { √ 2 e j 2 π q /Q : q = 0 , . . . , Q − 1 } in the co mplex case o r th e BPSK con stellation C = {− 1 , + 1 } in th e real case, since all message sequ ences h ave equ al energy . Clearly , for this special family of equ alizer and target filters there is no p erfor mance loss in sequence d etection if we minimize th e surroga te cost f unction ˜ D ( z , x ) instead of th e original cost D ( y , x ) . The practical implications of this result are that in the IIR limit we can ac hieve op timal sequ ence detection u sing any solution from the family (see also [21] ). In general, th ese targets are as lo ng a s the channel itself. Howe ver , we re quire a short FIR target for a V iterb i-based implementatio n. W e address th is pr oblem in Section VI where we show how to design goo d FIR targets to min imize the detection err or rates. Note that the param eter α is merely a scaling factor ( the target and eq ualizer scale as √ α ) but β affects the shape of the filters. Thus, we have a degre e o f freedo m in design represented by β . W e also have the freedo m to cho ose the phase respon se of G ( ω ) . Howe ver , the mo st lo gical ch oice would be to choo se G ( ω ) as the c ausal minimu m-ph ase spectral factor of (3 4). W e now consider sev eral intere sting cases in the family of optim al solutions: 1) Th e case α = 1 an d β = 0 pr oduces | G ( ω ) | 2 = | H ( ω ) | 2 and F ( ω ) = H ∗ ( ω ) G ∗ ( ω ) = G ( ω ) H ( ω ) which is an all- pass zero-fo rcing equalizer filter which keeps the noise white. 2) Setting α = λ/σ 2 w and β = σ 2 w yields the monic solution (see (2 3) and (24)) fo r S x ( ω ) = 1 , proving its con- jectured o ptimality in the asympto tic (II R) case. When β 6 = σ 2 w , the solutio n correspon ds to an monic design fo r a d ifferent n oise level. Howe ver , th is mismatch causes no perfor mance loss in sequence detection. Curiously , some n egativ e values β ∈ ( − inf ω | H ( ω ) | 2 , 0) also y ield optimal solutio ns even thoug h they do not repr esent the variance o f any meaningf ul no ise. Remark 3 : The above argumen t shows that the monic de- sign is an optimal choice if the in put spectrum is white. Howe ver , suppose that cha nnel inp ut sp ectrum is co lored, perhap s by the u se o f spectral shapin g cod es. Then , the monic design (2 1) has the requ ired form in Theore m 2 . However , the target (2 0) does not b ecause it depends on S x ( ω ) . H ence, the monic design may be suboptima l fo r colored inpu ts. In fact, for optim ality we must perfo rm the m onic desig n for the target and eq ualizer with an IID inpu t regard less of whether the actual inpu t is white or color ed. This is particular ly tru e at low SNRs wh ere the σ w is large and the seco nd term in (20) do minates. At high SNR values, the effect of the inp ut spectral co lor o n tra ining dim inishes. A. Matched F ilter Equalization W e br iefly examine the special case o f the solutions in Theorem 2 when we let β → ∞ . This corr esponds to the monic solution for a very low SNR, i.e., σ 2 w → ∞ . For conv enience, we let α = β witho ut loss of g enerality . Then, (33) an d (3 4) imply that | G ( ω ) | 2 = β 2 (1 + | H ( ω ) | 2 β − 1 ) (36) and F ( ω ) = β H ( ω ) ∗ /G ∗ ( ω ) . For β ≫ 1 , we use (3 6) to express G ( ω ) as G ( ω ) = β + A ( ω ) + O ( β − 1 ) where A ( ω ) mu st b e causal if G ( ω ) is minimu m-pha se. Thus, as β → ∞ we h av e F ( ω ) appr oaches the matched filter H ∗ ( ω ) . Now , o bserve that | G ( ω ) | 2 = β 2 h 1 + ( A ( ω ) + A ∗ ( ω )) β − 1 + O ( β − 2 ) i Comparing this with (36), we obtain A ( ω ) + A ∗ ( ω ) = | H ( ω ) | 2 + O ( β − 1 ) . Therefo re, in the time-dom ain a n =      r h n if n > 0 r h 0 / 2 if n = 0 0 , if n < 0 where r h = ¨ h ⋆ h is the auto -correla tion f unction of h . Using g = β δ + a + O ( β − 1 ) it is readily verified that ˜ D ( z , x ) = k z − g ⋆ x k 2 = k z k 2 − 2 ℜh g ⋆ x , z i + h x , ¨ g ⋆ g ⋆ x i = β 2 k x k 2 − 2 β ( ℜh x , z i − h x , a ⋆ x i ) + O (1) Since k x k 2 is constant for all inp uts sequences and β → ∞ , we deduce that the ML estimatio n ru le beco mes arg min x ˜ D ( z , x ) = arg max x ℜh x , z − a ⋆ x i . (37) VERSION NO VEMBER 6, 2018 7 W e inter pret th e above calculations as follows. The equalizer is a matched filter: f = ¨ h and the term z − a ⋆ x repr e- sents the equalizer outpu t with the post-cu rsor I SI r emoved using decision feedbac k. The estimator simply maximizes the correlation between this sequence with the input. It is easy to verify th at h x , a ⋆ x i = 1 2 k h ⋆ x k 2 . Thus, the matched filter equ alization structur e may b e der iv ed alterna- ti vely directly fr om Lemma 1 by letting g = δ a nd f = ¨ h . This approach gives u s th e fo llowing ru le for ML estimation ˆ x = a rg max x ℜh x , z i − 1 2 k h ⋆ x k 2 which is equivalent to (37). V I . O P T I M A L F I R T A R G E T D E S I G N In the pr evious section s we showed the existence a family of eq ualizers and targets that achieve the op timal seq uence detection pe rforma nce if we pr etend that the equ alizer outp ut came fr om the target chan nel. Unfo rtunately , the optimal target, bein g the minimu m p hase spectral factor of (2 4), has the same length as the original ch annel (except in rare cases where it can be sho rter). As such, we hav e not redu ced th e detector co mplexity by equalization. In this section, we co nsider th e more practical problem of th e d esign o f FIR targets to a chieve the best detection perfor mance. W e con sider only re al chann els with BPSK input symbols ( C = {− 1 , +1 } ). W ith som e effort, these resu lt c an be gen eralized to complex cha nnels or n on-bin ary inpu ts as well. Suppose that x ◦ is the actual inpu t to the chan nel, and ˆ x is the ML sequen ce estimate. Then e = ( ˆ x − x ◦ ) is an err or sequence . W e say that two error seq uences b elong to the same equivalen ce class if they ar e related to eac h other b y a time- shift or phase-rotatio n ( or sign-change ). Of all erro r sequences, a dom inant err or seq uence is on e that which min imizes k ˜ e k 2 where ˜ e = h ⋆ e is th e n oise-free ch annel respon se to the input e . W e call ˜ e a do minant ou tput err or sequence . Clearly , domina nt error sequence s are not un ique becau se all seq uences in the equivalence class of a do minant err or sequences are also dominan t. Howe ver , we shall assum e that there is a uniq ue dom inant equ iv alence class whose repr esen- tati ve elem ent e has th e can onical for m: e 0 6 = 0 a nd e n = 0 for n < 0 . Indeed , some ch annels co uld h av e a multiplicity of d ominan t events that belo ng to the different eq uiv alence classes. In th at ca se our pr obability o f error estimate would be scaled by the multiplicity factor . Let Q g ( · ) be the Gaussian Q -function Q g ( x ) = 1 √ 2 π Z ∞ x e − t 2 / 2 dt. W e n ow estimate the pro bability of seq uence detection error for any cho ice of target an d equalizer in terms of the Q - function . Theor em 3 : At hig h SNR, the pro bability of sequence de- tection error for a r eal BPSK chann el is given by P seq e ≃ κQ g ( √ SNR ) for some con stant κ with SNR is the effective signal-to- noise ratio of the system SNR = max v |ℜh e , p ⋆ h ⋆ e i| 2 k ( q − ¨ p ⋆ ¨ h ) ⋆ e − v k 2 + σ 2 w k p ⋆ e k 2 where p = f ⋆ ¨ g , q = g ⋆ ¨ g , and v is any sequ ence with th e same temp oral su pport as the dominant er ror sequence e . Theorem 3 is proved in A ppend ix I using er ror an alysis similar to th at of stan dard V iterbi detection [9, 22]. Note that the bit erro r rate ( BER) a lso takes the same fo rm as P seq e but has a different constant th an κ . The above result is a pplicable for FIR an d I IR equ alizers and targets. The op timal equalizer f and target g ar e chosen to m aximize SNR subject to relevant constraints. For practical reasons, we seek FIR targets, since the detector implementatio n co mplexity is expon ential in the target length. The co nstraint on the eq ualizer length is less impo rtant since the complexity growth is on ly linear . For simplicity we assum e that th e equ alizer is IIR but the target is FIR with len gth L . In this case, it is m ore convenient to max imize SNR over p an d q b ecause f and g can be recovered un iquely from p and q by sp ectral factor ization. Note th at p is I IR but q , b eing the autocorr elation fun ction o f g , is FIR. Furthermo re, we have Q ( ω ) ≥ 0 . W e write SNR = max v SNR ( p , q , v ) wh ere SNR ( p , q , v ) def = |ℜh e , p ⋆ h ⋆ e i| 2 k ( q − ¨ p ⋆ ¨ h ) ⋆ e − v k 2 + σ 2 w k p ⋆ e k 2 Now o bserve that SNR ( p , q , v ) = SNR ( p , q + β δ , v − β e ) for any ( p , q , v ) and β ∈ R . Mo reover , if v has the sam e temporal su pport as e , then so does v ′ = v − β e . Since we are min imizing SNR ( p , q , v ) over all v , we c onclud e th at the quantity max p , v SNR ( p , q , v ) . (38) would remain u nchan ged if w e rep lace q by q + β δ . This enables u s to temp orarily replace constrain t Q ( ω ) ≥ 0 b y q 0 = 0 for th e sake of the ma ximization. Having rid of the constraint on Q ( ω ) , the max imization is readily tra nsformed into a quadratic minimization . As a final step, we add a sufficiently large β to the so lution Q ( ω ) to m ake it satisfy Q ( ω ) ≥ 0 . The an alytical solution to (38) is p resented in th e Appen dix II. W e also show there that th e n oise variance in th e h ypoth et- ical target chann el noise variance (31) is set to σ 2 v = λ , the Lagrang e mu ltiplier u sed in the optimization. Clearly , the above maximizatio n admits infinitely m any solutions parameter ized by β . As the target length approa ches infinity , these solutio ns conv erge pr ecisely to the family o f solutions in Th eorem 2. In this limit the equ alizer and target filters of Theorem 1 ma ximize the ef fectiv e SNR. Furthe rmore, this m aximum v alue is SNR max = k h ⋆ e k 2 σ 2 w . (39) In p ractice we are interested in FIR equalizers for ease of implementatio n. W e point ou t that we c ould still m aximize the effecti ve SNR, albe it numerically , over all FIR targets a nd 8 VERSION NO VEMBER 6, 2018 2 4 6 8 10 12 14 16 0 0.05 0.1 0.15 0.2 0.25 Target Length FIR Approximation Loss (dB) Fig. 3. FIR approxima tion loss vs. targ et length equalizers with length constrain ts. If we choo se to use FIR equalizers, we would have the additional task of op timizing the proce ssing delay which is an im portant design par ameter [6–8] . V I I . E X A M P L E S W e now il lustrate o ur results of the pr eceeding sections with an example. Consider the real ISI chan nel (2) with impulse response h n = ( e − n/ 2 , 0 ≤ n ≤ 8 0 otherwise. with IID binary inpu t symbols ( x n ∈ C = {− 1 , +1 } ) and SNR defined as k h k 2 /σ 2 w where σ 2 w is the no ise variance. W e first study the effect of the target length o n the effectiv e SNR o f the system. The optimal eq ualizers a nd targets ar e computed fo r target leng ths of 2 and lo nger an d the resulting values o f SNR are ca lculated. Indeed, in the II R limit for the target leng th we obtain the maximum value SNR max giv en b y (39). Fig. 3 shows the FIR ap pr oximation loss , ( SNR max − SNR ) , for various finite target leng ths at an SNR of 10dB. In this example the o ptimal length-3 target in curs about 0.075d B penalty in perform ance and the per forman ce loss fo r longer targets diminishes quickly . Next, we ev aluate the BER p erform ance of the red uced complexity detec tors. At each SNR we design th e optimal length-3 target and II R equa lizer truncated to 21-taps ( cen- tered at the origin ). Th e equalizer is su fficiently long since it captures most of the e nergy in the equalizer taps. The dominan t error event for this channe l is e = { 1 , − 1 } . W e also design length-2 1 MMSE eq ualizers (c entered at 0 ) and length-3 targets describe d in Sectio n III for the monic target constraint. Using computer simu lations we compar e th e two de signs in term s of th eir BER perf orman ce for IID binar y inputs. T he two systems use the V iterbi algo rithm to perfo rm the seq uence detection. The results ar e shown in Fig . 4 along w ith th e BER of the fu ll co mplexity V iterbi detector (with 2 8 -states) that uses no chann el shortenin g equalizatio n. It is clear that both the red uced co mplexity detector s perf ormanc iden tically with a small pen alty relative to the full com plexity detecto r . Th e optimality o f the m onic design is p redicted b y T heorem 2 fo r the case o f IIR filter s. Indee d, w e observe n umerically that the monic desig n is near ly optimal for FI R filters as well. 0 5 10 15 SNR (dB) 10 -4 10 -3 10 -2 10 -1 10 0 Bit Error Rate Monic Design Proposed Design Full Complexity BER Performance Fig. 4. Comparison of BER performa nce of two designs for binary signaling Next, we consider th e same ISI chanel with an IID ternary input ( x n ∈ C = {− p 3 / 2 , 0 , + p 3 / 2 } ) which has unit av erage sym bol energy . This in put symb ols themselves have unequ al energy . Recall the results for th e IIR case in Section IV that the op timal seq uence detector for the equa lized c hannel needs to pr etend that it sees the o utput of the targ et channel with the inpu t prio r distribution is given by (32): ˜ P ( x ) ∝ exp  h x , s ⋆ x i 2 σ 2 v  where s = ¨ g ⋆ g − α ¨ h ⋆ h . Thus, the op timal detector needs to min imize the cost function min x  k z − g ⋆ x k 2 − h x , s ⋆ x i  where the second term is correction ter m that orig inates fr om the input p rior distribution ˜ P ( x ) . For the choice of equalizer and ta rget in Th eorem 2, we hav e s def = ¨ g ⋆ g − α ¨ h ⋆ h = αβ δ . Therefo re, h x , s ⋆ x i = αβ k x k 2 , which de pends on the energy of the sequen ce. The correctio n term is an issue o nly for sign al constellations un equal sym bol energies. For the monic target and MMSE equalizer design, we have αβ equals the variance of the equalizatio n er ror, λ . Th us, the c ost function reduc es to min x  k z − g ⋆ x k 2 − λ k x k 2  . W e directly adap t this expression to the FI R case as well by subtracting λ | x n | 2 from the trellis b ranch metric a t time n . I n fact, the detector would be suboptima l with out the corre ction term, a s we confir m below . VERSION NO VEMBER 6, 2018 9 W e design a leng th-3 m onic GPR target and a length- 21 MMSE eq ualizer fo r this ch annel and calculate the sy mbol error rates (SER) nu merically using the V iter bi alg orithm. Fig. 5 shows the SER o btained with and withou t the cor rection term in the trellis b ranch metric. The figure also shows the SER for the f ull com plexity V iterbi detector (with 3 8 states) that uses n o ch annel sho rtening eq ualization. Ther e is a sm all but noticeable ga in in detec tion perfo rmance with the correctio n term. I t must be no ted that th is m odification doe s not require much more d etector complexity . As λ beco mes smaller (at higher SNRs) the correction term to becomes smaller also and in deed, th e per forman ces gain due to the correction term diminishes at hig h SNRs. 0 5 10 15 SNR (dB) 10 -3 10 -2 10 -1 10 0 Symbol Error Rate Without Correction Term With Correction Term Full Complexity SER Performance Fig. 5. Comparison of SER performance for ternary input signaling V I I I . S U M M A RY Although a large body of literatur e exists for the d esign of optimal FIR targets and equalize rs, the implicit assump tion in v irtually all existing work on this subject is that MMSE equalization is optimal. T he purpo se of this work was to question tha t assumptio n. The ma in con tribution of this work is a new per spectiv e for the pro blem o f channel shortenin g equalization in terms of the und erlying a po steriori p roba- bilities unlike the tradition al app roach of using th e MSME equalization er ror as the criterio n. W e introduced the idea o f a posteriori eq uiv alence (APE) betwe en the equ alized and target channels. Under th is form of equiv alence, a ny M AP-based decodin g algor ithm design ed for th e target chan nel would also work optimally when ap plied to the equalized chan nel. In other words, as far as MAP decodin g is con cerned we can prete nd that the equ alized chann el is the target channel. In our analy sis of the p roblem w e treat f , g , σ 2 v (noise variance in the target channe l) an d in some cases even the input PDF ˜ P ( x ) fo r th e hypo thetical target chann el as design parameters. The e quiv alence is expressed as a set of alg ebraic condition s on the design p arameters. The A PE conditio ns admit an infinite family solutions or designs for the equalizer and target. I n the special c ase that the inp ut is IID and all the co de seq uences have equal en ergy , we showed that the “monic solu tion, ” i.e., the MMSE equalizer d esigned fo r a monic constrained target, is shown to belon g to this optimal family of d esigns. W e also observed that the mo nic solu tion must b e designed f or spec trally white in puts even if the actual input is co lored. The family of d esigns p rodu ces IIR filters in general, ma king their p ractical use somewhat lim ited, whe re as for a low com plexity imp lementation of op timal sequenc e detection (using V iterbi or BCJR-like algorithms) we r equire short FI R targets. W e also derived an expression for the p robab ility of se- quence detectio n error assuming II D inputs fo r arbitrar y FIR or I IR targets and equalizer s. Using this as a perf ormanc e measure, we propose a design algorithm to fin d the optimal IIR equalizer and FIR target. Indeed , in the IIR limit for the target these solution coincide with the previously de riv ed optimal IIR family of designs th at satisfy APE. These results are app lied to an examp le ISI channel. Nu- merically , we observe that for IID in puts, we obtain near ly optimal per forman ce using the monic de sign. f or in put signal constellations with uneq ual symbo l energies we also n eed to treat the input PDF ˜ P ( x ) for the target channel as a d esign parameter . T he optimal d etector is d esigned for the target channel with th e p rior ˜ P ( x ) incorpo rated into the V iterbi branch m etric as a cor rection ter m, wh ich would nor mally have been igno red if we simp ly u se the mo nic design. This is illustrated for the IID ter nary signaling examp le (Fig. 5) wh ere we see a small but n oticeable gain by using th e co rrection term. A P P E N D I X I P R O O F O F T H E O R E M 3 Suppose that x ◦ ∈ X is the tran smitted sequen ce, where X is the set o f sequen ces that are equally likely to be tran smitted. The chann el an d equalizer o utputs are y = h ⋆ x + w and z = f ⋆ y respectively . All sequences in the co deboo k hav e equal en ergy because the inp ut sym bols are IID and binary . Thus, the target c hannel inpu t is also treated as b eing II D: ˜ P ( x ) = P ( x ) . The V iterb i detector fo r th e equ alized c hannel com putes the sequence x that min imizes ˜ D ( z , x ) . Thus, the prob ability o f sequence d etection error is P seq e = P  ˜ D ( z , x ) < ˜ D ( z , x ◦ ) fo r som e x 6 = x ◦  ≤ 1 |X | X x ◦ ∈X X x ∈X \ x ◦ P  ˜ D ( z , x ) < ˜ D ( z , x ◦ )  (40) where the secon d step follows from the un ion bound . Using the pr operty th at k a k 2 − k b k 2 = ℜh a − b , a + b i (41) for any a and b , where ℜ deno tes the real p art, we obtain ˜ D ( z , x ) − ˜ D ( z , x ◦ ) = k z − g ⋆ x k 2 − k z − g ⋆ x ◦ k 2 = − 4 ℜh g ⋆ x − , z − g ⋆ x + i (42) where x ± def = x ± x ◦ 2 . 10 VERSION NO VEMBER 6, 2018 Applying ( 1) to (42) an d writin g z = f ⋆ y where y = h ⋆ ( x + − x − ) + w we obtain ˜ D ( z , x ) − ˜ D ( z , x ◦ ) = 4 ℜh x − , ¨ g ⋆ f ⋆ h ⋆ x − i + 4 ℜh x − , ¨ g ⋆ ( g − f ⋆ h ) ⋆ x + i − 4 ℜh x − , ¨ g ⋆ f ⋆ w i ≡ 4( φ ( x − ) + ∆( x − , x + ) − ψ ( x − )) where φ ( x − ) def = ℜh x − , p ⋆ h ⋆ x − i (43) ∆( x − , x + ) def = ℜh x − , ( q − p ⋆ h ) ⋆ x + i ( 44) ψ ( x − ) def = ℜh x − , p ⋆ w i (45) p def = ¨ g ⋆ f (46) q def = ¨ g ⋆ g . (47) Note that ψ ( x − ) ∼ N (0 , σ 2 w k p ⋆ x − k 2 ) is nor mally d is- tributed. T herefo re, Π( x − , x + ) def = P  ˜ D ( z , x ) < ˜ D ( z , x ◦ )  = P  ψ ( x − ) − ∆( x − , x + ) > φ ( x − )  . (48) Thus, (40) can be re written as P seq e ≤ 1 |X | X x − 6 =0 X x + ∈X + ( x − ) Π( x − , x + ) where X + ( x − ) is the set of sequences x + such that x + + x − and x + − x − are valid sequ ences in X . Note that x + is unifor mly distributed in X + ( x − ) when con ditioned on x − . In the hig h SNR regime, it is a good app roximatio n to assum e that dom inant erro r sequ ences are the only source of de tection errors. This allows u s to fix x − = e for any error sequenc e e ∈ E in the equiv alence class E of domin ant er ror seq uences. This yields P seq e ≤ |E | |X | X x + Π( e , x + ) = |E ||X + ( e ) | |X | E Π( e , x + ) with the expectation taken over x + giv en that x − = e . For analytical tractab ility , we assume that ∆( e , x + ) is appro xi- mately n ormally d istributed. T hus, ( 48) y ields P seq e ≃ κQ  φ ( e ) σ ( e )  (49) where σ 2 ( e ) = v ar(∆( e , x + )) + σ 2 w k p ⋆ x − k 2 (50) κ = |E | |X + ( e ) | |X | . (51) The co nstant κ is evidently the prod uct o f the nu mber of allow able domina nt erro r sequences |E | and the pro bability , |X + ( e ) | / |X | , that x ◦ will allow that er ror sequenc e. Th e b it error ra te ( BER) is appr oximated by P bit e = w H ( e ) M P seq e (52) where M is th e length o f the input codewords. The ab ove calculations are similar to proba bility o f error an alysis for classical V iterbi detec tion [ 9]. The only remainin g step is to estimate the variance of ∆( e , x + ) . Fir st no te that ∆( e , x + ) = ℜh e , ( q − p ⋆ h ) ⋆ x + i = ℜh a , x + i where a = ( q − ¨ p ⋆ ¨ h ) ⋆ e . Now , ∆( e , x + ) is zero-m ean because x + is zero-mean . Hen ce, the conditional variance of ∆( e , x + ) is v ar(∆( e , x + )) = 1 |X + ( e ) | X x + ∈X + ( e ) (∆( e , x + )) 2 = 1 |X + ( e ) | X x + ∈X + ( e ) |ℜh a , x + i| 2 . Since the inpu t is b inary with symbols being ± 1 , X + ( x − ) contains all seque nces x + that satisfy e n 6 = 0 = ⇒ x + n = 0 . It is an easy exercise to check that v ar(∆( e , x + )) = X { n : e n =0 } | a n | 2 which m ay also be written as v ar(∆( e , x + )) = min v k a − v k 2 = min v k ( q − ¨ p ⋆ ¨ h ) ⋆ e − v k 2 (53) where v is a vector whose tem poral suppor t is th e same as that of e . Comb ining (43), (4 9), (50), and (53), we obtain P seq e ≃ κQ g ( √ SNR ) where SNR ≃ |ℜh e , p ⋆ h ⋆ e i| 2 min v k ( q − ¨ p ⋆ ¨ h ) ⋆ e − v k 2 + σ 2 w k p ⋆ e k 2 = max v |ℜh e , p ⋆ h ⋆ e i| 2 k ( q − ¨ p ⋆ ¨ h ) ⋆ e − v k 2 + σ 2 w k p ⋆ e k 2 is the effective SNR of the system. A P P E N D I X I I A N A LY T I C A L S O L U T I O N T O ( 3 8 ) The m aximization ( 38) ma y be rewritten as min p , q , v k ( q − ¨ p ⋆ ¨ h ) ⋆ e − v k 2 + σ 2 w k p ⋆ e k 2 subject to q 0 = 0 and ℜh e , p ⋆ h ⋆ e i = 1 (54) thereby removing th e scaling inv ariance of the solution s. Define S = { l : e l 6 = 0 } = { s 1 , . . . , s J } . Then V ( ω ) = X l ∈S v l e − j lω Q ( ω ) = 2 L X l =1 q l cos( lω ) . VERSION NO VEMBER 6, 2018 11 where v l , l ∈ S and q l : l = 1 , . . . , L are the FIR p arameters. Therefo re, Q ( ω ) E ( ω ) − V ( ω ) = B ( ω ) x where B ( ω ) =  B 1 ( ω ) B 2 ( ω )  B 1 ( ω ) = 2 E ( ω )  cos( ω ) co s(2 ω ) · · · cos( Lω )  B 2 ( ω ) = −  e − j s 1 ω , · · · , e − j s J ω  x =  q 1 , · · · , q L , v s 1 , · · · , v s J  T Finally , let R ( ω ) def = P ( ω ) H ( ω ) E ( ω ) . (55) In ter ms of the ab ove quan tities, we can rewrite the o ptimiza- tion as min 1 2 π h Z | B ( ω ) x − R ( ω ) | 2 dω + σ 2 w Z | R ( ω ) / H ( ω ) | 2 dω i subject to 1 2 π ℜ Z R ∗ ( ω ) E ( ω ) dω = 1 . (56) All integrals are taken over [ − π , π ] . The cost fun ction reduc es to min h 1 2 π ℜ Z  A ( ω ) | R ( ω ) | 2 − 2 R ( ω ) ∗ B ( ω ) x  dω + x ∗ C x i where A ( ω ) = 1 + σ 2 w / | H ( ω ) | 2 and C = 1 2 π Z B ∗ ( ω ) B ( ω ) dω . Using variational calcu lus we obtain A ( ω ) R ( ω ) − B ( ω ) x = λE ( ω ) − 1 2 π Z B ∗ ( ω ) R ( ω ) dω + C x = 0 where λ is a Lagran ge multip lier . Solving the above simulta- neous equations yields R ( ω ) = B ( ω ) x + λE ( ω ) A ( ω ) x = λ ( C − D ) − 1 Z B ∗ ( ω ) E ( ω ) 2 π A ( ω ) dω where D = 1 2 π Z B ∗ ( ω ) B ( ω ) A ( ω ) dω . Finally P ( ω ) can be solved from ( 55). Note that λ is uniqu ely determined by the constrain t (5 6). Howe ver , we could choose an arbitrar y value for λ (such as λ = 1 ) since it merely scales the solution withou t altering the value of SNR . In th e long target (IIR) limit, it is easy to see that th e solutions c onv erge to the fo llowing limits: f ⋆ ¨ g = p → λ ¨ h σ 2 w , ¨ g ⋆ g = q → λ ( h ⋆ ¨ h + β ) σ 2 w for some β , which have the re quired for m in T heorem 2 fo r α = λ/σ 2 w . Therefore, (31) suggests that we must set σ 2 v = ασ 2 w = λ. In the FIR case, howe ver , the pro blem o f choosing the c orrect value o f σ 2 v is som ewhat ambiguo us because FI R solu tions do not satisfy the hyp otheses in The orem 2. W e n ominally set σ 2 v = λ in the FIR case as well. Th is is a goo d first approx imation and fine-tunin g this param eter may pro duce better r esults. R E F E R E N C E S [1] D. Messerschmitt, “Design of a finite impulse response for the V iterbi algorithm and decision-fee dback equaliz ation, ” in Int. Conf. on Communicat ion , Minneapoli s, June 1974, pp. 37D.1–37D.5. [2] D. Falconer and F . Magee, “ Adapti ve channel truncat ion for m aximum lik elihood sequence estimation, ” Bell Syst. T ec h. J. , vol. 52, no. 9, pp. 1541–1562, Nov . 1973. [3] W . L ee and F . Hill, “ A maximum-likel ihood sequence estimator with decisio n-feedback equalizati on, ” IEEE T rans. Commun, , vol. COM-25, no. 9, pp. 971–979, Sept. 1977. [4] F . Magee, “ A comparison of compromise V iter bi algorit hm and standard equali zation techn iques ov er band-limit ed channels, ” IEEE T rans. Commun. , vol. COM-23, no. 3, pp. 361–367, Mar . 1975. [5] S. Quer eshi and E. Newha ll, “ An ada pti ve recei ve r for da ta transmission ov er time-dispersi ve channels, ” IEE T rans. Inform. Theory , vol. IT -19, no. 4, pp. 448–457, July 1973. [6] N. Al-Dha hir and J. M. Cioffi, “Efficie ntly computed redu ced-paramet er input-a ided mmse equali zers for ml detecti on: a unified approach, ” IEEE T ransactions on Information Theory , vol. 42, no. 3, pp. 903–915, 1996. [7] N. Al-Dhahir and J. M. Ciof fi, “Mmse deci sion feedback equa lizers: finite-l ength result s, ” IT T rans. on Inform. Theory , vol. 41, no. 4, pp. 961–975, July 1995. [8] N. Al-Dhahir , “ A closed-form performance upper bound on the finite- length MMSE-DFE with arbitrary decisio n delay , ” IT T rans. on Inform. Theory , vol. 41, no. 4, pp. 961–975, July 1995. [9] John G. Proakis, Digital Communications , McGraw-Hill, Inc., New Y ork, NY , 10020, 4th edition, 2001. [10] A. J. V ite rbi, “Error bounds for con volut ional codes and ans asymptot- icall y optimumde coding algorithm, ” IEEE T rans. Inform. Theory , vol. IT -13, pp. 260–269, Apr . 1969. [11] A. J. V iterbi, “Con vol utional code s and their performance in commu- nicat ion systems, ” IEEE T rans. Commun. T echnol. , vol. COM-19, pp. 751–772, Oct. 1971. [12] Alan V . Oppenhei m and Ronald W . Schafer , Digital Signal Pr ocessing , Prentic e-Hall, Inc., E ngle wood Cliffs, Ne w Jersey , 1975. [13] J. Salz, “Optimum mean-square decision feedback equaliza tion, ” Bell syst. T ech. J . , vol. 52, no. 8, pp. 1341–1373, Oct. 1973. [14] D. D. Falcone r and G. J. Foschi ni, “Theory of m inimum mean-square - error QAM system employin g deci sion fee dback equ alizat ion, ” Bell syst. T ec h. J . , vol. 52, no. 10, pp. 1821–1849, Dec. 1973. [15] C. A. Belfiore and J r . J. H. Park, “Decision feedback equalizat ion, ” Pr oc. IEEE , vol. 66, pp. 1143–1156, Aug. 1979. [16] J. M. Ciof fi, G. P . Dude voir , M. V . Eyuboglu, and G. D. F orney , “MMSE decisio n feedbac k equali zers and coding–part I: Equalizat ion results, ” IEEE T rans. Inform. Theory , vol. 43, no. 10, pp. 2582–2594, October 1995. [17] J. M. Ciof fi, G. P . Dude voir , M. V . Eyuboglu, and G. D. F orney , “MMSE decisio n feedback equalize rs and coding –part I: Coding results, ” IEEE T rans. Inform. Theory , vol. 43, no. 10, pp. 2595–2604, October 1995. [18] S. S hamai (Shitz) and R. L aroia, “The intersymbol interference channe l: Lowe r bounds on capaci ty and channel precoding loss, ” IEE E T rans. Inform. Theory , vol. 42, no. 5, pp. 1388–1404, Sept. 1996. [19] T . Guess and M. K. V arana si, “ An inormation theoretic frame work for deri ving canonical decision-f eedback recei ver s in gaussian channels, ” IEEE T rans. Inform. Theory , vol. 51, no. 1, pp. 173–187, Jan. 2005. [20] J. Moon and W . Zeng, “Equalizat ion for maximum like lihood detec tors, ” IEEE T rans. Magnetics , vol. 31, no. 2, pp. 1083–88, March 1995. [21] R. V enkata ramani and M. F . Erden, “ A family of equalizers for optimal sequence detect ion, ” in Proc. Int. Conf. Acoustics and Speec h Signa l Pr oc. , T oulouse , France, May 2006. 12 VERSION NO VEMBER 6, 2018 [22] G. D. Forney , “Maximum likeliho od sequence estimation of digita l sequence s in presence of intersymbol interference , ” IEE E T rans. Info. Theory , vol. IT -18, no. 3, pp. 363–378, May 1972.