D-optimal Bayesian Interrogation for Parameter and Noise Identification of Recurrent Neural Networks
We introduce a novel online Bayesian method for the identification of a family of noisy recurrent neural networks (RNNs). We develop Bayesian active learning technique in order to optimize the interrogating stimuli given past experiences. In particul…
Authors: Barnabas Poczos, Andras Lorincz
D-optimal Ba y esian In terrogation for P arameter and Noise Iden tiation of Reurren t Neural Net w orks Barnabás Pózos ⋆ and András L®rinz Departmen t of Information Systems, Eötv ös Loránd Univ ersit y , Pázmán y P . sétán y 1/C, Budap est H-1117, Hungary WWW home page: http://nipg.info pozoss.ualberta.a, lorinzinf.elte.hu Abstrat. W e in tro due a no v el online Ba y esian metho d for the iden tiation of a family of noisy reurren t neural net w orks (RNNs). W e dev elop Ba y esian ativ e learning te h- nique in order to optimize the in terrogating stim uli giv en past exp erienes. In partiular, w e onsider the unkno wn parameters as sto hasti v ariables and use the D-optimalit y priniple, also kno wn as ` infomax metho d ', to ho ose optimal stim uli. W e apply a greedy te hnique to maximize the information gain onerning net w ork parameters at ea h time step. W e also deriv e the D-optimal estimation of the additiv e noise that p erturbs the dy- namial system of the RNN. Our analytial results are appro ximation-free. The analyti deriv ation giv es rise to attrativ e quadrati up date rules. 1 In tro dution When studying online systems it is of high relev ane to failitate fast information gain on- erning the system [1,2 ℄. As an example, onsider the resear h on real neurons. In one of the exp erimen tal paradigms, resear hers lo ok for the stim ulus that maximizes the resp onse of the neuron [3,4 ℄. Another approa h sear hes for stim ulus distribution that maximizes m utual infor- mation b et w een stim ulus and resp onse [5 ℄. A reen t te hnique assumes that the unkno wn system b elongs to the family of generalized linear mo dels [6℄ and treats the parameters as probabilis- ti v ariables. Then the goal is to nd the optimal stim uli b y maximizing m utual information b et w een the parameter set and the resp onse of the system. W e are in terested in the ativ e learning [7,8 ,9,10 ℄ of noisy reurren t artiial neural net w orks (RNNs), when w e ha v e the freedom to in terrogate the net w ork and to measure the resp onses. Our framew ork is similar to the generalized linear mo del (GLM) approa h used b y [6℄: w e w ould lik e to ho ose in terrogating, or ` ontr ol ' inputs in order to (i) iden tify the parameters of the net w ork and (ii) estimate the additiv e noise eien tly . F rom no w on, w e use the terms ontr ol and interr o gation in ter hangeably; on trol is the on v en tional expression, whereas the w ord in terrogation expresses our aims b etter. W e apply online Ba y esian learning [11 ,12 ,13 ,14 ℄ to aomplish our task. F or Ba y esian metho ds prior up dates often lead to in tratable p osterior distributions su h as a mixture of exp onen tially n umerous distributions. Here, w e sho w that in our mo del omputations are b oth tratable and appro ximation-free. F urther, the emerging learning rules are simple. W e also sho w that dieren t stim uli are needed for the same RNN ⋆ Presen t address: Departmen t of Computing Siene, Univ ersit y of Alb erta, A thabasa Hall, Edmon- ton, Canada, T6G 2E8 mo del dep ending on whether the goal is to estimate the w eigh ts of the RNN or the additiv e noise that p erturbs the RNN. Hereafter w e will refer to this noise as the `driving noise' of the RNN. Our approa h, whi h optimizes on trol online in order to gain maxim um information on- erning the parameters, falls in to the realm of Optimal Exp erimen tal Design, or Optimal Ba y esian Design [15 ,1 ,16 ,17 ,18℄. Sev eral optimalit y priniples ha v e b een w ork ed out and their eienies ha v e b een studied extensiv ely in the literature. F or a review see, e.g., [19 ℄. Our ap- proa h orresp onds to the so-alled D-optimalit y [20 ,21 ℄, whi h is equiv alen t to the information maximization (infomax) priniple applied b y [6℄. W e use b oth terms, the term D-optimality and the term infomax , to designate our approa h. T o the b est of our kno wledge, D-optimalit y has not b een applied to the t ypial non-spiking sto hasti artiial reurren t neural net w ork mo del that w e treat here. The on tribution of this pap er an b e summarized as follo ws: W e use the D-optimalit y (infomax) priniple and deriv e ost funtions and algorithms for (i) the parameter learning of the sto hasti RNN and (ii) the estimation of its driving noise. W e sho w that, (iii) using the D- optimalit y in terrogation te hnique, these t w o tasks are not ompatible with ea h other: greedy on trol signals deriv ed from the D-optimalit y priniple for parameter estimation are sub optimal (basially the w orst p ossible) for the estimation of the driving noise and vie v ersa. W e sho w that (iv) D-optimal ost funtions lead to simple greedy optimization rules b oth for the parameter estimation and for the noise estimation, resp etiv ely . In v estigation of non-greedy m ultiple step optimizations, whi h ma y a hiev e more eien t estimation of the net w ork parameters and the noise, seems diult and is b ey ond the sop e of the presen t pap er. Ho w ev er, (v) for the task of estimating the driving noise w e in tro due a non-greedy m ultiple step lo ok-ahead heuristis. The pap er is strutured as follo ws: In Setion 2 w e in tro due our mo del. Setion 3 onerns the Ba y esian equations of the RNN mo del. Setion 4 deriv es the optimal on trol for its parameter iden tiation starting from the D-optimalit y (infomax) priniple. Setion 5 deals with our seond task, when the goal is the estimation of the driving noise of the RNN. The pap er ends with a short disussion and some onlusions (Setion 6). 2 The Mo del W e in tro due our mo del here. Let P ( e ) = N e ( m , V ) denote the probabilit y densit y of a normally distributed sto hasti v ariable e with mean m and o v ariane matrix V . Let us assume that w e ha v e d simple omputational units alled ` neur ons ' in a reurren t neural net w ork: r t +1 = g I X i =0 F i r t − i + J X j =0 B j u t +1 − j + e t +1 , (1) where { e t } , the driving noise of the RNN, denotes temp orally indep enden t and iden tially distributed (i.i.d.) sto hasti v ariables and P ( e t ) = N e t ( 0 , V ) , r t ∈ R d represen ts the observ ed ativities of the neurons at time t . Let u t ∈ R c denote the on trol signal at time t . The neural net w ork is formed b y the w eigh ted dela ys represen ted b y matries F i ( i = 0 , . . . , I ) and B j ( j = 0 , . . . , J ), whi h onnet neurons to ea h other and also the on trol omp onen ts to the neurons, resp etiv ely . Con trol an also b e seen as the means of in terrogation, or the stim ulus to the net w ork [6℄. W e assume that funtion g : R d → R d in (1) is kno wn and in v ertible. The omputational units, the neurons, sum up w eigh ted previous neural ativities as w ell as w eigh ted on trol inputs. These sums are then passed through iden tial non-linearities aording to Eq. (1). Our goal is to estimate the parameters F i ∈ R d × d ( i = 0 , . . . , I ), B j ∈ R d × c ( j = 0 , . . . , J ) and the o v ariane matrix V , as w ell as the driving noise e t b y means of the on trol signals. In artiial neural net w ork terms, (1) is in the form of r ate o de mo dels . In our rate o de mo del, noise, on trol, and the reurren t ativities inuene the ring rates similarly . W e sho w that analyti ost funtions emerge for this mo del that are free of appro ximation. 3 Ba y esian Approa h Here w e em b ed the estimation task in to the Ba y esian framew ork. First, w e in- tro due the follo wing notations: x t +1 = [ r t − I ; . . . ; r t ; u t − J +1 ; . . . ; u t +1 ] , y t +1 = g − 1 ( r t +1 ) , A = [ F I , . . . , F 0 , B J , . . . , B 0 ] ∈ R d × m . With these notations, mo del (1) redues to a linear equa- tion y t = Ax t + e t . (2) T o fulll our goal, the online estimation of the unkno wn quan tities (parameter matrix A , noise e t and its o v ariane matrix V ), w e rely on Ba y es' metho d. W e assume that prior kno wledge is a v ailable and w e up date our p osteriori kno wledge on the basis of the observ ations. Con trol will b e hosen at ea h instan t to pro vide maximal exp eted information onerning the quan tities w e ha v e to estimate. Starting from an arbitrary prior distribution of the parameters the p osterior distribution needs to b e omputed. This an b e highly omplex, ho w ev er, so appro ximations are ommon in the literature. F or example, assumed densit y ltering, when the omputed p osterior is pro jeted to simpler distributions, has b een suggested [22,23 ,11 ℄. W e shall use the metho d of onjugated priors [24 ℄ instead. F or matrix A w e assume a matrix v alued normal distribution prior. F or o v ariane matrix V in v erted Wishart (IW) distribution will b e our prior. One an sho w for these hoies that the funtional form of the p osteriori distributions is not aeted. W e dene the normally distributed matrix v alued sto hasti v ariable A ∈ R d × m b y using the follo wing quan tities: M ∈ R d × m is the exp eted v alue of A . V ∈ R d × d is the o v ariane matrix of the ro ws, and K ∈ R m × m is the so-alled preision parameter matrix that w e shall mo dify in aordane with the Ba y esian up date. They are b oth p ositiv e semi-denite matries. The densit y funtion of the sto hasti v ariable A is dened as: N A ( M , V , K ) = | K | d/ 2 | 2 π V | m/ 2 exp( − 1 2 tr (( A − M ) T V − 1 ( A − M ) K )) , where tr , | · | , and sup ersript T denote the trae op eration, the determinan t, and transp osition, resp etiv ely . See e.g. [25 ,26 ℄. W e assume that Q ∈ R d × d is a p ositiv e denite matrix and n > 0 . Using these notations, the densit y of the In v erted Wishart distribution with parameters Q and n is as follo ws [25 ℄: I W V ( Q , n ) = 1 Z n,d 1 | V | ( d +1) / 2 V − 1 Q 2 n/ 2 exp( − 1 2 tr ( V − 1 Q )) , where Z n,d = π d ( d − 1) / 4 d Q i =1 Γ (( n + 1 − i ) / 2) and Γ ( . ) denotes the gamma funtion. No w, one an rewrite mo del (2) as follo ws: P ( A | V ) = N A ( M , V , K ) , (3) P ( V ) = I W V ( Q , n ) , (4) P ( e t | V ) = N e t ( 0 , V ) , (5) P ( y t | A , x t , V ) = N y t ( Ax t , V ) . (6) 4 The Infomax Approa h for P arameter Learning Let us ompute the parameter estimation strategy for task (1) (i.e., for task (3)-(6)) as presrib ed b y the infomax priniple. Let us in tro due t w o shorthands; θ = { A , V } , and { x } j i = { x i , . . . , x j } . W e ho ose the on trol v alue in (1) at ea h instan t su h that it pro vides the most exp eted information onerning the unkno wn parameters. Assuming that { x } t 1 , { y } t 1 are giv en, aording to the infomax priniple our goal is to ompute arg max u t +1 I ( θ , y t +1 ; { x } t +1 1 , { y } t 1 ) , (7) where I ( a, b ; c ) denotes the m utual information of sto hasti v ariables a and b for xed param- eters c . Let H ( a | b ; c ) denote the onditional en trop y of v ariable a onditioned on v ariable b and for xed parameter c . Note that I ( θ , y t +1 ; { x } t +1 1 , { y } t 1 ) = H ( θ ; { x } t +1 1 , { y } t 1 ) − H ( θ | y t +1 ; { x } t +1 1 , { y } t 1 ) , holds [27 ℄ and H ( θ ; { x } t +1 1 , { y } t 1 ) = H ( θ ; { x } t 1 , { y } t 1 ) is indep enden t from u t +1 , hene our task is redued to the ev aluation of the follo wing quan tit y: arg min u t +1 H ( θ | y t +1 ; { x } t +1 1 , { y } t 1 ) = (8) = arg min u t +1 − Z d y t +1 P ( y t +1 |{ x } t +1 1 , { y } t 1 ) Z d θ P ( θ |{ x } t +1 1 , { y } t +1 1 ) log P ( θ |{ x } t +1 1 , { y } t +1 1 ) . In order to solv e this minimization problem w e need to ev aluate P ( y t +1 |{ x } t +1 1 , { y } t 1 ) , the p osterior P ( θ |{ x } t +1 1 , { y } t +1 1 ) , and the en trop y of the p osterior, that is R d θ P ( θ |{ x } t +1 1 , { y } t +1 1 ) log P ( θ |{ x } t +1 1 , { y } t +1 1 ) , where P ( a | b ) denotes the onditional probabilit y of v ariable a giv en ondition b . The main steps of these omputations are pro vided b elo w. Assume that the a priori distributions P ( A | V , { x } t 1 , { y } t 1 ) = N ( A | M t , V , K t ) and P ( V |{ x } t 1 , { y } t 1 ) = I W V ( Q t , n t ) are kno wn. Then the p osterior distribution of θ is: P ( A , V |{ x } t +1 1 , { y } t +1 1 ) = P ( y t +1 | A , V , x t +1 ) P ( A | V , { x } t 1 , { y } t 1 ) P ( V |{ x } t 1 , { y } t 1 ) P ( y t +1 |{ x } t +1 1 , { y } t 1 ) = N y t +1 ( Ax t +1 , V ) N A ( M t , V , K t ) I W V ( Q t , n t ) R A R V N y t +1 ( Ax t +1 , V ) N A ( M t , V , K t ) I W V ( Q t , n t )) . This expression an b e rewritten in a more useful form: let K ∈ R m × m and Q ∈ R d × d b e p ositiv e denite matries. Let A ∈ R d × m , and let us in tro due the densit y funtion of the matrix v alued Studen t-t distribution [28 ,26 ℄ as follo ws: T A ( Q , n, M , K ) = | K | d/ 2 π dm/ 2 Z n + m,d Z n,d | Q | n/ 2 | Q + ( A − M ) K ( A − M ) T | ( m + n ) / 2 , No w, w e need the follo wing lemma: Lemma 41 N y ( Ax , V ) N A ( M , V , K ) I W V ( Q , n ) = N A (( MK + yx T )( xx T + K ) − 1 , V , xx T + K ) × ×I W V Q + ( y − Mx ) (1 − x T ( xx T + K ) − 1 x ) ( y − Mx ) T , n + 1 × ×T y Q , n, Mx , 1 − x T ( xx T + K ) − 1 x . Pr o of. It is easy to sho w that the follo wing equations hold: N y ( Ax , V ) N A ( M , V , K ) = N A ( M + , V , xx T + K ) N y ( Mx , V , γ ) , N A ( M , V , K ) I W V ( Q , n ) = I W V ( Q + H , n + 1) T A ( Q , n, M , K ) , (9) where M + = ( MK + yx T )( xx T + K ) − 1 , γ = 1 − x T ( xx T + K ) − 1 x , H = ( A − M ) K ( A − M ) T for the sak e of brevit y . Then w e ha v e N y ( Mx , V , γ ) I W V ( Q , n ) = I W V ( Q + ( y − Mx ) γ ( y − Mx ) T , n + 1 ) T y ( Q , n, Mx , γ ) , and the statemen t of the lemma follo ws. Using this lemma, w e an ompute the p osterior probabilities. Let us in tro due the follo wing quan tities: γ t +1 = 1 − x T t +1 ( x t +1 x T t +1 + K t ) − 1 x t +1 , n t +1 = n t + 1 , M t +1 = ( M t K t + y t +1 x T t +1 )( x t +1 x T t +1 + K t ) − 1 , Q t +1 = Q t + ( y t +1 − M t x t +1 ) γ t +1 ( y t +1 − M t x t +1 ) T . (10) F or the p osterior probabilities w e ha v e determined that P ( A | V , { x } t +1 1 , { y } t +1 1 ) = N A ( M t +1 , V , x t +1 x T t +1 + K t ) , (11) P ( V |{ x } t +1 1 , { y } t +1 1 ) = I W V ( Q t +1 , n t +1 ) , (12) P ( y t +1 |{ x } t +1 1 , { y } t 1 ) = T y t +1 ( Q t , n t , M t x t +1 , γ t +1 ) . Ha ving done so, w e an ompute the en trop y of the p osterior distribution of θ = { A , V } b y means of the follo wing lemma: Lemma 42 The entr opy of a sto hasti variable with density funtion P ( A , V ) = N A ( M , V , K ) I W V ( Q , n ) assumes the form − d 2 ln | K | + ( m + d +1 2 ) ln | Q | + f 1 ( d, n ) , wher e f 1 ( d, n ) dep ends only on d and n . Pr o of. Let v ec ( A ) denote a v etor of dm dimensions where the ( d ( i − 1) + 1) th , . . . , ( id ) th ( 1 ≤ i ≤ m ) elemen ts of this v etor are equal to the elemen ts of the i th olumn of matrix A ∈ R d × m in the appropriate order. Let ⊗ denote the Krone k er-pro dut. It is kno wn that for P ( A ) = N A ( M , V , K ) , P ( v ec ( A )) = N vec ( A ) ( v ec ( M ) , V ⊗ K − 1 ) holds [26 ℄. Using the w ell- kno wn form ula for the en trop y of a m ultiv ariate and normally distributed v ariable [27 ℄ and applying the relation | V ⊗ K − 1 | = | V | m / | K | d , w e ha v e that H ( A ; V ) = 1 2 ln | V ⊗ K − 1 | + dm 2 ln(2 π e ) = m 2 ln | V | − d 2 ln | K | + dm 2 ln(2 π e ) . Exploit ertain prop erties of the Wishart distribution, w e ompute the en trop y of distribution I W V ( Q , n ) . The densit y of the Wishart distribution is dened b y W V ( Q , n ) = 1 Z n,d | V | ( n − d − 1) / 2 Q − 1 2 n/ 2 exp − 1 2 tr ( VQ − 1 ) . Let Ψ denote the digamma funtion, and let f 2 ( d, n ) = − P d i =1 Ψ ( n +1 − i 2 )) − d ln 2 . Replaing V − 1 with S , w e ha v e for the Jaobian that | d V d S | = | d S − 1 d S | = | S | − ( d +1) [25 ℄. T o pro eed w e use that E W S ( Q ,n ) S = n Q , and E W S ( Q ,n ) ln | S | = ln | Q | − f 2 ( d, n ) , [29 ℄ and substitute them in to E I W V ( Q ,n ) ln | V | , and E I W V ( Q ,n ) tr ( QV − 1 ) : E I W V ( Q ,n ) ln | V | = = Z 1 Z n,d 1 | V | ( d +1) / 2 V − 1 Q 2 n/ 2 exp − 1 2 tr ( V − 1 Q ) ln | V | d V = − Z 1 Z n,d | S | ( d +1) / 2 SQ 2 n/ 2 exp − 1 2 tr ( SQ ) ln | S || S | − d − 1 d S = − Z 1 Z n,d | S | ( n − d − 1) / 2 Q 2 n/ 2 exp − 1 2 tr ( SQ ) ln | S | d S = − E W S ( Q − 1 ,n ) ln | S | = ln | Q | + f 2 ( d, n ) . (13) One an also sho w that E I W V ( Q ,n ) tr ( QV − 1 ) = = Z 1 Z n,d 1 | V | ( d +1) / 2 V − 1 Q 2 n/ 2 exp − 1 2 tr ( V − 1 Q ) tr ( QV − 1 ) d V = Z 1 Z n,d | S | ( d +1) / 2 SQ 2 n/ 2 exp − 1 2 tr ( SQ ) tr ( QS ) | S | − d − 1 d S , = Z 1 Z n,d | S | ( n − d − 1) / 2 Q 2 n/ 2 exp − 1 2 tr ( SQ ) tr ( QS ) d S , = E W S ( Q − 1 ,n ) tr ( QS ) , = tr ( Q Q − 1 n ) = nd (14) W e alulate the en trop y of sto hasti v ariable V with distribution I W V ( Q , n ) . It follo ws from Eq. (13) and Eq. (14) that H ( V ) = = − E I W V ( Q ,n ) − ln( Z n,d ) + n 2 ln Q 2 − n + d + 1 2 ln | V | − 1 2 tr ( V − 1 Q ) = ln( Z n,d ) − n 2 ln Q 2 + n + d + 1 2 " ln | Q | − d X i =1 Ψ ( n + 1 − i 2 )) − d ln 2 # + nd 2 = d + 1 2 ln | Q | + f 3 ( d, n ) , where f 3 ( d, n ) dep ends only on d and n . Giv en the results ab o v e, w e omplete the omputation of en trop y H ( A , V ) as follo ws: H ( A , V ) = H ( A | V ) + H ( V ) = H ( V ) + Z d V I W V ( Q , n ) H ( A ; V ) = Z d V I W V ( Q , n ) m 2 ln | V | − d 2 ln | K | + dm 2 ln(2 π e ) + H ( V ) = − d 2 ln | K | + dm 2 ln(2 π e ) + m 2 [ln | Q | + f 2 ( d, n )] + d + 1 2 ln | Q | + f 3 ( d, n ) = − d 2 ln | K | + ( m + d + 1 2 ) ln | Q | + f 1 ( d, n ) . This is exatly what w as laimed in Lemma 42. Lemmas 41 and 42 lead to the follo wing: Corrolary 43 F or the entr opy of a sto hasti variable with p osterior distribution P ( A , V | x , y ) it holds that H ( A , V ; x , y ) = − d 2 ln | xx T + K | + f 1 ( d, n ) + ( m + d + 1 2 ) ln | Q + ( y − Mx ) γ ( y − M x ) T | . W e note that the follo wing lemma also holds: Lemma 44 Z T y ( Q , n, µ , γ ) ln | Q + ( y − µ ) γ ( y − µ ) T | d y is indep endent fr om b oth µ and γ , and th us w e an ompute the onditional en trop y expressed in (8) : Lemma 45 H ( A , V | y ; x ) = Z p ( y | x ) H ( A , V ; x , y ) d y = − d 2 ln | xx T + K | + g 1 ( Q , d, n ) . wher e g 1 ( Q , d, n ) dep ends only on Q , d and n . Colleting all the terms, w e arriv e at the follo wing intriguingly simple expression u t +1 opt = ar g min u t +1 Z p y t +1 |{ x } t +1 1 , { y } t 1 H ( A , V |{ x } t +1 1 , { y } t 1 , y t +1 ) d y t +1 = ar g min u t +1 − d 2 ln | x t +1 x T t +1 + K t | = arg max u t +1 x T t +1 K − 1 t x t +1 , (15) where x t +1 . = [ r t − I ; . . . ; r t ; u t − J +1 ; . . . ; u t +1 ] , and w e used that | xx T + K | = | K | (1 + x T K − 1 x ) aording to the Matrix Determinan t Lemma [30 ℄. W e assume a b ounded domain U for the on trol, whi h is neessary to k eep the max- imization pro edure of (15) nite. This is, ho w ev er, a reasonable ondition for all pratial appliations. So, u t +1 opt = ar g max u ∈U x T t +1 K − 1 t x t +1 , (16) In what follo ws D-optimal on trol will b e referred to as `infomax interr o gation sheme' . The steps of our algorithm are summarized in T able 1. T able 1: Pseudo o de of the algorithm Con trol Calulation u t +1 = arg max u ∈U ˆ x T t +1 K − 1 t ˆ x t +1 where ˆ x t +1 = [ r t − I ; . . . ; r t ; u t − J +1 ; . . . ; u t ; u ] set x t +1 = [ r t − I ; . . . ; r t ; u t − J +1 ; . . . ; u t ; u t + 1 ] Observ ation observ e r t +1 , and let y t +1 = g − 1 ( r t +1 ) Ba y esian up date M t +1 = ( M t K t + y t +1 x T t +1 )( x t +1 x T t +1 + K t ) − 1 K t +1 = x t +1 x T t +1 + K t n t +1 = n t + 1 γ t +1 = 1 − x T t +1 ( x t +1 x T t +1 + K t ) − 1 x t +1 Q t +1 = Q t + ( y t +1 − M t x t +1 ) γ t +1 ( y t +1 − M t x t +1 ) T Computation of the in v erse ( x t +1 x T t +1 + K t ) − 1 in T able 1 an b e simplied onsiderably b y the follo wing reursion: let P t = K − 1 t , then aording to the Sherman-Morrison form ula [31 ℄ P t +1 = ( x t +1 x T t +1 + K t ) − 1 = P t − P t x t +1 x T t +1 P t 1 + x T t +1 P t x t +1 , (17) In this expression matrix in v ersion disapp ears and instead only a real n um b er is in v erted. 5 Estimating the Noise One migh t wish to ompute the optimal on trol for estimating noise e t in (1), instead of the iden tiation problem ab o v e. Based on (1) and b eause e t +1 = y t +1 − I X i =0 F i r t − i − J X j =0 B j u t +1 − j , (18) one migh t think that the b est strategy is to use the optimal infomax on trol of T able 1, sine it pro vides go o d estimations for parameters A = [ F I , . . . , F 0 , B J , . . . , B 0 ] and so for noise e t . Anotherand dieren tthough t is the follo wing. A t time t , let us denote our estimations as ˆ e t , ˆ F t i (i=0,. . . ,I), and ˆ B t j (j=0,. . . ,J). Using (18) , w e ha v e that e t +1 − ˆ e t +1 = I X i =0 ( F i − ˆ F t i ) r t − i + J X j =0 ( B j − ˆ B t j ) u t +1 − j . (19) This hin ts that the on trol should b e u t = 0 for all times in order to get rid of the error on tribution of matrix B j in (19). Straigh tforw ard utilization of D-optimalit y onsiderations, opp osed to the ob jetiv e of (7), suggests the optimization of the follo wing quan tit y: arg max u t +1 I ( e t +1 , y t +1 ; { x } t +1 1 , { y } t 1 ) , That is, for the estimation of the noise w e w an t to design a on trol signal u t +1 su h that the next output is the b est from the p oin t of view of greedy optimization of m utual information b et w een the next output y t +1 and the noise e t +1 . It is easy to sho w that this task is equiv alen t to the follo wing optimization problem: arg min u t +1 Z d y t +1 P ( y t +1 |{ x } t +1 1 , { y } t 1 ) H ( e t +1 ; { x } t +1 1 , { y } t +1 1 ) , (20) where H ( e t +1 ; { x } t +1 1 , { y } t +1 1 ) = H ( Ax t +1 ; { x } t +1 1 , { y } t +1 1 ) , b eause e t +1 = y t +1 − Ax t +1 . T o ompute this quan tit y w e need the follo wing lemma [26 ℄: Lemma 51 If P ( A ) = N A ( M , V , K ) , then P ( Ax ) = N Ax Mx , V , x T K − 1 x − 1 Applying this lemma and using (11) one has that P ( Ax t +1 | V , { x } t +1 1 , { y } t 1 ) = N Ax t +1 M t +1 x t +1 , V , x T t +1 K − 1 t +1 x t +1 − 1 (21) W e in tro due the notations ˜ K t +1 = x T t +1 K − 1 t +1 x t +1 − 1 ∈ R , (22) λ t +1 = 1 + ( Ax t +1 − M t +1 x t +1 ) T ( ˜ K t +1 Q − 1 t +1 )( Ax t +1 − M t +1 x t +1 ) ∈ R and use (9) and (12) for the p osterior distribution (21). Then w e arriv e at P ( Ax t +1 |{ x } t +1 1 , { y } t 1 ) = T Ax t +1 Q t +1 , n t +1 , M t +1 x t +1 , ˜ K t +1 = π − d/ 2 | ˜ K − 1 t +1 Q t +1 | − 1 / 2 Γ ( n t +1 +1 2 ) Γ ( n t +1 +1 − d 2 ) λ n t +1 +1 2 t +1 The Shannon-en trop y of this distribution aording to [32℄ equals: H ( Ax t +1 ; { x } t +1 1 , { y } t +1 1 ) = f 4 ( d, n t +1 ) + d 2 log | ˜ K − 1 t +1 | + log | Q t +1 | where f 4 ( d, n t +1 ) = − log Γ ( n t +1 +1 2 ) π d/ 2 Γ ( n t +1 +1 − d 2 ) + n t +1 + 1 2 Ψ n t +1 + 1 2 − Ψ n t +1 + 1 − d 2 . Using the notations in tro dued in (10) and in (22) the ab o v e expressions an b e transrib ed as follo ws: H ( Ax t +1 ; { x } t +1 1 , { y } t +1 1 ) = f 4 ( d, n t +1 ) − d 2 log | ˜ K t +1 | + log | Q t +1 | = f 4 ( d, n t +1 ) + d 2 log | x T t +1 ( K t + x t +1 x T t +1 ) − 1 x t +1 | + log | Q t +1 | = f 4 ( d, n t +1 ) + d 2 log | x T t +1 ( K t + x t +1 x T t +1 ) − 1 x t +1 | + + log | Q t + ( y t +1 − M t x t +1 ) γ t +1 ( y t +1 − M t x t +1 ) T | No w, w e are in a p osition to alulate (20) b y applying Lemma 44 as b efore. W e get that Z d y t +1 P ( y t +1 |{ x } t +1 1 , { y } t 1 ) H ( e t +1 ; { x } t +1 1 , { y } t +1 1 ) = = g 2 ( Q t , d, n t +1 ) + d 2 log | x T t +1 ( K t + x t +1 x T t +1 ) − 1 x t +1 | , where g 2 ( Q t , d, n t +1 ) dep ends only on Q t , d and n t +1 . Th us, w e ha v e that arg max u t +1 I ( e t +1 , y t +1 ; { x } t +1 1 , { y } t 1 ) = arg min u t +1 log | x T t +1 ( K t + x t +1 x T t +1 ) − 1 x t +1 | = ar g min u t +1 log x T t +1 K − 1 t − K − 1 t x t +1 x T t +1 K − 1 t 1 + x T t +1 K − 1 t x t +1 ! x t +1 = ar g min u t +1 log x T t +1 K − 1 t x t +1 1 + x T t +1 K − 1 t x t +1 = arg min u t +1 x T t +1 K − 1 t x t +1 In pratie, w e p erform this optimization in an appropriate domain U . Th us, the D-optimal in terrogation s heme for noise estimation is as follo ws u opt t +1 = ar g min u ∈U x T t +1 K − 1 t x t +1 . (23) It is w orth noting that this D-optimal ost funtion for noise estimation and the D-optimal ost funtion deriv ed for parameter estimation in (15) are not ompatible with ea h other. Estimating one of them qui kly will neessarily dela y the estimation of the other. In Setion 5.1 w e see that for large enough t v alues, expression (23) giv es rise to on trol v alues lose to u t = 0 . 5.1 Greedy and Non-Greedy Optimization Greedy optimization of (23) is simple, pro vided that K t is xed during the optimization of u t +1 . If so, then the optimization task is quadrati. T o see this, let us partition matrix K t as follo ws: K t = K 11 t K 12 t K 21 t K 22 t , where K 11 t ∈ R d × d , K 21 t ∈ R m − d × d , K 22 t ∈ R m − d × m − d . It is easy to see that if domain U in (23) is large enough then u opt t +1 = ( K 22 t ) − 1 K 21 t r t . (24) Ho w ev er, for non-greedy solutions, expression K t in x T t +1 K − 1 t x t +1 hanges, b eause it ma y dep end on previous on trol inputs u 1 , . . . , u t , the sub jet of previous optimization steps. The optimal strategy for long-term non-greedy optimization falls outside of the sop e of the presen t w ork. Here w e prop ose the follo wing heuristis for this problem: Use the strategy of T able 1 for the rst τ steps. It inreases | K t | qui kly in (23). Then after τ -steps swit h to the on trol desrib ed in (24) . This will derease the ost funtion (23) further. W e will all this non-greedy in terrogation heuristis in tro dued for noise estimation ` τ -infomax noise interr o gation ' . It is w orth noting that in the τ -infomax noise in terrogation, if the τ swit hing time is large enough then for large t v alues | K 22 t | will b e large, and hene aording to (24) the optimal u t in terrogation will b e lose to 0 . The appro ximation of the ` τ -infomax noise in terrogation' when w e use the in terrogation desrib ed in T able 1 for τ steps and then swit h to zer o-interr o gation will b e alled the ` τ -zer o interr o gation ' s heme. 6 Disussion and Conlusions W e ha v e treated the iden tiation problem of reurren t neural net w orks desrib ed b y mo del (1). W e applied ativ e learning to solv e this task. In partiular, the online D-optimalit y priniple w as applied and w e in v estigated the learning prop erties for parameter and noise estimations. W e note that the D-optimal in terrogation s heme is also alled infomax on trol in the literature [6℄. This name originates from the ost funtion that optimizes the m utual information. The GLM mo del used b y [6 ℄ is as follo ws: r t +1 = g I X i =0 F i r t − i + J X j =0 B j u t +1 − j + e t +1 , (25) where { e t } is i.i.d. noise with 0 ∈ R d mean. The authors mo del spiking neurons and assume that the main soure of the noise is this spiking, whi h app ears at the output of the neurons and adds linearly to the neural ativit y . They in v estigated the ase in whi h the observ ed quan tit y r t had a P oisson distribution. Unfortunately , in this mo del Ba y esian equations b eome in tratable and the estimation of the p osterior ma y b e sp oiled, b eause the distribution is pro jeted to the family of normal distributions at ea h instan t. A serious problem with this approa h is that the exten t of the information loss aused b y this appro ximation is not kno wn. Our sto hasti RNN mo del r t +1 = g I X i =0 F i r t − i + J X j =0 B j u t +1 − j + e t +1 , diers only sligh tly from the GLM mo del of (25) , but it has onsiderable adv an tages, as w e shall disuss b elo w. Ba y esian designs of dieren t kinds w ere deriv ed for the linear regression problem in [35 ℄: y = X θ + e (26) P ( e ) = N e (0 , σ 2 I ) . (27) This problem is similar to ours ((3)-(6)), but while the goal of [35 ℄ w as to nd an optimal design for the explanatory v ariables θ , w e w ere onerned with the parameter ( X in (26)) and the noise ( e ) estimation task. In V erdinelli's pap er in v erted gamma prior and v etor-v alued normal distribution w ere assumed on the isotropi noise and on the explanatory v ariables, resp etiv ely . By on trast, w e w ere in terested in the matrix-v alued o eien ts and in general, non-isotropi noises. W e used matrix-v alued normal distribution for the o eien ts and in v erted Wishart distribution for the o v ariane matrix as onjugate priors. Due to the in v erted Wishart distribution that w e used, the o v ariane matrix of the noise is not restrited to the isotropi form, but an b e general in our ase. The Ba y esian online learning framew ork allo w ed us to deriv e analyti results for the greedy optimization of the parameters as w ell as the driving noise. Optimal in terrogation strategies (16) and (23) app eared in attrativ e, in triguingly simple quadrati forms. W e ha v e sho wn that these t w o tasks are inompatible with ea h other. P arameter and noise estimations require the maximization and the minimization of expression x T t +1 K − 1 t x t +1 , resp etiv ely . The problem of non-greedy optimization of the full task has b een left op en. Ho w ev er, w e put forth a heuristi solution for the estimation of the driving noise that w e alled τ -infomax noise in terrogation. It uses the D-optimal in terrogation of T able 1 up to τ -steps, and applies the noise estimation on trol of (23) afterw ards. This heuristis dereases the estimation error of the o eien ts of matries F and B up to time τ and th us up on turning o the explorativ e D-optimization tries to minimize the estimation error of the v alue of the noise at time τ + 1 . W e in tro dued the τ -zero in terrogation s heme and sho w ed that it is a go o d appro ximation of the τ -infomax noise s heme for large τ v alues. Finally , it seems desirable to determine the onditions under whi h the algorithms deriv ed from the D-optimal (infomax) priniple are b oth onsisten t and eien t. The tratable form of our appro ximation-free results is promising in this resp et. 7 A kno wledgmen ts This resear h has b een supp orted b y the EC NEST `P ereptual Consiousness: Expliation and T esting' gran t under on trat 043261. Opinions and errors in this man usript are the author's resp onsibilit y , they do not neessarily reet the opinions of the EC or other pro jet mem b ers. Referenes 1. F edoro v, V.V.: Theory of Optimal Exp erimen ts. A ademi Press, New Y ork (1972) 2. Cohn, D.A.: Neural net w ork exploration using optimal exp erimen t design. In: A dv anes in Neural Information Pro essing Systems. V olume 6. (1994) 679686 3. deCharms, R.C., Blak e, D.T., Merzeni h, M.M.: Optimizing sound features for ortial neurons. Siene 280 (1998) 14391444 4. Földiák, P .: Stim ulus optimization in primary visual ortex. Neuro omputing 3840 (2001) 1217 1222 5. Ma hens, C.K., Gollis h, T., K olesnik o v a, O., Herz, A.V.M.: T esting the eieny of sensory o ding with optimal stim ulus ensem bles. Neuron 47 (2005) 447456 6. Lewi, J., Butera, R., P aninski, L.: Real-time adaptiv e information-theoreti optimization of neu- roph ysiology exp erimen ts. In: A dv anes in Neural Information Pro essing Systems. V olume 19. (2007) 7. MaKa y , D.J.C.: Information-based ob jetiv e funtions for ativ e data seletion. Neural Compu- tation 4 (1992) 590604 8. Cohn, D.A., Ghahramani, Z., Jordan, M.I.: A tiv e learning with statistial mo dels. Journal of Artiial In telligene Resear h 4 (1996) 129145 9. F ukumizu, K.: Statistial ativ e learning in m ultila y er p ereptrons. IEEE T ransations on Neural Net w orks 11 (2000) 1726 10. Sugiy ama, M.: A tiv e learning in appro ximately linear regression based on onditional exp etation of generalization error. The Journal of Ma hine Learning Resear h 7 (2006) 141166 11. Opp er, M., Win ther, O.: A Ba y esian approa h to online learning. In: Online Learning in Neural Net w orks. Cam bridge Univ ersit y Press (1999) 12. Solla, S., Win ther, O.: Optimal p ereptron learning: An online Ba y esian approa h. In: Online Learning in Neural Net w orks. Cam bridge Univ ersit y Press (1999) 13. Honk ela, A., V alp ola, H.: On-line v ariational Ba y esian learning. In: 4th In ternational Symp osium on Indep enden t Comp onen t Analysis and Blind Signal Separation. (2003) 803808 14. Ghahramani, Z.: Online v ariational Ba y esian learning (2000) Slides from talk presen ted at NIPS 2000 w orkshop on Online Learning. 15. Kiefer, J.: Optim um exp erimen tal designs. Journal of the Ro y al Statistial So iet y , Series B 21 (1959) 272304 16. Stein b erg, D.M., Hun ter, W.: Exp erimen tal design: review and ommen t. T e hnometrixs 26 (1984) 7197 17. T oman, B., Gast wirth, J.L.: Robust ba y esian exp erimen tal design and estimation for analysis of v ariane mo dels using a lass of normal mixtures. Journal of statistial planning and inferene 35 (1993) 383398 18. Puk elsheim, F.: Optimal Design of Exp erimen ts. John Wiley & Sons (1993) 19. Chaloner, K., V erdinelli, I.: Ba y esian exp erimen tal design: A review. Statist. Si. 10 (1995) 273304 20. Bernardo, J.M.: Exp eted information as exp eted utilit y . The Annals of Statistis 7 (1979) 686690 21. Stone, M.: Appliation of a measure of information to the design and omparison of regression exp erimen ts. Ann. Math. Statist 30 (1959) 5570 22. Bo y en, X., K oller, D.: T ratable inferene for omplex sto hasti pro esses. In: F ourteen th Con- ferene on Unertain t y in Artiial In telligene. (1998) 3342 23. Mink a, T.: A family of algorithms for appro ximate Ba y esian inferene. PhD thesis, MIT Media Lab, MIT (2001) 24. Gelman, A., Carlin, J.B., Stern, H.S., Rubin, D.B.: Ba y esian Data Analysis. CR C Press, 2nd erdition (2003) 25. Gupta, A.K., Nagar, D.K.: Matrix V ariate Distributions. V olume 104 of Monographs and Surv eys in Pure and Applied Mathematis. Chapman and Hall/CR C (1999) 26. Mink a, T.: Ba y esian linear regression (2000) MIT Media Lab note. 27. Co v er, T.M., Thomas, J.A.: Elemen ts of Information Theory . Wiley-In tersiene (1991) 28. K otz, S., Nadara jah, S.: Multiv ariate T-Distributions and Their Appliations. Cam bridge Univ er- sit y Press (2004) 29. Beal, M.J.: V ariational algorithms for appro ximate Ba y esian inferene. PhD thesis, Gatsb y Com- putational Neurosiene Unit, Univ ersit y College London (2003) 30. Harville, D.A.: Matrix Algebra F rom a Statistiian's P ersp etiv e. Springer-V erlag (1997) 31. Golub, G.H., V an Loan, C.F.: Matrix Computations. 3rd ed. edn. Johns Hopkins, Baltimore, MD (1996) 32. Zografos, K., Nadara jah, S.: Expressions for Rén yi and Shannon en tropies for m ultiv ariate distri- butions. Statistis and Probabilit y Letters 71 (2005) 7184 33. Y amakita, M., Iw ashiro, M., Sugahara, Y., F uruta, K.: Robust swing-up on trol of double p endulum (1995) 34. Gäfv ert, M.: Mo delling the furuta p endulum. T e hnial rep ort ISRN LUTFD2/TFR T7574SE, Departmen t of Automati Con trol, Lund Univ ersit y , Sw eden (1998) 35. V erdinelli, I.: A note on ba y esian design for the normal linear mo del with unkno wn error v ariane. Biometrik a 87 (2000) 222227
Original Paper
Loading high-quality paper...
Comments & Academic Discussion
Loading comments...
Leave a Comment