Tiny noise, big mistakes: Adversarial perturbations induce errors in Brain-Computer Interface spellers

Tin y noise, big mistak es: Adv ersarial p erturbations induce errors in Brain-Computer In terface sp ellers Xiao Zhang 1 , Dongrui W u 1 * , Lieyun Ding 2 * , Han bin Luo 2 , Chin-T eng Lin 3 , Tzyy-Ping Jung 4,5 & Ricardo Cha v a rriaga 6 1 Ministry of Education Key Laboratory of Image Pro cessing and Intelligen t Control, School of Ar tificial In telligence and Automation, Huazhong Universit y of Science and T ec hnology , W uhan 430074, China. 2 Sc ho ol of Civil Engineering and Mechanics, Huazhong Universit y of Science and T echnology , W uhan 430074, China. 3 Cen tre of Ar tificial In telligence, F aculty of Engineering and Inform ation T echno logy , Unive rsi t y of T ec hnology Sydney , Sydney 2007, Australia. 4 Sw artz Center for Computational Neuroscience, Institute for Neural Computation, University of California San Diego (UCSD), La Jolla, CA 92093, U SA. 5 Cen ter f or Adv anced Neurological Engineering, Institute of Engineering in Medicine, UCSD, La Jolla, CA 92093, USA. 6 ZHA W DataLab, Z¨ urich Univ ersity of Applied Sciences, Winterth ur 8401, Switzerland. Corr esp onding authors E-mail: drwu@hust.e du.cn, d ly@hust.e du.cn ABSTRACT An ele ctr o enc ephalo gr am (EEG) ba se d br ain-c omputer interfac e (BCI) sp el ler al lows a user t o input te xt to a c omputer by thought. It is p articularly useful to seve rely disable d individuals, e.g., amyotr ophic later al scler osis p atients, who have no other eff e ctive me ans of c ommunic ati on with another p erson or a c omputer. Most studies so far fo cuse d on making EEG-b ase d BCI sp e l lers faster and mor e r eliable; however, few have c onsider e d their se curity. This study, for the first time, shows that P300 and ste ady-state visual evoke d p otential BCI sp e l lers ar e ve ry vulner able, i.e., they ca n b e sever ely attacke d by adversarial p erturb ations, which ar e to o t iny to b e notic e d when adde d to EEG signals, b ut c an misle ad the sp el lers to sp el l anything the attacker wants. The c onse quenc e c ould r ange f ro m mer ely user frustr ation to se v er e misdiagnosis in clinic al applic ations. We hop e our r ese ar ch c an attr act mor e att ention to t he se curity of EEG-b ase d BCI sp el lers, and mor e br o ad ly, EEG-b ase d BCIs, which has r e c eive d little attention b efor e. Keywords El ectroencephalogram, brain-computer interfaces, BCI sp el l ers, adve rsar ial examples RESEARCH ARTICLE INFO RM A TION SCIENCE 1 INTR ODUCTION A brain-computer interface (BCI), which ha s b een extensively us ed in neur oscience, ne ur al engineering and clinical rehabilitation, o ffers a communication pathw ay that allows p eople to in terac t with computers using bra in sig nals directly [1–4]. There are man y approaches to collecting signals from the brain. Electro encephalo gram (EEG), usually measured from the sc a lp, may be the most p opular one due to its simplicity and low cost [5]. An EEG-based BCI sp eller a llows a user to input text to a co mputer by thought [6, 7 ]. It enables p eo ple with severe disabilities, e.g., amy otrophic la teral sc le rosis (ALS) patients, to communicate with computers or other p eople. The t wo main types o f EEG-bas e d BCI speller s are P3 00 sp ellers [6] a nd stea dy -state visual evoked p otential (SSVEP) sp ellers [7], which elicit differ ent EEG patterns, a s illustr ated in Figure 1a. A P 300 spelle r, which uses P300 evok ed po ten tials as its input signal [8], was firs t inv ented by F arwell and Donchin in 198 8 [6] and further developed by many o thers [9 – 12]. P30 0 is a p os itive deflection in voltage, typically app ear ing around 250 to 500 ms after a rare target stim ulus o c c ur s [13]. It is an endoge no us potential link ed to p eople’s cog nitive pro cesses, such as information pro cessing and decision making [14, 15]. The standard o ddball par a digm is usually used to elicit P3 00, in which ra re tar get s tim uli are mixed with high-pro bability non- ta rget one s . The P300 s pelle r considered in this article uses a 6 × 6 character ma trix, which c o nsists of 26 letters a nd 10 other symbols, a s shown in Figure 1 b. The us e r star es a t the character he/she w ants to input, while a r ow or column is rapidly intensified sequentially . The cor resp onding EE G signa ls are r ecorded and class ified as a target (containing P 300) or non-tar get (not containing P300 ) for each intensification. Then, the co mputer iden tifies the character at the intersection of the target row and the tar get column, w hich elicit the lar gest P300 s, as the o utput. F or re liable p erforma nce, each row and column may hav e to b e intensified multiple times, which reduces the sp eed of the P300 sp eller. Compared with the P3 00 sp eller, an SSVEP sp eller has the adv antages of high information transfer rate (ITR), little user training, a nd so me immunit y to artifacts [16–18]. When the user sta res at a visual target flick ering at a sp ecific frequency , usually b etw een 3.5 Hz and 75 Hz, electrical signals of the sa me freq uency , as w ell as its corres p o nding harmonics, can b e o bserved from the EEG signa ls [16]. In a n SSVEP sp eller, the pictures of different characters ar e flick ering a t different frequencies, so that a clas s ifier can directly iden tify the output character fro m a large num b er of ca ndida tes by matching their flick ering frequencies with the user ’s EE G oscilla tion frequency . Since all characters in an SSVEP speller are flick ering s imultaneously (in contrast to sequential intensification in a P30 0 sp eller), they can have muc h higher ITRs. The SSVEP sp eller considere d in this study has 40 characters (Figure 1 c), whose stimulation frequencies ar e from 8 Hz to 15 .8 Hz with 0 .2 Hz incr ement [19]. Machine learning is used in BCI sp ellers to construct the classifiers to detect the bra in r esp onses to s tim uli (i.e. the P300 or SSVEPs patterns). Most studies so far fo cused on making the BCI c la ssifiers faster and more reliable; how ever, few have co nsidered their secur ity . It has b een found in other application domains that a dversarial examples [20], which are nor mal exa mples contaminated by delib erately designed tiny pe rturbations, can eas ily fo o l machine le a rning mo dels. These pertur bations ar e usually so small that they are indistinguishable to human eyes. Existing studies on adversarial exa mples fo cused la rgely on deep learning mo dels for computer visio n. F or example, it was found that a picture of a panda, after a dding a weak a dversarial p erturba tio n, can b e misclass ified as a gibb on by a dee p learning classifier [2 1]. Kurakin et al. [22] found that pr int ed pho tos of a dversarial exa mples can degr a de the p erfor ma nce of an Imag eNet Inception cla ssifier. Athaly e et a l. [23] 3 D printed a turtle with an adversarial texture, whic h w as classified as a riffle from almost ev ery viewp oint. Recen tly , adv ersa rial examples w ere also found in traditional machine-learning mo dels [2 4] and many other application domains, e.g., sp eech reco gnition [25], text cla s sification [2 6], ma lware identification [2 7], etc. Due to the hig h r is k of a dversar ial a ttacks, many defense mechanisms hav e b een prop osed, such as defensive distillation [28], adversarial tra ining [21, 29, 30], and so on [31 – 33]. How ever, these appr oaches only improve empirical adversarial robus tnes s , which is not certified and may be bro ken by a stronger attack approach [34, 35]. Recently , res earchers sta rted to inv estigate prov able guarantees of adversaria l robustness, yet there is s till a huge gap b etw een certified ro bus tness and e mpirical robustness [36 – 39]. This a rticle aims to e xp o se a critica l sec ur ity concer n in EEG-base d BCI sp ellers , a nd mor e broadly , E E G-based BCIs, which has r eceived little atten tion b efore. It shows for the first time that one can gener a te tiny a dversarial EEG p erturbation templates for target attacks for b oth P300 and SSVEP sp ellers, i.e., mis le ad the clas sification to any character the attacker wan ts, rega rdless o f what the user intended character is. The conse q uence co uld range from merely user frustration to severe misdia gnosis in clinical applications [40]. W e b elieve a new and mor e detailed understanding o f how adversarial EEG p ertur bations affect BCI classification can inform the design of BCIs to defend against such attacks. There hav e b e en some s tudies on adversar ial attacks of time-series signals [25, 40–42]. They treated time-series signals just as images, and then a pplied ess ent ially the same attack a pproaches in image classificatio n to gener ate adversarial per tur bations. As a result, they need to know the full time-series b efore computing the adversarial p ertur- bations, which means these a pproaches are not caus a l a nd hence cannot b e implemented in real-world a pplications. F or example, to attack a voice command, pr e vious approaches need to reco rd the entire v oice command first, and then design the p erturbation. How ever, once the p erturbatio n is o btained, the voice command has alrea dy be e n sent out (e.g., to a smartphone or Amazon E cho), so there is no chance to a dd the pe rturbation to the voice command to actually p erform the attack. What distinguis he s the attack a pproaches in this article mo st fro m pre vious ones is that it explicitly co nsiders the ca usality in designing the p e rturbations. The adversarial p erturbation template is co nstructed dire c tly fro m the training set and then fixed. So, there is no need to know the test EEG trial and co mpute the p er turbation spe cifically for it. The p erturba tion can b e directly added to a test EEG tr ial as so on a s it starts, hence sa tisfies causa lit y and can be implemen ted in practice. Thus, it calls for an urg ent need to be a ware of such attacks a nd defend against them. A clos ely rela ted concept is universal adversaria l p erturba tions [43], whic h can also be viewed as a dversarial per turbation templates and hav e b een used to attack deep learning mo dels in image classification. This s tudy fo cuses on the secur it y of a traditional and most frequen tly used BCI pip eline, which consists of separ a te feature 2 extraction and class ification steps, wherea s universal adversarial p ertur bations are usually designed for non-ta rget attacks of end-to-end deep lea rning mo dels . T o summarize, our contributions ar e: 1. W e show, for the first time, tha t tiny noise can significantly manipulate the outputs o f P300 a nd SSVEP sp ellers, exp osing a cr itica l s ecurity co ncern in BCIs. 2. Instead of Deep Learning models , we cons ide r the classica l BCI pip eline consisting of feature extractio n and classification as our victim mo dels , which dominate practical BCI sp eller s. 3. Our gener ated adversarial p e r turbation templates satisfy the causality of time-s e r ies sig nals, which r arely drew m uch attention b efore. 2 RESUL TS 2.1 P erformance ev aluation W e used tw o measures to e v aluate the perfo rmance of a BCI sp eller : The classifica tion a ccuracy and the ITR [44], which measure s the typing sp eed of the sp eller: ITR = 1 T  log 2 Q + R log 2 R + (1 − R ) log 2 1 − R Q − 1  , (1) where T is the average time (minutes) sp ent to input a user character, Q the num b er of differ ent characters (which was 36 in our P3 00 sp eller a nd 40 in the SSVEP s p eller ), and R the classificatio n accura cy . The unit of ITR is bits/min. When the classification accura cy is lower than a rando m guess, i.e., R 6 1 Q , the ITR is directly set to 0. T o distinguish b etw een the character the user wan ts to spell, and the one the a ttack er w ants to mislea d to, we denote the former u ser char acter , and the latter att acker char acter . Accor dingly , user sc or e and user ITR a re used to de s crib e the cla ssification accuracy o f user characters and the corr esp onding ITR, resp ectively . An attacker sc or e is defined as the r atio that the p erturbation template leads the spe ller to output an attack er c harac ter, and the corres p o nding att acker ITR is calculated by replacing R in equation (1) with the attacker scor e . A higher attacker score or attack er ITR represents a b etter ta rget a ttack p erformance. 2.2 Securit y of the P300 s p ell e r Dataset : W e used a public P 300 datase t (datas et I I) intro duced by W olpaw et al. [45]. It reco rded 64-channel EEG signals from tw o sub jects (A a nd B). The EE G data were sampled at 240 Hz, bandpas s filter e d to 0.1- 40 Hz, then z -no r malized for ea ch channel. There were 8 5 training character tria ls and 100 test ones for ea ch sub ject. F or each trial, a set of 12 r a ndom int ensifica tio ns (six r ows and six columns) were rep eated 15 times (i.e., each r ow was int ensified 15 times, and ea ch column w as also in tensified 15 times). E ach intensification lasted for 100 ms, after which the character matrix was blanked for 75 ms. So, it to ok (10 0 + 75 ) × 1 2 × 15 = 31 , 500 ms, or 3 1.5 s, to input a character. The sp elling s pee d can b e improved by using fewer r ep eats, e.g., 1 0 or 5; how ever, the sp elling a ccuracy generally decreas e s with a smalle r nu mber of r ep eats. Note that a ll the following exp eriments were also succe ssfully p erfor med o n a public ALS P 300 dataset with eight ALS patients (see Supplementary Infor mation for the details). The victim mo de l : The victim mo del was a Riemannian geometry based appro ach, which won the Kag gle BCI challenge 1 in 20 15. Firs t, 16 xDA WN spatial filters [46], eight fo r the targ et trials and ano ther eight for the non-targ e t 1 h ttps://www.k aggle.com/c/inria-bci- c hallenge 3 trials, were des igned to filter a ll the tria ls. The template-sig nal cov a r iance matrices o f the EE G ep o chs were pro jected onto the tangent space of a Riemannian manifold [47–49], using Affine Inv a riant Riemannian Metric as its distanc e metric. Finally , we classified the feature vectors with a Lo gistic Regress ion mo del in the tangent space. The details can b e fo und in the Supplementary Informa tio n. The mo del was trained with class- s pe c ific weights to ac c ommo date class imbalance. All op er ations in these blo cks a re differentiable, so we re-implemented them using T enso rflow [50] to fa cilitate the gr adient calcula tion. T o get the lab el (tar get or non-targ e t) of an intensification, an epo ch b etw een 0-600 ms fr o m the b eginning of the int ensifica tio n was extracted and fed into the victim mo del to ca lculate the tar g et probability . B ecause ea ch r ow and column was intensified m ultiple times, voting was p erformed for e ach trial to ge t the target row and targ et co lumn, and hence the targ et character. Baseline p erformance : The fir s t par t of T able 1 shows the bas eline p erfor mance of the clean E EG data (without adding any p erturbations). As the n umber of intensification rep eats increased, the user score increased, indicating that the c la ssification accuracy o f the user c haracter s increa sed. Mea nwhile, the user ITR decreased, b ecause the time needed to input e a ch character significantly increa sed. The s econd par t o f T able 1 shows the baseline p erfor ma nce when we added Gaussian noise to the raw EEG da ta, av eraged ov er 10 r uns . The Gaussia n noise p erturba tions were prepro cesse d in the s ame wa y as the adversarial per turbations, by r eplacing the p erturbation – e P in eq ua tion (6)– with standard Gauss ian noise , so that they had the same ener gy . W e use signal- to-p erturbation ratio (SPR) to qua ntif y the ma gnitude of the p ertur bation, which is also presented in the s econd part of T a ble 1. Gaussia n nois e p er tur bations had almost no impact on the user sc o re and the user ITR at all, not to men tion forcing the P 300 speller to output a s pe cific attack er character. These results suggest that more sophistica ted a dversarial p erturbations ar e needed to a ttack the P30 0 sp eller. P erformance under adv ersarial attac ks : Then, we added the adversarial p erturbatio n template to the test EEG tr ials to v alidate w he ther it was effective in misleading the P300 sp eller . Figur e 2 a shows the attack er scor es of the 36 characters. The a ttack er c a n manipulate the P300 sp eller to s p ell whatever character he/she wan ts, reg ardless of what the user intended character is, with a higher than 9 0% av erage s uccess r ate. The third part of T a ble 1 shows the av erag e user sco r es a nd ITRs with different n umbers o f intensification r epe ats. The user scor es and ITRs w ere close to zero , sug gesting that the user a lmost ca nnot co rrectly input the character he/she wan ted. The four th part of T able 1 shows the a verage a ttack er scores and ITRs with different num b ers of intensification rep eats. The attacker score increased with the num b er of intensification r e pea ts, b ecause mor e rep ea ts incr eased the nu mber of times that the attacker can inject the p erturbation into the b enign EEG trial. T o b etter quantify the ma g nitude of the p ertur bations, we also calc ulated t wo SPRs. The adversarial p e rturbation template w as only added at some sp ecific p erio ds of the E EG tr ial, as shown in Figure 2b, therefore we defined a p erio d SPR to measur e the SPR o f the p er turb e d p erio d, and also a trial SPR to mea sure the SP R of the en tire trial. The last part of T able 1 shows thes e SP Rs . They w ere higher than 20 dB, suggesting that the a dversarial per turbation template may b e undetectable when added to b enign E EG trials. Visualization of the adversar ial p erturbations : In a ddition to hig h a ttack per formance, ano ther r equirement in adversarial attacks is that the pe r turbations should not b e detected easily . Figure 2b shows a typical EE G trial b efor e and after the adversarial p erturba tion on Sub ject A. F or clarity , we only show c hannels F3, F4, Cz, P3 a nd P4, whic h evenly distribute on the scalp. O ne can barely distinguish the adversarial EEG trial fro m the origina l EEG trial. A traditional wa y to visualize the P3 00 signal is to take the average of multiple P300 trials. W e also to ok this approach to chec k if ther e w as a noticeable difference b etw een the average target (or no n-target) trials, befor e and after p erturba tion. Figure 2c shows the results fro m the C z channel. One ca n hardly obs erve any differences. 4 Figure 2c also s hows the sp ectro grams and top oplots of the difference b etw een the av erage targ et E EG tria l a nd the av erage non-target EE G trial. The original and adv ers a rial sp ectr ogra ms (or to po plots) show v ery simila r energ y distributions, and ar e ha rdly distinguis hable by human eyes. 2.3 Securit y of the SSVEP Sp eller Dataset : The dataset was fir s t intro duce d by W ang et al. [19] as a b enchmark datas et for SSVEP-ba s ed B C Is . The 64-channel signals were recor ded from 35 sub jects using an ex tended 10 -20 system. During the exper iments, the sub jects were facing a monito r , in which a 5 × 8 character matrix was flick ering. Differen t flick ering frequencies were assigned to the 40 character s r esp ectively , ranging from 8 Hz to 15.8 Hz w ith 0.2 Hz increment, as shown in Figure 1c. Six blo cks of EEG signals were recorded from each sub ject, ea ch with 40 tria ls, co rresp onding to the 4 0 target characters. Each trial was downsampled to 2 50 Hz and lasted 6 seco nds, including 0.5 s b efore stimulus onset, 5 s for stimulation, and 0.5 s after stimulus offset. Chen et al. [51] show ed that an SSVEP at the stimulation frequency a nd its har monics usually star ts to b e e vok ed with a delay aro und 13 0-140 ms; hence, we extr a cted E EG sig na ls be tw een [0.13, 1.38 ] s after the stimulus o nset as the input to the victim mo del. Nine channels over the o cc ipita l and parietal a reas (Pz, PO z, PO3 , PO 4 , PO5, PO6, Oz, O1 and O2) were chosen. The sig na ls were ba ndpass filter e d to 7- 9 0 Hz with a fourth-order Butterworth filter. The victim mo del : Extr acting the frequency infor mation o f SSVEPs is an essential step in recog nizing the stimu latio n frequency , and hence the us er c hara cter. A natural solution is to utilize fast F our ier tra nsform to estimate the sp ectrum, so that the ener gy p eaks can be matched to the stim ulation frequency; howev er, cano nical correla tion a na lysis (CCA) was rec ent ly shown to b e more pr omising in identifying the stimulation frequency [51, 52]. Thu s, CCA-bas e d freq uency recognition was used in the victim mo del. CCA is a statistic a pproach that can b e used to extra c t the under lying corr elation b etw een tw o mult i-channel time series [5 3]. Its ma in idea is to find a linear co mb ination of channels for ea ch time ser ie s, so that their correla tion is maximized. When applied to SSVEP sp ellers , CCA is utilized to calculate the ma ximum cor relation be tw een the input EE G signals and a standard refer ence signal, w hich consists of the sinusoidal signal of a stimulation frequency and its ( N q − 1) har monics ( N q = 5 in our case). Mathematically , let X ∈ R N e × N s denote an EEG trial w ith N e channels and N s samples, and Y f a standar d reference signa l of stimulation freq uency f . The ( c, n )-th ent ry o f Y f is: Y f ( c, n ) =    sin  ( c + 1 ) π f f s n  , c is o dd cos  cπ f f s n  , c is even , (2) where f s is the sampling ra te, 1 6 c 6 2 N q , and 1 6 n 6 N s . T o calculate the maximum c o rrelatio n co e fficient ρ ( X , Y f ), X and Y f are first z -no rmalized, and then ρ ( X , Y f ) is computed a s the squar e ro ot of the la rgest eige nv alue of matrix S ( X , Y f ) = ( X X T ) − 1 X Y T f ( Y f Y T f ) − 1 Y f X T , (3) i.e., ρ ( X , Y f ) = q λ max ( S ( X , Y f )) . (4) More detailed deriv ations can b e found in the Supplemen tar y Info r mation. Let F = { f i } K i =1 be the set of K candida te stimulation frequencies ( K = 40 in our case). Then, the SSVEP sp eller outputs the character cor resp onding to the following stimulation frequency : f ∗ = a rg max f ∈ F ρ ( X , Y f ) . (5) 5 Baseline p erformance : Among the 35 s ub jects, eight with the best baseline p erformanc e s (shown in the first part of T able 2) were used in our exp eriments. Because SSVEPs are highly susceptible to p erio dic nois e, we ev a lua ted the robustness o f the v ictim mo del to Gaussian nois e and s inu so ida l noise of a random s ingle frequency chosen from 40 stimulation frequencies, and a random phase chosen from − π 2 to π 2 . W e a lso co nsidered comp ound sin usoida l noise, which can b e re garded a s the summation of single sinusoidal noise of different frequencies, r andom amplitudes, and random phases . The SPRs were all set to 25 dB, so that the energ y of the Gaussia n noise and single/co mpo und (S/C) p er io dic noise was co mparable to that of the adversaria l p erturbatio n templates. The ‘Gaussia n No ise’ a nd ‘S/C Perio dic Nois e’ panels o f T able 2 show the r esults on these noisy data, av erage d ov er 1 0 runs, res pe c tively . The victim mo del was almos t completely imm une to the Gaussian noise. The sing le p erio dic noise degra ded the mo del p er formance mor e than the Gaussia n noise or comp ound p erio dic noise. P erformance unde r adv ersarial attac ks : W e gener ated 40 adversaria l pe rturbation templates, each for cing the SSVEP sp eller to o utput a sp ecific character. Fig ure 3a shows their attack er s cores. F o r s ix of the eight sub jects, their output character ca n b e manipulated to any character the attack er wan ted, at 70 %-100% succes s rate. Interestingly , due to individual differences , Sub jects 3 and 25 s howed so me r esistance to adversarial p erturbation templates. The fifth and sixth pa rts o f T able 2 show the av era ged us e r and attack er p erfor mances, resp ectively . The adversarial per turbation templates were v ery effective on most sub jects (exc e pt Sub jects 3 and 25), reducing b oth the user s c ores and the user ITRs to almos t zero, i.e., the user almos t cannot corr ectly input any c hara cter he/she wan ted. The attack er score s for five sub jects were clo s e to one, i.e., the atta cker was a ble to force the SSVEP sp eller to output any character he/she wan ted. The SPRs were all around 25 dB, compa r able to the SPRs for ra ndom no ise. Visualization of the adv ersarial p erturbations : This subse c tion shows the character is tics of the adversarial per turbation templates, and verifies their imp e rceptibility to some widely-used a pproaches for ev aluating the quality of SSVEPs. Figure 3b shows the EEG signa ls b efore a nd after a dversarial p erturba tions, alo ng with the magnified differe nc e . The SSVEP sp eller misclassified the use r character, which was supp os e d to b e Y (8.6 Hz), int o N (13.2 Hz). Human eyes can bar ely reco g nize the difference b etw een the benign and the adversar ial EE G tria ls. After b eing ma gnified by 1 0 times, the p erturba tion lo oks p erio dical, which can mo dify the user frequency to the attack er fr equency . W e compa r ed the cle an a nd adversaria l EE G signals w ith standar d sinusoidal signals in Figure 3c, using Sub ject 26 as an example. W e to ok the av erag e of the clean tempo ral wav eforms o f 8 Hz SSVEP s fro m Channel POz , and did the same for their adversaria l sig nals with δ 13Hz added (whic h forced the SSVEP sp eller to output the c hara cter of 13 Hz stimulation frequency). W e chose Channel POz be cause the a dversar ial per turbation on this channel had one of the lar g est amplitudes, a s shown in Fig ure 3b. Figure 3c shows that b oth clean a nd adversarial EE G signals were synchronized with the standa r d 8 Hz sinusoidal signal, indicated b y the gree n do t- dashed lines. Compar ing the 13 Hz s in uso ida l s ignal with the magnified difference, the synchronization can als o be obs erved, sugges ting that the adversarial p erturbation template in tro duced a fre quency component matching the attack er character , which was imper ceptible to human eyes but powerful enough to mislead the SSVEP sp eller. Figure 3d shows the sp ectrum analy sis of SSVEP s for 40 s tim ulation fr equencies. W e av era ged the sp ectra o f the benign EEG signals o f the sa me stimulation frequency fr o m a ll the sub jects and all c hosen ch annels , so that background a ctivities can b e suppressed. The first row o f Figur e 3d, for b enign trials , clearly shows that the visual stimu lus, flickering at a stim ulation frequency , can evoke SSVEPs of the same frequency and its harmonics. The second row of Figure 3d shows the sa me pro per ty of adversarial trials , whose attack er character was ra ndomly chosen and fixed for each stim ulation frequency . W e canno t observe no ticeable differences betw een the tw o rows in Figure 3 d, demonstrating the challenge in de tecting the adversaria l p erturbatio n templates. 6 3 CONCLUSION AND DISCUSSION This article s hows that one can generate adversarial EEG p erturbation templa tes for target attacks for b oth P300 and SSVEP sp ellers , i.e., delib era tely -designed tiny p erturbatio ns ca n manipulate an EEG-based BCI sp eller to output anything the attack er w ants with high succes s rate, demonstr a ting the vulner ability of BCI sp ellers. W e should emphasize that the attack fra mework used here is no t specific to the victim mo dels used in this study . They may also b e utilized to attack many other classifier s in BCIs with little mo dification. Limitations: The cur r ent appro aches have tw o limitations: (a) they re quire some sub ject-/mo del- sp ecific EEG trials to construct the adversarial p erturba tion template; and, (b) they need to know the exact timing of the stimulus to achiev e the best attack p erforma nce. The adversarial attacks c ould b e more dangero us if these limitations are resolved. The first limitatio n may b e alleviated by utilizing the transfer a bility of adversaria l examples, which was one of the most dangerous pr op erties o f adversarial exa mples. It was firs t discovered by Szegedy et al. [20] in 2014 a nd further inv estiga ted by many others [24, 54 – 56]. The tra nsferability means that adversarial examples g enerated from one mo del ca n also b e use d to attack another mo del, which may hav e a completely different architecture and/or b e trained from a different dataset. Thus, it may b e po s sible to co nstruct the adversaria l p erturbation template fr o m some existing sub jects/mo de ls and then apply it to a new sub ject/mo del. Our Supplementary Informa tion presents exp erimental results on b oth cr oss-sub ject and cro ss-mo del transfera bilit y of the g enerated adversar ia l per turbations. The se c o nd limitation is that the attack er needs to know the precise time synchronization b etw een adversarial per turbation templates and EEG signals . T o study how the synchronization time delay affects the attack p erformance, we show the rela tionship betw een the user/a ttack er scores and the time delay in adding the p er turbation template (see Supplementary Fig ure 1). It can be obse rved that the SSVEP p erturbation template w as fairly robust to the time delay wher eas the P30 0 adversarial template was s ensitive to the synchronization. F or the P3 0 0 sp eller, when the time delay increased, the user score s incr eased rapidly while the attack er score decreased rapidly , sugges ting that hiding the time synchronization infor ma tion may help defend aga inst adversar ia l a ttacks in the P300 s pelle r s. How ever, atta cks insensitive to the synch ro nization may als o b e po s sible. F or ex a mple, the idea of adversarial p atch [57], which is a tiny picture pa tch that ca n mislead the class ifier when added anywhere to a lar ge picture to be cla ssified, may b e us e d to increase the robus tness to the synchronization time delay . Thus, defending a gainst the attack ers may not be an easy task. Closed-l o op B C I appli cation considerations: In a t ypical closed-lo op BCI sp eller , the use r could re c eive real-time feedback o f his/her chosen character from the screen. If the adversarial p erturbation constantly misleads the sp eller and retur ns wrong characters that do not ma tch the user’s inten tional input, the user would most likely stop us ing the sp eller. The consequent may not seem serio us for a use r that has other means o f communication; how ever, for pa tients with severe impairments that rely o n BCI sp ellers as their so le mean of communication, e.g ., ALS patients, either the attack er changes the meaning o f their s entences and they canno t do anything ab out it, or the patients stop res po nding, misleading do ctors/ resear chers into thinking they ar e not a ble to communicate at all. Both consequents can significa nt ly impact the patients. Although this article fo cused on a dversar ial a ttacks of P300 and SSVEP sp ellers, P300 and SSVEP are a lso widely us ed in neuro -ergo nomics a nd a s sessment o f co gnitive s ta tes, e .g., diagnosis of patients with disor der of consciousness [5 8]. The prop osed appr oach can be used to a ttack these BCI systems with little modifica tion. The adversarial p erturba tion could also be a serio us conce r n if the BCI system is used in o ther scenarios such as wheelchair control or e x oskeleton control, where the feedback could b e to o late and the cost of one step mistake could b e fatal. Moreov er, the a tta cker may only start the attack in some cr itica l conditions. The user is completely unprepare d, and the consequents co uld b e mor e catastrophic. Finally , we need to emphasize aga in that the goa l of this study is not to damage EEG-bas ed BCIs. Instead, we aim 7 to demonstrate that s erious adversarial attacks to E EG-based BCIs a re p os sible, and hence exp ose a cr itica l security concern, which has r e ceived little attention b efore . Our future r esearch will develop strateg ies to defend aga inst such attacks. Meanwhile, we ho pe our study can a ttract more r esearchers’ attention to the s ecurity of EEG-ba s ed BCIs. 4 METHODS W e detail our appro a ches to ev aluating the vulner ability of P30 0 and SSVEP sp eller s in this section. 4.1 Datasets and Co de The P30 0 spelle r datas et can b e downloaded fro m http ://ww w.bbc i.de/competition/iii/#data_set_ii (Dataset II). The P300 sp eller dataset of ALS pa tient s ca n be downloaded from h ttp:// bnci- h orizon- 2020.eu/database/data- sets (P300 sp eller with ALS pa tient s (008-20 14)). The SSVEP dataset can b e downloaded from htt p://bc i.med .tsinghua.edu.c n / d o w n l o a d . h t m l . All source co de is av a ilable on GitHub ( http s://g ithub .com/ZhangXiao96/Speller- A ttacks ). 4.2 A ttac k the P300 Sp el ler The ma in idea to construct the a dversarial p er turbation template was to find a universal per turbation tha t lea ds the P300 cla ssifier to classify no n-target ep o chs into target o nes. The appr oach was to get the direc tio ns p ointing from non-targe t ep o chs to the decisio n b oundary of the victim mo del, and then sum up these dir ections as the universal per turbation. These dir ections can b e iden tified b y simply calculating the gr adients of the loss with resp ect to the input no n-target EEG ep o chs, ass uming the decision bounda ry is linear. Though the victim mo del includes nonlinea r op erations, the attack appr oach s till work ed surpr isingly well. Let X b e an EE G tria l, y its label (0 for non-ta r get, and 1 for ta rget), f the victim mo del which gives the lab el probability for each input X , J ( X , y , f ) the loss function (cross-entropy loss in our ca se), and D N T the datase t containing all non-tar get ep o chs in the training set. Then, the ov era ll dir ection can b e co mputed as : e P = X ( X,y ) ∈ D N T ∇ X J ( X , 1 − y , f ) k∇ X J ( X , 1 − y , f ) k F . (6) After obtaining e P , we filtered it by a four th-order Butterworth bandpass filter of [0 . 1 , 15] Hz, extracted the first 350ms signal, a nd then no r malized it in each channel so that the L2 nor m is 1. Denote the result as b P . Then, the adversarial p erturbatio n P was computed as: P = ǫ · b P , (7) where ǫ is a consta n t controlling the ener gy o f the p erturbation ( ǫ = 0 . 5 in o ur exp eriments). T o mislead the P300 speller , o ne only needs to tamper with so me sp ecific signal p er io ds according to the onset of the targ et stimuli. Because in a practica l P3 00 sp eller the same r ow or column is nev er in tensified successively , the p erturba tion template can last mor e than one in tensification perio d. In our exp eriments, the template las ted 2 × 17 5 = 35 0 ms, i.e., tw o intensification p erio ds. Figure 4 illustr ates the a ttack pro cedure. The b enign EEG trial would output character 7 , since the last row and the third column o f the character matrix hav e the highest P3 00 probability , and their intersection is 7 . Howev er, after applying the p erturbatio n template, the trial outputs the c haracter Z , beca us e the fifth row and the sec o nd column have the highest P3 00 proba bility . In teresting ly , the a dversarial template a cts like r andom noise when it is not synchronized with an intensification onse t. As shown in Figure 4, the last 175 ms o f the template do es not influence the class ification of the corr esp onding intensification. 8 4.3 A ttac k the SSVEP Sp e ller There are tw o difficulties in a ttacking the victim mo del of the SSVEP sp eller . First, the v ictim mo del is not fixed, as the para meters of CCA v ar y in differen t EEG trials. Second, unlik e the P30 0 sp eller whos e base victim mo del only needs to classify the input into t wo classes, there are many more classes in the SSVEP spe lle r. These mak e adversarial attacks of the SSVE P sp eller muc h mor e challenging. The remedy was to genera te the adversarial p erturba tion template δ b f ∈ R N e × N s , which can lea d the SSVEP sp eller to o utput the attacker character of stimulation frequency b f . F or e a ch user, we used the first blo ck D = { X i } N i =1 to craft δ b f , and the remaining five blo cks to ev aluate its attack p erformance . According to the victim mo del, δ b f should be able to maximize ρ ( X + δ b f , Y b f ) in e quation (4), such that arg max f ∈ F ρ ( X + δ b f , Y f ) = b f . (8) In other words, δ b f can be crafted by solving max δ b f X X ∈D λ max ( S ( X + δ b f , Y b f )) , (9) where S ( X , Y ) is defined in equatio n (3). Since S ( X + δ b f , Y f ) is not sy mmetric, it is difficult to ca lculate the deriv atives of its la r gest eigenv alue , resulting in challenges in optimization. Because of the fa ct that the largest eig env alue is a lwa ys no smaller tha n the av erag e of all eigenv a lues: λ max ( S ( X + δ b f , Y f )) > 1 N e X j λ j ( S ( X + δ b f , Y f )) = 1 N e tr( S ( X + δ b f , Y f )) , (10) instead of solving equation (9) dir ectly , we can maximize its lower b o und to reduce the o ptimization difficult y: max δ b f X X ∈D tr( S ( X + δ b f , Y b f )) . (11) Because the effective frequency ba nd of SSVEP signals is 7-9 0 Hz, we intro duced a new v ariable r b f so that δ b f = filt( r b f ) , (12) where filt( · ) means retaining only the 7-9 0 Hz effective signal frequency comp onents. As a result, we can ensure the int egr ity o f the adversarial template dur ing s ignal filtering. In addition, w e added α · k δ b f k F to p enalize the ene r gy of the p erturba tio n, wher e α is the p enalty co e fficie nt. Finally , the problem b ecomes: min r b f − X X ∈D tr( S ( X + filt( r b f ) , Y b f )) + α · k filt( r b f ) k F . (13) Gradient descent was used to up date r b f , and the itera tion stopp ed when the SPR was low er tha n a threshold, which was set to 25 dB in our ex per iments. D A T A A V AILA BILITY ST A TEMENT Publicly av a ila ble BCI da tasets were used in this study . The P3 00 sp eller datase t c a n b e downloaded fr o m http: //www .bbci.d e / c o m p e t i t i o n / i i i / # d a t a _ s e t _ i i (Dataset I I). The P300 s pe lle r dataset o f ALS pa tient s was first used in [59] and ca n be downloaded from h ttp:// bnci- h orizon - 2 0 2 0 . e u / d a t a b a s e / d a t a - s e t s 9 (P300 sp eller with ALS pa tient s (008-20 14)). The SSVEP dataset can b e downloaded from htt p://bc i.med .tsinghua.edu.c n / d o w n l o a d . h t m l . All source co de is av a ilable on GitHub ( http s://g ithub .com/ZhangXiao96/Speller- A ttacks ). REFERENC ES 1. G raiman n, B., Alli son , B. & Pfurt scheller, G. Brai n-computer interfac es: A gentle intr o duction , 1–27 (Springe r, Berlin, Heidel b erg, 2009). 2. Lin, C. T. et al. EEG-based b r ai n-compute r interfaces: A nov el neu rotechnology and computati onal intelligence metho d. IEEE Systems, Man, and Cybernetics Magazine 3 , 16–26 (2017). 3. He, H. & W u, D. T ransfer l earning for brain-comp uter interfaces: A Euc lidean space data al ignment approach. IEEE T rans. on Biomedic al Engine ering 67 , 399–410 (2020). 4. C hav arriaga, R. & Mill´ an, J. d. R. Learning from EEG error-relate d p otentials in noninv asiv e brain-compu ter interfaces. IEEE T r ans. on Neur al Systems and R ehabilitation Engineerin g 18 , 381–388 (2010). 5. Nicolas-Alonso, L. F. & Gomez-Gil, J. Brain compute r interfaces, a re view. Sensors 12 , 1211–1279 (2012). 6. F arw el l, L. A. & Donchin, E. T alking off the top of your head: To ward a mental pr osth esis utiliz ing even t-related brain p otentials. Ele ctro e nc ephalogr aphy and Clinic al Neurophy siolo gy 70 , 510–523 (1988). 7. C he n , X. et al. High-sp eed sp elli ng with a noninv asiv e brain-compu t e r interface. Pr o c. National Ac ademy of Scienc es 112 , E6058–E6067 (2015). 8. Sutton, S., Brare n, M ., Zubin , J. & Joh n, E . R. Evok ed -pote ntial corre lates of stimulus unc ertaint y . Science 15 0 , 1187–1188 (1965). 9. Donc hin, E., Sp ence r, K. M. & Wijesingh e , R. The me ntal prosthesis: Assessing the sp eed of a P300-based brain -computer interface. IEEE T r ans. on Rehabilitation Engine ering 8 , 174–179 (2000). 10. Meinick e, P ., Kap er, M., Hopp e , F., Heu mann, M. & Ritter, H. Im proving transfer rates in b rain c om puter interfacing: A case study . In Pr o c. A dvanc es in Neural Information Pro c essing Systems , 1131–1138 (BC, Canada, 2003). 11. Xu, N. et al. BCI com p etition 2003-data se t I Ib: En hancing P 300 w av e detecti on using ICA-based subspace pro ject i ons for BCI applicat i ons. IEEE T r ans. on Biomedi cal Engine ering 51 , 1067–1072 (2004). 12. G uan, C., Thulasidas, M. & W u, J. High p erform ance P300 sp eller for brain-comp uter interface . In Pro c . IEEE Int’l Workshop on Biome dical Circuits and S ystems , S3–5 (Singap ore , 2004). 13. P olich, J. Up dating P 300: An integrative theory of P3a and P3b . Clinic al Neur ophysiolo gy 118 , 2128–2148 (2007). 14. C hapm an, R. M. & Bragdon, H. R. Evok ed re sp on ses to numerical and non-numerical visual stimuli while problem solving. Natur e 203 , 1155 ( 1964). 15. Sutton, S., T ueting, P ., Zubin, J. & John, E . R. I n formation del ivery and the sensory evok ed p otential. Scienc e 155 , 1436–1439 (1967). 16. B everina, F. et al. User adaptive BCIs: SS VEP and P300 based interfaces. PsychNolo gy Jou rnal 1 , 331–354 (2003). 17. W ang, Y., Gao, X., Hong, B., Jia, C. & Gao, S. Brain-c omputer interfaces base d on visual evok ed p otentials. IEEE Engine ering in Me dicine and Biolo gy Mag azine 27 , 64–71 (2008). 18. Vialatte, F.-B., Maurice , M ., Dauw els, J. & Cichocki, A. Steady-state visually evoked p otentials: Fo cus on e ssential paradi gms and future p ersp ectives. Pr o gr ess in Neur obiology 90 , 418–438 (2010). 19. W ang, Y., Chen, X., Gao, X. & Gao, S. A b en chmark dataset f or SSVE P-based brain-comp uter interface s. IEEE T r ans. on Neural Systems and R ehabilitation Engineering 25 , 1746–1752 (2017). 20. Szegedy , C. et al. Intriguing prop e r ties of neural n etw orks. In Pr o c. Int’l Conf. on L earni ng Repr esentations (Banff, Canada, 2014). 21. G o o dfel l ow , I. J., S hlens, J. & Sz egedy , C. Expl aining and harn essing adversarial exampl es. In Pr o c. Int’l Conf. on Le arning R epresenta- tions (S an Die go, CA, 2015). 22. Kurakin, A., Go o dfellow, I. J. & Be ngio, S. Adversarial examples in the physical world. In Pro c . Int’l Conf. on L earni ng Repr esentations (T oul on, F rance, 2017). 23. A thalye, A., E ngstrom, L., Ilyas , A. & Kw ok, K. Synthesizing robust adversarial example s. In Pr o c. 35th Int’l Conf. on Machine Le arnin g , 284–293 (S to ckholm, Sw ed en, 2018). 24. P ap ernot, N., McDanie l, P . & Go od fellow, I. T ran sferability in machine learning: From p henome na to black-box attacks usin g adversarial samples. CoR R abs/1 605.072 77 (2016). URL https://arxi v.org/abs/1605.0 7277 . 25. C arlini , N. & W agner, D. A. Audio adversarial examples: Targete d attacks on sp ee ch-to-text. In Pro c. IEEE Symposium on Security and Privacy , 1–7 (San F rancisco, CA, 2018). 26. Jia, R. & Liang, P . Adv ersarial examples for ev aluating reading compreh ension systems. CoR R abs/1 707.073 28 (2017). URL https://arxiv.o rg/abs/1707.0732 8 . 27. G rosse, K., Pap ernot, N., Man oharan, P ., Back es, M. & McDanie l, P . Adversarial p e rturbations again st dee p n eural netw orks f or malware classification. CoR R abs/16 06.0443 5 (2016). URL https://ar xiv.org/abs/1606 .04435 . 10 28. P ap ernot, N., McDan iel, P ., W u, X., Jha, S. & Swami, A. Distillation as a d efense to adversarial p erturbation s against deep ne ural netw orks. In Pro c. IEEE Sympo sium on Se curity and Privacy , 582–597 (S an Jose, CA, 2016). 29. Madry , A., Makelov, A., Schmidt , L., Tsipras, D. & Vladu, A. T o w ards d e ep learnin g mo de ls resistant to adversarial attacks. In Pr oc. Int’ l Conf. on Le arning R epresentation s (V ancouver, Canada, 2018). 30. T ramr, F. et al. Ensemble adversarial trainin g: Attacks and defen ses. In Pro c . Int’l Conf. on L earn ing Repr esentations (V an couver, Canada, 2018). 31. Samangouei, P ., K abk ab, M. & Che l lappa, R. Def ense-GAN: Protec ting c lassifiers against adversarial attacks u sin g gene rative mo de ls. In Pr o c. Int’l C onf. on L e arning R epresentati ons (V ancouver, Canada, 2018). 32. Xie, C., W ang, J., Zhan g, Z., Ren, Z. & Y uille, A. Mi tigating adversarial effects th rough ran domization . In Pro c. Int’l Conf. on L earn ing R epresentati ons (V ancouver, Canada, 2018). 33. Qin, C. et al. Adv ersarial rob u stness throu gh lo cal lineari zation. In Pro c. A dvanc es in N eural Information Pr o c essing Systems , 13824– 13833 ( V ancouver, Canada, 2019). 34. C arlini , N. & W agner, D. T o wards ev aluating the robustne ss of ne ural netw orks. In Pr o c. IEEE Sy mposium on S ecurity and Privacy , 39–57 ( S an Jose , CA, 2017). 35. A thalye, A., Carlini, N. & W agner, D. Obfuscate d gradie nts giv e a false sense of security: Circ umv enting defe nses to adversarial examples. In Pro c. 35th Int’l Conf. on Machine Le arning , 274–283 (Sto ckholm, S w ed en, 2018). 36. G ow al, S. et al. On the effec tiveness of interv al b ound propagation for training verifiably robust mo de ls. C oRR a bs/1810 .12715 (2018). URL http://arXiv.org/a bs/1810.12715 . 37. C ohen , J., Rosenfel d, E. & Kolter, Z. Certifi ed adversarial robustness via randomi zed smo othing. In Pr o c. 36th Int’l Conf . on Machine L earn ing , 1310–1320 (Long Beach, CA, 2019). 38. Li, B., Chen, C., W ang, W. & Carin, L. Certifie d adversarial robustness with additi ve n oise. In Pr o c. Advanc es in Neural Information Pr o cessi ng Systems , 9464–9474 (V ancouver, Canada, 2019). 39. B alunovic, M. & V echev, M. Adversarial training and prov abl e d efense s: Bridgi ng the gap. In Pr o c. Int’l Conf. on Le arning R epresentation s (Addis Ababa, E thiopia, 2020). 40. Z hang, X. & W u, D. On the vulnerab ility of CNN classifi e rs in EE G-based BCIs. IEEE T r ans. on Neural Systems and R ehabilitation Engine ering 27 , 814–825 (2019). 41. F aw az, H. I., F orestier, G., W eb er, J., Id oumghar, L. & Muller , P .-A. Adversarial attacks on de ep neural netw orks for ti me series cl assification. In Int’l Joint Conf. on Neur al Ne tworks , 1–8 (Bud ap est, Hungary , 2019). 42. Qin, Y. , Carlini, N., Cottrel l, G., Go o dfell o w, I. & Raffel, C. Imp erce ptible , robu st, and targeted adversarial examp les f or automatic sp ee ch recognition . In Pro c. 36th Int’l Conf. on Machine Le arnin g , 5231–5240 (Long Beach, CA, 2019). 43. Moosa vi-Dez fo oli, S.-M., F a wzi, A., F a wzi, O. & F rossard, P . Universal adversarial p e rturbations. In Pro c . IEEE Conf. on Computer Vision and Pattern R e c o gnition , 1765–1773 (Honolulu, HI, 2017). 44. W olpaw, J. R., Ramoser, H., McF arland, D. J. & Pfurtscheller, G. EE G-based communicati on: Imp rov ed accu racy by resp onse verification. IEEE T r ans. on Rehabilitat ion Engine ering 6 , 326–333 (1998). 45. W olpaw, J. R., Krusienski, D. & Schalk, G. Doc umentation W adsworth BCI dataset (P 300 evok ed p ote ntials) (2004). URL http://www.bbci .de/competition/ iii/desc_II.pdf . 46. R ivet, B., S ouloumiac, A., Attina, V. & Gib e rt, G. xDA WN algorithm to enhan c e evok ed p otentials: Applic ation to brain-comp uter interface. IEEE T r ans. on Biomedi cal Engine ering 56 , 2035–2043 (2009). 47. B arachant, A., Bonne t, S., Congedo, M. & Jutte n, C. Multic lass brain -compute r interface cl assification by Riem annian geometry . IEEE T r ans. on Biomedic al Engineering 59 , 920–928 (2012). 48. B arachant, A., Bonnet , S., Congedo, M. & Jutten, C. Classific ation of c ov arianc e matric es using a Riemannian-b ase d kernel for BCI application s. Neur o c ompu ting 112 , 172–178 (2013). 49. Yger, F., Be r ar, M . & Lotte , F. Rie mannian ap proaches in b rain-compute r interfaces: A r e view. IEEE T r ans. on Neural Systems and R ehabilitation Engineerin g 25 , 1753–1762 (2017). 50. Abadi, M. et al. T ensorflow: A system for large-scale m achine learn ing. In Pr o c. 12th USENIX Symp osium on Op eratin g Systems Design and Implementation , 265–283 (Sav annah, GA, 2016). 51. C hen , X., W ang, Y., Gao, S., Ju ng, T.-P . & Gao, X. Filter ban k canonic al corre lation analysis for i m pleme nting a high-sp ee d SSVEP-based brain–comp uter interface. Journal of Neural Engine ering 12 , 046008 (2015). 52. Lin, Z., Zhan g, C., W u, W. & Gao, X. F requen cy re cognition based on canonic al correlation anal ysis for S SVEP-based BCIs. IEEE T r ans. on Biomedi cal Engine ering 53 , 2610–2614 (2006). 53. Ak aike, H. Canon ical correlation analysis of time series and the use of an inf ormation crit erion. In Mathem atics in Science and Engine ering , vol. 126, 27–96 (E lsevier, 1976). 11 54. Liu, Y. , Chen , X., Li u, C. & Song, D. De lving into transferable adversarial examp les an d black-box attacks. CoRR abs/16 11.0277 0 (2016). URL http://arxiv.org/a bs/1611.02770 . 55. T ramr, F., Pap ernot, N., Go o dfellow, I., Boneh, D. & McDaniel, P . The space of transferable adversarial examples. CoRR abs/17 04.0345 3v2 (2017). URL http:/ /arXiv.org/abs/1 704.03453v2 . 56. W u, L., Zhu, Z., T ai , C. & E , W. Understand ing and enhanci ng the transferability of adversarial example s. CoRR ab s/1802 .09707 ( 2018). URL http://arXiv.org/a bs/1802.09707 . 57. B rown, T. B., Man ´ e , D., Ro y , A., Abad i, M. & Gilmer, J. Adversarial p atch. CoRR a bs/171 2.09665 (2017). URL http://arxiv.or g/abs/1712.09665 . 58. Li, Y. et al. Multimo dal BCIs: T arget detecti on, multidim ensional control, and aw areness e v aluation in patie nts with disorder of consciou s- ness. Pro c e edin gs of the IEEE 1 04 , 332–352 (2016). 59. R ic c io, A. et al. Attention and P300-based BCI p erf ormance in p eop l e with amy otroph ic l ateral sc lerosis. F rontiers in Hum an Neur oscienc e 7 , 732 (2013). 60. Hestenes, M. R. Multiplie r and gradie nt m etho ds. Journal of Optimization Theory and Applic ations 4 , 303–320 (1969). 12 Figure 1 A P300 sp eller and an SSVEP sp eller. a , W orkflow of a P300 sp eller (top path) and an SSVEP s pell er (b ottom path). F or eac h sp eller, the user wat ches the stimulation interface, f ocusing on the c haracter he/she wan ts to i nput, and EEG signals are recorded and analyzed b y the s pell er. The P300 sp eller first i den tifies the ro w and the column that elicit the largest P300, and then outputs the character at their i ntersection. The SSVEP sp eller iden tifies the output charact er directly b y matching the user ’s EEG oscill ation frequency with the flick ering frequency of each candidate c haracter. b , Stimulation i nt erface of a P300 sp eller, where the second column is inte nsified. c , Stimulation interface of an SSVEP sp eller. The n umber below eac h charact er indicates its flic kering frequency (Hz). 13 T able 1 P300 sp eller attac k results. Befor e attac k: Baselines on clean EEG data (without adding any p erturbations) and Gaussian- noise-per turbed EEG data, and the corresponding SPRs (dB). After attac k: Average user/attac k er scores/ITRs of the 36 attack er c haracters in target attac ks, and the corresponding per i od and trial SPRs (dB). Sub. Num b er Before a ttack After a ttack of Clean Gaussian noise User A ttack er Period T rial rep eats Scor e ITR Score ITR SPR Score ITR Score ITR SPR SPR 5 0.64 13.0 7 0.65 13.4 0 20.8 0.072 0.2 48 0.82 5 19.8 20.8 25.8 A 10 0.85 10.6 2 0.84 10.4 0 21.0 0.049 0.0 52 0.90 0 11.7 21.0 25.9 15 0.91 8.03 0.92 8.19 2 1.0 0.040 0.0 2 1 0.950 8.7 21.0 25.9 5 0.79 18.4 1 0.79 18.4 1 25.2 0.107 0.5 78 0.71 3 15.6 25.2 30.2 B 10 0.91 11.9 6 0.89 11.5 0 25.5 0.061 0.0 93 0.86 0 10.9 25.5 30.4 15 0.93 8.35 0.91 8.03 2 5.6 0.049 0.0 3 4 0.907 8.0 25.6 30.5 T able 2 SSVEP sp eller attack r esults. Befor e attac k: Baselines on clean data (without adding any perturbations), Gaussian-noise- perturb ed EEG data and p erio dic-noise-p erturb ed EEG data. After attac k: Av erage user/attac ker scores/ITRs of 40 attac k er ch aracters in target attac ks, and the corr esponding SPRs (dB). Sub. Before attack After attack Clean Gaussian Noise S/C Perio dic Nois e SPR User A ttack er SPR Score ITR Score ITR Score ITR Score ITR Sco re ITR 3 0.88 18 2.5 0.8 8 18 1.6 0.71/0.87 129.0/ 178.6 25.0 0 .44 61.1 0.58 93.3 25.3 4 0.90 186.9 0.9 0 187.1 0 .6 8/0.8 7 12 1.0/17 7.5 25 .0 0.07 2.3 0.95 2 10.1 25 .7 12 0.90 188.8 0.9 0 188.0 0 .7 8/0.8 6 15 0.0/17 4.3 25 .0 0.26 2 6.6 0.75 139.5 2 5.5 22 0.82 16 0 .0 0.7 9 150.2 0 .7 4/0.7 5 13 7 .5/14 0.7 25.0 0.11 6.1 0.9 1 191.0 25.1 25 0.90 18 9 .1 0.8 9 184.1 0 .8 4/0.8 7 16 8 .3/17 7.2 25.0 0.78 14 8 .2 0.17 13.8 26.7 26 0.90 18 7 .8 0.8 8 180.3 0 .5 8/0.8 4 94.4/1 68.1 25.0 0.03 0.1 1.00 2 29.9 24 .8 32 0.87 17 6 .9 0.8 7 179.6 0 .5 9/0.8 2 97.2/1 63.6 25.0 0.03 0.0 1.00 2 31.4 24 .9 34 0.80 15 4 .7 0.7 9 151.8 0 .4 8/0.7 2 66.8/1 30.4 25.0 0.03 0.0 1.00 2 31.2 25 .9 14 Figure 2 P300 sp eller attac k results. a , At tack er scores of manipulating the P300 spell er to misclassi fy the 100 test charact er trials int o a sp ecific attac ker ch aracter. The P300 sp eller used 15 intensificat ion r epeats for eac h charact er. b , EEG trials b efore and after adv ersarial p erturbation, which are almost completely ov erlapping (the SPRs are sho wn in T able 1), and the difference (magnified ten times) b etw een the adversarial trial and the b enign trial. The non-zero part of the difference is the adversarial p erturbation template, which i s added to a benign EEG trial according to the attac k er ch aracter. The adversarial p erturbation led the P300 sp eller to m isclassify letter Y into N . c , left column: the av erage of 100 × 15 × 2 = 3 , 000 target trials (con taining P300) and the av erage of 100 × 15 × 10 = 15 , 000 non-target trials (not containing P300) at ch annel Cz, for benign and adversarial trials; middle column: sp ectrogram of the difference betw een the av erage target trial and the av erage non-target trial in c hannel Cz, for b enign and adv ersarial trials; right column: topoplot of the difference b etw een the a verage target trial and the av erage non-target trial, for b enign and adve rsar i al trials. b and c present the visualization of the adversarial perturbations for Sub ject A. 15 Figure 3 SSVEP spell er attac k r esults. a , At tack er scores of manipulating the SSVEP sp eller to mi sclassify the 5 × 40 = 200 test c haracter trials into a sp ecific attack er frequency (cha racter). b , lef t column: EEG trials b efore and af ter adv ersari al p erturbation, f or Sub ject 26; righ t column: the difference (adve rsar i al p erturbation) b et ween the adv ersarial EEG tri al and the b enign EEG tri al for Sub ject 26, magnified by ten times to make them visibl e. The adversarial perturbation led the SSVEP sp el l er to m i sclassify the letter Y int o N . c , detailed si gnal analysis for Channel POz of Sub ject 26. The clean s i gnal was the av erage of all six trials of 8 Hz stimulation frequency , and the adversarial trial was the av erage of the same trials wi th δ 13Hz added. Stan dard 8 Hz and 13 Hz sinusoidal si gnals are shown as references. The green dot-dashed lines mar k the 8 Hz p erio dicity . d , Normalized sp ectra of SSVEPs for 40 stimulation frequencies, av eraged ov er all the chosen c hannels and all 40 sub jects. 16 Figure 4 Illustration of the attac k pro cedure in the P300 protocol. The attack er charac ter is Z , whereas the user character i s 7 . F or the benign EEG tri al, the P300 sp eller can correctly identify that P300 is elicited b y the intensifica tions of the last row and the third column. T o mislead the P300 sp eller, adve rsari al p erturbation template i s added during the p erio ds of 0-350ms and 700-1050ms, so that the fif th row and the second column are beli ev ed to elicit P300 with the highest probability . The added adv ersarial perturbation templates do not influence the results of the second and the last stimuli, b ecause their corresponding peri ods are out of sync hronization with the templates. As a result, the P300 sp eller misclassifies the p erturb ed tri al to attac k er charact er Z . 17 Supplemen tary Information 5 The victim mo del of the P300 sp e ller The details of the v ictim mo del of the P30 0 sp eller a re intro duced. 5.1 xDA WN spatial filters The original x D A WN filter [4 6] was de s igned for P300 evoked potentials by enhancing the tar get r esp onse with resp ect to the no n-target res po nse. W e used a generalized version in our ex p er iment s, which was implemen ted in pyR iemann 2 . More sp ecifica lly , let D = { ( X i , y i ) } N i =1 be the training set, where X i ∈ R N e × N s is the i -th mean-centered EE G epo ch ( N e is the num ber of channels, and N s the num ber of time domain samples), and y i ∈ { 0 , 1 } its corr esp onding lab el (0 for non-t ar get , and 1 for t ar get ). The average epo ch X c , c ∈ { 0 , 1 } , is first calcula ted. Spatial filters U c ∈ R N f × N e were then de s igned to maximize the sig nal to signal-plus-no ise ra tio for each cla ss: U c = arg max U tr  U X c X T c U T  tr  U X all X T all U T  , (14) where N f is the num ber o f filters ( N f = 8 was used in our e xp e r iments), X all is obtained by concatenating all EE G epo chs in D along the channels, and tr is the tr ace of a matrix. Gener alized eigenv alue dec o mpo sition can b e used to s olve equation (14). After obtaining the filters for b oth class es, the conca tenated spatial filters U = [ U 0 ; U 1 ] can b e us ed to filter each EEG ep o ch: e X i = U X i . (15) 5.2 T angen t s pace pro jection Cov aria nce matrices of the EEG trials ar e widely -used in BCIs. How ever, they lie on a Riemannian manifold of Symmetric Positive Definite (SPD) matrices, and hence cannot be dire c tly used by Euclidea n space clas s ifiers, such as Log istic Reg ression and Supp or t V ector Machines. T o solve this problem, the cov aria nce ma trices are pro jected onto the Euclidea n tangent space o f a r eference SPD ma tr ix, and then the vectorized features ar e used by Euclidea n space classifier s. More s pec ific a lly , we first calcula te the augmented cov ar iance matr ix C i for each e X i : C i = " Z Z T , Z e X T i e X i Z T , e X i e X T i # , (16) where Z =  U X 0 ; U X 1  . Then, C i is pro jected onto the tangent spa ce of the r eference SPD matr ix C f , which is the geometric mean of { C i } N i =1 , i.e., C f = a rg min C N X i =1 δ ( C, C i ) 2 ! , (17) where δ ( C A , C B ) is the Affine Inv a riant Riema nnian Metric distance: δ ( C A , C B ) =    logm  C − 1 / 2 A C B C − 1 / 2 A     . (18) 2 h ttps://pyriemann.readt hedo cs.io/en/latest/inde x.html 18 The vectorized features ar e: s i = upp er  logm  C − 1 / 2 f C i C − 1 / 2 f  , (19) where upp er( · ) v ector izes the upper triangula r pa rt of a symmetric matrix. A weight o f √ 2 is a pplied to the off- diagonal elements, and a w eight of 1 to the rest, during the vectorization. s i can then be fed in to a ny Euclidean space classifier . 6 Canonical correlation analysis (CCA) This section introduces CCA, which c a n b e used to extrac t the under lying corre lation b etw een tw o time series. 6.1 Probl em setup Let X ∈ R C 1 × N and Y ∈ R C 2 × N be tw o multi-c hannel time series, where C 1 and C 2 represent the num ber o f channels, and N the num ber o f time domain samples. X and Y a re z -normaliz e d in each channel. The main idea of CCA is to find a pair of c a nonical v ariables, denoted as a ∈ R C 1 × 1 and b ∈ R C 2 × 1 , for X a nd Y r e s pe c tively , s o tha t the co rrelatio n co efficie nt ρ b etw een a T X and b T Y ca n b e max imized. The pro blem can b e mathematically formulated as : max a , b a T X Y T b √ a T X X T a √ b T Y Y T b , (20) which can b e re- expressed as : max a , b a T X Y T b , (21) s.t. a T X X T a = 1 , b T Y Y T b = 1 . 6.2 S o lution of CCA There are several approa ches to so lve equation (21). Here we introduce the Lag range multiplier metho d [60]. Denote AB T by S AB . Then, eq uation (21) can b e rewritten as: max a , b a T S X Y b , (22) s.t. a T S T X X a = 1 , b T S T Y Y b = 1 . According to the Lag range multiplier metho d, equation (22) is eq uiv alen t to max a , b ,λ,θ J ( a , b , λ, θ ), wher e: J ( a , b , λ, θ ) = a T S X Y b − λ 2 ( a T S X X a − 1) − θ 2 ( b T S Y Y b − 1) . (23) By setting the firs t par tial deriv ativ es to zero , i.e., ∇ a J = S X Y b − λ · S X X a = 0 , (24) ∇ b J = S Y X a − θ · S Y Y b = 0 , (25) ∂ J ∂ λ = − 1 2 ( a T S X X a − 1) = 0 , (26) ∂ J ∂ θ = − 1 2 ( a T S Y Y b − 1) = 0 , (27) 19 we have λ = θ = a T S X Y b . (28) It should b e noted that equation (28) is als o the definition of the cor relation co efficient ρ . According to equations (24) a nd (25), we hav e: S − 1 X X S X Y S − 1 Y Y S Y X a = λ 2 a = ρ 2 a , (29) which implies that ρ 2 equals the larg est eigenv alue of S − 1 X X S X Y S − 1 Y Y S Y X , a nd a is the corr esp onding eig env ector. b ca n b e obtained in a simila r wa y . 7 Securit y of a P300 sp elle r for Amy otrophic Lateral Sclerosis (ALS) patients W e pe rformed additional exp eriments to inv estiga te ho w adversarial p erturba tio ns impact ALS patien ts on P30 0 Spelle r s [59]. The eight-c hannel (Fz, Cz, P z, O z, P3, P4, PO7 , and PO 8 ) EEG signals were recorded from eig ht ALS patients. The EEG data were digitized at 256 Hz, bandpass filtered to 0.1-30 Hz, and then z - no rmalized for e a ch channel. F or each sub ject, there were 21 characters for training a nd 1 4 fo r testing. E ach character corres po nds to a set o f 12 r andom intensifications, which were rep eated 20 times. Each int ensifica tion lasted for 1 25 ms, follow ed by a 125 ms blank. In o ur exp eriments, 10 rep eats were utilized to output a character during the test. W e applied the same Riemannian geometr y ba sed a pproach to reco gnizing the e xistence of P 300 p otentials. The only difference fr om the pr evious study was that the n umber o f xD A WN spatial filters was eight. A s shown in the ‘Before a ttack’ panel o f T able 3, the victim mo dels demonstra ted go o d p erforma nce without attacks, a nd als o high robustness to Ga us sian noise p erturbations. How ever, the ‘After atta ck’ panel shows that all us e r scores and ITRs were mor e or les s reduced after a dversarial p erturba tions. F or ha lf of the sub jects (sub jects 1, 2, 5 and 7), the user scores a nd ITRs a pproached zer o, i.e., the P30 0 sp eller b ecame almost completely useless, indicating a ser io us security concer n of P3 00 s pelle r s to ALS patients. T able 3 P300 sp eller attac k r esults f or eigh t ALS patients. Before attac k: Baseli nes on clean EEG data (without adding an y p ertur- bations) and Gaussian-noise-p erturb ed EEG data, and the corr esp onding SPRs (dB). After attac k: Average user/attac k er scores/ITRs of the 36 attac ker characters in target attac ks, and the corresp onding p erio d and trial SPRs (dB). ǫ = 0 . 8 for all the p erturbations. Sub. Before attack After a ttack Clean Gaussian noise User Attac ker Period T rial Score ITR Score ITR SP R Score ITR Score ITR SPR SPR 1 0.79 6.57 0.79 6.57 2 2.6 0 .03 0.04 1.00 10.22 22.6 27.4 2 0.93 8.76 0.9 3 8.76 22.4 0.10 0.26 0.7 4 6.09 22.4 27.5 3 1.00 10.2 2 1.00 10.22 22.9 0.53 3 .59 0.17 0.67 22 .9 27.7 4 1.00 10.2 2 0.93 8.76 23.1 0.4 5 2.8 3 0.22 1.10 23.1 27.9 5 1.00 10.2 2 1.00 10.22 22.2 0.05 0 .12 0.86 7.72 22 .2 27.1 6 0.93 8.76 0.8 6 7.60 22.4 0.21 0.86 0.4 4 2.75 22.4 27.2 7 1.00 10.2 2 1.00 10.22 22.9 0.03 0 .05 0.96 9.45 22 .9 27.7 8 1.00 10.2 2 1.00 10.22 23.1 0.98 9 .73 0.03 0.05 23 .1 28.0 8 T ransferabili t y of adv ersarial p erturbations W e hav e mentioned that a limitation of the attack approaches is that they requir e some s ub ject-/mo del-sp ecific information to construct adversarial p erturbation templates. O ne p os sible solution to alleviate this problem is to 20 enhance the transfer a bility of adversarial per turbations: the a ttack er can genera te adversarial pe r turbations based on EEG signals g athered by himself/ he r self, or a ny mo del he/ s he choo ses to use, and then utilize them to a ttack another sub ject/mo del. Here, we pre sent some exp erimental res ults on the cross - sub ject and cr o ss-mo del transferability of our adversarial p erturbatio ns (for P 300 sp ellers, the ALS patient dataset was used due to its la rge num b e r of sub jects). W e found that adversar ia l p erturba tio ns fo r SSVEP sp ellers s eem to have b etter transfer ability than P 300 sp ellers. 8.1 C ross-sub ject transferability W e used adversarial pertur bations generated from o ne sub ject to attack the victim mo del of a nother sub ject. Figure 5 shows the average attack er s cores of cro ss-sub ject attacks. There was almo s t no cros s -sub ject transferability of adversarial per turbation templates for the P 300 sp ellers , wher eas p erturbatio ns for SSVEP sp eller s can usually successfully attack the victim mo dels of different s ub jects. Additionally , some sub jects were muc h mo re robust to transfer attacks, e.g., Sub jects 1 2, 22 and 25 in Figur e 5b. Wh y adversarial per turbation templates demonstra ted p o or cross -sub ject transfera bilit y for the P300 sp ellers would b e inv estigated in more depth in our future resear ch. Figure 5 Cross-sub ject transferability of adversarial per turbations. The heatmap shows the av erage attac ker scores w hen using the adv ersarial p erturbations of one sub ject to attac k another sub ject. a , attac ker scores f or the P300 s p ell er . b , attack er scores for the SSVEP sp eller. 8.2 C ross-mo del transferabili ty Cross-mo del tr a nsferability requir es adv ersar ial p erturba tions to be able to attack different E EG classification pipelines , which means the attacker do es not need acces s to victim mo dels anymore, implying a more serious threat to the security o f BCI sp ellers. This subsection prese n ts the attack p erfo rmance of our ge ne r ated adversarial p ertur- bations on new EE G cla s sification pip elines. F or the P300 sp ellers , the new classifica tio n pip eline co nsisted of xD A WN filtering a nd Logistic Re g ression clas - sification, and the adversarial p er turbation templates were again genera ted from the Riemannia n ge ometry based approach. The ‘Before attack’ pa nel of T a ble 4 shows that the new pipeline had hig h classifica tion accur acy witho ut attacks, and it was also ro bust to Gauss ian noise. The ‘After atta ck’ panel shows that the new pip eline can still be manipulated by a dversarial p erturbatio n templates constr ucted from a different pipeline, though not as muc h a s that in T able 3. Comparing the attack p erformanc e s in T ables 3 and 4, it se e ms that an adversarial p erturbation template 21 with b etter attack pe rformance on the mo del it was gener ated from may also hav e b etter cr oss-mo del transfera bilit y to a ttack another mo del. T able 4 P300 s p ell er cross-mo del attac k results for eigh t ALS patient s. The victim mo del (xDA WN and Logistic R egression) was differen t from the attac ke r mo del (a Riemannian geometry based approac h), based on which adv ersari al p erturbations were generated. Before attac k: Baselines on clean EEG data (without adding any p erturbations) and Gaussian-noise-perturb ed EEG data, and the corresponding SPRs (dB). After attac k: Average user/attac k er scores/ITRs of the 36 attac k er c haracters in target attacks, and the corresponding per i od and trial SPRs (dB). ǫ = 0 . 8 for al l per turbations. Sub. Before attack After a ttack Clean Gaussian noise User Attac ker Period T rial Score ITR Score ITR SP R Score ITR Score ITR SPR SPR 1 0.86 7.60 0.8 6 7.60 22.6 0.03 0.04 1.0 0 10.22 22 .6 27.4 2 1.00 10.2 2 1.00 10.22 22.4 0.53 3 .56 0.24 0.99 22 .4 27.5 3 1.00 10.2 2 1.00 10.22 22.9 0.62 4 .66 0.15 0.60 22 .9 27.7 4 0.79 6.57 0.7 9 6.57 23.1 0.49 3.21 0.2 0 0.91 23.1 27.9 5 0.86 7.60 0.8 6 7.60 22.2 0.17 0.60 0.6 0 4.31 22.2 27.1 6 1.00 10.2 2 1.00 10.22 22.4 0.36 1 .90 0.31 1.66 22 .4 27.2 7 1.00 10.2 2 1.00 10.22 22.9 0.17 0 .61 0.53 3.59 22 .9 27.7 8 1.00 10.2 2 1.00 10.22 23.1 0.94 9 .04 0.04 0.06 23 .1 28.0 F or the SSVEP sp eller s, we utilized Filter Bank Canonical Correla tio n Analysis (FBCCA) 3 as our new victim mo del [5 1]. T able 5 shows the bas e line p erfo rmance o f FBCCA and the attack per formance of adversarial p er turba- tions (g enerated from CCA) on this mo del. FBCCA demonstra ted promising p erformance on clea n and randomly per turb ed E EG signa ls. How ever, adversarial pertur bations ge ne r ated from CCA can still manipulate the output characters of FBCCA, verifying that c r oss-mo del transfera bilit y also exists in the SSVE P sp ellers. T able 5 SSVEP sp eller cross-mo del attac k results. The victim m odel (FBCCA) was different fr om the attac ker mo del (CCA), based on which adve rsari al per turbations were generated. Before attac k: Baselines on clean data (without adding an y p erturbations), Gaussian- noise-per turbed EEG data and p erio dic-noise-p erturb ed EEG data (single/comp ound). After attac k: Ave rage user/attac k er scores/ITRs of 40 attac ke r c haracters in target attac ks, and the corresp onding SPRs (dB). Sub. Before attack After attack Clean Gaussian Noise S/C Perio dic Nois e SPR User A ttack er SPR Score ITR Sco re ITR Scor e ITR Score ITR Score ITR 3 0.98 21 8.7 0.9 8 21 9.0 0.96/0.97 212.7/ 217.7 25.0 0 .06 2 .3 0.9 2 204.2 2 5.3 4 0.88 18 1 .4 0.8 8 182.2 0 .8 4/0.8 8 16 9 .0/18 0.1 25.0 0.03 0 .0 1.00 230.6 25.7 12 0.85 17 3 .3 0.8 2 163.1 0 .8 1/0.8 1 15 9 .9/15 9.9 25.0 0.05 2 .2 0.97 219.0 25.5 22 0.95 20 7 .4 0.9 5 208.1 0 .9 4/0.9 5 20 4 .7/20 6.4 25.0 0.03 0 .4 0.99 228.6 25.1 25 0.87 17 6 .8 0.8 4 168.1 0 .8 3/0.8 4 16 3 .4/16 6.4 25.0 0.10 7 .8 0.90 194.0 26.7 26 0.87 17 7 .9 0.8 6 174.1 0 .7 5/0.8 2 13 9 .8/15 9.6 25.0 0.03 0 .0 1.00 231.4 24.8 32 0.88 18 1 .2 0.8 7 180.1 0 .8 0/0.8 5 15 7 .1/17 0.5 25.0 0.03 0 .0 1.00 231.4 24.9 34 0.88 18 0 .7 0.8 5 171.3 0 .7 2/0.8 0 13 2 .6/15 5.9 25.0 0.03 0 .0 0.98 226.4 25.9 The k ey to the tr a nsferability is to find the mo st common patterns sha red by different mo dels, hence the a dversarial per turbations a ffecting these patter ns can a ttack as many mo dels as p ossible. F ro m this po int of view, generating adversarial p er turbations based on an ensemble o f multip le mo dels presents a serious threa t to the s ecurity o f BCIs. W e will explor e this in o ur future resea rch. 3 Our implementa tion was adapted f rom https://gi thub.com/hisu n jiang/CCAforSSVEP . 22 9 Additional Figures Figure 6 shows how the s ynchronization time delay affects the a ttack p erfo rmance. Figure 6 User and attac k er scores with resp ect to the synchronizat ion time delay . The curve r epresen ts the mean of all attac ke r c haracters, and the shadow the standard deviation. a , scores for the P300 sp eller, where 100 test tri als for Sub ject A we re per turb ed to be mis cl ass i fied as eac h of the 36 attac ker charact ers. b , scores for the SSVEP s p ell er , where 5 × 40 = 200 test trials for Sub ject 26 were perturb ed to b e misclassi fied as each of the 40 attac ker c haracters. 23