Eye-Tracking Evolutionary Algorithm to minimize users fatigue in IEC applied to Interactive One-Max problem

Eye-Tracking Ev olutio nary A lgorithm to min imize user fatigue in IEC appl ied to I nteractiv e One-Max prob lem Deni s P ALLEZ LI RIS La b Uni ve rsi ty o f L y on1 Ly o n – France deni s.pa lle z @uni ce.fr Phi lip pe CO LLAR D I3 S L ab Uni ve rsi ty o f Ni ce Ni ce – France ph ili pp e.co ll ard @u nic e.fr Thie rry BA CCI NO LP EQ Lab Uni ve rsi ty o f Ni ce Ni ce – France thi err y.b acc in o @u nic e.fr Laur ent DUM ERC Y LP EQ Lab Uni ve rsi ty o f Ni ce Ni ce – France la ure nt.d ume rcy @u nic e.fr ABSTRACT In this paper, we describe a new algorithm that consists in combining an ey e-tracker f or minimizing the fatigue of a user during the evaluation process of Interactive Evolutionary Computation . The approach is then ap plied to the Interactive One- Max optimization p roblem. Categories and Sub ject Descr iptors D.3.3 [Program m ing Langua ges ]: Language Contructs and Features – a bstract da ta types, po lymorphism, control stru ctures. I.2.10 [Artificial Intelligence] : Vision an d Scene Understanding – Perceptua l reasonin g J.4 [Social And Behavioral Sciences] : Psychology General Terms Algorithms, Measurement, Experimentation, Human Factors. Keywords Interactive evolution ary com putatio n, user fa tigue minimization, eye-track ing system, interactive one-max problem. 1. INTRODUCTION Interactive Evoluti onary Computation (IEC) often suffers from user fatigue. In this paper, we present a new techniqu e, totally independ ent o f the d omain used, to minimize t his fatigue b y combining an IEC and an inp ut device. This d evice allows capturing where th e user is looking on a monitor on which individu als are presented. This i s p ossible b y using eye-tracking system s such as Tobii  which are to tally n on-int rusive for the user. Thus, we ensure there is no need for explicit user action (choosin g and clicking the most p romising individ ual, evaluating all the solutions etc.) durin g the evaluation p rocess o f the IEC; h e just has to watch the screen and t he presented individu als and t o tell when he has finished eva luatin g/looking. The evolutio nary algorithm then determines automatically which presented individu als are b etter by combinin g parameters ob tained by a Tobii  f or eac h presented ind ividual. We have applied to the Interactive On e-Max pro blem [3 ]. Thus, by using to tally implicit evaluation, we minimize the fatigue of the u ser in in teractive computation, indepen dently of the problem to be optimized. This approach may be used in any computer graphics app lication in which optimization or decision making is used. In this paper, we first present related work in In teractive Evolutio nary Co mputation, as well as an eye -tracking system and how it can b e used with evol utionary algorith ms. Next, we present the application we have developed to si mulate this approach (Interactive On e-max pro blem). We finish by presenting some results and futu re work. 2. IEC RELATE D WORK IEC is an optimization tech nique based on evolutio nary computation (genetic algorithm, genetic p rogramm ing, evolutio n strategy, or evolu tionary programming) and used when it is hard or impossible t o formalize efficiently the fitness functio n (th e method that gives the performance of a solut ion to a given problem) and where the fitn ess function is th erefore replaced by a human user. F or instance, IEC is o ften used for op timization of subjective criteria such as aesth etics. A large survey of more than 250 papers can be obtained in [1 6], b ut the generally accepted first work on IEC i s Dawkins [5 ], who stud ied the evolutio n of creatures called “biomorphs” by se lecting them m anually. Subsequ ently, much work was d one in t he area o f computer graphics: for instan ce usin g IE C for op timizing li ghting conditi ons for a given impression [ 1] , applied to fashion design [9] , or transforming drawing sketches into 3 D models rep resented by superqu adric functio ns and implicit surfaces, and evolving them b y usin g divergence o perators (ben ding, twisting, shearin g, tapering) to m odify the inpu t drawing in order to converge to more satisfactory 3D pieces [ 12] . We can also mention work in combining human in teractions w ith an artificial ant, applied t o non-p hotorealisti c renderi ng [1 5]. Another use o f IEC involves a human patient using a P DA on which an IEC is launched to define b est parameter values for co chlear implants [ 2] . First results show that patients using PDAs o btain a better parameterization t han previously thro ugh lengthy interaction with a do ctor. F ollowing t he same idea of using o ther human senses for human in teraction, we can also mention the o ptimization of coffee blends [ 7] . As mentioned before, IEC is used when a fitness functio n is difficult and sometim es im possible to formalize. Human-Based Genetic Algorithms (HBGA ) go furth er by allowing evolutio nary computation where a good representati on of individu als i s h ard or impossible t o find [3 ], for instance they can be u sed in storytelling or in develop ment of marketing slogans. To pro ve the usefulness of such techniqu es, the authors changed t he classical One-Max optimization problem int o an interactive one by in terpreting th e individu als (strings of bits – 0 or 1) as co lors to b e in teractively presented and manipulated. We use the same app roach to test ou r propo sition. Characteristics of IEC are inconsisten cies of ind ividuals fitness values given by the user, slo wness of the evolutio nary computation due to the in teractivity, and fatigue of th e user due to the obligation to ev aluate m anually all the i ndividuals of each generation [14 , 1 6]. For in stance, most often th e user is asked to give a mark to each in dividual o r to select the most promising individu als accordi ng: it still requ ires active time consuming participatio n during the interaction . The number of individ uals o f a classical IEC i s abo ut 20 (the maximum th at can be represented on th e screen), and abo ut the same for the nu mber of generations. However, som e tricks are used to overcom e those lim its, e.g., trying to accelerate the convergence of IEC by sho wing the fitness landscape mapped in 2 D or 3D, and b y asking the user to determine where the IEC should search for a better op timum [6 ]. Other work tr ies to predict fitness values of new ind ividuals based on previous sub jective evaluation. This can be done either by constructin g and appro aching the su bjective fitness function of th e user b y using genetic programming [4] o r neur al networks, o r also with Sup port Vector Machin e [10, 1 1]. In the latter case, inconsisten t respo nses can also be detected t hanks to graph based modeling. Nonetheless, p revious work is mostly algorithmic-oriented and not r eally u ser-oriented, which seems to be the future domain for IEC [13 , 16] . In the next section , we will present material that can be co mbined with Interactive Evo lutionary Computatio n in order to significantly reduce the active participati on of the user durin g the evaluation p rocess and to con sequently reduce considerab ly the fatigue of the user and the slowness of IEC ap proaches. 3. EYE-TRACKING EVOLUTIONARY ALGORITHM (E-TEA) 3.1 What is an eye-tracking sys tem? An ey e-tracking system consists of following th e eye’s motion s while a user wa tches a screen on which something is p resented. It pinpo ints in real time the p osition where the eye is lo oking, with the help of one or two video cam eras f ocusin g on a re flected infrared ray sent to th e user’s cornea (cf. F igure 1). This d evice coupled with a computer regularly samples the space p osition of the ey e an d the p upil diameter. This latt er parameter lets us kno w the co gnitive int ensity of the user: th e more th e user is concentrated on looking at something, the smaller the diameter [8] . Nowaday s, ey e-tracking systems are very u seful because t hey can an alyze in real time what a user i s focused o n withou t any effort an d in a completely non-r estrictive manner, in fact, the user does not kno w he is bein g o bserved b y the machine. With such equipment, on e can f inally captu re when, h ow much time, and with which cognitive int ensity a screen area is looked at. 3.2 How to us e an eye-tracker in IEC? If we consider that either phenotype or genotype of individuals are graphically disp layable on a screen, we can easily envisage using an eye-tracker d uring the evaluation process of IEC. Our propo sal consists in u sing th is h ypothesis: the more an in dividual is examined, the better the fitness of this particular in dividual will be. So, a new evolut ionary algori thm called E ye-T racking Evolutio nary A lgorith m (E-TEA ) is pro posed: 1. generate initial po pulatio n; 2. present the p opulati on to the user; 3. let the user watch th e individu als 4. compute how much ti me, how many times and with which cogniti ve intensity the p resented individ uals are looked at th anks to an eye-tracker; 5. combine previously obtained parameters an d compute a fitness for each individu al; 6. select the most pro mising individ uals from th e computed fitness 7. make crossover and mutation 8. return to step 2 unti l n o further good individuals are found Thus, the user ju st has to watch the screen and say when h e has finished watching/evaluating. There is no need f or the user to mark each ind ividual, no r to choo se th e b est o r the most promising one. This will save consid erable time and th e u ser will be capable evaluating more solutio ns consequen tly there will be more evaluated generation s. At a minimum, we estimate to d ouble the number of generations. The p rincipal difficulty is to determine how to combine di ff erent parameters obtain ed b y the eye-trac ker in ord er to define a computab le fitness. 3.3 Estimated fitness formalization As seen in pr evious sections, an eye-tracker like Tobii  is able to provide at l east 3 parameters for a screen region: – let d be t he time the user has focused o n a screen region; – let t be the number of transitions towards a particular screen region; – let p b e the average of the pu pil diameter when t he user has focused on a screen region. If we con sider a screen region as an individual and if we sup pose that the more an individual is observed, th e better will be its fitness, we can define an estimated fitness of the region as: (1) p t d f u γ β α + + = ˆ Unfortunately, α , β , γ values have to be defined empirically. In order to verify ou r hypoth esis, we have conducted some experiments. 4. APPLICATION TO THE INTE RACTIVE ONE-MAX OPTIMIZATION PROBLEM Our optimization problem will be borrowed from [ 3] where th e One-Max problem is considered as an interactive optimization problem in order to compare Interactive Genetic Algorithm (IGA) and Hu man-Based Genetic Algorithm (HBGA), and also in order to demonstrate the advantages o f usin g HBGA. Recall that t he video camera tracking left eye infrared ray video camera tracking right eye Figure 1: How w ork s an eye-track er like Tobii     ? classical One-Max op timization p roblem consists in maxim izing the number of 1s in a strin g of bit s (0 o r 1 ). I t is the simplest optimization problem and it is used here in ord er to parameterize our system . In the n ext paragraph, we will verify whether one-max optimization could b e adap ted to RGB colors. Then we present our in teractive one-max problem. 4.1 One-max optimization vs. color optimization In th is section, we try to sh ow th at one-max op timization is rather equivalent to white color o ptimization in t he RGB model even if it is not the best ch oice. Three distances for an objective fitness have been pro posed [ 3]: (2) B G R B G R M + + = ) , , ( 1 (3) − × = 3 255 ) , , ( 2 B G R M 2 2 2 ) 255 ( ) 255 ( ) 255 ( B G R − + − + − (4) ) , , min( ) , , ( B G R B G R M S = We have studied th e fitness-d istance-correlatio n between each of the p revious distances and the Hamm ing d istance (number of 1s in the stri ng). With 40 00 samples, we found that FDC(M 1 ) ≈ - 0.59 , FDC(M 2 ) ≈ -0.57 an d FDC(M S ) ≈ -0.48 . This means that M 1 , representin g the brightn ess, o r M 2 , rep resenting the E uclidean distance between the consid ered and the white colo rs, are both correlated. Thus, on e-max o ptimization can be adapted to interactive op timization by cho osing the brighter colo r. 4.2 Implementation As an eye-tracker is still very expen sive, we have simulated such equipment with t he help of a mouse. In fact, we ask the user to move the mouse to where he is lo oking. We know th is is t edious, but it i s the onl y way to simulate a Tobii  . Unfortu nately, it is impossible to o btain values o f th e th ird parameter p . Ho wever, we think it is no t u nreasonabl e as a test. With t his restrictio n, we have developed an app lication in Java 1.6 based on t he ECJ library 1 . Rather th an op timizing the simple one-max problem, we have decided to sho w i ndividu als as colors [3] . I ndividu als are represented b y a strin g of 24 bi ts, 8 bi ts each for red, green and blue. As we c aptu re simulated eye motion, the screen presents only 8 zones (o ne indi vidual per zo ne) and no in dividual in the center o f the screen as sh own in Figur e 2 . We avoid presentin g solutio ns in the center b ecause eyes are natu rally attracted to the center. Also, if t he user wants to co mpare two solutio ns that are diametrically op posite, ey es are obl iged to cross the center. Consequ ently, the n umber o f transitio ns for the center will increase considerab ly an d will disrupt th e estimated fitness o f the solutio n which co uld b e in th e center. When th e user estimates he has finished watching soluti ons of a generation, we give him t he possib ility to click o n h is preferred color among the 8 pr esented. In th at case, the estimated fitn ess is empirically cubed. The user also has the possib ility to choo se none of them. Thus, in F igure 2, we can see that durin g only the first 9 iteratio ns colors a re converging towards bright er co lors. 1 http ://www .cs.gmu.edu/~ eclab/proj ects/ecj/ Consequ ently, th e estimated fitness we used for the j th ind ividual depend s whether the u ser has cho sen it and is defined as: (5) ∑ + ∑ ∑ + ∑ = = = = = 8 .. 1 8 .. 1 8 .. 1 8 .. 1 2 2 3 2 2 : ? ) ( ˆ i i j i i j i i j i i j d d t t d d t t u chosen j f Equatio n (5 ) is equivalent to equati on ( 1) but we have normalized u f ˆ in [ 0,1] . If solutio n j is chosen , the first term is used, o therwise the secon d term. 4.3 Results For the moment, it is difficult to give significantly quan titative results in so far as the app lication developed is o nly restricted to the use of a mouse and movements the user woul d give to it in order to si mulate an eye-tracker. It is tediou s work, b ut, we can say that it i s easier to only move th e mou se than to choo se and click o n the most promising individ uals, or t o evaluate them. In the future, it shou ld be faster because int eractions woul d be only with th e eyes of th e user. We estimate doub ling, at a minimum the number o f iterations in th e Interactive Evo lution ary Compu tation exploring a larger search space. 5. DISCUSSIONS The Eye-Tracking Evolut ionary Algorithm is a very simple but very inno vative p roposi tion that is at the in tersection of two different domains: co mputer and cogni tive scien ces. This approach presents many advantages: – First, it is th e first ti me that an eye-tracker takes a very active part in a compu ter app lication. Mo re trad itionally, eye- tracking system s are used for analyzing hu man b ehavior when lo oking at an image, a text, a 3D model, a webpage, etc. – Second , with such a combinatio n we automate interactive evaluation of individuals with no constrain ts f or the user. The on ly thing he has to do is to watch individu als and t o say when he has finished. There is no explicit task imposed on the user, and th us no additio nal fatigue. – Next, such material is co mpletely n on-in trusive, i.e., the user could forget that h e is being observed. In teractive evaluation is as natur al as possib le. – Finally, by analyzing the cognitive activity of the u ser, we can easily detect when the u ser sho ws signs o f fatigue. Fo r 1 2 9 3 4 5 6 8 7 Figure 2 : Screenshots of our In teractive One-m ax optim ization pro blem (num bers repr esents the generation ) instance, when th e n umber o f transitio ns b etween ind ividuals is serio usly decreasing or when th e total time u sed to watch a generation is also decreasing, th ere is a ch ance that the user is bored. A pause can b e made an d t he in teractive evolutio nary algorith m can be resumed l ater. Ho wever, the time used to watch indi viduals could be interpreted differently: th e user is qu ickly converging toward a very good solu tion. More r esearch has to be done to detect this fatigue. Of course, each n ew system h as its drawbacks, bu t th ey are few compared to t he advantages: – The eye-tracker can follo w eyes if and on ly if it has b een calibrated to the user. Ho wever, this takes o nly few seconds, and th e user just has to focu s on con centric moving circles. – The oth er small con straint is that th e user does no t have total freedom of head movement. Fo r instance, h e can not lo ok away and th en resume evaluat ing. However, th e freedom is large en ough (3 0x16x20 cm) because of the use of two video cameras. I f the signal is lost for one eye, t he eye-tracker uses the oth er eye. 6. CONCLUSION & FUTURE WORK In this article, w e have presen ted a n ew algorit hm that should considerab ly improve th e speed of In teractive Evolutio nary Computatio n. To do so , we h ave presented the Eye-Tracking Evolutio nary Algorith m (E-TEA) that u ses an eye-track er in ord er to mini mize user i nteraction for evaluating ind ividuals. We have tested the appr oach by simulatin g an eye-tracker with a mou se durin g an interactive one-max op timization p roblem. The user had to move the mouse exactly to where h e is interested by an individ ual. The o nly difference with a real eye-tracker is th e loss of crucial information abo ut cognit ive inten sity represented by th e pupi l diameter. Noneth eless, w e are convin ced that time taken durin g the evaluatio n process can be significantly reduced . In th e future, we will first create an applicatio n interfacing the interactive one-max pro blem and a real eye-tracke r in ord er t o correctly parameterize ou r in teractive evolu tionary algorith m. Next, we want to test it on a real world appl ication. 7. ACKNOWLEDGMENTS We would like to thank the Institute of Te chno logy at Nice University (htt p://www.iut-nice.fr) which let this research work be possib le and also to th ank Pr. Peter S ander for hi s precious help. 8. REFERENCES [1] Aoki K, Takagi H. 3 -D CG lighting with an interactive GA. 1st Int ernation al Co nference Con ventiona l and Know ledge- Based In telligen t Electroni c Systems . Adelaide, Australi a 1997 :296 -301. [2] Bo urgeois-Repu bliqu e C, Vali giani G, Collet P . An Interactive Evolu tionary Algorithm for Coch lear Implant Fittin g: First Result s. In: al. HHe, ed. S AC05, ACM Symposiu m on Applied Compu ting . Santa Fe, New Mexico 2005 . [3] Ch eng CD, Koso rukoff A. Interactive One-Max P roblem Allows to Compare th e Performance of Interactive and Human-Based Genetic Algorithms. Gen etic and Evoluti onary Co mputa tion (GECCO) . Seattle, Washin gton, USA 2004 :983 -94. [4] Co stelloe D, Ryan C. Genetic Programming for Sub jective Fitness F unction Identi fication. In: LNCS, ed. Gen etic Progra mming 2 004: 259-6 8. [5] Dawkins R. The Blind Watchmaker 198 6. [6] Hayashida N, Takagi H. Acce leration of EC con vergence with landscap e visualization and h uman interventio n. App lied Soft Computi ng . 2 002; 1 :245 . [7] Herdy M. Evolutio nary optimization based on subjective selection - Evolving bl ends of coffee. 5th Europ ean Cong ress on Int elligent Techniqu es and soft comp uting 1997 :640 -4. [8] Just MA, Carpenter P A. The int ensity dimension o f thought: Pup illometric in dices of sentence p rocessing. Ca nadia n Journa l of Exp erimental Psycholog y, Specia l Issue: Rea ding and langu age p rocessing . 1993 ; 47 :310 -39. [9] Kim H-S, Cho S-B . Applicatio n of interactive geneti c algorithm to fashion design. Engin eering Ap plicat ions of Artificial Intellig ence . 200 0; 13 :635 . [10 ] Llorà X, Sastr y K. Combating User Fatigu e in iGAs: Partial Ordering, S uppo rt Vecto r Machin es and Syntheti c Fitness. GECCO . Washin gton, D.C ., USA 200 5:13 63 - 7 0. [11 ] Llorà X, Sastr y K, Alias F. Analyzing Active Interactive Genetic Algorithms using V isual Analytics. Genetic a nd Evoluti onary Co mputa tion (GECCO) . Seattle, Washin gton, USA 2006 :141 7-8. [12 ] Nishino H, Takagi H, Saga S, Utsumiya K. A virtual modeling system for intuitive 3 D shape con ceptualizatio n. IEEE In ternatio nal Co nference on Systems, Man and Cybernetics 2 002. [13 ] Parmee I. http ://www .ad-comtech.co .uk/Workshop s.htm (accessed Date 200 7). [14 ] Semet Y. Interactive Evolu tionary Computati on: a sur vey of existing theo ry. University of Illinois 2 002. [15 ] Semet Y, O’Reilly U-M, Duran d F. An In teractive Artificial Ant Approach t o Non -photo realistic Rend ering. Geneti c and Evoluti onary Co mputa tion (GECCO) . Seattle, Washin gton, USA 2004 :188 -200. [16 ] Takag i H. In teractive Evolut ionary Computatio n: Fu sion o f the Capaciti es of EC Optimization and Hu man Evaluation. IEEE 2 001:1 275- 96.