Cognitive Assessment Estimation from Behavioral Responses in Emotional Faces Evaluation Task -- AI Regression Approach for Dementia Onset Prediction in Aging Societies

Cognitiv e Assessment Estimation fr om Beha vioral Responses in Emotional F aces Ev aluation T ask - AI Regr ession A ppr oach f or Dementia Onset Pr ediction in Aging Societies - T omasz M. Rutkowski † , Masato S. Abe † , Marcin K oculak ‡ , and Mihoko Otake-Matsuura † † RIKEN Center for Advanced Intelligence Project (AIP), T ok yo, Japan http://aip.riken.jp/ tomasz.rutkowski@riken.jp ‡ Consciousness Lab, Institute of Psychology , Jagiellonian Univ ersity , Krakow , Poland Abstract W e present a practical health-theme machine learning (ML) application concerning ‘ AI for social good’ domain for ‘Producing Good Outcomes’ track. In particular , the solution is concerning the problem of a potential elderly adult dementia onset prediction in aging societies. The paper discusses our attempt and encouraging preliminary study results of beha vioral responses analysis in a w orking memory- based emotional ev aluation experiment. W e focus on the dev elopment of digital biomarkers for dementia progress detection and monitoring. W e present a be- havioral data collection concept for a subsequent AI-based application together with a range of regression encouraging results of Montreal Cogniti ve Assessment (MoCA) scores in the leav e-one-subject-out cross-validation setup. The regressor input v ariables include e xperimental subject’ s emotional v alence and arousal recog- nition responses, as well as reaction times, together with self-reported education lev els and ages, obtained from a group of twenty older adults taking part in the reported data collection project. The presented results sho wcase the potential social benefits of artificial intelligence application for elderly and establish a step forw ard to de velop ML approaches, for the subsequent application of simple behavioral objectiv e testing for dementia onset diagnostics replacing subjective MoCA. 1 Introduction Dementia, especially the age-related memory decline, is one of the most significant global challenges in the 21 st century’ s mental well-being and social welfare. W orldwide, the increased longevity and mainly for elderly adults of abov e 65 years old, dementia numbers, and costs are rising [ 8 ]. The Cabinet Of fice in Japan announces annual reports on an aging society to address the dif ficulty . United Nations Sustainable Development Goal #3 entitled “Good Health and W ell-being” also stresses a necessity to address the aging problem with a focus on healthy li ves, and it promotes well-being for all at all ages. Recent approaches to dementia and Alzheimer’ s disease (AD) patient support suggest a necessity to develop personalized therapies relying not only on traditional pharmacological interventions b ut also on lifestyle modifications [ 2 ] as well as cognitiv e support approaches [ 10 ]. There is also a social expectation for the dementia early-onset prediction and subsequent prophylactics steps, as broadly discussed in [ 8 ]. All the classical pharmacological and the novel ‘beyond-a-pill, ’ or the so-called ‘digital-pharma, ’ therapeutical interv entions require trustful biomarkers, which would offer a comfortable alternati ve to more advanced in application brainwav e-related techniques [ 6 , 11 ], which usually require a more clinical-le vel en vironment for a successful application. W e propose a AI for Social Good workshop at NeurIPS (2019), V ancouv er , Canada. machine-learning (ML) approach, belonging to a broad spectrum of AI for the social or common good, which allows for automatic and objective estimation of a cognitiv e decline. A self-reported working-memory decline, the so-called subjecti ve cognitiv e impairment (SCI), is one of the early biomarkers used in the medical community [ 4 ]. A mild cognitiv e impairment (MCI), often preceding a dementia onset, is also characterized by emotional contagion [ 5 ] and hippocampus atrophy related spatial memory problems [ 8 ]. On the other hand, there is no clear evidence about the working- implicit/procedural-memory impairment, and only the long-term equiv alent is kno wn to be unaf fected in dementia subjects [ 12 , 5 ]. The contemporary methods for dementia diagnostics rely on pencil- and-paper subjecti ve psychometric e valuations, for example, the Montreal Cognitiv e Assessment (MoCA) [ 7 ], the more complex physiological or imaging analyses [ 6 ], or massiv e multi-sensory datasets [ 3 ]. The latter methods often require expensiv e devices, very long recording periods, or clinical settings. W e dev elop a machine-learning-based biomarker to replace the traditional MoCA, which utilizes beha vioral responses in the spatial and working-implicit/procedural-memory testing task. Other work has been done predicting MCI using non-in vasi ve data, but it has relied on subjecti ve self-reports [ 5 ]. W e improv e on their work by collecting behavioral data, which is less biased, when carrying out predictions. W e present an experimental and subsequent ML/AI behavioral response analysis approach in which we ask elderly subjects to learn a reasonable new emotional faces (Mind Reading database [ 1 ]) ev aluation skill using a two-dimensional map, a so-called emoji- grid [ 13 ], of arousal and valence scores, which is an effortless spatial- and implicit-working-memory task. After a short training, the subjects perform a testing trial in which response time, arousal and valence user inputs together with self-reported age and education le vel features are used to train ML models as described in the follo wing methods section. First, we conduct a pilot study with a small sample of univ ersity students, middle-age, high- ( MoCA > 25) and low-scoring ( 20 6 MoCA 6 25 in this study) on MoCA-scale elderly , as well as reference ‘super -normal’ acti ve-seniors of 80+ years old. The pilot study produces encouraging results of reaction time (R T) and emotional v alence/arousal (V A) behavioral responses, recorded with a touchpad, as biomarker candidates. The results of the pilot study inspire the main project in this paper with 20 elderly MoCA-e valuated subjects. W e report on promising regression results allowing for estimation of MoCA le vels from beha vioral responses and without the necessity for very subjects paper -and-pencil tests. 2 Methods W e conducted experiments with human subjects with guidelines and approval of the RIKEN Ethical Committee for Experiments with Human Subjects in the Center for Advanced Intelligence Project (AIP). In the experimental session, twenty elder participants (number of females = 11 ; mean age = 76 . 5 years old; age standard de viation = 4 . 95 ; recruited from Silver Human Resources Center) took part. All participants gav e informed written consents, and they receiv ed a monetary gratification for their participation in the study . Each subject experiment consisted of 72 video presentation trials ( 5 ∼ 7 seconds each) with 24 dif ferent emotion categories [ 1 ]. Three different videos portrayed e very emotion with actors differing in age, gender , and skin color . The order of the videos was randomized before the experiment b ut was the same for ev ery participant. During the data recording experiments valence and arousal responses, as well as the reaction times were recorded together by the stimulus presentation application dev eloped in a visual programming en vironment MAX by Cycling ’74, USA . W e calculated absolute response errors and reaction times as, v e ( i, s ) = | v d ( i ) − v t ( i, s ) | , a e ( i, s ) = | a d ( i ) − a t ( i, s ) | , r t ( i, s ) = t t ( i, s ) − t o ( i ) , with s = 1 , . . . , 20 identifying a participant in our study; i = 1 , . . . , 72 representing the video clip sho wn in a trial number i ; v e ( i, s ) and a e ( i, s ) as valence and arousal errors related to emotional stimulus i and subject s , respectiv ely; v d ( i ) and a d ( i ) were the video clip assigned ground truth emotional scores from [ 1 ]; v t ( i, s ) and a t ( i, s ) the actual response inputs by a user number s on a touchpad after the video clip number i , which reflected the learned emotion e v aluation in the spatial- and working-memory task; r t ( i ) was a reaction time obtained as an interval between user response t t ( i, s ) and the i th video clip end at a timestamp t o ( i ) . A single video clip e v aluation feature vector F i,s related to video clip i and participant s for each ev aluated next regressor in training and subsequent leave-one-subject-out cross-v alidation procedure has been built as, F i,s = [ v e ( i, s ) , a e ( i, s ) , r t ( i, s ) , e ( s ) , g ( s )] , where e ( s ) ∈ { 0 , 1 } denoted a self- reported education lev el, and g ( s ) an age of 65 ∼ 80 years old in this study . A pairwise comparison scatter-plots of input features, together with linear regression fits, are summarized Figure 1a. W e tested regressors av ailable in the scikit-learn library version 0 . 21 . 3 [ 9 ] for continues prediction of MoCA values characterizing cognition stages of the 20 participants in our study using input features 2 20 30 MoCA 0 50 valence error 0 50 arousal error 5 10 reaction time 70 80 age 20 30 MoCA 0 1 education level 0 50 valence error 0 50 arousal error 0 10 reaction time 60 80 age 0 1 education level (a) Pairwise plots of beha vioral responses and MoCA HuberR linearR linearSVR rbfSVR polySVR RFR Tested regression methods 2.5 2.0 1.5 1.0 0.5 0.0 0.5 1.0 1.5 2.0 2.5 Median prediction error Median regression error ditributions of MoCA level prediction from behavioral respones in elderly adults in leave-one-subject cross-validation (b) Regression median error results Figure 1: Panel (a) presents a collection of pair-plots sho wing relationships of output/target (MoCA) and input features used in the subsequent regression analysis. Red lines depict linear regressi on fits with shaded confidence intervals of the pairwise data distributions. Panel (b) summarizes median regression errors from all the methods tested (nai ve mean regressor resulted in +1 . 5 median errors). F i,s described previously . W e implemented the leave-one-subject-out cross-validation procedure for our proposal e valuation. The following methods and appropriate steps were implemented with default parameters set to as the scikit-learn, except for changes mentioned next to the methods: Huber regressor ( HuberR ); linear regressor ( linearR ); linear support vector regressor ( linearSVR ); radial basis function support vector re gressor ( rbfSVR ): with a kernel coef ficient γ = 1 / 7 representing an in verse of feature vector F i,s length; polynomial support vector re gressor ( polySVR ): with a degree set to 3 , also here γ = 1 / 7 , an independent term in k ernel function coef 0 = 1 . 0 ,  = 0 . 10 ; random forest regressor ( RFR ): with maximum depth set to 10 , and a number of estimators equal to 200 . A binary classification trial for MCI ( MoCA 6 25) versus normal cognition subjects resulted with median leav e-one-subject out cross-validation accuracies of 99% for LD A, LR, RFR; 98% for linear SVM; 92% for sigmoid SVM; 89% and 83% for RBF and polynomial SVMs, respecti vely (chance lev el of 50% ). 3 Results and Conclusions The current project resulted in encouraging results with a sample of 20 older adults confirming a possibility of regression-based prediction of MoCA le vels using only subject behavioral responses in the spatial- and implicit-working-memory task. The study results are summarized in the form of behavioral feature distributions and prediction errors in Figures 1a and 1b, respectiv ely . W e expect that after collecting a more extensi ve database in near-future, a deep learning application would allo w for an ev en more successful lowering of re gression errors in MoCA predictions. The study resulted in MoCA pathology prediction from behavioral responses in the spatial- and implicit- working-memory task of emotional valence and arousal le vels estimation together with reaction times in simple video clips watching task. In the study inv olving older adults with known MoCA scores, we were able to e valuate se veral shallow learning regressors. The regression-based prediction of MoCA scores (usually MoCA 6 25 has been considered as mild cogniti ve impairment (MCI) stage already , while abov e this threshold an elderly adult cognition has been e valuated as standard) in the simple emotional f aces e valuation task resulted in rob ust and small errors as summarized in Figure 1b. The presented nov el approach for behavioral responses in emotional faces ev aluation task in volving spatial- and implicit-working-memory-based ne w skill acquisition together with linear 3 MoCA regression-based prediction results of fer a step forward in research and development of no vel dementia-related behavioral biomarkers for elderly adults, for whom possible early diagnosis of cognitiv e decline, as well as a life improv ement, are essential. The successful application of such AI/ML-based dementia onset prediction shall lead to a healthcare cost lowering benefiting the aging societies. W e also acknowledge the potential limitations of the current approach as we only infer human-error-prone MoCA scores, which are only proxy estimators of dementia. AI-based dementia estimators, if used without proper ev aluation, might also pose a danger of misuse or abuse; thus, proper ethical standards will need to be in place too. In the next step of our research project, we plan to e valuate the de veloped methods with a lar ger sample of ordinary v ersus SCI/MCI, or e ven dementia diagnosed members of the society . W e also plan to combine the proposed behavioral measures with neurophysiological, especially EEG and fNIRS signals for e ven more solid final classification. W e also belie ve that future in volv ement of AI methods for fully interacti ve stimuli in closed-loop user behavior and brainw av e monitoring shall lead to ev en more impactful results. References [1] S. Baron-Cohen. Mind Reading - The Interactive Guide to Emotions . Jessica Kingsley Publishers, London, UK, 2004. URL http://www.jkp.com/mindreading/ . [2] D. Bredesen. The End of Alzheimer’ s: The F irst Pr ogramme to Pre vent and Reverse the Cognitive Decline of Dementia . V ermilion, 2017. [3] R. Chen, F . Jankovic, N. Marinsek, L. Foschini, L. Kourtis, A. Signorini, M. Pugh, J. Shen, R. Y aari, V . Maljk ovic, et al. Developing measures of cogniti ve impairment in the real world from consumer -grade multimodal sensor streams. In Pr oceedings of the 25th A CM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 2145–2155. ACM, 2019. [4] S. Emrani, D. J. Libon, M. Lamar, C. C. Price, A. L. Jef ferson, K. A. Gifford, T . J. Hohman, D. A. Nation, L. Delano-W ood, A. Jak, K. J. Bangen, M. W . Bondi, A. M. Brickman, J. Manly , R. Swenson, and R. Au. Assessing working memory in mild cogniti ve impairment with serial order recall. J. Alzheimer’s Dis. , 61 (3):917–928, 2018. ISSN 18758908. doi: 10.3233/J AD- 170555. [5] A. Gutchess. Cognitive and Social Neur oscience of Aging . Cambridge Fundamentals of Neuroscience in Psychology . Cambridge Univ ersity Press, 2019. doi: 10.1017/9781316026885. [6] A. Horvath, A. Szucs, G. Csukly , A. Sakovics, G. Stefanics, and A. Kamondi. EEG and ERP biomarkers of Alzheimer’ s disease: a critical revie w . F rontier s in Bioscience , 23:183–220, 2018. [7] P . Julayanont, Z. Nasreddine, M. Brousseau, M. Borrie, H. Chertkow , and N. Phillips. The Montreal cognitiv e assessment memory index score (MOCA-MIS) and total MOCA score to help predict mci con version to Alzheimer’ s disease. Alzheimer’ s & Dementia: The Journal of the Alzheimer’s Association , 8(4):P372, 2012. [8] G. Livingston, A. Sommerlad, V . Or geta, S. G. Costafreda, J. Huntley , D. Ames, C. Ballard, S. Banerjee, A. Burns, J. Cohen-Mansfield, et al. Dementia prev ention, intervention, and care. The Lancet , 390(10113): 2673–2734, 2017. [9] F . Pedregosa, G. V aroquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P . Prettenhofer , R. W eiss, V . Dubourg, J. V anderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay . Scikit-learn: Machine learning in Python. Journal of Machine Learning Resear ch , 12:2825–2830, 2011. [10] R. M. G. Reinhart and J. A. Nguyen. W orking memory revi ved in older adults by synchronizing rhythmic brain circuits. Nature Neur oscience , 22(5):820–827, 2019. doi: 10.1038/s41593- 019- 0371- x. URL https://doi.org/10.1038/s41593- 019- 0371- x . [11] T . M. Rutko wski, Q. Zhao, M. S. Abe, and M. Otake. AI neurotechnology for aging societies - task-load and dementia EEG digital biomarker dev elopment using information geometry machine learning methods. In AI for Social Good W orkshop at the Neural Information Pr ocessing Systems (NeurIPS) 2018 , pages 1–4, Montreal, Canada, December 8, 2018. URL https://aiforsocialgood.github.io/2018/pdfs/ track1/68- tomek_nips_2018_workshop- FINAL.pdf . [12] S. D. Slotnick. Cognitive Neur oscience of Memory , chapter Implicit Memory , pages 129–149. Cambridge Fundamentals of Neuroscience in Psychology . Cambridge University Press, 2017. doi: 10.1017/9781316026687. [13] A. T oet, D. Kaneko, S. Ushiama, S. Hoving, I. de Kruijf, A.-M. Brouwer , V . Kallen, and J. B. F . van Erp. EmojiGrid: A 2D Pictorial Scale for the Assessment of F ood Elicited Emotions. Fr ont. Psychol. , 9:2396, nov 2018. ISSN 1664-1078. doi: 10.3389/fpsyg.2018.02396. URL https://www.frontiersin.org/ article/10.3389/fpsyg.2018.02396/full . 4