Wrist Sensor Fusion Enables Robust Gait Quantification Across Walking Scenarios

Wrist Sensor Fusion Enables Rob ust Gait Quantification Acr oss W alking Scenarios Zeev W aks, Itzik Mazeh, Chen Admati, Michal Afek, Y onatan Dolan, A vishai W agner Intel - Advanced Analytics {zeev.waks, itzik.mazeh, chen.admati, michal.afek, yonatan.dolan, avishai.wagner}@intel.com Abstract Quantifying step abundance via single wrist-worn accelerometers is a common approach for encouraging activ e lifestyle and tracking disease status. Nonetheless, step counting accuracy can be hampered by fluctuations in walking pace or de- meanor . Here, we assess whether the use of various sensor fusion techniques, each combining bilateral wrist accelerometer data, may increase step count rob ustness. By collecting data from 27 healthy subjects, we find that high-le vel step fusion leads to substantially improv ed accuracy across div erse walking scenarios. Gait cycle analysis illustrates that wrist de vices can recurrently detect steps proximal to toe-off ev ents. Collectiv ely , our study suggests that dual-wrist sensor fusion may enable robust gait quantification in free-li ving en vironments. 1 Introduction The growth of the quantified self mo vement has sparked substantial interest in measuring the e xtent of physical activity an indi vidual performs. One widespread approach for tracking daily activity is by counting walking steps. This is often accomplished using wrist-worn pedometers containing triaxial accelerometers. Beyond the public health value of increased ambulation, step counting interventions yield modest weight loss [1], and may aid in diabetes, cardiov ascular disorders, and COPD care [2]. Despite their pre valence, commercial wrist-w orn pedometers often display mark edly inferior accuracy compared to their counterparts located on other body parts, for instance the waist or ankles [ 3 , 4 , 5 ]. Indeed, wrist-based step counting does not appear robust to walking pace and other walking style variations, with double digit error rates being fairly common [ 3 , 5 , 6 ]. In fact, it has been argued that step counts deri ved using accelerometers w orn on the waist and wrist are generally not comparable under both laboratory and free-living conditions [7], with w aist placement being superior [5]. Body-worn micro electromechanical system (MEMS) inertial sensors including accelerometers and gyroscopes can also be used to quantify gait abnormalities in mov ement disorders. This is primarily achiev ed in controlled settings by detecting specific gait phases, especially the heel strike (initial contact) and toe-of f (terminal contact) ev ents. For example, inertial sensors ha ve been used to track cadence inconsistencies and axial symptoms such as step time asymmetry in patient cohorts [8, 9]. In free-living environments, wrists would presumably serve as attracti ve sites for monitoring gait pathologies gi ven the wide adoption of wrist-worn devices. Ho wev er , due to arm mo vement, use of individual wrist-worn sensors for measuring lo wer -limb, time-dependent aspects such as initial foot contact and terminal contact is challenging. W e hypothesized that concurrent use of opposing wrist sensors, via sensor fusion, may help ov ercome the limitations of single-wrist devices. Sensor fusion refers to the combining of sensory data such that resulting information may be better than that from individual sources (re vie wed in [ 10 ]). Algorithms used for synchronous multi-sensor fusion include, among others, av eraging, Borda count voting, fuzzy logic, Kalman filters, and 31st Conference on Neural Information Processing Systems (NIPS 2017), Long Beach, CA, USA. inference methods. Sensor fusion architectures generally fall under three categories: lo w-lev el (raw data), intermediate-lev el (features), or high-le vel (combining decisions, in this paper detected steps). Perhaps surprisingly , there are limited examples of sensor fusion using body-worn inertial sensors. Indeed, multi-sensor studies often compare the outputs of individual sensors rather than perform fusion to improv e a measure of interest. Existing fusion examples typically focus on acti vity classification via intermediate-le vel fusion, for e xample by classifying daily acti vities [ 11 ] or detecting falls [ 12 ]. Sensor fusion for gait quantification is scarce, with one e xception using ankle, thigh, and waist sensors, but not wrist de vices, and focusing on a single high-le vel fusion approach [13]. Gi ven the lack of precedent, methodical comparison of multiple fusion approaches for gait quantifica- tion would be of high v alue, especially using wrist sensors. The present study e xplores this possibility by comparing sensor fusion with single sensor results. Using eight types of walking tasks, we assess whether dual-wrist sensor fusion can facilitate rob ust step counting and gait phase detection. 2 Data collection and labels Data was collected from 27 healthy volunteers (18 male, 9 female) aged between 18 and 50. Subjects receiv ed instructions and subsequently performed eight separate walking tasks that simulate multiple walking types (T able 1). A total of N=214 tasks were successfully collected. T asks were performed while wearing six synchronized, wireless de vices, each containing a 128Hz triaxial accelerometer and 128Hz triaxial gyroscope (Opal sensors, APDM). The six devices were placed on both ankles, both wrists, the lumbar spine, and the trunk, per manufacturer configuration. The entire process including instructions, placement, and remov al of the de vices was performed in less than 30 minutes. Each walking task consisted of a 74.6m uninterrupted walk in a flat, rectangular course (33.5m X 3.8m) with a mean duration of 64.3s [range 30.4s-110.6s]. Subjects self-counted their steps, with a mean step count of 104.4 steps [range 67-151]. These counts were not used for training or ev aluation. Ankle gyroscope signals were used to deri ve step count labels and to determine heel strike and toe-of f timings [ 14 , 15 ]. The method, implemented by the device manufacturer (APDM), is considered highly reliable and has been previously used as a label [ 3 ]. Gait e vents were identified by matching a template, specifically a double pendulum model for le g swing and an in verse double pendulum for stance (foot on ground). Segments analyzed by the manufacturer softw are, accounting for the lar ge majority of gait cycles, were used for step count e v aluation. T o further ensure accuracy , we discarded the 5% of samples with the lar gest cadence difference between subject step counts and ankle-deriv ed val ues. This increased the correlation between the measures from r=0.81 (N=214) to r=0.96 (N=203). T able 1: Data collection walking tasks. Sample sizes shown after outlier remo v al. Category W alking type (N) Unconstrained Slow pace (25), Comfortable pace (26), F ast pace (24) Arms constrained Holding bag in right hand (27), Holding cellphone with two hands (26), Hands not swinging alongside body (27) Asymmetrical W ithout right shoe (25), pretending to use cane in right hand (23) T otal 203 3 Methods This work e valuates no fusion (single side), low-le vel fusion, and high-lev el fusion step detec- tion approaches. All approaches first con verted axis-lev el raw data into signal magnitude (norm) p x 2 t + y 2 t + z 2 t , followed by smoothing with a centered moving av erage. Peak detection was used for step counting as pre viously recommended for non-fixed accelerometers gi v en its high performance [ 16 ]. Peaks were detected using first-order dif ference with parameters for minimum peak height and minimum windo w between adjacent peaks [ 17 ]. In the case of multiple peaks within the minimum window , the highest peak was chosen. Each peak was considered a single step. All parameters were tuned by five-fold cross validation using all N=203 samples by minimizing the root mean square error (values sho wn in T able 2). This resulted in negligible smoothing for a fe w of the approaches. 2 T able 2: Mean parameter values of the six algorithms using fi v e-fold cross v alidation. Minimum peak amplitude values are for signal v alues ∈ [0, 1], min-max normalized using data from all samples. Parameter Algorithm No fusion Low-lev el fusion High-lev el fusion L R Sum Dif f Intersect Union Moving average smoothing Single sensor window (sec) 0.03 0.03 0.18 0.02 0.02 0.40 Fused signal window (sec) - - 0.08 0.02 - - P eak detection Min peak amplitude (range 0-1) 0.27 0.36 0.15 0.08 0.24 0.06 Min window between peaks (sec) 0.36 0.34 0.23 0.40 0.34 0.32 Event fusion Max window for peak fusion (sec) - - - - 0.32 - Min window between peaks (sec) - - - - - 0.29 The six step detection approaches are listed below . W e denote N L and N R the signal magnitude (norm) of left and right sensors after smoothing, respectiv ely . W e denote T L and T R the set of times that match the peaks of N L and N R , respectiv ely , as identified by peak detection. No fusion (two approaches, one per side): Peaks from only one sensor were used, either T R or T L . Low-lev el fusion (two approaches): Fusion was performed at the raw data level after signal magnitude smoothing. Sum low-le vel fusion, N R + L = M ov ing Av g ( N R + N L ) , or dif ference lo w-le vel fusion, N | R − L | = M ov ing Av g ( | N R − N L | ) , were followed by peak detection to detect steps. T R + L and T | R − L | are the sets of times that match the peaks of N R + L and N | R − L | . High-level fusion (two approaches, see algorithms 1, 2): High-le vel fusion consisted of signal smoothing, peak detection, and fusion (intersection or union) of detected peaks. Fusion parameters included the maximum allow able distance max _ dist to be considered an intersection (upper bounded by the peak detection minimum windo w parameter), or the minimum allo wable distance min _ dist between adjacent peaks (union). In both cases, the peak with the highest amplitude was selected. Algorithm 1 High-Lev el Intersection Input: N R , N L , T R , T L and max _ dist Output: T R ∩ L (set of times) 1: T R ∩ L = ∅ 2: for t l ∈ T L do 3: if ∃ t ∈ T R : | t l − t | ≤ max _ dist then 4: t r = ar g min t ∈ T R ( | t l − t | ) 5: if ∀ t ∈ T L | t l − t r | < | t − t r | then 6: t r ∩ l = ar g max t r , t l ( { N R ( t r ) , N L ( t l ) } ) 7: T R ∩ L = T R ∩ L ∪ t r ∩ l 8: end if 9: end if 10: end for 11: return T R ∩ L Algorithm 2 High-Lev el Union Input: N R , N L , T R , T L and min _ dist Output: T R ∪ L (set of times) 1: S = T R ∪ T L 2: T R ∪ L = ∅ 3: while S 6 = ∅ do 4: P R ∪ L = N R ( T R ∩ S ) ∪ N L ( T L ∩ S ) 5: p i = max ( P R ∪ L ) 6: T R ∪ L = T R ∪ L ∪ t i 7: S = S/ { t : | t − t i | ≤ min _ dist } 8: end while 9: return T R ∪ L 4 Results 4.1 Step count accuracy Each one of the six algorithms were each trained using fi v e-fold cross validation. T raining was not per-task, b ut rather used all samples in order to e valuate algorithm rob ustness to different indi viduals and walking types. W e observed that signal summation (lo w-le vel fusion) or union of peaks (high- lev el fusion) were better at quantifying step abundance than single side algorithms as measured by 3 the percent error (Figure 1a). Union fusion also had substantially higher correlation to the label (ankle step count) than single sensor methods (union, r=0.98; left sensor , r=0.78; right sensor , r=0.88), with 90% of the samples having a small step count error rate between -2.2% and 2.5% (see boxplot whiskers). Raw data fusion using dif ference of signal magnitudes performed poorly . Specific walking type comparisons of the better no fusion algorithm (right side) with the best fusion approach (high-le vel union) highlighted the potential for rob ust step counting using fusion (Figure 1b). While both methods performed well in multiple w alking tasks, particularly where at least one arm was constrained, the no fusion algorithm was less accurate in fast or slow w alking and in simulations of pathological gait (cane). In contrast, the robustness of union high-lev el fusion is strikingly evident. Figure 1: Union fusion has the lowest step counting error among sensor fusion approaches. Boxplot whiskers capture 90% of data (5th and 95th percentiles). Negati ve error is under counting. Left: Overall step count error . Right: Comparison of union (high-le vel) fusion with the better no fusion algorithm (only right sensor) per task sho ws the union approach is robust to div erse types of walking. 4.2 Gait phase detection The gait phase detected by all methods was expectedly similar as all methods share the same peak detection technique (data not sho wn due to space limitations). Focusing specifically on union (high- lev el) fusion, we observ ed that peaks were identified relati vely consistently adjacent to the toe-of f phase (Figure 2). The low v ariation in ∆ t of fset relativ e to toe-of f suggests that wrist-de vice fusion may enable quantification of gait asymmetries and cadence inconsistencies to a certain degree. Figure 2: Gait phase detection by the peaks of the union (high-le vel) fusion approach. Left: Offset from heel strike and toe-of f for all samples. Right: Offset from toe-of f per walking task. 5 Conclusions Our work has two primary contributions. First, we illustrate that mathematically simple fusion of wrist accelerometer output can ov ercome the shortcomings of single-wrist step counters. Further work is required to assess whether this holds in free-li ving en vironments. The second major implication of our work is the potential for dual wrist-devices to capture timing-based, gait abnormalities. As above, additional work can help e v aluate this promise in patient populations. Acknowledgments W e thank all volunteers that participated in the study . W e also thank Amitai Armon, Or Shimshi, Nir Darshan, Jeremie Dreyfuss, Gilad W allach, and Abraham Israeli for insightful discussions. 4 References [1] Caroline R Richardson, T if fan y L Newton, Jobby J Abraham, Ananda Sen, Masahito Jimbo, and Ann M Swartz. A meta-analysis of pedometer-based walking interventions and weight loss. The Annals of F amily Medicine , 6(1):69–77, 2008. [2] David R Bassett, Lindsay P T oth, Samuel R LaMunion, and Scott E Crouter . Step counting: A revie w of measurement considerations and health-related applications. Sports Medicine , 47(7): 1303–1315, 2017. [3] Fabio A Storm, Ben W Heller , and Claudia Mazzà. Step detection and activity recognition accuracy of se ven ph ysical activity monitors. PloS one , 10(3):e0118723, 2015. [4] Hannah M Husted and T amra L Llewellyn. The accurac y of pedometers in measuring walking steps on a treadmill in college students. International Journal of Exercise Science , 10(1):146, 2017. [5] Jessica J Chow , Jeanette M Thom, Michael A W ewege, Rachel E W ard, and Belinda J Parmenter . Accuracy of step count measured by physical acti vity monitors: The effect of gait speed and anatomical placement site. Gait & P osture , 57:199–203, 2017. [6] Ming-De Chen, Chang-Chih Kuo, Christine A Pellegrini, and Miao-Ju Hsu. Accuracy of wristband acti vity monitors during ambulation and activities. Medicine and science in sports and exer cise , 48(10):1942–1949, 2016. [7] Catrine T udor -Locke, T iago V Barreira, and John M Schuna Jr . Comparison of step outputs for waist and wrist accelerometer attachment sites. Medicine and science in sports and exer cise , 47 (4):839–842, 2015. [8] Silvia Del Din, Alan Godfrey , Brook Galna, Sue Lord, and L ynn Rochester . Free-li ving gait characteristics in ageing and parkinson’ s disease: impact of en vironment and ambulatory bout length. J ournal of neur oengineering and rehabilitation , 13(1):46, 2016. [9] Anat Mirelman, Hagar Bernad-Elazari, A vner Thaler, Eytan Giladi-Y acobi, T anya Gure vich, Mali Gana-W eisz, Rachel Saunders-Pullman, Deborah Raymond, Nancy Doan, Susan B Bress- man, et al. Arm swing as a potential new prodromal marker of parkinson’ s disease. Movement Disor ders , 31(10):1527–1534, 2016. [10] W ilfried Elmenreich. Sensor fusion in time-triggered systems. 2002. [11] Lei Gao, AK Bourke, and John Nelson. Evaluation of accelerometer based multi-sensor versus single-sensor acti vity recognition systems. Medical engineering & physics , 36(6):779–785, 2014. [12] JD López, C Ocampo, A Sucerquia, and JF V arg as-Bonilla. Analyzing multiple accelerometer configurations to detect falls and motion. In VII Latin American Congr ess on Biomedical Engineering CLAIB 2016, Bucaramanga, Santander , Colombia, October 26th-28th, 2016 , pages 169–172. Springer , 2017. [13] E Fortune, V A Lugade, Shreyasee Amin, and K enton R Kaufman. Step detection using multi- versus single tri-axial accelerometer -based systems. Physiological measurement , 36(12):2519, 2015. [14] Arash Salarian, Heike Russmann, François JG V ingerhoets, Catherine Dehollain, Yves Blanc, Pierre R Burkhard, and Kamiar Aminian. Gait assessment in parkinson’ s disease: toward an ambulatory system for long-term monitoring. IEEE transactions on biomedical engineering , 51 (8):1434–1443, 2004. [15] Kamiar Aminian, B Najafi, C Büla, P-F Leyvraz, and Ph Robert. Spatio-temporal parameters of gait measured by an amb ulatory system using miniature gyroscopes. Journal of biomechanics , 35(5):689–699, 2002. [16] Agata Brajdic and Robert Harle. W alk detection and step counting on unconstrained smart- phones. In Pr oceedings of the 2013 ACM international joint confer ence on P ervasive and ubiquitous computing , pages 225–234. A CM, 2013. 5 [17] Lucas H Negri. P eakUtils 1.1.0 , 2017. URL https://pypi.python.org/pypi/PeakUtils . 6