BrailleBand: Blind Support Haptic Wearable Band for Communication using Braille Language

BrailleBand: Blind Support Haptic W earable Band f or Communication using Braille Language Savindu H.P . 1 , Iroshan K.A., Panangala C.D., Perera W .L.D.W .P ., De Silv a A.C. (MIEEE) Abstract — V isually impair ed people ar e neglected from many modern communication and interaction procedur es. Assistive technologies such as text-to-speech and braille displays are the most commonly used means of connecting such visually impaired people with mobile phones and other smart devices. Both these solutions face usability issues, thus this study focused on de- veloping a user friendly wearable solution called the ‘Braille- Band’ with haptic technology while preser ving affordability . The ‘BrailleBand’ enables passive reading using the Braille language. Connectivity between the BrailleBand and the smart device (phone) is established using Bluetooth protocol. It consists of six nodes in three bands worn on the arm to map the braille alphabet, which ar e actuated to giv e the sense of touch corresponding to the characters. Three mobile applications were developed for training the visually impaired and to integrate existing smart mobile applications such as navigation and short message service (SMS) with the device BrailleBand. The adaptability , usability and efficiency of reading was tested on a sample of blind users which reflected progr essive results. Even though, the reading accuracy depends on the time duration between the characters (character gap) an a verage Character T ransfer Rate of 0.4375 characters per second can be achieved with a character gap of 1000 ms. I . I N T RO D U C T I O N The global completely blind population is estimated to be 40 to 45 million and nearly 135 million are estimated to hav e lo w vision. The cause for blindness for youth is mainly due to birth defects in the brain or the eye, and uncorrected refractiv e errors. While visual impairment and blindness due to infections hav e significantly reduced with the rapid progress of health care services, there is a notable increase in blind and visually impaired people ov er 65 years of age due to long life expectanc y . Unfortunately , the blind population is expected to double by 2020 [9]. Sight is the main human sense which possess the main influence on perception of all sensations, in collaboration with other senses such as hearing. Therefore, the lack of sight is the greatest challenge the blind face in performing their daily tasks such as navigation, information access, interpersonal interac- tions and safety . Hence, the blind are unemployed and depri ved of the privilege of education under normal circumstances. Approximately 75% of the blind are unemployed while only 10% of the blind children receive special braille education [1]. In the context of the modern society a blind person and his/her family faces many socioeconomic problems. Consequently , *This work was supported by the University of Moratuwa, Faculty of Engineering. 1 Savindu H.P . is with the Department of Electronic and T elecommuni- cation Engineering, Faculty of Engineering, University of Moratuwa, 10400 Katubedda, Sri Lanka. savinduherath@gmail.com the need for assistiv e technologies which enable the blind to li ve independent, productiv e and better lives emerged as in vesting on nursing homes, blind welfare, health care and blind care specialists were perceiv ed to be costly and unsustainable solutions [11]. A. Existing technologies and their dr awbacks Many research projects hav e been conducted focusing on independent communication for the blind, the deaf and the deaf-blind communities. W earable systems such as mechanical hands for automated fingerspelling and communication glov e systems hav e been de veloped using various alphabets [5]. The Lorm glove uses the Lorm alphabet, a form of communication for deaf and blind. It translates the hand-touch Lorm alphabet to text and vice versa using flat fabric sensors embedded on the glove [7]. DB-HAND is a similar wearable device with Malossi alphabet interface for the deaf-blind community . This glov e is equipped with sensors and actuators enabling two way communication with pressing of tactile sensors enabling the inputs to the device [3]. In another braille based mobile com- munication and translation glove, uses capaciti ve touch sensors and actuators to enable two way communication by translating text to braille patterns and vice versa [4]. W earabraille is a keyboard for the deaf and blind which uses finger mount accelerometers to identify the tapping action of fingers in a similar fashion to typing in a traditional keyboard [8]. Howe ver , the main drawback of glov e systems is that they limit the use of hands for other acti vities since the sense of touch for communication is giv en to the palm or the fingers and it gets interrupted if hands are being used for another activity . T ext-to-speech is another mode of assisting the blind. How- ev er , it possesses usability issues such as priv acy issues, disturbs hearing which is the main sense of blind people etc. Even if a pair of headsets is used it permanently disconnects the visually impaired person from the surrounding en vironment increasing the chance of meeting with accidents. Moreov er , Braille terminals and displays carry an exorbitant cost, hence unaffordable. B. Proposed solution The hypothesis of the proposed solution is to use the sense of touch of the arm to transfer information to a blind person using a wearable band having six vibration nodes corresponding to the standardized braille dot code. The system developed to test this hypothesis integrated with a smart device was named as the ‘BrailleBand’. W earing a band on the arm is similar to wearing a wrist- watch, hence does not hinder the use of the hand for other essential tasks. Moreover , wearable assistiv e solutions are preferred than the portable assiti ve solutions such as braille displays and mobile phones (text to speech) as they provide hands-free con venient interactions. Assistiv e technologies basically provide the ability to dis- abled people to work independently without having a need for someone else attending. The main focus areas of assisti ve technologies are [2]: 1) Information transmission Issues with regard to information transmission are read- ing, character identification and interpreting two and three-dimensional graphics.The most successful and most popular blind reading tool is braille dot code which is the same dot code used in the BrailleBand making it readily usable for the blind community for information transmission. T actile displays and braille displays enable character and graphic recognition by transferring infor- mation to tangible forms to be felt by the blind. 2) Mobility assistance/Na vigation Mobility and navigation in volves information of the immediate dynamic en vironment and obstacle avoid- ance. The basic functionality of all electronic travel aids (ET As) is scanning the surrounding en vironment and transferring the gathered information through other senses to the user . BrailleBand can transfer directional data through sense of touch to the user and gi ve feedback on the traveling directions. 3) Computer and smart device accessibility W ith the increased use of computers and smart devices the need of the visually impaired to access smart de vices emerged. The solutions currently lev eraged are text-to- speech and braille displays. BrailleBand can transfer information on simple computer screens to blind users using the vibrating braille dot patterns and subsequent to the development of the replying system two way communication will be established with computers and smart de vices. The product BrailleBand caters the needs of all the above three areas and its scope of applications are broader and can be customized according to di verse user needs (refer to Fig. 1). The study showed promising results with blind users being able to read mobile phone text messages and tweets using the BrailleBand. Also, the study rev ealed increase in reading speed and accurate character identification as the usage increased. I I . M E T H O D O L O G Y A. Hardwar e implementation T o represent nodes of the univ ersal Braille dot code, six haptic motors (8mm V ibration Motor - 2mm T ype from Preci- sion Microdriv es T M ) along with haptic motor controllers were used to generate the vibration sensation to be felt at the forearm Fig. 1. The BrailleBand corresponding to the characters to be read. Refer to Fig. 3 for a fe w samples of Braille dot code. The Fig. 2 summaries the hardw are implementation of the Braileband. A Rechargeable 1200 mAh Li-Ion battery was used as the source of power to the whole system which through a 5V V oltage booster circuit. The battery was charged through the USB type-C port using a Battery Management 0.8A single input battery charger . The microcontroller used was A TMEGA328P-A U which interacts with the Bluetooth module and I2C multiplex ers to generate the required signals corresponding to the information receiv ed from the smart de vice via Bluetooth. Haptic motor controllers are I2C controllable. Howe ver , they hav e the same static I2C address and therefore I2C multiplexers were used. The circuit was designed using Altium T M and the enclosure of the prototype product was de veloped using Solidworks T M which was realized through three-dimensional (3D) printing. Fig. 2. Interconnections and the functionality of the modules Fig. 3. Braille alphabet and braille numbers (www .pharmabraille.com) B. V ibration motor placement and sense of touch When testing the hypothesis of this study , placement of vibration motors on the forearm was a critical task. Main aspects considered were: 1) T wo-point discrimination threshold (TPDT) 2) Axes of vibration TPDT is a measure of the minimum distance between two pressure points on the skin to be felt as two distinct points. TPDT varies along body due to une ven distrib ution of mechanoreceptors. As BrailleBand is worn on the forearm the TPDT on the forearm which is 40mm is critical for design specifications [6]. The axis of vibration of the motors were horizontal and Eccentric Rotating Mass (ERM) haptic motors were used to generate vibrations. Even though the sense of touch is a supplementary sensory modality to a healthy-sighted person, it is the primary sensory modality for a blind person for non-audible information. Braille is a prime example of the success of tactile mode of transferring information. The skin is responsible for the sense of touch through three types of sensory modalities: 1) Thermoreceptors - for thermal sensations 2) Nociceptors - pain sensing 3) Mechanoreceptors - mechanical sensations due to exter - nal forces There are four types of mechanoreceptors in the human skin and out of them Meissner senses touch while Pacini corpuscles sense vibration. Since the BrailleBand stimulus is vibration our interest was on Meissner and Pacini sensors [10]. • Meissner - high amplitude lo w frequency stimuli • Pacini corpuscles - low amplitude high frequency stimuli The frequencies and the amplitudes of the haptic motors were determined in such a manner that they stimulate both Pacini and Meissner vibration sensors in the skin to enhance the sense of vibration. C. Software implementation 1) Connectivity: Mobile applications which are connected with the de vice are dev eloped using android. By selecting the Media Access Control (MA C) address of the Bluetooth module of the de vice (HC06), application can establish a socket connection between the BrailleBand and the smart phone. After the establishing the connection, a thread is created in the application which handles the socket connection. When the thread is started, the input stream and the output stream is set with the BrailleBand. The gaps between characters and words are controlled by changing the threads to postDelayed state. The program continuously read the input stream and control haptic motors depending on the received character . 2) Mobile applications: Three mobile applications were dev eloped for the BrailleBand: 1) BrailleBand T eacher Used to get the blind person familiarized with the de vice and to teach the method of reading braille from Braille- Band. 2) BrailleBand Messenger Enables the user to read text messages received to the mobile phone through the BrailleBand. 3) BrailleBand Na vigator This application can navigate a blind person to a desti- nation by giving haptic feedback about the direction and the distance to travel through the BrailleBand. Howe ver , the accurac y of na vigation still needs to be developed. The existing social media and utility mobile applications such as facebook, twitter , linkedin and google maps can be integrated with the BrailleBand application and is readily usable with the BrailleBand device since information from these applications can be con ve yed to the blind user through the de vice. Refer to Fig.4. An android library was also developed and is readily a vail- able at https://github .com/BrailleBand/BrailleBand-mobile-app to facilitate other de velopers to de velop mobile applications customized to dif ferent user requirements. Hence, the product BrailleBand is open for inno vations. Fig. 4. BrailleBand mobile applications and integration of existing applica- tions with product BrailleBand 3) Microcontr oller pr ogramming: The microcontroller is programmed using Arduino software to con vert the character which is being receiv ed from the bluetooth module to the respectiv e braille pattern. Then using the I2C communication protocol the microcontroller addresses the desired haptic mo- tors and activ ate them. D. T esting and verification Three experiments were conducted to test and validate the hypothesis which resulted in the following parameters: 1) Reading speed 2) Character transfer rate (CTR) 3) Device usability score A sample of 10 blind subjects (7 male and 3 female) were chosen from the Blind School, Rathmalana, Sri Lanka within an age range of 16 to 19 years based on the criteria of having experience with Braille language for more than 5 years. No initial training was giv en to the students but 15 minutes were taken to familiarize the students with the device using the BrailleBand T eacher mobile application. The sample was tested with the device where 3 character words were sent with different character gaps. The critical parameter which gov erns the Reading Speed and the CTR is the Character gap and hence we focused on reducing the character gap and identifying the minimum character gap which does not significantly differ statistically with the character gap which giv es the best average reading accurac y (90.1%) using Analysis of V ariance (ANO V A), Bonferroni and Holm tests. It was found that the minimum character gap satisfying the above conditions is 1000 ms. Refer to T able I. C T R = number of char acters sent time tak en (1) An exact CTR cannot be calculated since the number of dots to be vibrated vary with characters. Hence, a minimum (for “q”) and a maximum (for “a”) CTR was calculated and a av erage CTR was deri ved by using (2). C T R av er age = C T R maximum + C T R minimum 2 (2) The subject was giv en a score ranging from 0 to 10 to rate the usability of the BrailleBand based on the preference to adopt and use this de vice for communication purposes. I I I . R E S U L T S A. Development of the BrailleBand BrailleBand is a blind support wearable which is light weight, portable and easy to use. Moreover , it mitigates the usability and affordability issues associated with the current solutions av ailable. Blind support wearable solutions are still progressing in the experimental stages and hence the market is open with minimum competition. BrailleBand caters a broader scope of user needs such as information transmission, na vi- gation and mobility assistance, accessibility to computers and smart de vices. B. Reading speed test The sample was tested with random words consisting of three characters and the character gap was reduced as in T able I and the following reading accuracies were noted. W ith decreasing character gap the reading speed increases as characters are being transmitted faster . T ABLE I R E AD I N G S P EE D T E S T R E SU LT S Character gap(ms) A verage reading accuracy(%) Standard deviation 2000 90.1 10.94 1500 90.1 6.24 1200 87.9 6.24 1000 87.9 12.11 800 69.1 18.03 500 53.2 21.02 400 46.4 22.84 It is notable that with practice the reading speed can be improv ed to a greater extent. C. Character T ransfer Rate (CTR) A dot in the braille cell is vibrated for a period of 300 ms and the gap between the vibration of two consecuti ve dots is also 300 ms. Therefore, for a single dot to be vibrated it will take 600 ms. An ANO V A test can be done to check whether the av erage reading accuracies obtained in T able I are statistically signifi- cantly dif ferent with a confidence le vel of 95%. T ABLE II O N E - W AY A N OV A F O R 7 I N DE P E N DE N T C H A RA CT E R G A P S source sum of squares df mean square F statistic p-value treatment 21,168.37 6 3,528.06 15.1239 0 error 14,696.5 63 233.28 - - total 35,864.87 69 - - - The conclusion of the ANO V A test as tabulated in T able II is that the p-value corresponding to the F-statistic of one-way ANO V A is lower than 0.05 (95% confidence), suggesting that the sample data of one or more treatments (character gaps) are significantly dif ferent to the others. Therefore, a multiple comparison method should be used to assess the statistical significance of the dif ferences between the means of the sample data under different treatments. Hence, Bonferroni and Holm multiple comparison tests follow to test only pairs relati ve to a certain treatment (character gap) to verify statistical significant dif ferences. Since the char - acter gap 1500 ms gives the highest reading accuracy (90.1%) while having the least standard de viation (refer to T able I), we choose character gap = 1500 ms as the treatment against which other treatments are compared for statistical significant differences. From the above results table of Bonferroni and Holm mul- tiple comparison tests, we can observe that there is statisti- cally insignificant difference between the sample data under Character Gap=1500 ms, Character Gap=2000 ms, Character Gap=1200 ms, and Character Gap=1000 ms. Therefore, we can use the character gap of 1000 ms for Character Transfer Rate (CTR) calculations as the sample data do not sho w a statistically significant dif ference with the T ABLE III B O NF E R RO N I A N D H O LM R E SU LT S : O N LY PA IR S R E L A T IV E T O C H A RA CT E R G A P = 1 5 0 0 M S S I M U L TAN E O U SLY C O MPA R ED T reatment pairs Bonferroni and Holm T -statistic Bonferroni inference Holm infer- ence 1500 Vs 2000 0.00 Insignificant Insignificant 1500 Vs 1200 0.32 Insignificant Insignificant 1500 Vs 1000 0.32 Insignificant Insignificant 1500 Vs 800 3.07 Insignificant Significant 1500 Vs 500 5.40 Significant Significant 1500 Vs 400 6.40 Significant Significant treatment which giv es the highest reading accuracy of 90.1% (Character Gap=1500 ms). Therefore, we can calculate the maximum CTR which is corresponding to the character ”a” and minimum CTR corre- sponding to the character ”q” using Equation 1. C T R maximum = 1 0 . 6 + 1 = 0 . 625 character s s − 1 C T R minimum = 1 0 . 6 × 5 + 1 = 0 . 25 character s s − 1 Therefore, from Equation 2, C T R av er age = 0 . 625 + 0 . 25 2 = 0 . 4375 character s s − 1 It is observed that the average CTR is 0.4375 characters per second and this also can be improv ed with practise. D. Usability scor e All the subjects were requested to rate the usability of the BrailleBand out of 10 and the resultant av erage usability score was 8.7. I V . D I S C U S S I O N BrailleBand caters a wider range of blind support needs and it has many applications such as reading, navigation and smart device accessibility . It is a complete package of the device and the mobile applications for smart phones. Howe ver , the replying system of the BrailleBand is still under de velopment. The capaciti ve touch pads and capacitive touch controllers are the constituents of the replying system which allows the user to reply back to messages using the same braille dot pattern. Refer to Fig. 2. Additionally , further research should be done on the health and safety aspect of the product as well. The go-to-market plan should include a market research to identify the additional features demanded by the users and to identify the price the users are willing to pay . Moreover , long term sustainable usability of the product should also be v erified as BrailleBand will be a lifetime product for a blind person focusing on robustness of the device and biological effects as well (for example: nerve adaptations for vibrations). The reading speed and the CTR can be further improv ed with practise. Moreov er , from the device end we can reduce the dot vibration time which is currently at 300 ms to a lesser value and transmit characters faster . But the compromise here is the effect of the vibrating sensation to the user . As blind people usually lack financial strength to afford so- phisticated devices. Therefore, the BrailleBand will be a better solution. Y et, the price of BrailleBand should be subsidized through corporate and government funding to reach the wider blind community . W earable assisti ve technology solutions for blind are still at an emer ging lev el with lot of research and development being done around the world. BrailleBand integrates haptics in de veloping an assistiv e de vice for blind support. V . C O N C L U S I O N S The blind community uses the sense of hearing and sense of touch to interact and understand the surrounding environ- ment. Hence, the sense of touch becomes the primary sensory modality to communicate non-audible information to and from a blind person. The ‘BrailleBand’ haptic wearable blind support de vice con- nected to smart phone applications helps the blind community to lead an independent quality life. This assistiv e technology enables information transmission, na vigation and smart de vice accessibility through the sense of touch. W ith the development of the BrailleBand we have successfully implemented a new mode of communication to the blind using haptic technology . The product feasibility tests conducted showed promissing results with regard to reading speed and reading accuracy which impro ved with practice. Based on e xperimental results an average CTR of 0.4375 characters per second was achie ved. AC K N OW L E D G M E N T W e e xtend our gratitude to Dr . Ajith Pasqual and all the academic staff of Department of Electronic and T elecommuni- cation Engineering, University of Moratuwa for veteran assis- tance and guidance gi ven in developing the product Braille- Band. W e acknowledge the Blind School, Rathmalana, Sri Lanka for accommodating the testing and verification process by granting patient access. R E F E R E N C E S [1] Blind W orld Magazine (2006). Br eaking the chains of paternalism [online]. A vailable online at: http://home.earthlink.net/ ˜ blindworld/NEWS/6- 06- 14- 02.htm . [2] J Brabyn, KD Seelman, and S Panchang. Aids for people who are blind or visually impaired. An Intr oduction to Rehabilitation Engineering, ed. RA Cooper , H. Ohnabe, and DA Hobson , pages 287–313, 2007. [3] Nicholas Caporusso. A wearable malossi alphabet interface for deafblind people. In Pr oceedings of the working conference on Advanced visual interfaces , pages 445–448. A CM, 2008. [4] T anay Choudhary , Saurabh Kulkarni, and Pradyumna Reddy . A braille- based mobile communication and translation glove for deaf-blind people. In P ervasive Computing (ICPC), 2015 International Conference on , pages 1–4. IEEE, 2015. [5] Ben A G Elsendoorn. Assistiv e technology for the hearing-impaired, deaf and deafblind, by marion e. hersch and michael a. johnson, eds. T echnology and Disability , 16(2):111–113, 2004. [6] SR Geiger . Handbook of physiology , Section 1: The nervous system . American Physiological Society , 1984. [7] Ulrike Gollner , T om Bieling, and Gesche Joost. Mobile lorm glov e: introducing a communication de vice for deaf-blind people. In Pr oceed- ings of the Sixth International Conference on T angible, Embedded and Embodied Interaction , pages 127–130. ACM, 2012. [8] JA Miele. W earabraille: dev elopment of a wireless, virtual, braille keyboard. in 25th Annu. Int. T echnology & Persons with Disabilities Conf. (2010). [9] W orld Health Organization et al. V isual impairment and blindness. fact sheet no. 282. updated august 2014, 2014. [10] R. Sekuler and R. Blake. P erception . Number v . 1 in Perception. McGraw-Hill, 2002. [11] Ramiro V el ´ azquez. W earable assistiv e devices for the blind. In W earable and autonomous biomedical devices and systems for smart envir onment , pages 331–349. Springer, 2010.