A low-cost real-time 3D imaging system for contactless asthma observation

A L O W - C O S T R E A L - T I M E 3 D I M A G I N G S Y S T E M F O R C O N T A C T L E S S A S T H M A O B S E RV A T I O N A P R E P R I N T Sheona M.M.D.P . Sequeira, Beril Sirmacek Robotics and Mechatronics Univ ersity of T wente Enschede, The Netherlands b.sirmacek@utwente.nl Nov ember 5, 2019 A B S T R AC T Asthma is becoming a very serious problem with ev ery passing day , especially in children. Howe ver , it is very dif ficult to detect this disorder in them, since the breathing motion of children tends to change when they reach an age of 6. This, thus mak es it very difficult to monitor their respiratory state easily . In this paper, we present a cheap non-contact alternati ve to the current methods that are av ailable. This is using a stereo camera, that captures a video of the patient breathing at a frame rate of 30Hz. For further processing, the captured video has to be rectified and con verted into a point cloud. The obtained point clouds need to be aligned in order to hav e the output with respect to a common plane. They are then conv erted into a surf ace mesh. The depth is further estimated by subtracting ev ery point cloud from the reference point cloud (the first frame). The output data, ho wev er , when plotted with respect to real time produces a v ery noisy plot. This is filtered by determining the signal frequency by taking the Fast F ourier T ransform of the breathing signal. The system was tested under 4 dif ferent breathing conditions: deep, shallo w and normal breathing and while coughing. On its success, it was tested with mixed breathing (combination of normal and shallow breathing) and was lastly compared with the output of the e xpensiv e 3dMD system. The comparison showed that using the stereo camera, we can reach to similar sensiti vity for respiratory motion observation. The experimental results sho w that, the proposed method provides a major step to wards de velopment of low-cost home-based observ ation systems for asthma patients and care-givers. K eywords Computer vision · 3D human body · Respiratory motion · Asthma monitoring · Home monitoring systems 1 Introduction and Backgr ound The need for monitoring respiratory motion is increasing rapidly . There are a lot of people that tend to suffer from breathing disorders, asthma being the prime one. Asthma is a very serious condition in which it becomes e xtremely dif ficult to breathe, due to the airway passage becoming very narro w and swollen. This, thus prevents the proper flow of air into the lungs. This disorder , in turn results in the patient getting extremely tired fast, making it dif ficult for them to perform daily acti vities that a normal individual would do. Asthma is sadly a condition which cannot be healed permanently , but the intensity of the asthmatic attacks can be reduced by taking proper treatment and care. Howe ver , this condition is extremely dif ficult to be properly diagnosed in children [ 1 ]. This is due to the fact that children who sho w recurring symptoms of asthma at ages belo w 6 not necessarily will continue to suf fer from asthma later on in their liv es. Thus, this usually tends to go undiagnosed with the hope of it being a passing phase. While on one hand the children with mild asthma-like symptoms might no longer suffer from it after the age of 6 or more, children with more se vere asthmatic conditions often tend to still suffer . For this very purpose, monitoring the respiratory motion especially in children is vital. A P R E P R I N T - N OV E M B E R 5 , 2 0 1 9 There are many de vices that can be used for tracking respiratory motion [ 2 ]. These devices might be further categorized based on the motion of the body , that is, they can be in variant to rigid body motion or not. The devices that are in variant to body motion ideally assume that every motion that occurs in the patient is the motion due to respiration. On the other hand, the non-in variant systems basically in volv es only the monitoring of the chest motion from multiple cameras and then obtaining the depth of the re gion of interest. Some of the devices might also be classified as contact and non-contact de vices. An example of a contact based system that is in variant to rigid body motion would be a belt strapped around the chest region [ 3 ]. Bentur et al. [ 4 ] used contact-based for measuring ho w children with asthma are responding to medical therapy sessions. Their study underlines that our proposed system could also support medical practitioners or other researchers to monitor therapy response and ev olution of the disease. Another widely used commercial de vice which was introduced by LeoSound, also uses a contact-based method for the respiratory motion measurement during sleep [ 5 ]. Such devices, ho wev er , might both produce not completely accurate results since every motion is considered as respiratory motion, while, at the same time having an object strapped around the chest also causes a lot of discomfort, as a result of which it might actually mak e the person breathe inefficiently . For this reason, we propose to use a non-in variant contactless system. This can be done by obtaining a video using a stereo camera. The solution thereby reduces any of the ne gati ve outcomes of the systems that might hinder the breathing motion while at the same time being cheaper and very user friendly , thus enabling it to be used for tracking child respiratory motion at homes on a frequent basis. Massaroni et al. [ 6 ] proposed a contactless respiratory motion monitoring system based on usage of regular RGB cameras. Even though the system looks ef fecti ve for homebased usage simply using a webcam, the missing aspect of the 3D reconstruction would highly limit the system to measure 3D motion of the chest. Besides, since the approach works using the color bands of the RGB camera, the system would need white light in the sleeping room to guarantee good results. W e belie ve that our solution w orks more reliably under lo w light conditions (because the algorithm doesn’t rely on the color components of the images) while pro viding 3D chest motion which we con vert to respiratory signal for detecting abnormalities. 2 Methods For simple and handy utilization, we have implemented a system that only makes use of a stereo camera. A stereo camera is a passiv e camera that can be used to produce a depth image of the object under test since it in volves two individual cameras. These together provide a distinctiv e outlook on the same object much as ho w the human eye works. This significant property of the camera can be used to appropriately determine the depth of an object. The depth of an image as percei ved by the camera is directly proportional to the focal length, baseline of the camera and in versely proportional to the disparity . The camera can be used to ideally cover infinite depth, ho wev er as the depth increases the error rate increases; thus, maintaining a distance between 0.50 - 1m from the camera produces more accurate results since the breathing motion is more easily visible within this range. At the time of testing, it should be noted that the patient is able to breathe in two dif ferent manners: that is either through thoracic breathing or abdominal breathing. A video of the patient breathing should be recorded with a frame rate of 30Hz and a resolution of 480x1280. The sampling rate of the video for [ 7 ]. Having a sampling frequency as high as 30Hz can result in a large video size, thus halving the sampling rate might be a better solution. Attention should be paid to the extent to which the sampling frequency is lowered, since lo wering the sampling rate might af fect the quality of data by making it noisier . For specific analysis, a region of interest can also be selected. Howe ver , as mentioned earlier , by taking a region of interest, the analysis is being limited to either one of the breathing mechanisms which might thus result in inaccurate results. On the positive side, selecting a region of interest decreases the computation time greatly . 2.1 Stereo Camera Calibration Stereo camera calibration: Several images (20 images) of a 7x10 checkerboard with each of the checkerboard squares having dimensions of 10mm are taken from dif ferent angles and different aspect points. For the sake of calibration, the Stereo Camera Calibration App of Matlab is used wherein the improper images are directly eliminated thereby reducing the calibration error rate. The stereo parameters obtained are then used for processing the breathing frames. 2.2 3D reconstruction of chest and tracking respiratory motion For the purpose of tracking the respiratory motion of a patient, a video of about 30 seconds to 1 minute is first tak en by placing the stereo camera at a fixed distance of about 1m from the patient. Since the obtained video consists of frames from both the cameras of the stereo camera, the video should be split into two indi vidual videos. Each indi vidual video frame then has to be rectified. Image rectification is e xtremely important in order to reduce the trouble of finding matching points between the frames later on. Upon rectification, it is possible to determine the distance between alike 2 A P R E P R I N T - N OV E M B E R 5 , 2 0 1 9 Figure 1: Block Diagram on the process flow of the setup objects in the two images, which in turn helps in computing the disparity . It is very important to set the disparity range correctly in order to obtain a proper output. The disparity data needs to be filtered before it is further con verted into a 3D point cloud as can be seen in figure 2. The 3D point cloud is basically a collection of the three-dimensional coordinate system that is obtained from the disparity data in order to produce an accurate 3D digital output of the chest. For further processing of the data, the in valid points that are present in the point cloud need to be eliminated and the point cloud should be de-noised An extremely essential step that should be taken after obtaining the point cloud is aligning all the video frames along the same axis, thereby making the data suitable for tracking respiratory motion. This is done by using the Iterativ e Closest Point Algorithm [ 8 ]. The algorithm re-determines the closest point set and continues until it finds the local minimum match between the two surfaces. It works in a fe w stages, that is, identification of the closest model point for each data point and then finding the least square rigid body transformation relating these point sets. The initial frame is taken as the reference frame for aligning the point clouds to a common axis, thus obtaining the transformation angle. In order to obtain the depth data, the aligned 3D point cloud should then be con verted into a surface mesh. The depth information can be acquired by k eeping the initial frame as the reference frame (to be taken such that it is the intermediary stage between inhalation and exhalation) and subtracting the depth of ev ery subsequent frame from it. The obtained output will howe ver consist of a lot of noisy data apart from the input signal and needs to be filtered. The noise can be of v arious types including salt and pepper noise, quantization noise and white noise. This filtered output should then be plotted with respect to time. Based on the number of breaths that the person tak es and the frequency with which it occurs, the person’ s breathing condition can be determined. This can be done by determining the number of peaks in the output plot, where in each peak represents a maximum breath that is taken. Based on literature, it is seen that for children below the age of 6, the number of breaths should be between 22-34 breaths per minute while children between 6 to 12 years should be 18-30 breaths per minute [ 9 ]. If the patient(child) breathes more than 15-17 breaths or less than 10 breaths in 30 seconds a conclusion can be drawn about the breathing condition of the person being tested. A conclusion can be dra wn easily if a person is taking more breaths than required that the patient is showing symptoms of asthma. 3 Experimental setup and results The following materials ha ve been used for implementation and testing of this method: Stereo Camera: Non-Distortion Dual Lens USB3.0 Camera Module Synchronization HD 960P O TG UVC Plug Play Driv erless 3D VR Stereo W ebcam ( 75euro). Output of this camera is stereo images with a resolution of 960x1280 for each separate left and right image. Software: MA TLAB R2019a, including the Image Processing T oolbox. The accuracy of the system was tested by checking the precision with which the output plot was produced with respect to time and visually matching that output with the input video that w as taken of the patient. The input video taken was of a relativ ely small time frame since it was done in order to test the accuracy of determining respiration. The sampling rate for processing was also maintained at 30Hz. Reducing the sampling rate to 6.7Hz increased the amount of noise that was present in the output. 3 A P R E P R I N T - N OV E M B E R 5 , 2 0 1 9 Figure 2: Initial image processing (a) Image Rectification (b) T riangular surface mesh (c) disparity map (d) Point cloud After obtaining the initial output, the data needed to be further filtered. This was done by designing a band pass filter with the help of using the Fast F ourier T ransform (FFT). From the output of the FFT , the suitable frequency range for only the required breathing data was selected and the rest of the noise was thus ignored. The system was tested by having a v olunteer breathe in 4 dif ferent manners, including normal breathing, deep breathing, shallow breathing and coughing. For the deep breathing, the volunteer w as asked to inhale air very slo wly and then again exhale very slo wly . On the other hand, the volunteer when asked for shallo w breathing was asked to partially inhale and exhale very rapidly . This data served as input into the FFT , thus obtaining a clear output signal from which the breathing pattern can be determined. The number of breaths that are taken are displayed on a dialog box of the user interface (on a pc at the moment) in order to indicate the current health status. Howe ver , the frequency range that was selected for e very test k ept changing and thus could not be generalized to be within a specific range for each different breathing condition as can be seen in figure 4. The filtered output plots of the dif ferent breathing conditions can be seen in figure 3. The data for normal breathing was further also ev aluated by considering a small region of interest. This sav ed a lot of computation time, howe ver , the output produced by limiting the size of the region of interest to just a small black point on the chest resulted in a much noisier FFT output response. Thus, the region of interest was changed to the entire chest area. Another issue that also arose was that, the v olunteer switched between breathing mechanisms in the middle of the test run as a result of which when only a small region of interest was considered, there were certain cycles that were missed. This can be seen in figure 5. On successful establishment of the pre vious stages of testing, we performed a test where in a volunteer was ask ed to shuffle between normal breathing and shallo w breathing in order to accurately determine whether the system works ef ficiently well. The output of the test can be seen in the figure 6. In order to conclusi vely determine whether the designed system w as successful, we compared our output with that of the more e xpensiv e and precise medical system, the 3dMD. This was done by calculating the maximum expansion of the chest at the time of deep breathing from the intermediary state between inhalation and exhalation. Due to the extreme comple xity of doing this comparison in MatLab, the point clouds from the 3dMD and the stereo camera were entered into point cloud processing software called as CloudCompare [ 10 ]. The maximum depth of the 3dMD w as found to be 36.0661mm with an error rate of 1.783 while our alternati ve had a depth of 38.4005mm and an error rate of 1.542. In Figure 7 we can see the output image from CloudCompare for both the 3dMD and the stereo camera. A simple lo w cost system that is easy to use and comprehend was thus de veloped. Howe ver , there is still scope for more research when it comes to elimination of the noise, since the frequency range that could be used for determining the breathing pattern of the patient k ept changing with e very dif ferent test that was performed. The dif ference in the measurement that arose between the depth estimation of the 3dMD system and the Stereo camera could be due to either the difference in the induced noise(that could be due to the difference in the relativ e thickness of the clothes) or it could also be due to the camera calibration. The system should also be generalized to be independent of whether the first frame is the intermediate breathing position or not. This could be done by taking a cycle of breathing before starting the 4 A P R E P R I N T - N OV E M B E R 5 , 2 0 1 9 Figure 3: Breathing output for each test (a) long breathing (b) breathing while coughing (c) breathing while coughing (d) short breathing Figure 4: FFT for different breathing input signals Figure 5: Region of interest. Left: entire chest area. Right: limited region of interest. 5 A P R E P R I N T - N OV E M B E R 5 , 2 0 1 9 Figure 6: Mixture of breathing input. Left: Output signal. Right: FFT result. Figure 7: Depth comparison. Left: 3D reconstruction with stereo camera. Right: 3D reconstruction with 3dMD. data and further storing the reference position from that. The setup could also not be tested with children, b ut that’ s not a major problem since it can be solv ed easily by adjusting the maximum and minimum number of breaths that can be taken by the children in 30 seconds. 4 Conclusions and future w ork The output performance of this system w as v erified by testing under se veral breathing conditions and w as also compared with the benchmark system that is 3dMD, that produced a small depth dif ference of about 2mm. This system can be incorporated without hindering the persons breathing, in a completely non-contact manner . It can thus successfully be used to monitor the patients breathing on a re gular basis. This is especially required for children since after a certain age the child that used to hav e asthmatic symptoms might not necessarily continue to suffer from the same. Thus, parents can keep a check of their child’ s breathing and on unfortunate events can further go on to seeing a doctor . Conclusiv ely , with a little bit more research, this will be a very successful method for the detection of respiratory disorders like asthma. References [1] W im M. C. van Aalderen. Childhood asthma: Diagnosis and treatment. Scientifica (Cair o) , 2012. [2] Stefan W iesner and Zi v Y ani v . Monitoring patient respiration using a single optical camera. Annual International Confer ence of the IEEE Engineering in Medicine and Biology Society . IEEE Engineering in Medicine and Biology Society . Conference , 2007:2740–3, 02 2007. [3] Carlo Massaroni, Andrea Nicolò, Daniela Lo Presti, Massimo Sacchetti, Sergio Silvestri, and Emiliano Schena. Contact-based methods for measuring respiratory rate. Sensors , 19(4), 2019. [4] L. Bentur , R. Beck, M. Shinawi, T . Nav eh, and N. Gavriely . Wheeze monitoring in children for assessment of nocturnal asthma and response to therapy . Eur opean Respiratory Journal , 21(4):621–626, 2003. https: //erj.ersjournals.com/content/21/4/621 . 6 A P R E P R I N T - N OV E M B E R 5 , 2 0 1 9 [5] Ulrich K oehler , Olaf Hildebrandt, Andreas W eissflog, Angela Zacharasie wicz, Ke ywan Sohrabi, Niklas K oehler, and V olker Groß. Leosound - a new de vice for long-term recording of wheezing and cough in pediatric and adult patients with asthma (during sleep). The Eur opean r espiratory journal : official journal of the Eur opean Society for Clinical Respiratory Physiology , 21, 2018. [6] Carlo Massaroni, Daniela Presti, Domenico Formica, Sergio Silvestri, and Emiliano Schena. Non-contact monitoring of breathing pattern and respiratory rate via rgb signal measurement. Sensors , 19:2758, 06 2019. [7] Jochen K empfle and Kristof V an Laerhoven. Respiration rate estimation with depth cameras: An e valuation of parameters. Pr oceedings of the 5th International W orkshop on Sensor -based Activity Recognition and Interaction , (4):4:1–4:10, 2018. http://doi.acm.org/10.1145/3266157.3266208 . [8] D. L.G. Hill, P . G. Batchelor , M. Holden, and D. J. Ha wkes. Medical image registration. Physics in Medicine and Biology , 3(46), 2001. [9] Susannah Fleming, Matthew Thompson, Richard Stev ens, Carl Heneghan, Annette Plüddemann, Ian Maconochie, L. T arassenko, and David Mant. Normal ranges of heart rate and respiratory rate in children from birth to 18 years of age: A systematic revie w of observational studies. Lancet , 377:1011–8, 03 2011. [10] CloudCompare. https://www.danielgm.net/cc/ . 7