An Efficient Real Time Method of Fingertip Detection

An Efficient Real Time Meth od of Fingertip Detection Jagdish Lal Raheja*, K aren Das**, Ankit Chaud hary*** *Digital Systems Group, Central Electronics Enginee ring Research Institute (CEERI) / Council of Scientific and Industrial Research (CSIR), Pilani 333 031 INDIA (e-mail:ja gdish@ceeri.ernet.in) **Tezpur University, Assam, INDIA (e-mail: karenkdas@gmail.com) *** BITS, Pilani, INDIA (e-mail: ankitc.bitspila ni@gmail.com) Abstract— Fingertips detection has been used in many a pplications, and it is very popular and commonly used in the area of Human Comput er Interaction these da ys. This paper presents a novel time efficient method that will lead to fingertip detection after cropping the irreleva nt parts of input imag e. Binary silhouette of the input image was generated, usin g HS V color space based skin filter and hand cropping was performed based on skin pixel histogram of the hand im age. The cropped image woul d be used to determine the fingertips in the image fram e. Keywords : Human Computer Interface, Skin Filter, Image Segm entation, Binar y Silhoue tte, Interactive Real Time Systems. 1. INTRODUCTI ON Interactive system s based on gesture reco gnition needs a real time implementati on to work with accepta ble performance. In the literature many examples can be find out where gesture is used to control the system based on fingertip detection. Our focus is on hand gesture recognition in natural way without using an y marker of sensor ba sed gloves. M any researchers have proposed di fferent m ethods for dy namic hand gestu re recognition us ing fingertip detecti on, but several limitations can be seen in these a pproaches. Garg [ Garg, P. et al., 2009 ] uses 3D im ages in his m ethod to recognize the hand gesture, but this process is complex and also not time efficient. Processing time is v ery critical factor in real tim e applications as Ozer [ Oz er, I.B.,, 2005 ] states “Designing a real-time video analysis is truly a complex task”. Yang [Yang] analyses the hand contour to sele ct fingerti p candidates and fi nd peaks in their spatial distribution a nd checks local variance to locate fingertip s. These methods are not i nvariant to the orientati on of the han d. There are ot her methods , which are using directionally Variant templates to d etect fingertips [ Kim, J.M. and Lee, W.K., 2008 ], [ Sanghi, et al., 2008 ]. Few other m ethods are depen dent on specialized instruments and setup like the use of infrared cam era [ Oka, K. et al., 2002 ], stereo camera [Ying], a fixed back ground [ C rowley, J.L., et al., 1995], [ Quek, F.K.H. et al., 1995 ] or use of m arkers on hand. This paper describes a novel m ethod of moti on patterns reco gnition ge nerated by the hand without any kind of sensor or marker. The detection of moving fi ngertips in vi deo needs a fast and robust im plementat ion of method. M any fingertip detection m ethods are based on ha nd Segment ation technique because it decreases pixel area which is going to process, by selecting only areas of in terest. However most hand segmentation methods cannot do a clearly hand segmentation und er some conditions lik e fast hand motion, clutter ed background, poor light condition [ Christian ]. Poor han d segmentati on method performance usually invalidates fingertip detectio n methods. Re searchers [ Ok a, et al., May 2 002 ], [ Oka, et al., Dec 2002], [Sato, 200 0 ] uses infrared camera to get a reliable segmentation. Few researchers [ Crowle, et al., 1995], [Quek, et al., 1995], [Christian], [Tomita, et al., 1994 ], [Keaton, et al., 20 02 ], [ Wu et al., 2000 ] limits the de gree of the backgr ound clutter, finger motion speed or light conditions to get a reliable segmentation in their work. Some of fingertip detection methods cannot localize accurately multidirectional fingertips. Researchers [ Crowley, et al., 1995], [Quek, et al., 1995], [Brown, et al., 2000], [Tomita, et al., 1995 ] assumes that the hand is always poi nting upwa rd to get precise localization. Fig. 1: Algorithm Flow of Fingertip Detection Method 2. APPROACH TO FINGERTIP DE TECTION Algorithm flow of Fingerti p detection m ethod has been shown in figure 1. It includes five steps. F irst of all a camera captur e a real tim e video of moving ha nd in front of syst em and ha nd segmentati on is done ba sed on skin filter in second step. In th e next step wrist end is detected, based o n histogram of skin pixel s and after this perform s hand crop ping using di fferent param eters in current image frame of video. Finally fin gertips will be detected in the cropped hand image, which is a continuous process for different image frames in the video. 448  International Conference on “Trends in Indu st rial Measurements and Automation” TIMA–2011 TS10-1 Fig. 2: Skin Filtering Process. Images Shown are (a) Initia l Hand Image, (b) HSV Conversion, (c) Filtered Image in HSV format, (d) Smoothen Image afte r Applying Averaging Filter , (e) Binary Sil houette Respec tively. 2.1 Skin Filter The skin filter is used on the current inp ut image frame of video. It i s based on HSV (ca n also be based o n YC b C r ) colour space. In the HSV c olour space the skin would be filtered using the chromacity (hu e and saturation) values while in the YC b C r colour space, the C b , C r values would be used fo r filtering skin. The skin filters are used to create a binary im age with background i n black colou r and the ha nd region i n white. In the next step the binary image need to be smoothened using th e averaging filter. Figure 2 shows different steps of skin filte ring process. There can be many errors in the output image of skin filter step because of wrong pixel detect ion or some skin pi xels in the backgro und of han d. Fig. 3(a): Biggest BLOB Fig. 3(b): Hand after Filtration To remove these errors, the biggest BLOB (Binary Linked Object) i s considered as the hand a nd rest the background as shown in figure 3(a). The biggest BLOB represents hand coo rdinates in ‘1’ a nd ‘0’ to the background. The filtered out hand is shown in figure 3(b) after removing all errors. The only limitatio n of this filter is that the BLOB for h and should be the biggest one. 2.2 Wrist End Detection Wrist end detect ion is based on the hist ogram of the binary silho uette. Histo grams generati ng functi ons are: TS10-1 An Efficient Real Time Method of Fingertip Detection  449 Here imb represents the binary silhou ette and m, n represents the row and columns of the matrix imb . After a 4-way scan of imag e, we choose the maxim um value of ‘on’ pixels coming out of all scanne d (‘1’ in the binary si lhouette). It was noted that m aximum value of ‘on’ pixels represe nts the wrist end and opp osite end of this scan would represent the finger end. Figure 4 shows the scanning process. The ye llow bar showed in figure 4 corres ponds to the first ‘o n’ pixel in the bina ry silhouette scanned from the left to right direction. Similarly the green bar corresponds to right to left, red bar corresponds to down to up, and p ink bar corresponds to up to downward sca n ‘on’ pi xels in the binary silhouette. Now, it is clear that red bar had greater ma gnitude tha n other bars for that particular image frame. So we can infer that the wrist end is in downward di rection of the fram e and conse quently t he direction of finger is in the up ward direct ion. Here the direction from wrist to finger is known. Fig. 4: Image Scanning and Corresponding Bars 2.3 Ha nd Croppi ng Hand croppin g minimizes the n umber of pixels t o be taken into account for processing which leads to minimizat ion of comput ation time . In the next step Histogram would be generat ed from the bi nary silhouette of the image, as shown in figure 5. It was observed from the histog ram th at at the point where the wrist ends, a steeping inclination of th e magnitude of the histogram starts, whose sl ope, m can be defined as: As starting point of image where inclination is found, and then the poi nts correspond to t he first ‘on’ pixel scanning from ot her three sides are found, which gives the coordinates where the image should be cropped. The equations for cropping the im age are: Where imcrop represents the cropped image, Xm in, Ymin, Xmax, Ymax represent the boundar y of the hand in the image. Fig. 5: Hand Cropping. Images shown are (a) Initial Image, (b) Histogram of Binary Sil houette wher e wrist end can be seen clearly, (c) Cropped Hand Im age respectively. Some results wit h processing steps for ha nd cropping are shown in fi gure 6. The arrows sh owed in the m ain frames indicate the directions of scan which were found from wrist end detection step. In all the histograms in figure 6 it is clearly seen that at the wrist point, a steeping inclination starts in the scanning direction. 450  International Conference on “Trends in Indu st rial Measurements and Automation” TIMA–2011 TS10-1 Fig. 6: Results of Hand Cropp ing Process from Initial Im ages 2.4 Fingertip Detection Now in the cropped hand image, fing ertips will be figured out. Again scanning the cropped bi nary image and calculate the number of pixels for each row or column based on the ha nd direction in up-down or left - right. Then intensity values for each pi xel are assigned from 1 to 255 in increasing order from wris t to finger end by proportionality. So, each ‘on ’ pixel on the edges of the finge rs would be as signed a hi gh intensity value of 255. Now det ection of the edge of the fi ngers is done by just detecti ng pixels ha ving, inten sity of 255. T his can be represented mathematically as: Here Finger edge gives the bounda ry of the fi nger. The line having hi gh intensi ty pixel, i s first indexe d and check whether differentiated value li e inside a threshold, if it is then it represents a fingertip. The threshold value changes t oward the di rection of hand. That threshold can be set after the detection of the direction of hand to t he finger which we already k now. 3. CONCLUSION The detection of fingertip usi ng a time effi cient method has been discu ssed which will be used in our project ‘Controlling the robot using hand gesture’. In this project user will pass con trolling information to rob ot using hand ge stures in natural way. The Movement of user’s finger will control the robot hand and its working, by moving hand in fro nt of camera with out wearing any glove s or markers. REFERENCES Brown, T. and Thomas, R.C. (2000) . “Finger Tracking for the Digital Desk”, Proceedings of First Australasian User Interface Conference (AUIC 2000) , 11–16, Canberra , Australia. Crowley, J.L., Berardand F. and Co utaz, J. (1995). “Finger T acking as an Input Device for Augmented Reality”, Proceedings of. International Workshop on Auto matic Face and Gesture Recognition, 195–200, Zurich, Switzerland Garg, P., Aggarwal, N. and Sofa t, S. ( 2009). “Vision Based Hand Gesture Recognition”, World Academy of Science, Engineering and Technology , vol. 49, 972–9 77. Hardenberg, C.V. and Berard, F. (2001). “Bare Hand Human Computer Interaction”, Proceedings of the ACM Workshop on Perceptive user Interfaces, 1–8 , Orlando, Florida, USA. Keaton, T., Dominguez, S.M. and Sayed, A.H. (2002). “SNAP & TELL: A Multimodal Wearable Co mputer Interface for Browsing the Environment”, Proceedings of Sixth Inter national Symposium on Wearable Computers, (ISWC 2002), 75–82, Seattle, WA, USA. Kim, J.M. and Lee, W.K. (2008). “Hand Shape Recognition using Fingertips”, In the Proceedings of Fifth International Conference on Fuzzy Systems and Knowledge Discovery, 44–48, Jinan, Shandong, China. Oka, K., Sato, Y. and Koike, H. (Dec, 2002). “Real Tim e Fingertip Tracking and Gesture Recognition” Computer Graphics and Applications, IEEE , vol. 22(6), 64–71. Oka, K., Sato, Y. and Koike, H. (May, 2002). “Real Time Tracking of Multiple Fingertips and Gesture Recognition for Augmented Desk Interface Systems”, Proceedings of the Fifth IEEE International Conference on Automatic F ace and Gesture Recognition (FGR.02), 411–4 16, Washington, D.C., USA. Ozer, I.B., Lu, T. and Wolf, W. ( 2005). “Design of a Real Time Gesture Recognition System: High Perform ance through Algorithms and Software”, IEEE Signal Processing Magazine, 57–64. Quek, F.K.H., Mysliwiec, T. and Zhao, M. (1995). “Finger Mouse: A Free Hand Pointing Computer Interface”, Proceedings of International Workshop on Auto matic Face and Gesture Recognition , 372–377, Zurich, Switzer land. Sanghi, A., Arora, H., Gupta, K. , Vats and V.B. (2008). A Fingertip Detection and Tracking System as a Virtual Mouse, a Signature Input Device and an Application Selector”, Seventh International Conference on Devices, Circuits and Syste ms, 1–4, Cancun, Mexico. Sato, Y., Kobayashi, Y. and Koike, H. (2000). “Fast T racking of Hands and Fingertips in Infrared Images for Augmented Desk Interface”, Proceedings of Fou rth IEEE International Conference on Automatic Face and Gesture Recognition, 462– 467, Grenoble, France. Tomita, A. and Ishii. R.J. (1994). “Hand Shape Extraction from a Sequence of Digitized Grayscale Images”, Proceedings of Tweenth International Conference on Industrial Electronics, Control and Instrumentation, IECON’94, vol. 3, 1925–1930, Bologna, Italy. Wu, Y., Shan, Y., Zhangy, Z. and Shafer, S. (2000). “Visual Panel: From an Ordinary Paper to a Wireless and Mobile Input Device”, Technical Report, MSRTR 2000 , Microsoft Research Corporation. Yang, D., Jin, L.W., Yin, J., and Others. (2005) “An Effective Robust Fingertip Detection Method for Finger Writing Character Recognition System”, Proceedings of the Fourth International Conference on Machine Learning and Cybernetics , 4191–4196, Guangzhou, China. Ying H., Song, J., Renand, X. and Wang, W. (2008). “Fingertip Detection and Tracking Using 2D and 3D Inform ation”, Proceedings of the Seventh World Congress on I ntelligent Control and Automation , 1149–11 52, Chongqing, Chi na.