Detection Of Fingers With A Depth Based Hand-detector In Static Frames

Abstract

This thesis presents a method for a finger detection system. It is assumed that the user taps their fingers on a table, and that the camera is placed on the same table in front of their fingers. This setup is motivated by the application of analyzing the movement of fingers in patients engaging in physical therapy. Fingers are detected in static images, which is a more challenging task than detecting and tracking fingers in videos which are based on motion.The Microsoft Kinect sensor has been used as a source for data, and it provides color and depth images at each video frame. Detection of fingers is performed using two different methods: Template Matching and Principal Component Analysis (PCA). Additional information present in the image, such as skin color and depth data, is used to improve accuracy and efficiency. The depth frames are used to separate the foreground from the background, and also to provide additional features for detecting hands. A face detector is also utilized and the position of face is used as a reference to determine where the hands are located.An additional contribution of the thesis is a graphical interface, developed in Matlab, for annotating finger positions. This tool provides abilities for users to load various sequences of images and manually annotate the position of fingers in those images. Using this tool, we have annotated a large number of video frames, and these annotations have been used for training and testing the proposed method. In addition, these annotations remain as a valuable resource for future research on finger detection and tracking. For testing purposes, the Matlab system also allows running the proposed method and measuring the accuracy of the results, based on the manual annotations. The thesis includes a comprehensive study on the effect of possible design decisions, as well as accuracy of user-dependent and user-independent settings