Real Time Eye Tracking and Blink Detection Using Low Resolution

1、real time eye tracking and blink detection using low resolution web camyassir nawaz1 and shiladitya sircar1,2faculty of engineering and applied science1/ remote sensing group c-core2 memorial university st. johns, nf, canada a1b 3x5e-mail: yassir, sircarengr.mun.caabstract: in this paper we present

2、a computationally efficient and cost effective solution for blink detection and tracking of eyes in real time. we also present a framework that can be used to investigate various strategies for human computer interaction. we commence with the extraction of human face and its various features from a

3、low-resolution video stream. these extracted features, in particular eyes, are tracked and monitored in real time to detect any movements and variations. these variations then form the basis of various actions or responses. lastly we demonstrate the effectiveness of our technique through experimenta

4、l results and provide recommendations for future development and improvement.1. introductionin recent years, scientists are aiming to develop non-intrusive man-machine-interface with vision systems in a constrained domain. this has led to the emergence of the field of automatic face processing and i

5、dentification of visual facial behaviors (such as blinking, smiling, frowning etc.), which are instinctively inherent to every human being and thus bringing cognitive sciences closer to computer sciences. this paper presents an efficient technique for eye tracking and blink detection for low resolut

6、ion web cameras.2. feature point initializationin order to track a facial feature it must be first identified in an image stream. the initialization process starts with the location of face in the first image. human face is extracted from the image using skin segmentation techniques. skin pixels can

7、 have very different rgb values for different people depending on their skin color, illumination etc, however their chromatic red and chromatic blue values defined as chr = r/(r+g+b) and chb = b/(r+g+b) are clustered in a limited region of (chr, chb) space and are largely independent of skin color o

8、r lighting conditions. based on their chromatic red and chromatic blue values, the pixels are classified into skin or non skin pixels. after face segmentation the image is subject to blob analysis to locate the exact position of the eyes. the technique makes use of the fact that eyes when open have

9、color different from skin and therefore appear as holes in the face, in the skin segmented image. these holes give the approximate location of the eyes. these locations are then used to track the eyes in the remaining video stream. blink detection is used to further improve these location estimates

10、during tracking.3. eye tracking using optical flowoptical flow is a very powerful and popular technique used in motion estimation. the optical flow algorithm (lucas & kanade) used in this technique tracks a feature point by trying to predict the apparent motion of image brightness around that point.

11、 lucas & kanade technique examine the brightness of the pixel neighborhood and then tries to locate the position of the pixel in the subsequent images assuming that all neighborhood moves with the same velocity. the selection of the feature points to be used in lucas & kanade tracking algorithm is c

12、rucial to the performance of tracking. a good feature point should have a distinct neighborhood so that it can be identified easily in the subsequent images. in this work it has been found that inner corner of the eye serves as a very god feature point to track. the tracking algorithm returns the ne

13、w position of the eyes in each image of video stream. these positions are then examined to monitor head orientation or loss of a feature point. in case one or more feature points are lost due to a rapid movement or turning of the head feature points are initialized once again as described in section

14、 2. figure 1 shows the tracking of eye using lucas & kanade optical flow technique.4. blink detectionthe eye tracking reduces the blink detection search space to two eye blobs. the location of these blobs or eyes is passed on to the blink detection algorithm by the eye tracker in each image. when a

15、person blinks the intensity of the pixels that represent the eye also changes. this forms the basis of blink detection. a variance map of the change in intensity values in the corresponding pixels of the two consecutive frames, in an image sequence is created on a figure 1 eye tracking using optical

16、 flowpixel by pixel basis. two maps, one representing the mean and the other variance 2are initialized. the arrival of a new image updates these maps recursively according to the following relations: (1) (2) the variance map is thresholded after each update. if the ratio of the number of thresholded

17、 variance pixels to the total number of pixels in the eye blob exceeds a threshold a blink is detected. figure 2 shows two images taken from a blink sequence and the corresponding variance map generated.figure 2 blink detection using variance maps5. eyeball detectioneyeball detection algorithm recei

18、ves the coordinates of two eye boxes and finds the location of eyeballs within these boxes. several techniques were considered for the purpose of eyeball detection. geometrical pattern search or ellipse fitting techniques provide very good results however they are computationally intensive and were

19、rejected in favor of circular hough transform. circular hough transform itself is computationally intensive however it can be improved considerably by pre processing the eye image. this pre processing reduces the number of pixels which can vote in the hough transform making it much faster. 5.1 pre p

20、rocessing of eye imagethe purpose of pre processing is to minimize the number of on-pixels in the binary image on which hough transform is computed. an ideal image for hough transform will only contain a circle representing the outline of the eyeball, however it is difficult to obtain and attempt is

21、 made to remove unwanted pixels without removing the eyball pixels. the pre processing steps are:5.1.1 color filtering: three channel rgb image is converted to a single channel image for edge detection. it has been found experimentally that green channel image provides best results for edge detectio

22、n even though difference with other channels is not huge. 5.1.2 histogram equalization: the single channel image is equalized to compensate for different lighting conditions. since eye image contains two regions of comparable areas and distinct intensities, one representing the eyeball and the other

23、 white area around it, equalization tends to highlight the edges of the eyeballs.5.1.3 edge detection: the green channel image is then subjected to edge detection to get an outline of the eyeball. since the eyeball pixels have different intensity than the white pixels around it an edge is readily ob

24、tained however it is occluded. a 3 by 3 vertical prewitt operator is used for this purpose. horizontal prewitt operator was also tried but since the outline of the eyes form horizontal edges, extra edges appear in the edge image. vertical operator returns the vertical edges which are sufficient for

25、hough transform as all the pixels are not required to obtain the best fit circle.the steps explained above are shown in figure 3. figure 3 pre processing of eye image for eyeball detection5.2 circular hough transformhough transform can be used to extract various features from an image. the transform

26、 takes the geometric equation of the feature to be found and then inverts it so that x and y become constants in the equation, thus resulting in an inverse function space. the intersections in this space reveals the constants associated with the equation of the features discovered in the original im

27、age. circular hough transform is used to find the coordinates of circles in an image. in order to determine the location of the eyeball in the processed image a range for the radius of the desired circle is specified. each thresholded pixel in the binary image is examined to determine whether it fit

28、s the desired circle. the best circle gives the outline of the eyeball. the technique is very powerful since it can fit circles even when a partial outline of a circle is available which is common in an eye image due to the occluded eyeball. figure 4 shows the extraction of eye ball from eye image u

29、sing circular hough transform. the image has been equalized and vertical prewitt operator has been used to get the approximate outline of the eyeball as explained in section 5.1.figure 4 circular hough transform to locate eyeball6. eyeball trackinghough transform can be used to locate eye ball in ea

30、ch eye image thus determining the location of the eye ball. this technique however is computationally intensive and slows down the throughput of the algorithm. the advantage of computing hough transform on the other hand is that we do not have to worry about the loss of tracking. if the hough transf

31、orm fails to find a circle in one image, this will not affect the transform computation in the subsequent images. another technique is to use lucas & kanade feature tracker explained in section 3 to track the eyeball once its center and radius has been obtained through circular hough transform. this

32、 vastly improves the speed of the algorithm however the tracking will be lost upon the loss of a feature point representing the eyeball. this feature point could be lost due to a blink or a rapid movement of the eyeball or eye itself. we address this problem by constantly monitoring the tracked feat

33、ure points to determine if the tracking has been lost, in which case hough transform is used to re initialize the eyeball location. the best feature points for eyeball tracking are the center of the eyeball and its two outer edges. the relative distance among these three points should remain constan

34、t as the eyeball moves within the eye. a significant change in the above distance will indicate a loss of a feature point.figure 5 eyeball tracking7. results and future workfigure 5 shows the results of our technique. three images taken from the eye tracking video demonstrate the tracking of the eye

35、ball along with between-the-eyes feature point during the movement of the head in various directions. the results for blink detection are satisfactory for this project. the eye tracker can tolerate up to 45 degrees head turn in either direction. this can be further improved by using between-the-eyes

36、 feature point. this point can tolerate up to 90 degrees rotation of the head and can be used to reinitialize the lost feature points by using the distance measurements from the other tracked eye. the other alternative is to reinitialize all the feature points using blink detection. hough transform

37、gives satisfactory results for this project where a very low-resolution web cam has been used. the images shown above have a resolution of 320240. if more accuracy is required i.e. very subtle eyeball movements need to be detected a high-resolution web camera can be used. this will however increase

38、the number of pixels processed by the hough transform thus reducing the overall speed. additional hardware to increase the computing power may be used to offset the additional calculations. the eye tracker and blink detector implemented in this project can be used to built various intelligent applic

39、ations some of which are suggested below tracker and blink detector can be interfaced with a pc mouse so that cursor can be moved on the screen using eye movements. voluntary blink detections can be used to simulate mouse clicks thus enabling a hands free browsing. this application can provide severely disabled individuals, who are unable to operate ordinary mouse, a way to use personal computers. various human monitoring applications measure the blink rates of human to determine several biological parameters. this software can be used to monitor and detect the blink rate of a person