Using Watershed Transform for Vision-based Two-Hand Occlusion in an Interactive AR Environment

Peng Peng Leim, Guat Yew Tan, Kah Pin Ng, Miin Huey Ang


To achieve a natural interaction in augmented reality environment, we have suggested to use markerless vision-based two-handed gestures for the interaction; with an outstretched hand and a pointing hand used as virtual object registration plane and pointing device respectively. However, two-handed interaction always causes mutual occlusion which jeopardizes the hand gesture recognition. In this paper, we present a solution for two-hand occlusion by using watershed transform. The main idea is to start from a two-hand occlusion image in binary format, then form a grey-scale image based on the distance of each non-object pixel to object pixel. The watershed algorithm is applied to the negation of the grey scaled image to form watershed lines which separate the two hands. Fingertips are then identified and each hand is recognized based on the number of fingertips on each hand. The outstretched hand is assumed to contain 5 fingertips and the pointing device contains less than 5 fingertips. An example of applying our result in hand and virtual object interaction is displayed at the end of the paper.


Watershed Algorithm; Two-Hand Occlusion; Augmented Reality; Vision-Based Hand Detection


P. P. Leim and G. Y. Tan, "Component level interaction of a 3D model in an interactive augmented reality environment," International Journal on Future Computer and Communication, vol. 2, no. 5, pp. 539-542, October 2013.

H. S. Hasan and S. A. Kareem, "Human computer interaction for vision based hand gesture recognition: a survey," in Proceedings of the IEEE International Conference on Advanced Computer Science Applications and Technologies (ACSAT), 2012, pp. 55-60.

S. Reifinger, F. Wallhoff, M. AblaBmeier, T. Poitschke, and G. Rigoll, "Static and dynamic hand-gesture recognition for augmented reality applications," International Conference on Human-Computer Interaction, C. Stephanidis, Eds. Beijing: Springer, July 2007, pp. 728-737.

J. D. Sturman and D. Zeltzer, "A survey of glove-based input," Computer Graphics and Applications, IEEE, vol.14, no. 1, pp. 30-39, 1994.

V. Buchmann, "FingARtips: gesture based direct manipulation in Augmented Reality," in Proceedings of the 2nd International Conference on Computer Graphics and Interactive Techniques, Australasia and South East Asia, ACM, 2004, pp. 212-221.

K. P. Ng, G. Y. Tan and Y. P. Wong, "Vision-Based Hand Detection for Registration of Virtual Objects in Augmented Reality," International Journal of Future Computer and Communication, vol. 2, no. 5, pp. 423-427, October 2013.

A. Utsumi and J. Ohya, "Multiple Hand Gesture Tracking using Multiple Cameras," in Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition (CVPR), 1999, pp. 473-478.

J. Shi and J Malik, "Normalized cuts and image segmentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, no. 8, pp. 888-905, 2000.

T. Inaguma, H. Saji, and H Nakatani, "Two-handed gesture tracking in the case of occlusion of hands," in IAPR Workshop on Machine Vision Applications, 2002, pp. 306-309.

K. A. Barhate, K. S. Patwardhan, S. D. Roy, S. Chaudhuri, and S. Chaudhury, "Robust two hand tracker using predictive eigentracking," in Proceedings of the National Conference on Communication (NCC), 2004, pp. 101-105.

J. M. Black and A. D. Jepson, - Eigentracking: Robust matching and tracking of articulated objects using a view-based representation," International Journal of Computer Vision, vol. 26, no.1, pp. 63-84, 1998.

N. Gupta, P. Mittal, S. D. Roy, S. Chaudhury, and S. Banerjee, "A Predictive Scheme for Appearance-based Hand Tracking," in Proceedings of the National Conference on Communications (NCC), 2002, pp. 513-522.

S. Beucher, "The watershed transformation applied to image segmentation," Scanning Microscopy Supplement, pp. 1-26, 1992.

L. J. Belaid and W. Mourou, "Image Segmentation: A Watershed Transformation Algorithm," Image Analysis & Stereology, vol. 28, no. 2, 2009, pp. 93-102.

H. Kato and T. Kato, "A marker-less Augmented Reality based on fast fingertip detection for smart phones," IEEE International Conference on Consumer Electronics (ICCE), 2011, pp. 127-128.

Q. Chen, X. Yang, and E. M. Petriu, "Watershed segmentation for binary images with different distance transforms," in Proceedings of the 3rd IEEE International Workshop on Haptic, Audio and Visual Environments and Their Applications, 2004, pp. 111-116.

Full Text: PDF


  • There are currently no refbacks.

Creative Commons License
This work is licensed under a Creative Commons Attribution 3.0 License.

IT in Innovation IT in Business IT in Engineering IT in Health IT in Science IT in Design IT in Fashion

IT in Industry (2012 - ) ISSN (Online): 2203-1731; ISSN (Print): 2204-0595