Automatic gesture recognition is a key technology used in Human Computer Interaction. In this paper we introduce a hand gesture recognition system which consists of 4 modules, image segmentation, feature extraction, HMM training and gesture recognition. Image or video is divided into multiple frames and segmentation process which uses colour based detection of the path of the object is applied to each frame. Feature extraction process mainly considers the orientation of the state tracked. This is done using HSV[hue-saturation value] image and contour mapping of the image. The training part of the HMM model works on basis of LRB(Left-Right-Banded)topology and uses the BW (Baum Welch) algorithm. We have used Viterbi algorithm for mapping the state to a symbol i.e. recognition. HMM is used to predict the gesture and increase the tolerance of the system to incorporate human errors.