A novel approach for gesture recognition is developed in this paper based on template matching from motion depth image. The proposed method uses a single example of an action as a query to find similar matches from a good number of test samples. No prior knowledge about the actions, the foreground/background segmentation, or any motion estimation or tracking is required. A novel approach to separate different gestures from a single video is also introduced. The proposed method is based on the computation of space-time descriptors from the query video which measures the likeness of a gesture in a lexicon. The descriptor extraction method includes the standard deviation of the depth images of a gesture. Moreover, two dimensional discrete Fourier transform is employed to reduce the effect of camera shift. Classification is done based on correlation coefficient of the image templates and an intelligent classifier is proposed to ensure better recognition accuracy. Extensive experimentation is done on a vast and very complicated dataset to establish the effectiveness of employing the proposed method.