This paper proposes a novel approach for gesture recognition from motion depth images based on template matching. Gestures can be represented with image templates, which in turn can be used to compare and match gestures. The proposed method uses a single example of an action as a query to find similar matches and thus termed one-shot-learning gesture recognition. It does not require prior knowledge about actions, foreground/background segmentation, or any motion estimation or tracking. The proposed method makes a novel approach to separate different gestures from a single video. Moreover, this method is based on the computation of space–time descriptors from the query video which measures the likeness of a gesture in a lexicon. These descriptor extraction methods include the standard deviation of the depth images of a gesture as well as the motion history image. Furthermore, two dimensional discrete Fourier transform is employed to reduce the effect of camera shift. The comparison is done based on correlation coefficient of the image templates and an intelligent classifier is proposed to ensure better recognition accuracy. Extensive experimentation is done on a very complicated and diversified dataset to establish the effectiveness of employing the proposed methods.