Human Action Recognition (HAR) attempts to recognize the human action from images and videos. The major challenge in HAR is the design of an action descriptor that makes the HAR system robust for different environments. A novel action descriptor is proposed in this study, based on two independent spatial and spectral filters. The proposed descriptor uses a Difference of Gaussian (DoG) filter to extract scale-invariant features and a Difference of Wavelet (DoW) filter to extract spectral information. To create a composite feature vector for a particular test action picture, the Discriminant of Guassian (DoG) and Difference of Wavelet (DoW) features are combined. Linear Discriminant Analysis (LDA), a widely used dimensionality reduction technique, is also used to eliminate duplicate data. Finally, a closest neighbor method is used to classify the dataset. Weizmann and UCF 11 datasets were used to run extensive simulations of the suggested strategy, and the accuracy assessed after the simulations were run on Weizmann datasets for five-fold cross validation is shown to perform well. The average accuracy of DoG + DoW is observed as 83.6635% while the average accuracy of Discrinanat of Guassian (DoG) and Difference of Wavelet (DoW) is observed as 80.2312% and 77.4215%, respectively. The average accuracy measured after the simulation of proposed methods over UCF 11 action dataset for five-fold cross validation DoG + DoW is observed as 62.5231% while the average accuracy of Difference of Guassian (DoG) and Difference of Wavelet (DoW) is observed as 60.3214% and 58.1247%, respectively. From the above accuracy observations, the accuracy of Weizmann is high compared to the accuracy of UCF 11, hence verifying the effectiveness in the improvisation of recognition accuracy.