With the rapid progress of mobile computational devices, human action counting has emerged as a promising application. It could revolutionize individual fitness routines, school physical education, and military training. However, existing studies suffer from low counting accuracy or efficiency. In this paper, we first provide our contributed real-world video dataset, including 84 videos from 54 recruited volunteers. Next, we propose a Skeleton-based Real-time Human Action Counting (SRHAC) architecture with spatial-temporal optimization. SRHAC analyzes human skeletons to interpret action semantics, offering finer granularity and higher accuracy. Moreover, a dynamic frame filtering algorithm and a region-of-interest generator algorithm are designed to further improve the accuracy and efficiency of SRHAC. Extensive experiments demonstrate our method achieves an advanced 98.03% counting accuracy under real-time level counting efficiency.