Vision-Based Sign Language Translation via a Skeleton-Aware Neural Network

Sign languages are mainly expressed by human actions, such as arm, hand, and finger motions. Thus a skeleton which reflects human pose information can provide an important cue for distinguishing signs (i.e., human actions), and can be used for sign language translation (SLT), which aims to translate sign language to spoken language. However, the recent neural networks typically focus on extracting local-area or full-frame features, while ignoring informative skeleton features. Therefore, this paper proposes a novel skeleton-aware neural network, SANet, for vision-based SLT. Specifically, to introduce skeleton modality, we design a self-contained branch for skeleton extraction. To efficiently guide the feature extraction from videos with skeletons, we concatenate the skeleton channel and RGB channels of each frame for feature extraction. To distinguish the importance of clips (i.e., segmented short videos), we construct a skeleton-based graph convolutional network, GCN, for feature scaling, i.e., giving an importance weight for each clip. Finally, to generate spoken language from features, we provide an end-to-end method and a two-stage method for SLT. Besides, based on SANet, we provide an SLT solution on the smartphone for benefiting communication between hearing-impaired people and normal people. Extensive experiments on three public datasets and case studies in real scenarios demonstrate the effectiveness of our method, which outperforms existing methods.