Multi-locomotion mode human-robot interaction technology for self-paced treadmills

A self-paced treadmill (SPT) is key human-robot interactive equipment for virtual reality, which can enable a user to walk at the intended speed by using a re-positioning technology. Realizing multimode interactions of SPTs is crucial for enriching their applications. However, existing studies only realizes a few interaction modes. To realize multimode interactions in self-paced treadmills, a novel multilayer control framework is proposed in this paper.

In this study, the control system is divided into two layers: the recognition layer and the control layer. First, a novel hybrid spatial-temporal graph convolutional neural network is proposed to realize user-independent human locomotion mode recognition based on plantar pressure insoles in the recognition layer. The proposed network separates the pressure and acceleration signals and dealt with them individually. It is also utilized to extract the natural spatial topology between pressure nodes. Long short-term memory layers are used to individually extract temporal-dependent features of pressure and acceleration signals and to fuse multimodal features for final recognition. A multilayer perceptron is utilized to map the fusion features to the locomotion modes. By extracting the natural spatial-temporal features of multimodal data during human locomotion, a high generalization capability of the recognition results can be expected. Second, control strategies for different locomotion modes are designed in the control layer according to the stability condition of different human locomotion modes. Meanwhile, a walking speed feedforward control strategy is proposed to re-position the user and ensure natural gaits for the walking mode. Variable gain control strategies are adopted to manipulate the acceleration for the running and back walking modes. A buffer control strategy is proposed to improve the stability during jump landing for the jumping mode. Then, a finite state machine is used to automatically switch the control strategies. The states are transited based on the recognition results.

1) The proposed locomotion mode recognition method was evaluated on a dataset that comprises eight subjects with five locomotion modes through the leave-one-subject-out cross validation. Then, it was compared with the convolutional neural network (CNN) and domain-adversarial neural network (DANN). Experimental results indicated that the mean and standard deviation classification accuracies of the CNN, DANN, and HSTGCN are (90.26±8.54)%, (97.71±3.60)%, and (97.37±1.40)%, respectively. These results validated that the proposed method can achieve high generalization capability without any dependency on the data of target subjects. Hence, the burden of repeated data collection and network training was reduced. 2) Based on the recognition results, experiments on the multi-locomotion mode human-robot interaction were conducted using a finite state machine. Experimental results indicated that a user can freely change the locomotion modes on the treadmill, and the balance was not significantly affected by the treadmill acceleration.

The proposed framework can automatically combine the recognition results with the treadmill control and can realize the control of multi-locomotion mode human-robot interactions. Further, experimental results validate that the proposed multilayer control strategy can achieve a stable and smooth multi-locomotion mode human-robot interaction, ensure natural gaits and posture stability of the user, and meet the requirements of multi-locomotion mode human-robot interactions for self-paced treadmills.