To identify the driving activities of urban buses while avoiding privacy infringement on drivers and other road users caused by the use of on-board surveillance cameras, this study establishes a bus vehicle behavior prediction model that takes vehicle motion and driving operation data as inputs. First, experiments were carried out to collect the natural driving data of urban buses, and vehicle movement and driver behavior operation data were collected through the CAN protocol. Then, segments corresponding to station entry, station exit, intersections, turning and lane changing were selected. Based on Takens’ delay embedding method, phase space reconstruction was performed to map time-series data into a high-dimensional space to generate two-dimension recurrence plots. Afterwards, multi-channel stacking was applied to construct RGB images. To address the issue of class imbalance, Focal Loss function was adopted to enhance the model’s feature extraction capability for minority classes. On this basis, an E-bus vehicle behavior prediction model marked as E-VBPM was developed using the ConvNeXt network. The results indicate that E-VBPM achieves an accuracy of 84.62% in predicting 5 kinds of driving activities. As compared with the machine learning algorithm that uses time-series data as the input, the proposed model achieves an absolute increase in accuracy, precision, and recall by 6.79%, 10.98% and 8.86%, respectively. The results of this research provide support for electric bus on-board systems to identify the current operating modes and assist the driver in a safer and more intelligent way.