A Brain-Computer Interface (BCI) is designed for human-computer interactions without body movement. To improve the expression of input features closely related to a given BCI task, we propose a convolution network called GANet to analyze Event-Related Potential (ERP) in the BCI task. This model introduces the parallel convolution to extract multi-scale features in electroencephalogram (EEG) data. In addition, a Graph-based Attention (GA) mechanism is used to model interdependencies among different EEG channels. Experiments are conducted on a public dataset of 15 subjects in the specific-subject and cross-subject scenarios. The results indicate that the GANet achieves state-of-the-art performance with an accuracy of 99.75% in the specific-subject scenario and an accuracy of 81.37% in the cross-subject scenario. Different structures are discussed to analyze the contributions of the parallel convolution in feature extraction and the GA module in feature expression. GANet shows satisfied performance and good generalization in the BCI task. Our codes are publicly available at https://github.com/Debbie-85/GANet.