Graph neural networks (GNNs) are a prominent paradigm in dealing with graph structured data as they are able to explore rich relational information between nodes with attributes. That said, improving the robustness of GNNs becomes increasingly important since recent studies have shown that attackers can catastrophically ruin the performance of GNNs through injecting unnoticeable adversarial edges to graphs. Prior defense approaches focused on reducing the adversarial effect via deleting adversarial edges or designing a robust framework. However, this defense strategy is limited in (1) preserving the integrity of graph structure and (2) improving the expressive power of node representations. To address this limitation, we propose a lightweight purified graph neural network (Pure-GNN), which handles the adversarial effect by assigning importance weights to edges and utilizes a residual mechanism to improve the quality of node representations. With this design, it cannot only protect the intact information of graph topology but also learn the representations with significantly improved runtime efficiency. Extensive experiments over real-world datasets demonstrate that Pure-GNN outperforms the state-of-the-art approaches in defending against various adversarial attacks.