Most traffic accidents are caused by improper driver operation, so autonomous driving based on rapidly developing artificial intelligence technology has attracted much attention. Inspired by the biological visual perception and neural decision-making mechanism, this paper constructs a two-branch bionic network for autonomous driving, which learns to map the driver’s perspective image directly to the steering commands. On the real-world driving dataset we collected, extensive experiments prove the efficiency, robustness, superior structure and biological interpretability of this end-to-end algorithm. Moreover, the flexible scalability of this network greatly supports real-time inference and deployment.