Journal Home > Volume 5 , Issue 3
Journal of Intelligent and Connected Vehicles 2022, 5(3): 316-332 https://doi.org/10.1108/JICV-07-2022-0030
Research paper | Open Access | Issue | Published: 13 September 2022

An enhanced eco-driving strategy based on reinforcement learning for connected electric vehicles: cooperative velocity and lane-changing control

Show Author's Information Haitao Ding,  Wei Li( ),  Nan Xu,  Jianwei Zhang
State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun, China
Keywords:
Electric vehicles, Ecological driving, Reinforcement learning in hybrid action space, Velocity and lane-changing control, Reward function
Cite this article:
Ding H, Li W, Xu N, et al. An enhanced eco-driving strategy based on reinforcement learning for connected electric vehicles: cooperative velocity and lane-changing control. Journal of Intelligent and Connected Vehicles, 2022, 5(3): 316-332. https://doi.org/10.1108/JICV-07-2022-0030
Download citation
EndNote(RIS)
BibTeX

501

Views

30

Downloads

Citations

19

Crossref

N/A

WoS

14

Scopus

N/A

CSCD

Abstract Full text About this article
Purpose

This study aims to propose an enhanced eco-driving strategy based on reinforcement learning (RL) to alleviate the mileage anxiety of electric vehicles (EVs) in the connected environment.

Design/methodology/approach

In this paper, an enhanced eco-driving control strategy based on an advanced RL algorithm in hybrid action space (EEDC-HRL) is proposed for connected EVs. The EEDC-HRL simultaneously controls longitudinal velocity and lateral lane-changing maneuvers to achieve more potential eco-driving. Moreover, this study redesigns an all-purpose and efficient-training reward function with the aim to achieve energy-saving on the premise of ensuring other driving performance.

Findings

To illustrate the performance for the EEDC-HRL, the controlled EV was trained and tested in various traffic flow states. The experimental results demonstrate that the proposed technique can effectively improve energy efficiency, without sacrificing travel efficiency, comfort, safety and lane-changing performance in different traffic flow states.

Originality/value

In light of the aforementioned discussion, the contributions of this paper are two-fold. An enhanced eco-driving strategy based an advanced RL algorithm in hybrid action space (EEDC-HRL) is proposed to jointly optimize longitudinal velocity and lateral lane-changing for connected EVs. A full-scale reward function consisting of multiple sub-rewards with a safety control constraint is redesigned to achieve eco-driving while ensuring other driving performance.


menu
Abstract
Full text
Outline
About this article

An enhanced eco-driving strategy based on reinforcement learning for connected electric vehicles: cooperative velocity and lane-changing control

Show Author's information Haitao Ding, Wei Li( ), Nan Xu, Jianwei Zhang
State Key Laboratory of Automotive Simulation and Control, Jilin University, Changchun, China

Abstract

Purpose

This study aims to propose an enhanced eco-driving strategy based on reinforcement learning (RL) to alleviate the mileage anxiety of electric vehicles (EVs) in the connected environment.

Design/methodology/approach

In this paper, an enhanced eco-driving control strategy based on an advanced RL algorithm in hybrid action space (EEDC-HRL) is proposed for connected EVs. The EEDC-HRL simultaneously controls longitudinal velocity and lateral lane-changing maneuvers to achieve more potential eco-driving. Moreover, this study redesigns an all-purpose and efficient-training reward function with the aim to achieve energy-saving on the premise of ensuring other driving performance.

Findings

To illustrate the performance for the EEDC-HRL, the controlled EV was trained and tested in various traffic flow states. The experimental results demonstrate that the proposed technique can effectively improve energy efficiency, without sacrificing travel efficiency, comfort, safety and lane-changing performance in different traffic flow states.

Originality/value

In light of the aforementioned discussion, the contributions of this paper are two-fold. An enhanced eco-driving strategy based an advanced RL algorithm in hybrid action space (EEDC-HRL) is proposed to jointly optimize longitudinal velocity and lateral lane-changing for connected EVs. A full-scale reward function consisting of multiple sub-rewards with a safety control constraint is redesigned to achieve eco-driving while ensuring other driving performance.

Keywords: Electric vehicles, Ecological driving, Reinforcement learning in hybrid action space, Velocity and lane-changing control, Reward function

References(55)

Afshar, S., Macedo, P., Mohamed, F. and Disfani, V. (2021), “Mobile charging stations for electric vehicles – a review”, Renewable and Sustainable Energy Reviews, Vol. 152, p. 111654, doi: 10.1016/j.rser.2021.111654.
DOI Google Scholar

Bai, Z., Hao, P., Shangguan, W., Cai, B. and Barth, M.J. (2022), “Hybrid reinforcement learning-based eco-driving strategy for connected and automated vehicles at signalized intersections”, IEEE Transactions on Intelligent Transportation Systems, pp. 1-14, doi: 10.1109/TITS.2022.3145798.
DOI Google Scholar

Bertoni, L., Guanetti, J., Basso, M., Masoero, M., Cetinkunt, S. and Borrelli, F. (2017), “An adaptive cruise control for connected energy-saving electric vehicles”, IFAC-PapersOnLine, Vol. 50 No. 1, pp. 2359-2364, doi: 10.1016/j.ifacol.2017.08.425.
DOI Google Scholar

Chen, R., Cassandras, C.G., Tahmasbi-Sarvestani, A., Saigusa, S., Mahjoub, H.N. and Al-Nadawi, Y.K. (2020), “Cooperative time and energy-optimal lane change maneuvers for connected automated vehicles”, IEEE Transactions on Intelligent Transportation Systems, Vol. 23 No. 4, pp. 3445-3460, doi: 10.1109/tits.2020.3036420.
DOI Google Scholar

Deng, S., Li, W. and Wang, T. (2020), “Subsidizing mass adoption of electric vehicles with a risk-averse manufacturer”, Physica A: Statistical Mechanics and Its Applications, Vol. 547, p. 124408, doi: 10.1016/j.physa.2020.124408.
DOI Google Scholar

Dong, H., Zhuang, W., Chen, B., Yin, G. and Wang, Y. (2021), “Enhanced eco-approach control of connected electric vehicles at signalized intersection with queue discharge prediction”, IEEE Transactions on Vehicular Technology, Vol. 70 No. 6, pp. 5457-5469, doi: 10.1109/TVT.2021.3075480.
DOI Google Scholar

Dong, J., Chen, S., Li, Y., Du, R., Steinfeld, A. and Labi, S. (2021), “Space-weighted information fusion using deep reinforcement learning: the context of tactical control of lane-changing autonomous vehicles and connectivity range assessment”, Transportation Research Part C: Emerging Technologies, Vol. 128, p. 103192, doi: 10.1016/j.trc.2021.103192.
DOI Google Scholar

dos Santos, T.C. and Wolf, D.F. (2019), . “Automated conflict resolution of lane change utilizing probability collectives”, 2019 19th International Conference on Advanced Robotics (ICAR), IEEE, pp. 623-628, doi: 10.1109/ICAR46387.2019.8981609.
DOI Google Scholar

Du, Y., Chen, J., Zhao, C., Liu, C., Liao, F. and Chan, C. -Y. (2022), “Comfortable and energy-efficient speed control of autonomous vehicles on rough pavements using deep reinforcement learning”, Transportation Research Part C: Emerging Technologies, Vol. 134, p. 103489, doi: 10.1016/j.trc.2021.103489.
DOI Google Scholar
Erdmann, J. (2015), “SUMO’s lane-changing model”, Modeling Mobility with Open Data, Lecture Notes in Mobility, Springer, Cham.

Fan, Z., Su, R., Zhang, W. and Yu, Y. (2019), “Hybrid actor-critic reinforcement learning in parameterized action space”, arXiv preprint arXiv: 1903.01344, doi: 10.48550/arXiv.1903.01344.
DOI Google Scholar

Galvin, R. (2017), “Energy consumption effects of speed and acceleration in electric vehicles: laboratory case studies and implications for drivers and policymakers”, Transportation Research Part D: Transport and Environment, Vol. 53, pp. 234-248, doi: 10.1016/j.trd.2017.04.020.
DOI Google Scholar

Greenberg, H. (1959), “An analysis of traffic flow”, Operations Research, Vol. 7 No. 1, pp. 79-85, doi: 10.1287/opre.7.1.79.
DOI Google Scholar
Greenshields, B., Bibbins, J., Channing, W. and Miller, H. (1935), “A study of traffic capacity”, Highway Research Board Proceedings, Vol. 14, pp. 448-477, National Research Council (USA), Highway Research Board.

Guo, J., Li, W., Wang, J., Luo, Y. and Li, K. (2021), “Safe and energy-efficient car-following control strategy for intelligent electric vehicles considering regenerative braking”, IEEE Transactions on Intelligent Transportation Systems, Vol. 23 No. 7, pp. 1524-9050, doi: 10.1109/TITS.2021.3066611.
DOI Google Scholar

Guo, Q., Angah, O., Liu, Z. and Ban, X.J. (2021), “Hybrid deep reinforcement learning based eco-driving for low-level connected and automated vehicles along signalized corridors”, Transportation Research Part C: Emerging Technologies, Vol. 124, p. 102980, doi: 10.1016/j.trc.2021.102980.
DOI Google Scholar

Hardman, S., Jenn, A., Tal, G., Axsen, J., Beard, G., Daina, N., Figenbaum, E., Jakobsson, N., Jochem, P. and Kinnear, N. (2018), “A review of consumer preferences of and interactions with electric vehicle charging infrastructure”, Transportation Research Part D: Transport and Environment, Vol. 62, pp. 508-523, doi: 10.1016/j.trd.2018.04.002.
DOI Google Scholar

Hausknecht, M. and Stone, P. (2015), “Deep reinforcement learning in parameterized action space”, arXiv preprint arXiv: 1511.04143, doi: 10.48550/arXiv.1511.04143.
DOI Google Scholar

He, J., Yang, H., Huang, H. -J. and Tang, T. -Q. (2018), “Impacts of wireless charging lanes on travel time and energy consumption in a two-lane road system”, Physica A: Statistical Mechanics and Its Applications, Vol. 500, pp. 1-10, doi: 10.1016/j.physa.2018.02.074.
DOI Google Scholar

He, X. and Wu, X. (2018), “Eco-driving advisory strategies for a platoon of mixed gasoline and electric vehicles in a connected vehicle system”, Transportation Research Part D: Transport and Environment, Vol. 63, pp. 907-922, doi: 10.1016/j.trd.2018.07.014.
DOI Google Scholar

He, X., Fei, C., Liu, Y., Yang, K. and Ji, X. (2020), “Multi-objective longitudinal decision-making for autonomous electric vehicle: a entropy-constrained reinforcement learning approach”, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 1-6, doi: 10.1109/ITSC45102.2020.9294736.
DOI Google Scholar

Jan, L.E., Zhao, J., Aoki, S., Bhat, A., Chang, C. -F. and Rajkumar, R. (2020), “Speed trajectory generation for energy-efficient connected and automated vehicles”, Dynamic Systems and Control Conference, American Society of Mechanical Engineers, Vol. 84287, p. V002T23A001, doi: 10.1115/DSCC2020-3148.
DOI Google Scholar

Kang, L., Shen, H. and Sarker, A. (2017), “Velocity optimization of pure electric vehicles with traffic dynamics consideration”, 2017 IEEE 37th International Conference on Distributed Computing Systems (ICDCS), IEEE, pp. 2206-2211, doi: 10.1109/ICDCS.2017.220.
DOI Google Scholar

Kendall, A., Hawke, J., Janz, D., Mazur, P., Reda, D., Allen, J. -M., Lam, V. -D., Bewley, A. and Shah, A. (2019), “Learning to drive in a day”, 2019 International Conference on Robotics and Automation (ICRA), IEEE, pp. 8248-8254, doi: 10.1109/ICRA.2019.8793742.
DOI Google Scholar

Krasowski, H., Wang, X. and Althoff, M. (2020), “Safe reinforcement learning for autonomous lane changing using set-based prediction”, 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 1-7, doi: 10.1109/ITSC45102.2020.9294259.
DOI Google Scholar

Krizhevsky, A., Sutskever, I. and Hinton, G.E. (2012), “Imagenet classification with deep convolutional neural networks”, Advances in Neural Information Processing Systems, Vol. 25, pp. 1097-1105.
Google Scholar

Li, M., Cao, Z. and Li, Z. (2021), “A reinforcement learning-based vehicle platoon control strategy for reducing energy consumption in traffic oscillations”, IEEE Transactions on Neural Networks and Learning Systems, Vol. 32 No. 12, pp. 5309-5322, doi: 10.1109/TNNLS.2021.3071959.
DOI Google Scholar

Li, Y., Zhong, Z., Zhang, K. and Zheng, T. (2019), “A car-following model for electric vehicle traffic flow based on optimal energy consumption”, Physica A: Statistical Mechanics and Its Applications, Vol. 533, p. 122022, doi: 10.1016/j.physa.2019.122022.
DOI Google Scholar

Lopez, P.A., Behrisch, M., Bieker-Walz, L., Erdmann, J., Flötteröd, Y. -P., Hilbrich, R., Lücken, L., Rummel, J., Wagner, P. and Wießner, E. (2018), “Microscopic traffic simulation using sumo”, 2018 21st International Conference on Intelligent Transportation Systems (ITSC), IEEE, pp. 2575-2582, doi: 10.1109/ITSC.2018.8569938.
DOI Google Scholar

Lu, C., Dong, J. and Hu, L. (2019), “Energy-efficient adaptive cruise control for electric connected and autonomous vehicles”, IEEE Intelligent Transportation Systems Magazine, Vol. 11 No. 3, pp. 42-55, doi: 10.1109/MITS.2019.2919556.
DOI Google Scholar

Mnih, V., Kavukcuoglu, K., Silver, D., Graves, A., Antonoglou, I., Wierstra, D. and Riedmiller, M. (2013), “Playing Atari with deep reinforcement learning”, arXiv preprint arXiv: 1312.5602, doi: 10.48550/arXiv.1312.5602.
DOI Google Scholar

Olovsson, T., Svensson, T. and Wu, J. (2022), “Future connected vehicles: communications demands, privacy and cyber-security”, Communications in Transportation Research, Vol. 2, doi: 10.1016/j.commtr.2022.100056.
DOI Google Scholar

Park, S., Oh, C., Kim, Y., Choi, S. and Park, S. (2019), “Understanding impacts of aggressive driving on freeway safety and mobility: a multi-agent driving simulation approach”, Transportation Research Part F: traffic Psychology and Behaviour, Vol. 64, pp. 377-387, doi: 10.1016/j.trf.2019.05.017.
DOI Google Scholar

Qu, X., Zhang, J. and Wang, S. (2017), “On the stochastic fundamental diagram for freeway traffic: model development, analytical properties, validation, and extensive applications”, Transportation Research Part B: methodological, Vol. 104, pp. 256-271, doi: 10.1016/j.trb.2017.07.003.
DOI Google Scholar

Qu, X., Yu, Y., Zhou, M., Lin, C. -T. and Wang, X. (2020), “Jointly dampening traffic oscillations and improving energy consumption with electric, connected and automated vehicles: a reinforcement learning based approach”, Applied Energy, Vol. 257, p. 114030, doi: 10.1016/j.apenergy.2019.114030.
DOI Google Scholar

Rezaee, K., Yadmellat, P., Nosrati, M.S., Abolfathi, E.A., Elmahgiubi, M. and Luo, J. (2019), “Multi-lane cruising using hierarchical planning and reinforcement learning”, 2019 IEEE Intelligent Transportation Systems Conference (ITSC), IEEE, pp. 1800-1806, doi: 10.1109/ITSC.2019.8916928.
DOI Google Scholar

Schulman, J., Wolski, F., Dhariwal, P., Radford, A. and Klimov, O. (2017), “Proximal policy optimization algorithms”, arXiv pre-print server, doi: 10.48550/arXiv.1707.06347.
DOI Google Scholar

Shi, X., Wang, Z., Li, X. and Pei, M. (2021), “The effect of ride experience on changing opinions toward autonomous vehicle safety”, Communications in Transportation Research, Vol. 1, p. 100003, doi: 10.1016/j.commtr.2021.100003.
DOI Google Scholar

Silgu, M.A., Erdai, S.G., Gksu, G. and Celikoglu, H.B. (2021), “Combined control of freeway traffic involving cooperative adaptive cruise controlled and human driven vehicles using feedback control through sumo”, IEEE Transactions on Intelligent Transportation Systems, Vol. 23 No. 8, pp. 11011-11025, doi: 10.1109/TITS.2021.3098640.
DOI Google Scholar
Silver, D., Lever, G., Heess, N., Degris, T., Wierstra, D. and Riedmiller, M. (2014), “Deterministic policy gradient algorithms”, International Conference on Machine Learning, PMLR, pp. 387-395, available at: https://proceedings.mlr.press/v32/silver14.html

Srisomboon, I. and Lee, S. (2021), “Efficient position change algorithms for prolonging driving range of a truck platoon”, Applied Sciences, Vol. 11 No. 22, p. 10516, doi: 10.3390/app112210516.
DOI Google Scholar

Tajeddin, S., Ekhtiari, S., Faieghi, M. and Azad, N.L. (2019), “Ecological adaptive cruise control with optimal lane selection in connected vehicle environments”, IEEE Transactions on Intelligent Transportation Systems, Vol. 21 No. 11, pp. 4538-4549, doi: 10.1109/TITS.2019.2938726.
DOI Google Scholar

Tran, M. -K., Bhatti, A., Vrolyk, R., Wong, D., Panchal, S., Fowler, M. and Fraser, R. (2021), “A review of range extenders in battery electric vehicles: current progress and future perspectives”, World Electric Vehicle Journal, Vol. 12 No. 2, p. 54, doi: 10.3390/wevj12020054.
DOI Google Scholar

Treiber, M., Hennecke, A. and Helbing, D. (2000), “Congested traffic states in empirical observations and microscopic simulations”, Physical Review E, Vol. 62 No. 2, pp. 1805-1824, doi: 10.1103/PhysRevE.62.1805.
DOI Google Scholar

Vahidi, A. and Sciarretta, A. (2018), “Energy saving potentials of connected and automated vehicles”, Transportation Research Part C: Emerging Technologies, Vol. 95, pp. 822-843, doi: 10.1016/j.trc.2018.09.001.
DOI Google Scholar

Vaz, W.S., Nandi, A.K. and Koylu, U.O. (2015), “A multiobjective approach to find optimal electric-vehicle acceleration: simultaneous minimization of acceleration duration and energy consumption”, IEEE Transactions on Vehicular Technology, Vol. 65 No. 6, pp. 4633-4644, doi: 10.1109/TVT.2015.2497246.
DOI Google Scholar

Wang, S., Wang, Z., Jiang, R., Yan, R. and Du, L. (2022), “Trajectory jerking suppression for mixed traffic flow at a signalized intersection: a trajectory prediction based deep reinforcement learning method”, IEEE Transactions on Intelligent Transportation Systems, doi: 10.1109/TITS.2022.3152550.
DOI Google Scholar

Wegener, A., Pirkowski, M., Raya, M., Hellbröck, H., Fischer, S. and Hubaux, J. -P. (2008), “Traci: an interface for coupling road traffic and network simulators”, Proceedings of the 11th Communications and Networking Simulation Symposium, pp. 155-163, doi: 10.1145/1400713.1400740.
DOI Google Scholar
Wilson, T.B., Butler, W., McGehee, D.V. and Dingus, T.A. (1997), “Forward-looking collision warning system performance guidelines”, SAE Transactions, Vol. 106, pp. 701-725, available at: www.jstor.org/stable/44731227

Xiong, J., Wang, Q., Yang, Z., Sun, P., Han, L., Zheng, Y., Fu, H., Zhang, T., Liu, J. and Liu, H. (2018), “Parametrized deep q-networks learning: reinforcement learning with discrete-continuous hybrid action space”, arXiv pre-print server, doi: 10.48550/arXiv.1810.06394.
DOI Google Scholar

Xu, L., Yin, G., Li, G., Hanif, A. and Bian, C. (2018), “Stable trajectory planning and energy-efficience control allocation of lane change maneuver for autonomous electric vehicle”, Journal of Intelligent and Connected Vehicles, Vol. 1 No. 2, pp. 55-65, doi: 10.1108/JICV-12-2017-0002.
DOI Google Scholar

Xu, W., Chen, H., Zhao, H. and Ren, B. (2019), “Torque optimization control for electric vehicles with four in-wheel motors equipped with regenerative braking system”, Mechatronics, Vol. 57, pp. 95-108, doi: 10.1016/j.mechatronics.2018.11.006.
DOI Google Scholar

Ye, F., Cheng, X., Wang, P., Chan, C. -Y. and Zhang, J. (2020), “Automated lane change strategy using proximal policy optimization-based deep reinforcement learning”, 2020 IEEE Intelligent Vehicles Symposium (IV), IEEE, pp. 1746-1752, doi: 10.1109/IV47402.2020.9304668.
DOI Google Scholar

Yu, S., Fu, R., Guo, Y., Xin, Q. and Shi, Z. (2019), “Consensus and optimal speed advisory model for mixed traffic at an isolated signalized intersection”, Physica A: Statistical Mechanics and Its Applications, Vol. 531, p. 121789, doi: 10.1016/j.physa.2019.121789.
DOI Google Scholar

Zhu, M., Wang, Y., Pu, Z., Hu, J., Wang, X. and Ke, R. (2020), “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving”, Transportation Research Part C: Emerging Technologies, Vol. 117, p. 102662, doi: 10.1016/j.trc.2020.102662.
DOI Google Scholar
Publication history
Copyright
Rights and permissions

Publication history

Received: 12 July 2022
Revised: 07 August 2022
Accepted: 08 August 2022
Published: 13 September 2022
Issue date: October 2022

Copyright

© 2022 Haitao Ding, Wei Li, Nan Xu and Jianwei Zhang. Published in Journal of Intelligent and Connected Vehicles. Published by Emerald Publishing Limited.

Rights and permissions

This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence maybe seen at http://creativecommons.org/licences/by/4.0/legalcode

Return