Journal Home > Volume 28 , Issue 5
Tsinghua Science and Technology 2023, 28(5): 965-975 https://doi.org/10.26599/TST.2022.9010037
Open Access | Issue | Published: 19 May 2023

Autonomous Vehicles Testing Considering Utility-Based Operable Tasks

Show Author's Information Jingwei Ge1,  Jiawei Zhang1,  Yi Zhang2( ),  Danya Yao1,  Zuo Zhang1,  Rui Zhou3
Department of Automation, Tsinghua University, Beijing 100084, China
Department of Automation, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China, and with Tsinghua-Berkeley Shenzhen Institute (TBSI), Shenzhen 518055, China, and also with Jiangsu Province Collaborative Innovation Center of Modern Urban Traffic Technologies, Nanjing 210096, China
Macau University of Science and Technology, Macau 999079, China, and also with Waytous Inc., Shenzhen 518000, China
Keywords:
Autonomous Vehicle (AV), intelligence testing, operable tasks
Cite this article:
Ge J, Zhang J, Zhang Y, et al. Autonomous Vehicles Testing Considering Utility-Based Operable Tasks. Tsinghua Science and Technology, 2023, 28(5): 965-975. https://doi.org/10.26599/TST.2022.9010037
Download citation
EndNote(RIS)
BibTeX

364

Views

34

Downloads

Citations

2

Crossref

1

WoS

2

Scopus

0

CSCD

Abstract Full text About this article

Virtual simulation testing of Autonomous Vehicles (AVs) is gradually being accepted as a mandatory way to test the feasibility of driving strategies for AVs. Mainstream methods focus on improving testing efficiency by extracting critical scenarios from naturalistic driving datasets. However, the criticalities defined in their testing tasks are based on fixed assumptions, the obtained scenarios cannot pose a challenge to AVs with different strategies. To fill this gap, we propose an intelligent testing method based on operable testing tasks. We found that the driving behavior of Surrounding Vehicles (SVs) has a critical impact on AV, which can be used to adjust the testing task difficulty to find more challenging scenarios. To model different driving behaviors, we utilize behavioral utility functions with binary driving strategies. Further, we construct a vehicle interaction model, based on which we theoretically analyze the impact of changing the driving behaviors on the testing task difficulty. Finally, by adjusting SV’s strategies, we can generate more corner cases when testing different AVs in a finite number of simulations.


menu
Abstract
Full text
Outline
About this article

Autonomous Vehicles Testing Considering Utility-Based Operable Tasks

Show Author's information Jingwei Ge1, Jiawei Zhang1, Yi Zhang2( ), Danya Yao1, Zuo Zhang1, Rui Zhou3
Department of Automation, Tsinghua University, Beijing 100084, China
Department of Automation, Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing 100084, China, and with Tsinghua-Berkeley Shenzhen Institute (TBSI), Shenzhen 518055, China, and also with Jiangsu Province Collaborative Innovation Center of Modern Urban Traffic Technologies, Nanjing 210096, China
Macau University of Science and Technology, Macau 999079, China, and also with Waytous Inc., Shenzhen 518000, China

Abstract

Virtual simulation testing of Autonomous Vehicles (AVs) is gradually being accepted as a mandatory way to test the feasibility of driving strategies for AVs. Mainstream methods focus on improving testing efficiency by extracting critical scenarios from naturalistic driving datasets. However, the criticalities defined in their testing tasks are based on fixed assumptions, the obtained scenarios cannot pose a challenge to AVs with different strategies. To fill this gap, we propose an intelligent testing method based on operable testing tasks. We found that the driving behavior of Surrounding Vehicles (SVs) has a critical impact on AV, which can be used to adjust the testing task difficulty to find more challenging scenarios. To model different driving behaviors, we utilize behavioral utility functions with binary driving strategies. Further, we construct a vehicle interaction model, based on which we theoretically analyze the impact of changing the driving behaviors on the testing task difficulty. Finally, by adjusting SV’s strategies, we can generate more corner cases when testing different AVs in a finite number of simulations.

Keywords: Autonomous Vehicle (AV), intelligence testing, operable tasks

References(37)

[1]
S. Chen, Z. Jian, Y. Huang, Y. Chen, Z. Zhou, and N. Zheng, Autonomous driving: Cognitive construction and situation understanding, Sci. China Inf. Sci., vol. 62, no. 8, p. 81101, 2019.
DOI Google Scholar
[2]
W. Schwarting, J. Alonso-Mora, and D. Rus, Planning and decision-making for autonomous vehicles, Annu. Rev. Control Rob. Auton. Syst., vol. 1, pp. 187–210, 2018.
DOI Google Scholar
[3]
L. Li, W. L. Huang, Y. Liu, N. N. Zheng, and F. Y. Wang, Intelligence testing for autonomous vehicles: A new approach, IEEE Trans. Intell. Veh., vol. 1, no. 2, pp. 158–166, 2016.
DOI Google Scholar
[4]
S. Khastgir, G. Dhadyalla, S. Birrell, S. Redmond, R. Addinall, and P. Jennings, Test scenario generation for driving simulators using constrained randomization technique, presented at the SAE World Congress Experience, Detroit, MI, USA, 2017.
DOI Google Scholar
[5]
L. Huang, K. Wang, Y. Lv, and F. Zhu, Autonomous vehicles testing methods review, in Proc. 2016 IEEE 19th Int. Conf. on Intelligent Transportation Systems, Rio de Janeiro, Brazil, 2016, pp. 163–168.
DOI Google Scholar
[6]
J. Zhou and L. del Re, Reduced complexity safety testing for ADAS & ADF, IFAC-PapersOnLine, vol. 50, no. 1, pp. 5985–5990, 2017.
DOI Google Scholar
[7]
D. Bezzina and J. Sayer, Safety Pilot Model Deployment: Test Conductor Team Report. Washington, DC, USA: National Highway Traffic Safety Administration, 2014.
[8]
D. Zhao, H. Lam, H. Peng, S. Bao, D. J. LeBlanc, K. Nobukawa, and C. S. Pan, Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques, IEEE Trans. Intell. Transp. Syst., vol. 18, no. 3, pp. 595–607, 2017.
DOI Google Scholar
[9]
D. Zhao, X. Huang, H. Peng, H. Lam, and D. J. LeBlanc, Accelerated evaluation of automated vehicles in car-following maneuvers, IEEE Trans. Intell. Transp. Syst., vol. 19, no. 3, pp. 733–744, 2018.
DOI Google Scholar
[10]
S. Feng, Y. Feng, C. Yu, Y. Zhang, and H. X. Liu, Testing scenario library generation for connected and automated vehicles, Part I: Methodology, IEEE Trans. Intell. Transp. Syst., vol. 22, no. 3, pp. 1573–1582, 2021.
DOI Google Scholar
[11]
X. Yan, S. Feng, H. Sun, and H. X. Liu, Distributionally consistent simulation of naturalistic driving environment for autonomous vehicle testing, arXiv preprint arXiv: 2101.02828, 2021.
Google Scholar
[12]
S. Feng, X. Yan, H. Sun, Y. Feng, and H. X. Liu, Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment, Nat. Commun., vol. 12, no. 1, p. 748, 2021.
DOI Google Scholar
[13]
M. Koren, S. Alsaif, R. Lee, and M. J. Kochenderfer, Adaptive stress testing for autonomous vehicles, in Proc. 2018 IEEE Intelligent Vehicles Symp., Changshu, China, 2018, pp. 1–7.
DOI Google Scholar
[14]
M. Klischat, I. E. Liu, F. Holtke, and M. Althoff, Scenario factory: Creating safety-critical traffic scenarios for automated vehicles, in Proc. 2020 IEEE 23rd Int. Conf. on Intelligent Transportation Systems, Rhodes, Greece, 2020, pp. 1–7.
DOI Google Scholar
[15]
L. Li, Y. L. Lin, N. N. Zheng, F. Y. Wang, Y. Liu, D. Cao, K. Wang, and W. L. Huang, Artificial intelligence test: A case study of intelligent vehicles, Artif. Intell. Rev., vol. 50, no. 3, pp. 441–465, 2018.
DOI Google Scholar
[16]
A. Corso and J. M. Kochenderfer, Interpretable safety validation for autonomous vehicles, in Proc. 2020 IEEE 23rd Int. Conf. on Intelligent Transportation Systems, Rhodes, Greece, 2020, pp. 1–6.
DOI Google Scholar
[17]
L. Li, N. Zheng, and Y. F. Wang, A theoretical foundation of intelligence testing and its application for intelligent vehicles, IEEE Trans. Intell. Transp. Syst., vol. 22, no. 10, pp. 6297–6306, 2021.
DOI Google Scholar
[18]
G. Wang, J. Hu, Z. Li, and L. Li, Harmonious lane changing via deep reinforcement learning, IEEE Trans. Intell. Transp. Syst., vol. 23, no. 5, pp. 4642–4650, 2022.
DOI Google Scholar
[19]
C. Zhao, L. Li, X. Pei, Z. Li, F. Y. Wang, and X. Wu, A comparative study of state-of-the-art driving strategies for autonomous vehicles, Accid. Anal. Prev., vol. 150, p. 105937, 2021.
DOI Google Scholar
[20]
U. Khan, P. Basaras, L. Schmidt-Thieme, A. Nanopoulos, and D. Katsaros, Analyzing cooperative lane change models for connected vehicles, in Proc. 2014 Int. Conf. on Connected Vehicles and Expo, Vienna, Austria, 2014, pp. 565–570.
DOI Google Scholar
[21]
Z. Li, J. Jiang, and W. H. Chen, Automatic lane change maneuver in dynamic environment using model predictive control method, in Proc. 2020 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems, Las Vegas, NV, USA, 2020, pp. 2384–2389.
DOI Google Scholar
[22]
W. E. Armstrong, Uncertainty and the utility function, Econ. J., vol. 58, no. 229, pp. 1–10, 1948.
DOI Google Scholar
[23]
J. C. Harsanyi, Cardinal utility in welfare economics and in the theory of risk-taking, J. Polit. Econ., vol. 61, no. 5, pp. 434–435, 1953.
DOI Google Scholar
[24]
L. Chen, B. Wang, X. Chen, X. Zhang, and D. Yang, Utility-based resource allocation for mixed traffic in wireless networks, in Proc. 2011 IEEE Conf. on Computer Communications Workshops, Shanghai, China, 2011, pp. 91–96.
DOI Google Scholar
[25]
G. Arslan, R. J. Marden, and S. J. Shamma, Autonomous vehicle-target assignment: A game-theoretical formulation, J. Dyn. Syst. Meas. Control, vol. 129, no. 5, pp. 584–596, 2007.
DOI Google Scholar
[26]
T. Toledo, N. H. Koutsopoulos, and M. E. Ben-Akiva, Modeling integrated lane-changing behavior, Transp. Res. Rec., vol. 1857, no. 1, pp. 30–38, 2003.
DOI Google Scholar
[27]
E. Altendorf, C. Schreck, G. Weßel, Y. Canpolat, and F. Flemisch, Utility assessment in automated driving for cooperative human–machine systems, Cognition, Technology & Work, vol. 21, no. 4, pp. 607–619, 2019.
DOI Google Scholar
[28]
J. Ge, H. Xu, J. Zhang, Y. Zhang, D. Yao, and L. Li, Heterogeneous driver modeling and corner scenarios sampling for automated vehicles testing, J. Adv. Transp., vol. 2022, p. 8655514, 2022.
DOI Google Scholar
[29]
W. H. Ma and H. Peng, A worst-case evaluation method for dynamic systems, J. Dyn. Syst. Meas. Control, vol. 121, no. 2, pp. 191–199, 1999.
DOI Google Scholar
[30]
Y. Kou, Development and evaluation of integrated chassis control systems, PhD dissertation, University of Michigan, Ann Arbor, MI, USA, 2010.
[31]
S. Feng, Y. Feng, X. Yan, S. Shen, S. Xu, and H. X. Liu, Safety assessment of highly automated driving systems in test tracks: A new framework, Accident Analysis & Prevention, vol. 144, p. 105664, 2020.
DOI Google Scholar
[32]
D. Zhao, Accelerated evaluation of automated vehicles, PhD dissertation, University of Michigan, Ann Arbor, MI, USA, 2016.
[33]
J. Zhang, C. Chang, H. Pei, X. Peng, Y. Guo, R. Lian, Z. Chen, and L. Li, CAVSIM: A microscope traffic simulator for connected and automated vehicles environment, in Proc. 2022 IEEE 25thInt. Conf. on Intelligent Transportation Systems, Macau, China, 2022, pp. 3719–3724.
DOI Google Scholar
[34]
J. Zhang, H. Pei, J. X. Ban, and L. Li, Analysis of cooperative driving strategies at road network level with macroscopic fundamental diagram, Transp. Res. Part C Emerging Technol., vol. 135, p. 103503, 2022.
DOI Google Scholar
[35]
J. Ge, H. Pei, D. Yao, and Y. Zhang, A robust path tracking algorithm for connected and automated vehicles under i-VICS, Transp. Res. Interdiscip. Perspect., vol. 9, p. 100314, 2021.
DOI Google Scholar
[36]
Y. Meng, L. Li, Y. F. Wang, K. Li, and Z. Li, Analysis of cooperative driving strategies for nonsignalized intersections, IEEE Trans. Veh. Technol., vol. 67, no. 4, pp. 2900–2911, 2018.
DOI Google Scholar
[37]
L. Li, X. Peng, Y. F. Wang, D. Cao, and L. Li, A situation-aware collision avoidance strategy for car-following, IEEE/CAA J. Autom. Sin., vol. 5, no. 5, pp. 1012–1016, 2018.
DOI Google Scholar
Publication history
Copyright
Acknowledgements
Rights and permissions

Publication history

Received: 31 May 2022
Revised: 18 August 2022
Accepted: 14 September 2022
Published: 19 May 2023
Issue date: October 2023

Copyright

© The author(s) 2023.

Acknowledgements

This work was supported in part by the National Key Research and Development (No. 2021YFB2501200).

Rights and permissions

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).

Return