Current autonomous vehicles (AVs) primarily respond to external traffic conditions and treat humans as passive occupants, limiting their ability to personalize driving behavior and handle ambiguous scenarios that could benefit from occupant input. This study proposes PACE-ADS (Psychology and Cognition Enabled Automated Driving Systems), a human-centered autonomy framework that enables AVs to interpret and respond to both external traffic conditions and internal occupant states. PACE-ADS employs three collaborating foundation model agents: the Driver Agent interprets the external environment; the Psychologist Agent infers occupant state from passive psychological signals (e.g., facial expressions) and interprets active cognitive inputs (e.g., verbal commands); and the Coordinator Agent integrates these inputs to generate high-level driving behavior decisions. Operating at the low-frequency semantic planning layer, PACE-ADS complements existing AV stacks and activates selectively in response to changes in occupant state, cognitive instructions, or vehicle immobilization. Closed-loop CARLA simulations under prescribed occupant-state trajectories show that PACE-ADS adapts driving behavior along kinematic dimensions associated with ride comfort and supports recovery from operational immobilization through self-reasoning and occupant-in-the-loop guidance.

As large language models (LLMs) are increasingly integrated into autonomous vehicles (AV), understanding their reasoning and behavioral tendencies becomes essential. Trained on vast datasets, LLMs carry behavioral priors and social biases that may shape their driving decisions. Without such insight, developers may struggle to align models for AV needs. To address this, we probe prompt-conditioned high-level action choices of LLMs, each with 1,500 contextual variants. Three widely used LLMs are evaluated with multilingual prompts to select from predefined behavioral options ordered by aggressiveness. An Ordered Logit Model quantifies how contextual factors influence decisions, complemented by thematic analysis to reveal underlying reasoning tendencies. Results show that LLM decisions reflect a mix of model characteristics, linguistic framing, and scenario context. Across conditions, models remain sensitive to rider urgency, traffic complexity, and road-user types. GPT is more conservative, while DeepSeek and LLaMA act more assertively, especially in vehicle interactions. Prompt language also matters. Chinese and French prompts are associated with more assertive behavior than English, with French strongest. Across scenarios, all models shift toward more protective behavior when vulnerable road users appear, reducing aggressiveness and prioritizing safety and smoother flow. These findings characterize model-level behavioral priors relevant to LLM choice and prompt design in AV applications. 

This study presents the development of a comprehensive dataset capturing interactions between autonomous vehicles (AVs) and traffic control devices, specifically traffic lights and stop signs. Derived from the Waymo Motion dataset, our work addresses a critical gap in the existing literature by providing real-world trajectory data on how AVs navigate these traffic control devices. We propose a methodology for identifying and extracting relevant interaction trajectory data from the Waymo Motion dataset, incorporating over 37,000 instances with traffic lights and 44,000 with stop signs. Our methodology includes defining rules to identify various interaction types, extracting trajectory data, and applying a wavelet-based denoising method to smooth the acceleration and speed profiles and eliminate anomalous values, thereby enhancing the trajectory quality. Quality assessment metrics indicate that trajectories obtained in this study have anomaly proportions in acceleration and jerk profiles reduced to near-zero levels across all interaction categories. By making this dataset publicly available, we aim to address the current gap in datasets containing AV interaction behaviors with traffic lights and signs. Based on the organized and published dataset, we can gain a more in-depth understanding of AVs’ behavior when interacting with traffic lights and signs. This will facilitate research on AV integration into existing transportation infrastructures and networks, supporting the development of more accurate behavioral models and simulation tools.