Multi-modal intelligent situation awareness in real-time air traffic control: Control intent understanding and flight trajectory prediction^☆

With the advent of the next-generation Air Traffic Control (ATC) system, there is growing interest in using Artificial Intelligence (AI) techniques to enhance Situation Awareness (SA) for ATC Controllers (ATCOs), i.e., Intelligent SA (ISA). However, the existing AI-based SA approaches often rely on unimodal data and lack a comprehensive description and benchmark of the ISA tasks utilizing multi-modal data for real-time ATC environments. To address this gap, by analyzing the situation awareness procedure of the ATCOs, the ISA task is refined to the processing of the two primary elements, i.e., spoken instructions and flight trajectories. Subsequently, the ISA is further formulated into Controlling Intent Understanding (CIU) and Flight Trajectory Prediction (FTP) tasks. For the CIU task, an innovative automatic speech recognition and understanding framework is designed to extract the controlling intent from unstructured and continuous ATC communications. For the FTP task, the single- and multi-horizon FTP approaches are investigated to support the high-precision prediction of the situation evolution. A total of 32 unimodal/multi-modal advanced methods with extensive evaluation metrics are introduced to conduct the benchmarks on the real-world multi-modal ATC situation dataset. Experimental results demonstrate the effectiveness of AI-based techniques in enhancing ISA for the ATC environment.