Experimental assessment of situational awareness in air traffic management based on inquiry and probing measurement

Situational awareness (SA) among air traffic controllers during command operations is critical for safeguarding the operational safety of aircraft. The Situation Awareness Global Assessment Technique (SAGAT) and Situation Present Assessment Method (SPAM) are widely used SA measurement tools; however, their practical application has sparked extensive debate in academic circles. Therefore, in this study, a basic air traffic control (ATC) experimental scenario was developed, and participants’ SA levels were measured using SAGAT and SPAM, respectively. The performance of the two methods in ATC contexts across three key dimensions (intrusiveness, sensitivity, and predictability) was comparatively evaluated using statistical data analysis.

First, the measurement procedures and evaluation metrics of both methods were designed and selected based on a literature review. Second, an ATC scenario was simulated in an experimental setting, where SAGAT and SPAM were applied to quantify the participants’ SA. The two methods were compared using statistical analyses, with specific approaches tailored to each evaluation dimension. To assess the intrusiveness, participants completed the NASA Task Load Index (NASA-TLX) to rate their workload, which covers six dimensions: mental demand, physical demand, temporal demand, performance, effort, and frustration. The weight of each dimension was determined via the pairwise comparison method. After eliminating gross errors, an independent samples t-test was used to examine the differences in the data across each dimension for the two groups. To assess the sensitivity, the SA values measured by the two methods were divided into high and low groups based on the median, after which a test was conducted to determine whether there were significant differences in various command performance indicators for the high-SA versus low-SA groups. The command performance indicators were derived from the Key Performance Indicators specified in the Global Air Navigation Plan by the International Civil Aviation Organization and adjusted to fit the simulation scenario. Specifically, these indicators include the mean Lateral Separation (LS), average Flight Miles (FM), Level flight Holding Time (LHT), Handover Time (HT), and Remaining instruction Count (RC). To assess the predictability, a Multiple Linear Regression (MLR) model was used to analyze the relationship between the measured SA data at three hierarchical levels and the performance indicators. The evaluation metrics for the model include the adjusted R² value and p-value.

The comparative analysis indicated that: 1) independent samples t-tests on the NASA-TLX data for evaluating the intrusiveness using SAGAT and SPAM showed significant differences only for the mental demand (p = 0.037) and temporal demand (p = 0.044), with no significant differences in the other four dimensions. Overall, similar intrusiveness levels were obtained with the two methods. 2) Independent samples t-tests to assess the sensitivity indicated significant differences in three performance indicators (LS (p = 0.043), FM (p = 0.026), and LHT (p = 0.042)) for the high and low SA groups identified by SAGAT. In contrast, for the high and low SA groups identified by SPAM, significant differences were observed only for RC (p = 0.03). The results demonstrate that SAGAT is more sensitive to capturing variations in the operational performance of controllers. 3) MLR analysis of the predictability shows that the SA assessment using SAGAT provides more accurate predictions for most performance indicators. Using SAGAT, significant regression relationships were observed between the SA scores and four performance indicators: FM, LHT, HT, and RC. Using SPAM, a significant regression relationship was observed only for RC. Additionally, the adjusted R² values indicated that compared to SPAM, SAGAT better explained the variations in most performance indicators, confirming that SAGAT provides more accurate predictions of the operational performance of controllers.

Statistical data analysis was used to systematically compare SAGAT and SPAM using simulated ATC experiments, focusing on three critical evaluation dimensions. This study reveals the differences between the two methods in terms of the level of cognitive adaptability and measurement performance: SAGAT is suitable for probing high-level SA, whereas SPAM is more appropriate for measuring low-level SA. Based on these results, SAGAT should be prioritized for application in high-complexity ATC scenarios, whereas SPAM is preferred for real-time scenarios involving information-dense tasks. This research provides evidence-based guidance for selecting SA measurement tools in ATC practice and related experimental studies.