Construction of a standardized evaluation system and toolkit development for interdisciplinary eye-tracking experiments

As eye-tracking research increasingly transitions from constrained laboratory environments to complex, high-ecological-validity scenarios, such as virtual reality, autonomous driving, and clinical diagnostics, methodological heterogeneity has emerged as a primary barrier to reproducibility. Current research lacks a unified standard for documenting hardware constraints, data quality baselines, and event-parsing algorithms. This study aims to address these critical inconsistencies by establishing a standardized reporting protocol and providing a transparent computational infrastructure. This research is motivated by the widespread “black-box” nature of proprietary analysis software and the limited transparency in data quality disclosure within the interdisciplinary eye-tracking community.

The research was conducted in two distinct stages. First, a systematic review following the PRISMA 2020 guidelines was performed to quantify prevailing reporting practices. A final corpus of 253 empirical studies published between 2014 and 2025 was retrieved from major databases, including Web of Science and Scopus. Based on the identified reporting deficiencies, the Interdisciplinary Eye-Tracking Reporting Checklist (IER-Checklist) was developed, comprising 51 items organized into five dimensions: Participants and Design; Apparatus and Environment; Calibration and Validation; Processing and Event Detection; and Quality and Statistics. Concurrently, the IER-Metrics toolkit, an open-source, Python-based environment, was developed for automated metric extraction. The toolkit implements a multistage protocol, including automated handling of missing data through linear or cubic spline interpolation, physiological blink artifact removal, and oculomotor event parsing using the velocity-threshold identification (I-VT) algorithm. A key algorithmic innovation is the implementation of a dynamic bounding box for stationary gaze entropy (SGE) calculations. This approach isolates the instantaneous working visual field to prevent probability distribution bias caused by head movements in dynamic environments. To assess the framework’s effectiveness, four postdoctoral experts conducted a between-subjects validation study by evaluating 80 empirical manuscripts. Furthermore, the computational accuracy of the IER-Metrics toolkit was benchmarked against established commercial software using a dataset comprising 95 independent recording sessions and examining advanced spatiotemporal features, such as the nearest neighbor index (NNI) and SGE.

The systematic review revealed that only 35% of the literature reported essential measures of spatial accuracy and data loss rates, whereas fixation thresholds ranged from 60 ms to 200 ms without explicit justification. Experimental validation showed that application of the IER-Checklist significantly improved evaluative consistency, with inter-rater agreement (Cohen’s κ) increasing from 0.61 under traditional heuristic assessment to 0.86. Moreover, evaluators using the IER-Checklist identified a significantly higher mean number of methodological omissions (25.15 per manuscript) than those in the traditional assessment group (8.18 per manuscript, p < .001). In the computational benchmarking, IER-Metrics demonstrated high concordance with commercial software for the NNI (r = 0.923, R² = 0.853). For SGE calculations, although a correlation of r = 0.744 was observed, the self-developed toolkit demonstrated greater sensitivity. This outcome was attributed to the dynamic bounding box algorithm, which effectively handled ego-motion data that fixed-grid commercial algorithms often underestimate because of extensive zero-probability peripheral zones. Finally, the toolkit achieved a mean saccade detection accuracy exceeding 98% compared with industry gold standards.

This study establishes a standardized evaluation system and a reproducible computational environment for interdisciplinary eye-tracking research. By integrating the 51-item IER-Checklist with the open-source IER-Metrics toolkit, the proposed framework enables researchers to reduce selection bias and ensure algorithmic transparency. The findings underscore that standardized reporting is an operational necessity in naturalistic research settings. Ultimately, this framework supports the transition toward open science by reducing reliance on proprietary software and establishing a structured mechanism for rigorous experimental design in human–environment interaction research.