Performance Analysis of an AI-Based IDS xApp for Cyberattack Anomaly Detection in O-RAN Near-RT RIC

The introduction of the Open Radio Access Network (O-RAN) architecture enhances network flexibility but introduces novel security threats targeting open interfaces and the RAN Intelligent Controller (RIC). Particularly in the Near-RT RIC environment, an effective Intrusion Detection System (IDS) that satisfies strict near-real-time constraints of within 1 s is essential to defend against cyber attacks. This paper proposes an Artificial Intelligence (AI)-based IDS xApp designed for real-time cyber attack monitoring in the O-RAN Near-RT RIC environment, and quantitatively analyzes its anomaly detection performance and inference latency characteristics against multi-layer security threats utilizing Open RAN Centralized Unit(O-CU) network layer data and Open RAN Distributed Unit (O-DU) radio telemetry data. Evaluation using a public dataset (NetsLab 5G O-RAN IDD) on four deep learning models (LSTM, CNN, Transformer, Autoencoder) showed that supervised learning-based models achieved high F1-scores (reaching up to 0.99) on both datasets. Furthermore, their performance variation remained highly stable at approximately the $\pm$0.1 pp level upon transition from the training environment (the Service and Management Orchestration, SMO) to the deployment environment (Near-RT RIC). In the inference latency analysis, the system’s scalability was evaluated by increasing the number of prediction instances up to 80,000. The results confirmed that the latency follows a highly predictable linear time complexity ( $𝒪(N)$). Specifically, the LSTM, CNN, and Autoencoder models successfully maintained a response time within 1000 ms even under the maximum load of 80,000 instances across both datasets, whereas the computationally heavy Transformer model experienced resource exhaustion in the KServe inference pod at approximately 20,000 instances, causing the inference process to terminate and rendering further measurement infeasible. Comprehensively, the LSTM model demonstrated the most outstanding balance between performance and operational efficiency by recording stable detection performance, short tail latency (approximately 140 ms at P99), and low training resource consumption. This study experimentally demonstrates the anomaly detection performance of the IDS xApp in the O-RAN near-real-time control environment, and comprehensively verifies its practical effectiveness by considering both inference latency and resource consumption.