With the increasing emphasis on personal information protection, encryption through security protocols has emerged as a critical requirement in data transmission and reception processes. Nevertheless, IoT ecosystems comprise heterogeneous networks where outdated systems coexist with the latest devices, spanning a range of devices from non-encrypted ones to fully encrypted ones. Given the limited visibility into payloads in this context, this study investigates AI-based attack detection methods that leverage encrypted traffic metadata, eliminating the need for decryption and minimizing system performance degradation—especially in light of these heterogeneous devices. Using the UNSW-NB15 and CICIoT-2023 dataset, encrypted and unencrypted traffic were categorized according to security protocol, and AI-based intrusion detection experiments were conducted for each traffic type based on metadata. To mitigate the problem of class imbalance, eight different data sampling techniques were applied. The effectiveness of these sampling techniques was then comparatively analyzed using two ensemble models and three Deep Learning (DL) models from various perspectives. The experimental results confirmed that metadata-based attack detection is feasible using only encrypted traffic. In the UNSW-NB15 dataset, the f1-score of encrypted traffic was approximately 0.98, which is 4.3% higher than that of unencrypted traffic (approximately 0.94). In addition, analysis of the encrypted traffic in the CICIoT-2023 dataset using the same method showed a significantly lower f1-score of roughly 0.43, indicating that the quality of the dataset and the preprocessing approach have a substantial impact on detection performance. Furthermore, when data sampling techniques were applied to encrypted traffic, the recall in the UNSW-NB15 (Encrypted) dataset improved by up to 23.0%, and in the CICIoT-2023 (Encrypted) dataset by 20.26%, showing a similar level of improvement. Notably, in CICIoT-2023, f1-score and Receiver Operation Characteristic-Area Under the Curve (ROC-AUC) increased by 59.0% and 55.94%, respectively. These results suggest that data sampling can have a positive effect even in encrypted environments. However, the extent of the improvement may vary depending on data quality, model architecture, and sampling strategy.

The introduction of the Open Radio Access Network (O-RAN) architecture enhances network flexibility but introduces novel security threats targeting open interfaces and the RAN Intelligent Controller (RIC). Particularly in the Near-RT RIC environment, an effective Intrusion Detection System (IDS) that satisfies strict near-real-time constraints of within 1 s is essential to defend against cyber attacks. This paper proposes an Artificial Intelligence (AI)-based IDS xApp designed for real-time cyber attack monitoring in the O-RAN Near-RT RIC environment, and quantitatively analyzes its anomaly detection performance and inference latency characteristics against multi-layer security threats utilizing Open RAN Centralized Unit(O-CU) network layer data and Open RAN Distributed Unit (O-DU) radio telemetry data. Evaluation using a public dataset (NetsLab 5G O-RAN IDD) on four deep learning models (LSTM, CNN, Transformer, Autoencoder) showed that supervised learning-based models achieved high F1-scores (reaching up to 0.99) on both datasets. Furthermore, their performance variation remained highly stable at approximately the $\pm$0.1 pp level upon transition from the training environment (the Service and Management Orchestration, SMO) to the deployment environment (Near-RT RIC). In the inference latency analysis, the system’s scalability was evaluated by increasing the number of prediction instances up to 80,000. The results confirmed that the latency follows a highly predictable linear time complexity ( $𝒪(N)$). Specifically, the LSTM, CNN, and Autoencoder models successfully maintained a response time within 1000 ms even under the maximum load of 80,000 instances across both datasets, whereas the computationally heavy Transformer model experienced resource exhaustion in the KServe inference pod at approximately 20,000 instances, causing the inference process to terminate and rendering further measurement infeasible. Comprehensively, the LSTM model demonstrated the most outstanding balance between performance and operational efficiency by recording stable detection performance, short tail latency (approximately 140 ms at P99), and low training resource consumption. This study experimentally demonstrates the anomaly detection performance of the IDS xApp in the O-RAN near-real-time control environment, and comprehensively verifies its practical effectiveness by considering both inference latency and resource consumption.

The advent of 5th Generation (5G) mobile networks has introduced Network Slicing as a core mechanism for supporting heterogeneous vertical services—such as enhanced Mobile Broadband (eMBB), Ultra-Reliable Low-Latency Communication (URLLC), and massive Machine-Type Communication (mMTC) over a shared physical infrastructure, thereby significantly expanding the attack surface at the User Plane Function (UPF). Securing this multi-slice environment requires intrusion detection systems that can simultaneously accommodate the statistical heterogeneity of per-slice traffic and the stringent Quality of Service (QoS) constraints of real-time slices, yet the practical cost of obtaining high-quality labeled traffic in operational 5G cores remains prohibitive. This study proposes 5G-SliceMatch, a Slice-Aware Semi-Supervised Learning framework, to address the challenge of malicious traffic detection in 5G network slicing environments under severe label scarcity. Traditional intrusion detection systems fail to account for the heterogeneous traffic characteristics of different 5G slices, leading to significant performance degradation in resource-constrained slices like URLLC. To overcome this, 5G-SliceMatch integrates a Slice-Aware Teacher Model with dedicated classification heads, slice-specific differential Feature Masking, and a progressive Self-Training strategy. Experimental results on the 5G-SliciNdd dataset demonstrate that 5G-SliceMatch consistently outperforms state-of-the-art baselines. Specifically, in an extreme scenario with only 1% of labeled data, 5G-SliceMatch achieved an F1-Score of 0.9393, outperforming XGBoost by 2.82%. Moreover, it achieves 98.1% of the performance of a fully supervised XGBoost (100% labels) using only 5% of the data, effectively reducing the manual labeling workload by 95%. This work proves that 5G-SliceMatch—by integrating slice-aware architectural design with efficient semi-supervised learning is critical for achieving high-resolution security visibility while satisfying the stringent QoS requirements of complex 5G network environments.

This study proposes an efficient traffic classification model to address the growing threat of distributed denial-of-service (DDoS) attacks in 5th generation technology standard (5G) slicing networks. The proposed method utilizes an ensemble of encoder components from multiple autoencoders to compress and extract latent representations from high-dimensional traffic data. These representations are then used as input for a support vector machine (SVM)-based metadata classifier, enabling precise detection of attack traffic. This architecture is designed to achieve both high detection accuracy and training efficiency, while adapting flexibly to the diverse service requirements and complexity of 5G network slicing. The model was evaluated using the DDoS Datasets 2022, collected in a simulated 5G slicing environment. Experiments were conducted under both class-balanced and class-imbalanced conditions. In the balanced setting, the model achieved an accuracy of 89.33%, an F1-score of 88.23%, and an Area Under the Curve (AUC) of 89.45%. In the imbalanced setting (attack:normal = 7:3), the model maintained strong robustness, achieving a recall of 100% and an F1-score of 90.91%, demonstrating its effectiveness in diverse real-world scenarios. Compared to existing AI-based detection methods, the proposed model showed higher precision, better handling of class imbalance, and strong generalization performance. Moreover, its modular structure is well-suited for deployment in containerized network function (NF) environments, making it a practical solution for real-world 5G infrastructure. These results highlight the potential of the proposed approach to enhance both the security and operational resilience of 5G slicing networks.

In the IoT (Internet of Things) domain, the increased use of encryption protocols such as SSL/TLS, VPN (Virtual Private Network), and Tor has led to a rise in attacks leveraging encrypted traffic. While research on anomaly detection using AI (Artificial Intelligence) is actively progressing, the encrypted nature of the data poses challenges for labeling, resulting in data imbalance and biased feature extraction toward specific nodes. This study proposes a reconstruction error-based anomaly detection method using an autoencoder (AE) that utilizes packet metadata excluding specific node information. The proposed method omits biased packet metadata such as IP and Port and trains the detection model using only normal data, leveraging a small amount of packet metadata. This makes it well-suited for direct application in IoT environments due to its low resource consumption. In experiments comparing feature extraction methods for AE-based anomaly detection, we found that using flow-based features significantly improves accuracy, precision, F1 score, and AUC (Area Under the Receiver Operating Characteristic Curve) score compared to packet-based features. Additionally, for flow-based features, the proposed method showed a 30.17% increase in F1 score and improved false positive rates compared to Isolation Forest and OneClassSVM. Furthermore, the proposed method demonstrated a 32.43% higher AUC when using packet features and a 111.39% higher AUC when using flow features, compared to previously proposed oversampling methods. This study highlights the impact of feature extraction methods on attack detection in imbalanced, encrypted traffic environments and emphasizes that the one-class method using AE is more effective for attack detection and reducing false positives compared to traditional oversampling methods.