Explore Advanced Hybrid Deep Learning for Enhanced Wireless Signal Detection in 5G OFDM Systems

Single-signal detection in orthogonal frequency-division multiplexing (OFDM) systems presents a challenge due to the time-varying nature of wireless channels. Although conventional methods have limitations, particularly in multi-input multioutput orthogonal frequency division multiplexing (MIMO-OFDM) systems, this paper addresses this problem by exploring advanced deep learning approaches for combined channel estimation and signal detection. Specifically, we propose two hybrid architectures that integrate a convolutional neural network (CNN) with a recurrent neural network (RNN), namely, CNN-long short-term memory (CNN-LSTM) and CNN-bidirectional-LSTM (CNN-Bi-LSTM), designed to enhance signal detection performance in MIMO-OFDM systems. The proposed CNN-LSTM and CNN-Bi-LSTM architectures are evaluated and compared with both traditional methods and standalone deep learning models. Training was conducted offline using a dataset generated from a 2 × 2 MIMO-OFDM system with a 3GPP 5G channel model. The trained models are evaluated using accuracy, loss, and computational time, and further analysis of signal detection performance is based on bit error rate, optimal cyclic prefix length, and optimal pilot subcarrier configurations under various noise conditions and channel uncertainty scenarios. The results demonstrate that the proposed CNN-based architectures, particularly the CNN-Bi-LSTM trained model, significantly reduce the need for pilot and cyclic prefix symbols while delivering superior performance, especially at SNRs. All the hybrid deep learning architectures (CNN-LSTM, CNN-Bi-LSTM) demonstrated greater robustness and adaptability under dynamic channel conditions, outperforming conventional methods and benchmark deep learning architectures. These results indicate the effectiveness of CNN-based feature extractors in learning generalized spatial patterns, positioning these hybrid models as highly efficient and reliable solutions for MIMO-OFDM signal detection in 5G and future wireless communication systems.