The Internet of Things (IoT) is integral to modern infrastructure, enabling connectivity among a wide range of devices from home automation to industrial control systems. With the exponential increase in data generated by these interconnected devices, robust anomaly detection mechanisms are essential. Anomaly detection in this dynamic environment necessitates methods that can accurately distinguish between normal and anomalous behavior by learning intricate patterns. This paper presents a novel approach utilizing generative adversarial networks (GANs) for anomaly detection in IoT systems. However, optimizing GANs involves tuning hyper-parameters such as learning rate, batch size, and optimization algorithms, which can be challenging due to the non-convex nature of GAN loss functions. To address this, we propose a five-dimensional Gray wolf optimizer (5DGWO) to optimize GAN hyper-parameters. The 5DGWO introduces two new types of wolves: gamma ( $\gamma$) for improved exploitation and convergence, and theta ( $\theta$) for enhanced exploration and escaping local minima. The proposed system framework comprises four key stages: 1) preprocessing, 2) generative model training, 3) autoencoder (AE) training, and 4) predictive model training. The generative models are utilized to assist the AE training, and the final predictive models (including convolutional neural network (CNN), deep belief network (DBN), recurrent neural network (RNN), random forest (RF), and extreme gradient boosting (XGBoost)) are trained using the generated data and AE-encoded features. We evaluated the system on three benchmark datasets: NSL-KDD, UNSW-NB15, and IoT-23. Experiments conducted on diverse IoT datasets show that our method outperforms existing anomaly detection strategies and significantly reduces false positives. The 5DGWO-GAN-CNN_AE exhibits superior performance in various metrics, including accuracy, recall, precision, root mean square error (RMSE), and convergence trend. The proposed 5DGWO-GAN-CNN_AE achieved the lowest RMSE values across the NSL-KDD, UNSW-NB15, and IoT-23 datasets, with values of 0.24, 1.10, and 0.09, respectively. Additionally, it attained the highest accuracy, ranging from 94% to 100%. These results suggest a promising direction for future IoT security frameworks, offering a scalable and efficient solution to safeguard against evolving cyber threats.