TranSegWGAN: Segmentation of Histology Images Through Stain Style Transfer Based on A Wasserstein Generative Adversarial Network

The segmentation of tumor epithelial tissue from hematoxylin and eosin (HE)-stained pathology images is crucial for the clinical diagnosis of oropharyngeal cancer (OPC). However, such segmentation is complicated by the heterogeneity of pathological features associated with OPC, with tumor epithelial tissue and other tissues, such as tumor stroma, exhibiting similar characteristics in HE-stained images. Immunohistochemistry (IHC)-stained images can distinguish epithelial tissue from other tissues but are costly to acquire. Therefore, a suitable technique for converting HE-stained images into IHC-stained images is urgently required to better perform segmentation. Consequently, we propose a two-stage framework, termed stain transfer and segmentation Wasserstein generative adversarial network (TranSegWGAN). First, an enhanced stain style transfer network based on Wasserstein generative adversarial Network with Gradient Penalty (WGAN-GP) is proposed, which improves transformation quality and feature representation through encoder replacement, multiscale fusion and deep supervision strategy, and a hybrid loss function integrating WGAN-GP and the regularized relativistic generative adversarial network (R3GAN). Second, the generated pseudo-IHC-stained images are superimposed onto original HE-stained images and fed into a multilayer perceptron model optimized by particle swarm optimization to obtain OPC epithelial tissue mask. Experimental results indicate that TranSegWGAN not only generates more stable and higher-fidelity images than other generative adversarial network (GAN)-based models but also achieves an accuracy of 90.85%, surpassing current mainstream segmentation models.