In image analysis, high-precision semantic segmentation predominantly relies on supervised learning. Despite significant advancements driven by deep learning techniques, challenges such as class imbalance and dynamic performance evaluation persist. Traditional weighting methods, often based on pre-statistical class counting, tend to overemphasize certain classes while neglecting others, particularly rare sample categories. Approaches like focal loss and other rare-sample segmentation techniques introduce multiple hyperparameters that require manual tuning, leading to increased experimental costs due to their instability. This paper proposes a novel CAWASeg framework to address these limitations. Our approach leverages Grad-CAM technology to generate class activation maps, identifying key feature regions that the model focuses on during decision-making. We introduce a Comprehensive Segmentation Performance Score (CSPS) to dynamically evaluate model performance by converting these activation maps into pseudo mask and comparing them with Ground Truth. Additionally, we design two adaptive weights for each class: a Basic Weight (BW) and a Ratio Weight (RW), which the model adjusts during training based on real-time feedback. Extensive experiments on the COCO-Stuff, CityScapes, and ${\text{ADE}}_{20k}$ datasets demonstrate that our CAWASeg framework significantly improves segmentation performance for rare sample categories while enhancing overall segmentation accuracy. The proposed method offers a robust and efficient solution for addressing class imbalance in semantic segmentation tasks.