Cross-subject emotion recognition based on electroencephalography (EEG) remains challenging due to non-stationary distribution shifts among individuals. Existing domain adaptation methods typically follow fixed paradigms, either indiscriminately merging sources, which risks negative transfer, or isolating them as independent branches, which leads to high computational redundancy. To address this dilemma, we propose a novel flexible source aggregation net-work (FSA-Net) for coarse-to-fine domain adaptation. Unlike static methods, our approach first introduces a similarity-based aggregation mechanism that dynamically reconfigures source domains based on their affinity to the target, effectively balancing model complexity with transfer performance. Subsequently, we construct a hierarchical alignment strategy that minimizes global discrepancy through coarse alignment using maximum mean discrepancy (MMD), as well as fine-grained subdomain adaptation incorporating contrastive learning with local MMD. This ensures that samples with identical emotion labels are clustered while distinct categories are separated in the latent space. Furthermore, a domain-aware weighted ensemble strategy is designed to integrate predictions based on the adaptation quality of each aggregated branch. Extensive experiments on the benchmark SEED and SEED-IV datasets demonstrate that FSA-Net achieves superior state-of-the-art accuracy of 90.80% and 82.74%, respectively, validating its efficacy in robust cross-subject emotion decoding.