The relentless pursuit of next-generation photonic systems for advanced information processing and terahertz-scale wireless communications necessitates unprecedented precision in arbitrary electromagnetic wave manipulation. By engineering the local phase of individual meta-atoms, metasurfaces enable the precise control of multidimensional optical parameters. Nevertheless, conventional meta-atom design paradigms, heavily reliant on iterative trial-and-error processes, are computationally intensive and resource-demanding. Herein, we propose a task-specific, multifunctional lightweight deep neural network that integrates forward and inverse design modules. This framework enables efficient and accurate exploration within the function space and parameter space. It successfully reconciles the inherent trade-off between computational efficiency and prediction accuracy, thereby enabling a combinatorial inverse design paradigm. This approach, rooted in functional modularity, significantly enhances the efficiency and flexibility of developing highly integrated, multifunctional optical devices. To address the inherent data dependency, we implement a data distillation strategy that reduces the required dataset to just 20% of its original volume while simultaneously enhancing design precision by 12.06%. As a proof of concept, we demonstrate the entire design and validation process of the polarization-multiplexed metalens and vortex-generation metasurfaces. This versatile approach could be readily extended to develop other types of wavefront modulation devices, heralding a new era of customizable and multifunctional meta-devices.