Accurate building energy simulation (BES) is essential for developing effective energy conservation strategies and implementing evidence-based policy interventions in the built environment. However, BES accuracy is often undermined by unrealistic weather data, as conventional Typical Meteorological Year (TMY) files fail to adequately capture urban microclimate variations. This study proposes a deep learning model that integrates wind-driven building morphology maps for high-resolution temporal microclimate prediction. By combining macro-scale wind dynamics with urban morphological features, encoded as frontal area maps, the model captures seasonal microclimate variations influenced by prevailing wind conditions. Validation conducted on a university campus demonstrates that the proposed model outperforms benchmark approaches in predicting air temperature and relative humidity (RH). The ground truth for validation is the real-time microclimate data collected by weather stations installed across the campus. Compared to TMY files, a standard deep learning model, and a deep learning model with wind directions, the proposed model reduces the root mean squared error (RMSE) in air temperature by 36.3%, 14.2%, and 14.0%, and RMSE in RH by 30.5%, 17.3%, and 17.3%, respectively. When integrated into BES for three test buildings, the model’s weather data enabled cooling energy prediction with less than 2% error, significantly outperforming alternative methods. Overall, the proposed model allows cross-building temporal microclimate prediction without requiring long-term weather data collection at the target building.