Wind turbine gears, operating at high speeds under complex environmental conditions, exhibit subtle early-stage fault signals that are easily masked, resulting in low accuracy with conventional diagnostic methods. To address this issue, this paper proposes a fault diagnosis method for wind turbine gears based on an improved sailfish optimizer (ISFO) algorithm optimising variational modal decomposition (VMD) and convolutional neural networks (CNN). Firstly, the ISFO algorithm is enhanced by incorporating initialisation via the Logistic chaotic map, optimisation principles from Lévy flight theory, and genetic algorithm techniques. This yields an ISFO algorithm based on hybrid strategies, effectively resolving the algorithm’s local optimum issue. Subsequently, using ISFO algorithm refines VMD parameter decomposition of signals, extracting fault feature information from the modal component with the highest correlation coefficient. A short-time Fourier transform (STFT) is then employed to construct a time-frequency map. Finally, the time-frequency map is input into an optimised CNN for fault diagnosis classification. Experimental comparisons and analyses demonstrate that the proposed method achieves high diagnostic accuracy on both public and self-test datasets, with an average accuracy rate of 98.67%, effectively addressing wind turbine gear fault diagnosis challenges.