Mesh denoising algorithms explore recovering smooth surfaces from 3D meshes that are corrupted with noise. They aim to remove noise while maintaining geometric features. Although existing mesh denoising methods have shown promising results, they typically involve many parameters and settings that can be complex to non-experts. Also, parameter tuning is usually manual and realized through trial and error. To address this, we propose a new fairer benchmark for mesh denoising algorithms. This benchmark consolidates existing algorithms into a single platform, organizes various noisy mesh data, and provides optimal parameter settings for different algorithms. Specifically, we propose a more effective and scientific method for setting parameters by utilizing Bayesian optimization to find optimal parameters. We use the upper confidence bound strategy to balance exploration and exploitation needs. We conduct extensive experiments to compare existing mesh denoising algorithms quantitatively and qualitatively. These experiments show that there does not exist a single method that is optimal for all shapes and noise types. Additionally, we provide all the optimal parameter settings of the state-of-the-art methods that are available to the community.