Due to the complex products and irradiation-induced defects, it is hard to understand and even predict the thermal conductivity variation of materials under fast neutron irradiation, such as the abrupt degradation of thermal conductivity of boron carbide (B₄C) at the very beginning of the irradiation process. In this work, the contributions of various irradiation-induced defects in B₄C primarily consisting of the substitutional defects, Frenkel defect pairs, and helium bubbles were re-evaluated separately and quantitatively in terms of the phonon scattering theory. A theoretical model with an overall consideration of the contributions of all these irradiation-induced defects was proposed without any adjustable parameters, and validated to predict the thermal conductivity variation under irradiation based on the experimental data of the unirradiated, irradiated, and annealed B₄C samples. The predicted thermal conductivities by this model show a good agreement with the experimental data after irradiation. The calculation results and theoretical analysis in light of the experimental data demonstrate that the substitutional defects of boron atoms by lithium atoms, and the Frenkel defect pairs due to the collisions with the fast neutrons, rather than the helium bubbles with strain fields surrounding them, play determining roles in the abrupt degradation of thermal conductivity with burnup.