Ablative thermal protection is an important thermal protection method for hypersonic vehicles. Porous pyrolyzed carbon for thermal protection materials are a type of ablative thermal protective material, whose permeability significantly influences transport characteristics. To address the issue of difficulty in obtaining empirical coefficients in permeability formulas for thermal protective porous materials, a multi-source heterogeneous dataset is constructed, incorporating material microstructure images and macroscopic structural characteristic parameters (maximum flow and fractal dimension). A particle-based method named direct simulation Monte Carlo is employed to calculate the permeability of the microstructure. Based on this, three permeability prediction methods using multi-source heterogeneous data fusion strategies are proposed: a decision-level fusion strategy, a fusion strategy based on direct concatenation of multi-source data features, and a fusion strategy based on a cross-modal attention mechanism for multi-source data. By comparing the predictive performance of the three fusion strategies, the strategy based on the cross-modal attention mechanism demonstrates the best performance. This strategy captures the relationship between image convolutional features and structural parameters and dynamically adjusts the weights between them. On the test set, the coefficient of determination is 0.949 7, and the Mean Absolute Percentage Error is 5.29%. Compared with single-source data-driven permeability prediction models, the coefficient of determination improves by 6%, and the Mean Absolute Percentage Error decreases by 41%. This method enables efficient and accurate prediction of permeability, providing technical support for the refined design of thermal protection structures in actual hypersonic vehicles.