Component Evaluation and Preference Optimization for Multimodal Product Summary Generation

The task of multimodal product summary generation aims to produce concise and accurate summary texts that highlight the core selling points of products by integrating both textual and visual information. However, existing methods still face two major challenges. First, traditional evaluation metrics such as ROUGE, which are based on lexical overlap, fail to precisely measure the completeness of a summary in expressing key product information. Second, mainstream supervised fine-tuning paradigms struggle to capture users’ implicit preferences regarding the relative importance of different product information components, resulting in a misalignment between generated summaries and actual usage requirements. To address these issues, this paper proposes a component-based summarization evaluation metric, termed CSE (Component-based Summary Evaluation), which comprehensively evaluates the expression of key information from two perspectives: Component Hit Accuracy (CHA) and Component Quantity Ratio (CQR). Furthermore, a preference optimization-based multimodal product summary generation model, PAMPS, is designed. The model aligns its outputs with users’ preferences for product component expression through four training stages: supervised fine-tuning, summary resampling, component evaluation-guided preference pair construction, and direct preference optimization. Experimental results on the large-scale Chinese e-commerce dataset CEPSUM show that PAMPS achieves significant improvements on ROUGE metrics. In particular, compared with Qwen2-VL-SFT, Qwen2-VL-DPO-ROUGE improves ROUGE-1, ROUGE-2, and ROUGE-L by 0.25, 0.44, and 1.21 percentage points on average, respectively, demonstrating superior overall summary generation quality. Under the proposed CSE evaluation framework, the Qwen2-VL-DPO-CSE model demonstrates particularly prominent improvements in the CHA metric, with an average relative improvement of about 4%, indicating that component-oriented preference optimization can effectively enhance the model’s ability to capture and express core product features. The comprehensive experimental results fully validate the effectiveness and practical applicability of the proposed method in improving the quality of multimodal product summary generation.