Large visual language models (LVLMs) have revolutionized the multimodal domain, demonstrating exceptional performance in tasks requiring fusing visual and textual information. However, the current evaluation benchmarks fail to adequately assess the knowledge alignment between images and text, focusing primarily on answer accuracy rather than the reasoning processes behind them. To address this gap and enhance the understanding of LVLMs’ capabilities, we introduce KnowBench, a novel benchmark designed to assess the alignment of knowledge between images and text for LVLMs. KnowBench comprises 1 081 image-question pairs, each with four options and four pieces of corresponding knowledge across 11 major categories. We evaluate mainstream LVLMs on KnowBench, including proprietary models like Gemini, Claude, and GPT, and open-source models like LLaVA, Qwen-VL, and InternVL. Our experiments reveal a notable discrepancy in the models’ abilities to select correct answers and corresponding knowledge whether the models are open-source or proprietary. This indicates that there is still a significant gap in the current LVLMs’ knowledge alignment between images and text. Furthermore, our further analysis shows that model performance on KnowBench improves with increased parameters and version iterations. This indicates that scaling laws have a significant impact on multimodal knowledge alignment, and the iteration of the model by researchers also has a positive effect. We anticipate that KnowBench will foster the development of LVLMs and motivate researchers to develop more reliable models. We have made our dataset publicly available at https://doi.org/10.57760/sciencedb.29672.

Relation extraction is a pivotal task within the field of natural language processing, boasting numerous real-world applications. Existing research predominantly centers on monolingual relation extraction or cross-lingual enhancement for relation extraction. However, there exists a notable gap in understanding relation extraction within mix-lingual (or code-switching) scenarios. In these scenarios, individuals blend content from different languages within sentences, generating mix-lingual content. The effectiveness of existing relation extraction models in such scenarios remains largely unexplored due to the absence of dedicated datasets. To address this gap, we introduce the Mix-Lingual Relation Extraction (MixRE) task and construct a human-annotated dataset MixRED to support this task. Additionally, we propose a hierarchical training approach for the mix-lingual scenario named Mix-Lingual Training (MixTrain), designed to enhance the performance of large language models (LLMs) when capturing relational dependencies from mix-lingual content spanning different semantic levels. Our experiments involve evaluating state-of-the-art supervised models and LLMs on the constructed dataset, with results indicating that MixTrain notably improves model performance. Moreover, we investigate the effectiveness of using mix-lingual content as a tool to transfer learned relational dependencies across different languages. Additionally, we delve into factors influencing model performance for both supervised models and LLMs in the novel MixRE task.