Large visual language models (LVLMs) have revolutionized the multimodal domain, demonstrating exceptional performance in tasks requiring fusing visual and textual information. However, the current evaluation benchmarks fail to adequately assess the knowledge alignment between images and text, focusing primarily on answer accuracy rather than the reasoning processes behind them. To address this gap and enhance the understanding of LVLMs’ capabilities, we introduce KnowBench, a novel benchmark designed to assess the alignment of knowledge between images and text for LVLMs. KnowBench comprises 1 081 image-question pairs, each with four options and four pieces of corresponding knowledge across 11 major categories. We evaluate mainstream LVLMs on KnowBench, including proprietary models like Gemini, Claude, and GPT, and open-source models like LLaVA, Qwen-VL, and InternVL. Our experiments reveal a notable discrepancy in the models’ abilities to select correct answers and corresponding knowledge whether the models are open-source or proprietary. This indicates that there is still a significant gap in the current LVLMs’ knowledge alignment between images and text. Furthermore, our further analysis shows that model performance on KnowBench improves with increased parameters and version iterations. This indicates that scaling laws have a significant impact on multimodal knowledge alignment, and the iteration of the model by researchers also has a positive effect. We anticipate that KnowBench will foster the development of LVLMs and motivate researchers to develop more reliable models. We have made our dataset publicly available at https://doi.org/10.57760/sciencedb.29672.

Relation extraction is a pivotal task within the field of natural language processing, boasting numerous real-world applications. Existing research predominantly centers on monolingual relation extraction or cross-lingual enhancement for relation extraction. However, there exists a notable gap in understanding relation extraction within mix-lingual (or code-switching) scenarios. In these scenarios, individuals blend content from different languages within sentences, generating mix-lingual content. The effectiveness of existing relation extraction models in such scenarios remains largely unexplored due to the absence of dedicated datasets. To address this gap, we introduce the Mix-Lingual Relation Extraction (MixRE) task and construct a human-annotated dataset MixRED to support this task. Additionally, we propose a hierarchical training approach for the mix-lingual scenario named Mix-Lingual Training (MixTrain), designed to enhance the performance of large language models (LLMs) when capturing relational dependencies from mix-lingual content spanning different semantic levels. Our experiments involve evaluating state-of-the-art supervised models and LLMs on the constructed dataset, with results indicating that MixTrain notably improves model performance. Moreover, we investigate the effectiveness of using mix-lingual content as a tool to transfer learned relational dependencies across different languages. Additionally, we delve into factors influencing model performance for both supervised models and LLMs in the novel MixRE task.

Graph neural networks (GNNs) have shown great power in learning on graphs. However, it is still a challenge for GNNs to model information faraway from the source node. The ability to preserve global information can enhance graph representation and hence improve classification precision. In the paper, we propose a new learning framework named G-GNN (Global information for GNN) to address the challenge. First, the global structure and global attribute features of each node are obtained via unsupervised pre-training, and those global features preserve the global information associated with the node. Then, using the pre-trained global features and the raw attributes of the graph, a set of parallel kernel GNNs is used to learn different aspects from these heterogeneous features. Any general GNN can be used as a kernal and easily obtain the ability of preserving global information, without having to alter their own algorithms. Extensive experiments have shown that state-of-the-art models, e.g., GCN, GAT, Graphsage and APPNP, can achieve improvement with G-GNN on three standard evaluation datasets. Specially, we establish new benchmark precision records on Cora (84.31%) and Pubmed (80.95%) when learning on attributed graphs.