Recent advancements in financial large language models (FinLLMs) have shown strong performance in high-resource languages like English and Chinese, but are limited in low-resource settings, particularly in Southeast Asia (SEA), where labeled data resources are extremely scarce and challenging to annotate. Existing benchmarks primarily focus on high-resource finance, neglecting low-resource finance. To address these issues, we introduce CroFinBen, to the best of our knowledge, the first multilingual benchmark specifically designed to bridge the language-resource gap between high- and low-resource finance. It includes four key financial NLP tasks—financial sentiment analysis (FinSA), financial stock prediction (FinSP), financial text summarization (FinTS), and financial text classification (FinTC)—across both high-resource languages (English and Chinese) and low-resource Southeast Asian (SEA) languages (Indonesian, Malay, Thai, Filipino, and Vietnamese), comprising over 50 000 samples from 16 datasets, providing a comprehensive and balanced evaluation of LLMs. Unlike others that rely on full translations or overlook local context, CroFinBen incorporates localized annotations to reflect financial terms and cultural nuances in SEA languages. Evaluating 25 LLMs shows significant performance differences, with no clear proficiency in either high- or low-resource languages, especially for existing language-biased fine-tuned FinLLMs. The 1800B large-parameter closed-source GPT-4o excels, while DeepSeek-V3 and ChatGPT-3.5 also perform well. By bridging language-resource barriers, CroFinBen enhances the fairness and robustness of FinLLMs, providing strong support for improving performance in global financial scenarios. Our data resources are available at https://jcst.ict.ac.cn/en/supplement/afc97c89-905b-4397-941c-60dd0c720248.
- Article type
- Year
- Co-author
Although the popular database systems perform well on query optimization, they still face poor query execution plans when the join operations across multiple tables are complex. Bad execution planning usually results in bad cardinality estimations. The cardinality estimation models in traditional databases cannot provide high-quality estimation, because they are not capable of capturing the correlation between multiple tables in an effective fashion. Recently, the state-of-the-art learning-based cardinality estimation is estimated to work better than the traditional empirical methods. Basically, they used deep neural networks to compute the relationships and correlations of tables. In this paper, we propose a vertical scanning convolutional neural network (abbreviated as VSCNN) to capture the relationships between words in the word vector in order to generate a feature map. The proposed learning-based cardinality estimator converts Structured Query Language (SQL) queries from a sentence to a word vector and we encode table names in the one-hot encoding method and the samples into bitmaps, separately, and then merge them to obtain enough semantic information from data samples. In particular, the feature map obtained by VSCNN contains semantic information including tables, joins, and predicates about SQL queries. Importantly, in order to improve the accuracy of cardinality estimation, we propose the negative sampling method for training the word vector by gradient descent from the base table and compress it into a bitmap. Extensive experiments are conducted and the results show that the estimation quality of q-error of the proposed vertical scanning convolutional neural network based model is reduced by at least 14.6% when compared with the estimators in traditional databases.
京公网安备11010802044758号