CroFinBen: A Multilingual Benchmark of Large Language Models Across High- and Low-Resource Languages in Finance

Recent advancements in financial large language models (FinLLMs) have shown strong performance in high-resource languages like English and Chinese, but are limited in low-resource settings, particularly in Southeast Asia (SEA), where labeled data resources are extremely scarce and challenging to annotate. Existing benchmarks primarily focus on high-resource finance, neglecting low-resource finance. To address these issues, we introduce CroFinBen, to the best of our knowledge, the first multilingual benchmark specifically designed to bridge the language-resource gap between high- and low-resource finance. It includes four key financial NLP tasks—financial sentiment analysis (FinSA), financial stock prediction (FinSP), financial text summarization (FinTS), and financial text classification (FinTC)—across both high-resource languages (English and Chinese) and low-resource Southeast Asian (SEA) languages (Indonesian, Malay, Thai, Filipino, and Vietnamese), comprising over 50 000 samples from 16 datasets, providing a comprehensive and balanced evaluation of LLMs. Unlike others that rely on full translations or overlook local context, CroFinBen incorporates localized annotations to reflect financial terms and cultural nuances in SEA languages. Evaluating 25 LLMs shows significant performance differences, with no clear proficiency in either high- or low-resource languages, especially for existing language-biased fine-tuned FinLLMs. The 1800B large-parameter closed-source GPT-4o excels, while DeepSeek-V3 and ChatGPT-3.5 also perform well. By bridging language-resource barriers, CroFinBen enhances the fairness and robustness of FinLLMs, providing strong support for improving performance in global financial scenarios. Our data resources are available at https://jcst.ict.ac.cn/en/supplement/afc97c89-905b-4397-941c-60dd0c720248.