A Review of Causal Reasoning in Foundation Models

This comprehensive review examines the integration of causal reasoning in foundation models, focusing on three types of causal reasoning: commonsense causal reasoning, quantitative causal reasoning, and formal causal reasoning. Current foundation models exhibit different capabilities in processing or reasoning these three types of causal relationships with various methodologies. We propose a structured way that categorize existing methods into three tiers: data-level, where input data, such as prompts or knowledge, are crafted for causal inferences; component-level, where model internals, like structures or modules, are adjusted to yield causal insights; and model-level, where some existing foundation models have been discussed about their potential in causal inference, while several new models are designed inherently with causal reasoning capabilities. The reviews highlight the significance of these strategies in advancing AI systems towards reliable and ethical causal inference, essential for real-world applications.