Generative artificial intelligence (GAI) has garnered considerable attention from the scientific community and driven a paradigm shift in scientific research, namely AI for science (AI4S). To evaluate the potential of GAI in biomedical research, this study explores the capability of using large language models (LLMs) to discover biomedical knowledge by conducting a case study to extract disease causalities from large-scale biomedical literature using ChatGPT. In particular, two groups of the most frequent causal knowledge in the biomedical domain are discussed in this study, especially the causalities among different diseases and chemically induced causations. Quantitatively, ChatGPT’s responses are compared with biocurators’ annotations in identifying causalities from a random sampling of PubMed citations. Guided by six predefined prompts of three mainstream paradigms, namely zero-shot, few-shot, and long chain-of-thought (CoT), ChatGPT’s results are very close to human annotations in terms of a high degree of precision (94.73% for the disease-disease group and 92.47% for the chemical-disease group individually). These findings suggest that ChatGPT has the capacity to discover biomedical knowledge efficiently, as well as the capability to facilitate biomedical research. In the near future, more state-of-the-art LLMs, more real-world data, such as electronic medical records, and more biomedical tasks will be pursued for a series of holistic studies. Overall, this work supports the vision and development of AI4S, especially the knowledge discovery of biomedical research.