Multi-source and multi-modal data in the medical area include structured data, texts, images, and continuous vital sign monitoring data generated by multiple devices of the Internet of Things, known as the Internet of medical things (IoMT) data. The IoMT system integrates a multitude of sensors, medical devices, and intelligent equipment in hospitals, leveraging perceptual and communication technologies, is popularized increasingly. The devices communicate through diverse protocols, and the absence of standardized IoMT interfaces presents a realistic dilemma in integrating IoMT data for holistic clinical analysis. Additionally, the scarcity of computing resources poses a constraint for the extensive training of models and the execution of complex reasoning processes, particularly in high-stakes settings such as intensive care unit (ICU). To address these challenges, we introduce a novel framework designed to facilitate semantic interoperability across multiple devices and to transform multi-source and multi-modal data into a unified data structure. Furthermore, we propose an innovative cloud-edge collaborative system, which could conduct intelligent computing in resource-constrained environments. Our approach was rigorously tested across various metrics, including system response time, data transmission latency, and overall system accuracy. The outcomes demonstrate clear advantages, and offer promising prospects for the future of medical data integration and analysis.