AI Chat Paper
Note: Please note that the following content is generated by AMiner AI. SciOpen does not take any responsibility related to this content.
{{lang === 'zh_CN' ? '文章概述' : 'Summary'}}
{{lang === 'en_US' ? '中' : 'Eng'}}
Chat more with AI
Powered by AMiner. Clicking this button will take you away from SciOpen.
Home Big Data Mining and Analytics Article
PDF (7 MB)
Cite
EndNote(RIS) BibTeX
Collect
Submit Manuscript AI Chat Paper
Show Outline
Outline
Show full outline
Hide outline
Outline
Show full outline
Hide outline
Open Access

A Large-Scale Spatio-Temporal Multimodal Fusion Framework for Traffic Prediction

Technical Consulting Department, Shanghai EchoBlend Internet Technology Co. Ltd., Shanghai 201111, China
Department of Electrical and Electronic Engineering, The University of Hong Kong, Hong Kong 999077, China
Department of Industrial and Manufacturing Systems Engineering, The University of Hong Kong, Hong Kong 999077, China
Show Author Information

Abstract

Traffic prediction is crucial for urban planning and transportation management, and deep learning techniques have emerged as effective tools for this task. While previous works have made advancements, they often overlook comprehensive analyses of spatio-temporal distributions and the integration of multimodal representations. Our research addresses these limitations by proposing a large-scale spatio-temporal multimodal fusion framework that enables accurate predictions based on location queries and seamlessly integrates various data sources. Specifically, we utilize Convolutional Neural Networks (CNNs) for spatial information processing and a combination of Recurrent Neural Networks (RNNs) for final spatio-temporal traffic prediction. This framework not only effectively reveals its ability to integrate various modal data in the spatio-temporal hyperspace, but has also been successfully implemented in a real-world large-scale map, showcasing its practical importance in tackling urban traffic challenges. The findings presented in this work contribute to the advancement of traffic prediction methods, offering valuable insights for further research and application in addressing real-world transportation challenges.

Keywords

spatio-temporaltraffic predictionmultimodal fusionlearning representation

References

[1]
China Association of Automobile Manufacturers, Motor vehicle ownership reaches 430 million in China, http://www.caam.org.cn/chn/7/cate_120/con_5236191.html, 2023.
[2]

K. Nellore and G. P. Hancke, A survey on urban traffic management system using wireless sensor networks, Sensors, vol. 16, no. 2, p. 157, 2016.
Crossref Google Scholar
[3]

J. Rios-Torres and A. A. Malikopoulos, Automated and cooperative vehicle merging at highway on-ramps, IEEE Trans. Intell. Transp. Syst., vol. 18, no. 4, pp. 780–789, 2017.
Crossref Google Scholar
[4]

Q. Shi and M. Abdel-Aty, Big data applications in real-time traffic operation and safety monitoring and improvement on urban expressways, Transp. Res. Part C Emerg. Technol., vol. 58, pp. 380–394, 2015.
Crossref Google Scholar
[5]
S. Muthuramalingam, A. Bharathi, S. Rakesh Kumar, N. Gayathri, R. Sathiyaraj, and B. Balamurugan, IoT based intelligent transportation system (IoT-ITS) for global perspective: A case study, in Internet of Things and Big Data Analytics for Smart Generation, V. E. Balas, V. K. Solanki, R. Kumar, and M. Khari, eds. Cham, Germany: Springer, 2019, pp. 279–300.
Crossref
[6]

Z. Lv and W. Shang, Impacts of intelligent transportation systems on energy conservation and emission reduction of transport systems: A comprehensive review, Green Technol. Sustain., vol. 1, no. 1, p. 100002, 2023.
Crossref Google Scholar
[7]
J. Liu, C. Chang, J. Liu, X. Wu, L. Ma, and X. Qi, MarS3D: A plug-and-play motion-aware model for semantic segmentation on multi-scan 3D point clouds, in Proc. 2023 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Vancouver, Canada, 2023, pp. 9372–9381.
Crossref
[8]
Y. Zhao, H. Zheng, Z. Wang, J. Luo, and E. Y. Lam, Point cloud denoising via momentum ascent in gradient fields, in Proc. 2023 IEEE Int. Conf. Image Processing, Kuala Lumpur, Malaysia, 2023, pp. 161–165.
Crossref
[9]
X. Lai, Y. Chen, F. Lu, J. Liu, and J. Jia, Spherical transformer for LiDAR-based 3D recognition, in Proc. 2023 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Vancouver, Canada, 2023, pp. 17545–17555.
Crossref
[10]
J. Liu, Y. Chen, X. Ye, Z Tian, X. Tan, and X. Qi, Spatial pruned sparse convolution for efficient 3D object detection, in Proc. 36 th Conf. Neural Information Processing Systems, New Orleans, LA, USA, 2022, pp. 6735–6748.
Crossref
[11]
J. Yang, S. Shi, Z. Wang, H. Li, and X. Qi, ST3D: Self-training for unsupervised domain adaptation on 3D object detection, in Proc. 2021 IEEE/CVF Conf. Computer Vision and Pattern Recognition, Nashville, TN, USA, 2021, pp. 10363–10373.
Crossref
[12]
A. Faramarzi, M. Heidarinejad, B. Stephens, and S. Mirjalili, Equilibrium optimizer: A novel optimization algorithm, Knowledge-Based Systems, vol. 191, p. 105190, 2020.
Crossref
[13]
M. C. Popescu, V. Balas, L. P. Popescu, and N. Mastorakis, Multilayer perceptron and neural networks, WSEAS Transactions on Circuits and Systems, vol. 8, no.7, pp. 579−588, 2009.
[14]

B. M. Williams and L. A. Hoel, Modeling and forecasting vehicular traffic flow as a seasonal ARIMA process: Theoretical basis and empirical results, J. Transp. Eng., vol. 129, no. 6, pp. 664–672, 2003.
Crossref Google Scholar
[15]

Y. Zhang and Y. C. Liu, Traffic forecasting using least squares support vector machines, Transportmetrica, vol. 5, no. 3, pp. 193–213, 2009.
Crossref Google Scholar
[16]

X. Ma, Z. Tao, Y. Wang, H. Yu, and Y. Wang, Long short-term memory neural network for traffic speed prediction using remote microwave sensor data, Transp. Res. Part C Emerg. Technol., vol. 54, pp. 187–197, 2015.
Crossref Google Scholar
[17]

J. Tang, F. Liu, Y. Zou, W. Zhang, and Y. Wang, An improved fuzzy neural network for traffic speed prediction considering periodic characteristic, IEEE Trans. Intell. Transp. Syst., vol. 18, no. 9, pp. 2340–2350, 2017.
Crossref Google Scholar
[18]

D. W. Xu, Y. D. Wang, L. M. Jia, Y. Qin, and H. H. Dong, Real-time road traffic state prediction based on ARIMA and Kalman filter, Front. Inf. Technol. Electron. Eng., vol. 18, no. 2, pp. 287–302, 2017.
Crossref Google Scholar
[19]

H. Yang, L. Du, G. Zhang, and T. Ma, A traffic flow dependency and dynamics based deep learning aided approach for network-wide traffic speed propagation prediction, Transp. Res. Part B Methodol., vol. 167, pp. 99–117, 2023.
Crossref Google Scholar
[20]

M. Levin and Y. D. Tsao, On forecasting freeway occupancies and volumes (abridgment), Transp. Res. Record, vol. 773, pp. 47–49, 1980.
Google Scholar
[21]

T. Bogaerts, A. D. Masegosa, J. S. Angarita-Zapata, E. Onieva, and P. Hellinckx, A graph CNN-LSTM neural network for short and long-term traffic forecasting based on trajectory data, Transp. Res. Part C Emerg. Technol., vol. 112, pp. 62–77, 2020.
Crossref Google Scholar
[22]

M. S. Ahmed and A. R. Cook, Analysis of freeway traffic time-series data by using Box-Jenkins techniques, Transp. Res. Record, vol. 722, pp. 1–9, 1979.
Google Scholar
[23]

D. A. Tedjopurnomo, Z. Bao, B. Zheng, F. M. Choudhury, and A. K. Qin, A survey on modern deep neural network for traffic prediction: Trends, methods and challenges, IEEE Trans. Knowl. Data Eng., vol. 34, no. 4, pp. 1544–1561, 2022.
Google Scholar
[24]
Y. Zhao, G. Li, and E. Y. Lam, Cross-camera human motion transfer by time series analysis, in Proc. 2024 IEEE Int. Conf. Acoustics, Speech and Signal Processing, Seoul, Republic of Korea, 2024, pp. 4985–4989.
Crossref
[25]
C. De Fabritiis, R. Ragona, and G. Valenti, Traffic estimation and prediction based on real time floating car data, in Proc. 2008 11 th Int. IEEE Conf. Intelligent Transportation Systems, Beijing, China, 2008, pp. 197–203.
Crossref
[26]

Z. Liu, Z. Li, K. Wu, and M. Li, Urban traffic prediction from mobility data using deep learning, IEEE Network, vol. 32, no. 4, pp. 40–46, 2018.
Crossref Google Scholar
[27]
P. P. Dubey and P. Borkar, Review on techniques for traffic jam detection and congestion avoidance, in Proc. 2015 2 nd Int. Conf. Electronics and Communication Systems, Coimbatore, India, 2015, pp. 434–440.
Crossref
[28]

Y. Yan, Y. Deng, S. Cui, Y. H. Kuo, A. H. F. Chow, and C. Ying, A policy gradient approach to solving dynamic assignment problem for on-site service delivery, Trans. Res. Part E Logist. Transp. Rev., vol. 178, p. 103260, 2023.
Crossref Google Scholar
[29]

Y. Yan, A. H. F. Chow, C. P. Ho, Y. H. Kuo, Q. Wu, and C. Ying, Reinforcement learning for logistics and supply chain management: Methodologies, state of the art, and future opportunities, Transp. Res. Part E Logist. Transp. Rev., vol. 162, p. 102712, 2022.
Crossref Google Scholar
[30]

Z. Ning, H. Chen, E. C. H. Ngai, X. Wang, L. Guo, and J. Liu, Lightweight imitation learning for real-time cooperative service migration, IEEE Trans. Mobile Comput., vol. 23, no. 2, pp. 1503–1520, 2024.
Crossref Google Scholar
[31]
H. Chen, X. Wang, Z. Ning, and L. Guo, SDN-enabled 3C resource integration in green internet of electrical vehicles, in Proc. of CECNet 2021.
Crossref
[32]
Y. Zhao, Q. Zeng, and E. Y. Lam, Adaptive compressed sensing for real-time video compression, transmission, and reconstruction, in Proc. 2023 IEEE 10 th Int. Conf. Data Science and Advanced Analytics, Thessaloniki, Greece, 2023, pp. 1–10.
Crossref
[33]
Y. Zhao and E. Y. Lam, SASA: Saliency-aware self-adaptive snapshot compressive imaging, in Proc. 2024 IEEE Int. Conf. Acoustics, Speech and Signal Processing, Seoul, Republic of Korea, 2024, pp. 2370–2374.
Crossref
[34]
Y. Zhao, H. Zheng, J. Luo, and E. Y. Lam, Improving video colorization by test-time tuning, in Proc. 2023 IEEE Int. Conf. Image Processing, Kuala Lumpur, Malaysia, 2023, pp. 166–170.
Crossref
[35]

Y. Qi and S. Ishak, A hidden Markov model for short term prediction of traffic conditions on freeways, Transp. Res. Part C Emerg. Technol., vol. 43, pp. 95–111, 2014.
Crossref Google Scholar
[36]
B. Yu, H. Yin, and Z. Zhu, Spatio-temporal graph convolutional networks: A deep learning framework for traffic forecasting. in Proc. 27 th Int. Joint Conf. Artificial Intelligence, Stockholm, Sweden, 2018, pp. 3634–3640.
Crossref
[37]
R. Vinayakumar, K. P. Soman, and P. Poornachandran, Applying deep learning approaches for network traffic prediction, in Proc. 2017 Int. Conf. Advances in Computing, Communications and Informatics, Udupi, India, 2017, pp. 2353–2358.
Crossref
[38]
Y. Tian and L. Pan, Predicting short-term traffic flow by long short-term memory recurrent neural network, in Proc. 2015 IEEE Int. Conf. Smart City/SocialCom/SustainCom, Chengdu, China, 2015, pp. 153–158.
Crossref
[39]
Z. Pan, Y. Liang, W. Wang, Y. Yu, Y. Zheng, and J. Zhang, Urban traffic prediction from spatio-temporal data using deep meta learning, in Proc. 25 th ACM SIGKDD Int. Conf. Knowledge Discovery & Data Mining, Anchorage, AK, USA, 2019, pp. 1720–1730.
Crossref
[40]
Y. Zhao, H. Zheng, Z. Wang, J. Luo, and E. Y. Lam, MANet: Improving video denoising with a multi-alignment network, in Proc. 2022 IEEE Int. Conf. Image Processing, Bordeaux, France, 2022, pp. 2036–2040.
Crossref
[41]

S. Zhao, R. Y. Zhong, J. Wang, C. Xu, and J. Zhang, Unsupervised fabric defects detection based on spatial domain saliency and features clustering, Comput. Ind. Eng., vol. 185, p. 109681, 2023.
Crossref Google Scholar
[42]
S. Guo, Y. Lin, N. Feng, C. Song, and H. Wan, Attention based spatial-temporal graph convolutional networks for traffic flow forecasting, in Proc. 33 rd AAAI Conf. Artificial Intelligence, Honolulu, HI, USA, 2019, pp. 922–929.
Crossref
[43]
Y. Zhao, Z. Wang, and E. Y. Lam, Improving source localization by perturbing graph diffusion, in Proc. 2022 IEEE 9 th Int. Conf. Data Science and Advanced Analytics, Shenzhen, China, 2022, pp. 1–9.
Crossref
[44]
B. Zhou, J. Liu, S. Cui, and Y. Zhao, Large-scale traffic congestion prediction based on multimodal fusion and representation mapping, in Proc. 2022 IEEE 9 th Int. Conf. Data Science and Advanced Analytics, Shenzhen, China, 2022, pp. 1–9.
Crossref
[45]

Z. Zhou, X. Dong, Z. Li, K. Yu, C. Ding, and Y. Yang, Spatio-temporal feature encoding for traffic accident detection in VANET environment, IEEE Transactions on Intelligent Transportation Systems, vol. 23, no. 10, pp. 19772–19781, 2022.
Crossref Google Scholar
[46]
Y. Li, R. Yu, C. Shahabi, and Y. Liu, Diffusion convolutional recurrent neural network: Data-driven traffic forecasting, in Proc. of 6 th Int. Conf. Learning Representations, Vancouver, Canada.
[47]
X. Wang, C. Chen, Y. Min, J. He, B. Yang, and Y. Zhang, Efficient metropolitan traffic prediction based on graph recurrent neural network, arXiv preprint arXiv: 1811.00740, 2018.
[48]

K. Zhang, L. Zheng, Z. Liu, and N. Jia, A deep learning based multitask model for network-wide traffic speed prediction, Neurocomputing, vol. 396, pp. 438–450, 2020.
Crossref Google Scholar
[49]
T. Nguyen, G. Nguyen, and B. M. Nguyen, EO-CNN: An enhanced CNN model trained by equilibrium optimization for traffic transportation prediction, Procedia Comput. Sci., vol. 176, pp. 800–809, 2020.
Crossref
[50]
D. Yang, S. Li, Z. Peng, P. Wang, J. Wang, and H. Yang, MF-CNN: Traffic flow prediction using convolutional neural network and multi-features fusion, IEICE Trans. Inf. Syst., vol. E102, no. 8, pp. 1526–1536, 2019.
Crossref
[51]

M. Z. Mehdi, H. M. Kammoun, N. G. Benayed, D. Sellami, and A. D. Masmoudi, Entropy-based traffic flow labeling for CNN-based traffic congestion prediction from meta-parameters, IEEE Access, vol. 10, pp. 16123–16133, 2022.
Crossref Google Scholar
[52]

J. Wang, S. Zhao, C. Xu, J. Zhang, and R. Zhong, Brain-inspired interpretable network pruning for smart vision-based defect detection equipment, IEEE Trans. Ind. Inf., vol. 19, no. 2, pp. 1666–1673, 2023.
Crossref Google Scholar
[53]

W. Min and L. Wynter, Real-time road traffic prediction with spatio-temporal correlations, Transp. Res. Part C Emerg. Technol., vol. 19, no. 4, pp. 606–616, 2011.
Crossref Google Scholar
[54]

H. Yu, Z. Wu, S. Wang, Y. Wang, and X. Ma, Spatiotemporal recurrent convolutional networks for traffic prediction in transportation networks, Sensors, vol. 17, no. 7, p. 1501, 2017.
Crossref Google Scholar
[55]

C. Qiu, Y. Zhang, Z. Feng, P. Zhang, and S. Cui, Spatio-temporal wireless traffic prediction with recurrent neural network, IEEE Wirel. Commun. Lett., vol. 7, no. 4, pp. 554–557, 2018.
Crossref Google Scholar
[56]

X. Luo, D. Li, Y. Yang, and S. Zhang, Spatiotemporal traffic flow prediction with KNN and LSTM, J. Adv. Transp., vol. 2019, p. 4145353, 2019.
Crossref Google Scholar
[57]

J. Wang, R. Chen, and Z. He, Traffic speed prediction for urban transportation network: A path based deep learning approach, Transportation Research Part C: Emerging Technologies, vol. 100, pp. 372–385, 2019.
Crossref Google Scholar
[58]
J. Chung, C. Gulcehre, K. Cho, and Y. Bengio, Empirical evaluation of gated recurrent neural networks on sequence modeling, arXiv preprint arXiv: 1412.3555, 2014.
[59]
Y. Zhao, S. Shi, R. Ravi, Z. Wang, E. Y. Lam, and J. Zhao, H4M: Heterogeneous, multi-source, multi-modal, multi-view and multi-distributional dataset for socioeconomic analytics in the case of Beijing, in Proc. 2022 IEEE 9 th Int. Conf. Data Science and Advanced Analytics, Shenzhen, China, 2022, pp. 1–10.
Crossref
[60]
Y. Zhao, R. Ravi, S. Shi, Z. Wang, E. Y. Lam, and J. Zhao, PATE: Property, amenities, traffic and emotions coming together for real estate price prediction, in Proc. 2022 IEEE 9 th Int. Conf. Data Science and Advanced Analytics, Shenzhen, China, 2022, pp. 1–10.
Crossref
[61]
Y. Zhao, J. Zhao, and E. Y. Lam, House price prediction: A multi-source data fusion perspective, Big Data Mining and Analytics.
[62]
J. Devlin, M. W. Chang, K. Lee, and K. Toutanova, BERT: Pre-training of deep bidirectional transformers for language understanding, in Proc. 2019 Conf. North American Chapter of the Association for Computational Linguistics : Human Language Technologies, Volume 1, Minneapolis, MN, USA, 2019, pp. 4171–4186.
[63]
D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning internal representations by error propagation, in Parallel Distributed Processing : Explorations in the Microstructures of Cognition, D. E. Rumelhart and J. L. McClelland, eds. Cambridge, MA, USA: MIT Press, 1986, pp. 318−362.
[64]

S. Hochreiter and J. Schmidhuber, Long short-term memory, Neural Comput., vol. 9, no. 8, pp. 1735–1780, 1997.
Crossref Google Scholar
Big Data Mining and Analytics
Volume 7 Issue 3,
September 2024
Pages 621-636
DOI: 10.26599/BDMA.2024.9020020
Cite this article:
Zhou B, Liu J, Cui S, et al. A Large-Scale Spatio-Temporal Multimodal Fusion Framework for Traffic Prediction. Big Data Mining and Analytics, 2024, 7(3): 621-636. https://doi.org/10.26599/BDMA.2024.9020020

308

Views

51

Downloads

2

Crossref

1

Web of Science

2

Scopus

0

CSCD

Altmetrics
Received: 16 October 2023
Revised: 23 February 2024
Accepted: 22 March 2024
Published: 28 August 2024
© The author(s) 2024.

The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).
Return