The metaverse is attracting considerable attention recently. It aims to build a virtual environment that people can interact with the world and cooperate with each other. In this survey paper, we re-introduce metaverse in a new framework based on a broad range of technologies, including perception which enables us to precisely capture the characteristics of the real world, computation which supports the large computation requirement over large-scale data, reconstruction which builds the virtual world from the real one, cooperation which facilitates long-distance communication and teamwork between users, and interaction which bridges users and the virtual world. Despite its popularity, the fundamental techniques in this framework are still immature. Innovating new techniques to facilitate the applications of metaverse is necessary. In recent years, artificial intelligence (AI), especially deep learning, has shown promising results for empowering various areas, from science to industry. It is reasonable to imagine how we can combine AI with the framework in order to promote the development of metaverse. In this survey, we present the recent achievement by AI for metaverse in the proposed framework, including perception, computation, reconstruction, cooperation, and interaction. We also discuss some future works that AI can contribute to metaverse.
A. Scavarelli, A. Arya, and R. J Teather, Virtual reality and augmented reality in social learning spaces: A literature review, Virtual Reality, vol. 25, no. 1, pp. 257–277, 2021.
G. W. Wei, Protein structure prediction beyond alphafold, Nat. Mach. Intell., vol. 1, no. 8, pp. 336–337, 2019.
S. H. M. Mehr, M. Craven, A. I. Leonov, G. Keenan, and L. Cronin, A universal system for digitization and automatic execution of the chemical synthesis literature, Science, vol. 370, no. 6512, pp. 101–108, 2020.
A. Mirhoseini, A. Goldie, M. Yazgan, J. W. Jiang, E. Songhori, S. Wang, Y. J. Lee, E. Johnson, O. Pathak, A. Nazi, et al., A graph placement methodology for fast chip design, Nature, vol. 594, no. 7862, pp. 207–212, 2021.
Q. Li, J. Pellegrino, D. J. Lee, A. A. Tran, H. A. Chaires, R Wang, J. E. Park, K. Ji, D. Chow, N. Zhang, et al., Synthetic group a streptogramin antibiotics that overcome vat resistance, Nature, vol. 586, no. 7827, pp. 145–150, 2020.
X. Yuan, M. Ji, J. Wu, D. J. Brady, Q. Dai, and L. Fang, A modular hierarchical array camera, Light: Sci. Appl., vol. 10, no. 1, p. 37, 2021.
B. Zhang, Y. Guo, Y. Li, Y. He, H. Wang, and Q. Dai, Memory recall: A simple neural network training framework against catastrophic forgetting, IEEE Trans. Neural Netw. Learn. Syst., vol. 33, no. 5, pp. 2010–2022, 2022.
X. Lin, Y. Rivenson, N. T. Yardimci, M. Veli, Y. Luo, M. Jarrahi, and A. Ozcan, All-optical machine learning using diffractive deep neural networks, Science, vol. 361, no. 6406, pp. 1004–1008, 2018.
T. Zhou, X. Lin, J. Wu, Y. Chen, H. Xie, Y. Li, J. Fan, H. Wu, L. Fang, and Q. Dai, Large-scale neuromorphic optoelectronic computing with a reconfigurable diffractive processing unit, Nat. Photonics, vol. 15, no. 5, pp. 367–373, 2021.
L. I. Rudin, S. Osher, and E. Fatemi, Nonlinear total variation based noise removal algorithms, Phys. D:Nonlinear Phenom., vol. 60, no. 1-4, pp. 259–268, 1992.
X. Yuan, T. H. Tsai, R. Zhu, P. Llull, D. J. Brady, and L. Carin, Compressive hyperspectral imaging with side information, IEEE J. Sel. Top. Signal Process., vol. 9, no. 6, pp. 964–976, 2015.
J. Yang, X. Yuan, X. Liao, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, Video compressive sensing using Gaussian mixture models, IEEE Trans. Image Process., vol. 23, no. 11, pp. 4863–4878, 2014.
J. Yang, X. Liao, X. Yuan, P. Llull, D. J. Brady, G. Sapiro, and L. Carin, Compressive sensing by learning a Gaussian mixture model from measurements, IEEE Trans. Image Process., vol. 24, no. 1, pp. 106–119, 2015.
Y. Liu, X. Yuan, J. Suo, D. J. Brady, and Q. Dai, Rank minimization for snapshot compressive imaging, IEEE Trans. Pattern Anal. Mach. Intell., vol. 41, no. 12, pp. 2990–3006, 2019.
X. Yuan, D. J. Brady, and A. K. Katsaggelos, Snapshot compressive imaging: Theory, algorithms, and applications, IEEE Signal Process. Mag., vol. 38, no. 2, pp. 65–88, 2021.
M. Qiao, Z. Meng, J. Ma, and X. Yuan, Deep learning for video compressive sensing, APL Photonics, vol. 5, no. 3, p. 030801, 2020.
S. Zheng, C. Wang, X. Yuan, and H. Xin, Super-compression of large electron microscopy time series by deep compressive sensing learning, Patterns, vol. 2, no. 7, p. 100292, 2021.
M. Qiao, X. Liu, and X. Yuan, Snapshot temporal compressive microscopy using an iterative algorithm with untrained neural networks, Opt. Lett., vol. 46, no. 8, pp. 1888–1891, 2021.
V. Sitzmann, S. Diamond, Y. Peng, X. Dun, S. Boyd, W. Heidrich, F. Heide, and G. Wetzstein, End-to-end optimization of optics and image processing for achromatic extended depth of field and super-resolution imaging, ACM Trans. Graphics, vol. 37, no. 4, p. 114, 2018.
L. Wang, T. Zhang, Y. Fu, and H. Huang, HyperReconNet: Joint coded aperture optimization and image reconstruction for compressive hyperspectral imaging, IEEE Trans. Image Process., vol. 28, no. 5, pp. 2257–2270, 2019.
J. Zhang, C. Zhao, and W. Gao, Optimization-inspired compact deep compressive sensing, IEEE J. Sel. Top. Signal Process., vol. 14, no. 4, pp. 765–774, 2020.
J. W. Han, J. H. Kim, H. T. Lee, and S. J. Ko, A novel training based auto-focus for mobile-phone cameras, IEEE Trans. Consum. Electron., vol. 57, no. 1, pp. 232–238, 2011.
Y. S. Rawat and M. S. Kankanhalli, Context-aware photography learning for smart mobile devices, ACM Trans. Multimedia Comput. , Commun. , Appl., vol. 12, no. 1s, p. 19, 2015.
Y. S. Rawat and M. S. Kankanhalli, ClickSmart: A context-aware viewpoint recommendation system for mobile photography, IEEE Trans. Circuits Syst. Video Technol., vol. 27, no. 1, pp. 149–158, 2017.
C. Wang, Q. Fu, X. Dun, and W. Heidrich, Megapixel adaptive optics: Towards correcting large-scale distortions in computational cameras, ACM Trans. Graphics, vol. 37, no. 4, p. 115, 2018.
A. Ashok, P. K. Baheti, and M. A. Neifeld, Compressive imaging system design using task-specific information, Appl. Opt., vol. 47, no. 25, pp. 4457–4471, 2008.
F. Zhou and Y. Chai, Near-sensor and in-sensor computing, Nat. Electron., vol. 3, no. 11, pp. 664–671, 2020.
Y. Chai, In-sensor computing for machine vision, Nature, vol. 579, no. 7797, pp. 32–33, 2020.
M. A. Zidan, J. P. Strachan, and W. D. Lu, The future of electronics based on memristive systems, Nat. Electron., vol. 1, no. 1, pp. 22–29, 2018.
L. Mennel, J. Symonowicz, S. Wachter, D. K. Polyushkin, A. J. Molina-Mendoza, and T. Mueller, Ultrafast machine vision with 2D material neural network image sensors, Nature, vol. 579, no. 7797, pp. 62–66, 2020.
W. Wang, G. Pedretti, V. Milo, R. Carboni, A. Calderoni, N. Ramaswamy, A. S. Spinelli, and D. Ielmini, Learning of spatiotemporal patterns in a spiking neural network with resistive switching synapses, Sci. Adv., vol. 4, no. 9, p. eaat4752, 2018.
U. A. Butt, M. Mehmood, S. B. H. Shah, R. Amin, M. W. Shaukat, S. M. Raza, D. Y. Suh, and M. J. Piran, A review of machine learning algorithms for cloud computing security, Electronics, vol. 9, no. 9, p. 1379, 2020.
A. A. Grusho, M. I. Zabezhailo, A. A. Zatsarinnyi, and V. O. Piskovskii, On some artificial intelligence methods and technologies for cloud-computing protection, Autom. Doc. Math. Linguist., vol. 51, no. 2, pp. 62–74, 2017.
H. M. El-Boghdadi and R. A. Ramadan, Resource scheduling for offline cloud computing using deep reinforcement learning, Int. J. Comput. Sci. Netw. Secur., vol. 19, no. 4, pp. 54–60, 2019.
L. Floridi and M. Chiriatti, GPT-3: Its nature, scope, limits, and consequences, Minds Mach., vol. 30, no. 4, pp. 681–694, 2020.
S. Yuan, H. Zhao, Z. Du, M. Ding, X. Liu, Y. Cen, X. Zou, Z. Yang, and J. Tang, WuDaoCorpora: A super large-scale Chinese corpora for pre-training language models, AI Open, vol. 2, pp. 65–68, 2021.
J. Jumper, R. Evans, A. Pritzel, T. Green, M. Figurnov, O. Ronneberger, K. Tunyasuvunakool, R. Bates, A. Žídek, A. Potapenko, et al., Highly accurate protein structure prediction with AlphaFold, Nature, vol. 596, no. 7873, pp. 583–589, 2021.
P. Li, J. Li, Z. Huang, T. Li, C. Gao, S. M. Yiu, and K. Chen, Multi-key privacy-preserving deep learning in cloud computing, Future Gener. Comput. Syst., vol. 74, pp. 76–85, 2017.
Y. Li, H. Li, G. Xu, T. Xiang, X. Huang, and R. Lu, Toward secure and privacy-preserving distributed deep learning in fog-cloud computing, IEEE Internet Things J., vol. 7, no. 12, pp. 11460–11472, 2020.
T. Li, A. K. Sahu, A. S. Talwalkar, and V. Smith, Federated learning: Challenges, methods, and future directions, IEEE Signal Process. Mag., vol. 37, no. 3, pp. 50–60, 2020.
J. McCarthy, Generality in artificial intelligence, Commun. ACM, vol. 30, no. 12, pp. 1030–1035, 1987.
The HEP Software Foundation, J. Albrecht, A. A. Alves Jr, G. Amadio, G. Andronico, N. Anh-Ky, L. Aphecetche, J. Apostolakis, M. Asai, L. Atzori, et al., A roadmap for HEP software and computing R&D for the 2020s,Comput. Softw. Big Sci., vol. 3, no. 1, p. 7, 2019.
Y. LeCun, Y. Bengio, and G. Hinton, Deep learning, Nature, vol. 521, no. 7553, pp. 436–444, 2015.
V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. A. Riedmiller, A. K. Fidjeland, G. Ostrovski, et al., Human-level control through deep reinforcement learning, Nature, vol. 518, no. 7540, pp. 529–533, 2015.
Y. LeCun, L. Bottou, Y. Bengio, and P. Haffner, Gradient-based learning applied to document recognition, Proc. IEEE, vol. 86, no. 11, pp. 2278–2324, 1998.
J. Zhang, Z. Wang, and N. Verma, In-memory computation of a machine-learning classifier in a standard 6T SRAM array, IEEE J. Solid-State Circuits, vol. 52, no. 4, pp. 915–924, 2017.
A. Biswas and A. P. Chandrakasan, CONV-SRAM: An energy-efficient SRAM with in-memory dot-product computation for low-power convolutional neural networks, IEEE J. Solid-State Circuits, vol. 54, no. 1, pp. 217–230, 2019.
G. W. Burr, B. N. Kurdi, J. C. Scott, C. H. Lam, K. Gopalakrishnan, and R. S. Shenoy, Overview of candidate device technologies for storage-class memory, IBM J. Res. Dev, vol. 52, no. 4-5, pp. 449–464, 2008.
E. Chen, D. Apalkov, Z. Diao, A. Driskill-Smith, D. Druist, D. Lottis, V. Nikitin, X. Tang, S. Watts, S. Wang, et al., Advances and future prospects of spin-transfer torque random access memory, IEEE Trans. Magn., vol. 46, no. 6, pp. 1873–1878, 2010.
V. Joshi, M. Le Gallo, S. Haefeli, I. Boybat, S. R. Nandakumar, C. Piveteau, M. Dazzi, B. Rajendran, A. Sebastian, and E. Eleftheriou, Accurate deep neural network inference using computational phase-change memory, Nat. Commun., vol. 11, no. 1, p. 2473, 2020.
H. Wu and Q. Dai, Artificial intelligence accelerated by light, Nature, vol. 589, no. 7840, pp. 25–26, 2021.
Y. Shen, N. Harris, S. Skirlo, M. Prabhu, T. Baehr-Jones, M. Hochberg, X. Sun, S. Zhao, H. Larochelle, D. Englund, et al., Deep learning with coherent nanophotonic circuits, Nat. Photonics, vol. 11, no. 7, pp. 441–446, 2017.
J. Feldmann, N. Youngblood, C. D. Wright, H. Bhaskaran, and W. H. P. Pernice, All-optical spiking neurosynaptic networks with self-learning capabilities, Nature, vol. 569, no. 7755, pp. 208–214, 2019.
J. Feldmann, N. Youngblood, M. Karpov, H. Gehring, X. Li, M. Stappers, M. Le Gallo, X. Fu, A. Lukashchuk, A. S. Raja, et al., Parallel convolutional processing using an integrated photonic tensor core, Nature, vol. 589, no. 7840, pp. 52–58, 2021.
X. Xu, M. Tan, B. Corcoran, J. Wu, A. Boes, T. G. Nguyen, S. T. Chu, B. E. Little, D. G. Hicks, R. Morandotti, et al., 11 tops photonic convolutional accelerator for optical neural networks, Nature, vol. 589, no. 7840, pp. 44–51, 2021.
E. Khoram, A. Chen, D. Liu, L. Ying, Q. Wang, M. Yuan, and Z. Yu, Nanophotonic media for artificial neural inference, Photonics Res., vol. 7, no. 8, pp. 823–827, 2019.
F. Tao, W. Liu, J. Liu, X. Liu, Q. Liu, T. Qu, T. Hu, Z. Zhang, F. Xiang, W. Xu, et al., Digital twin and its potential application exploration, Comput. Integr. Manuf. Syst., vol. 24, no. 1, pp. 1–8, 2018.
X. F. Han, H. Laga, and M. Bennamoun, Image-based 3D object reconstruction: State-of-the-art and trends in the deep learning era, IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 5, pp. 1578–1604, 2021.
M. Zollhöfer, P. Stotko, A. Görlitz, C. Theobalt, M. Nießner, R. Klein, and A. Kolb, State of the art on 3D reconstruction with RGB-D cameras, Comput. Graphics Forum, vol. 37, no. 2, pp. 625–652, 2018.
M. Zollhöfer, J. Thies, P. Garrido, D. Bradley, T. Beeler, P. Pérez, M. Stamminger, M. Nießner, and C. Theobalt, State of the art on monocular 3D face reconstruction, tracking, and applications, Comput. Graphics Forum, vol. 37, no. 2, pp. 523–550, 2018.
S. Kumar, Y. Dai, and H. Li, Superpixel soup: Monocular dense 3D reconstruction of a complex dynamic scene, IEEE Trans. Pattern Anal. Mach. Intell., vol. 43, no. 5, pp. 1705–1717, 2021.
J. N. P. Martel, D. B. Lindell, C. Z. Lin, E. R. Chan, M. Monteiro, and G. Wetzstein, Acorn: Adaptive coordinate networks for neural scene representation, ACM Trans. Graphics, vol. 40, no. 4, p. 58, 2021.
B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, NeRF: Representing scenes as neural radiance fields for view synthesis, Commun. ACM, vol. 65, no. 1, pp. 99–106, 2022.
A. Tewari, O. Fried, J. Thies, V. Sitzmann, S. Lombardi, K. Sunkavalli, R. Martin-Brualla, T. Simon, J. Saragih, M. Nießner, et al., State of the art on neural rendering, Comput. Graphics Forum, vol. 39, no. 2, pp. 701–727, 2020.
S. M. Ali Eslami, D. J. Rezende, F. Besse, F. Viola, A. S. Morcos, M. Garnelo, A. Ruderman, A. A. Rusu, I. Danihelka, K. Gregor, et al., Neural scene representation and rendering, Science, vol. 360, no. 6394, pp. 1204–1210, 2018.
X. Chen, C. Wu, T. Chen, H. Zhang, Z. Liu, Y. Zhang, and M. Bennis, Age of information aware radio resource management in vehicular networks: A proactive deep reinforcement learning perspective, IEEE Trans. Wireless Commun., vol. 19, no. 4, pp. 2268–2281, 2020.
L. Li, Y. Xu, J. Yin, W. Liang, X. Li, W. Chen, and Z. Han, Deep reinforcement learning approaches for content caching in cache-enabled D2D networks, IEEE Internet Things J., vol. 7, no. 1, pp. 544–557, 2020.
L. T. Tan and R. Q. Hu, Mobility-aware edge caching and computing in vehicle networks: A deep reinforcement learning, IEEE Trans. Veh. Technol., vol. 67, no. 11, pp. 10190–10203, 2018.
M. Yan, G. Feng, J. Zhou, Y. Sun, and Y. C. Liang, Intelligent resource scheduling for 5G radio access network slicing, IEEE Trans. Veh. Technol., vol. 68, no. 8, pp. 7691–7703, 2019.
H. Li, K. Ota, and M. Dong, Learning IoT in edge: Deep learning for the internet of things with edge computing, IEEE Netw., vol. 32, no. 1, pp. 96–101, 2018.
D. Ravì, C. Wong, B. Lo, and G. Z. Yang, A deep learning approach to on-node sensor data analytics for mobile or wearable devices, IEEE J. Biomedical Health Inform., vol. 21, no. 1, pp. 56–64, 2017.
H. Ye, G. Y. Li, and B. H. Juang, Power of deep learning for channel estimation and signal detection in OFDM systems, IEEE Wireless Commun. Lett., vol. 7, no. 1, pp. 114–117, 2018.
C. K. Wen, W. T. Shih, and S. Jin, Deep learning for massive MIMO CSI feedback, IEEE Wireless Commun. Lett., vol. 7, no. 5, pp. 748–751, 2018.
X. Ma, Z. Gao, F. Gao, and M. Di Renzo, Model-driven deep learning based channel estimation and feedback for millimeter-wave massive hybrid MIMO systems, IEEE J. Sel. Areas Commun., vol. 39, no. 8, pp. 2388–2406, 2021.
F. Liang, C. Shen, and F. Wu, An iterative BP-CNN architecture for channel decoding, IEEE J. Sel. Top. Signal Process., vol. 12, no. 1, pp. 144–159, 2018.
H. Xie, Z. Qin, G. Y. Li, and B. H. Juang, Deep learning enabled semantic communication systems, IEEE Trans. Signal Process., vol. 69, pp. 2663–2675, 2021.
Z. Weng and Z. Qin, Semantic communication systems for speech transmission, IEEE J. Sel. Areas Commun., vol. 39, no. 8, pp. 2434–2444, 2021.
A. Maxmen, AI researchers embrace bitcoin technology to share medical data, Nature, vol. 555, no. 7696, pp. 293–294, 2018.
K. Salah, M. H. Ur Rehman, N. Nizamuddin, and A. Al-Fuqaha, Blockchain for AI: Review and open research challenges, IEEE Access, vol. 7, pp. 10127–10149, 2019.
Y. Rizk, M. Awad, and E. W. Tunstel, Decision making in multiagent systems: A survey, IEEE Trans. Cognit. Dev. Syst., vol. 10, no. 3, pp. 514–529, 2018.
N. Dhieb, H. Ghazzai, H. Besbes, and Y. Massoud. A secure AI-driven architecture for automated insurance systems: Fraud detection and risk measurement, IEEE Access, vol. 8, pp. 58546–58558, 2020.
Z. Zhang, H. Ning, F. Shi, F. Farha, Y. Xu, J. Xu, F. Zhang, and K. K. R. Choo, Artificial intelligence in cyber security: Research advances, challenges, and opportunities, Artif. Intell. Rev., vol. 55, no. 2, pp. 1029–1053, 2022.
Y. Xin, L. Kong, Z. Liu, Y. Chen, Y. Li, H. Zhu, M. Gao, H. Hou, and C. Wang, Machine learning and deep learning methods for cybersecurity, IEEE Access, vol. 6, pp. 35365–35381, 2018.
X. Lu, L. Xiao, T. Xu, Y. Zhao, Y. Tang, and W. Zhuang, Reinforcement learning based PHY authentication for VANETs, IEEE Trans. on Veh. Technol., vol. 69, no. 3, pp. 3068–3079, 2020.
Y. Zhang, X. Chen, D. Guo, M. Song, Y. Teng, and X. Wang, PCCN: Parallel cross convolutional neural network for abnormal network traffic flows detection in multi-class imbalanced network traffic flows, IEEE Access, vol. 7, pp. 119904–119916, 2019.
J. J. Vidal, Toward direct brain-computer communication, Annu. Rev. Biophys. Bioeng., vol. 2, pp. 157–180, 1973.
H. Zhang, M. Zhao, C. Wei, D. Mantini, Z. Li, and Q. Liu, EEGdenoiseNet: A benchmark dataset for deep learning solutions of EEG denoising, J. Neural Eng., vol. 18, no. 5, p. 056057, 2021.
W. Sun, Y. Su, X. Wu, and X. Wu, A novel end-to-end 1D-rescnn model to remove artifact from EEG signals, Neurocomputing, vol. 404, pp. 108–121, 2020.
F. R. Willett, D. T. Avansino, L. R. Hochberg, J. M. Henderson, K. V. Shenoy, High-performance brain-to-text communication via handwriting, Nature, vol. 593, no. 7858, pp. 249–254, 2021.
D. A. Moses, M. K. Leonard, J. G. Makin, and E. F. Chang, Real-time decoding of question-and-answer speech dialogue using human cortical activity, Nat. Commun., vol. 10, no. 1, p. 3096, 2019.
M. Capogrosso, T. Milekovic, D. Borton, F. Wagner, E. M. Moraud, J. B. Mignardot, N. Buse, J. Gandar, Q. Barraud, D. Xing, et al., A brain-spine interface alleviating gait deficits after spinal cord injury in primates, Nature, vol. 539, no. 7628, pp. 284–288, 2016.
K. W. Scangos, A. N. Khambhati, P. M. Daly, G. S. Makhoul, L. P. Sugrue, H. Zamanian, T. X. Liu, V. R. Rao, K. K. Sellers, H. E. Dawes, et al., Closed-loop neuromodulation in an individual with treatment-resistant depression, Nat. Med., vol. 27, no. 10, pp. 1696–1700, 2021.
J. Gao, P. Li, Z. Chen, and J. Zhang, A survey on deep learning for multimodal data fusion, Neural Comput., vol. 32, no. 5, pp. 829–864, 2020.
L. Shi, B. Li, C. Kim, P. Kellnhofer, and W. Matusik, Towards real-time photorealistic 3D holography with deep neural networks, Nature, vol. 591, no. 7849, pp. 234–239, 2021.
This work was supported by the National Key Research and Development Program of China (Nos. 2020AAA0105500 and 2021ZD0109901), the National Natural Science Foundation of China (Nos. 62088102, 62125106, and 61971260), and the Beijing Municipal Science and Technology Commission (No. Z181100003118014).
The articles published in this open access journal are distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/).