Journal Home > Volume 17 , Issue 4
Nano Research 2024, 17(4): 2971-2980 https://doi.org/10.1007/s12274-023-6104-1
Research Article | Issue | Published: 23 September 2023

Accurate atomic scanning transmission electron microscopy analysis enabled by deep learning

Show Author's Information Tianshu Chu1,2,3,§,  Lei Zhou1,2,3,§,  Bowei Zhang1,2,3( ),  Fu-Zhen Xuan1,2,3( )
Shanghai Key Laboratory of Intelligent Sensing and Detection Technology, East China University of Science and Technology, Shanghai 200237, China
School of Mechanical and Power Engineering, East China University of Science and Technology, Shanghai 200237, China
Key Laboratory of Pressure Systems and Safety of Ministry of Education, East China University of Science and Technology, Shanghai 200237, China

§ Tianshu Chu and Lei Zhou contributed equally to this work.

Keywords:
deep learning, single atoms, low-dimensional materials, atomic defects
Topics:
Catalyst activityChemical sensors High resolution transmission electron microscopyImage processing
Cite this article:
Chu T, Zhou L, Zhang B, et al. Accurate atomic scanning transmission electron microscopy analysis enabled by deep learning. Nano Research, 2024, 17(4): 2971-2980. https://doi.org/10.1007/s12274-023-6104-1
Download citation
EndNote(RIS)
BibTeX

671

Views

59

Downloads

Citations

1

Crossref

1

WoS

1

Scopus

0

CSCD

Abstract Full text Electronic supplementary material About this article

Currently, the machine learning (ML)-based scanning transmission electron microscopy (STEM) analysis is limited in the simulative stage, its application in experimental STEM is needed but challenging. Herein, we built up a universal model to analyze the vacancy defects and single atoms accurately and rapidly in experimental STEM images using a full convolution network. In our model, the unavoidable interference factors of noise, aberration, and carbon contamination were fully considered during the training, which were difficult to be considered in the past. Even toward the simultaneous identification of various vacancy types and low-contrast single atoms in the low-quality STEM images, our model showed rapid process speed (45 images per second) and high accuracy (> 95%). This work represents an improvement in experimental STEM image analysis by ML.


menu
Abstract
Full text
Outline
Electronic supplementary material
About this article

Accurate atomic scanning transmission electron microscopy analysis enabled by deep learning

Show Author's information Tianshu Chu1,2,3,§, Lei Zhou1,2,3,§, Bowei Zhang1,2,3( ), Fu-Zhen Xuan1,2,3( )
Shanghai Key Laboratory of Intelligent Sensing and Detection Technology, East China University of Science and Technology, Shanghai 200237, China
School of Mechanical and Power Engineering, East China University of Science and Technology, Shanghai 200237, China
Key Laboratory of Pressure Systems and Safety of Ministry of Education, East China University of Science and Technology, Shanghai 200237, China

§ Tianshu Chu and Lei Zhou contributed equally to this work.

Abstract

Currently, the machine learning (ML)-based scanning transmission electron microscopy (STEM) analysis is limited in the simulative stage, its application in experimental STEM is needed but challenging. Herein, we built up a universal model to analyze the vacancy defects and single atoms accurately and rapidly in experimental STEM images using a full convolution network. In our model, the unavoidable interference factors of noise, aberration, and carbon contamination were fully considered during the training, which were difficult to be considered in the past. Even toward the simultaneous identification of various vacancy types and low-contrast single atoms in the low-quality STEM images, our model showed rapid process speed (45 images per second) and high accuracy (> 95%). This work represents an improvement in experimental STEM image analysis by ML.

Keywords: deep learning, single atoms, low-dimensional materials, atomic defects

References(58)

[1]

Chu, T. S.; Rong, C.; Zhou, L.; Mao, X. Y.; Zhang, B. W.; Xuan, F. Z. Progress and perspectives of single-atom catalysts for gas sensing. Adv. Mater. 2023, 35, 2206783.
DOI Google Scholar
[2]

Mitterreiter, E.; Schuler, B.; Micevic, A.; Hernangómez-Pérez, D.; Barthelmi, K.; Cochrane, K. A.; Kiemle, J.; Sigger, F.; Klein, J.; Wong, E. et al. The role of chalcogen vacancies for atomic defect emission in MoS2. Nat. Commun. 2021, 12, 3822.
DOI Google Scholar
[3]

Hong, J. H.; Hu, Z. X.; Probert, M.; Li, K.; Lv, D. H.; Yang, X. N.; Gu, L.; Mao, N. N.; Feng, Q. L.; Xie, L. M. et al. Exploring atomic defects in molybdenum disulphide monolayers. Nat. Commun. 2015, 6, 6293.
DOI Google Scholar
[4]

Sun, H. M.; Yan, Z. H.; Liu, F. M.; Xu, W. C.; Cheng, F. Y.; Chen, J. Self-supported transition-metal-based electrocatalysts for hydrogen and oxygen evolution. Adv. Mater. 2020, 32, 1806326.
DOI Google Scholar
[5]

Zhang, N. Q.; Ye, C. L.; Yan, H.; Li, L. C.; He, H.; Wang, D. S.; Li, Y. D. Single-atom site catalysts for environmental catalysis. Nano Res. 2020, 13, 3165–3182.
DOI Google Scholar
[6]

Zhang, Q. Q.; Guan, J. Q. Applications of single-atom catalysts. Nano Res. 2022, 15, 38–70.
DOI Google Scholar
[7]

Zhang, J. Q.; Zhao, Y. F.; Guo, X.; Chen, C.; Dong, C. L.; Liu, R. S.; Han, C. P.; Li, Y. D.; Gogotsi, Y.; Wang, G. X. Single platinum atoms immobilized on an MXene as an efficient catalyst for the hydrogen evolution reaction. Nat. Catal. 2018, 1, 985–992.
DOI Google Scholar
[8]

Du, Q. H.; Wu, L. J.; Cao, H. B.; Kang, C. J.; Nelson, C.; Pascut, G. L.; Besara, T.; Siegrist, T.; Haule, K.; Kotliar, G. et al. Vacancy defect control of colossal thermopower in FeSb2. npj Quantum Mater. 2021, 6, 13.
DOI Google Scholar
[9]

Gao, W.; Li, S.; He, H. C.; Li, X. N.; Cheng, Z. X.; Yang, Y.; Wang, J. L.; Shen, Q.; Wang, X. Y.; Xiong, Y. J. et al. Vacancy-defect modulated pathway of photoreduction of CO2 on single atomically thin AgInP2S6 sheets into olefiant gas. Nat. Commun. 2021, 12, 4747.
DOI Google Scholar
[10]

Dou, Y. H.; He, C. T.; Zhang, L.; Yin, H. J.; Al-Mamun, M.; Ma, J. M.; Zhao, H. J. Approaching the activity limit of CoSe2 for oxygen evolution via Fe doping and Co vacancy. Nat. Commun. 2020, 11, 1664.
DOI Google Scholar
[11]

Sun, Y. H.; Zhao, H. F.; Zhou, D.; Zhu, Y. C.; Ye, H. Y.; Moe, Y. A.; Wang, R. M. Direct observation of epitaxial alignment of Au on MoS2 at atomic resolution. Nano Res. 2019, 12, 947–954.
DOI Google Scholar
[12]

Gu, H. Y.; Liu, X.; Liu, X. F.; Ling, C. C.; Wei, K.; Zhan, G. M.; Guo, Y. B.; Zhang, L. Z. Adjacent single-atom irons boosting molecular oxygen activation on MnO2. Nat. Commun. 2021, 12, 5422.
DOI Google Scholar
[13]

Hobbs, C.; Downing, C.; Jaskaniec, S.; Nicolosi, V. TEM and EELS characterization of Ni-Fe layered double hydroxide decompositions caused by electron beam irradiation. npj 2D Mater. Appl. 2021, 5, 29.
DOI Google Scholar
[14]

Zhai, P. L.; Xia, M. Y.; Wu, Y. Z.; Zhang, G. H.; Gao, J. F.; Zhang, B.; Cao, S. Y.; Zhang, Y. T.; Li, Z. W.; Fan, Z. Z. et al. Engineering single-atomic ruthenium catalytic sites on defective nickel-iron layered double hydroxide for overall water splitting. Nat. Commun. 2021, 12, 4587.
DOI Google Scholar
[15]

Cui, W. J.; Hu, Z. Y.; Unocic, R. R.; Van Tendeloo, G.; Sang, X. H. Atomic defects, functional groups and properties in MXenes. Chin. Chem. Lett. 2021, 32, 339–344.
DOI Google Scholar
[16]

Zhang, B. W.; Zhu, C. Q.; Wu, Z. S.; Stavitski, E.; Lui, Y. H.; Kim, T. H.; Liu, H.; Huang, L.; Luan, X. C.; Zhou, L. et al. Integrating Rh species with NiFe-layered double hydroxide for overall water splitting. Nano Lett. 2020, 20, 136–144.
DOI Google Scholar
[17]

Pattengale, B.; Huang, Y. C.; Yan, X. X.; Yang, S. Z.; Younan, S.; Hu, W. H.; Li, Z. D.; Lee, S.; Pan, X. Q.; Gu, J. et al. Dynamic evolution and reversibility of single-atom Ni(II) active site in 1T-MoS2 electrocatalysts for hydrogen evolution. Nat. Commun. 2020, 11, 4114.
DOI Google Scholar
[18]

Kalinin, S. V.; Ophus, C.; Voyles, P. M.; Erni, R.; Kepaptsoglou, D.; Grillo, V.; Lupini, A. R.; Oxley, M. P.; Schwenker, E.; Chan, M. K. Y. et al. Machine learning in scanning transmission electron microscopy. Nat. Rev. Methods Primers 2022, 2, 11.
DOI Google Scholar
[19]
Jimenez-Del-Toro, O.; Otálora, S.; Andersson, M.; Eurén, K.; Hedlund, M.; Rousson, M.; Müller, H.; Atzori, M. Analysis of histopathology images: From traditional machine learning to deep learning. In Biomedical Texture Analysis. Depeursinge, A.; Al-Kadi, A. S.; Mitchell, J. R., Eds.; Elsevier: Amsterdam, 2017; pp 281–314.
[20]

Castiglioni, I.; Rundo, L.; Codari, M.; Di Leo, G.; Salvatore, C.; Interlenghi, M.; Gallivanone, F.; Cozzi, A.; D’Amico, N. C.; Sardanelli, F. AI applications to medical images: From machine learning to deep learning. Phys. Medica 2021, 83, 9–24.
DOI Google Scholar
[21]

Lecun, Y.; Bottou, L.; Bengio, Y.; Haffner, P. Gradient-based learning applied to document recognition. Proc. IEEE 1998, 86, 2278–2324.
DOI Google Scholar
[22]
Simonyan, K.; Zisserman, A. Very deep convolutional networks for large-scale image recognition. In 3rd International Conference on Learning Representations, San Diego, USA, 2015, pp 1–14.
[23]
Liu, J. J.; Hou, Q. B.; Cheng, M. M.; Feng, J. S.; Jiang, J. M. A simple pooling-based design for real-time salient object detection. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, USA, 2019, pp 3912–3921.
[24]

Vaisman, R.; Kroese, D. P.; Gertsbakh, I. B. Improved sampling plans for combinatorial invariants of coherent systems. IEEE Trans. Reliab. 2016, 65, 410–424.
DOI Google Scholar
[25]
Wang, T. T.; Borji, A.; Zhang, L. H.; Zhang, P. P.; Lu, H. C. A stagewise refinement model for detecting salient objects in images. In 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017, pp 4039–4048.
[26]

Cheng, M. M.; Fan, D. P. Structure-measure: A new way to evaluate foreground maps. Int. J. Comput. Vis. 2021, 129, 2622–2638.
DOI Google Scholar
[27]
Zhang, P. P.; Wang, D.; Lu, H. C.; Wang, H. Y.; Yin, B. C. Learning uncertain convolutional features for accurate saliency detection. In 2017 IEEE International Conference on Computer Vision, Venice, Italy, 2017, pp 212–221.
[28]
Chen, S. H.; Tan, X. L.; Wang, B.; Hu, X. L. Reverse attention for salient object detection. In 15th European Conference on Computer Vision, Munich, Germany, 2018, pp 236–252.
[29]

Zhang, Z. X.; Liu, Q. J.; Wang, Y. H. Road extraction by deep residual U-Net. IEEE Geosci. Remote Sens. Lett. 2018, 15, 749–753.
DOI Google Scholar
[30]

Qin, X. B.; Zhang, Z. C.; Huang, C. Y.; Dehghan, M.; Zaiane, O. R.; Jagersand, M. U2-Net: Going deeper with nested U-structure for salient object detection. Pattern Recognit. 2020, 106, 107404.
DOI Google Scholar
[31]

Lee, C. H.; Khan, A.; Luo, D.; Santos, T. P.; Shi, C. Q.; Janicek, B. E.; Kang, S. M.; Zhu, W. J.; Sobh, N. A.; Schleife, A. et al. Deep learning enabled strain mapping of single-atom defects in two-dimensional transition metal dichalcogenides with sub-picometer precision. Nano Lett. 2020, 20, 3369–3377.
DOI Google Scholar
[32]

Yang, S. H.; Choi, W.; Cho, B. W.; Agyapong-Fordjour, F. O. T.; Park, S.; Yun, S. J.; Kim, H. J.; Han, Y. K.; Lee, Y. H.; Kim, K. K. et al. Deep learning-assisted quantification of atomic dopants and defects in 2D materials. Adv. Sci. 2021, 8, 2101099.
DOI Google Scholar
[33]

Lin, R. Q.; Zhang, R.; Wang, C. Y.; Yang, X. Q.; Xin, H. L. TEMImageNet training library and AtomSegNet deep-learning models for high-precision atom segmentation, localization, denoising, and deblurring of atomic-resolution images. Sci. Rep. 2021, 11, 5386.
DOI Google Scholar
[34]

Aguiar, J. A.; Gong, M. L.; Unocic, R. R.; Tasdizen, T.; Miller, B. D. Decoding crystallography from high-resolution electron imaging and diffraction datasets with deep learning. Sci. Adv. 2019, 5, eaaw1949.
DOI Google Scholar
[35]

Sang, X. H.; Xie, Y.; Lin, M. W.; Alhabeb, M.; Van Aken, K. L.; Gogotsi, Y.; Kent, P. R. C.; Xiao, K.; Unocic, R. R. Atomic defects in monolayer titanium carbide (Ti3C2Tx) MXene. ACS Nano 2016, 10, 9193–9200.
DOI Google Scholar
[36]

Trentino, A.; Madsen, J.; Mittelberger, A.; Mangler, C.; Susi, T.; Mustonen, K.; Kotakoski, J. Atomic-level structural engineering of graphene on a mesoscopic scale. Nano Lett. 2021, 21, 5179–5185.
DOI Google Scholar
[37]

Ziatdinov, M.; Dyck, O.; Maksov, A.; Li, X. F.; Sang, X. H.; Xiao, K.; Unocic, R. R.; Vasudevan, R.; Jesse, S.; Kalinin, S. V. Deep learning of atomically resolved scanning transmission electron microscopy images: Chemical identification and tracking local transformations. ACS Nano 2017, 11, 12742–12752.
DOI Google Scholar
[38]

Zhao, L.; Pan, Y. L.; Wang, S.; Zhang, L.; Islam, M. S. A hybrid crack detection approach for scanning electron microscope image using deep learning method. Scanning 2021, 2021, 5558668.
DOI Google Scholar
[39]

Mitchell, S.; Parés, F.; Faust Akl, D.; Collins, S. M.; Kepaptsoglou, D. M.; Ramasse, Q. M.; Garcia-Gasulla, D.; Pérez-Ramírez, J.; López, N. Automated image analysis for single-atom detection in catalytic materials by transmission electron microscopy. J. Am. Chem. Soc. 2022, 144, 8018–8029.
DOI Google Scholar
[40]

Qi, K.; Cui, X. Q.; Gu, L.; Yu, S. S.; Fan, X. F.; Luo, M. C.; Xu, S.; Li, N. B.; Zheng, L. R.; Zhang, Q. H. et al. Single-atom cobalt array bound to distorted 1T MoS2 with ensemble effect for hydrogen evolution catalysis. Nat. Commun. 2019, 10, 5231.
DOI Google Scholar
[41]

Addou, R.; Colombo, L.; Wallace, R. M. Surface defects on natural MoS2. ACS Appl. Mater. Interfaces 2015, 7, 11921–11929.
DOI Google Scholar
[42]

Rasool, H. I.; Ophus, C.; Zettl, A. Atomic defects in two dimensional materials. Adv. Mater. 2015, 27, 5771–5777.
DOI Google Scholar
[43]

Stobinski, L.; Lesiak, B.; Malolepszy, A.; Mazurkiewicz, M.; Mierzwa, B.; Zemek, J.; Jiricek, P.; Bieloshapka, I. Graphene oxide and reduced graphene oxide studied by the XRD, TEM and electron spectroscopy methods. J. Electron Spectros. Relat. Phenomena 2014, 195, 145–154.
DOI Google Scholar
[44]

Peng, W.; Han, J. H.; Lu, Y. R.; Luo, M.; Chan, T. S.; Peng, M.; Tan, Y. W. A general strategy for engineering single-metal sites on 3D porous N, P Co-doped Ti3C2Tx MXene. ACS Nano 2022, 16, 4116–4125.
DOI Google Scholar
[45]

Zhao, D.; Chen, Z.; Yang, W. J.; Liu, S. J.; Zhang, X.; Yu, Y.; Cheong, W. C.; Zheng, L. R.; Ren, F. Q.; Ying, G. B. et al. MXene (Ti3C2) vacancy-confined single-atom catalyst for efficient functionalization of CO2. J. Am. Chem. Soc. 2019, 141, 4086–4093.
DOI Google Scholar
[46]

Cheng, Y. W.; Dai, J. H.; Zhang, Y. M.; Song, Y. Transition metal modification and carbon vacancy promoted Cr2CO2 (MXenes): A new opportunity for a highly active catalyst for the hydrogen evolution reaction. J. Mater. Chem. A 2018, 6, 20956–20965.
DOI Google Scholar
[47]

Wan, Q.; Li, S. N.; Liu, J. B. First-principle study of Li-ion storage of functionalized Ti2C monolayer with vacancies. ACS Appl. Mater. Interfaces 2018, 10, 6369–6377.
DOI Google Scholar
[48]

Wu, H.; Guo, Z. L.; Zhou, J.; Sun, Z. M. Vacancy-mediated lithium adsorption and diffusion on MXene. Appl. Surf. Sci. 2019, 488, 578–585.
DOI Google Scholar
[49]

Hao, J. C.; Zhuang, Z. C.; Cao, K. C.; Gao, G. H.; Wang, C.; Lai, F. L.; Lu, S. L.; Ma, P. M.; Dong, W. F.; Liu, T. X. et al. Unraveling the electronegativity-dominated intermediate adsorption on high-entropy alloy electrocatalysts. Nat. Commun. 2022, 13, 2662.
DOI Google Scholar
[50]

Chu, C. H.; Huang, D. H.; Gupta, S.; Weon, S.; Niu, J. F.; Stavitski, E.; Muhich, C.; Kim, J. H. Neighboring Pd single atoms surpass isolated single atoms for selective hydrodehalogenation catalysis. Nat. Commun. 2021, 12, 5179.
DOI Google Scholar
[51]

Zhang, C. Y.; Liang, S. X.; Liu, W.; Eickemeyer, F. T.; Cai, X. B.; Zhou, K.; Bian, J. M.; Zhu, H. W.; Zhu, C.; Wang, N. et al. Ti1-graphene single-atom material for improved energy level alignment in perovskite solar cells. Nat. Energy 2021, 6, 1154–1163.
DOI Google Scholar
[52]

Li, Z.; Xiao, Y.; Chowdhury, P. R.; Wu, Z. W.; Ma, T.; Chen, J. Z.; Wan, G.; Kim, T. H.; Jing, D. P.; He, P. L. et al. Direct methane activation by atomically thin platinum nanolayers on two-dimensional metal carbides. Nat. Catal. 2021, 4, 882–891.
DOI Google Scholar
[53]

Bao, H. H.; Qiu, Y.; Peng, X. Y.; Wang, J. A.; Mi, Y. Y.; Zhao, S. Z.; Liu, X. J.; Liu, Y. F.; Cao, R.; Zhuo, L. C. et al. Isolated copper single sites for high-performance electroreduction of carbon monoxide to multicarbon products. Nat. Commun. 2021, 12, 238.
DOI Google Scholar
[54]

Madsen, J.; Liu, P.; Kling, J.; Wagner, J. B.; Hansen, T. W.; Winther, O.; Schiøtz, J. A deep learning approach to identify local structures in atomic-resolution transmission electron microscopy images. Adv. Theory Simul. 2018, 1, 1800037.
DOI Google Scholar
[55]

Seo, S. Y.; Yang, D. H.; Moon, G.; Okello, O. F. N.; Park, M. Y.; Lee, S. H.; Choi, S. Y.; Jo, M. H. Identification of point defects in atomically thin transition-metal dichalcogenide semiconductors as active dopants. Nano Lett. 2021, 21, 3341–3354.
DOI Google Scholar
[56]

Maksov, A.; Dyck, O.; Wang, K.; Xiao, K.; Geohegan, D. B.; Sumpter, B. G.; Vasudevan, R. K.; Jesse, S.; Kalinin, S. V.; Ziatdinov, M. Deep learning analysis of defect and phase evolution during electron beam-induced transformations in WS2. npj Comput. Mater. 2019, 5, 12.
DOI Google Scholar
[57]
PyTorch. https://pytorch.org/docs/stable/generated/torch.nn.BCELoss.
[58]
Kirkland, E. J. Advanced Computing in Electron Microscopy; 2nd ed. Springer: New York, 2010; pp 1–351.
File
12274_2023_6104_MOESM1_ESM.pdf (3.1 MB)
Publication history
Copyright
Acknowledgements

Publication history

Received: 11 July 2023
Revised: 14 August 2023
Accepted: 16 August 2023
Published: 23 September 2023
Issue date: April 2024

Copyright

© Tsinghua University Press 2023

Acknowledgements

Acknowledgements

This work was supported by the National Natural Science Foundation of China (Nos. 52105145 and 12274124), the Shanghai Pilot Program for Basic Research (No. 22TQ1400100-6), and the Fundamental Research Funds for the Central Universities. Additional support was provided by the Feringa Nobel Prize Scientist Joint Research Center of the East China University of Science and Technology.

Return