Geometrically frustrated antiferromagnetic order transition in two-dimensional van der Waals semiconductor MnIn₂Se₄

The geometrically frustrated antiferromagnetism, revealing non-mean-field ordering mechanisms through partial order, quantum fluctuations, and criticality, has long been a central focus in condensed matter physics for probing and manipulating novel phases of matter. However, the direct experimental observation of the phase transition associated with this frustrated antiferromagnetic order, termed “hidden order”, in two-dimensional systems remains a significant challenge. This is primarily due to the scarcity of suitable candidate materials and the limitations of conventional measurement techniques. Here, we investigate the magnetic properties of MnIn₂Se₄, a promising candidate for studying such transitions. Using a combination of polarized Raman spectroscopy, density functional theory (DFT), and Monte Carlo simulations calculations, we observe this geometrically frustrated antiferromagnetism order that persists from the spin-freezing temperature of 3.5 K up to 180 K. This transition is characterized by pronounced magneto-optical scattering phenomena, manifested by two distinct broad spectral regions of continuous magnetic states, with inflection points emerging at 180 K. Additionally, we detect temperature-dependent quasi-elastic scattering (QES) and Fano resonance, further confirming the magnetic order change. The presence of geometrically frustrated interactions in this material provides a natural explanation for the observed extended temperature range of the partial order phase. These findings establish MnIn₂Se₄ as a promising material platform for investigating frustration-driven phase transitions and the underlying mechanisms of high-temperature superconductivity and quantum bits. The insights gained into the interplay between thermal fluctuation effects on magnetic interactions may open new avenues for designing functional semiconductor devices with tunable magnetic and topological characteristics.