Septuple XBi2Te4 (X=Ge, Sn, Pb) intercalated MnBi2Te4 for realizing interlayer ferromagnetism and quantum anomalous hall effect

The magnetic topological insulators (MTIs), empowering magnetism with topological band structures, have attracted tremendous interests in condensed matter physics and materials science. Different from the conventional three-dimensional topological insulators (3DTIs), MTIs exhibit a broken time-inversion symmetry, leading to the emergence of various exotic topological quantum phenomena, including the quantum anomalous Hall effect (QAHE) that is characterized by dissipationless chiral edge states^1,2,3; topological axion states which exhibit quantized magnetoelectric effects^4,5,6; magnetic Weyl states that host peculiar quasiparticles known as Weyl fermions^7,8; and Majorana states in topological superconductors which obey non-Abelian statistics⁹. Typically, doping transition metals emerges as a highly effective method for achieving MTIs. A prominent example is Cr-doped (Bi,Sb)₂Te₃, which enabled the first experimental observation of QAHE³. However, this approach introduces randomness into the system, resulting in narrow topological band gaps and low Curie temperatures^{10,11,12,13,14}. Consequently, QAHE in such systems is restricted to ultra-low temperatures, limiting their practical applicability.

Intrinsic MTIs present a promising alternative, as they naturally combine magnetism with topological band structures, eliminating the need for random doping. MnBi₂Te₄ (MBT)^{15,16,17,18,19,20,21,22,23,24,25,26,27,28,29}, an intrinsic and quasi-two-dimensional (2D) MTI synthesized by intercalating a magnetic MnTe layer into one Bi₂Te₃ (BT) layer for one MBT layer consisting of Te-Bi-Te-Mn-Te-Bi-Te septuple layer (SL) blocks stacked along the [001] direction, has emerged as a robust platform for exploring topological quantum states. The MBT bulk intrinsically exhibits intralayer ferromagnetic (FM) coupling but favors interlayer antiferromagnetic (AFM) coupling^30,31. This layer-dependent magnetic structure allows QAHE to occur only in odd MBT multilayers (MBTs) and at low temperatures (~ 1.4 K)²³. Aligning all layers into an FM configuration via an external magnetic field raises the quantization temperature to 45 K³², but achieving intrinsic interlayer FM coupling remains a key challenge for realizing high-temperature QAHE.

While generating FM spin-polarization in an AFM system can bring many advantages^33,34, efforts to address this challenge have explored various strategies, including constructing heterostructures with materials like VBi₂Te₄ and CrI₃^35,36 or alloying MBTs with hole-doping elements such as V^37,38. Recent studies have investigated the intercalation of Bi₂Te₃ (BT) layers into MBTs, forming MBT/(BT)_n heterostructures^{39,40,41,42,43,44,45,46,47,48,49,50,51,52}. Although these structures exhibit a transition from AFM to FM coupling when n ≥ 3⁴⁷, the interlayer exchange is weakened as more BT layers are added, reducing the magnetic transition temperature and limiting their effectiveness.

In this study, we propose a new approach to achieve interlayer FM coupling in a MBT bilayer by intercalating septuple-layer topological insulator XBi₂Te₄ (X=Ge, Sn, Pb) [labeled as GBT, SBT, and PBT, respectively]. Using first-principles calculations, we demonstrate that the p_z orbital of the X atom mediates the interaction between the interlayer Mn atoms, creating a channel for FM coupling. Monte Carlo simulations predict a magnetic transition temperature (T_c) of 38 K for the MBT/PBT/MBT trilayer. Electronic band structure and topological analyses confirm that QAHE is preserved in all these heterostructures, with the MBT/PBT/MBT trilayer exhibiting a topological band gap as large as 72 meV, significantly larger than the pure MBT bilayer. Additionally, a continuum model provides insights into the mechanism behind these nontrivial topological states.

Crystal structures

It has been demonstrated that the MBT bilayer accommodates the van der Waals (vdW) interlayer coupling, as illustrated in Fig. 1a. The intralayer Mn atoms are ferromagnetically coupled, which is well explained by the Goodenough-Kanamori 90° rule, while the interlayer Mn atoms are antiferromagnetically coupled (see Fig. 1a), which is attributed to the super-exchange. The exchange coupling between interlayer Mn atoms is mediated by the p orbitals of Bi and Te, and all five d orbitals of Mn²⁺ ions are fully-filled in one layer with the same spin orientation (Hund’s rule). This unique magnetic behavior results in the fact that the QAHE can only be realized in odd-layered MBTs.

a, b Structural configurations of the pure MBT bilayer with interlayer antiferromagnetic (AFM) coupling and MBT/PBT/MBT trilayer with interlayer ferromagnetic (FM) coupling. Here, “d” in a represents the interlayer distance between two MBT layers. c Schematic diagrams of realizing FM coupling in MBT/PBT/MBT.

Full size image

Here, we propose to realize interlayer FM coupling by constructing various MBT/XBT/MBT trilayers, as depicted in Fig. 1b. It should be noted that the three p orbitals of X in the XBT layer are not fully occupied, thus the two interlayer Mn atoms can be mediated by the p orbitals of X (see Fig. 1c). Consequently, the five d orbitals of Mn²⁺ ions are non-fully filled in one layer with the same spin orientation. Namely, the XBT intercalation indirectly allows the interlayer Mn atoms to open a channel within the same spin, enabling the interlayer FM coupling, as illustrated in Fig. 1c. To confirm this point, we first calculated the projected density of states of MBT/BT/MBT and MBT/PBT/MBT trilayers, as shown in Fig. 2. As mentioned above, the weak AFM coupling between two adjacent MBT layers is actually the exchange coupling between two interlayer Mn atoms (see Fig. 2a). The reason is that all five d orbitals of Mn²⁺ ions are fully-filled in one layer with the same spin orientation. Here, the mechanism for the interlayer AFM coupling in MBT/BT/MBT is the same as that of pure MBT bilayer¹⁹, as depicted in Fig. 2c. For MBT/PBT/MBT, the p_z orbital of Pb in PBT is nearly unoccupied (see Fig. 2b). The two interlayer Mn atoms can be mediated by p_z orbital of Pb, leading to a channel allowing the interlayer FM coupling, as illustrated in Fig. 2c. We emphasize that such unoccupied p_z orbital of Pb plays an important role for realizing the interlayer FM coupling, which is essentially different from that of MBT/BT/MBT, hosting the competitive advantage of realizing QAHE.

a, b PDOS of MBT/BT/MBT and MBT/PBT/MBT. c Illustration of p–d hopping in MBT/BT/MBT and MBT/XBT/MBT.

Full size image

Then, we check the stacking style of MBT/XBT/MBT trilayers. Similar to MBTs, our calculations show that all MBT/XBT/MBT prefer the ABC stacking style, as shown in Fig. 1b. Here, the optimized lattice constants of MBT/GBT/MBT, MBT/SBT/MBT, and MBT/PBT/MBT trilayers are 4.34, 4.36, and 4.38 Å, respectively. As the stability of XBT-intercalated MBTs is an important issue, we have also calculated the phonon spectrum of these MBT/XBT/MBT trilayers, as given in Supplementary Fig. 1, showing that there has no imaginary phonon mode for each system.

In addition, the interlayer binding energies (E_b) of MBT/GBT/MBT, MBT/SBT/MBT, and MBT/PBT/MBT trilayers are calculated to be 0.86, 0.93, and 0.97 eV, respectively. These binding energies are more than twice that of pure MBT trilayer with E_b = 0.41 eV. Here, E_b is defined as E_b = E_2SL-MBT + E_XBT – E_MBT/XBT/MBT, with E_2L-MBT, E_XBT, and E_MBT/XBT/MBT respectively denoting the total energies of MBT bilayer, XBT monolayer, and MBT/XBT/MBT trilayer. All those cross checks demonstrate the stability of XBT-intercalated MBT, suggesting that MBT/XBT/MBT trilayers may be experimentally fabricated.

Magnetism and curie temperature

Next, we have calculated the energy difference ΔE_AFM-FM of MBT/XBT/MBT with the AFM and FM configurations, where ΔE_AFM-FM is defined as ΔE_AFM-FM = E_AFM – E_FM, with E_AFM and E_FM denoting the total energies of AFM and FM orders, respectively. Our calculations show that ΔE_AFM-FM of pure MBT bilayer is −0.63 meV/Mn, which is consistent with previous calculations¹⁸. In contrast, ΔE_AFM-FM of MBT/PBT/MBT trilayer is calculated to be 1.33 meV/Mn, indicating the two MBT layers are ferromagnetically coupled, which is the same case for MBT/GBT/MBT and MBT/SBT/MBT trilayers. Thus, the MBT bilayer undergoes a magnetic phase transition from AFM to FM coupling via XBT intercalations. Specifically, we have confirmed that the interlayer FM coupling for each system is not sensitive to the choices of U_eff varying from 1.0 to 5.0 eV, and also not sensitive to the vdW corrections, as shown in the Supplementary Fig. 2.

Our calculations have also shown that the magnetic moment of each Mn atom is ~4.54 μ_B for pure MBT and XBT-intercalated MBTs, indicating that the XBT-intercalations nearly do not affect the d orbital occupancy of Mn. Namely, the topological states of MBT bilayer may be well preserved in MBT/XBT/MBT trilayers. On the other hand, the magnetic anisotropy energy (MAE) is a key issue for magnetic materials, thus we have also calculated the MAE of MBT/XBT/MBT, where MAE is defined as MAE = E_||− E_⊥, with E_|| and E_⊥ denoting the total energies when the magnetic moment is parallel and perpendicular to the plane of the 2D material, respectively. The calculated MAE values are respectively 0.47, 0.62, and 0.96 meV for MBT/GBT/MBT, MBT/SBT/MBT, and MBT/PBT/MBT, showing that the easy magnetization axis of each system prefers the out-of-plane.

Considering that the Curie temperature (T_c) is also an important factor for 2D FM materials, we have employed Monte Carlo (MC) simulation⁵³ to estimate the T_c of these MBT/XBT/MBT trilayers. Here, the effective Hamiltonian is described as (H=-{J}_{1}mathop{sum}nolimits_{ < i,j > }vec{{S}_{i}}{{cdot }}vec{{S}_{j}}-{J}_{2}mathop{sum}nolimits_{ll i,jgg }vec{{S}_{i}}{{cdot }}vec{{S}_{j}}-{J}_{3}mathop{sum}nolimits_{ < ll i,jgg > }vec{{S}_{i}}{{cdot }}vec{{S}_{j}}), where J₁, J₂, and J₃ denote the respective exchange coupling between the nearest ( < i,j > ), next-nearest (<<i,j > >), and third-nearest (<<<i,j > >>) neighboring Mn spins, and S_i is the magnetic moment of site i. Here J < 0 represents the AFM coupling between Mn spins, otherwise, it represents the FM coupling. The detailed results are given in Fig. 3a. A 2 × 2 supercell is used to calculate the total energies of four magnetic configurations, corresponding to one FM and three AFM [namely, Néel antiferromagnetic (N-AFM), stripy antiferromagnetic (S-AFM), and zigzag antiferromagnetic (Z-AFM)] states^54,55. In our calculations, we mainly considered the nearest, next-nearest, and third-nearest magnetic couplings, denoted as J₁, J₂, and J₃, respectively. By examining the total energies of the FM, N-AFM, S-AFM, and Z-AFM spin configurations, the three magnetic coupling constants can be extracted by the following equations:

where S_A and S_B are the spin operators on site A and B in the honeycomb lattice, respectively. Our calculations show that parameter J₁ is much larger than J₂ and J₃ for all systems, thus J₂ and J₃ are neglected in our MC calculations. The calculated J₁ values are respectively 0.65, 0.73, and 1.34 meV for MBT/GBT/MBT, MBT/SBT/MBT, MBT/PBT/MBT, demonstrating that the interlayer FM order is the ground state for each system. The estimated T_c of MBT/PBT/MBT trilayer is ~38 K (see Fig. 3b), where the MAE, intralayer, and interlayer couplings are considered in the MC simulations. For comparison, we have also calculated the Néel temperature of pure MBT with T_N = 23 K, which is very close to that of ref. ¹⁹ with T_N = 25.4 K, noting again here that the intercalated system shows the FM order while the pure one has the AFM order.

a For pure MBT bilayer and MBT/XBT/MBT trilayers, the Mn atoms between two nearest MBT layers form honeycomb lattice, thus the effective exchange coupling parameters are obtained by calculating the total energies of the FM, Néel-AFM (N-AFM), stripy-AFM (S-AFM), and zigzag-AFM (Z-AFM) spin configurations. b Normalized magnetic moment (blue dotted line) and specific heat C_ν (red dotted line) of the MBT/PBT/MBT as a function of temperature obtained from the Monte Carlo simulations.

Full size image

Electronic band structures and topological states

Now, we turn to investigate the electronic properties of MBT/XBT/MBT trilayers. Figure 4a–c give the respective electronic band structures of MBT/GBT/MBT, MBT/SBT/MBT, MBT/PBT/MBT without considering spin-orbit coupling (SOC). It is found that the MBT/GBT/MBT, MBT/SBT/MBT, and MBT/PBT/MBT host the band gaps of 0.56, 0.49, and 0.51 eV, which are smaller than that of pure MBT bilayer with a 0.68 eV band gap. After SOC is applied, the band gaps are decreased to 26.4, 38.7, and 73.5 meV for the MBT/GBT/MBT, MBT/SBT/MBT, and MBT/PBT/MBT, respectively (see Fig. 4d–f). It has been demonstrated that FM MBT bilayer is a QAH insulator, thus the topological states may be well preserved for all MBT/XBT/MBT trilayers. We first analyzed the topological properties of MBT/XBT/MBT through the projected bands. Notably, it is found that there exists the band inversion between Bi p_z and Te p_z orbitals for all systems (see Fig. 4g–i), suggesting that the XBT intercalations do not destroy the topological states of MBT bilayer.

a–c Electronic band structures of MBT/GBT/MBT, MBT/SBT/MBT, and MBT/PBT/MBT without SOC, respectively. Here, red and blue lines denote spin-up and spin-down channels. d–f The same as a–c with considering SOC. g–i Corresponding projected bands d–f near the Fermi levels with SOC. The Fermi level is set to be zero for each system.

Full size image

To confirm the above analyses, we have identified the topological properties of MBT/XBT/MBT trilayers by calculating the Berry curvature distribution (varOmega (k)) along the high symmetry paths, given by

where k denotes the electron wave vector, f_n is the Fermi-Dirac distribution function, E_n represents the eigenvalue of Bloch function (|{psi }_{{rm{nk}}}rangle), v_x and v_y are the x and y components of the electron velocity operator, and the summation is over all the occupied states. By integrating Ω(k) over the first Brillouin zone (BZ), the Chern number (C) is obtained by (C=frac{1}{2pi }{sum }_{n}{int }_{BZ}{d}^{2}k{Omega }_{n}(k)). Here, (|{psi }_{{rm{nk}}}rangle) is obtained from the maximally localized Wannier functions constructed via non-self-consistent calculations with a 12 × 12 × 1 k-point grid.

Specifically, we take MBT/PBT/MBT as an example to illustrate the topological properties of MBT/XBT/MBT trilayers. From Fig. 5a, we see that the band structures of MBT/PBT/MBT obtained from the Wannier function are nearly the same as that of calculated from the DFT method. The one-dimensional Berry curvature distribution along the high-symmetry paths is also given, indicated by the orange solid lines in Fig. 5a. We see that nonzero Berry curvatures are distributed in the global band gap near the Γ point (see Fig. 5a). By integrating these Berry curvatures over the first BZ, nonzero Chern number of C = 1 is obtained, and the quantized Hall plateau with e²/h locates near the Fermi level, as displayed in Fig. 5b. Most importantly, one notices that a chiral edge state appears along the edge (see Fig. 5c). Similarly, it is found that topological states are well preserved for MBT/GBT/MBT and MBT/SBT/MBT trilayers. The conclusion is not sensitive to the choices of U_eff values and vdW corrections, as shown in Supplementary Fig. 3. To further confirm the reliability of our DFT calculations, the hybrid functional HSE06 method⁵⁶ is employed to calculate the nontrivial band gaps of all systems. The calculated band gaps are 32.4, 47.6, and 90.5 meV for MBT/GBT/MBT, MBT/SBT/MBT, MBT/PBT/MBT, which are slightly larger those obtained from the PBE + U calculations. Meanwhile, the topological properties are well preserved. It should be noted again that although the structure of MBT/XBT/MBT is rather close to that of pure MBT trilayer and both of them host the similar transition temperatures and band gaps, there exists an essential difference. Namely, MBT/XBT/MBT trilayers are ferromagnetically coupled, while pure MBT trilayers are antiferromagnetically coupled, thus the latter is not beneficial for exploration of QAHE.

a Band structures obtained from the DFT calculations (black lines) and the Wannier package (blue lines) of MBT/PBT/MBT, where the one-dimensional Berry curvature distribution along the high-symmetry paths is also given (orange lines). b The corresponding Hall conductance as a function of Fermi energy. c Calculated edge state, where the red and blue colors represent the contributions of the edge states.

Full size image

Effective low-energy model

The mechanism for the nontrivial topological states of MBT/XBT/MBT trilayers is then revealed by adopting the effective low-energy model. Here, the model is described by the massive Dirac surface states of 3DTIs with considering Zeeman term. The generic Hamiltonian of MTI thin-film with the basis of (|{t}_{uparrow }rangle),(|{t}_{downarrow }rangle),(|{b}_{uparrow }rangle), and (|{b}_{downarrow }rangle) is written as⁵⁷

where t (b) represents the top (bottom) surface state, ↑ ( ↓ ) denotes spin-up (spin-down) channel, and v_F is the Fermi velocity. σ_i and τ_i (i = x, y, z) are Pauli matrices acting on spin and layer, and (m(k)={m}_{0}+{m}_{1}({k}_{x}^{2}+{k}_{y}^{2})) is the hybridization between the top and bottom surface states. The last two terms describe the Zeeman field splitting of top and bottom surface states, respectively. By diagonalizing the formula (7), we can obtain the electronic band dispersion of MBTs, namely, ({E}_{pm }^{2}({k}_{x},{k}_{y})={v}_{F}^{2}({k}_{x}^{2}+{k}_{y}^{2})+{(sqrt{m{(k)}^{2}+{g}_{a}^{2}}pm {g}_{f}^{2})}^{2}). The Chern number C only changes at the gap closing of (sqrt{{m}_{0}^{2}+{g}_{a}^{2}}=|{g}_{f}|). When (sqrt{{m}_{0}^{2}+{g}_{a}^{2}} ,>, |{g}_{f}|), the system hosts topologically trivial state with C = 0; while for (sqrt{{m}_{0}^{2}+{g}_{a}^{2}} < |{g}_{f}|), the system is QAH insulator with C = ({g}_{f}/|{g}_{f}|). The systematic parameters g_a,f can be obtained from the formula g_a,f = (g_t (mp) g_b)/2, where ({g}_{i}=sum _{j}mathrm{sgn}({s}_{j}^{z}){lambda }^{ij}), (i = t, b). Here, j is the Mn layer index, ({s}_{j}^{z}) is the z component of Mn local spin in layer j, and λ^ij is the effective exchange parameter between local moments in layer j and top/bottom surface, respectively.

As each MBT/XBT/MBT trilayer has two MBT layers, namely, top- and bottom-layer MBT, g_a,f can be simplified as: g_f = 0 and g_a = λ₁ − λ₂ for AFM system; while g_a = 0 and g_f = λ₁ + λ₂ for FM system. Specifically, we notice that (sqrt{{m}_{0}^{2}+{g}_{a}^{2}}) is always larger than 0, namely, the Chern numbers of AFM MBT bilayer and MBT/XBT/MBT trilayers are always 0. For FM MBT/XBT/MBT, whose topological property is simply determined by the hybridization gap m₀ and magnetic exchange gap λ₁ + λ₂. If m₀ < λ₁ + λ₂, the system is a QAH insulator, otherwise, it is a topologically trivial insulator. We can obtain the parameters of m₀, λ₁, and λ₂ by solving the following formulas

As illustrated in Fig. 6a, b, E_g and E_g1/g2 denote the band gaps with AFM and FM orders, which can be extracted from the electronic band structures of AFM and FM systems. The obtained parameters of pure MBT bilayer are m₀ = 30.1 meV, λ₁ = 34.7 meV, and λ₂ = 26.2 meV. Obviously, we see that the hybridization gap m₀ is smaller than the exchange gap λ₁ + λ₂, thus FM MBT bilayer is a Chern insulator. When GBT, SBT, and PBT are intercalated into MBT bilayers, the magnetic exchange gap λ₁ + λ₂ for each system is also larger than the hybridization gap m₀ (see Fig. 6c), resulting in QAH characters being well preserved for all MBT/XBT/MBT trilayers. Specifically, it is found that m₀ and λ₂ barely changes while λ₁ is gradually increased.

a, b Band structure illustrations of AFM and FM MBT/XBT/MBT around the Γ point. Here, E_g is the band gap of AFM system, E_g1 and E_g2 are the band gaps of FM system with different spin channels. c Extracted parameters of m₀, λ₁, and λ₂ for MBT/XBT/MBT.

Full size image

As the MBT/XBT/MBT trilayers are the ideal candidates to realize intrinsic QAHE, we discuss how to fabricate these systems. One possible method is to grow the single crystal of XBT-intercalated MBTs (like BT-intercalated MBT^{41,42,43,44,45,46,47,48,49}) by using self-flux⁴¹. Then, MBT/XBT/MBT thin films can be obtained by using an Al₂O₃-assisted exfoliation technique²³. Another typical method is to fabricate the MBT and XBT monolayers, and then sandwich one XBT layer between two MBT layers.

At last, we have investigated the electronic and topological properties of PBT-intercalated MBT with different terminations. Here, three types of structural configurations are mainly considered, namely, MBT/MBT/PBT, MBT/MBT/PBT/PBT, and PBT/MBT/MBT/PBT heterostructures, as shown in Supplementary Fig. 4 and 5. Their corresponding electronic band structures are displayed in Supplementary Fig. 6. It is obviously seen that the electronic structures are very sensitive to the surface termination. For MBT/MBT/PBT, the topologically nontrivial band gap is remarkably decreased to 8.5 meV, while there has no global band gap for MBT/MBT/PBT/PBT and PBT/MBT/MBT/PBT. All those demonstrate that intercalating XBT between two MBT layers is more effective to explore large-gap QAHE.

In summary, by employing first-principles calculations, we have systematically investigated the structural, magnetic, electronic, and topological properties of MBT/XBT/MBT trilayers. It is found that the interlayer AFM to FM phase transition can be realized for all MBT/XBT/MBT trilayers, which is attributed to that the p_z orbital of the X atom mediates interactions between interlayer Mn atoms, enabling FM coupling. Monte Carlo simulations predict a magnetic transition temperature T_c of 38 K for the MBT/PBT/MBT trilayer. Electronic band structure and topological analyses demonstrate that the QAH states are well preserved for all MBT/XBT/MBT, with MBT/PBT/MBT harboring the topological band gap of 72 meV. The mechanism for the nontrivial topological states of MBT/XBT/MBT is also revealed by employing continuum model analysis. Our work establishes a practical pathway to achieving interlayer FM coupling in MBTs, offering a platform for realizing high-temperature QAHE and advancing the development of magnetic topological insulators for quantum and spintronic applications.

DFT calculations

The structural, electronic, magnetic, and topological properties of pure MBT bilayer and MBT/XBT/MBT trilayers were calculated by using the projector augmented wave (PAW)^58,59,60 formalism in the Vienna ab initio simulation package (VASP)^61,62. The Perdew-Burke-Ernzerhof (PBE) approximation was used to describe the exchange and correlation functional⁶³. To better describe the 3 d electrons of Mn, the GGA + U method⁶⁴ was employed, a moderate effective Hubbard U of U_eff = 4.0 eV was adopted, where the on-site U and exchange interaction J parameters were set to 5.0 and 1.0 eV, respectively. A vacuum space larger than 20 Å was used to avoid the interaction between two adjacent slabs. The plane-wave cutoff energy was set to 500 eV and all the atoms in the supercell were relaxed until the Hellmann–Feynman force on each atom was smaller than 0.01 eV/Å. The gamma-centered Monkhorst–Pack k-point mesh of 18 × 18 × 1 was adopted for structural optimization. The lattice constants of MBT/XBT/MBT trilayers were fully optimized. The vdW correction with the Grimme (DFT-D3) method⁶⁵ was included. The Berry curvature, Chern number, Hall conductance, and edge states were calculated to identify the topological properties using the Wannier90 and Wannier-Tools packages^66,67.