Machine-learned phase diagrams of generalized Kitaev honeycomb magnets

Abstract

We use a recently developed interpretable and unsupervised machine-learning method, the tensorial kernel support vector machine, to investigate the low-temperature classical phase diagram of a generalized Heisenberg-Kitaev-Γ (J-K-Γ) model on a honeycomb lattice. Aside from reproducing phases reported by previous quantum and classical studies, our machine finds a hitherto missed nested zigzag-stripy order and establishes the robustness of a recently identified modulated S_{3}×Z_{3} phase, which emerges through the competition between the Kitaev and Γ spin liquids, against Heisenberg interactions. The results imply that, in the restricted parameter space spanned by the three primary exchange interactions—J, K, and Γ, the representative Kitaev material α-RuCl_{3} lies close to the boundaries of several phases, including a simple ferromagnet, the unconventional S_{3}×Z_{3}, and nested zigzag-stripy magnets. A zigzag order is stabilized by a finite Γ^{′} and/or J_{3} term, whereas the four magnetic orders may compete in particular if Γ^{′} is antiferromagnetic.