In this article, the modulation instability (MI) of open Bose–Bose mixtures with helicoidal spin–orbit coupling (SOC) was studied. Unlike previous spin–orbit (SO)-coupled Bose–Einstein condensate system with helicoidal gauge potential, the purpose of this article to study the input of the helicoidal SOC in the emergence of MI in open Bose–Bose mixtures by taking into account the Lee–Huang–Yang corrections to the coupled Gross–Pitaevskii equations. We present the detailed analyses of system parameters on the characteristics of MI and analytically conclude the parameter conditions for MI occurrence. Our results provide a potential way to manipulate the MI in the helicoidal SO-coupled open Bose–Bose mixtures.
