Ragulator is a pentamer composed of p18, MP1, p14, C7orf59, and hepatitis B virus X‐interacting protein (HBXIP; LAMTOR 1—5) which acts as a lysosomal scaffold of the Rag GTPases in the amino acid sensitive branch of TORC1 signaling. Here, we present the crystal structure of human HBXIP‐C7orf59 dimer (LAMTOR 4/5) at 2.9 Å and identify a phosphorylation site on C7orf59 which modulates its interaction with p18. Additionally, we demonstrate the requirement of HBXIP‐C7orf59 to stabilize p18 and allow further binding of MP1‐p14. The structure of the dimer revealed an unfolded N terminus in C7orf59 (residues 1–15) which was shown to be essential for p18 binding. Full‐length p18 does not interact stably with MP1‐p14 in the absence of HBXIP‐C7orf59, but deletion of p18 residues 108–161 rescues MP1‐p14 binding. C7orf59 was phosphorylated by protein kinase A (PKA) in vitro and mutation of the conserved Ser67 residue to aspartate prevented phosphorylation and negatively affected the C7orf59 interaction with p18 both in cell culture and in vitro. C7orf59 Ser67 was phosphorylated in human embryonic kidney 293T cells. PKA activation with forskolin induced dissociation of p18 from C7orf59, which was prevented by the PKA inhibitor H‐89. Our results highlight the essential role of HBXIP‐C7orf59 dimer as a nucleator of pentameric Ragulator and support a sequential model of Ragulator assembly in which HBXIP‐C7orf59 binds and stabilizes p18 which allows subsequent binding of MP1‐p14.