In vertebrate retinal development, the axonal terminals of retinal neurons make synaptic contacts within narrow fixed regions, and these locations are maintained thereafter. However, the mechanisms and biological logic of the organization of these fixed synapse locations are poorly understood. We show here that a membrane scaffold protein, 4.1G, is highly expressed in retinal photoreceptors and is essential for the arrangement of their correct synapse location. The 4.1G-deficient retina exhibits mislocalization of photoreceptor terminals, although their synaptic connections are normally formed. The 4.1G protein binds to the AP3B2 protein, which is involved in neuronal membrane trafficking, and promotes neurite extension in an AP3B2-dependent manner. 4.1G mutant mice showed visual acuity impairments in an optokinetic response, suggesting that correct synapse location is required for normal visual function. Taken together, the data in this study provide insight into the mechanism and importance of proper synapse location in neural circuit formation.