The growing pollen tube has become one of the most fascinating model cell systems for investigations into cell polarity and polar cell growth in plants. Rapidly growing pollen tubes achieve tip-focused cell expansion by vigorous anterograde exocytosis, through which various newly synthesized macromolecules are directionally transported and deposited at the cell apex. Meanwhile, active retrograde endocytosis counter balances the exocytosis at the tip which is believed to recycle the excessive exocytic components for multiple rounds of secretion. Therefore, apical exocytosis and endocytosis are the frontline cellular processes which drive the polar growth of pollen tubes, although they represent opposite vesicular trafficking events with distinct underpinning mechanisms. Nevertheless, the molecular basis governing the spatiotemporal crosstalk and counterbalance of exocytosis and endocytosis during pollen tube polarization and growth remains elusive. Here we discuss recent insight into exocytosis and endocytosis in sculpturing high rates of polarized pollen tube growth. In addition, we especially introduce the novel integration of mathematical modeling in uncovering the mysteries of cell polarity and polar cell growth.