Carbon molecular sieve (CMS) membranes have been developed to replace or support energy-intensive cryogenic distillation for olefin/paraffin separation. Olefin and paraffin have similar molecular properties, but can be separated effectively by a CMS membrane with a rigid, slit-like pore structure. A variety of polymer precursors can give rise to different outcomes in terms of the structure and performance of CMS membranes. Herein, for olefin/paraffin separation, the CMS membranes derived from a number of polymer precursors (such as polyimides, phenolic resin, and polymers of intrinsic microporosity, PIM) are introduced, and olefin/paraffin separation properties of those membranes are summarized. The effects from incorporation of inorganic materials into polymer precursors and from a pyrolysis process on the properties of CMS membranes are also reviewed. Finally, the prospects and future directions of CMS membranes for olefin/paraffin separation and aging issues are discussed.