In recent years, the Swept-Source Phase Microscope (SS-PM) has gained more attention due to its greater robustness to sample motion and lower signal decay with depth. However, the mechanical wavelength tuning of the swept source creates small variations in the wavenumber sampling of spectra that introduce serious phase noise. We present a software post-processing method to eliminate phase noise in SS-PM. This method does not require high-quality swept light sources or high-precision synchronization devices and achieves ~72 pm displacement sensitivity using a conventional SS-PM system. We compare the performance of this method with traditional software-based methods by measuring phase fluctuations. The phase fluctuations in the traditional software-based method are five times those of the proposed method, which means the proposed method has better sensitivity. Using this method, we reconstructed phase images of air wedges and resolution plates to demonstrate the SS-PM’s potential for high-sensitivity surface profiling measurement. Finally, we discuss the advantages of SS-PM over traditional Spectral-Domain PM techniques.