The fabric defect models based on deep learning often demand numerous training samples to achieve high accuracy. However, obtaining a complete dataset containing all possible fabric textures and defects is a big challenge due to the sophisticated and various fabric textures and defect forms. This study created a two-stage deep pix2pixGAN network called Dual Deep pix2pixGAN Network (DPGAN) to address the above problem. The defect generation model was trained based on the DPGAN network to automatically “transfer” defects from defected fabric images to clean, defect-free fabric images, thus strengthening the training data. To evaluate the effectiveness of the defect generation model, extensive comparative experiments were conducted to assess the performance of the fabric defect detection before and after data enhancement. The results indicate that the detection accuracy was improved regarding the belt_yarn, hole, and stain defect.