X-ray fluorescence spectroscopy analysis is a major method for quantitative analysis of metal elements in coal. The accuracy of background subtraction directly affects mineral metal sorting. As traditional background subtraction methods feature low background fit and large diffraction peak area errors, this study proposes an optimized method of complex wavelet transform with inverse derivative fitting for background subtraction. We obtained a simulated background spectral line through reversed peak searching and separated the overlapping peaks and valleys using Largrange's interpolation to push the simulated background close to the actual background. We then decompose and reconstruct the simulated background spectral line through complex wavelet transform to accomplish background subtraction of XRF analysis. We used the optimized method of complex wavelet with inverse derivative fitting for background subtraction of the simulated XRF spectrum and the real XRF spectrum, and compared with Continuous Wavelet and Dual-tree Complex Wavelet. The experimental results indicate that the optimized method outperforms the traditional methods in background subtraction accuracy, with less than 1 % of diffraction peak area error and less than 0.1 % of background area.