We investigate the magnetic field-dependent fluorescence lifetime of microdiamond powder containing a high density of nitrogen-vacancy centers. This constitutes a non-intensity quantity for robust, all-optical magnetic field sensing. We propose a fiber-based setup in which the excitation intensity is modulated in a frequency range up to 100MHz. The change in magnitude and phase of the fluorescence relative to B=0 is recorded where the phase shows a maximum in magnetic contrast of 5.8∘ at 13MHz. A lock-in amplifier-based setup utilizing the change in phase at this frequency shows a 100 times higher immunity to fluctuations in the optical path compared to the intensity-based approach. A noise floor of 20μT/Hz and a shot-noise-limited sensitivity of 0.95μT/Hz were determined.