The charge carrier dynamics of improved InP-based InAs/AlGaInAs quantum dot (QD) semiconductor optical amplifiers are examined employing the multi-wavelength ultrafast pump-probe measurement technique. The transient transmission response of the continuous wave probe shows interesting dynamical processes during the initial 2-3 ps after the pump pulse, when carriers originating from two photon absorption contribute the least to the recovery. The effects of optical excitations and electrical bias levels on the recovery dynamics of the gain in energetically different QDs are quantified and discussed. The experimental observations are validated qualitatively using a comprehensive finite-difference time-domain model by recording the time evolution of the charge carriers in the QDs ensemble following the pulse.