Using localized surface plasmon resonance (LSPR) as an optical probe we demonstrate the presence of free carriers in phosphorus doped silicon nanocrystals (SiNCs) embedded in a silica matrix. In small SiNCs, with radius ranging from 2.6 to 5.5 nm, the infrared spectroscopy study coupled to numerical simulations allows us to determine the number of electrically active phosphorus atoms with a precision of a few atoms. We demonstrate that LSP resonances can be supported with only about 10 free electrons per nanocrystal, confirming theoretical predictions and probing the limit of the collective nature of plasmons. We reveal the appearance of an avoided crossing behavior linked to the hybridization between the localized surface plasmon in the doped nanocrystals and the silica matrix phonon modes. Finally, a careful analysis of the scattering time dependence versus carrier density in the small size regime allows us to detect the appearance of a new scattering process at high dopant concentration, which can be explained by P clustering inside the SiNCs.