Light detection and ranging (LIDAR) sensors using a polarization-diverse receiver are able to capture polarimetric information about the target under measurement. We demonstrate this capability using a silicon photonic receiver architecture that enables this on a shot-by-shot basis, enabling polarization analysis nearly instantaneously in the point cloud, and then use this data to train a material classification neural network. Using this classifier, we show an accuracy of 85.4% for classifying plastic, wood, concrete, and coated aluminum.