This study is directed towards developing a fast, non-destructive, and easy-to-use handheld multimode spectroscopic system for fish quality assessment. We apply data fusion of visible near infra-red (VIS-NIR) and short wave infra-red (SWIR) reflectance and fluorescence (FL) spectroscopy data features to classify fish from fresh to spoiled condition. Farmed Atlantic and wild coho and chinook salmon and sablefish fillets were measured. Three hundred measurement points on each of four fillets were taken every two days over 14 days for a total of 8400 measurements for each spectral mode. Multiple machine learning techniques including principal component analysis, self-organized maps, linear and quadratic discriminant analyses, k-nearest neighbors, random forest, support vector machine, and linear regression, as well as ensemble and majority voting methods, were used to explore spectroscopy data measured on fillets and to train classification models to predict freshness. Our results show that multi-mode spectroscopy achieves 95% accuracy, improving the accuracies of the FL, VIS-NIR and SWIR single-mode spectroscopies by 26, 10 and 9%, respectively. We conclude that multi-mode spectroscopy and data fusion analysis has the potential to accurately assess freshness and predict shelf life for fish fillets and recommend this study be expanded to a larger number of species in the future.