A warning on separation in multinomial logistic models

Abstract

Oppenheim et al. (2015) provides the first empirical analysis of insurgent defection during armed rebellion, estimating a series of multinomial logit models of continued rebel participation using a survey of ex-combatants in Colombia. Unfortunately, many of the main results from this analysis are an artifact of separation in these data – that is, one or more of the covariates perfectly predicts the outcome. We demonstrate that this can be identified using simple cross tabulations. Furthermore, we show that Oppenheim et al.’s (2015) results are not supported when separation is explicitly accounted for. Using a generalization of Firth’s (1993) penalized-likelihood estimator – a well-known solution for separation – we are unable to reproduce any of their conditional results. While our (re-)analysis focuses on Oppenheim et al. (2015), this problem appears in other research using multinomial logit models as well. We believe that this is both because the discussion on separation in political science has primarily focused on binary-outcome models, and because software (Stata and R) does not warn researchers about seperation in multinomial logit models. Therefore, we encourage researchers using multinomial logit models to be especially vigilant about separation, and discuss simple red flags to consider.