Expert-Driven and Citational Approaches to Assessing Journal Publications of Brazilian Political Scientists

Abstract

In this study, we seek to contribute to discussions on how the quality of academic production in the field of political science should be evaluated using Brazil as a case study. We contrast the 'expert-driven approach' that is followed by CAPES, an agency of the Brazilian federal government with the 'citational' approach, which is based on the ranking of journals by mainstream indices of scientific research impact. With data provided by CAPES from 2010 to 2014, we examine to what extent journals that are ranked as having high quality by CAPES also have high impact indexes in the SCImago Journal rank index (SJR), the Hirsch index (h-index) calculated by SCImago, the h5-index and h5-median (based on the h-index period 05 years, calculated by Google Scholar Metrics), and the SNIP indicator (calculated by the CWTS Journal Indicators, included in the Scopus database). Our findings show that there is a positive, but weak correlation between citational criteria and the Qualis evaluation of the same journals. In ordered logistic regressions, we show that a journal's past Qualis scores are the most important factor for explaining its grades in the next evaluation. We show that once a journal's past Qualis score is considered, a journal's citational ranking does not influence its Qualis score with the exception of the SJR in the 2013-4 evaluation. Moreover, a journal's Qualis score is not influenced by the country of publication, language, or social science focus, all else equal.

Keywords

• Journal evaluation
• political science
• Qualis
• citational approach
• expert-driven approach