Supporting grant reviewers through the scientometric ranking of applicants.

Abstract

IntroductionComparing the scientific output of different researchers applying for a grant is a tedious work. In Hungary, to help reviewers to rapidly rank the scientific productivity of a researcher, a grant decision support tool was established and is available at www.scientometrics.org. In the present study, our goal was to assess the impact of this decision support tool on grant review procedures.MethodsThe established, publicly available scientometric portal uses four metrics, including the H-index, the yearly citations without self-citations, the number of publications in the last five years, and the number of highly cited publications of a researcher within eleven independent scientific disciplines. Publication-age matched researchers are then ranked and the results are provided to grant reviewers. A questionnaire was completed by reviewers regarding utilization of the scientometric ranking system. The outcome of the grant selection was analyzed by comparing scientometric parameters of applying and funded applicants. We compared three grant allocation rounds before to two grant allocation rounds after the introduction of the portal.ResultsThe scientometric decision support tool was introduced in 2020 to assist grant selection in Hungary and all basic research grant applicants (n = 6,662) were screened. The average score of funded proposals compared to submitted proposals increased by 94% after the introduction of the ranking. Correlation between ranking scores and actual grant selection was strong in life and material sciences but some scientific panels had opposite correlation in social sciences and humanities. When comparing selection outcome to H-index across all applicants, both type I and type II errors decreased. All together 540 reviewers provided feedback representing all eleven scientific disciplines and 83.05% of the reviewers (especially younger reviewers) found the ranking useful.ConclusionsThe scientometric decision support tool can save time and increase transparency of grant review processes. The majority of reviewers found the ranking-based scientometric analysis useful when assessing the publication performance of an applicant.