The Use of Technology in Evaluation Practice

Abstract

Background: Evaluation practice is no longer limited to pencil and paper questionnaires, today technological advances allow evaluators to collect data with handheld devices, visualize information in interactive ways, and communicate instantaneously with stakeholders across the globe. These advances have changed how we conduct our practice and they will continue to redefine how we design our evaluations, interact with stakeholders, and communicate our findings. There are few published articles that examine the interface between evaluation and technology and this study represents an initial attempt at examining the technological tools that evaluators use in their practice, the reasons they are adopted, and future technological interest of practicing evaluators. Purpose: This research on evaluation study attempts to (1) identify the types of technology tools evaluators use in their practice, (2) describe the factors that predict technology adoption, and (3) understand the tools that evaluators are interested in learning more about. This inquiry offers the evaluation community a broader perspective on the technologies that evaluators are implementing in their practice, offers insights on future technological trends within the field, and introduces the evaluation community to tools that can potentially enhance practice. Setting: Virtual on-line community of evaluation practitioners. Intervention: This was an exploratory research on evaluation study with no intervention. Research Design: A panel of experts on technology and evaluation were recruited to brainstorm a comprehensive list of technologies that could be adopted by evaluators as part of their practice. The comprehensive list of technology tools was then embedded within a larger survey instrument that was distributed to a sample of evaluation practitioners from the American Evaluation Association. The survey asked evaluators to select the technology tools they have or currently use in their practice, how often each were used, their satisfaction with each tool, and why they were utilizing each tool in their practice. Data Collection and Analysis: Data were collected through a web-based survey from members of the American Evaluation Association. The analysis utilized descriptive statics to represent trends in technology use and adoption. Multiple statistical comparisons using ANOVA were also utilized to examine the relationship between technological adoption and use and evaluator background characteristics. Open-ended responses in the survey were also presented as part of the analysis. Findings: Analyses revealed that technological tools were adopted by evaluators because they helped to produce quality products, increased timeliness, reduced errors, and increased cost efficiencies, and the most adopted tools tended to aid in quantitative data analysis, project management, and productivity. Many evaluators expressed interest in learning more about the use of qualitative analysis tools, web-based data collection tools, and relational database creation and management. Keywords: technology adoption; technology use; research on evaluation