Do we really measure what we think we are measuring?
Dario Gordillo,
Janir Ramos da Cruz,
Dana Moreno,
Simona Garobbio,
Michael H. Herzog
Affiliations
Dario Gordillo
Laboratory of Psychophysics, Brain Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland; Corresponding author
Janir Ramos da Cruz
Laboratory of Psychophysics, Brain Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland; Institute for Systems and Robotics – Lisboa (LARSyS), Department of Bioengineering, Instituto Superior Técnico, Universidade de Lisboa, 1049-001 Lisbon, Portugal; Wyss Center for Bio and Neuroengineering, CH-1202 Geneva, Switzerland
Dana Moreno
Laboratory of Psychophysics, Brain Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland
Simona Garobbio
Laboratory of Psychophysics, Brain Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland
Michael H. Herzog
Laboratory of Psychophysics, Brain Mind Institute, School of Life Sciences, École Polytechnique Fédérale de Lausanne (EPFL), CH-1015 Lausanne, Switzerland
Summary: Tests used in the empirical sciences are often (implicitly) assumed to be representative of a given research question in the sense that similar tests should lead to similar results. Here, we show that this assumption is not always valid. We illustrate our argument with the example of resting-state electroencephalogram (EEG). We used multiple analysis methods, contrary to typical EEG studies where one analysis method is used. We found, first, that many EEG features correlated significantly with cognitive tasks. However, these EEG features correlated weakly with each other. Similarly, in a second analysis, we found that many EEG features were significantly different in older compared to younger participants. When we compared these EEG features pairwise, we did not find strong correlations. In addition, EEG features predicted cognitive tasks poorly as shown by cross-validated regression analysis. We discuss several explanations of these results.
