Induction in Neuroscience with Classification: Issues and Solutions

Machine learning and pattern recognition techniques are increasingly adopted in neuroimaging-based neuroscience research. In many applications a classifier is trained on brain data in order to predict a variable of interest. Two leading examples are brain decoding and clinical diagnosis. Brain decoding consists of predicting stimuli or mental states from concurrent functional brain data. In clinical diagnosis it is the presence or absence of a given medical condition that is predicted from brain data. Observing accurate classification is considered to support the hypothesis of variable-related information within brain data. In this work we briefly review the literature on statistical tests for this kind of hypothesis testing problem. We claim that the current approaches to this hypothesis testing problem are suboptimal, do not cover all useful settings, and that they could lead to wrong conclusions. We present a more accurate statistical test and provide examples of its superiority.