Index of Neuroimaging Datasets for Visual Perception Reconstruction
This repository indexes open neuroimaging datasets for reconstructing visual perception from human fMRI data.
This guide is primarily aimed at researchers from AI and machine learning backgrounds who may not be familiar with neuroimaging methodology. Reconstruction from neuroimaging data has recently gained popularity at major AI conferences, but many approaches fall into common traps that are well known within neuroscience. These pitfalls can lead to misleading results, often due to misunderstandings about the nature of fMRI data or the limitations of datasets originally collected for other research questions. For a detailed discussion of such issues in recent reconstruction pipelines, see: Shirakawa, K. et al. (2025). Spurious reconstruction from brain activity, Neural Networks .
Table of Contents
Basics: Identification vs. Decoding vs. Reconstruction
These terms are now often conflated. In foundational reconstruction and decoding literature their separation has been strict for good reasons: the tasks differ in difficulty, failure modes, what can be concluded from a result and what a method can realistically achieve.
True reconstruction requires potential to generalize to stimuli and categories that were not present during training. If the correct answer is (also implicitly) selected from a predefined candidate category or image set, the wording is decoding or identification, not reconstruction.
Neuroscience term ML framing Search space Difficulty Decoding classification closed label/category set easy Identification retrieval finite set of images moderate Reconstruction generative inverse problem infinite, open set of perception hard
Decoding (category level)
Decoding refers to predicting (classifying) pre-defined labels or cognitive states from brain activity patterns. This type of classification has long been used for neuroscientific insight. In multivariate pattern analysis (MVPA), voxel activity patterns are treated as feature vectors and classifiers are trained to distinguish experimental conditions (for example risky vs. safe decision, add vs. subtract, rule A vs. rule B). Decoding is scientifically useful because it tests whether information about a stimulus or mental state is present in a brain region. However, this is constrained to a predefined closed set, arbitrary mind or visual content can not be recovered. Many recent papers labeled as “reconstruction” are effectively performing n-way decoding: they decode a category and then use a generative model to produce a visually plausible sample inside the predicted class. This can look convincing, but remains a restricted classification problem. Similar n-way decoding setups have long been standard in the MVPA literature and can achieve quite high performance when the candidate set is limited.
... continue reading