DCNN-GAN: Reconstructing Realistic Image from fMRI

Visualizing the perceptual content by analyzing human functional magnetic resonance imaging (fMRI) has been an active research area. However, due to its high dimensionality, complex dimensional structure, and small number of samples available, reconstructing realistic images from fMRI remains challenging. Recently with the development of convolutional neural network (CNN) and generative adversarial network (GAN), mapping multi-voxel fMRI data to complex, realistic images has been made possible. In this paper, we propose a model, DCNN-GAN, by combining a reconstruction network and GAN. We utilize the CNN for hierarchical feature extraction and the DCNN-GAN to reconstruct more realistic images. Extensive experiments have been conducted, showing that our method outperforms previous works, regarding reconstruction quality and computational cost.

💡 Research Summary

The paper “DCNN‑GAN: Reconstructing Realistic Image from fMRI” addresses the long‑standing challenge of generating high‑fidelity visual reconstructions from functional magnetic resonance imaging (fMRI) data. The authors observe that fMRI signals are extremely high‑dimensional, noisy, and typically available in only a few hundred samples, which makes direct image reconstruction ill‑posed. Recent advances in deep convolutional neural networks (CNNs) and generative adversarial networks (GANs) have opened the possibility of mapping multi‑voxel brain activity to complex visual representations, but prior work either suffers from poor texture detail, over‑fitting to a narrow set of categories, or computationally expensive iterative optimization.

To overcome these limitations, the authors propose a three‑stage pipeline called DCNN‑GAN. First, an encoder network E based on a pre‑trained VGG‑19 extracts hierarchical visual features from the original image. Rather than using the full convolutional feature maps, they take the output of the first fully‑connected layer (fc7), yielding a 4096‑dimensional vector z that preserves semantic content and the image’s categorical label c while dramatically reducing dimensionality. Second, a decoder D_f learns a mapping from fMRI voxel data X (≈4,400 voxels from the visual cortex) to the feature vector z using ridge regression (linear least‑squares with L2 regularization). This regularization stabilizes the solution, improves the coefficient of determination (R² from –0.31 to 0.32) and reduces root‑mean‑square error (RMSE from 0.496 to 0.361) compared with the ordinary least‑squares approach used in earlier studies. Third, a reconstruction network R (a deconvolutional decoder) transforms the decoded feature vector back into a coarse image R(z) of size 112×112×3. Because R(z) captures overall shape but lacks realistic texture, a conditional GAN (pix2pix) is employed. The generator G receives both R(z) and the categorical label c, producing a refined image x̂ that is encouraged to resemble natural images of the same class. The discriminator D learns to distinguish real images from generated ones. The overall loss combines the conditional GAN loss, an L1 pixel‑wise loss, and an L2 reconstruction loss, enabling the generator to add semantically plausible details while preserving the decoded structure.

Experiments were conducted on the ILSVRC‑2012 dataset (1.2 M training images) and a corresponding fMRI dataset collected while subjects viewed 1,200 ImageNet images (training) and 50 test images (repeated 35 times). The encoder, decoder, and reconstruction network were implemented in PyTorch; the reconstruction network was trained for 200 epochs, and the GAN for 500 epochs, using Adam optimization with learning‑rate schedules and Gaussian noise augmentation for robustness.

Quantitative evaluation shows that ridge regression markedly improves decoding accuracy over the sparse linear regression used in prior work. Qualitative visual comparison demonstrates that DCNN‑GAN produces images with sharper edges, richer textures, and more faithful color reproduction than the baseline method, which tended to preserve shape but produce blurry textures. A double‑blind human perceptual study with over 40 volunteers revealed that 55.7 % of participants preferred the DCNN‑GAN reconstructions over the baseline (44.3 % preference), and satisfaction rates rose from 12.4 % to 20.1 %.

The paper’s contributions are threefold: (1) leveraging a deep CNN encoder to compress visual information into a manageable feature vector, (2) applying ridge regression for stable and accurate fMRI‑to‑feature mapping, and (3) integrating a conditional GAN that injects category‑specific priors to refine coarse reconstructions into realistic images. The combined system operates in a single forward pass, enabling near real‑time reconstruction—a significant advantage over iterative optimization approaches. The authors suggest future work on generalizing to unseen categories and integrating the framework into brain‑computer interfaces for real‑time visual feedback.