[LR] Reconstruction of Dynamic Natural Vision from Slow Brain Activity
[This review is intended solely for my personal learning] Paper Info arXiv: 2405.03280v2 Title: Animate Your Thoughts: Decoupled Reconstruction of Dynamic Natural Vision from Slow Brain Activity Authors: Yizhuo Lu, Changde Du, Chong Wang, Xuanliu Zhu, Liuyun Jiang, Xujin Li, and Huiguang He Prior Knowledge Brain decoding with fMRI: Functional MRI measures brain activity through BOLD (Blood-Oxygen-Level Dependent) signals, which reflect neural activation across 3D voxel grids. Decoding methods aim to map these signals back to perceptual experiences like images or videos. Contrastive learning and CLIP: Contrastive loss functions (e.g., InfoNCE) train models to align representations from different modalities. CLIP is a pre-trained vision-language model that embeds text and images into a shared space, useful for representing high-level semantics. VQ-VAE (Vector Quantized Variational Autoencoder): A generative model that discretizes image data into latent tokens, enabling compression and reconstruction. It’s commonly used in diffusion models for reducing the input dimensionality while preserving structure. Transformer and sparse causal attention: Transformers model long-range dependencies via attention. Sparse causal attention masks future tokens and restricts attention to a sparse set, ensuring temporal consistency while reducing computational cost—critical for decoding motion from fMRI. Stable Diffusion and U-Net: Stable Diffusion is a text-to-image model that generates visuals via iterative denoising in a latent space. Its U-Net backbone processes image latents at multiple resolutions and is commonly inflated for video generation. Goal To accurately reconstruct dynamic natural vision (videos) from fMRI recordings by explicitly disentangling and decoding semantic, structural, and motion features — each aligned with distinct neural processes — in order to improve both reconstruction fidelity and neurobiological interpretability. ...