Mind Video

What is Mind Video?

Mind Video is a two-part model that reconstructs video from brain fMRI signals, bridging the gap between decoding images and decoding video from brain activity. It learns progressively across several stages to capture the meaning behind what a person is seeing.

Training happens in three steps. First, unsupervised masked brain modeling learns general visual fMRI features. Second, contrastive learning within the CLIP space, using annotated datasets, distills semantic features. Third, these features are fine-tuned by co-training with a modified stable diffusion model tailored for video generation and guided by fMRI data.

The result is high-quality video reconstructions with accurate semantic meaning, outperforming previous methods on both semantic and pixel-level metrics by 45%.

Who created Mind Video?

Mind Video was developed by Zijiao Chen, Jiaxin Qing, Tiange Xiang, and Professor Juan Helen Zhou. It comes from Mind-X, a research group focused on decoding multimodal brain signals using large-scale models. Their goal is to advance brain decoding toward general-purpose models for brain-computer interfaces, neuroimaging, and neuroscience research.

What is Mind Video used for?

• Reconstructing human vision from brain activity to study cognitive processes
• Recovering continuous visual experiences as videos
• Connecting image and video brain-signal decoding
• Producing semantically accurate video reconstructions
• Supporting work in brain-computer interfaces, neuroimaging, and neuroscience

Technically, the model uses hierarchical encoder layers that preserve volume and time-frames, masked brain modeling, large-scale unsupervised learning, multi-modal contrastive learning, progressive semantic learning, and analytical attention analysis. It also demonstrates the dominance of the visual cortex.

Who is Mind Video for?

• Neuroscientists and neurologists
• Brain-computer interface (BCI) researchers and specialists
• Neuroimaging scientists and researchers
• Psychiatrists
• Anyone working in neuroscience