Multimodal brain decoding through deep learning

Understanding the neural mechanisms underlying cognition and perception is a fundamental pursuit in neuroscience. This thesis addresses the challenge of de-coding brain activity to interpret and reconstruct human cognitive processes using functional magnetic resonancei maging (fMRI). By lever aging advancements in deep learning, we develop robust pipelines for decoding various sensory and cognitive modalities, including vision, language, music, and video. Central to our approach is the alignment of brain representations with computational models, assuming brain representations can be mapped on vectorial spaces of pretraining multimodal models, enabling seamless mappings between neural data and external stimuli. Through the integration of encoding and decoding models, we explore tasks ranging from image reconstruction and cross-modal decoding to language generation and video retrieval. Furthermore,this thesis delves into the concept of "brain algebra", examining how neural representations adhere to composition a land transformational principles ak into vector spaces. By perturbing brain activity in this high-dimensional semantic space, we uncover insightsin to the brain’s capacity for concept manipulation and compositionality. This work highlights the synergy between neuroscience and artificial intelligence, showcasing how multimodal, data-driven approaches can deepen our understanding of brain function and pave the way for innovative applications in brain-computer interfaces and cognitive modeling.