Multimodal Learning to Enhance Diagnosis, Monitoring and Accessibility in Healthcare

Artificial intelligence (AI) is increasingly adopted in diagnosis, monitoring, and assistive technologies, yet real-world impact is often limited by practical constraints on data collection: heterogeneous acquisition protocols, privacy-sensitive modalities, and deployment requirements in resource-constrained settings. This dissertation studies multimodal learning across three independent research tracks in healthcare, showing how combining complementary inputs and modern representation learning can improve robustness, generalization, and usability in realistic workflows. Part I focuses on cardiovascular analysis from coronary angiography. We develop spatio-temporal deep models that operate on angiographic sequences to quantify stenosis severity and to estimate functional indices (FFR/iFR) without invasive probes. The contributions span 3D and attention-based modeling, hybrid convolutional-transformer designs for temporal contrast-flow representation, and learning under distributed multi-center settings via federated learning, including detection-transformer-based pipelines on key frames. We also investigate how foundation-model paradigms can be adapted to angiography with parameter-efficient strategies, leveraging contrastive vision-language representations and learnable prompts to improve transfer under domain shift. Part II targets automated cognitive and behavioral assessment from scalable, non-invasive signals compatible with commodity devices and consumer wearables. We study the extraction of subtle facial dynamics through phase-based motion amplification and their integration with affective and task-performance descriptors for estimating cognitive screening scores. We also investigate transformer-based modeling of eye-gaze traces collected during structured tasks, emphasizing temporal representation learning and evaluation protocols aligned with remote assessment. In addition, we include a long-term wearable sensing study that analyzes multimodal smartwatch time series under realistic conditions of missingness and heterogeneity, covering both subject identification and personalized relapse detection in the psychotic spectrum. Part III addresses accessibility in clinical communication through sign language recognition for medical vocabulary. We consolidate multimodal dataset design and benchmarking for Italian Sign Language, and we advance privacy-preserving recognition using mmWave radar representations and sequence models. The work also explores semantic grounding through text-aligned embeddings and reports benchmarking outcomes through community evaluation initiatives. Overall, the dissertation shows that multimodal learning, by aligning and fusing complementary signals and by enabling modality substitution when constraints arise, provides a practical route to robust and deployable healthcare AI across diagnosis, monitoring, and accessibility.