Ghost in the VR Shell: Capturing Spectral Cardio-Respiratory Rates from Subtle VR Device Movements

This work examines the estimation of heart rate and respiratory rate using only the kinematic data captured by a consumer-grade standalone VR devices. The high-resolution motion tracking offered by these devices creates an opportunity for indirect vital sign detection through spectral analysis of subtle VR device movement data. In our study, kinematic data were collected from a Meta Quest 3 head-mounted display, controllers and MX Ink pen across multiple posture configurations (e.g., seated, standing, lying down), both at rest and after moderate exercise. These postures emulate real-world XR scenarios for rest, fitness, and meditation. The collected data was processed using what we refer to as the Ghost approach, a simple yet effective method that applies a Fast Fourier Transform to capture the spectral components associated with respiratory and cardiac rhythms. Ground-truth biosignals were simultaneously recorded using wearable physiological sensors for validation. Results clearly reveal that both heart rate and respiratory rate can be reliably estimated from subtle micro-movements in the head-mounted display, VR controllers, or VR pen, revealing the potential for non-contact physiological monitoring within immersive environments. Finally, we demonstrate a use case of a VR stethoscope, where a standard VR controller is repurposed to estimate heart and respiratory rates.