Towards meta-reinforcement learning-based computational phenotyping

Major depressive disorder (MDD) is a debilitating disorder, both at an individual and a societal level. It is highly prevalent and difficult to diagnose and treat; both diagnosis and treatment selection are based on symptoms, leading to suboptimal treatment selection. This thesis aims to improve treatment selection in major depressive disorder by moving away from a symptomatic approach towards an approach of treatment selection based on the underlying (causal) mechanisms. To this end, we propose to implement computational phenotyping, a method that can identify differences in parameters describing mechanisms altered in MDD. In this thesis, we describe and develop a computational phenotyping process, aiming to detect differences in mechanisms affected in MDD. We aim to discuss its required components, with emphasis on a valid computational model and model inversion procedure, and aim to test its effectiveness in two empirical studies. In this thesis, we furthermore aim to investigate the validity of the computational model we used in computational phenotyping: the reinforcement meta-learner model (RML). We attempt to validate this computational model across several decision-making paradigms, showing that it is able to correctly simulate the behaviour and neural activity of individuals in various conditions, showing that it is a biologically valid model of the mechanisms underlying decision-making. To achieve these goals, we proceed as follows. In the first chapter, we provide background information on MDD and describe the process of computational phenotyping. Based on earlier literature, we propose two of the components of this process. In the second chapter, we provide a technical description of the computational model (the RML) and propose a model inversion procedure, two components of the computational phenotyping procedure that are tested in the remainder of the thesis. In the third chapter, we validate the RML as a biologically valid computational model for effort-based decision-making. By implementing this model across three different tasks used in earlier studies, we show that it is able to simulate the neural activity found in several different decision-making paradigms. With this, we found evidence that the RML is able to provide a biologically valid account of several mechanisms during effort-based decision-making and, therefore, would be a suitable candidate for the computational phenotyping procedure. In the fourth chapter, we describe the validation of the RML in an empirical study, implementing an effort-based decision-making task in a stationary environment. In this study, we included an acute stress induction, a procedure known to change several mechanisms involved in MDD. In this study, we validated the RML by showing that it is able to predict behavioural and neural data from participants in this task following our computational phenotyping procedure. Furthermore, we showed that our computational phenotyping procedure using the RML could identify a difference in behaviour after stress and after a control procedure. In the fifth chapter, we propose an alternative to the model inversion method, one of the components of the computational phenotyping procedure, proposed in the first chapter, and compare these two methods using computational simulations. This model inversion method is empirically tested in our sixth chapter, where we perform a study investigating effort-based decision-making in a non-stationary environment. In the final chapter, we provide a summary of the findings and outline possible directions for future research. The results from the different studies in this thesis are mixed. First, they provide evidence validating the RML as a biologically valid computational model describing the mechanisms underlying effort-based decision-making. We show that it accurately simulates participants' behavioural and neural activity across all but one of the tested task paradigms: the effort-based decision-making paradigm in a non-stationary environment. We also show that the computational phenotyping procedure is able to successfully identify stress using behaviour from an effort-based decision-making task in a stationary environment. However, we were unable to confirm the validity of the computational phenotyping procedure in detecting differences in depressive symptom traits in a population of healthy individuals using behaviour from an effort-based decision-making task in a non-stationary environment. All in all, these findings provide strong evidence for the validity of the reinforcement meta-learner as a biologically valid computational model explaining behaviour and neural activity during effort-based decision-making. Furthermore, we found mixed evidence that the computational phenotyping procedure (including the RML) is promising for detecting changes in traits related to MDD. With this, we have provided a foundation for a method to apply the RML to advance precision medicine for MDD. However, given the mixed results in the non-stationary environment, further research testing this procedure – preferably in a patient study – is required to further develop a mechanism-based approach for aiding diagnosis and treatment selection in MDD.