Exploring New Technologies in Healthcare: Advancing Natural Language Processing

In the last few years, Natural Language Processing (NLP) has gained impressive momentum in both academic and industrial research. Texts portray distinct characteristics from other kinds of data (such as images, audio, etc.), being inherently discrete, compositional, and hierarchical. NLP techniques allow the manipulation of such a peculiar source of information, providing researchers and practitioners with a way to automatize the analysis of textual data, enabling humans to communicate with machines (and vice versa) through natural language. Understanding and generating natural language revealed to be a valuable ally in many fields, including healthcare. The process of digitalization taking place nowadays in healthcare, as well as in everyday life (e.g., social media), is pushing the need for tools to manage the high volumes of textual data available today. Leveraging unstructured, textual data from Electronic Health Records (EHRs) and Internet resources has disclosed plenty of applications, paving the way for improvements in the care of patients and their diseases. The exploitation of NLP techniques in the healthcare domain is not only driven by the digitalization process we are living in but also by the advancements in the NLP field over the past years. In the last decade, in particular, we shifted the NLP paradigm from classical, machine learning-driven pipelines to end-to-end, deep learning ones. Especially in the very last few years, the NLP field was ruled by the Transformers architectures, which achieved state-of-the-art performance on numerous tasks. Besides the large improvements obtained with these kinds of architectures, concerns about their explainability have risen. The end-to-end paradigm, together with the complexity of deep learning models, makes it difficult to understand the motivations behind their decisions, which inhibits the interpretation from final users, linguists, or domain experts. Such an issue is particularly felt in a sensitive domain such as healthcare. Furthermore, being unable to understand the mechanisms behind their reasoning inhibits the researchers from getting rid of the current models and providing new solutions. The present manuscript thus explores the landscape of NLP solutions in healthcare and provides significant contributions to the field. It demonstrates the worth of investigating such technology for improving healthcare, with particular focus on the explainability of the state-of-the-art models, i.e., Transformers, providing new solutions and analyses. After providing an extensive background of NLP and its advancements, focusing on the solutions proposed in the healthcare literature, we investigated the use of Transformers in both Natural Language Understanding (NLU) and Generation (NLG). For the former, we collected the first dataset for sentiment analysis in Italian for healthcare. In our work we compared Transformer-based and Machine Learning (ML)-based NLP. Quite surprisingly, the classical model outperformed the other, for which we highlighted its sensitivity to data class imbalance. For the latter, we faced the problem of reducing the expertise gap for patients reading medical texts by proposing a new system to simplify such documents. We employed Transformer-based bi-encoders (also known as Sentence Transformers) to collect new parallel datasets we analyzed in quality and then used to train an encoder-decoder model (again, based on the Transformers architecture). The analysis we conducted with human evaluators assesses without doubts our system to outperform models proposed in the past literature, while providing relevant insights on the automatic evaluation metrics usually employed in this kind of tasks. Finally, we contributed to overcome the explainability issues, both from the end-user and researchers standpoints. First, we proposed two hierarchical architectures based on Transformers to perform document classification tasks while providing document summaries as an explanation of the decisions made. Using a well-known benchmark in sentiment analysis, we evaluated the two proposed models, highlighting their strengths and weaknesses. Both systems achieved good results, not so far from previous literature, while providing extractive summaries as an explanation of the sentences that were most relevant for the decision. Our proposed evaluation protocols ensured their ability to explain their reasoning. Then, we conducted a study to investigate the robustness of Transformers when adapting to new domains through the further pre-training paradigm. By inducing minimal variations we disclosed surprising instabilities in fine-tuning.After testing a very large number of combinations, which we briefly summarize, our experiments focused on an intermediate phase consisting of a single-step and single-sentence masked language modeling stage and its impact on a sentiment analysis task. We discuss a series of these unexpected findings which leave some open questions over the nature and stability of further pre-training and Transformers themselves.