From
Time -
STL, Thesis
Natural Language Inference for clinical trials
Speaker : Mathilde AGUIAR
Jury
- Elena CABRIO, Rapporteur & Examinatrice, Professeure des universités, I3S, Université Nice Côte d’Azur
- André FREITAS, Rapporteur & Examinateur, Senior Lecturer, équivalent HDR, Department of Computer Science at the University of Manchester, CRUK Manchester Institute, Idiap Research Institute
- Claire GARDENT, Examinatrice, Directrice de recherche, LORIA, Université de Lorraine
- Albert RILLIARD, Examinateur, Chargé de recherche, LISN, CNRS
- Fatiha SAÏS, Examinatrice, Professeure des universités, LISN, Université Paris-Saclay
Keywords
Natural Language Processing, Natural Language Inference, Clinical trials, Inference types, Large Language Models, Finetuning
Abstract
Clinical trials are essential for developing novel treatments that are both efficient and safe for patients.
Clinical trials provide detailed information about the interventions being evaluated, the study populations involved, the outcome measures assessed, and the adverse events observed. Interpreting and analyzing this information requires substantial domain expertise and a deep understanding of clinical research. Automating the interpretation of clinical trial reports remains challenging, even for language models, as the relevant evidence is often spread across multiple sections and expressed through complex semantic relationships. Natural Language Inference (NLI), which determines whether a hypothesis is entailed or contradicted by a given premise, provides a natural framework for reasoning over such evidence. The example applications of NLI in this context include clinical evidence assessment and the improvement of the patient-trial matching process.
This thesis focuses on enhancing the interpretation of clinical trials by using Natural Language Inference.
It provides a comparative analysis of several approaches based on model finetuning and prompting in this context. The analysis shows that generative language models outperform masked language models, which have traditionally been used for the NLI task, even without additional finetuning, solely through prompting.
This thesis also provides additional annotations for the NLI4CT dataset to enable a fine-grained evaluation of language models across different types of inference. Our experiments demonstrate that models vary in their sensitivity to the types of inferences involved. However, for models that are sensitive to inference types, logical and numerical inferences are often the ones that negatively impact performance.
Finally, this thesis proposes a novel dataset, NLI4PR, which addresses the patient-to-trial matching. This dataset uses NLI to assess whether a patient profile, expressed in medical or lay language, meets the trial’s eligibility criteria. The results show that, across the best-performing models tested, using lay language results in only a slight decrease in performance. This suggests that lay language could also be used for NLP-based tools designed for patient-trial matching.
Codebases and datasets resulting from this thesis are all freely available on GitHub (Explore).