Efficient contextual embeddings to improve the transferability of disease trajectory projections with generative models

Abstract

A growing global burden of non-communicable diseases demands enhanced preventive healthcare measures to avert escalating societal costs. Electronic Health Records (EHRs) hold considerable promise in this regard, as they facilitate the leveraging of past disease trajectories to predict future risks. However, the transferability of current projection methods is limited due to their dependence on the specific biomedical vocabularies on which they were trained, restricting applicability across countries.

This thesis proposes a new embedding model based on the BERT architecture that integrates codes from disparate medical vocabularies. The model is trained using a contrastive objective that ensures the produced embeddings reflect the semantics and relationships of the underlying concepts. An integration into Delphi, a transformer-based method for producing disease risk predictions, ultimately aims to enable disease trajectory projections based on medical events encoded in numerous medical vocabularies, possibly not included in the training data of the model.

In an external evaluation, the proposed embedding model demonstrated superior performance in its capacity to align and differentiate related and unrelated concepts compared to other models for embedding biomedical concepts, excluding the OpenAI embeddings. This is achieved while maintaining only 20% of the size of the smallest embedding models in the comparison. Moreover, following the integration with the embedding model, Delphi exhibits effective transfer between synonymous codes of varying vocabularies, demonstrating minimal impact on performance. Transferability was also achieved between countries, specifically from the United Kingdom to Finland. Here, the model improved upon the original Delphi by incorporating codes not part of the training vocabularies in addition to also utilizing completely different vocabularies.

Given these results, the proposed model has the potential to facilitate enhanced disease prevention, for instance, through the development of cross-country risk prediction models and by utilizing the complete array of available EHR data to make more precise risk predictions.