Automating knowledge extraction from large dermatopathological datasets to establish objective and interpretable diagnostic frameworks

ongoing

The continuous and overwhelming increase in the amount of data in health and science, including image collections, textbooks and scientific literature, renders it difficult to meaningfully understand, validate and query the available body of knowledge. Modern developments in artificial intelligence hold promise to solve these shortcomings by enabling the collection, understanding, curation, use of, as well as interaction with, large bodies of data. The goal of this project is to make this case in the explicit context of a highly specialized scientific field, dermatopathology, which is in charge of obtaining reliable diagnoses of skin diseases. We aspire to improve understanding in four different settings: 1) Unstructured real-life data: To obtain population-scale structured knowledge on the real-life burden of skin disease beyond cancer, we will use modern language models to automatically map historic unstructured pathology reports to disease entities. 2) Scientific corpora: Guidance of general-purpose language-models will be used to transform medical scientific corpora towards an open knowledge graph. 3) Cross-modality entity representation: Training a long-tail multi-modal and multi-task foundation model with enable to objectively define diagnostic grey zones and narrow them. 4) Interaction with large image data: Retrieval-, annotation-, and language-based interaction modalities will be developed into a digital microscopy viewing interface to find the most efficient avenues in human-computer interaction for the analysis of large-size medical image data. In sum, this project employs a multifaceted approach to standardize pathways to understand large, multi-modal open-set data, and further explore further the field of human-computer interaction in highly specialized imaging professions. Results will be transferable to other medical specialties, and vision-based scientific fields in general.