3D Scene Reconstruction, Segmentation and Augmented Reality for Robotic and Endoscopic Surgery

ongoing

The 3D-Intel-Surgery project aims to advance robotic-assisted and minimally invasive surgery (RMIS) to improve the patient outcomes through novel 3D intelligent algorithm development, inter-disciplinary collaborations, skills training and knowledge transfer between world leading researchers in the EU. The innovative aspects of the project are as follows: (a) New methods for medical image processing for RMIS to advance the field significantly by developing a robust deep learning-based segmentation algorithm, such as the Segment Anything-guided method, for endoscopy images, resulting enhanced accuracy of image segmentation in medical diagnoses. (b) Augmented reality for RMIS through the rapid and accurate 3D reconstructions of soft tissues using monocular endoscopic videos for augmented reality of surgical scenes to improve the visual quality of the medical procedure. (c) Adaptive stereo vision transfer for enhanced depth perceptions through the development of adaptive stereoscopic transformation methods based on the 3D reconstruction, 3D segmentation, and viewing conditions. Interdisciplinary collaborations underpin the success of this project since it is at the cross-section of medical image processing, computer graphics and Augmented Reality, and computer vision. By advancing precision medical technologies such as robotic and endoscopic surgery, the EU can address critical societal challenges, including ageing populations and increased costs and demands for cancer treatment, and sustainable and high-quality healthcare for all its citizen. The project supports knowledge exchange and future academic developments of the research teams and trains the PF into a leading European multi-disciplinary scientist through collaboration with world class researchers in the UK and the digital innovation medical device SEM in Italy to provide essential research expertise and resources for knowledge exchange, log-term collaboration, and industry innovation.