Biomechanics-assisted (medical) image processing & Augmented reality

In medical imaging, image registration is the process of transforming different sets of data into one coordinate system, i.e. projecting images into a common reference framework. Clinical data could include magnetic resonance imaging (MRI) scans, computed-tomography (CT), laser surface scans (Surf3D), multiple photographs; also, it may involve data by combining different sensors/cameras, at different time points, depths or even viewpoints. As far as breast cancer management is concerned, this usually involves acquisition of MRI and CT data. Specifically, prone-to-supine breast image registration can be applied into surgical and radiotherapy planning, image-guided interventions, multi-modal cancer diagnosis, staging, and prognosis (response to therapy prediction). However, registration of 3D image data, acquired with the patient in different postures is a challenging problem, due to large deformations induced to the soft breast tissue caused by the change in gravity. To circumvent this, we have integrated biomechanical in silico models with image processing techniques into a registration framework such that it aligns pertinent (MRIs, CTs, Surface) images in a central (reference) configuration by minimizing the target registration error to the image resolution accuracy.

This work was carried out at UCL’s Centre for Medical Image Computing, together with Dr B. Eiben, Dr J. Hipwell and Professor David Hawkes (UCL), and partners from PHILIPS (Germany), Royal Free Hospital (UK) and Champalimaud Foundation (Portugal).

Selected relevant works:

Eiben et al. 2015. Annals of Biomedical Engineering, doi: 10.1007/s10439-015-1496-z

Eiben et al. 2016. SPIE Medical Imaging, doi:10.1117/12.2216728

Eiben et al. 2016. In Lecture Notes in Computer Science, Springer, pp.274-281

Registration of a prone MRI scan data (red-frame illustration) to upright 3dMD or MS-Kinect scans of a female torso (blue frame illustration).
Data projection after image registration visualized in an animation. Notice the internal breast tissue structure underneath the skin texture. The animation above was produced by Dr Bjoern Eiben (UCL). ( cannot be embeded)

LEFT: Registration of prone MRI (left) to supine CT (right) breast images, where the end simulation result transforms the breast image data into an “unloaded state” – used to quantitively assess the registration result.
RIGHT: Animation of the registration algorithm with MRI and CT images starting from corresponding breast configurations, which both ultimately converge into a reference ”zero-gravity” configuration to accomplish registering the image data. ( cannot be embed into the websites)


Breast conservative surgery combined with radiotherapy has become the treatment of choice for the majority of women presenting with early breast cancer. Due to widespread screening an important percentage of early breast cancers are non-palpable, thus, tumour localization is carried out to “guide” the surgeon during surgery so as to successfully excise and resect the breast lesion. Existing however modalities for pre-operative localization are invasive requiring imaging guidance (e.g., wire-guided, carbon tattooing, radioactive seed localization, etc.). To overcome this, we are developing an innovative technology that is capable fusing augmented reality with in silico and imaging models.

Together with colleagues from the Champalimaud Foundation, INESC (Portugal) and Microsoft, we presented our work in Breast. In this paper we describe the first experimental test with a digital non-invasive method for intra-operative breast cancer localization using AR to guide breast conservative surgery.

Relevant work:

Gouveia et al. 2021. Breast, doi: 10.1016/j.breast.2021.01.004

Vavourakis et al. 2016. PLOS ONE, doi: 10.1371/journal.pone.0159766

LEFT to RIGHT: Fusion of structural data of computed tomography scans and 3D surface scans with the breast cancer (BC) patient being in the supine position. A fiducial marker (white arrow) is being used as a tracking reference for the Hololens system. Concept preliminary tested In the operating room, where the breast surgeon is wearing a Hololens headset to help guiding them during BC patient treatment for breast conservative surgery.

You might be interested in …