Mixed reality techniques in cardiovascular medicine: a new approach towards image-based diagnosis and procedural support

28 Feb 2020

Adriana Zlahoda-Huzior

MedApp S.A., Krakow, Poland and AGH University of Science and Technology, Department of Measurement and Electronics, Krakow, Poland

Maciej Stanuch

MedApp S.A., Krakow, Poland and AGH University of Science and Technology, Department of Measurement and Electronics, Krakow, Poland

Dariusz Dudek

Institute of Cardiology, Jagiellonian University Medical College, Krakow, Poland and Maria Cecilia Hospital S.p.A., Cotignola, Italy

Over the last few decades, society has observed considerable technological advances in the field of computation and in particular, the ever increasing processing power juxtaposed with the further drive to miniaturisation of computer components as seen for instance with smartwatches. These technological gains are obviously not exclusive to the general public but are also evident within, amongst others, the medical field. Within diagnostic imaging, one could consider for example the progressive steps from a single x-ray image, the transformation to computed tomography (CT) scans with further iterations in higher intricate image delineation (multiple slices) as also volumetric representations of structures (3D). In spite of the immense strides in imaging, one facet of imaging analysis has always remained the same – medical images are always evaluated on a computer screen inclusive of inherent limitations. Presently, a new field within diagnostic imaging is evolving, in which imaging has become extraordinarily interactive with the application of hand gesturing and associated voice commands. These interactive elements have kindled novel proposals in anatomical understanding which in turn enhance contemporary diagnostic protocols and intuitively, also procedural support.

Topic(s):

Electronic Health Records & Hospital Information Systems

Defining realities

Virtual reality (VR) is the application of computer technology to generate a three-dimensional simulated environment in which the user not only experiences but also interacts with the projected artificial environment, colloquially referred to as an immersive experience. This virtual reality is commonly rendered with a head-mounted display device. Augmented reality (AR) enhances real-world viewing with digitally generated components which provides the user with a more faithful representation of observations. Whereas VR engages a user in an artificial environment and AR superimposes digital objects on the real-world environment, mixed reality (MR) merges the fundaments of both VR and AR, in other words, a user can interact and manipulate both real and virtual items.

Figure 1: The spectrum of mixed reality techniques between real(10) and digital environments(11)

The three-dimensional digital models presented with display devices are in essence a form of holography, which is a diffraction technique that produces three-dimensional imagery effectuating amongst others the principles of depth perception such as parallax and perspective. Currently, the most popular head-mounted display devices are the Microsoft HoloLens (Redmond, Washington, USA) and Magic Leap (Plantation, Florida, USA). Notwithstanding, an inexpensive experience can be also attained by using a mobile phone with dedicated applications but this approach is limited in terms of user immersion and interactivity compared to the dedicated head-mounted display devices.

Mixed reality techniques within the cardiology field

Instinctively mixed reality technologies lends itself well to educational domains. A number of MR derived training applications have been previously reported. The HoloAnatomy programme at Case Western Reserve University (Cleveland, Ohio, USA) resulted in the first MR healthcare application¹. The Lucile Packard Children’s Hospital Stanford (Palo Alto, California, USA) developed novel interactive visualisations as part of their Stanford Virtual Heart Project so that paediatric cardiologists can explain complex congenital heart defects to both trainees and patients alike.²

Additionally mixed reality technologies are gradually being introduced into pre-procedural planning protocols^3,4. Cardiac DICOM image visualization in three-dimensional simulated environment provides the user an enhanced appreciation of depth perception entailing accurate volume measurements and higher quality data management. It is principally correlated to computed tomography or magnetic resonance studies performed prior to specific cardiac procedures⁵. When comparing three-dimensional displays generated by AR or VR techniques to standard flat screens monitor displays data interpretation time is reduced with similar results in terms accuracy.

Figure 2: An example of a minimal invasive cardiac surgical procedure with mixed reality support. a) A physician wearing a Microsoft HoloLens headset manoeuvering the hologram by hand gestures. b) The physician's field of view. The holographic visualisation of three dimensional echocardiography data (CarnaLife Holo, MedApp S.A., Poland) streamed in real-time. In the background conventional computer monitors displaying patient's angiography and two dimensional echocardiography.

As a consequence, AR is being deployed for peri-procedural data visualisation with real-time three-dimensional rotational angiography or echocardiography data streaming from devices inside the surgical theatre ⁶ (Fig.2). Image / data manipulation using voice commands and hand gestures helps to establish a flexible and convenient workflow for physicians even throughout the procedure. A number of companies are developing AR technology for data streaming and holographic visualisation quality improvement such as RealView Medical Imaging (Yokneam, Israel) ⁷,CarnaLife Holo - MedApp (Krakow, Poland) ⁸. Within the field of rehabilitation, MindMaze (Lausanne, Switzerland ) has applied VR technology to improve patient limb mobility in a post CVA setting ⁹.

Challenges and further works

It is clear that the promise presented by MR techniques in the medical field are almost infinite. Nevertheless, numerous iteration cycles will be fundamental before a comprehensive adoption can be considered. Among the current limitations are the concerns with precise stereoscopy views in VR and small view fields in AR devices. Conventional developments factors such as cost, size, weight and computer power will be detrimental to success.

Conflict of Interest

Zlahoda-Huzior and Maciej Stanuch are scientific developers for MedApp SA, Krakow, Poland. Dariusz Dudek is a member of the Scientific Board, MedApp SA, Poland.

References

http://case.edu/hololens/. Accessed October 20, 2019.
http://www.stanfordchildrens.org/en/about/news/releases/2017/virtual-reality-program. Accessed October 20, 2019.
Brun H, Bugge RAB, Suther LKR, et al. Mixed reality holograms for heart surgery planning: first user experience in congenital heart disease. Eur Heart J Cardiovasc Imaging. 2019;20(8):883–888. doi:10.1093/ehjci/jey184
Mendez A, Hussain T, Hosseinpour AR, Valverde I. Virtual reality for preoperative planning in large ventricular septal defects. Eur Heart J. 2019;40(13):1092. doi:10.1093/eurheartj/ehy685
E G Milano, E Pajaziti, E Sauvage, A M Taylor, J Marek, K Mortensen, A Cook, S Schievano, M Kostolny, C Capelli, P358 .Taking surgery out of reality: a repair of double outlet right ventricle planned by means of virtual reality, European Heart Journal - Cardiovascular Imaging, Volume 20, Issue Supplement_2, June 2019, jez109.001, https://doi.org/10.1093/ehjci/jez109.001
Kasprzak JD, Pawlowski J, Peruga JZ, Kaminski J, Lipiec P. First-in-man experience with real-time holographic mixed reality display of three-dimensional echocardiography during structural intervention: balloon mitral commissurotomy [published online ahead of print, 2019 Apr 12]. Eur Heart J. 2019;ehz127. doi:10.1093/eurheartj/ehz127
Bruckheimer E, Rotschild C, Dagan T, et al. Computer-generated real-time digital holography: first time use in clinical medical imaging. Eur Heart J Cardiovasc Imaging. 2016;17(8):845–849. doi:10.1093/ehjci/jew087
http://en.medapp.pl/rd-projects/holo-project/. Accessed October 20, 2019
Chevalley, Odile Hélène, et al. "Intensive upper limb neurorehabilitation with virtual reality in chronic stroke: a case report." Annual Meeting of American Society of Neurorehabilitation. No. CONF. 2015.
https://www.livescience.com/34655-human-heart.html. Accessed October 20, 2019.
https://chw.org/newshub/stories/new-virtual-reality-heart. Accessed October 20, 2019.

The content of this article reflects the personal opinion of the author/s and is not necessarily the official position of the European Society of Cardiology.