AR is a metaverse technique wherein the external physical world is augmented with a virtual image. An AR system usually consists of image acquisition and three-dimensional (3D) reconstruction, registration, image rendering or display, and human-computer interaction. This section discusses the considerations of each component of AR technology in cardiovascular medicine.

Image Acquisition & Three-dimensional Reconstruction

For most AR technologies used in cardiology, holographic images obtained using conventional imaging techniques are applied to the physical world, such as in surgical or procedural fields. Studies have utilized various medical imaging techniques such as computed tomography (CT), magnetic resonance imaging (MRI), echocardiography, and angiography. However, CT is the most frequently used image acquisition method. Cardiac CT provides excellent visualization of vascular structures, allows 3D reinterpretation, and is frequently used for planning invasive procedures [8]. In several previous studies, cardiac CT has been the most popular imaging technique for acquiring 3D images of hearts [9,10,11,12].

Coronary angiography, an invasive image acquisition method, is the gold standard for evaluating coronary artery disease primarily because of its high spatial and temporal resolution. Two-dimensional (2D) C-arm X-ray images were acquired from various angles to enable geometric understanding [13]. Multiple-view geometry theories have been developed to reconstruct 3D coronary arteries from 2D angiographic images acquired at various angles [14]. Salavitabar et al. evaluated the feasibility of converting 3D rotational angiography images into AR models in patients with congenital heart disease [15]. Interventional cardiologists who participated in the study from a single center perceived the reconstructed models as beneficial in visualizing 3D relationships in anatomical structures and helpful in assisting interventional planning.

Cardiovascular MRIs can also be used for AR. Jang et al. reported a framework for the 3D holographic visualization of myocardial scars on AR, as evidenced by late gadolinium enhancement (LGE) [16]. The endocardial and epicardial contours from the high-resolution LGE data were manually delineated, and the surfaces were subsequently computed using the nearest neighbor method, followed by mesh generation. All surfaces and scar layers were blended with different transparencies and colors to enhance visibility and perception. The AR system also allows user interaction, as shown in Fig. 2 (A, B). The authors suggested that visualizing the complex 3D architecture of the myocardial scar could facilitate MR-guided complex radiofrequency ablation.

Fig. 2

(A) User interaction with the 3D image in an augmented reality environment. (B) Endoscopic view showing the scare from inside of the ventricular chamber. (C) A hologram virtually deployed using TAVIguide™ technology assists in device selection and positioning before the procedure [29]

Registration

The ‘registration’ process in Augmented Reality refers to accurately aligning virtual objects with their desired locations in the real world. In terms of position and scale, registration is required to overlay images in the real world accurately. The cardiac images mentioned above were typically obtained preoperatively. Before preoperative images can be used in intraoperative navigation, registration using the patient’s coordinate system or intraoperative images is required [17]. Approaches to AR registration can be classified into marker-based and marker-less methods. During registration, marker-based methods involve the use of physical devices with visual features as triggers or references for coordinates. In contrast, marker-less methods typically employ localization techniques through the global positioning system (GPS) or various sensors [18].

Studies on AR in cardiology are primarily marker-based, in which identifying markers or user-defined images triggers augmentation. Liu et al. proposed a marker-based registration method and described it in detail. The spine was used as a marker to align the fluoroscopic images with the CT scan geometrically to develop an AR environment for visualizing the real-time position of a catheter within the human body [9]. In a study by De Buck et al. [19], fluoroscopic images were registered to preoperative models using a modified version of the marker-based method proposed in De Buck et al. [20]. This approach identifies ablation sites during surgery by comparing them with a preoperative AR model.

Image Display (Rendering)

AR uses three major display types: HMD, handheld displays, and spatial displays [21, 22]. In most cardiology studies on AR, HMD devices are utilized. Previous studies have used optical see-through HMDs that resemble bulky glasses such as Google Glass [23] and HoloLens (Microsoft Corporation) [9, 24, 25]. The device allows users to view the real world through semi-transparent mirrors that reflect computer-generated images into their eyes to merge real and virtual objects. The video see-through HMD captures a view of the real world using multiple video cameras mounted on the device and directly outputs and combines these with computer-generated images to represent the real world [26]. These two systems have trade-offs regarding system latency, object occlusion, and the accuracy of real-world representation [27]. Cardiac simulation using AR can be performed on handheld devices such as tablets [28]. Bruckheimer et al. reported holographic imaging, a type of spatial display of a virtually deployed transcatheter aortic valve replacement (TAVR) [29].

Multiple triangles and textures represented by meshes must be processed simultaneously for real-time representation to create 3D models. Therefore, using graphics processing unit (GPU) hardware is beneficial. The study by Li et al. [24] employed five GPUs and 4 K monitors arranged in an arc configuration to maximize the visual display.

Human-Computer Interaction

Human-computer interaction (HCI) is a research field that focuses on the ways humans interact with computers and develops novel technologies that allow humans to interact with computers. Keyboard and mouse are well-established input devices commonly used in computing. Touchscreen has enabled evolution from a simple basic phone to widespread adoption of a smartphone.

AR poses unique challenges in human interactions in physical and augmented environments. Hand gestures and voice commands [12] are commonly used instead of keyboard or a mouse. HCI that enables sterility of surgical or interventional fields is an essential component of AR in healthcare.

Static and dynamic hand gestures can be captured using a hand gesture tracking camera [24, 25] or handheld controllers. Hand gestures are intuitive and natural, providing enhanced and immersive user experience. On the other hand, users may need time to learn and remember various hand gestures. Hand gestures may lack the precision of traditional input methods like keyboard and mouse [30]. Voice commands are the most natural way of HCI. Studies have exploited voice commands in image guidance AR systems for cardiovascular intervention [9, 23]. Privacy concerns and misinterpretation are potential drawbacks.

Application of AR in Clinical Cardiology

The lists of clinical studies on AR in clinical cardiology are summarized in Table 1.

Table 1 Clinical studies on augmented reality in cardiologyCoronary Intervention

Coronary angiography and intervention have become standard treatment methods for coronary artery disease [31, 32]. Because the complex coronary anatomy is displayed as 2D images on the monitor, operators should interpret them as 3D structures. Radiocontrast dyes, which can be harmful to patients, are frequently used to differentiate the coronary arteries from other structures. AR can help operators perform coronary intervention more accurately with less use of radiocontrast dyes. Opolski compared conventional and AR-assisted chronic total occlusion interventions [23]. During the procedure, AR images were reconstructed from preprocedural CT images and viewed using wearable computers. The AR-assisted procedure is feasible, safe, and reduces contrast use. While AR-assisted intervention showed longer fluoroscopic times, the radiation dose was similar.

Virtual percutaneous coronary intervention (PCI) is an AR-based approach that simulates the post-PCI quantitative flow ratio (QFR) value, assuming that the specified segment of the treated vessel is successfully dilated. Zhang et al. performed a retrospective analysis of a randomized trial and suggested that a residual QFR-guided PCI strategy might be superior to angiographic guidance alone [33]. The AQVA trial demonstrated the superiority of QFR-based virtual PCI over angiography-based PCI in multivessel coronary artery disease in terms of post-PCI QFR values, without increasing radiation exposure [34]. In an editorial, Bo et al. proposed the need for a large, prospective, multicenter, randomized trial with procedural planning using QFR-based virtual PCI powered by clinical endpoints [35]. AR technology can assist in procedures by offering additional information and reducing the procedure time and the amount of contrast agent.

Vascular Intervention

AR technology can be used in vascular interventions either as fusion imaging or with an HMD device. Fusion imaging guidance from a fluoroscopic navigation system is a convenient way to minimize radiation exposure and reintervention rates during peripheral artery interventions [36]. The “3D road map” using preoperative CT imaging that creates an image overlay to guide aortic endograft is also used in endovascular aneurysm repair (EVAR) [37].

A case report by Lu et al. described an AR navigation system for successful retrograde peroneal access [38]. The AR glasses displayed a 3D structure derived from CT angiography. The system successfully guided the operator in acquiring the needle trajectory for retrograde access. Garcia-Vasquez et al. proposed an AR system to display the position of an electromagnetically tracked catheter tip within a 3D virtual aorta [39]. The authors also developed a prototype to guide EVAR interventions in real time using an electromagnetic tracking system after attaching a sensor to the catheter tip and displaying this information on Microsoft HoloLens glasses [40]. This concept has shown promising progress, but issues related to registration accuracy persist.

Structural Heart Intervention

Structural heart intervention requires an understanding of complex anatomy and is a promising area where AR technology may be helpful for operators. Liu et al. demonstrated the possibility of an image guidance system with AR that provides a 3D visual environment and quantitative feedback on the position of the catheter within the heart [9]. Kiaii et al. conducted several interesting experiments using magnetic tracking sensors: AR-enhanced transesophageal echocardiography for mitral transcatheter edge-to-edge repair procedures [41] and AR guidance for TAVR procedures [42]. Aortic valve commissures were identified more reliably than conventional methods with reduced contrast use. Sadri et al. reported AR guidance for placing a cerebral embolic protection device to reduce the risk of periprocedural stroke during TAVR [43]. Successful placement of cerebral embolic protection was performed in the AR and control groups with no need for aortic angiography, resulting in no contrast use in the AR guidance group.

The number of left atrial appendage occlusion (LAAO) procedures is increasing as the bleeding risk in patients with atrial fibrillation rises. When selecting LAAO devices, an important consideration is the 3D morphology of the left atrial appendage. A case report in 2018 suggested that 3D reconstruction of the heart structure from pre-procedure CT and visualization using AR goggles may help in left atrial appendage occlusion (LAAO) [10]. Efforts have also been made to apply VR or mixed reality (MxR) to LAAO procedures [44, 45]. Using pre-procedural CT for 3D reconstruction, the left atrial appendage is classified into one of four morphology types, and the ostium, landing zone, and depth are measured using VR. While 3D-printed patient-specific left atrial appendage models can also aid in LAAO device sizing [46], AR or VR offers pre-procedural sizing at a much lower cost [47].

Electrophysiology

During electrophysiological procedures, electrical activity is measured using catheters with electrodes. Ablation procedures are performed by applying energy such as radiofrequency waves or cryoablation. Understanding the complex 3D anatomy of the heart is essential to interpreting the vast amount of information and assessing the effects of ablation procedures. The 3D mapping system used during electrophysiological procedures already contains an AR or VR component [48]. The system visualizes the endocardium using a mapping catheter and helps in the procedure by creating geometry and activation maps, which provide detailed anatomical and electrical information.

Additionally, studies have attempted to engage electrophysiological procedures in AR further to create virtual hologram images (Fig. 3). De Buck et al. reported the merging of a 3D anatomical model constructed from MR images with fluoroscopic images in an AR environment, which provided a 3D visualization of the cardiac activation pattern [19]. Prakosa et al. investigated the improvements in the accuracy of catheter ablation using AR with two virtual ventricular tachycardia models. AR-guided catheter navigation showed improved accuracy in one of the two models [49].

Fig. 3

Augmented reality-assisted invasive electrophysiology procedures [63].

CommandEP™ is a MxR system for cardiac electrophysiological procedures that provides a real-time 3D digital image of the cardiac geometry and catheter locations. A human study of 16 patients (with atrioventricular nodal reentrant tachycardia, accessory pathway-mediated tachycardia, and premature ventricular contractions) showed that the system improved ablation points accuracy and visualization [50]. It was also shown that using the system did not increase the total procedure time [51]. The benefits and risks of this system can be demonstrated further by future prospective randomized studies.

Vascular Access

Al-Abcha et al. demonstrated that vascular access obtained using a monocular HMD AR system seamlessly integrates ultrasound and physical visual cues into a consolidated field of view [52]. The MantUS system (Sentiar, Inc.), a product under development, uses a MxR interface to display US images and integrate real-time needle tracking. A trial enrolled physicians, randomized for vascular access using conventional US and MantUS in a preclinical study [53]. This study showed that vascular access improved intervention by providing improved spatial understanding, including improved quality of access.

Cardiovascular Surgery

AR has been applied in many surgical fields, such as orthopedic, visceral, and neurosurgery. However, its use in cardiovascular surgery is still nascent [54,55,56]. As suggested by two case reports, MxR holograms may be helpful as surgical planning tools for complex cardiac surgery. They may have high diagnostic value and contribute to the understanding of complex morphology [57, 58]. Efforts have been made to develop a catheter for AR-guided ultrasound during cardiac surgery [59, 60]. Experience from other surgical fields has suggested the benefits of AR, such as training, preoperative planning, and intraoperative assistance, especially in minimally invasive surgery [61]. It remains to be seen whether AR may prove beneficial in frequently performed cardiac surgeries, such as coronary artery bypass grafting or valve replacement.

AR in Other Purposes

AR potentially enables active engagement in healthcare providers’ training and education [62]. AR-based training provides effective and efficient opportunities for medical professionals to engage in real-world practice [63]. AR can be accessed through mobile applications or remote learning contexts, making it easy for trainees to use. AR education in anatomy is easy to use and results in higher examination scores and enhanced motivation [64]. AR and VR are expected to improve education in clinical practice and more complex procedures.

Additionally, AR can be used to educate patients. It is sometimes difficult to explain and provide information about cardiovascular interventions to patients and their caregivers. AR or VR may make it easier for patients to understand the process by visualizing the pathophysiology of cardiac anatomy [65]. Cardiac rehabilitation using VR or video games increases patient motivation and adherence. Cardiac rehabilitation using AR has not been widely reported, but it is expected to be widely adopted along with VR in the future [66]. Alexander et al. performed a MxR study in patients with acute myocardial infarction to elucidate the mechanisms of lipid-lowering drugs [67]. This improved the patients’ knowledge and drug adherence.

Challenges Ahead

AR often elicits visually induced motion sickness such as headache, nausea, disorientation, and visual discomfort known as “cybersickness”. Cybersickness varies according to the type and degree of immersion [68]. Cybersickness is a significant issue in VR, and concerns regarding AR have increased recently [69]. While VR tends to lead to overt manifestations of motion sickness, AR is generally associated with everyday ailments, such as headaches and eyestrain. A study applied a holographic anatomy program to medical students and found that 15% of students complained of headaches, and an additional 10% reported nausea, eye fatigue, or neck strain [