Cardiovascular magnetic resonance (CMR) imaging has become an indispensable tool in the diagnosis, treatment planning, and management of cardiovascular diseases. A critical component of advanced cardiac imaging is the extraction of accurate 3D meshes from CMR images. These meshes serve as the foundation for various applications, including computational simulations (Fedele and Quarteroni, 2021), biomarker discovery (Bonazzola et al., 2021), and analysis of heart deformation and dynamics (Beetz et al., 2022).

Despite its importance, cardiac mesh extraction remains a challenging task. Traditional methods, such as active shape models (Ordas et al., 2007) and multi-atlas segmentation (Bai et al., 2015), often require extensive computational resources and can be time-consuming. The inherent variability in heart shapes, sizes, and pathologies further complicates the extraction process, necessitating robust and adaptable methods.

Traditional mesh generation pipelines are complex, involving multiple steps and often requiring manual interventions. Fig. 1 and Table 1 illustrate this complexity, comparing different mesh generation approaches and highlighting the numerous steps, algorithms, and manual interventions typically required in common pipelines.

A particular challenge lies in transitioning from 2D image slices to a cohesive 3D representation, especially when modelling tetrahedral meshes. Current methodologies often require intricate post-processing steps to refine the meshes and make them suitable for simulations (Fedele and Quarteroni, 2021, Neic et al., 2020). These additional steps can introduce errors and prolong the overall processing time.

Existing approaches to cardiac mesh generation can be broadly categorised into two main strategies. The first strategy follows a multi-stage pipeline that begins with voxel-level segmentation using techniques like U-Net (Ronneberger et al., 2015) or V-Net (Milletari et al., 2016), followed by surface mesh extraction and volumetric mesh generation (Fedele and Quarteroni, 2021, Neic et al., 2020, Kim et al., 2018, Väänänen et al., 2019, Pak et al., 2024). However, this approach often introduces errors at each stage—segmentation models can produce unrealistic masks with holes or artifacts (Larrazabal et al., 2020), and the subsequent mesh generation steps can compound these errors. Recently, Chen et al. (2021) proposed MR-Net, which improves this pipeline by using a deep learning-based dense segmentation-to-point cloud registration approach. While MR-Net achieves faster inference times compared to traditional registration methods, it still relies on an initial segmentation step and is therefore limited by the quality of these segmentations. Other methods attempt to improve this pipeline by estimating mesh node displacements (Puyol-Anton et al., 2017, Pak et al., 2021) or by deforming a simulation-ready template (Kong and Shadden, 2021), but their accuracy remains constrained by the quality of the estimated deformations.

The second strategy aims to bypass these intermediate steps by generating meshes directly from images. Recent work has explored end-to-end neural networks that use convolutional architectures to estimate parameterized shapes (Tóthová et al., 2020, Xia et al., 2022). While these methods are promising, they typically rely on Principal Component Analysis (PCA) shape models, which are inherently limited by their linear nature and struggle to capture the full complexity of cardiac structures. More recent advances have focused on developing alternative approaches that learn to deform template meshes directly from medical images through various techniques such as differentiable mesh voxelization, graph convolutional networks, and mesh-based motion tracking (Joyce et al., 2022, Kong and Shadden, 2023, Meng et al., 2023). Although these methods successfully eliminate complex multi-step processing pipelines, they remain constrained by their reliance on template geometries and deformation field estimations, potentially limiting their ability to capture patient-specific anatomical variations.

We propose HybridVNet, a novel architecture that advances the direct image-to-mesh approach by combining the strengths of volumetric image processing and geometric deep learning. We work under the hypothesis that direct generation can improve the accuracy of the resulting meshes while being computationally efficient. Motivated by this hypothesis, our method produces high-quality surface and volumetric meshes directly from CMR images through an end-to-end learning approach. Unlike previous methods, HybridVNet uses a hybrid architecture that combines standard 3D convolutions for volumetric image encoding with a spectral graph convolutional decoder for mesh generation. This combination allows us to better capture both global anatomical context and local geometric details, producing meshes that are immediately suitable for computational models without requiring additional processing steps. Notably, while our primary contribution is in direct mesh generation, our experiments also demonstrate that HybridVNet significantly outperforms existing segmentation-to-mesh pipelines, including MR-Net, achieving substantially lower reconstruction errors and better mesh quality.

Contributions: Our primary contributions encompass the development of HybridVNet, a multi-view volumetric hybrid graph convolutional model capable of seamlessly integrating multiple CMR views within a jointly learned latent space, directly producing meshes from images. Our model exhibits versatility in creating both cardiac surface and tetrahedral meshes which could potentially be employed for finite element simulations, both from images or segmentations as input. We explore classic regularisation techniques for surface meshes and introduce a novel differentiable regularisation term specifically tailored for tetrahedral meshes, markedly enhancing element quality. Notably, while previous works often relied on cropped regions of volumetric images, our model demonstrates exceptional performance in both cropped areas and complete images, showcasing its robustness and adaptability. The performance of HybridVNet is evaluated using the UK Biobank CMR dataset (Petersen et al., 2015), providing a comprehensive assessment in the context of cardiac imaging.