In this work, we used a prototype of a standalone combined camera–CBCT system (Fig. 1a), which integrates two IR cameras into a mobile CBCT scanner. A first technical characterization of the system has been reported previously [18]. In summary, two Primex 13W IR cameras (OptiTrack, Corvallis, OR, USA) were integrated into the mobile CBCT scanner ImagingRing m (medPhoton, Salzburg, Austria), which is well suited for imaging in the lithotomy position [7, 19, 20]. The cameras emit IR light and receive corresponding reflections, e.g. from passive IR markers, within a field of view (FOV) of 82° in the horizontal and 70° in the vertical direction.

Shown is the standalone combined system integrating a tracking camera directly into the mobile ImagingRing m (a). For implementing needle tracking, we created a corresponding tool of the indicated dimensions (b), which was placed in the main operation area (marked red, c) for future clinical settings. For evaluating tracking accuracy, we inserted needles attached to the tool into a melon and compared the actual needle tip to the predicted (marked red) needle tip (d)

High-quality calibration and camera–CBCT registration are prerequisites for robust tracking. The optimal calibration and fusion procedures for achieving the system’s maximum performance of 0.46 ± 0.28 mm accuracy (for transferring marker positions from the camera into the CBCT coordinate system) have been reported previously [18]. These optimal procedures were also applied to the device prior to the present work. The purpose of this study was to evaluate for the first time the needle tracking and prediction accuracy achievable with the new system.

To predict needle courses on CBCT scans, a rigidly attachable tracking tool was created. The intention was that by tracking this tool fixed to the distal part of a needle and transferring the tool position into the CBCT coordinate system, the corresponding needle tip could be projected into a previously acquired CBCT scan due to the rigid relationships. Our tool consisted of four IR markers (12.7 mm diameter) attached at different heights and distances to ensure asymmetry of the configuration (Fig. 1b). In addition, a clamp allowed a needle to be fixed in position by tightening it with a screw, which eliminated any uncertainties regarding the exact needle–tool connection. Because of the resulting constant offset between the clamp and the markers, the setup appeared suitable for determining needle courses via tracking. In general, the tool was designed to have two finger recesses on the bottom to hold it during implantation. It could either be attached to an already inserted needle or attached to the needle before or during insertion to allow real-time tracking of the dynamic process. The tool was designed using Tinkercad (Autodesk, San Rafael, CA, USA) with the dimensions shown in Fig. 1b and printed on a 3D printer using polyactide (PLA) filament. In the present work, the tool was attached to 18G Trocar point titanium needles (Elekta, Netherlands) of lengths 200 and 160 mm, which represent the longest and median-length needles used in our standard gynecologic brachytherapy workflows.

To implement needle tracking, ground truth models of the tool attached to the studied needles of 200 and 160 mm length were created. Each needle was clamped into the tool and a corresponding CT scan with 0.3 × 0.3 × 1 mm3 voxel size was acquired using a SOMATOM go.Open Pro scanner (Siemens Healthineers, Forchheim, Germany). This conventional CT scanner was chosen for establishing the ground truth due to an improved handling of scatter and beam hardening potentially caused by the IR markers compared to the ImagingRing CBCT. Needle course and marker positions were then determined by thresholding and applying the MATLAB (MathWorks, Natick, MA, USA) “regionprops” function [21], which resulted in the desired ground truth model.

In the context of IR tracking, the position of the needle tip can only be deduced via its rigid relationship to the attached tool, but cannot be directly detected by the cameras due to a missing line-of-sight during/after needle insertion. Therefore, our approach was to create a match of the ground truth tool to the marker positions determined by corresponding IR tracking (note that the tracked marker positions were already available in the CBCT coordinate system due to the previously described [18] camera–CBCT registration). Matching was performed according to the rigid transformation principle [22] implemented in MATLAB (MathWorks, USA). In this principle, the four tracked and ground truth 3D marker positions were each considered as 4 × 3 matrices A and B, respectively, where a and b represent the corresponding centroids of both point clouds. The optimal transformation between the two clouds was determined by computing the covariance cov and its singular value decomposition SVD:

$$\begin cov=_^\left(B_-b\right)\left(A_-a\right)^,\\ \left[U,S,V\right]=SVD\left(cov\right). \end$$

This yielded the best-fit rotation r and translation t for matching the ground truth B onto the tracked markers by \(B'=r\cdot B+t\), where

$$r=VU^,\quad t=a-r\cdot b.$$

By applying this transformation to the ground truth model, including the rigidly attached needle, it became possible to project the needle course and especially the needle tip location into the CBCT coordinate system.

To analyze tool tracking for a stationary scenario, the tool was placed in five different positions within the area where its main operation will be performed in future clinical settings (indicated by the red rectangle in Fig. 1c). At each position, the markers were tracked every 2 s for 5 min. For each snapshot and marker, the positional differences from the individual mean marker positions were calculated. In a second step, to evaluate the effect of dynamic movements, the tool was moved randomly by hand within the camera FOV twice for 8 min. Tracking was performed every 2 s. For each snapshot, the distances between the individual markers were measured and compared with the corresponding distances obtained for the stationary scenario mentioned above. In this way, the inaccuracies introduced by tool movements were determined.

In addition, the agreement of the tracked marker positions (which were already available in the CBCT coordinate system due to the previously described [18] camera–CBCT registration) with the positions determined directly on CBCT scans was investigated. For this purpose, the tool was placed at 10 different locations within the overlapping camera and CBCT FOV. Note that in clinical scenarios, the patient has to be positioned within the CBCT FOV and, therefore, there would be no space to place the tool here as well. However, our procedure was the only way to truly compare tracked markers and markers identified directly via CBCT. At each location, tracking was performed every 2 s for 10 s, and the average marker positions were determined within the CBCT coordinate system. Additionally, CBCT scans with a voxel size of 0.3 × 0.3 × 1 mm3 were acquired and the marker centroids were calculated by thresholding and applying the MATLAB (MathWorks, USA) “regionprops” function [21]. The positions obtained in both ways were compared by calculating the corresponding Euclidean distances as a measure of tracking accuracy.

To evaluate the accuracy of needle tip prediction, the tool with the examined needle clamped (we investigated titanium needles of 200 mm and 160 mm length, as mentioned above) was placed at 10 different positions within the area where its main operation will be performed in future clinical settings (Fig. 1c). At each position, it was tracked every 2 s for 10 s. The respective ground truth was the matched according to the section “Implementation of needle tracking” to the averaged tracked marker positions in order to project the needle course into the CBCT coordinate system. In this way, the corresponding needle tip location was predicted. The Euclidean distance between the matched and averaged positions was calculated as a measure of the match quality.

In addition, a CBCT scan (voxel size of 0.3 × 0.3 × 1 mm3) of the region around the needle tip was acquired for each examined position. On this scan, the needle was manually reconstructed using the treatment planning system Oncentra Brachy (Nucletron, Veenendaal, Netherlands). The Euclidean distance between the predicted and reconstructed needle tip was calculated, both in total and separately for the direction transverse and longitudinal to the needle course.

Two sets of measurements were performed. First, the clamped needles were suspended freely in air. Second, the needles were inserted into a watermelon (Fig. 1d) to simulate implantations potentially associated with needle bending.

As a last step, it was investigated whether the uncertainties in needle tip prediction determined in this work were caused by our experimental setup or were mainly due to the tracking system itself. In addition, our aim was to theoretically determine the uncertainties to be expected in the prediction of the needle tips in clinical practice (without taking needle bending into account). It should be noted that the uncertainty analysis did not address potential effects of needle bending during implantation, as these are considered to be highly operator dependent, and our goal was to assess the maximum prediction accuracy achievable with the new system.

To accomplish this, we performed a simulation of the propagation of individual tracking uncertainties to obtain an estimate of the needle tip prediction uncertainty. First, note that the distance between the camera and the main clinical operation area is about 1.5-times larger than the distance set for the performed camera–CBCT marker comparison (Sect. Analysis of tool tracking), where the tool was placed within the CBCT FOV. This could, in the worst case (i.e., in the case of rotational camera–CBCT registration errors), lead to up to 1.5-times larger positional shifts (compared to the results of Sect. Analysis of tool tracking) between tracked and actual marker locations. This increased tracking uncertainty will be referred to as µa ± σa in the following.

Second, we previously investigated the spatial fidelity of the camera coordinate system that can be achieved by its corresponding calibration [18]. The optimal mean calibration accuracy was determined to µb ± σb = 0.18 ± 0.03 mm, which will also be propagated as respective uncertainty in the tracking workflow. Third, the tracking itself features a statistical position uncertainty µc ± σc, which was experimentally determined in Sect. Analysis of tool tracking as well.

To provide a conservative estimate of the total uncertainties to be expected in needle tip predictions, we considered a propagation of all three individual uncertainties. For this purpose, the ground truth model of the examined needle clamped in the tool was shifted by a random distance (taken from a Gaussian distribution with mean µa and standard deviation σa) in an arbitrary spatial direction. This was done because a camera–CBCT registration error affects all markers approximately the same due to the small tool extent compared to the tool–camera distance. Each marker was then individually shifted in random directions considering another Gaussian distribution (mean µb, standard deviation σb) to account for the calibration inaccuracies and a Gaussian distribution (µc, σc) to account for the tracking uncertainty. This sequence of three shifts was simulated a total of 100,000 times. The ground truth tool model (including the rigidly attached needle) was then matched to each simulated tool according to Sect. Implementation of needle tracking. The Euclidean distances between the resulting and original needle tips were computed in total and separately for the direction transverse and longitudinal to the original needle course. This served as a measure of the needle tip prediction uncertainties inherent to the combined camera–CBCT system.