We aimed to describe three different AODS algorithms for the detection of pathogen-related clusters and describe their utility. We show the performances of the systems across four different case studies: the introduction of a new pathogen of interest with subsequent low-level endemicity, an endemic organism, slow rise in numbers of an endemic organism, and sporadic occurrence of an organism, and compared their outcomes.

Strikingly, we found a paucity of alerts from the WHONET-SaTScan software as compared to CLAR and the P75 system. However, the clusters that were picked up by WHONET-SaTScan seemed to be worth investigating. This may help reduce the overall number of alerts and allow clinicians to focus on the statistically relevant cluster alerts and reduce user fatigue. For sporadic organisms WHONET-SaTScan did not generate any alerts. There was congruency seen for all three systems for endemic pathogens. However, WHONET-SaTScan did not detect an increase in cases when this happened gradually, as in the case study of Aspergillus fumigatus in the adult ICU, while this was constantly alerted by both the CLAR and the P75 systems. Of note, this slow rise in Aspergillus fumigatus was detected in the adult ICU since the start of the COVID-19 pandemic, postulated reasons could include COVID-19 associated pulmonary aspergillosis or heightened detection by clinicians due to increased awareness brought on by the pandemic.

Our study is the first to compare the performance of three different outbreak detection methods across four distinct pathogen occurrence case studies for both endemic and sporadic organisms. Our findings align with previous research, indicating that both the CLAR and the WHONET-SaTScan system can detect relevant clusters [16, 21]. The CLAR system can even detect clusters for sporadic pathogens [6], because of its high sensitivity. The trade-off of this increased sensitivity is lower specificity. It is probable that a large portion of increases in numbers, that may trigger an alarm in the P75 and CLAR system, are random events, rather than true outbreaks. In contrast, WHONET-SaTScan can return only the significant outbreaks and ignores small clusters, but the risk of missing a true outbreak, or detecting it too late (when the outbreak is already at its peak), is greater. In any case, a manual verification of the results by infection prevention practitioners remains crucial [22]. The choice of which system to implement should be guided by the specific goals and priorities of the healthcare facility. Therefore, the next step should incorporate an assessment of clinical relevance of the alerts, which can then help to define a best fit system for use by the institution.

In the process of implementing any AODS, it is evident that data pre-processing is considered by institutions before adoption of any system. Both the CLAR and the P75 system require curated data and can only identify clusters of micro-organism-phenotype combinations that are pre-specified. Furthermore, the P75 method requires manual yearly threshold updates, this may not always happen due to resource constraints. In contrast, WHONET-SaTScan has the advantage of analyzing extensive datasets and detecting emerging pathogens within the hospital ecosystem. For CLAR and P75 unfortunately this is not the case, as data needs to be curated prior to being run by the AODS, hence if a novel organism is not specified in the pre-processing phase – this will be missed by both systems. However, due to its comprehensive dataset analysis, WHONET-SaTScan required longer run-times compared to the other two systems but required less curated input data. In addition, it is recognized that in each approach there is the risk that (recent) outbreaks influence future thresholds as the background incidence increases. Finally, it may be considered to improve detection systems by adding a generic rule for new (and hence rare) micro-organism-phenotype combinations based on a simple rule of two or more cases in a 30-day period.

While this study provided valuable insights for infection prevention practitioners who would like to use automated systems for cluster detection, or evaluate their current system, there are some limitations to this study. The analyzed dataset excluded individuals who objected to the use of their medical data. Since the number of such objections was low, this did not significantly impact the final outcomes but may do so if this number continues to rise. A second limitation of this study is that WHONET-SaTScan analyzed the data on a day-by-day basis, while the P75 and CLAR systems both conducted retrospective analysis on the entire dataset. However, it is worth noting that in real-time implementation that the P75 and CLAR systems can be configured to run daily analyses. As this was a retrospective study, time to identification was not studied but should be included in future studies as this would impact upon infection prevention processes whereby timely intervention is key. Denominator information was not incorporated in the analyses, and this could potentially affect cluster analysis with fluctuations in overall patient numbers, this lack of denominator data is current practice in many systems. Differences in endemic and sporadic organisms may need to be factored into the AODS methods, as sporadic species can potentially lower the threshold and increase the number of false positive alerts should there be long periods without occurrences. Likewise, clusters among endemic microorganisms can be missed when the numbers do not exceed the calculated or stipulated thresholds. Data output across all three systems exhibited heterogeneity. To facilitate a fair comparison, we made assumptions regarding maximum cluster length and minimum cluster size. These assumptions may differ among institutions, and our results might not be generalizable to those institutions. Therefore, the assumptions should be tested prior to adoption of each system for every institution. Lastly, with regards to implementation of WHONET SaTScan, although the freeware has improved usability over the original SaTScan, it is less adaptable. Therefore, exploration of the dataset with the original SaTScan package may be important to consider for future studies.

Future studies may improve these methods by including groups of wards that are closely related as epidemiological units and account for difference in underlying number of patient days over time. As this is a retrospective cohort study, real-time assessment of each system will need to be performed to better understand implementation difficulties, technical and data storage issues, and relevance to infection prevention practitioners [23]. In addition, simulation studies may help to investigate the performance of the automated cluster outbreak systems as early warning systems. NGS can serve as a valuable tool to determine clonality within the clusters detected by AODS, and may serve as a gold standard for the comparison of such systems in the future. Integrating NGS data into cluster analysis can enhance the accuracy and reliability of cluster detection, offering insights into the genetic relatedness of microbial isolates within detected clusters other than species-based definitions. For example, transmission of a resistance element (e.g., on a plasmid) may occur in multiple species, and such a transmission event would not be identified using species-based definitions. Moreover, AODS cannot differentiate between hospital acquired infections and community acquired infections, therefore patients within a potential cluster alerted by an AODS can be unrelated [22]. AODS can also help prioritize isolates for NGS testing especially working within real world limitations such as availability, costs, and turnaround time. As such, the choice for a cluster alert system should be tailored to the specific needs of an individual institution. Instead of entirely replacing an existing system, a possible approach is to introduce an additional system as a complementary solution, addressing any shortcomings of the existing one.