Controversies and disagreements may arise between dental and medical communities or between different areas of the same medical specialty. This may be particularly the case when the statements presented are supported by evidence but have not been evaluated for quality and risk of bias. In the earlier mentioned controversy in Germany, the paediatricians provided evidence for the high efficacy of fluoride tablets in caries prophylaxis and recommended their use in early childhood, while the paediatric dentists recommended low concentrated fluoridated toothpaste instead of tablets and presented evidence to support their recommendations. Therefore, despite having evidence background for each recommendation, the reliability of the underlying literature should be thoroughly assessed to reach consensus. Although the level of evidence is essential to assess the reliability of results of studies, but it is not sufficient, because an RCT or a systematic review of RCTs has a high evidence level, but might have high risk of bias, which is a systematic error or deviation from the truth that may distort the results (Higgins et al. 2020). A correct evaluation of the risk of bias can signalize the reliability of the results, which may improve the quality of dental treatments (Faggion 2015). The evaluation of the risk of bias in the evidence underlying the recommendations is one of the factors, that influences the strengths of recommendations and was performed in this study.

As recommended by the Cochrane Collaboration, tools for Risk of Bias assessment should not use scales and scores and may rather use the terms “low risk”, “high risk”'', or “unclear risk”' for the assessment (Lundh and Gotzsche 2008; Higgins et al. 2020). The three tools used in this study fulfilled this criterion. (1) The first tool was RoB 2, which is a Revised Cochrane risk-of-bias tool for randomized trials. The tool was found to be quite simple and easy to use with its detailed guidance. (2) The ROBINS-I tool was the second tool, which was developed by members of the Cochrane Bias Methods Group and the Cochrane Non-Randomized Studies of Interventions Methods Group and is based on the concepts of the Cochrane RoB tool for randomized trials. Although the ROBINS-I tool provides a useful method to assess the risk of bias in non-randomized trials, many reports have criticized it because of the difficulty in its application, the weak guidance, and poor reliability (Sun et al. 2018; Thomson et al. 2018; Minozzi et al. 2019). However, the ROBINS-I tool provides a substantial improvement over the formerly widely used Newcastle–Ottawa Scale, which has also been sharply criticized (Stang 2010; Huffman and Thomas 2018). A modification on this tool called ROBINS-E for trials of exposure is developed and newly updated (Morgan et al. 2019), but it was at the time of assessments in this study only a preliminary tool and was criticized for not meeting international standards for evaluating human observational studies (Bero et al. 2018). In comparison to other available tools for risk of bias assessment of non-randomized trials, the use of ROBINS-I might be acceptable (Schünemann et al. 2019). (3) The third tool (ROBIS) is reported to be the first tool that is specially designed for the risk of bias assessment of systematic reviews (Hu et al. 2018). Moreover, this tool is recently increasingly used in the literature and as found in our study is reported to be reliable and easy to use (Buhn et al. 2017).

It is noteworthy that although all the assessed studies are published before the development of the used risk of bias assessment tools, the studies varied from low to high risk of bias in the assessments. This shows that also before the recently increased attention to risk of bias, well performed older studies managed to control the risk of bias without having these modern assessment tools to aid them in avoiding bias. Future studies should use the opportunity of having such tools to guide them in minimizing the risk of bias in the planning stages.

Unfortunately, but not surprisingly, 44.8% of the assessed RCTs showed high risk of bias (n = 13). Previous studies assessing risk of bias even in RCTs mostly reported similar results (Papageorgiou et al. 2015; Elangovan et al. 2016). This underlines the necessity of improving the quality of study design and reporting of RCTs because biased results of RCTs tend to increase the effect size estimation and raise suspicion about their reliability (Page et al. 2016; Saltaji et al. 2018).

None of the included non-randomized trials was assessed to have low or moderate risk of bias. This is mostly due to the difficulty of avoiding risk of bias in such study designs in general. While a randomized study design should be considered for proving the efficiency of therapeutic and preventive clinical measurements, it is in many cases not possible to randomize the participants, especially in matters such as the caries prevention effect of fluoridated salts or fluoridated water supply. Therefore, when it is not avoidable to include non-randomized trials in systematic reviews or in clinical guidelines and recommendations, the included trials should be assessed for risk of bias, and their results should be interpreted with caution (Schünemann et al. 2019).

Systematic reviews are mostly considered of high level in the medical field. However, this should not be taken for granted, as there are many components, that affect the quality and the reliability of systematic reviews. Risk of bias of systematic reviews is one of these components and should always be considered when interpreting the results of a systematic review. In this review, 68.8% of the included systematic reviews were assessed with a low risk of bias (n = 11), including all the six assessed “’Cochrane reviews’’, which agrees with recent systematic and umbrella reviews, where all the included “’Cochrane reviews’’ had always low risk of bias compared to other systematic reviews (Faggion et al. 2015; BaniHani et al. 2021). Two systematic reviews did not report the specific search methods or an overview of the included studies in detail and had, therefore, a high risk of bias.

The findings of the overall risk of bias assessments of the cited papers in the 2013 German guidelines for the use of fluoride in caries prevention showed a high risk of bias for nearly half of the papers regardless of their study design (48.3%; n = 28). This result agrees with a previous review, that reported a high percentage of risk of bias in cited papers of the American Heart Association guidelines (Cho et al. 2019). Authors of clinical trials and systematic reviews as well as clinical guidelines should be therefore encouraged to consider risk of bias during planning to ensure a good reliability of their results and to raise the quality of evidence to a trustworthy level (Bradley et al. 2020).

Regarding risk of bias in the papers cited in different sections of the German fluoride guidelines, it was found that the percentage of cited papers having a low or moderate risk of bias was high in the sections concerning fluoridated mouthwash and fluoridated gel. On the other hand, the sections regarding fluoride tablets, fluoridated toothpastes, and the paediatrician’s recommendations were based on papers, from which more than 50% had a high risk of bias (85.7%, 54.5%, and 71.4% respectively). This could explain the failure to reach consensus regarding the use of fluoride for caries prevention in early childhood, where the paediatricians recommended the use of fluoride tablets or lozenges, while the paediatric dentists recommended the use of low concentrated fluoridated toothpaste starting with the eruption of the first primary tooth. A further look in the recommendations of the guidelines showed that 13 out of 20 recommendations and statements could not be supported with evidence of low or moderate risk of bias, many of which were in these three controversial sections. Considering risk of bias of underlying literature in forming the recommendations would probably change the outcome or the strengths of recommendations, which results in avoidance of confusion for the clinicians or the patients, due to discrepancies in the statements.

This study, as any other study has its limitations, which should be considered when interpreting the results; 22 cited papers were excluded from risk of bias assessment due to different reasons, which are mentioned in appendix 1. The exclusion of four studies, where the full text was not found might have an effect on the overall assessment of the recommendations. As these studies are almost equally distributed between the sections of the guidelines, the impact of the exclusion on the overall results and conclusions of this study is thought to be minimal and rather neutral, but could still be possible. Moreover, although the assessors had sufficient time and experience to understand and train on the use of the risk of bias assessment tools, a formal training or piloting was not performed, and should be considered in future similar studies. However, all assessors are members of teaching staff of universities and have the knowledge as well as the ability to perform such assessments. Moreover, the fact, that a total of four researchers participated in the assessments of risk of bias, should eliminate any performance bias or personal preferences in the assessments.

As the results of this review show, clinical guidelines should not be taken for granted, without a critical consideration of their methodology. It is essential to assess the risk of bias of cited papers before providing updates on clinical recommendations and guidelines that would likely affect clinical decisions of practitioners. It may be, therefore, advantageous to follow the AGREE checklist (Brouwers et al. 2016) and/or the GRADE methodology (Guyatt et al. 2008) in the process of guideline development, as the EAPD guidelines for the use of Fluoride in caries prevention did, where not only GRADE ratings were used, but also the recommendations were categorised in STRONG and CONDITIONAL for patients, clinicians and policy makers (Toumba et al. 2019). This will ensure a sound methodology and deliver clear recommendations without dropping any major aspects such as risk of bias assessments. Using specially developed digital software for this purpose such as GRADEpro (GDT 2022) or MAGICAPP (2022) may also increase the transparency during development to have guidelines with high quality and trustworthy recommendations.

It is noteworthy to state, that within the preparation of this paper, a consensus in Germany was announced and a new unified recommendation from the paediatricians and the dentists regarding caries prevention in early childhood was published (Berg et al. 2021).