In this study, we investigated the performance of ChatGPT-3.5 and ChatGPT-4 on basic healthcare leadership and management questions in three different areas of healthcare management and leadership. For that issue, we used a novel scoring system to evaluate the performance of the two chatbot versions.

Hospitals worldwide are confronted with escalating demands in times of limited resources [11], and the COVID-19 pandemic intensified these issues [12]. For today’s hospital department leaders, special skills, characteristics, and behavioral competencies [12, 13] are needed to address workforce challenges, changing consumer expectations, increasing demands for care, and increasing levels of quality and safety of patient care [14]. Mumford et al. suggest four main categories of skills that healthcare leaders should comprise: cognitive skills, interpersonal skills, business skills, and strategic skills [15].

Effective clinical leadership is associated with the hospital’s performance and has to ensure a high quality of safe and efficient patient care [14]. Therefore, it is of high importance for aspiring hospital department leaders to gain leadership skills. Many HCPs are unfamiliar with gaining leadership skills apart from the classical way of earning a master’s degree in economics or business administration. However, there is a positive correlation between the presence of HCPs in hospital leadership positions and social performance and community benefits [16, 17]. Here comes the role of ChatGPT, which could fill the gap. First, ChatGPT could help access leadership/management training by providing information about available programs and resources. Second, ChatGPT could be asked for solutions on different hospital leadership and management levels, such as management basics or sophisticated leadership/management issues that aspiring department leaders could ask. However, ChatGPT cannot replace sophisticated leadership and management training, often requiring business school attendance and leadership/management experience.

Although ChatGPT is quite a novel application, there are many already performed and ongoing studies about the potential benefit of ChatGPT in medicine and education [7, 18]. A recent study evaluated the performance of ChatGPT on the United States Medical Licensing Examination (USMLE). The study revealed that ChatGPT passed all three exams (Step 1, Step 2 CK, Step 3) near the passing threshold without any previous training [19] and was comparable to a third-year medical student [20]. In a case study by Jeblick et al., radiologists evaluated the quality of simplified radiology reports generated with ChatGPT. The results showed that the reports were “correct, complete, and not potentially harmful to patients” [21].

We used a novel quality scoring system developed for this study. This scoring system could also be used for other LLM-based applications. A score of at least 3 in the overall analysis or per answer indicates a valuable source of education. The answers provided by ChatGPT-3.5 and ChatGPT-4 showed good and very good mean scores and did not present any inaccurate results. Only a few answers offered by ChatGPT-3.5 and ChatGPT-4 were challenging to learn. Also, only one answer provided by ChatGPT-3.5 was rated insufficiently, making ChatGPT a valuable source for accessing hospital leadership/management education and gaining information about that topic.

However, in a direct comparison of both chatbot versions, ChatGPT-4 showed a higher performance than ChatGPT-3.5 and should, therefore, primarily be considered the superior source for accessing leadership and management education. This trend also indicates that accessing leadership/management education could be significantly improved with further chatbot development beyond ChatGPT.

Despite the enormous potential of ChatGPT, challenges and limitations remain and should be taken seriously, also for leadership/management education and knowledge forming. One major limitation of ChatGPT is that it does not standardly present references. This issue becomes problematic when HCPs want to dive deeper into each topic. HCPs should also be cautious, as even OpenAI states that ChatGPT could sometimes “respond to harmful instructions” [22, 23]. Another issue is plagiarism, which was unveiled by different studies [24, 25]. Also, data privacy and legal issues remain and need major approaches to be solved in the future [26].

ChatGPT is a good point at which to start the journey, as leadership is required at all medical levels. However, anyone has to find their leadership style, as Steve Jobs once showed us with his unconventional but highly effective leadership style [27].

Our study concept has some limitations itself. First, we performed a simple question-and-answer scheme for our research. Second, we did not investigate the variation of ChatGPT's answers to the same question. Third, we did not check the sources used by ChatGPT to provide the answers, as references were not provided standardly.