Evaluating the performance of ChatGPT in preventive and social medicine in medical university examination

INTRODUCTION

The integration of advanced artificial intelligence (AI) language models like ChatGPT into medical education and public health initiatives represents a significant step towards revolutionizing how information is disseminated and utilized in these fields. The collaboration between OpenAI and Microsoft has further expanded the reach of this technology, making it potentially more accessible to a wider audience.^1,2

In the context of medical education, ChatGPT can serve as a valuable tool for students and professionals alike. Its ability to generate coherent and contextually relevant responses can enhance the learning experience by providing instant access to a vast amount of medical knowledge. This can be particularly beneficial in disciplines like preventive and social medicine, where a comprehensive understanding of public health principles is crucial.³

However, it is important to approach the implementation of ChatGPT in medical education with a critical mindset. While the model has impressive capabilities, it also has inherent limitations such as occasional inaccuracies, lack of real-world experience, and the potential for biased outputs. Therefore, educators and learners should use ChatGPT as a supplementary resource rather than a sole reference.⁴

In the realm of public health, ChatGPT can play a pivotal role in empowering individuals and communities to make informed decisions about their health. It can act as a virtual health advisor, offering guidance on preventive measures, healthy lifestyle choices, and general health information. This can be particularly beneficial in reaching populations with limited access to traditional healthcare resources.^5,6

To assess ChatGPT's knowledge in preventive and social medicine, it would be essential to evaluate its accuracy and depth of understanding in topics such as epidemiology, health promotion, disease prevention, and social determinants of health. Additionally, addressing any potential biases in the model's responses and ensuring that the information aligns with current medical guidelines is crucial for its reliable application in public health initiative.

Objectives

This study aims to assess the knowledge and potential applications of ChatGPT, an advanced AI language model, in the field of preventive and social medicine. The focus is on evaluating the model's proficiency in providing accurate and contextually relevant information related to public health principles. By conducting this assessment, the study aims to determine the feasibility and effectiveness of integrating ChatGPT into medical education and public health initiatives, with the ultimate goal of contributing to the improvement of physical and mental health outcomes for the population.

METHODS

ChatGPT 3.5 was used for this study. Our evaluation used university examination question papers, encompassing both part 1 and part 2 of preventive and social medicine in the third year of the Bachelor of medicine and bachelor of surgery (MBBS) course. The examination included various question types such as essays (2), short notes (10), and multiple-choice questions (MCQs) (20), each scored for a maximum of 15, 10 and 1 respectively. The maximum mark was 100. We typed the questions into ChatGPT and recorded the responses generated without making any alterations. For MCQs, we provided the question and answer choices and saved ChatGPT's answers for evaluation by faculty members from the department of community medicine.

RESULTS

The evaluation results indicate that ChatGPT possesses a solid understanding of preventive and social medicine concepts and performed well in providing detailed and suitable answers to the questions. An example of response generated to risk approach in maternal health is provided in Figure 1. In the case of MCQs, ChatGPT also offered explanations for the chosen answers, which enhanced the educational value of the responses.

However, it's worth noting that a few answers lacked completeness according to references in standard textbooks and there was some repetition in content, which might affect the accuracy and comprehensiveness of the information provided. Overall, ChatGPT demonstrated commendable performance, achieving a score of 62% in Paper 1 and 65% in Paper 2. Out of the 20 MCQs in each paper, ChatGPT correctly answered 18 questions in Paper 1 and 14 questions in Paper 2.

DISCUSSION

Our results suggest that ChatGPT possesses a good grasp of the subject matter. However, the scope of this evaluation was limited to a specific set of undergraduate medical exam question papers, and therefore, the findings may not be generalized. To effectively incorporate ChatGPT as a tool to enhance public health, further studies are necessary to validate the accuracy and reliability of generated content and to explore its practical implications. While ChatGPT shows promise in assisting with medical education and public health, several noteworthy drawbacks need to be considered. Despite its capabilities, ChatGPT is not immune to errors or misinformation. The information it provides should be cross-referenced with trusted sources to ensure accuracy, especially in medical and public health contexts where accuracy is critical.⁷ ChatGPT may inadvertently violate privacy and ethical principles in healthcare. It may lack the ability to uphold the same ethical standards as human practitioners and might lack cultural sensitivity, which is crucial in public health initiatives.⁸ ChatGPT should not replace the role of healthcare professionals, epidemiologists, or public health specialists. It cannot solely diagnose or provide personalized medical advice and may lack the nuance required for complex health issues.⁹

ChatGPT may struggle to understand the unique socio-cultural, economic, and political contexts of different regions and populations. Preventive and social medicine often requires tailoring interventions to specific communities, which ChatGPT may not fully grasp. As observed in our evaluation, ChatGPT occasionally provided repetitive or incomplete answers, which can be frustrating for users seeking comprehensive information.

The utilization of ChatGPT in medical education and public health has promising implications. It could serve as a supplementary tool for students to gain insights and explanations in their study of preventive and social medicine. Moreover, ChatGPT can potentially provide accessible and instant information to the public on various health-related topics, promoting health literacy and informed decision-making. However, caution must be exercised, as the accuracy of the generated content is subject to limitations. The repetition and lack of completeness in some responses raise concerns about the reliability of ChatGPT in certain contexts. As it stands, ChatGPT should be considered a supportive resource rather than a sole authoritative source in medical education and public health.

To harness the full potential of ChatGPT in these domains, future studies should focus on refining its responses, addressing accuracy concerns, and evaluating its effectiveness in real-world educational and healthcare settings. These investigations can help ensure that ChatGPT truly enhances the quality of medical education and positively impacts public health outcomes.

CONCLUSION

The integration of ChatGPT into medical education and public health holds great promise for transforming the way information is disseminated and healthcare decisions are made. However, it is imperative to approach this technology with a vigilant mindset, acknowledging and addressing its limitations. As we aspire to harness the full potential of ChatGPT, future studies should focus on refining the model's responses, ensuring accuracy, and evaluating its practical effectiveness in real-world settings. Through continuous improvement and validation efforts, we can optimize ChatGPT to truly enhance the quality of medical education and positively impact public health outcomes. This iterative process will contribute to the responsible integration of AI technologies, fostering a more informed and healthier society.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: Not required

REFERENCES

1. 1. Baidoo-Anu D, Ansah LO. Education in the Era of Generative Artificial Intelligence (AI): Understanding the Potential Benefits of ChatGPT in Promoting Teaching and Learning. Available at: https://ssrn.com/abstract=4337484 or http://dx.doi.org/10.2139/ssrn.4337484. Accessed on 26 March 2023.
2. 2. Microsoft announces new Bing and Edge browser powered by upgraded ChatGPT AI. Available at: https://www.theverge.com/2023/2/7/23587454/microsoft-bing-edge-chatgpt-ai. Accessed on 26 March 2023.
3. 3. Khan RA, Jawaid M, Khan AR, Sajjad M. ChatGPT - Reshaping medical education and clinical management. Pak J Med Sci. 2023;39(2):605-7.
4. 4. Lee H. The rise of ChatGPT: Exploring its potential in medical education. Anat Sci Educ. 2023;14.
5. 5. Biswas SS. Role of Chat GPT in Public Health. Ann Biomed Eng. 2023;51(5):868-9.
6. 6. Kahambing JG. ChatGPT, public health communication and 'intelligent patient companionship'. J Public Health (Oxf). 2023;45(3):e590.
7. 7. Cascella M, Montomoli J, Bellini V, Bignami E. Evaluating the Feasibility of ChatGPT in Healthcare: An Analysis of Multiple Clinical and Research Scenarios. J Med Syst. 2023;47(1):33.
8. 8. Dave T, Athaluri SA, Singh S. ChatGPT in medicine: an overview of its applications, advantages, limitations, future prospects, and ethical considerations. Front Artif Intell. 2023;6:1169595.
9. 9. Liu J, Wang C, Liu S. Utility of ChatGPT in Clinical Practice. J Med Internet Res. 2023;25:e48568.