Accuracy and validity of common dental terms using chat GPT model: a cross-sectional study

INTRODUCTION

In recent years, natural language processing (NLP) and machine learning (ML) techniques have revolutionized the field of question-answering systems.¹ Artificial intelligence (AI) is advancing in medical and biomedical literature. AI has the potential to significantly impact the diagnosis of diseases by improving the accuracy, speed, and efficiency of decision-making. AI algorithms can process vast amounts of data, identify patterns, and make predictions that may be beyond the capabilities of human physicians.² Higher cognitive thinking in AI refers to the ability of AI systems to perform advanced cognitive processes, such as problem-solving, decision-making, reasoning, and perception. This type of thinking goes beyond simple data processing and involves the ability to understand and manipulate abstract concepts, interpret, and use information in a contextually relevant way, and generate new insights based on past experiences and accumulated knowledge. However, it still has some limitations as it lacks the human ability to think creatively, understand emotions, and exhibit ethical judgment.³

With the increasing amount of medical and dental data and complexity of clinical decision-making, NLP tools could theoretically assist dentist in making timely, informed decisions and improve the overall quality and efficiency of healthcare. Chat GPT performed at or near the passing threshold for United States Medical Licensing Exam (USMLE) without any specialized training, suggesting its potential for medical education and clinical decision support.^4,5

Chat GPT is one such artificial intelligence (AI) language model that has been trained to understand the nuances of natural language and provide accurate answers to a wide range of questions.⁶

Chat Generative Pre-Trained Transformer (Chat GPT) is a 175-billion-parameter natural language processing model that uses deep learning algorithms trained on vast amounts of data to generate human-like responses to user prompts.⁷ As a general-purpose dialogic agent, Chat GPT is designed to be able to respond to a wide range of topics, potentially making it a useful tool for customer service, chatbots, and a host of other applications. Since its release, it has garnered significant press for both seemingly incredible feats such as automated generation of responses in the style of Shakespearean sonnets while also failing to answer simple mathematical questions.⁸

Chat GPT is the latest among a class of large language models (LLMs) known as autoregressive language models. Chat GPT builds on Open AI’s previous GPT-3.5 language models with the addition of both supervised and reinforcement learning techniques.⁹ Chat GPT is a direct descendant of Instruct GPT, a fine-tuned version of GPT-3.5 trained on human-derived responses to prompts submitted to the Open AI application programming interface (API) Playground. Instruct GPT was developed by first being tasked to generate a set of responses to a particular prompt and having human annotators label the preferred answer. These preferences are then maximized in a reward model trained using proximal policy optimization, a reinforcement learning algorithm, to tune Instruct GPT. Chat GPT is reported to be specifically trained on conversational prompts to encourage dialogic output.¹⁰

While Chat GPT has shown promising results in solving problems in various fields, its ability to accurately answer first- and second-order knowledge questions in dentistry has not been extensively evaluated. First-order questions are straightforward questions that ask for factual information or seek a direct answer. Second-order questions, on the other hand, are more complex and require higher-level thinking and interpretation. These questions demand analysis or evaluation of information or an opinion or prediction based on evidence.¹¹

Automated question-answering systems have the potential to transform dental education by providing instant and accurate responses to complex questions. The use of Chat GPT as an automated question-answering system in the field of dentistry could aid in the development of effective educational tools that can help medical students learn and understand the subject better. However, before implementing Chat GPT as an educational tool, it is crucial to evaluate its ability to accurately answer first-and second-order knowledge questions related to dentistry.

Further, technology advancements have led to the democratization of knowledge, where patients no longer solely rely on healthcare professionals for medical information. Instead, they are increasingly turning to search engines, and now AI chatbots, as convenient and accessible sources of medical and dental information.

However, despite its potential, Chat GPT often produces seemingly credible but incorrect outputs, thus warranting caution when considering its applications in dental practice and research. The reliability and accuracy of these engines has not been assessed, particularly in the context of open-ended medical questions that dentists and patients are likely to ask.

Hence the aim of this study was to compare the oral health terms generated from chat GPT with that of the standard text book definitions. This will provide an early evidence base on the reliability of Chat GPT for providing accurate and complete information. The results of this study could also shed light on the strengths and weaknesses of Chat GPT as an automated question-answering system with regard. Also, to assess and evaluate the accuracy of dental terms generated by chat GPT model with standard textbook definitions. The study objectives were 1) To assess the responses for 30 common dental terms from WHO oral health assessment form 2013 using chat GPT model, 2) To assess the responses of chat GPT using plagiarism detector software (Plagiarismdetector.net), 3) To compare the responses of the common dental terms from chat GPT and standard textbook using subject expert’s opinion on a Likert scale.

METHODS

Study type, setting, and ethics

This cross-sectional study was conducted in Department of Public Health Dentistry, Ragas Dental College and Hospital, Chennai over the second and third week of February 2023. For data collection, we used personal laptops with institutional Wi-Fi broadband internet connection. A detailed protocol describing the need for the present study and the methodology that would be adopted was prepared and submitted to the Institutional Review Board (IRB) for scrutiny. After scrutiny, the protocol was approved by the IRB without any alterations and suggestions.

Study tool

In the present study we used the current version (January 30, 2023) of Chat GPT (https://chat.openai.com) for generating the solution to higher-order reasoning in dentistry. This version is for trial for the public and research purposes. Chat GPT is capable of responding to complex commands by using its advanced natural language processing capabilities and its vast training data to analyze and understand the input text. The model can generate relevant and meaningful responses to a wide range of questions and commands, including those that are complex in nature.

Questions

We randomly selected 30 dental terms from WHO oral health assessment form 2013 using lottery method. These terms were of a higher order; the answer to the question requires an in-depth knowledge of the subject matter. It focuses on underlying concepts and principles, rather than just rote memorization of facts.

Data collection

The selected 30 dental terms were used as input for the conversation with Chat GPT. Each dental term was entered individually in the chatbot and a definition for the entered term was generated by chat GPT model. All the terms were entered either in a new chat/session. The answer provided by the program was copied into a notepad. It was saved on the computer for further analysis. The data collection ranged from February 5 to February 6, 2023. The questions and collected text were then printed for evaluation by reviewers.

Reliability of Chat GPT

To check the reliability of chat GPT, we evaluated chat GPT generated dental terms for plagiarism detection using a web-crawling plagiarism detection tool which gives an originality score of 0-100%, giving 100% meaning no plagiarism is detected.

Accuracy of Chat GPT

To evaluate the accuracy of chat GPT generated terms, we used standard textbook definition, with subject expert’s opinion, by conducting a blind survey questionnaire among 15 dental teaching staffs of various specialities who have an experience in the field for more than 4 years.

Study participants

The study participants included in this study were current teaching dental facilities of our institution who had a teaching experience of 4 years in various field of dentistry, they were mostly Readers and Professors of various specialities. The selection of the participant’s was based on convenience sampling using lottery method.

Inclusion criteria

All faculty who was present on the day of study, faculty who was willing to participate in the study, and faculty with more than four years of teaching experience were included.

Exclusion criteria

All faculty who was absent on the day of study, faculty who was not willing to participate in the study, and faculty with less than four years of teaching experience were excluded.

Study tool

Chat GPT terms and standard textbook definitions was evaluated for subject expert’s opinion using a self-developed questionnaire. Thus, a survey was conducted among the participants which consists of 30 questions with 3 subdivisions for each question. The questionnaire includes 30 dental terms in which 2 set of definition for an individual term is given. One definition is generated from the chat GPT model and another definition is referred from a standard textbook. First a question comprises of 3 parts, first definition with a Likert scale; second definition with a Likert scale and finally a question stating that which definition among the 2 is more appropriate for the specific dental terms. The definitions are labelled as “DEFINITION 1 and “DEFINTION 2”. For each definition a 5-point Likert scale is given individually (Strongly agree, Agree, Neutral, Disagree, strongly disagree) with scores ranging from of 1 to 5 to assess the quality of the definitions.

The participants will be asked to give an option in the Likert scale for each of the definitions based on their perspective and finally, among the two definitions, one option i.e., option (A) or option (B) is to be selected based on which definition explains the given dental term more accurately (Figure 1).

Statistical analysis

The data were expressed as number, mean, median, standard deviation using descriptive statistical tests. The Kolmogorov-Smirnov test was performed with the dataset, which showed the distribution was not normally distributed. Mann-Whitney U test was to compare the data. For all the statistical analyses, we utilized SPSS version 21 (IBM Inc., Armonk, New York). A p-value of less than 0.05 was regarded as statistically significant.

RESULTS

Accuracy of Chat GPT response checked using plagiarism detector tool. Chat GPT generated definitions were checked for plagiarism by individually entering the definitions in the online plagiarism detector software and checked for plagiarism. It gave an originality score of 0-100% unique, with 100% meaning no plagiarism. It was observed that all selected 30 dental terms showed 100% unique score and no plagiarism was detected for those 30 dental terms.

Frequency distribution of reviewers’ perceptions for accuracy of definitions generated from Chat GPT compared with text book definitions was shown in Table 1. Among 30 dental terms, 13 terms which were generated from the chat GPT model were found to be more appropriate when compared to text book definition.

Comparison of reviewers’ perceptions for accuracy of definitions generated from chat GPT compared with text book definitions was shown in Table 2. Among the 30 dental terms, 9 terms were found to be statistically significant (P<0.05*). A statistically significant results were found for following dental terms: Erosion, abutment, periodontal pocket, loss of attachment, enamel fluorosis, oral candidiasis, hard and soft palate, alveolar ridge and angular cheilitis.

DISCUSSION

Chat GPT is an artificial intelligence (AI) language model that has been trained to process and respond to questions across a wide range of topics. It is also capable of solving problems in medical educational topics. However, the capability of Chat GPT to accurately answer questions in the field of dentistry has not been explored so far. Hence, in this study, we aimed to assess and evaluate the accuracy of dental terms generated by chat GPT model with standard textbook definitions. It was observed that all selected 30 dental terms showed 100% unique score and no plagiarism was detected for those 30 dental terms. Thus, Chat GPT paraphrased the answers to dental definitions with was unique, simple and easily understandable to dental students and lay person. Moreover among 30 dental terms, 13 terms which were generated from the chat GPT model were found to be more appropriate when compared to text book definition.

Numerous past investigations have evaluated the potential of Chat GPT for medical education purposes. The observation is similar to a recent study conducted by Sinha et al. demonstrating the utility of Chat GPT for solving higher-order problems in pathology.¹² The study conducted by Das et al. had a similar outcome where Chat GPT was shown to answer first and second-order questions of microbiology with 80% accuracy.¹³ According to Gilson et al, Chat GPT can provide answers to medical queries through natural language processing, equivalent to that of a third-year medical student in the United States. In addition, they have noted that Chat GPT has the ability to provide reasoning and informative context in most of its responses, which is due to its dialogic nature when answering questions.¹⁵

Additionally, Kung et al conducted a study that discovered that Chat GPT could successfully complete the United States Medical Licensing Examination without any human assistance. Moreover, Chat GPT demonstrated clear, logical thinking and provided accurate clinical insights in its responses.¹⁶ On the other hand, Huh's study discovered that Chat GPT's performance in parasitology is still lacking compared to that of a Korean student.¹⁷ The study by Juhi et al also shows that Chat GPT is only partially reliable in predicting and explaining drug-drug interactions in pharmacology.¹⁸

Hence, we suggest that dental institutions may not restrict the use of AI but train the students to take advantage of it with judicial use. Furthermore, future AI systems must be carefully designed, developed, and validated to ensure they provide accurate and trustworthy information to dental students. Further development of AI especially for health-related information would enhance the capability of AI to be used in education and healthcare systems. The current Chat GPT has limitations in that they have information on 2021. Hence, recent advances may not be available in its output. It is essential that AI systems are monitored and updated regularly to ensure they remain relevant and up to date with the latest advances in the field of dentistry.

In many institutions, self-directed learning (SDL) is practiced where a student collects relevant information from various sources to study a topic. SDL is also an important component of the competency-based curriculum recently introduced in India by the National Medical Commission of India in 2019. This SDL learning approach places medical professionals in charge of their own learning, allowing them to tailor their learning experiences to their individual needs and take ownership of their learning outcomes. By promoting a learner-centred approach, SDL can empower dental professionals to identify their strengths and weaknesses, set goals for improvement, and apply new knowledge and skills to their clinical practice.¹⁹

Chat GPT can assist medical students in self-directed learning by providing access to relevant information and answering questions. Medical students can ask questions about specific diseases, treatments, or procedures. The major advantage of the model is personalized learning it provides for the individual needs of medical students. Chat GPT can provide answers with the aid of literature resources to help students understand the concepts and cross-check them.²⁰

This study has certain limitations like we only evaluated the capability of Chat GPT in answering dental terms available in oral health survey 2013, hence the findings may not be generalizable to other subjects or domains. Secondly, the study only used a single user to converse with Chat GPT conversations with another user at different time points may generate a different response. AI may be able to recognize patterns and classify data, but it lacks the ability to truly understand the underlying meaning and context of information. Although AI can process and analyse large amounts of data, it may not be able to identify the relationships between different pieces of information in any complex dental situation. AI cannot make subjective judgments or ethical evaluations, as it lacks the ability to understand personal values and biases.²¹ AI may be able to generate new information based on existing data, but it cannot create truly original and innovative ideas without human input. Hence, in healthcare and dental education, careful use of technology is needed so that it can facilitate human decisions, not replace them.²²

CONCLUSION

The present study showed that Chat GPT is a potential tool for answering knowledge based questions with equal vigor in the field of dentistry. Moreover, the accuracy of Chat GPT to solve questions in dentistry has a relational level of accuracy. Hence, the text output had connections among its parts to provide a meaningful response. This level of cognition in AI can help students, academicians, and lay persons to get a handy response to their queries. However, as AI, all over the world are evolving its applicability in the field of dentistry should be tested further in future studies.

Funding: No funding sources

Conflict of interest: None declared

Ethical approval: The study was approved by the Institutional Ethics Committee

REFERENCES

1. 1. Johnson D, Goodman R, Patrinely J, Stone C, Zimmerman E, Donald R, et al. Assessing the accuracy and reliability of AI-generated medical responses: an evaluation of the Chat-GPT model. Research Square. 2023;28:2023.
2. 2. Gao CA, Howard FM, Markov NS, Dyer EC, Ramesh S, Luo Y, et al. Comparing scientific abstracts generated by ChatGPT to original abstracts using an artificial intelligence output detector, plagiarism detector, and blinded human reviewers. bioRxiv. 2022:2022-12.
3. 3. Shen Y, Heacock L, Elias J, Hentel KD, Reig B, Shih G, Moy L. ChatGPT and other large language models are double-edged swords. Radiol. 2023;307(2):e230163.
4. 4. Stokel-Walker C. Chat GPT listed as author on research papers: many scientists disapprove. Nature. 2023;613(7945):620-1.
5. 5. Iftikhar L. Docgpt: Impact of chatgpt-3 on health services as a virtual doctor. EC Paediatrics. 2023;12(1):45-55.
6. 6. Khanagar SB, Al-ehaideb A, Maganur PC, Vishwanathaiah S, Patil S, Baeshen HA, et al. Developments, application, and performance of artificial intelligence in dentistry-a systematic review. J Dent Sci. 2021;16(1):508-22.
7. 7. Khanagar SB, Al-Ehaideb A, Vishwanathaiah S, Maganur PC, Patil S, Naik S, et al. Scope and performance of artificial intelligence technology in orthodontic diagnosis, treatment planning, and clinical decision-making-a systematic review. J Dent Sci. 2021;16(1):482-92.
8. 8. Mahmood H, Shaban M, Indave BI, Santos-Silva AR, Rajpoot N, Khurram SA. Use of artificial intelligence in diagnosis of head and neck precancerous and cancerous lesions: a systematic review. Oral Oncol. 2020;110:104885.
9. 9. Kalla D, Smith N. Study and analysis of Chat GPT and its impact on different fields of study. Inter J Innovat Sci Res Technol. 2023;8(3).
10. 10. Hosseini M, Rasmussen LM, Resnik DB. Using AI to write scholarly publications. Accountability Res. 2023:1-9.
11. 11. Thorp HH. ChatGPT is fun, but not an author. Sci. 2023;379(6630):313.
12. 12. Sinha RK, Roy AD, Kumar N, Mondal H, Sinha R. Applicability of ChatGPT in assisting to solve higher order problems in pathology. Cureus. 2023;15(2).
13. 13. Das D, Kumar N, Longjam LA, Sinha R, Roy AD, Mondal H, et al. Assessing the capability of ChatGPT in answering first-and second-order knowledge questions on microbiology as per competency-based medical education curriculum. Cureus. 2023;15(3).
14. 14. Lemons PP, Lemons JD. Questions for assessing higher-order cognitive skills: It's not just Bloom’s. CBE-Life Sci Educat. 2013;12(1):47-58.
15. 15. Gilson A, Safranek CW, Huang T, Socrates V, Chi L, Taylor RA, et al. How does ChatGPT perform on the United States medical licensing examination? The implications of large language models for medical education and knowledge assessment. JMIR Med Educat. 2023;9(1):e45312.
16. 16. Kung TH, Cheatham M, Medenilla A, Sillos C, De Leon L, Elepaño C, et al. Performance of ChatGPT on USMLE: Potential for AI-assisted medical education using large language models. PLoS Digital Health. 2023;2(2):e0000198.
17. 17. Huh S. Are ChatGPT’s knowledge and interpretation ability comparable to those of medical students in Korea for taking a parasitology examination?: a descriptive study. J Educ Eval Health Prof. 2023;20(1).
18. 18. Juhi A, Pipil N, Santra S, Mondal S, Behera JK, Mondal H, et al. The capability of ChatGPT in predicting and explaining common drug-drug interactions. Cureus. 2023;15(3).
19. 19. Patterson C, Crooks D, Lunyk-Child O. A new perspective on competencies for self-directed learning. J Nurs Educat. 2002;41(1):25-31.
20. 20. Bhandari B, Chopra D, Singh K. Self-directed learning: assessment of students' abilities and their perspective. Adv Physiol Educ. 2020;44(3):383-6.
21. 21. Goisauf M, Cano Abadía M. Ethics of AI in radiology: a review of ethical and societal implications. Frontiers in Big Data. 2022;5:850383.
22. 22. Karn A, Priyadarshi A, Roy AD. A review on digitalization of healthcare with SWOC analysis of digital pathology in the backdrop of COVID-19. Global J Res Analysis. 2022;11(7):1-2.