ChatGPT: is it able to guide parents and caregivers on paediatric tonsillectomy-related questions?
DOI:
https://doi.org/10.34631/sporl.2209Keywords:
ChatGPT, OpenAI, Artificial intelligence, Pediatric tonsillectomy, Parents and caregivers questionsAbstract
Aim: To assess the accuracy of ChatGPT in answering parents and caregivers’ questions about pediatric tonsillectomy, comparing its versions 3.5 and 4.0.
Study Design: Instrument validation study of ChatGPT in answering questions about pediatric tonsillectomy.
Material & Methods: We prompted ChatGPT versions 3.5 and 4.0 with 21 questions. The answers were assessed in accordance with the latest American Academy of Otolaryngology’s guideline on Tonsillectomy in Children. The assessment of the responses generated by the two versions were compared using McNemar's test (exact version).
Results: Of the 21 ChatGPT-generated answers, 13 (61.9%) were deemed accurate using version 3.5, and 19 (90.5%) using version 4.0 (p=0.031). The inter-rater agreement was very good - Cohen’s Kappa=0.97 (v3.5) and 0.83 (v4.0).
Conclusion: ChatGPT version 3.5 should not be regarded as a sufficiently accurate tool for guiding caregivers on pediatric tonsillectomy-related questions. Version 4.0 seems to be a significantly more reliable tool.
References
De Angelis L, Baglivo F, Arzilli G, Privitera GP, Ferragina P, Tozzi AE, et al. ChatGPT and the rise of large language models: the new AI-driven infodemic threat in public health. Front Public Health. 2023 Apr 25:11:1166120. doi: 10.3389/fpubh.2023.1166120.
Roumeliotis KI, Tselikas ND. ChatGPT and open-AI models: a preliminary review. Future Internet. 2023; 15(6):192. https://doi.org/10.3390/fi15060192
Carlbring P, Hadjistavropoulos H, Kleiboer A, Andersson G. A new era in Internet interventions: the advent of Chat-GPT and AI-assisted therapist guidance. Internet Interv. 2023 Apr 11:32:100621. doi: 10.1016/j.invent.2023.100621.
Vaishya R, Misra A, Vaish A. ChatGPT: Is this version good for healthcare and research? Diabetes Metab Syndr. 2023 Apr;17(4):102744. doi: 10.1016/j.dsx.2023.102744.
Liévin V, Hother CE, Motzfeldt AG, Winther O.Can large language models reason about medical questions? Patterns (N Y). 2024 Mar 1;5(3):100943. doi: 10.1016/j.patter.2024.100943
Qu RW, Qureshi U, Petersen G, Lee SC. Diagnostic and management applications of ChatGPT in structured otolaryngology clinical scenarios. OTO Open. 2023 Aug 22;7(3):e67. doi: 10.1002/oto2.67
Hoch CC, Wollenberg B, Lüers JC, Knoedler S, Knoedler L, Frank K. et al. ChatGPT's quiz skills in different otolaryngology subspecialties: an analysis of 2576 single-choice and multiple-choice board certification preparation questions. Eur Arch Otorhinolaryngol. 2023 Sep;280(9):4271-4278. doi: 10.1007/s00405-023-08051-4.
Long C, Lowe K, Zhang J, Santos AD, Alanazi A, O'Brien D. et al. A novel evaluation model for assessing ChatGPT on otolaryngology-head and neck surgery certification examinations: performance study. JMIR Med Educ. 2024 Jan 16:10:e49970. doi: 10.2196/49970.
Radulesco T, Saibene AM, Michel J, Vaira LA, Lechien JR. ChatGPT-4 performance in rhinology: A clinical case series. Int Forum Allergy Rhinol. 2024 Jan 24. doi: 10.1002/alr.23323.
Lechien JR, Gorton A, Robertson J, Vaira LA. Is ChatGPT-4 accurate in proofread a manuscript in otolaryngology-head and neck surgery? Otolaryngol Head Neck Surg. 2023 Sep 17. doi: 10.1002/ohn.526.
Nachalon Y, Broer M, Nativ-Zeltzer N. Using ChatGPT to generate research ideas in dysphagia: a pilot study. Dysphagia. 2024 Jun;39(3):407-411. doi: 10.1007/s00455-023-10623-9.
Frosolini A, Franz L, Benedetti S, Vaira LA, de Filippis C, Gennaro P. et al. Assessing the accuracy of ChatGPT references in head and neck and ENT disciplines. Eur Arch Otorhinolaryngol. 2023 Nov;280(11):5129-5133. doi: 10.1007/s00405-023-08205-4.
Lechien JR, Briganti G, Vaira LA. Accuracy of ChatGPT-3.5 and -4 in providing scientific references in otolaryngology–head and neck surgery. Eur Arch Otorhinolaryngol. 2024 Apr;281(4):2159-2165. doi: 10.1007/s00405-023-08441-8.
Ayoub NF, Lee YJ, Grimm D, Balakrishnan K. Comparison between ChatGPT and Google search as sources of postoperative patient instructions. JAMA Otolaryngol Head Neck Surg. 2023 Jun 1;149(6):556-558. doi: 10.1001/jamaoto.2023.0704.
Moise A, Centomo-Bozzo A, Orishchak O, Alnoury MK, Daniel SJ. Can ChatGPT guide parents on tympanostomy tube insertion? Children (Basel). 2023 Sep 30;10(10):1634. doi: 10.3390/children10101634.
Campbell DJ, Estephan LE, Mastrolonardo EV, Amin DR, Huntley CT, Boon MS. Evaluating ChatGPT responses on obstructive sleep apnea for patient education. J Clin Sleep Med. 2023 Dec 1;19(12):1989-1995. doi: 10.5664/jcsm.10728.
Zalzal HG, Abraham A, Cheng J, Shah RK. Can ChatGPT help patients answer their otolaryngology questions? Laryngoscope Investig Otolaryngol. 2023 Dec 9;9(1):e1193. doi: 10.1002/lio2.1193.
Soto-Galindo GA, Capelleras M, Cruellas M, Apaydin F. Effectiveness of ChatGPT in identifying and accurately guiding patients in rhinoplasty complications. Facial Plast Surg. 2023 Dec 27. doi: 10.1055/a-2218-6984.
Langlie J, Kamrava B, Pasick LJ, Mei C, Hoffer ME. Artificial intelligence and ChatGPT: an otolaryngology patient's ally or foe? Am J Otolaryngol. 2024 May-Jun;45(3):104220. doi: 10.1016/j.amjoto.2024.104220.
Mitchell RB, Archer SM, Ishman SL, Rosenfeld RM, Coles S, Finestone SA. et al. Clinical practice guideline: tonsillectomy in children (update)- executive summary. Otolaryngol Head Neck Surg. 2019 Feb;160(2):187-205. doi: 10.1177/0194599818807917.
Taloni A, Borselli M, Scarsi V, Rossi C, Coco G, Scorcia V. et al. Comparative performance of humans versus GPT-4.0 and GPT-3.5 in the self-assessment program of American Academy of Ophthalmology. Sci Rep. 2023 Oct 29;13(1):18562. doi: 10.1038/s41598-023-45837-2.
Downloads
Published
How to Cite
Issue
Section
License
Copyright (c) 2024 Portuguese Journal of Otorhinolaryngology and Head and Neck Surgery
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.
