Enlarge / Dr. Greg House has a better rate of accurately diagnosing patients than ChatGPT.
ChatGPT is still no House, MD.
While the chatty AI bot has previously underwhelmed with its attempts to diagnose challenging medical caseswith an accuracy rate of 39 percent in an analysis last yeara study out this week in JAMA Pediatrics suggests the fourth version of the large language model is especially bad with kids. It had an accuracy rate of just 17 percent when diagnosing pediatric medical cases.
The low success rate suggests human pediatricians won't be out of jobs any time soon, in case that was a concern. As the authors put it: "[T]his study underscores the invaluable role that clinical experience holds." But it also identifies the critical weaknesses that led to ChatGPT's high error rate and ways to transform it into a useful tool in clinical care. With so much interest and experimentation with AI chatbots, many pediatricians and other doctors see their integration into clinical care as inevitable.
The medical field has generally been an early adopter of AI-powered technologies, resulting in some notable failures, such as creating algorithmic racial bias, as well as successes, such as automating administrative tasks and helping to interpret chest scans and retinal images. There's also lot in between. But AI's potential for problem-solving has raised considerable interest in developing it into a helpful tool for complex diagnosticsno eccentric, prickly, pill-popping medical genius required.
In the new study conducted by researchers at Cohen Childrens Medical Center in New York, ChatGPT-4 showed it isn't ready for pediatric diagnoses yet. Compared to general cases, pediatric ones require more consideration of the patient's age, the researchers note. And as any parent knows, diagnosing conditions in infants and small children is especially hard when they can't pinpoint or articulate all the symptoms they're experiencing.
For the study, the researchers put the chatbot up against 100 pediatric case challenges published in JAMA Pediatrics and NEJM between 2013 and 2023. These are medical cases published as challenges or quizzes. Physicians reading along are invited to try to come up with the correct diagnosis of a complex or unusual case based on the information that attending doctors had at the time. Sometimes, the publications also explain how attending doctors got to the correct diagnosis.
For ChatGPT's test, the researchers pasted the relevant text of the medical cases into the prompt, and then two qualified physician-researchers scored the AI-generated answers as correct, incorrect, or "did not fully capture the diagnosis." In the latter case, ChatGPT came up with a clinically related condition that was too broad or unspecific to be considered the correct diagnosis. For instance, ChatGPT diagnosed one child's case as caused by a branchial cleft cysta lump in the neck or below the collarbonewhen the correct diagnosis was Branchio-oto-renal syndrome, a genetic condition that causes the abnormal development of tissue in the neck, and malformations in the ears and kidneys. One of the signs of the condition is the formation of branchial cleft cysts.
Overall, ChatGPT got the right answer in just 17 of the 100 cases. It was plainly wrong in 72 cases, and did not fully capture the diagnosis of the remaining 11 cases. Among the 83 wrong diagnoses, 47 (57 percent) were in the same organ system.
Among the failures, researchers noted that ChatGPT appeared to struggle with spotting known relationships between conditions that an experienced physician would hopefully pick up on. For example, it didn't make the connection between autism and scurvy (Vitamin C deficiency) in one medical case. Neuropsychiatric conditions, such as autism, can lead to restricted diets, and that in turn can lead to vitamin deficiencies. As such, neuropsychiatric conditions are notable risk factors for the development of vitamin deficiencies in kids living in high-income countries, and clinicians should be on the lookout for them. ChatGPT, meanwhile, came up with the diagnosis of a rare autoimmune condition.
Though the chatbot struggled in this test, the researchers suggest it could improve by being specifically and selectively trained on accurate and trustworthy medical literaturenot stuff on the Internet, which can include inaccurate information and misinformation. They also suggest chatbots could improve with more real-time access to medical data, allowing the models to refine their accuracy, described as "tuning."
"This presents an opportunity for researchers to investigate if specific medical data training and tuning can improve the diagnostic accuracy of LLM-based chatbots," the authors conclude.
See the original post here:
ChatGPT bombs test on diagnosing kids' medical cases with 83% error rate - Ars Technica
- Johns Hopkins University Unveils Initiatives to Tackle Pediatrics TB in India and Empower Women Scientists - Devdiscourse - November 20th, 2024 [November 20th, 2024]
- Announcing Lau as interim chair of the Department of Pediatrics - Heersink School of Medicine News | UAB - University of Alabama at Birmingham - November 12th, 2024 [November 12th, 2024]
- Implementing artificial intelligence and the importance of transparency for patients - Contemporary Pediatrics - October 26th, 2024 [October 26th, 2024]
- Pediatrics with collaborative care explored in CDA Journal collection - CDA (California Dental Association) - October 13th, 2024 [October 13th, 2024]
- PAK Pediatrics offering drive-thru flu clinics - Yahoo Life - October 13th, 2024 [October 13th, 2024]
- Nonoperative treatment emerging for pediatric appendicitis - Contemporary Pediatrics - October 13th, 2024 [October 13th, 2024]
- Rep. Paul Lee honored as 'Legislative Hero' by American Academy of Pediatrics - WTVY - October 13th, 2024 [October 13th, 2024]
- Pediatrics Leveraging Yosi Health for Mental Health and Other Screenings to Ensure Healthy Children and Parents - Healthcare IT Today - October 2nd, 2024 [October 2nd, 2024]
- Chroming trend among teens linked to TikTok videos - Contemporary Pediatrics - October 2nd, 2024 [October 2nd, 2024]
- Water-absorbing beads pose growing threat to children, new study highlights - Contemporary Pediatrics - October 2nd, 2024 [October 2nd, 2024]
- Steven Selbst, MD: 'We have to be ready' to respond, help in mass shooting situations - Contemporary Pediatrics - October 2nd, 2024 [October 2nd, 2024]
- Pediatrics academy accused of fearmongering over GMO ingredients in kids diets - Science - September 23rd, 2024 [September 23rd, 2024]
- XpertDox Partners with Blue Fish Pediatrics to Enhance Medical Coding Efficiency with AI-Powered Healthcare Coding Software - 24-7 Press Release - September 23rd, 2024 [September 23rd, 2024]
- Exploring associations of inpatient food insecurity and pediatric hospitalizations - Contemporary Pediatrics - September 23rd, 2024 [September 23rd, 2024]
- Infants' tongue-tie may be overdiagnosed and needlessly treated, pediatrics group says - Tulsa World - August 5th, 2024 [August 5th, 2024]
- Infants' tongue-tie may be overdiagnosed and needlessly treated, American Academy of Pediatrics says - The Associated Press - August 5th, 2024 [August 5th, 2024]
- Infants tongue-tie may be overdiagnosed and needlessly treated, American Academy of Pediatrics says - Drgnews - August 5th, 2024 [August 5th, 2024]
- Pediatric patients with uveitis at elevated risk of cataracts, study finds - Contemporary Pediatrics - July 26th, 2024 [July 26th, 2024]
- Finding the Signal in the Noise on Pediatric Gender-Affirming Care - The Hastings Center - June 27th, 2024 [June 27th, 2024]
- Texas abortion law was followed by a rise in infant deaths: study | STAT - STAT - June 27th, 2024 [June 27th, 2024]
- Generation Next Dr. Lisa Costello, 37, Assistant Professor of Pediatrics, WVU - WV News - June 27th, 2024 [June 27th, 2024]
- Local pediatricians tips on keeping kids active and eating healthy this summer - WTOP - June 27th, 2024 [June 27th, 2024]
- Discussing health care sustainability, climate change, and WHO's One Health goal - Contemporary Pediatrics - June 27th, 2024 [June 27th, 2024]
- Pediatric chiropractic boosta physical, emotional well-being - The Times of Northwest Indiana - June 27th, 2024 [June 27th, 2024]
- Ask the Pediatrician: How firearm safety begins at home - Union Democrat - June 27th, 2024 [June 27th, 2024]
- Expert consensus and guidance for integrating discussions of racism in pediatric care - News-Medical.Net - June 27th, 2024 [June 27th, 2024]
- JAMA Pediatrics Published Study Rise in Infant Deaths in Texas to Abortion Ban - ASEAN NOW - June 27th, 2024 [June 27th, 2024]
- Supreme Court Will Take Up State Bans on Gender Care for Minors: Here's What to Know - Medpage Today - June 27th, 2024 [June 27th, 2024]
- Weekly review: Drowning prevention, coding updates, and more - Contemporary Pediatrics - June 9th, 2024 [June 9th, 2024]
- A call to action for pediatrics: Caring for students with learning disabilities in custody and community settings | Pediatric ... - Nature.com - June 9th, 2024 [June 9th, 2024]
- Milestones Pediatrics begins summer camp with sports at Rec Center - The Post-Searchlight - Post Searchlight - June 9th, 2024 [June 9th, 2024]
- Pediatric Associates to Deploy Innovaccer's AI Platform to Enhance Quality of Care for 1.5 Million Patients - HIT Consultant - June 9th, 2024 [June 9th, 2024]
- With drownings on the rise, prevention conversations with caregivers are crucial - Contemporary Pediatrics - June 9th, 2024 [June 9th, 2024]
- Chief Nursing Officer Honored with Miracle Maker Award - URMC - June 9th, 2024 [June 9th, 2024]
- EEG and ECG are overused in children with breath-holding spells - Contemporary Pediatrics - June 9th, 2024 [June 9th, 2024]
- New Close To Home Center to Open in Grove City - Newswise - June 9th, 2024 [June 9th, 2024]
- Dr. Robert Wilmott, pediatrics chair who gave parents advice as 'Dr. Bob,' dies at 75 - St. Louis Post-Dispatch - May 24th, 2024 [May 24th, 2024]
- American Academy of Pediatrics launches 'Rx4DC' initiative to address gun violence - DC News Now | Washington, DC - May 24th, 2024 [May 24th, 2024]
- Utah Valley Pediatrics expands to Sanpete County by opening Ephraim location - Daily Herald - May 24th, 2024 [May 24th, 2024]
- Pruitt named director of academic pediatrics division - The Source - Washington University in St. Louis - May 24th, 2024 [May 24th, 2024]
- Integrating behavioral health within primary care settings - Contemporary Pediatrics - May 24th, 2024 [May 24th, 2024]
- Phones and kids: new pediatric guidelines, expert advice and info on new school rules - Kidsburgh - May 24th, 2024 [May 24th, 2024]
- Suicide: Blueprint for Youth Suicide Prevention - American Academy of Pediatrics - May 24th, 2024 [May 24th, 2024]
- Dr Julie Linton Addresses Disparities in Pediatric Care for Immigrant Children in the US - AJMC.com Managed Markets Network - May 24th, 2024 [May 24th, 2024]
- CAR T-Cell Therapies Move Ahead in Pediatric and Adult ALL - OncLive - May 24th, 2024 [May 24th, 2024]
- American Academy of Pediatrics reverses long-standing ban on breastfeeding for mothers with HIV - WION - May 24th, 2024 [May 24th, 2024]
- In a Shift, Pediatricians' Group Says Breastfeeding Safe When HIV-Positive Mom Is Properly Treated - jacksonprogress-argus - May 24th, 2024 [May 24th, 2024]
- US Pediatricians Group Reverses Decades-Old Ban On Breastfeeding For Those With HIV - 1340 WJOL - 1340 WJOL - May 24th, 2024 [May 24th, 2024]
- Sang Pediatrics: When to see the doctor for a fever - YourCentralValley.com - May 12th, 2024 [May 12th, 2024]
- Should You Put Sunscreen on Infants? Not Usually - FDA.gov - May 12th, 2024 [May 12th, 2024]
- Thousands of bike helmets to be distributed to Ohio children | 10tv.com - 10TV - May 12th, 2024 [May 12th, 2024]
- How to talk to your teenaged patient about pregnancy and contraception - Contemporary Pediatrics - May 12th, 2024 [May 12th, 2024]
- Adult RSV Shots Given to More Than 30 Babies by Mistake - Medpage Today - May 12th, 2024 [May 12th, 2024]
- Pediatric Health Care Teams Can Help Prevent Youth Suicide - The Pew Charitable Trusts - May 12th, 2024 [May 12th, 2024]
- Small patients, big discoveries | Penn Today - Penn Today - May 12th, 2024 [May 12th, 2024]
- HCPs can play an important role in addressing the US infant formula shortage - Contemporary Pediatrics - May 12th, 2024 [May 12th, 2024]
- East Wenatchee Doctor Awarded For Boosting Child Vaccinations - KPQ - May 12th, 2024 [May 12th, 2024]
- HPV vaccination coverage of US-children aged 9-17 years in 2022 - Contemporary Pediatrics - May 12th, 2024 [May 12th, 2024]
- Weekly review: Profound genetic deafness gene therapy, measles increases, and more - Contemporary Pediatrics - May 12th, 2024 [May 12th, 2024]
- Pediatric Telehealth Platform Market is expected to Expand at a Massive CAGR of 26.6% through 2030 - openPR - May 12th, 2024 [May 12th, 2024]
- DB-OTO improved hearing to normal in child with profound genetic deafness - Contemporary Pediatrics - May 12th, 2024 [May 12th, 2024]
- The IT strategy behind a groundbreaking new $2B pediatric hospital - Healthcare IT News - May 12th, 2024 [May 12th, 2024]
- Modest Changes in Kids' Developmental Scores Seen From Pandemic - Medpage Today - April 23rd, 2024 [April 23rd, 2024]
- This pediatrics subspecialty has the best work-life balance - MD Linx - April 23rd, 2024 [April 23rd, 2024]
- PM Pediatrics Cobble Hill closing Pediatric urgent care options in Brooklyn and lower Manhattan - Brooklyn Bridge Parents - April 23rd, 2024 [April 23rd, 2024]
- Bethesda Pediatrics awarded $18,000 grant by GIVE - Yahoo! Voices - April 23rd, 2024 [April 23rd, 2024]
- Nemours Children's Health, UCF Partner to Elevate the Quality of Healthcare for Florida's Children - UCF - April 23rd, 2024 [April 23rd, 2024]
- Pediatricians told to talk to patients and parents about climate. - Grist - April 23rd, 2024 [April 23rd, 2024]
- Getting a good night's rest is vital for neurodiverse children pediatric sleep experts explain why - The Conversation - April 23rd, 2024 [April 23rd, 2024]
- Tina Cheng Honored as Exceptional CMO - Research Horizons - Research Horizons - April 23rd, 2024 [April 23rd, 2024]
- Let's talk about childhood injury prevention | Hub - The Hub at Johns Hopkins - April 23rd, 2024 [April 23rd, 2024]
- More young people with depression, anxiety in Virginia | Headlines - InsideNoVa - April 23rd, 2024 [April 23rd, 2024]
- Pediatrics fundraising event at Altru helps children with critical needs - Grand Forks Herald - April 15th, 2024 [April 15th, 2024]
- Public perceptions of families affected by pediatric cancer and educational work in pediatric oncology | Pediatric ... - Nature.com - April 15th, 2024 [April 15th, 2024]
- The pediatrician, the resident, the strawberry monster, and the zebra - VUMC News - VUMC Reporter - April 15th, 2024 [April 15th, 2024]
- Receptor targeting to identify medications for pediatric mental health concerns - Contemporary Pediatrics - April 15th, 2024 [April 15th, 2024]
- Hypoglossal nerve stimulation offers hope for pediatric Down syndrome patients with OSA - News-Medical.Net - April 15th, 2024 [April 15th, 2024]
- Efforts to reduce wait times for developmental evaluations - Baylor College of Medicine | BCM - April 15th, 2024 [April 15th, 2024]
- Nasal oxidative stress mediating the effects of colder temperature exposure on pediatric asthma symptoms | Pediatric ... - Nature.com - April 15th, 2024 [April 15th, 2024]
- Week in review: A new dermatology case study, and a new treatment for severe asthma - Contemporary Pediatrics - April 15th, 2024 [April 15th, 2024]