According to a study in JAMA Pediatrics, the fourth version of the big language model is even worse with kids had a hit rate of just 17 percent when spotting pediatric medical cases. The low score shows human kid doctors won't be sacked soon, in case you were worried. As the writers say: "This study shows how important clinical experience is."

But it spots the big flaws that made ChatGPT's error rate so high and ways to make it a handy tool in clinical care. With so much buzz and trying out AI chatbots, many child doctors and other medics see their use in clinical care as sure to happen.

For ChatGPT's test, the researchers copied the key text of the medical cases into the prompt, and then two trained doctor-researchers marked the AI-made answers as right, wrong, or "did not get the diagnosis."

In the last case, ChatGPT came up with a condition linked to the clinic that was too wide or vague to be the right diagnosis.

For example, ChatGPT said one kid's case was caused by a branchial cleft cyst -- a bump in the neck or under the collarbone -- when the right diagnosis was Branchio-oto-renal syndrome, a gene problem that causes the odd growth of tissue in the neck, and changes in the ears and kidneys.

One of the signs of the problem is the making of branchial cleft cysts. All in all, ChatGPT got the right answer in just 17 of the 100 cases. It was wrong in 72 cases, and the diagnosis was not made for the other 11 cases. Of the 83 wrong diagnoses, 47 (57 per cent) were in the same organ. To be fair though it didn't recommend a course of leeches.

Among the fails, researchers saw that ChatGPT had trouble seeing known links between conditions that a good doctor would hopefully spot. 

ChatGPT came up with the diagnosis of a rare illness when it was just a vitamin deficiency. Though the chatbot had a hard time in this test, the researchers say it could get better by being specially and carefully trained on true and trusted medical books -- not stuff on the Internet.

They also say chatbots could get better with more real-time access to medical data, letting the models make their accuracy better.