June 8, 2023
2 mins read

ChatGPT performs poorly at US’ urologists exam

The explanations provided by ChatGPT were longer than those provided by SASP, but “frequently redundant and cyclical in nature”, according to the authors…reports Asian Lite News

The much-acclaimed OpenAI’s ChatGPT chatbot has failed a urologist exam in the US, according to a study.

This comes at a time of growing interest in the potential role of artificial intelligence (AI) technology in medicine and healthcare.

The study, reported in the journal Urology Practice, showed that ChatGPT achieved less than a 30 per cent rate of correct answers on the American Urologist Association’s widely used Self-Assessment Study Program for Urology (SASP).

“ChatGPT not only has a low rate of correct answers regarding clinical questions in urologic practice, but also makes certain types of errors that pose a risk of spreading medical misinformation,” said Christopher M. Deibert, from University of Nebraska Medical Center.

The AUA’s Self-Assessment Study Program (SASP) is a 150-question practice examination addressing the core curriculum of medical knowledge in urology.

The study excluded 15 questions containing visual information such as pictures or graphs.

Overall, ChatGPT gave correct answers to less than 30 per cent of SASP questions, 28.2 per cent of multiple-choice questions and 26.7 per cent of open-ended questions.

The chatbot provided “indeterminate” responses to several questions. On these questions, accuracy was decreased when the LLM model was asked to regenerate its answers.

For most open-ended questions, ChatGPT provided an explanation for the selected answer.

The explanations provided by ChatGPT were longer than those provided by SASP, but “frequently redundant and cyclical in nature”, according to the authors.

“Overall, ChatGPT often gave vague justifications with broad statements and rarely commented on specifics,” Dr. Deibert said.

Even when given feedback, “ChatGPT continuously reiterated the original explanation despite it being inaccurate”.

The researchers suggest that while ChatGPT may do well on tests requiring recall of facts, it seems to fall short on questions pertaining to clinical medicine, which require “simultaneous weighing of multiple overlapping facts, situations and outcomes”.

“Given that LLMs are limited by their human training, further research is needed to understand their limitations and capabilities across multiple disciplines before it is made available for general use,” Dr. Deibert said.

“As is, utilisation of ChatGPT in urology has a high likelihood of facilitating medical misinformation for the untrained user.”

ALSO READ-US judge orders lawyers not to use ChatGPT-drafted content  

Previous Story

Here are the most expensive cities for expats

Next Story

Prez Murmu: India-Serbia ties defined in context of NAM

Latest from -Top News

Trump tariffs send world markets into panic

US benchmark crude oil shed $2.70 to $64.25 a barrel after major oil producers announced they plan to increase production. Brent crude, the international standard, was down $2.63 at $67.51 a barrel

EU prepares retaliation for Trump’s tariffs

The European Commission is assembling a fresh round of counter-tariffs aimed at US goods, adding to two existing lists of potential targets—one of which includes products that were hit by suspended tariffs

US, EU slam China’s war games near Taiwan

US President Donald Trump underscored the need to maintain peace in the Taiwan Strait, advocating for a diplomatic approach to cross-strait tensions while warning against the use of force The United States

£13.9 billion of R&D fund to boost innovation, jobs

Funding outlined to support transformational R&D in areas like life sciences, green energy, engineering and beyond More UK innovators like those developing treatment-transforming dementia tests or building world-leading testing facilities to power

OPEC+ accelerates oil output hikes

Despite the production boost, the group emphasised that future adjustments remain flexible and could be paused or reversed depending on market conditions. Eight OPEC+ nations have unexpectedly decided to accelerate their oil
Go toTop

Don't Miss

ChatGPT app for iOS now expands to more countries

The company also introduced a new feature called shared links.

ChatGPT bug may have exposed payment information

Due to the bug, some subscription confirmation emails generated during