split Edward tian GPTZero
CNN runs a human-written script through an AI-text detection app. See what happens
01:51 - Source: CNN
CNN  — 

Without cracking a single textbook, without spending a day in medical school, the co-author of a preprint study correctly answered enough practice questions that it would have passed the real US Medical Licensing Examination.

But the test-taker wasn’t a member of Mensa or a medical savant; it was the artificial intelligence ChatGPT.

The tool, which was created to answer user questions in a conversational manner, has generated so much buzz that doctors and scientists are trying to determine what its limitations are – and what it could do for health and medicine.

What ChatGPT is and isn’t

ChatGPT, or Chat Generative Pre-trained Transformer, is a natural language-processing tool driven by artificial intelligence.

The technology, created by San Francisco-based OpenAI and launched in November, is not like a well-spoken search engine. It isn’t even connected to the internet. Rather, a human programmer feeds it a vast amount of online data that’s kept on a server.

It can answer questions even if it has never seen a particular sequence of words before, because ChatGPT’s algorithm is trained to predict what word will come up in a sentence based on the context of what comes before it. It draws on knowledge stored on its server to generate its response.

ChatGPT can also answer followup questions, admit mistakes and reject inappropriate questions, the company says. It’s free to try while its makers are testing it.

Medical interest is strong

Artificial intelligence programs have been around for a while, but this one generated so much interest that medical practices, professional associations and medical journals have created task forces to see how it might be useful and to understand what limitations and ethical concerns it may bring.

Dr. Victor Tseng’s practice, Ansible Health, has set up a task force on the issue. The pulmonologist is a medical director of the California-based group and a co-author of the study in which ChatGPT demonstrated that it could probably pass the medical licensing exam.

Tseng said his colleagues started playing around with ChatGPT last year and were intrigued when it accurately diagnosed pretend patients in hypothetical scenarios.

“We were just so impressed and truly flabbergasted by the eloquence and sort of fluidity of its response that we decided that we should actually bring this into our formal evaluation process and start testing it against the benchmark for medical knowledge,” he said.

That benchmark was the three-part test that US med school graduates have to pass to be licensed to practice medicine. It’s generally considered one of the toughest of any profession because it doesn’t ask straightforward questions with answers that can easily found on the internet.

The exam tests basic science and medical knowledge and case management, but it also assesses clinical reasoning, ethics, critical thinking and problem-solving skills.

The study team used 305 publicly available test questions from the June 2022 sample exam. None of the answers or related context was indexed on Google before January 1, 2022, so they would not be a part of the information on which ChatGPT trained. The study authors removed sample questions that had visuals and graphs, and they started a new chat session for each question they asked.

Students often spend hundreds of hours preparing, and medical schools typically give them time away from class just for that purpose. ChatGPT had to do none of that prep work.

The AI performed at or near passing for all the parts of the exam without any specialized training, showing “a high level of concordance and insight in its explanations,” the study says.

Tseng was impressed.

“There’s a lot of red herrings,” he said. “Googling or trying to even intuitively figure out with an open-book approach is very difficult. It might take hours to answer one question that way. But ChatGPT was able to give an accurate answer about 60% of the time with cogent explanations within five seconds.”

Dr. Alex Mechaber, vice president of the US Medical Licensing Examination at the National Board of Medical Examiners, said ChatGPT’s passing results didn’t surprise him.