July 20, 2023
A study by Stanford researchers has found that ChatGPT, a large language model AI system, can outperform first- and second-year medical students in answering challenging clinical care exam questions.
The study, published in JAMA Internal Medicine, found that ChatGPT scored an average of 4.2 points higher than medical students on the case-report section of the exam. ChatGPT also had a higher passing rate, with 93% of its responses receiving a passing grade compared to 85% for medical students.
The researchers believe that ChatGPT’s success is due to its ability to quickly process and synthesize large amounts of medical data. ChatGPT can also generate text that is both informative and engaging, which is important for answering complex clinical questions. However, the study also found that ChatGPT can sometimes make mistakes. For example, ChatGPT may confabulate, or make up information that is not supported by the evidence.
The study’s findings suggest that AI has the potential to revolutionize medical education. ChatGPT could be used to help medical students learn about different diseases and treatments, and it could also be used to provide real-time clinical advice to doctors.
Stanford School of Medicine is already considering how to incorporate AI tools into its curriculum. The school is also considering switching exams from open book to closed book to address concerns about test-taking integrity.
The sources for this piece include an article in Stanford.
