Can GPT take medical licensing exams?

Link here


  • – Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation across various domains, including medicine.
  • – GPT-4 is a large language model that has been evaluated for its performance on the United States Medical Licensing Examination (USMLE).
  • – GPT-4 shows a remarkable improvement over its predecessor models on official USMLE exam questions, improving by over 30 percentage points on both exams when compared to GPT-3.5.
  • – GPT-4 is able to perform well on questions with media elements, obtaining 70-80% prediction accuracies for these questions on both exams.
  • – GPT-4 is significantly better calibrated than GPT-3.5, demonstrating a much-improved ability to predict the likelihood that its answers are correct.
  • – GPT-4 can provide rich explanations to the student about their errors and can explain medical reasoning and interactively support students on counterfactual scenarios around a medical case.
  • – GPT-4 and its successors may one day assist investigators with clinical and biomedical research and could help to raise the competency of physicians’ assistants and help with triage and communication with remote experts.
  • – The technology could also enable more time for physicians to learn, reflect, and engage in continuing medical education to become the best at what they are interested in doing.
  • – Great care must be taken with the introduction of various forms of automation in healthcare, including uses of machine learning, and best practices for quality assurance must be developed and shared among medical professionals to ensure safe and effective use.
  • – There are still limitations and challenges to the use of large language models in real-world settings, and further research and development are needed to optimize their advantages and mitigate risks associated with their applications.


Interesting discussion on HN Forums here:

2 thoughts on “Can GPT take medical licensing exams?

    1. Possibly yes. But that will require significant effort if the data is all over the place. Look at how PyTorch from FB has been open sourced and its all over the place. There are numerous models with almost similar output!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.