The Performance of Leading Language Models in USMLE Step 3 Examination and Implications for Future Medical Practice

Securing a medical license in the United States is no easy feat. Aspiring doctors must successfully navigate three stages of the U.S. Medical Licensing Examination (USMLE), with the third and final installment often considered the most challenging. This step, known as Step 3, requires candidates to answer around 60% of the questions correctly, with an average passing score historically hovering around 75%.

Recently, major large language models (LLMs) were put to the test with the Step 3 examination, and the results were quite remarkable. These LLMs, including platforms like ChatGPT, Claude, Google Gemini, Grok, and Llama, outperformed many doctors in their performance on the exam.

In a study that isolated 50 questions from the 2023 USMLE Step 3 sample test, these leading large language models were evaluated and compared in a head-to-head analysis. The results of this experiment provided valuable insights into the clinical proficiency of each platform.

OpenAI’s ChatGPT-4o emerged as the top performer, achieving an impressive score of 98%. This platform provided detailed medical analyses with extensive reasoning, explaining its decision-making process thoroughly. Claude, from Anthropic, followed closely behind with a score of 90%, offering more human-like responses with simple language structures. Google Gemini, Grok, and Llama also performed well, but with varying degrees of detailed reasoning and clarity in their answers.

Despite these models not being specifically designed for medical reasoning, they demonstrated a surprising aptitude for clinical analysis. As newer platforms like Google’s Med-Gemini, refined for medical applications, continue to evolve, the potential for these machines to assist in medical diagnoses, treatment recommendations, and clinical reasoning becomes increasingly promising.

While these platforms may not replace human providers entirely, they have the potential to offer a level of precision and consistency that can complement the work of doctors, particularly in scenarios where fatigue and human error may come into play. As technology continues to advance, the future of healthcare may involve a synergistic approach where machines and doctors work together to provide the best possible care for patients.