Chapter 1: Introduction to ChatGPT-4's Diagnostic Capabilities

Recent research has showcased the impressive diagnostic capabilities of ChatGPT-4, with its average score reaching an outstanding 4.2 out of 5 in assessing complex medical conditions.

A pivotal study published in JAMA, titled “Accuracy of a Generative Artificial Intelligence Model in a Complex Diagnostic Challenge,” was conducted by a team of medical professionals at Beth Israel Deaconess Medical Center (BIDMC) affiliated with Harvard Medical School. They meticulously examined the diagnostic abilities of the conversational AI, ChatGPT-4. The findings were remarkable, with ChatGPT-4 correctly identifying the diagnosis as its first choice in nearly 40% of intricate medical scenarios. More impressively, it included the correct diagnosis among its considered options in two-thirds of the cases evaluated.

Generative AI, such as ChatGPT, operates differently from traditional AI by utilizing patterns and insights from training data to create new content, rather than merely analyzing existing information. Many individuals have interacted with this technology via chatbots, which are becoming increasingly advanced digital assistants that utilize natural language processing (NLP) — a branch of AI that allows computers to comprehend and generate human-like language.

While the influence of chatbots is already being felt in customer service and education, their application in clinical settings, especially for complex diagnoses, is still largely unexplored.

“Recent breakthroughs in AI have led to generative models capable of producing comprehensive text-based responses that score high in standardized medical assessments,” stated Adam Rodman, MD, MPH, who led the study. “We aimed to determine whether such a generative model could ‘think’ like a doctor, so we tasked it with resolving standardized complex diagnostic cases used for educational purposes. The results were outstanding.”

Rodman and his research team assessed the chatbot’s diagnostic skills using a compilation of clinicopathological case conferences (CPCs). These cases contain detailed clinical data, lab results, imaging studies, and histopathological findings, typically published in the New England Journal of Medicine for educational purposes. The research involved 70 CPC cases analyzed by ChatGPT-4 AI. Using a five-point grading scale, a score of 5 indicates that the actual diagnosis was included in the differential, while a score of 0 signifies a complete miss. The differential diagnosis represents a list of potential conditions based on the patient's symptoms, medical history, and clinical findings.

Notably, the average score achieved was 4.2, with most cases receiving a score of 5 (Figure 1). The correct diagnosis appeared in the differential for 64% of cases and was the primary diagnosis in 39% of them.

Figure 1: Effectiveness of ChatGPT-4 in evaluating complex clinicopathological case conferences (CPCs). Source: Kanjee et al. (2023); JAMA.

“While chatbots cannot substitute the expertise and knowledge of trained medical professionals, generative AI shows promise as a valuable adjunct to human cognition in diagnostics,” remarked Zahir Kanjee, MD, MPH, the study’s lead author. “It has the potential to assist healthcare providers in interpreting complex medical data and refining or expanding our diagnostic approaches.”

Explore how ChatGPT could revolutionize healthcare diagnostics in this insightful video.

A recent study reveals that ChatGPT achieves accurate medical diagnoses only about half the time, raising questions about its reliability in clinical settings.