Dr. Optimus

Would you trust an AI to be your doctor?

Would you trust it to make a diagnosis… and then to prescribe your treatment?

Or would you trust a human doctor who used artificial intelligence (AI) to do the same?

Given the latest developments in applying AI to life sciences, biotechnology, and healthcare, these are quickly becoming relevant questions to ask.

Some research published last month – led by a team from Stanford University – might help us answer those questions.

I think you’ll find the results interesting, as they are not what most of us would expect.

Augmented With AI?

The design of the trial – titled Large Language Model Influence on Diagnostic Reasoning – was a randomized clinical trial.

Its objective was to determine if the use of OpenAI’s ChatGPT Plus would significantly improve the accuracy of doctors’ diagnoses compared to using non-AI-based resources available today.

Fifty physicians were given the task of diagnosing complex cases. Half were randomly assigned ChatGPT Plus for use to diagnose the cases. And the other half were permitted to use their conventional methods.

It would be logical for us to assume that the physicians leveraging a powerful large language model (LLM) like ChatGPT would outperform those without that resource.

After all, a well-designed LLM that has been trained on high-quality data, research, and literature can not only have perfect recall but also synthesize that information and even find connections that aren’t intuitive.

The results were surprising.

Source:  Goh, et. al., “Large Language Model Influence on Diagnostic Reasoning,” JAMA, October 28, 2024

The median diagnostic accuracy for ChatGPT Plus-augmented physicians was 76.3%. And the accuracy for those physicians who used conventional approaches to diagnosis was 73.7%. There was very little difference between the two groups of physicians.

Not significant at all.

The only major difference was that those augmented by ChatGPT Plus were able to reach their diagnosis in 519 seconds, on average… versus the conventional group at 565 seconds.

After reviewing the results, one researcher involved made an interesting comment:

We were surprised to find that adding a human physician to the mix actually reduced diagnostic accuracy though improved efficiency.

This is where it gets really interesting.

Just AI – No Human

Why didn’t the AI-augmented physicians wipe the floor with the non-augmented ones?

Why did they perform so poorly to expectation?

The poor performance of the ChatGPT Plus-augmented physicians wasn’t because ChatGPT was wrong in its diagnosis. It was because the physicians used their own bias to overrule the AI in its diagnosis.

The big surprise to all was that when ChatGPT Plus was tasked with diagnosing those complex cases alone, it had a median diagnostic accuracy of 92%!

To ask a simple question: As a patient, which accuracy would you prefer when being diagnosed, 73.7% or 92%? I know what I would choose.

Now, to be fair, this is just one study with 50 physicians. And most physicians are probably still skeptical about trusting AI from a diagnostic perspective.

But this isn’t the first analysis of this kind. AIs trained on the entire body of knowledge in a field of study are consistently outperforming humans. And the accuracy has been improving at an exponential pace.

But there is some very interesting context that I think most of us will find shocking.

The Healthcare Industry’s Dirty Secret

If we listen to the Centers for Disease Control and Prevention (CDC), it will tell us that the top five leading causes of death in the U.S. in 2023 were:

The above list probably won’t come as a surprise, except for the fact that it is inaccurate.

A recent study from the BMJ showed that about 371,000 deaths are caused by misdiagnosis in the U.S. That makes it the third largest cause of death.

The evidence-based table should look like this:

This is one of the dirtiest secrets in the healthcare industry.

Have you ever been told by a healthcare professional, a doctor, a nurse, someone at a hospital or a clinic that misdiagnosis is the third-leading cause of death?  And that the doctor may only be accurate with his/her diagnosis 75% of the time?

The truth is that it’s actually much worse.

If we consider the number of people who are permanently disabled by misdiagnosis, the damage caused is an additional 424,000 patients in the U.S. a year. In total, that’s 795,000 patients who die or are permanently disabled by misdiagnosis every year.

I don’t know about you, but those numbers make me extremely uncomfortable.

Unrigging the System

I know this might sound strange, but whenever I go see a doctor, I do some research before my visit. I read up on the latest research in their field and see if I find something interesting that I can ask about or something that might be useful for my own health.

Every time, the physicians are not aware of the research I share with them. I often print out and bring copies for them to review.  It’s not their fault actually. It’s just the way the healthcare system works.

Primary care physicians tend to see a range of 20–30 patients a day. Specialists might see 10–20.  A friend of mine, a cardiologist, told me he sees about 55 patients a day.

The reality is that they don’t have the time to keep up with all the latest methods and research. And the reality is that they are strongly “encouraged” to follow the standard practices and therapies as approved and prescribed by their medical licensing board for their area of practice. [Note: These boards are known for being “influenced” by the pharmaceutical and biomedical industries.]

This is why it is so important for us, as patients, to do our own research and be our own advocates for our health.

The healthcare system is rigged, and not in a good way. Death and disability due to misdiagnosis are just the “cost of doing business” – that’s why medical malpractice insurance is so expensive and necessary.

If it were up to me, I would work medically trained AIs into every doctor’s workflow as soon as is humanly possible. 92% accuracy is materially higher than 75%… and it would save hundreds of thousands of lives and permanent disability annually. And that’s just the U.S…

The numbers are far larger on a global basis. There is a shortage of millions of physicians and surgeons globally. The scale is so large that it is impossible to fix, even in a 10-year time frame.

A multimodal AI can immediately help solve the labor shortage and dramatically improve healthcare outcomes for patients, especially those who are underserved.

And manifest that AI into a humanoid robot form – such as Tesla’s Optimus – and the AI will be empowered with real-time computer vision, the ability to understand diagnostic data, question the patient, order additional tests, diagnose the illness, and even prescribe medicine. All with accuracy far better than a human physician.

So I’ll ask again, would you trust an AI to be your doctor?

Jeff