How Good Are AI Doctors?

You might think that big language models, like the ones powering AI doctors, can predict diseases as well as traditional classifiers. But hold on, let's dig into this! These AI doctors use something called next-word probability to make guesses. However, this isn't the same as the pre-test probability that traditional classifiers use. So, what's the deal? Well, when you type in symptoms, the AI guesses what comes next, like in a sentence. This isn't the same as estimating the likelihood of a disease before any tests are done. It's like asking Siri to finish your sentence versus asking a doctor to predict your health based on symptoms alone.Researchers have been comparing these AI doctors to traditional classifiers to see who's better at uncertainty estimation. Uncertainty estimation is a fancy way of saying how sure or unsure the AI is about its diagnosis. It's crucial because it helps doctors decide when to order more tests or seek a second opinion. The results? They're not as clear-cut as we'd like. While AI doctors are getting better, they still have a ways to go before they can match traditional classifiers at estimating uncertainty.