AI Chatbots in Gastro Pathology: How Well Do They Really Perform?

The world of artificial intelligence chatbots is buzzing with potential. These clever tools have been put through their paces in various tests. However, their performance in real-life medical scenarios, especially in gastrointestinal pathology, remains a bit of a mystery. Some chatbots, like ChatGPT and Gemini Advanced, have shown promise in helping with patient questions and assisting healthcare providers. But, they're not perfect and come with their own set of challenges. In a recent study, three large language models were put to the test. They were given 20 different clinicopathological scenarios in gastrointestinal pathology. Two experts then evaluated how well these models performed. They looked at things like how accurate the diagnoses were and how confident the models seemed. The study also checked out how well the models could provide a list of possible diagnoses, interpret certain medical tests, give a clear final diagnosis, and explain their thought process. Each of these areas was scored on a scale of one to five. The results were then compared across the three models. So, how did they do? Well, Gemini Advanced and ChatGPT-4. 0 showed significant improvements in certain areas compared to ChatGPT-3. 5. This suggests that these models are getting better with more data training. However, their overall performance was still just average. None of the models could accurately provide the stage of tumors in the scenarios. Plus, they had a habit of citing references that didn't actually exist. This is a big no-no in the medical world. The study highlights that while these AI chatbots are evolving, they still need human oversight. They're not ready to be used on their own in clinical medicine just yet. This was the first study of its kind in gastrointestinal pathology. It's a step forward in understanding how these tools can be used in medicine. But, there's still a long way to go before they can be trusted to make critical medical decisions.