AI Medical Tools Need More Proof and Better Eyes

Doctors and computers are learning to read medical pictures. A study looked at five big AI systems. These systems could answer questions about scans. The study asked them to spot problems. It wanted to know how well they did. The goal was to test real world use.

The team used many kinds of scans. They used CT scans, X rays, MRIs, ultrasounds, and nuclear images. They also used English and Chinese questions. Some questions had pictures. Some had only words. The team had 298 questions in all. Almost 200 were picture based. The rest were word based. The team grouped the questions in different ways. The team looked at many body areas. The team studied many kinds of problems.

The results showed big differences. One AI model was best overall. The top model was Gemini. It got a right answer rate near 90 percent. The other models scored lower. The gap was big. The best model also did better on word questions than picture questions. That means words helped more than pictures for some models. A few models liked English more than Chinese. The models did not all do well. The best model was not perfect.

The study focused on close visual checks. It looked at body parts, spots, and where things were. The models had trouble with some picture tasks. Their answers were not always steady. The best model still missed some visual hints. The team saw limits in the data.