Do Medical Search Tools Stay Reliable Across Hospitals?

Search tools for clinical notes depend on embedding models. A study checked whether the top model stays on top when the hospital record set changes. It kept the documentation style fixed. The goal was to see if model rankings hold steady across institutions.

The team looked at known-item search. A short query comes from a target document. The system has to find the full source. They tested 13 embedding models. Each query was a fixed two-sentence excerpt. The setup crossed two sites or corpora in a 2 by 2 design. One corpus came from Beth Israel Deaconess Medical Center through MIMIC-IV-Note. The other came from a university corpus.

The study asked three main questions. Do model rankings move from one institution to another? What matters more: the model, the site or corpus, or the genre? Does the answer change based on scoring? It also checked whether the top factor changes when the model panel changes. Many public tests use only one source. Multi-institution evaluations often compare models and query styles inside one task, not whether rankings stay stable across institutions. By keeping genre fixed, the study tries to isolate site and model effects.

actions