Medical AI has a measurement problem

NEWS AND VIEWS

28 July 2026 Medical AI has a measurement problem The development of two medical AI assistants highlights an unnerving challenge: as the technology races ahead, what is the best way to evaluate what works? By Arjun K. Manrai 0 Arjun K. Manrai Arjun K. Manrai is in the Department of Biomedical Informatics, Harvard Medical School, Boston, Massachusetts 02115, USA. View author publications PubMed Google Scholar

Evidence-based medicine rests on the assumption that the yardstick used to measure the effectiveness of a new drug or medical device is reliable. But this assumption is often shaky, especially in the context of the artificial-intelligence tools that are making their way into clinical practice. Two papers in Nature, one by Ferber et al.1 and one by Liévin et al.2 show how capable medical AI tools have become, but also demonstrate why that capability is increasingly difficult to assess and understand properly.

Nature 655, 1138-1139 (2026)

doi: https://doi.org/10.1038/d41586-026-02125-z

References Ferber, D. et al. Nature 655, 1282–1291 (2026). Liévin, V. et al. Nature 655, 1292–1299 (2026). Johnson, A. E. W. et al. Sci. Data 10, 1 (2023). Vishwanath, K. et al. Nature Med. 32, 2405–2409 (2026). Feng, J. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2606.28960 (2026). Begley, C. G. & Ioannidis, J. P. A. Circ. Res. 116, 116–126 (2015). Brodeur, P. et al. Preprint at arXiv https://doi.org/10.48550/arXiv.2603.08448 (2026). American Medical Association. Physician Survey on Augmented Intelligence. (American Medical Association; 2026). Download references

Competing Interests The author declares no competing interests.

Subjects