×
Home Current Archive Editorial board
News Contact
Opinion/Perspectives/Commentary

Discussing the Role of Explainable AI and Evaluation Frameworks for Safe and Effective Integration of Large Language Models in Healthcare

By
Sandeep Reddy Orcid logo ,
Sandeep Reddy
Contact Sandeep Reddy

Deakin School of Medicine , Victoria , Australia

Alexandre Lebrun Orcid logo ,
Alexandre Lebrun

Nabla , Paris , France

Adam Chee Orcid logo ,
Adam Chee

Saw Swee Hock School of Public Health, National University of Singapore , Singapore , Singapore

Dimitros Kalogeropoulos Orcid logo
Dimitros Kalogeropoulos

Global Health & Digital Innovation Foundation , London , United Kingdom

UCL Global Business School for Health , London , United Kingdom

Abstract

The integration of artificial intelligence (AI), specifically large language models (LLMs), into healthcare continues to accelerate, necessitating thoughtful evaluation and oversight to ensure safe, ethical, and effective deployment. This editorial summarizes key perspectives from a recent panel conversation among AI experts regarding central issues around implementing LLMs for clinical applications. Key topics covered include: the potential of explainable AI to facilitate transparency and trust; challenges in aligning AI with variable global healthcare protocols; the importance of evaluation via translational and governance frameworks tailored to healthcare contexts; scepticism around overly expansive uses of LLMs for conversational interfaces; and the need to judiciously validate LLMs, considering risk levels. The discussion highlights explainability, evaluation and careful deliberation with healthcare professionals as pivotal to realizing benefits while proactively addressing risks of larger AI adoption in medicine.

References

1.
Telehealth and Medicine Today.
2.
Wahl B, Cossy-Gantner A, Germann S, Schwalbe NR. Artificial intelligence (AI) and global health: how can AI contribute to health in resource-poor settings? BMJ Global Health. 2018;3(4):e000798.
3.
Char DS, Shah NH, Magnus D. Implementing Machine Learning in Health Care — Addressing Ethical Challenges. New England Journal of Medicine. 2018;378(11):981–3.
4.
Kelly CJ, Karthikesalingam A, Suleyman M, Corrado G, King D. Key challenges for delivering clinical impact with artificial intelligence. BMC Medicine. 2019;17(1).
5.
Adadi A, Berrada M. Peeking Inside the Black-Box: A Survey on Explainable Artificial Intelligence (XAI). IEEE Access. 2018;6:52138–60.
6.
Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence. 1(5):206–15.
7.
Reddy S, Rogers W, Makinen VP, Coiera E, Brown P, Wenzel M, et al. Evaluation framework to guide implementation of AI systems into healthcare settings. BMJ Health & Care Informatics. 2021;28(1):e100444.
8.
Kalogeropoulos D. Telehealth’s Role Enabling Sustainable Innovation and Circular Economies in Health. Telehealth and Medicine Today. 8(1).
9.
Miller T. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence. 2019;267:1–38.

Citation

This is an open access article distributed under the Creative Commons Attribution License which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited. 

Article metrics

Google scholar: See link

The statements, opinions and data contained in the journal are solely those of the individual authors and contributors and not of the publisher and the editor(s). We stay neutral with regard to jurisdictional claims in published maps and institutional affiliations.