Ethics Forum: The Current Landscape of Artificial Intelligence in Medicine

A handful of articles on artificial intelligence (AI) have graced the pages of this publication in the past six years, including one by Bharat Kumar, MD, in November 2022. Dr. Kumar highlighted the exciting potential of AI in rheumatology, including machine learning (ML) algorithms for the prediction of response to methotrexate and a predictive model for the early diagnosis of ankylosing spondylitis, while emphasizing the critical importance of humans, particularly physicians and patients, in the development and deployment of AI in medicine.¹

Since that article was published nearly two years ago—a generation in computer years—much has changed and much has remained the same. OpenAI’s ChatGPT appears on many of our home screens. As the company’s marketing material suggests, instead of consulting “Dr. Google,” patients can share their symptoms with an AI-powered chatbot, K Health, partially funded by Cedars-Sinai, a nonprofit academic healthcare organization.² We now attend grand rounds that seek to demystify the nomenclature of AI and to instill us with hope and fear.

Few of us in medicine, however, have harnessed the capabilities of large language models (LLMs) to increase the efficiency of our interactions with powerful, yet somehow obtuse, electronic health records (EHRs). In rheumatology, we are not yet the beneficiaries of the automated image processing software, described by Hügle et al., that promises improved efficiency and accuracy in interpreting images and diagnosing osteoarthritis.³ To my knowledge, none of the machine learning (ML) algorithms for the detection of osteoporosis using computed tomography (CT), evaluated by Ong et al. in a systematic review, have made it to clinical practice.^3,4 Still, it is difficult to look at the advancement of AI-enabled systems in fields such as gastroenterology and breast imaging and not appreciate what our future holds.^5-8

As of Oct. 19, 2023, the U.S. Food & Drug Administration (FDA) had authorized nearly 700 AI/ML-enabled medical devices, dating back to 1995, with 155 approved between 2022 and 2023 alone, the vast majority in the field of radiology.⁹ We will most certainly see this technology grow and expand into more fields of medicine in the coming years, including rheumatology.

As we await task-specific advances in our specialty, AI pushes forward in the harnessing of big data with foundation models. Howell et al. describe the acceleration in AI/ML over the past few years as our entry into the “third epoch,” or “AI 3.0,” marked by foundation models capable of generating new content without seeing new training data, encompassing LLMs that interpret complex medically related questions and produce comprehensive answers.¹⁰

As we enter a new era of AI, regulatory frameworks governing its use must contend with systems that are more powerful and more opaque than ever. The machine learning algorithms regulated by the FDA are trained on labeled data to perform a specific task via a supervised learning approach, whereas foundation models are trained via self-supervised learning on unlabeled data.¹¹ In the case of clinical language models (CLaMs) and foundation models for electronic medical records (FEMRs), those unlabeled data are biomedical text and patient medical history from the EHR, respectively.¹² Hundreds of billions of parameters may be fed into the model, but there is no labeled truth as there is in traditional machine learning—no specific task to learn up front.¹²

Training of clinical models requires massive datasets of patient information, shared among institutions and globally, presenting new challenges for data protection and appropriate use.¹³ In the case of most FEMRs, model weights—a parameter of neural networks that helps determine flow of information—are not widely available to the research community, necessitating retraining of models on new EMR data to validate performance.¹²

Bommasani et al. point out that accessibility is threatened by the massive scale of foundation models and the vast resources required to interrogate them, resources not often available to academic institutions, and by the concentration of foundation model development with a handful of large players in big tech.¹⁴

Assessing performance, algorithmic bias and reliability, and ensuring privacy and safety, along with a slew of other important metrics, will become increasingly difficult in the era of foundation models.

As of October 2023, the FDA had not approved any device using generative AI or foundation models, including LLMs. The regulatory landscape is well outside the scope of this article, but it is fair to say that innovation in AI is outpacing oversight and regulation, both nationally and globally.

There are ethical considerations and potential concerns at every juncture in the application of AI in medicine—some that are already well defined, as exemplified by our current ML-enabled devices and others that are only now emerging; and still more to be discovered in the rapidly changing landscape of generative AI.

In this article, we examine just a few of the many ethical considerations surrounding the current and projected use of AI in medicine, focusing on the bioethical principle of autonomy. Further discussion of ethics in medical AI is warranted, with an exploration of justice, including bias, fairness and equity, of great importance.

Autonomy

Autonomy is defined by Veatch et al. as “self-legislating, the ability to make decisions based on personal values, preferences and sense of self. The removal of barriers to informed and voluntary decision making.”¹⁵

Perhaps the most obvious example of where the principle of autonomy butts against AI in medicine is patient privacy—the ownership of health information and consent for its use.

My home institution’s version of a Patient Bill-of-Rights states, “Your medical information will be kept confidential. In general, we will only share it with others if you give us permission or as otherwise permitted by law.”¹⁶ This law is the Health Insurance Portability and Accountability Act of 1996 (HIPAA), with a subsequently added Privacy Rule, which, among other provisions, allows healthcare providers to use identifiable protected health information (PHI) for research purposes, with or without consent, as deemed appropriate by an institutional review board (IRB).¹⁷ This provision has governed a vast amount of research conducted by institutions over the past 25 years, advancing the field of medicine. But the Privacy Rule explicitly does not apply to de-identified health information, the ownership of which is undecided throughout much of the U.S. As of 2015, several states assign ownership rights to the healthcare facility, with only New Hampshire assigning ownership to the patient and over half of states without a comparable law.¹⁸

Patients in the U.S. may be unaware that their individual health information, while anonymized, can be included in the large datasets used to train, validate and test ML algorithms under cover of the Privacy Rule. Aggarwal et al. surveyed patients at a large teaching hospital in the U.K. and found most were comfortable sharing health data with the National Health Service and academic institutions, but only 26.4% were comfortable sharing with commercial organizations.¹⁹ Due to a paucity of research in this specific area, less is known about the attitudes toward AI in medicine in the U.S., but we do know trust in healthcare systems is declining.²⁰

Char et al. describe a tension between improving health outcomes and generating profit in U.S. healthcare, which only stands to increase with the explosion of AI and big tech in medicine.²¹ Industry overtook academia back in 2014, surging ahead and releasing 32 significant ML systems in 2022, compared with just three by academia in the same year.²² This gap has surely widened in the years since.

Collaboration between academic institutions and industry is well established and has proved fertile ground for many advances in medicine, including pharmaceuticals, but we must acknowledge that industry is synonymous with a for-profit business model.

In the simplest terms, anonymized patient health information, often collected by nonprofit medical centers, is used by for-profit companies to build AI-enabled systems that are then sold back to healthcare providers. This structure has allowed rapid innovation, but often lacks transparency, leaving little room for patient choice, their values or preferences. Establishing new research methodologies that protect patient autonomy and define the ethical responsibilities of researchers in AI is crucial if we are to prevent the further erosion of trust in our healthcare system.

If we include the removal of barriers to voluntary decision making in our conceptualization of patient autonomy, then we must also reconsider how we approach informed consent.

In the case of breast cancer screening, already grappling with balancing the benefits of early detection with risk of overdiagnosis, AI-enabled software may exacerbate these risks and benefits. Freeman et al. conducted a systematic review of AI mammographic systems and found mixed results on accuracy compared with single and double human readers, as well as variable sensitivity and specificity, some of which was ascribed to intrinsic differences in AI systems.⁷

As these AI-enabled systems enter clinical practice, should their use be explained to the patient as part of the informed consent process? Are we able to quantify the added risk or benefit of a particular system for a particular patient, and do they have a choice in receiving this AI-enabled care?

The right to privacy is often linked with autonomy because the violation of privacy threatens an individual’s sense of self. Generative AI poses novel risks to patient privacy not seen in ML algorithms. LLMs demonstrate memorization, and the larger the model, the more data are memorized.

Carlini et al. showed an LLM memorizing at least 1% of its training dataset, leaving significant data open to a variety of attacks, including “training data extraction attacks.”²³ These malicious attacks have the potential to leak identifiable patient information and may represent a risk that is difficult to quantify, let alone guard against, given the inherent opacity of foundation models and their development in the private sector.

In Sum

As with all technological endeavors, acceleration is a given and, as we witnessed with the explosion of information transmission across the globe upon the introduction of the World Wide Web, AI will quickly permeate our lives, inside and outside medicine. Although work remains to be done, we can already see the potential benefits of AI, for both our patients and our strained healthcare systems, in the use of ML algorithms for colon, breast and lung cancer detection.²⁴

In rheumatology, predictive models may enable the earlier diagnosis of our diseases and allow for treatment regimens to be tailored to individual patients. LLMs have the potential to discover new biomarkers of disease through the digestion of massive clinically tuned datasets and to tame our EHRs to improve our workflow as physicians.

As someone once said (Voltaire, Churchill and/or Spider-Man’s Uncle Ben, according to my AI-enabled search engine), “with great power comes great responsibility.” Medical researchers and physicians will assume a critical role in evaluating AI. We need to speak the language and be equipped with tools to evaluate the validity, reliability and applicability of ML algorithms and foundation models. Gone will be the days of P values and confidence intervals. The medical community will not be able to rely solely on government oversight or self-regulation by industry for the appropriate and ethical development and application of AI in medicine. Physicians must retain a voice in the planning and implementation of new technology.

Finally, we will be tasked with not only educating ourselves and each other, but with paving the way for the next generation of researchers and physicians to advance the field, while maintaining a patient-centered, ethically sound approach.

Jeanne Gosselin, MD, is an assistant professor and the associate director of the rheumatology fellowship program at The Robert Larner, M.D., College of Medicine at the University of Vermont Medical Center, Burlington.

References

1. Kumar B. Exploring the role of artificial intelligence in rheumatology. The Rheumatologist. 2022 Nov 4.
2. Jennings K. This AI chatbot has helped doctors treat 3 million people—and it may be coming to a hospital near you. Forbes. 2023 Jul 17.
3. Hügle M, Omoumi P, van Laar JM, et al. Applied machine learning and artificial intelligence in rheumatology. Rheumatol Adv Pract. 2020 Feb 19;4(1):rkaa005.
4. Ong W, Liu RW, Makmur A, et al. Artificial intelligence applications for osteoporosis classification using computed tomography. Bioengineering (Basel). 2023 Nov 27;10(12):1364.
5. Vulpoi RA, Luca M, Ciobanu A, et al. Artificial intelligence in digestive endoscopy-where are we and where are we going? Diagnostics (Basel). 2022 Apr;12(4):927.
6. Whooley S. Medtronic to install 115 GI Genius modules across VA facilities. MassDevice. 2022 Oct 5.
7. Freeman K, Geppert J, Stinton C, et al. Use of artificial intelligence for image analysis in breast cancer screening programmes: Systematic review of test accuracy. BMJ. 2021 Sep;374:n1872.
8. Ng AY, Oberije CJG, Ambrózay É, et al. Prospective implementation of AI-assisted screen reading to improve early detection of breast cancer. Nat Med. 2023 Dec;29(12):3044–3049.
9. Artificial intelligence and machine learning (AI/ML)-enabled medical devices. U.S. Food & Drug Administration. 2023 Dec 6.
10. Howell MD, Corrado GS, DeSalvo KB. Three epochs of artificial intelligence in health care. JAMA. 2024 Jan;331(3):242–244.
11. Keeling G. Algorithmic bias, generalist models, and clinical medicine. AI Ethics. 2023 Aug 14.
12. Wornow M, Xu Y, Thapa R, Patel B, et al. The shaky foundations of large language models and foundation models for electronic health records. NPJ Digit Med. 2023 Jul 29;6(1):135.
13. He J, Baxter SL, Xu J, et al. The practical implementation of artificial intelligence technologies in medicine. Nat Med. 2019 Jan;25(1):30–36.
14. Bommasani R, Hudson D, Adeli E, et al. On the opportunities and risks of foundation models. Center for Research on Foundation Models, Stanford University.
15. Veatch RM, Guidry-Grimes LK. The Basics of Bioethics. 4th ed. Routledge; 2020.
16. Patient bill of rights at the UVM Medical Center.
17. Health information privacy. U.S, Department of Health and Human Services. 2018 Jun 13 (last reviewed).
18. Who owns medical records: 50 state comparison. 2015 Aug 20 (last updated).
19. Aggarwal R, Farag S, Martin G, et al. Patient perceptions on data sharing and applying artificial intelligence to health care data: Cross-sectional survey. J Med Internet Res. 2021;23(8):e26162.
20. Blendon RJ, Benson JM.Trust in medicine, the health system & public health. Daedalus, the Journal of the American Academy of Arts & Sciences. 2022;151(4):67–82.
21. Char DS, Shah NH, Magnus D. Implementing machine learning in health care—Addressing ethical challenges. N Engl J Med. 2018 Mar 15;378(11):981–983.
22. Maslej N, Fattorini L, Brynjolfsson E, et al, Artificial intelligence index report 2023. Institute for Human-Centered Artificial Intelligence, Stanford University. https://aiindex.stanford.edu/report. 2023 April.
23. Carlini N, Ippolito D, Jagielski M, et al. Quantifying memorization across neural language models. ICLR 2023 Conference. 2023 Feb 01. https://openreview.net/forum?id=TatRHT_1cK.
24. Chiu HY, Chao HS, Chen YM. Application of artificial intelligence in lung cancer. Cancers (Basel). 2022 Mar 8;14(6):1370.

Editor’s note: If you have a comment for the author or a case you’d like to see in Ethics Forum, email us at [email protected].