Artificial Intelligence (AI) and Machine Learning (ML) are being increasingly embedded in healthcare, from diagnostics to administrative work to clinical decision support to healthcare delivery and even patient engagement. While there is a promise of improved efficiency, there is also a concern regarding ethics, patient safety and transparency.
Understanding AI and MACHINE LEARNING in HEALTHCARE
Artificial Intelligence refers to the simulation of human intelligence by computer systems, like pattern recognition, decision making, and language processing. Machine learning is a subset of AI where the computer learns from the sets of data provided for training.
The common uses of AI in healthcare are: 1) Medical image interpretation, 2) Clinical documentation,3) Prediction and early warning symptoms, 4) Patient-facing chart and 5) Symptom tools, operational and administrative optimisation. A diverse data set is used by large models to train them, and these models continue to evolve after being implemented.
Why is patient safety important?
Patient safety was focused originally on errors in medication, failure of systems, and the risk involved with procedures. Now, AI brings forward a new category of risk that may be less visible and requires more regulations.
So regulators such as the WHO and regional regulators continue to publish laws and guidance, specifically to address AI in healthcare. The WHO has published ethics and governance guidance for large language models in healthcare, and the European Union's AI Act 2024 defines the obligations for high-risk healthcare AI.
Key patient safety risks of AI in healthcare
Transparency in decision-making and explainability: Black box models can produce clinical suggestions, and fail to explain the rationale behind it, which makes it difficult for clinicians to assess whether this should be implemented.
Bias in data: The data used to train AI systems reflects the results produced. If underrepresented populations and demographics are not included in the data set, this would result in unequal care as the systems solely perform according to the data provided.
Insensitivity to context: There is a lack of access to cultural and contextual factors, which are key for clinical decisions, and thereby AI cannot be the sole decision maker.
Automation bias: Over-reliance on suggestions made by AI can result in conflicts with clinical judgments by the physicians, thereby increasing the risk of harm to the patient.
Ethics beyond compliance
Ethical AI in healthcare ethical use of AI in healthcare is beyond just technical performance. The core values of healthcare, such as beneficence, autonomy, non-maleficence, and justice, should be the key pillars for ethical deployment of AI in healthcare.
This includes: Respecting patient autonomy, Protecting patient privacy and health data, Equitable access to benefits and Accountability in case of harm.WHO in 2021 clarified that ethical governance must be the key factor of AI from design to development to deployment and monitoring.
The regulatory response
Regulatory authorities all over the world now emphasise the need for frameworks for adaptive and learning systems
Risk-based classification: The high-risk category includes AI systems that directly influence diagnosis and treatment and thereby require enhanced scrutiny.
Life cycle oversight, the US Food and Drug Administration emphasise the need for predefined change controls and post-marketing monitoring to manage adaptive algorithms (FDA 2021). Human oversight the European Union Artificial Intelligence Act 2021 has mandatorily called for human oversight and transparency for high-risk healthcare AI systems.
Patient advocacy as a safety mechanism
Patients are no longer passive recipients of healthcare, but they now directly interact with AI tools through trackers, healthcare apps, and symptom checkers. This makes awareness and advocacy of safe use of AI by patients more critical.
Patient of advocacy contributes by: Identifying harms, unintended by the AI, promoting transparency and explainability of AI and challenging any inappropriate use. The National Academy of Medicine and Ada Lovelace Institute have thereby emphasised that clinical engagement is critical while building trust in AI healthcare systems.
Industry experience and emerging best practices
Taking into account, the need for ensuring, intentional and safe use of AI in healthcare several leading organizations have adopted common practices, such as: Review boards, which are multidisciplinary, following up with Continuous monitoring, Local validation before clinical deployment or use, Clinician education and training, as well as Patient education, for ethical and safe use of AI in healthcare.
Balancing innovation with patient safety
Safety and innovation are often seen as competing goals. In healthcare, however, responsible innovation ensures that public trust and sustainable adoption is smooth. Evidence from health systems has shown that governance can reduce downstream risk and resistance.
Patient's experience, trust, and understanding.
From a patient's experience or point of view, AI in healthcare was often seen as an automated message or a risk score, which influences access to healthcare or a chatbot response. These interactions would thereby establish trust and expectations, even before a clinician is involved.
Patients have reported concerns while engaging with AI healthcare systems:
Understanding and informed concerns:
Many patients are unaware that AI systems are used in various stages of their health care. When unexplainable AI is used, patients may not understand why or how decisions were made or what role AI has played at which stage of the care.
International guidance from the WHO emphasises that patients should be informed of the use of AI in their care and also understand the limitations of such AI systems when used for diagnosis and treatment decisions.
Perceived authority and accuracy:
Patients assume that AI systems are more accurate than humans. This assumption can lead to an exaggerated trust and confidence in AI responses, especially in cases of AI chatbots or symptom checkers. But when these systems do not provide clear information without context, patients may delay seeking care, self-diagnose incorrectly, or even experience unnecessary anxiety. The National Academy of Medicine 2023 highlighted that AI systems should provide information rather than clinical judgment, which is not understood by patients.
Access and equity from a patient's perspective, patients often experience concerns about equity and access. Patients from underrepresented groups may experience incorrect risk scoring or misclassification without an explanation of why this occurred. Hence, biased data sets can translate into unequal access to care and thereby poor outcomes for certain populations, thereby increasing the existing health disparities.
For those affected, these harms become cumulative, and trust is compromised, not really in technology, but in healthcare broadly.
Looking ahead:
A patient-centred future for AI the frameworks for AI governance will need to formalise patient involvement as a key factor and not be treated as optional. Some of the emergent proposals may include: a) Patient representation on oversight committees, b) Mandatory disclosure of AI in healthcare, c) Public reporting of safety incidents where AI was deployed, and d) Clear and explainable pathways for patients to raise their concerns or seek any correction.
These measures would align with a shift towards participatory healthcare, where patients are recognised as the primary stakeholders rather than just passive recipients of AI-driven clinical decisions.
Conclusion
From a patient's perspective, AI thereby provides both opportunity as well as risk. While the promise of delivering improved access and efficiency comes along with AI systems in healthcare, there is also a new form of bias, imbalance, and opacity.
Ethical implementation of AI in healthcare requires more than just technical excellence and demands more accountability, meaningful patient engagement, and trust. Also, accountability in case of harm is a crucial factor. The opportunity now exists to shape a future where technology will enhance medical care without compromising patient safety.
I founded Aitheal to advocate for the ethical and safe use of AI by informed patients. And to empower them to make appropriate decisions. The focus is on educating patients on the use of AI in healthcare, the advantages and limitations, and when human clinical judgement should be prioritised. The initiative emphasises responsible use and building trust at the intersection of technology and healthcare.
WHAT PATIENTS SHOULD KNOW ABOUT AI IN HEALTHCARE
ARTIFICIAL INTELLIGENCE is used in health systems by hospitals and healthcare organisations but does not replace clinicians and human care.
How AI is used?
To analyse medical images such as X-rays, photographs, pathology slides and reports
To document and summarise clinical notes while speaking to the doctor, thereby giving more time with the clinician
To identify risk or early warning signs
To provide general information or after-care instructions
What AI can do?
Help analyse large amounts of data
Support clinicians in decision-making
Reduce administrative delays
What AI cannot do?
Replace clinical judgement
Understand personal, cultural, and behavioural contexts like clinicians
Make standalone decisions on diagnosis or treatment
What patients have a right to expect?
Patients should be informed
What AI tools are used
Clear explanation in simple language
Know that a clinician is accountable
Be able to raise concerns or ask questions
Remember
1. AI provides information, not assurance
2. AI can make mistakes
3. AI may not work the same for all
4. Clinical and final decisions are always made by the clinician
A CLINICIAN SHOULD ALWAYS BE YOUR FIRST OPINION SUPPORTED BY AI
