Part 1: AI in Healthcare: What Providers Need to Know Before Deployment

Artificial intelligence (AI) is being increasingly integrated into healthcare delivery, from clinical decision support tools to administrative tasks. While these technologies offer opportunities to improve efficiency and patient outcomes, they also create new legal, privacy, and human rights considerations.

Recognizing these challenges, the Office of the Information and Privacy Commissioner of Ontario (IPC) and the Ontario Human Rights Commission (OHRC) jointly released the Principles for the Responsible Use of Artificial Intelligence (the “IPC-OHRC Principles”). These principles provide guidance for the development, procurement, deployment, and oversight of AI systems used in Ontario's public sector and offer valuable insight for healthcare organizations adopting AI technologies. The IPC has also published AI Scribes: Key Considerations for the Health Sector and an accompanying checklist, which applies the IPC-OHRC Principles to the AI Scribes use case. Together, these professional guidelines send a clear message to the health sector: responsible AI deployment requires careful consideration of how this new technology will be incorporated into a health information custodian’s privacy framework before implementation and across the lifecycle of the technology.

In this three-part blog series, we discuss the takeaways from the IPC-OHRC Principles for the use, procurement, and development of AI systems in the healthcare sector. In Part 1, we discuss planning considerations for AI use and maintaining patient privacy across its lifecycle.

Understanding the AI Lifecycle

The IPC-OHRC Principles define AI as a machine-based system that generates outputs such as predictions, recommendations, content, or decisions that can influence physical or virtual environments. This aligns with the domestic Enhancing Digital Security and Trust Act and international Organization for Economic Co-operation and Development definitions.

For healthcare providers, responsible AI governance extends beyond implementation. It encompasses the entire AI lifecycle, including:

• Design, data collection, and model development;

• Verification and validation;

• Deployment;

• Ongoing operation and monitoring; and

• Eventual decommissioning.

Healthcare providers should understand that compliance and oversight obligations continue as the system is updated and used in practice, and plan ahead accordingly.

Validity, Reliability, and Safety in Clinical Settings

In healthcare settings, validity, reliability, and safety are important considerations because inaccurate, biased, or incomplete data can undermine even a technically sophisticated system. AI tools used to support diagnosis, patient triage, resource allocation, or administrative functions should be evaluated for accuracy and consistency across diverse patient populations and operating conditions (as appropriate to the size and patient population of the healthcare provider).

Before deployment, healthcare organizations should be satisfied that an AI system has been appropriately tested and performs as intended. According to the IPC-OHRC Principles, validity requires that a system meet independent testing standards and fulfill its stated purpose. Reliability requires consistent performance within the environment for which it was designed.

The IPC-OHRC Principles also emphasize safety. AI systems should support human life, physical and mental health, economic security, and environmental well-being. Organizations should ensure that cybersecurity protections, monitoring mechanisms, and safeguards are in place to prevent unintended harms or infringements on individual rights.

Privacy Protections Cannot Be an Afterthought

Healthcare providers routinely handle highly sensitive personal health information. The IPC-OHRC Principles recommend adopting a "Privacy by Design" approach, embedding privacy protections into AI systems from the outset rather than attempting to address privacy concerns after deployment.

Recommended measures include:

• Data minimization practices;

• Use of privacy-enhancing technologies;

• De-identification or synthetic data where appropriate;

• Strong information security controls; and

• Enhanced protections for vulnerable individuals and those unable to provide informed consent.

Healthcare providers should also ensure that any collection, use, or disclosure of personal information through AI-enabled systems has a lawful basis under applicable privacy legislation.

Human Rights Considerations in Healthcare AI

Healthcare providers should also consider whether AI systems could contribute to discriminatory outcomes. The IPC-OHRC Principles emphasize that AI systems should be human rights-affirming and designed to identify and mitigate bias within training data and operational processes. Healthcare providers should proactively address potential systemic discrimination based on protected grounds under the Ontario Human Rights Code.

Caution is especially warranted where AI systems affect access to healthcare services, prioritization decisions, eligibility assessments, or clinical recommendations. Applying uniform algorithms across diverse populations without consideration of demographic differences may create adverse impacts for certain groups.

Looking Ahead

Responsible AI deployment requires more than selecting an innovative product. Healthcare organizations must ensure that AI systems are accurate, safe, privacy-protective, and consistent with human rights obligations throughout their lifecycle.

In Part 2 of this blog series, we discuss the key due diligence considerations for procuring AI tools for your practice or healthcare organization.