Part 2: AI Procurement in Healthcare: Key Due Diligence Considerations

Artificial intelligence (AI) is being increasingly integrated into healthcare delivery, from clinical decision support tools to administrative tasks. While these technologies offer opportunities to improve efficiency and patient outcomes, they also create new legal, privacy, and human rights considerations.

Recognizing these challenges, the Office of the Information and Privacy Commissioner of Ontario (IPC) and the Ontario Human Rights Commission (OHRC) jointly released the Principles for the Responsible Use of Artificial Intelligence (the “IPC-OHRC Principles”). These principles provide guidance for the development, procurement, deployment, and oversight of AI systems used in Ontario's public sector and offer valuable insight for healthcare organizations adopting AI technologies. The IPC has also published AI Scribes: Key Considerations for the Health Sector and an accompanying checklist, which applies the IPC-OHRC Principles to the AI Scribes use case.

In this three-part blog series, we discuss the takeaways from the IPC-OHRC Principles for the use, procurement, and development of AI systems in the healthcare sector. In Part 2, we discuss planning considerations for AI use and maintaining patient privacy across its lifecycle. If you missed Part 1, find it here.

Procurement Is a Governance Exercise

For healthcare providers, procuring AI technology involves far more than evaluating functionality and cost. AI procurement decisions should be treated as accountability-focused governance decisions rather than ordinary technology purchases.

The IPC-OHRC Principles place particular emphasis on transparency and accountability during the procurement phase, requiring healthcare providers to understand how AI systems operate and establish appropriate oversight before implementation.

Transparency Requirements for AI Procurement

Healthcare providers should be able to understand, assess, and explain how an AI system functions before acquiring it. According to the IPC-OHRC Principles, AI systems should be explainable, traceable, and supported by sufficient technical documentation to allow organizations to assess both legal compliance and operational risks.

When evaluating vendors, healthcare organizations should consider whether they can obtain information relating to:

• System design and functionality;

• Training data sources;

• Known limitations and risks;

• Performance metrics and error rates;

• Validation and testing methodologies; and

• Ongoing monitoring requirements.

Organizations should also be able to explain how an AI system reaches conclusions or produces recommendations, particularly where outputs may affect patients, employees, or members of the public.

Documentation Matters

Meaningful transparency depends on adequate documentation. The IPC-OHRC Principles recommend maintaining documentation sufficient to:

• Explain system operations;

• Justify AI-assisted decisions;

• Track changes over time;

• Reconstruct system processes where necessary; and

• Demonstrate compliance with legal obligations

Healthcare organizations should consider addressing these requirements contractually during procurement negotiations rather than attempting to obtain information after implementation.

Accountability Must Be Built into Governance Structures

Transparency alone is insufficient. Organizations must also establish internal accountability mechanisms.

The IPC-OHRC Principles identify several governance measures that organizations should consider, including:

• Privacy impact assessments;

• Human rights assessments;

• Algorithmic risk assessments;

• Human-in-the-loop oversight processes; and

• Designated personnel responsible for AI governance and oversight.

Governance frameworks should also provide mechanisms to pause, restrict, or decommission systems that become unsafe, unreliable, or non-compliant.

Privacy Compliance and Legal Authority

Healthcare providers must determine whether an AI system's intended collection, use, and processing of information is authorized under applicable privacy legislation. In particular, healthcare providers should know where personal health information is stored or transmitted.

The IPC-OHRC Principles further recommend that individuals be informed when their personal information is used in connection with AI systems and, where appropriate, have opportunities to opt out of high-risk automated decision-making in favour of human review.

Organizations should also be attentive to broader constitutional and human rights considerations, particularly where AI systems may disproportionately affect vulnerable or marginalized populations.

Questions Healthcare Providers Should Ask Vendors

Before signing an AI vendor contract, healthcare organizations may wish to ask:

• How was the system trained and validated?

• What error rates have been identified?

• How are bias and discrimination risks assessed?

• What privacy protections are built into the system?

• What documentation will be provided?

• How will updates and changes be communicated?

• What audit rights are available?

• Under what circumstances can the system be suspended or withdrawn?

These questions can help organizations align procurement decisions with both regulatory expectations and emerging best practices.

Looking Ahead

Strong procurement processes create the foundation for responsible AI deployment. By prioritizing transparency, accountability, privacy, and human rights considerations before implementation, healthcare organizations can reduce legal risk while maintaining public trust.

In Part 3, we examine what AI developers should understand when designing systems for healthcare and other public-sector environments.