By Michael DavisThe proliferation of artificial intelligence across healthcare promises transformative efficiencies and diagnostic breakthroughs. Yet, for Health IT Professionals and Clinicians, a critical question looms large: When does the use of Protected Health Information (PHI) in AI training data trigger the stringent compliance requirements of HIPAA, and what does this mean for enterprise procurement of AI health tools? This isn’t merely a theoretical exercise; it’s a foundational inquiry shaping the future landscape of digital health and determining which AI solutions can realistically integrate into a compliant healthcare ecosystem.
Navigating the PHI Labyrinth: AI Vendors and HIPAA Applicability
The core principle is clear: HIPAA applies when AI tools process PHI. This often necessitates either robust de-identification of data or the establishment of a Business Associate Agreement (B.A.A.) for training data. However, the nuances of “processing PHI” and “de-identification” are where the complexities arise, creating a critical filter for healthcare organizations evaluating AI partners.
Consider companies operating directly with vast datasets that often contain PHI. Vendors like Tempus AI, which leverages clinical and genomic data to advance precision medicine, and Flatiron Health, known for its oncology real-world evidence platform, inherently deal with data that, in its raw form, is undeniably PHI. Flatiron Health recently launched “Flatiron Telescope,” an AI-powered oncology intelligence platform built on over 15 years of longitudinal real-world data. Similarly, IQVIA, a global leader in healthcare data science, frequently handles extensive patient information and is committed to using AI responsibly. For these entities, and their partners, the application of the HIPAA Privacy Rule and the HIPAA Security Rule is paramount. Their AI models, trained on such rich datasets, must adhere to strict protocols regarding data access, use, and disclosure.
The challenge intensifies when AI developers seek to build powerful diagnostic or prognostic tools. PathAI, focusing on AI-powered pathology, and Paige AI, specializing in computational pathology, rely on digital pathology slides and associated patient data. PathAI received FDA Breakthrough Device Designation for PathAssist Derm in March 2026 and 510(k) clearance for AISight® Dx2. Roche has entered into a definitive merger agreement to acquire PathAI, with the acquisition expected to close in the second half of 2026. Paige AI was acquired by Tempus AI on August 22, 2025, and Tempus AI launched “Paige Predict” in January 2026, leveraging Paige’s technology to predict biomarkers from H&E slides. This data, even when anonymized or de-identified, often carries a high risk of re-identification if not handled with extreme care, especially given the granular detail present in pathology images. As legal scholars I. Glenn Cohen and Carmel Shachar have highlighted, the line between de-identified data and re-identifiable data can be surprisingly thin, posing significant compliance challenges for AI developers and their healthcare clients. Harvard Law article on re-identification risks
Conversely, some AI health apps operate in a more ambiguous space, particularly those that engage directly with consumers. Companies like BetterHelp (online therapy) and Cerebral (mental health care) collect sensitive health information directly from individuals. BetterHelp is actively expanding its insurance coverage and partnerships. Cerebral has acquired companies such as Inflow and Resilience Lab. Its former CEO, David Mou, stepped down in December 2024, with Brian Reinken now serving as interim president and CEO. While they may not always be direct “covered entities” under HIPAA in the traditional sense, their relationships with healthcare providers or health plans can trigger B.A.A. requirements. Furthermore, the very nature of their services means they are handling highly sensitive personal health information, even if their primary legal obligation stems from other consumer privacy laws. Health IT Professionals must scrutinize the data flows and contractual agreements of such platforms to ensure no PHI is inadvertently exposed or used in training models without proper consent or de-identification.
The perspective offered by experts like Ziad Obermeyer, who studies the application of machine learning in clinical decision-making, underscores the inherent tension. The more granular and comprehensive the data, the more effective the AI model. Yet, this very granularity increases the risk of PHI exposure. Therefore, for AI health apps to be viable for large employer or health-plan contracts, their data practices must demonstrate an unwavering commitment to HIPAA compliance, mirroring the robust safeguards expected from traditional healthcare providers.
Regulatory Pillars: HIPAA’s Mandate for AI
The regulatory framework governing PHI in AI training data is anchored by several key HIPAA components. The HIPAA Privacy Rule establishes national standards to protect individuals’ medical records and other personal health information. It dictates how Covered Entities and their Business Associates can use and disclose PHI. When AI tools are trained on PHI, this rule directly applies, requiring explicit authorization or falling under permitted uses like treatment, payment, or healthcare operations, or for research under specific conditions. Proposed changes to the HIPAA Privacy Rule would shorten timelines for patient record access and expand patient rights.
Central to mitigating risk is adherence to HIPAA De-identification Standards. These standards provide two methods for rendering PHI de-identified: the “Safe Harbor” method, which requires the removal of 18 specific identifiers, and the “Expert Determination” method, where a qualified statistician determines that the risk of re-identification is very small. Many AI companies leverage de-identified datasets for training. However, the bar for effective de-identification, especially with complex, multi-modal datasets, is high. The HHS OCR, responsible for enforcing HIPAA, consistently emphasizes that mere obfuscation is not sufficient; the data must truly pose a negligible risk of re-identification.
Furthermore, the HIPAA Security Rule mandates administrative, physical, and technical safeguards to ensure the confidentiality, integrity, and availability of electronic PHI (ePHI). This is crucial for AI training environments, which often involve cloud infrastructure, complex data pipelines, and numerous access points. A major update to the HIPAA Security Rule is expected to take effect in 2026, introducing new cybersecurity requirements such as mandatory multi-factor authentication (MFA), encryption of ePHI at rest and in transit, annual penetration testing, biannual vulnerability scanning, and a 72-hour data restoration requirement. The HHS OCR also released a January 2026 Cybersecurity Newsletter focusing on system hardening and security baselines. Robust encryption, access controls, audit logs, and incident response plans are non-negotiable. The ONC, working to advance health information technology, also champions interoperability and secure data exchange, further underscoring the need for compliant data practices in AI development. In June 2026, the ONC announced a new AI-Cybersecurity Governance Framework for Healthcare.
Procurement as a Compliance Gatekeeper
For Health IT Professionals, the implication is clear: HIPAA compliance must serve as a primary enterprise procurement filter for AI health tools. Any vendor lacking transparent and verifiable adherence to these regulations, particularly concerning their AI training data practices, presents an unacceptable risk. This means going beyond surface-level assurances and demanding detailed evidence of compliance, including independent audits, comprehensive BAAs, and clear methodologies for data de-identification or patient consent. The benchmark for trust and compliance is not merely meeting the minimum legal requirements, but demonstrating a proactive and mature approach to data governance and patient privacy. HHS OCR guidance on AI and HIPAA
Frequently Asked Questions
When does HIPAA apply to AI tools used in healthcare?
HIPAA applies when AI tools process Protected Health Information (PHI). This means that if an AI solution uses patient data that identifies individuals, it falls under HIPAA regulations. Healthcare organizations must ensure that their AI partners either robustly de-identify data or establish a Business Associate Agreement (BAA).
What is the significance of ‘de-identification’ for AI training data under HIPAA?
De-identification is crucial for using PHI in AI training data without triggering full HIPAA compliance requirements for every data point. However, the article highlights that the line between de-identified and re-identifiable data can be thin, especially with granular data like pathology images. This poses significant compliance challenges if not handled with extreme care.
Are AI health apps that engage directly with consumers always subject to HIPAA?
Not always directly as ‘covered entities’ in the traditional sense, but their relationships with healthcare providers or health plans can trigger Business Associate Agreement (BAA) requirements. These apps handle sensitive personal health information, and Health IT Professionals must scrutinize their data flows and contractual agreements to prevent inadvertent PHI exposure or improper use in training models.
What HIPAA rules are most relevant when AI models are trained on PHI?
The HIPAA Privacy Rule and the HIPAA Security Rule are paramount. The Privacy Rule dictates how Covered Entities and their Business Associates can use and disclose PHI, requiring explicit authorization or falling under permitted uses. The Security Rule establishes national standards to protect electronic PHI.
