AI and Data Privacy: Legal Requirements (2026)

Artificial intelligence systems consume personal data at a scale that existing privacy frameworks were not designed to address. Training datasets scraped from the internet, real-time biometric analysis, automated hiring decisions, predictive policing, and algorithmic credit scoring all raise the same core question: what rights do individuals have when AI processes their data? Lawmakers across the US, EU, and beyond are scrambling to answer that question, and the regulatory landscape is shifting faster than most organizations can track.

The EU AI Act: A Data Privacy Framework for AI

The European Union's Artificial Intelligence Act, published in the Official Journal on July 12, 2024, is the world's first comprehensive AI regulation. While it is primarily a product safety law rather than a data protection law, its data governance provisions directly affect how organizations collect, process, and retain personal data for AI purposes.

Risk-Based Classification

The AI Act categorizes AI systems into four risk tiers:

Unacceptable risk (prohibited). AI systems that manipulate human behavior, exploit vulnerabilities, score individuals for social behavior (social scoring by governments), or perform real-time remote biometric identification in public spaces (with narrow law enforcement exceptions) are banned entirely. These prohibitions took effect on February 2, 2025.

High risk. AI systems used in critical areas are subject to extensive requirements. These areas include biometric identification, critical infrastructure management, education and vocational training, employment and worker management, access to essential services (credit scoring, insurance), law enforcement, migration and border control, and administration of justice. High-risk systems must comply with data governance, transparency, human oversight, accuracy, and cybersecurity requirements.

Limited risk. AI systems that interact with humans (chatbots), generate synthetic content (deepfakes), or perform emotion recognition must meet transparency obligations. Users must be informed they are interacting with AI.

Minimal risk. All other AI systems can be developed and deployed without additional obligations, though voluntary codes of conduct are encouraged.

Data Governance Requirements for High-Risk AI

Article 10 of the AI Act imposes specific data governance requirements for high-risk AI systems. Training, validation, and testing datasets must:

• Be subject to appropriate data governance and management practices
• Account for the specific geographic, contextual, behavioral, or functional setting of the system
• Be relevant, sufficiently representative, and to the extent possible, free of errors and complete
• Have appropriate statistical properties, including with regard to the persons or groups on whom the system is intended to be used

Organizations must document the design choices, data collection processes, and data preparation operations (annotation, labeling, cleaning, enrichment). For special categories of personal data (as defined under the GDPR), the AI Act allows processing strictly for bias detection and correction purposes, subject to safeguards.

Interaction with the GDPR

The AI Act operates alongside the GDPR, not as a replacement. Organizations deploying high-risk AI systems must comply with both frameworks simultaneously. Key tensions include:

Data minimization vs. bias detection. The GDPR's data minimization principle (Article 5(1)(c)) conflicts with the AI Act's requirement to ensure training data is "sufficiently representative." Detecting and correcting bias in AI models often requires collecting demographic data (race, gender, age) that the GDPR classifies as special category data subject to heightened protections.

Purpose limitation vs. model training. Using personal data collected for one purpose (providing a service) to train an AI model (a different purpose) requires a separate legal basis under GDPR Article 6. The legitimate interest basis is available but requires a balancing test, and several data protection authorities have questioned whether model training qualifies.

Right to erasure vs. model retention. When a data subject exercises the right to erasure under GDPR Article 17, must the organization retrain its AI model to remove that individual's influence? This "machine unlearning" problem remains legally unsettled, though the Italian data protection authority (Garante) raised the issue during its enforcement action against OpenAI in 2023.

US State AI Privacy Laws

Colorado's AI Act: SB 24-205, Repealed and Replaced by SB 26-189

Colorado SB 24-205, signed into law on May 17, 2024, was the first comprehensive AI legislation enacted by a US state. It targeted "high-risk AI systems" that make or substantially assist "consequential decisions" in areas including employment, education, financial services, healthcare, housing, insurance, and legal services, but its effective date was delayed twice and it was repealed before it ever took effect. Governor Polis signed SB 26-189 on May 14, 2026, repealing and reenacting the law as a narrower Automated Decision-Making Technology (ADMT) Act, effective January 1, 2027.

As originally enacted, SB 24-205 would have required deployers to provide consumers notice, plain-language system descriptions, an opt-out option, annual impact assessments, and human review for consequential decisions, and would have required developers to document system capabilities, limitations, and bias-testing results. None of that ever took legal effect.

Under SB 26-189, the requirements that actually apply from January 1, 2027 are narrower. Key requirements for deployers:

• Provide consumers clear and conspicuous notice when they interact with a covered AI system
• Disclose the AI system's role within 30 days of an adverse consequential decision
• Provide consumers a right to request the personal data used in the decision, correct factually inaccurate data, and request meaningful human review of an adverse outcome

Key requirements for developers:

• Provide deployers with technical documentation describing intended uses, training-data categories, known limitations, and usage instructions
• Notify deployers of material updates or modifications to the system
• Retain compliance records for at least three years

The Colorado Attorney General has exclusive enforcement authority under SB 26-189 as well; there is no general private right of action, and violators receive a 60-day cure notice before an enforcement action can proceed (through 2030). The law also allocates fault between developers and deployers in discrimination-related claims.

Illinois Artificial Intelligence Video Interview Act

The Illinois AI Video Interview Act (820 ILCS 42), effective January 1, 2020, was one of the earliest US AI-specific laws. It requires employers that use AI to analyze video interviews to:

• Notify applicants that AI will analyze the interview and explain how the AI works and what characteristics it evaluates
• Obtain the applicant's written consent before the interview
• Limit sharing of the video to those whose expertise is necessary to evaluate the applicant
• Destroy all video recordings within 30 days of a request by the applicant

The law was an early signal that AI-specific regulation would focus on transparency, consent, and data minimization.

NYC Local Law 144 (Automated Employment Decision Tools)

New York City Local Law 144, effective July 5, 2023, regulates automated employment decision tools (AEDTs) used in hiring or promotion within New York City. The law requires:

• An annual bias audit conducted by an independent auditor, with results published on the employer's website
• Notice to candidates at least 10 business days before use, including: that an AEDT will be used, the job qualifications the AEDT assesses, the data sources, and the data retention policy
• Candidates must have the option to request an alternative selection process or accommodation

The bias audit must test for disparate impact across race/ethnicity and sex categories using the selection rate or scoring rate for each group. The audit summary must be publicly available for at least 6 months.

Other State Developments

Several states have enacted or proposed AI-related privacy legislation as of early 2026:

• Utah enacted the Artificial Intelligence Policy Act (2024), which requires disclosure when interacting with generative AI and regulates AI-generated content in certain contexts, but does not directly regulate AI data privacy.
• California considered several AI bills in the 2024-2025 legislative sessions. AB 2013 (signed 2024) requires developers of generative AI systems to post high-level summaries of training data on their websites. SB 1047, which would have imposed strict safety requirements on large AI models, was vetoed by Governor Newsom in September 2024.
• Connecticut enacted the Artificial Intelligence Responsibility and Transparency Act (Substitute SB 5, Public Act 26-15), signed by Governor Lamont on May 27, 2026. AI-related WARN Act notice requirements take effect October 1, 2026; automated-employment-decision-technology disclosures take effect October 1, 2027; AI companion chatbot rules take effect January 1, 2027.
• Texas enacted the Texas Responsible Artificial Intelligence Governance Act (HB 149, "TRAIGA"), signed by Governor Abbott on June 22, 2025 and effective January 1, 2026. The original, broader HB 1709 draft, which would have required consumer notice and opt-out for consequential AI decisions in healthcare, financial services, and insurance, did not pass. TRAIGA instead prohibits specific AI uses, such as government social scoring and unlawful discrimination, and is enforced solely by the Texas Attorney General with a 60-day cure period.

CCPA Automated Decision-Making Rights

The California Privacy Rights Act (CPRA) added automated decision-making provisions to the CCPA. Cal. Civ. Code 1798.185(a)(15) directed the California Privacy Protection Agency (CPPA) to issue regulations governing:

• The right to opt out of the use of automated decision-making technology, including profiling
• The right to access information about the logic involved in automated decision-making processes and the likely outcome for the consumer
• Requirements for businesses that use automated decision-making to conduct cybersecurity and risk assessments

The CPPA Board adopted final automated decision-making, risk-assessment, and cybersecurity-audit regulations on July 24, 2025, and the Office of Administrative Law approved them on September 22, 2025. The regulations took effect January 1, 2026, with ADMT-specific business compliance required by January 1, 2027. The regulations define "automated decision-making technology" broadly to include any system that processes personal information to make or assist decisions replacing human decision-making.

Under the regulations, consumers have the right to opt out of automated decisions in contexts including employment, healthcare, financial services, housing, education, and insurance. Businesses must provide pre-use notices, access to the logic of the decision, and a human review option for significant decisions.

FTC Enforcement on AI

The Federal Trade Commission has emerged as the most active US federal enforcer on AI and data privacy. The FTC uses its existing authority under Section 5 of the FTC Act (unfair or deceptive practices) to address AI-related harms.

Key FTC AI Enforcement Actions

Rite Aid (2023). The FTC ordered Rite Aid to stop using facial recognition technology after the company deployed an AI system that misidentified customers as shoplifters, disproportionately affecting women and people of color. The consent order banned Rite Aid from using facial recognition for five years and required deletion of all data collected through the system.

Weight Watchers/Kurbo (2022). The FTC required WW International to delete algorithms and AI models trained on children's data collected without verifiable parental consent in violation of COPPA. This "algorithmic disgorgement" remedy, requiring deletion of not just the data but the AI models derived from it, established a precedent that has significant implications for AI companies.

California SB 1247 (2026). California's proposed SB 1247 would let children demand deletion of monetized content featuring them once they turn 18 - extending deletion rights into the social media era where parents routinely post images and videos of their children that generate revenue.

Amazon/Alexa (2023). The FTC alleged Amazon violated COPPA by retaining children's voice recordings and geolocation data indefinitely to improve Alexa's AI models. Amazon paid a $25 million penalty and agreed to delete the data and models derived from it.

FTC Guidance on AI Claims

The FTC has published multiple blog posts and guidance documents warning companies about:

• Exaggerated AI claims. Claiming AI capabilities that do not exist or overstating the accuracy of AI systems constitutes deceptive advertising.
• Biased AI outcomes. AI systems that produce discriminatory results can constitute unfair practices, particularly in credit, housing, and employment.
• Dark patterns for AI consent. Using manipulative design to obtain consent for AI data processing violates Section 5.

Training Data and Consent

One of the most contested areas of AI and privacy law involves the use of personal data to train AI models. The legal question is straightforward: does using someone's data to train an AI model require their consent?

The GDPR Perspective

Under the GDPR, model training constitutes "processing" of personal data and requires a lawful basis under Article 6. The available bases include:

• Consent (Article 6(1)(a)). Explicit, informed consent for the specific purpose of AI training. This is the most protective but least scalable option.
• Legitimate interest (Article 6(1)(f)). The controller must demonstrate a legitimate interest that is not overridden by the data subject's rights. Several DPAs have questioned whether commercial AI training qualifies.
• Contract performance (Article 6(1)(b)). If AI training is necessary to provide a service the user signed up for. This basis has been challenged by the Irish DPC in the context of social media companies training AI on user content.

The Italian Garante temporarily banned ChatGPT in March 2023 over concerns about the lawful basis for processing training data, transparency, and age verification. OpenAI was permitted to resume service after implementing changes, but the enforcement action signaled that European regulators view AI training data as subject to full GDPR compliance.

The US Perspective

The US has no federal law specifically addressing consent for AI training data. However:

• The CCPA grants consumers the right to know how their data is used, which includes AI training, and the CPPA's ADMT regulations, in effect since January 1, 2026, add further transparency requirements.
• Copyright law (currently being litigated in multiple federal courts) may limit the use of copyrighted content for training, though this is a separate issue from privacy. In early 2026, hundreds of authors returned book advances to protest unconsented use of their writing for AI training - a public pressure campaign running in parallel with the ongoing federal litigation.
• The FTC's algorithmic disgorgement precedent means that AI models trained on unlawfully collected data may need to be deleted entirely.

Scraping and Public Data

Many AI training datasets are built from publicly available internet data. Whether scraping public data for AI training violates privacy law depends on the jurisdiction:

• Under the GDPR, data being publicly available does not eliminate the need for a lawful basis. Data subjects retain their rights regardless of whether their data is publicly accessible.
• In the US, the Ninth Circuit ruled in hiQ Labs v. LinkedIn (2022) that scraping publicly available data does not violate the Computer Fraud and Abuse Act, but this does not address state privacy law claims.
• Several class action lawsuits filed in 2023-2025 allege that AI companies violated state privacy laws by scraping personal data for model training without notice or consent.

Emerging Federal Legislation

As of early 2026, no comprehensive federal AI privacy law has been enacted, but several proposals are progressing:

The American Privacy Rights Act (APRA). Introduced in April 2024, this House bill would have established federal data privacy standards including provisions for algorithmic decision-making, a right to opt out of AI-based profiling and targeted advertising, and requirements for civil rights impact assessments. The House Energy and Commerce Committee canceled its scheduled markup on June 27, 2024 amid opposition from privacy advocates and committee Republicans, and the bill never received a floor vote in either chamber. APRA expired with the end of the 118th Congress in January 2025 and has not been reintroduced as of mid-2026.

The AI Accountability Act. Proposed in 2024, this bill would require impact assessments for AI systems used in critical decisions and mandate transparency about AI system capabilities, limitations, and data sources.

Executive Order 14110 (October 2023, revoked January 2025). President Biden's Executive Order on Safe, Secure, and Trustworthy AI directed federal agencies to develop AI safety standards, required developers of powerful AI systems to share safety test results with the government, and ordered NIST to develop AI risk management standards. President Trump revoked EO 14110 on January 20, 2025, and on January 23, 2025 signed Removing Barriers to American Leadership in Artificial Intelligence, directing agencies to suspend, revise, or rescind any EO 14110-derived actions inconsistent with a deregulatory approach to AI. As of mid-2026, no comparable binding federal AI privacy directive has replaced it, and Congress has not enacted federal AI privacy legislation.

Sources and References

This article provides general legal information about the intersection of AI and data privacy law. This is a rapidly evolving area with new legislation, regulations, and enforcement actions emerging frequently. Consult an attorney for advice specific to your situation.

Related news

• Supreme Court Upholds FCC Fines Over Carrier Location Data (2026)