AI and Data Privacy: Legal Requirements (2026)

Artificial intelligence systems consume personal data at a scale that existing privacy frameworks were not designed to address. Training datasets scraped from the internet, real-time biometric analysis, automated hiring decisions, predictive policing, and algorithmic credit scoring all raise the same core question: what rights do individuals have when AI processes their data? Lawmakers across the US, EU, and beyond are scrambling to answer that question, and the regulatory landscape is shifting faster than most organizations can track.

The EU AI Act: A Data Privacy Framework for AI

The European Union's Artificial Intelligence Act, published in the Official Journal on July 12, 2024, is the world's first comprehensive AI regulation. While it is primarily a product safety law rather than a data protection law, its data governance provisions directly affect how organizations collect, process, and retain personal data for AI purposes.

Risk-Based Classification

The AI Act categorizes AI systems into four risk tiers:

Unacceptable risk (prohibited). AI systems that manipulate human behavior, exploit vulnerabilities, score individuals for social behavior (social scoring by governments), or perform real-time remote biometric identification in public spaces (with narrow law enforcement exceptions) are banned entirely. These prohibitions took effect on February 2, 2025.

High risk. AI systems used in critical areas are subject to extensive requirements. These areas include biometric identification, critical infrastructure management, education and vocational training, employment and worker management, access to essential services (credit scoring, insurance), law enforcement, migration and border control, and administration of justice. High-risk systems must comply with data governance, transparency, human oversight, accuracy, and cybersecurity requirements.

Limited risk. AI systems that interact with humans (chatbots), generate synthetic content (deepfakes), or perform emotion recognition must meet transparency obligations. Users must be informed they are interacting with AI.

Minimal risk. All other AI systems can be developed and deployed without additional obligations, though voluntary codes of conduct are encouraged.

Data Governance Requirements for High-Risk AI

Article 10 of the AI Act imposes specific data governance requirements for high-risk AI systems. Training, validation, and testing datasets must:

• Be subject to appropriate data governance and management practices
• Account for the specific geographic, contextual, behavioral, or functional setting of the system
• Be relevant, sufficiently representative, and to the extent possible, free of errors and complete
• Have appropriate statistical properties, including with regard to the persons or groups on whom the system is intended to be used

Organizations must document the design choices, data collection processes, and data preparation operations (annotation, labeling, cleaning, enrichment). For special categories of personal data (as defined under the GDPR), the AI Act allows processing strictly for bias detection and correction purposes, subject to safeguards.

Interaction with the GDPR

The AI Act operates alongside the GDPR, not as a replacement. Organizations deploying high-risk AI systems must comply with both frameworks simultaneously. Key tensions include:

Data minimization vs. bias detection. The GDPR's data minimization principle (Article 5(1)(c)) conflicts with the AI Act's requirement to ensure training data is "sufficiently representative." Detecting and correcting bias in AI models often requires collecting demographic data (race, gender, age) that the GDPR classifies as special category data subject to heightened protections.

Purpose limitation vs. model training. Using personal data collected for one purpose (providing a service) to train an AI model (a different purpose) requires a separate legal basis under GDPR Article 6. The legitimate interest basis is available but requires a balancing test, and several data protection authorities have questioned whether model training qualifies.

Right to erasure vs. model retention. When a data subject exercises the right to erasure under GDPR Article 17, must the organization retrain its AI model to remove that individual's influence? This "machine unlearning" problem remains legally unsettled, though the Italian data protection authority (Garante) raised the issue during its enforcement action against OpenAI in 2023.

US State AI Privacy Laws

Colorado SB 205 (Colorado AI Act)

Colorado SB 21-169/SB 205, signed into law in May 2024 and effective February 1, 2026, is the first comprehensive AI legislation enacted by a US state. It focuses on "high-risk AI systems" that make or substantially assist "consequential decisions" in areas including employment, education, financial services, healthcare, housing, insurance, and legal services.

Key requirements for deployers (organizations using AI systems):

• Provide consumers with notice that AI is being used to make or assist consequential decisions
• Provide a description of the AI system in plain language, including the type of decision it makes and how it factors into the final outcome
• Allow consumers to opt out when technically feasible, and provide an appeals process for adverse decisions
• Complete an impact assessment before deploying a high-risk AI system, updated annually, covering: the system's purpose, intended benefits and risks, data inputs, performance metrics, bias testing results, and mitigation measures
• Provide human review options for consequential decisions

Key requirements for developers (companies building AI systems):

• Provide deployers with documentation about the system's capabilities, limitations, known risks, and data governance practices
• Disclose known or foreseeable risks of algorithmic discrimination
• Make available a summary of training data types and known bias testing outcomes

The Colorado Attorney General has exclusive enforcement authority. There is no private right of action. The law explicitly provides an affirmative defense for organizations that discover and cure violations within a reasonable time and maintain a risk management framework consistent with recognized AI governance standards (such as NIST AI RMF).

Illinois Artificial Intelligence Video Interview Act

The Illinois AI Video Interview Act (820 ILCS 42), effective January 1, 2020, was one of the earliest US AI-specific laws. It requires employers that use AI to analyze video interviews to:

• Notify applicants that AI will analyze the interview and explain how the AI works and what characteristics it evaluates
• Obtain the applicant's written consent before the interview
• Limit sharing of the video to those whose expertise is necessary to evaluate the applicant
• Destroy all video recordings within 30 days of a request by the applicant

The law was an early signal that AI-specific regulation would focus on transparency, consent, and data minimization.

NYC Local Law 144 (Automated Employment Decision Tools)

New York City Local Law 144, effective July 5, 2023, regulates automated employment decision tools (AEDTs) used in hiring or promotion within New York City. The law requires:

• An annual bias audit conducted by an independent auditor, with results published on the employer's website
• Notice to candidates at least 10 business days before use, including: that an AEDT will be used, the job qualifications the AEDT assesses, the data sources, and the data retention policy
• Candidates must have the option to request an alternative selection process or accommodation

The bias audit must test for disparate impact across race/ethnicity and sex categories using the selection rate or scoring rate for each group. The audit summary must be publicly available for at least 6 months.

Other State Developments

Several states have enacted or proposed AI-related privacy legislation as of early 2026:

• Utah enacted the Artificial Intelligence Policy Act (2024), which requires disclosure when interacting with generative AI and regulates AI-generated content in certain contexts, but does not directly regulate AI data privacy.
• California considered several AI bills in the 2024-2025 legislative sessions. AB 2013 (signed 2024) requires developers of generative AI systems to post high-level summaries of training data on their websites. SB 1047, which would have imposed strict safety requirements on large AI models, was vetoed by Governor Newsom in September 2024.
• Connecticut enacted PA 24-40, requiring developers and deployers of high-risk AI to conduct impact assessments and provide transparency reports. It takes effect October 1, 2026.
• Texas passed HB 1709 (2025), requiring notice and opt-out for AI in consequential decisions in healthcare, financial services, and insurance.

CCPA Automated Decision-Making Rights

The California Privacy Rights Act (CPRA) added automated decision-making provisions to the CCPA. Cal. Civ. Code 1798.185(a)(16) directed the California Privacy Protection Agency (CPPA) to issue regulations governing:

• The right to opt out of the use of automated decision-making technology, including profiling
• The right to access information about the logic involved in automated decision-making processes and the likely outcome for the consumer
• Requirements for businesses that use automated decision-making to conduct cybersecurity and risk assessments

The CPPA published draft automated decision-making regulations in late 2024 and revised them in 2025. As of early 2026, the final regulations are expected to take effect in the second half of 2026. The draft regulations define "automated decision-making technology" broadly to include any system that processes personal information to make or assist decisions replacing human decision-making.

Under the proposed rules, consumers would have the right to opt out of automated decisions in contexts including employment, healthcare, financial services, housing, education, and insurance. Businesses would need to provide pre-use notices, access to the logic of the decision, and a human review option for significant decisions.

FTC Enforcement on AI

The Federal Trade Commission has emerged as the most active US federal enforcer on AI and data privacy. The FTC uses its existing authority under Section 5 of the FTC Act (unfair or deceptive practices) to address AI-related harms.

Key FTC AI Enforcement Actions

Rite Aid (2023). The FTC ordered Rite Aid to stop using facial recognition technology after the company deployed an AI system that misidentified customers as shoplifters, disproportionately affecting women and people of color. The consent order banned Rite Aid from using facial recognition for five years and required deletion of all data collected through the system.

Weight Watchers/Kurbo (2022). The FTC required WW International to delete algorithms and AI models trained on children's data collected without verifiable parental consent in violation of COPPA. This "algorithmic disgorgement" remedy, requiring deletion of not just the data but the AI models derived from it, established a precedent that has significant implications for AI companies.

Amazon/Alexa (2023). The FTC alleged Amazon violated COPPA by retaining children's voice recordings and geolocation data indefinitely to improve Alexa's AI models. Amazon paid a $25 million penalty and agreed to delete the data and models derived from it.

FTC Guidance on AI Claims

The FTC has published multiple blog posts and guidance documents warning companies about:

• Exaggerated AI claims. Claiming AI capabilities that do not exist or overstating the accuracy of AI systems constitutes deceptive advertising.
• Biased AI outcomes. AI systems that produce discriminatory results can constitute unfair practices, particularly in credit, housing, and employment.
• Dark patterns for AI consent. Using manipulative design to obtain consent for AI data processing violates Section 5.

Training Data and Consent

One of the most contested areas of AI and privacy law involves the use of personal data to train AI models. The legal question is straightforward: does using someone's data to train an AI model require their consent?

The GDPR Perspective

Under the GDPR, model training constitutes "processing" of personal data and requires a lawful basis under Article 6. The available bases include:

• Consent (Article 6(1)(a)). Explicit, informed consent for the specific purpose of AI training. This is the most protective but least scalable option.
• Legitimate interest (Article 6(1)(f)). The controller must demonstrate a legitimate interest that is not overridden by the data subject's rights. Several DPAs have questioned whether commercial AI training qualifies.
• Contract performance (Article 6(1)(b)). If AI training is necessary to provide a service the user signed up for. This basis has been challenged by the Irish DPC in the context of social media companies training AI on user content.

The Italian Garante temporarily banned ChatGPT in March 2023 over concerns about the lawful basis for processing training data, transparency, and age verification. OpenAI was permitted to resume service after implementing changes, but the enforcement action signaled that European regulators view AI training data as subject to full GDPR compliance.

The US Perspective

The US has no federal law specifically addressing consent for AI training data. However:

• The CCPA grants consumers the right to know how their data is used, which would include AI training. The proposed automated decision-making regulations would add further transparency requirements.
• Copyright law (currently being litigated in multiple federal courts) may limit the use of copyrighted content for training, though this is a separate issue from privacy.
• The FTC's algorithmic disgorgement precedent means that AI models trained on unlawfully collected data may need to be deleted entirely.

Scraping and Public Data

Many AI training datasets are built from publicly available internet data. Whether scraping public data for AI training violates privacy law depends on the jurisdiction:

• Under the GDPR, data being publicly available does not eliminate the need for a lawful basis. Data subjects retain their rights regardless of whether their data is publicly accessible.
• In the US, the Ninth Circuit ruled in hiQ Labs v. LinkedIn (2022) that scraping publicly available data does not violate the Computer Fraud and Abuse Act, but this does not address state privacy law claims.
• Several class action lawsuits filed in 2023-2025 allege that AI companies violated state privacy laws by scraping personal data for model training without notice or consent.

Emerging Federal Legislation

As of early 2026, no comprehensive federal AI privacy law has been enacted, but several proposals are progressing:

The American Privacy Rights Act (APRA). Introduced in April 2024, this bipartisan bill would establish federal data privacy standards including provisions for algorithmic decision-making. The bill includes a right to opt out of AI-based profiling and targeted advertising, requirements for civil rights impact assessments, and a private right of action. It passed committee markup but stalled on the Senate floor.

The AI Accountability Act. Proposed in 2024, this bill would require impact assessments for AI systems used in critical decisions and mandate transparency about AI system capabilities, limitations, and data sources.

Executive Order 14110 (October 2023). President Biden's Executive Order on Safe, Secure, and Trustworthy AI directed federal agencies to develop AI safety standards, required developers of powerful AI systems to share safety test results with the government, and ordered NIST to develop AI risk management standards. While not privacy legislation per se, several provisions address data privacy in AI contexts.

Sources and References

This article provides general legal information about the intersection of AI and data privacy law. This is a rapidly evolving area with new legislation, regulations, and enforcement actions emerging frequently. Consult an attorney for advice specific to your situation.