News of companies integrating AI into their operations has become increasingly common. While this reflects growing adoption at the top level, smaller firms are also turning to publicly available tools to remain competitive—sometimes aware of the risks, and sometimes not. There is often uncertainty about how these tools handle data, including whether user inputs may be used for model training. This article explores data protection in AI systems, the key risks involved, and the relevant legal and regulatory considerations.

Why Data Protection Matters in AI

Data protection is fundamental in the context of artificial intelligence due to the risks associated with large-scale use of personal data. AI systems rely on algorithms to detect patterns and make predictions, often enabling automated decision-making and profiling. In some cases, this can be intrusive and have significant effects on individuals.[1]

Machine learning models frequently process personal data to generate insights. For this reason, protecting personal data is essential to safeguard individuals’ rights and maintain trust in AI systems.[2]

A clear example can be seen in fintech, where AI is used during customer onboarding to verify identity documents. By analysing elements such as colour, patterns, and structure, the system determines whether a document is genuine or potentially fraudulent. Based on this, applications may be approved or rejected. This process involves profiling and facial recognition, which can significantly impact individuals, particularly where there is limited transparency or human oversight.

Because AI systems retain and process data, breaches of data protection laws can lead to reputational damage, financial penalties, and regulatory action. The ICO has issued guidance to support organisations in complying with these requirements, including the need for impact assessments, transparency, and appropriate safeguards.[3]

AI Systems and Personal Data

AI systems rely on large volumes of data, making it essential to distinguish between personal and non-personal data. In practice, datasets often contain both, requiring careful assessment to ensure compliance.

Personal data, as defined under the UK GDPR, refers to information relating to an identified or identifiable individual.[4] AI systems may process such data directly, or indirectly identify individuals by linking datasets or detecting patterns. This creates an inherent risk of non-compliance.[5]

The processing of personal data must follow key principles, including transparency, accuracy, and data minimisation. Organisations must ensure that data is handled lawfully, fairly, and securely.[6]

Non-personal data includes anonymised or aggregated datasets used for analysis. However, if individuals can be re-identified due to technological developments, the data must be treated as personal and protected accordingly.[7]

Key Data Protection Risks

AI systems present several data protection risks, particularly under the UK GDPR framework. These include bias and discrimination, security vulnerabilities, and challenges in ensuring transparency and accountability.

Bias can arise from training data or human decisions during development. For example, in legal technology, systems may produce harsher outcomes for certain individuals based on biased or inaccurate data.

Transparency is also critical. Organisations must inform individuals how their data is used and ensure they can exercise their rights, including access, correction, and deletion.[8] This can be more complex when using third-party AI tools, but responsibility remains with the organisation.

Data Protection Principles and Privacy by Design

There is no single UK authority responsible for regulating AI. Instead, AI is governed through existing legal frameworks and regulators.

The UK GDPR and the Data Protection Act 2018 form the primary legal framework. These laws regulate how personal data is processed and include provisions relevant to AI, particularly automated decision-making.

Organisations must process personal data lawfully, fairly, and transparently. They must also follow principles such as data minimisation, accuracy, and security. Article 25 requires “data protection by design and by default,” meaning safeguards must be built into systems from the outset.[9]

Article 22 addresses automated decision-making, giving individuals the right not to be subject to decisions made solely by automated processes that significantly affect them, unless certain conditions apply.[10] In such cases, organisations must provide safeguards, including human intervention and an explanation.

At a broader level, the EU AI Act introduces a comprehensive framework covering the full lifecycle of AI systems, including risk management, testing, and ongoing monitoring.

What Companies Should Do

In light of these risks, organisations must adopt a structured approach to data protection in AI systems. This requires combining legal, technical, and organisational measures.

First, organisations should establish clear governance and accountability structures. AI systems should always operate under human oversight, with clear responsibility assigned for decisions and outcomes.[11]

Second, AI should be integrated into existing risk and control frameworks, such as compliance and audit processes.[12] This ensures consistency and allows organisations to manage AI risks alongside other operational risks.

Third, organisations should focus on monitoring and testing AI systems. Given the complexity of some models, it is often more effective to validate outputs and monitor performance than rely solely on explainability.[13]

Fourth, where third-party tools are used, organisations must carry out robust vendor due diligence. This includes understanding how data is handled, stored, and potentially reused. Responsibility for compliance remains with the organisation.[14]

Fifth, organisations should implement clear internal policies on AI use, particularly regarding personal and sensitive data. This helps prevent misuse, especially when employees use publicly available tools.[15]

Finally, organisations must ensure strong data security and privacy safeguards, including appropriate technical and organisational measures to prevent unauthorised access or data breaches.[16]

Conclusion

As AI becomes more widely adopted, data protection must be treated as a core part of system design rather than an afterthought. The use of personal data, automated decision-making, and complex models creates significant legal and ethical risks for organisations. By embedding data protection principles, ensuring transparency, maintaining human oversight, and implementing strong governance structures, companies can reduce these risks while building trust in their AI systems. Ultimately, responsible AI use depends on aligning innovation with accountability, ensuring that technological progress does not come at the expense of individuals’ rights.

Disclaimer:

This article is published on 11 April 2026 by Dilmurod Erkinov at Edu-LegalTech. The information contained herein is accurate as of the date of publication and is provided for general informational and educational purposes only.

This article does not constitute legal, financial, or any other form of professional advice. It should not be relied upon as a substitute for obtaining independent advice tailored to specific circumstances. Organisations, individuals, and other readers are strongly encouraged to seek appropriate advice from qualified and certified professionals before taking or refraining from any action based on the content of this article.

While reasonable efforts have been made to ensure the accuracy and reliability of the information presented, no representations or warranties, express or implied, are made regarding its completeness, accuracy, or timeliness. The author accepts no liability for any loss or damage arising from reliance on this content.

The views expressed in this article are those of the author and do not necessarily reflect the views of any affiliated organisations.

Bibliography:
 

[1] Negotiation guide - AI contracts

[2] Negotiation guide - AI contracts

[3] Guidance on AI and data protection

[4] https://www.legislation.gov.uk/ukpga/2018/12/section/3/enacted

[5] AI related civil liability—risks and mitigation - AI related civil liability—risks and mitigation > What kinds of civil claim might AI use give rise to? > Data protection, Anonymisation, pseudonymisation and privacy enhancing technologies (PETs)

[6] REGULATION (EU) 2016/679 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 27 April 2016

[7] REGULATION (EU) 2018/1807 OF THE EUROPEAN PARLIAMENT AND OF THE COUNCIL of 14 November 2018

[8] https://ico.org.uk/for-the-public/getting-copies-of-your-information-subject-access-request/

[9] https://www.legislation.gov.uk/eur/2016/679/article/25

[10] https://www.legislation.gov.uk/eur/2016/679/article/22

[11] FMSB, Executive Summary, p.4 (“Human accountability”); Diligent, AI Governance Principles – Accountability & Oversight.

[12] FMSB, Executive Summary, p.4 (“Control frameworks”)

[13] FMSB, Executive Summary, p.4 (“Monitoring outputs”)

[14] CITMA, AI Procurement Guidance, Part A (“Trust and Partnership Assessment”); Series overview (Risk & Compliance framework)

[15] Diligent, AI Governance Policy Section

[16] Diligent, AI Governance Principles – Security and Privacy