Data Privacy and Security Risks in Scientific AI Applications

As AI technologies continue to develop, every industry must calibrate its risk management standards and practices to meet the moment.

Scientific fields are no exception. The data privacy and security risks of AI are particularly pronounced in areas of science that are focused on human subjects, especially medical and social sciences. For legal advice, reach out to a science and technology lawyer.

AI Risks in the Medical and Life Sciences

The potential for revolutionary AI applications in the medical sciences has been widely touted, even by critics of the broader AI industry. The use of new predictive AI tools in cancer detection has attracted particular interest.

Because artificial intelligence and machine learning systems are trained on massive amounts of data, and because, in medical science, that data is derived from human subjects, the ethical use of AI to deliver medical advances requires careful management of risks around informed consent and data security.

Get Help With Technology Law Matters

Technology law is changing and catching up. For legal help with tech law and regulation, use the Super Lawyers directory to find a technology lawyer near you.

Find a lawyer today

The Role of HIPAA in Securing Healthcare Data and Privacy Concerns

In the United States, the use of patient data in medical research is primarily governed by the Health Insurance Portability and Accountability Act (HIPAA).

HIPAA specifies what types of data healthcare providers can collect about patients, how it can be stored and used by the provider, and what consent is needed from patients before their data can be shared with other parties such as insurance companies and other providers.

Under HIPAA, providers can also obtain consent from patients for their data to be deidentified and used for medical research purposes. Since this data is no longer associated with a specific patient, HIPAA does not historically govern what researchers do with that data.

However, in recent years, the proliferation of datasets sourced from social media, public records, and other sources, together with the introduction of sophisticated AI pattern-recognition algorithms, creates a risk that a patient’s deidentified personal data could be reidentified with them through, for instance, distinguishing marks like tattoos.

In light of these developments, medical research institutions will need to review their data privacy, informed consent, and cybersecurity practices. Patients may need to be informed of the risk of reidentification, promised that their deidentified data will only be used for a single purpose and not sold or transferred for additional uses, and/or given more information about how exactly their data will be processed.

AI Considerations in Social Science Data Gathering and Management

The collection and use of human data is also a prominent aspect of social sciences such as political science, sociology, and economics.

Data may be collected on a one-time basis in the form of a survey that individuals are invited to complete online. Alternatively, a group of subjects may be interviewed or observed repeatedly over time in a longitudinal or panel study. In either case, demographic information about participants is an important part of the dataset, and personally identifying information may be shared with and retained by researchers.

Data privacy and informed consent considerations are important in the design of any study in the social sciences. The use of AI systems to analyze any data that is gathered creates additional risks that need to be taken into consideration. Generally, at least a portion of every AI model is a “black box” in which the algorithm operates on data in an indeterminate manner. This characteristic of AI increases the risk that personally identifying information could be shared or retained in a manner that the subject has not given consent for.

State-Level and International Data Privacy Regulations

In response to these types of risks, some states have started adopting laws specifying how data privacy obligations apply to AI systems.

For instance, in 2024, California amended their Consumer Privacy Act (CCPA) to specify that “personal information” protected by the law includes “abstract digital formats” and “artificial intelligence systems that are capable of outputting personal information.” The law also specifies how data collected by business entities (including the survey companies that power much of social science data-gathering) can and cannot be used for research purposes.

Laws in other jurisdictions, such as the European Union’s General Data Protection Regulation (GDPR), may also apply to data collected across national boundaries. It will become increasingly important for anyone collecting or retaining social science data to audit their practices and datasets regularly to ensure compliance with these laws and professional ethical obligations.

Concerns About How AI Impacts Social Science Surveys

Tangential to the data protection risks of AI models, there has also been significant concern about these models negatively impacting the integrity of social science surveys.

Typically, people completing these surveys are paid for their time, so there is an incentive for bad actors to develop AI tools that will complete surveys with no human involvement, or for human respondents to rely on generative AI to produce responses more quickly, making their responses less unique and personal.

Social scientists will need to scrutinize the practices of survey companies to ensure that they are taking adequate safeguards against this risk.

Identifying and Mitigating the Privacy Risks of AI Usage

In scientific research as in any other industry or academic field, institutions will need to evolve their practices, and potentially adopt new practices, to guard against the risks of using AI systems to process data, especially sensitive data from human subjects.

Only with the proper safeguards in place can science safely explore the potential benefits of artificial intelligence. For legal advice, speak with an experienced science and technology lawyer.