The rapid development of AI technologies like large language models (LLMs) as well as other ML continues to dominate the technology world. While AI is being incorporated into nearly every aspect of daily life, this rapid deployment is not without some significant growing pains. In order for LLMs and ML to function effectively, they must be trained on massive datasets that can range from works of literature to social media posts to personally identifiable information (PII). However, the vulnerability of AI models to cyberattacks creates serious privacy concerns. Although there are methods that can be used to mask or transform training data, these often yield less reliable and less effective results, which frequently prevents developers from employing these safeguards. Fortunately, there is work being done to reduce the risks of privacy.
Privacy Risks in AI Training Data
While the training data used in AI models presents many risks to privacy, there are three primary areas of concern according to Dr. Omer Akgul, Principal Researcher at RSAC.
1. Memorization of Sensitive Data: Given the right circumstances, LLMs are capable of memorizing and reproducing complete sets of the data used to train them. “LLMs often memorize input data and may reproduce it verbatim if prompted in the right way. This data might contain private user or company data, putting such data at risk if the models are deployed in the wild,” Akgul said this information can include personally identifiable information (PII), proprietary company data, as well as other sensitive sources of information. If these same LLMs are deployed for use without the proper safeguards, they present a very real security concern.
2. Persistence of Removed Data: Even if sensitive information is later removed from the training dataset, it is impossible to remove the influence of that data from the model. Akgul said, “Removing data from the training set is easy, however, fully removing the influence of such data from an already trained model (without significantly reducing model accuracy) is currently not possible.” The model will still retain patterns of behavior and other information based on the removed data, thus continuing to be a source of concern when it comes to safeguarding private information.
3. Ineffectiveness of Data Masking: Attempts to alter PII through masking or transforming the data have been unsuccessful due to the difficulty of ensuring that the data is rendered completely unrecognizable. Akgul said, “There are no privacy preserving methods to include private data in the training set… either the data can’t be properly transformed to be safe… or when used the data produces significantly less accurate models.” The loss in speed and efficacy completely undermines the purpose behind training the AI in the first place and is currently impractical.
The Right to Erasure
The right to erasure, also known as the right to be forgotten, is a data protection right that allows individuals to request that organizations delete their personal data. While this may seem like an attractive option for those looking to safeguard their PII, this right only applies in certain circumstances and is only effective in instances where a company or organization is actually capable of compliance.
Currently, the most effective way to meet such requests is to completely delete the current model and retrain it without the affected datasets in a process which is known as “machine unlearning.” Counterintuitively, this process can often create an even greater security risk to the sensitive data by opening the door to what are known as “reconstruction attacks.” These are attacks that specifically target LLMs and allow cybercriminals to completely recreate the deleted datasets by exploiting vulnerabilities in the model.
Mitigating Privacy Risks in AI Training
Because training data is so difficult, researchers have been studying how to best achieve a formal level of privacy for LLM, according to Dr. Dario Pasquini, Principal Researcher at RSAC. “Today, strictly provable privacy guarantees are achieved primarily through techniques such as differential privacy.”
Differential privacy techniques obscure data to the point of being effectively unrecognizable, according to Pasquini. Unfortunately, the technology is not scalable and is difficult to apply to larger models due to associated expenses, reduced speed and lower accuracy. However, this approach is being used to great effect in smaller models.
To mitigate the risks of memorization, researchers have developed techniques such as the “Goldfish” loss function which includes random sets of data that are not memorized during the training process. These “missing” sections of information reduce the likelihood that the model would recreate verbatim training data.
For these reasons, it is important for developers to carefully curate the data they use in training. Pasquini said, “Careful data engineering also substantially reduces the risk of accidental memorization. First, remove obvious personally identifiable information (PII) from the corpus before training. Second, perform thorough data deduplication.”
Removing all sensitive information, including PII, before using it to train AI prevents the model from having the ability to regurgitate protected data. In addition, it is important for developers to ensure that data is deduplicated. When duplicate sets of data are present in the training model, it significantly increases the likelihood that the data will be memorized and subject to recall.
Pasquini said, “Always red-team models for privacy leakage before deployment. Attacks such as membership inference and targeted extraction tests provide an empirical measure of memorization and reveal risky behavior that you can address prior to public release. This allows for developers to understand the model’s susceptibility to compromise and to determine what data it is capable of reproducing. Red teaming, combined with several other mitigating strategies, can provide a reasonable guarantee of safety regarding the information used in training AI models.
Staying Up to Date
Artificial intelligence is still in its infancy, and developers are only beginning to scratch the surface of its true potential. Part of this growth is understanding the risks associated with the training and deployment of various AI models. It is important to stay educated on the latest developments, trends, and tactics by visiting the new RSAC Community Platform, which contains content from some of the top minds in the industry, opportunities to network and communicate with cybersecurity peers, as well as a powerful Artificial Intelligence (AI) assistant that can help you find exactly what you need to stay up to date and one step ahead.