Under a newly released action plan for artificial intelligence, the technology will be integrated into U.S. government functions. The plan, announced July 23, is another step in the Trump administration’s push for an “AI-first strategy.”
In July, for instance, the U.S. Department of Defense handed out $200 million contracts to Anthropic, Google, OpenAI and xAI. Elon Musk’s xAI announced “Grok for Government,” where federal agencies can purchase AI products through the General Services Administration. And all that comes after months of reports that the advisory group called the Department of Government Efficiency has gained access to personal data, health information, tax information and other protected data from various government departments, including the Treasury Department and Veteran Affairs. The goal is to aggregate it all into a central database.
But experts worry about potential privacy and cybersecurity risks of using AI tools on such sensitive information, especially as precautionary guardrails, such as limiting who can access certain types of data, are loosened or disregarded.
To understand the implications of using AI tools to process health, financial and other sensitive data, Science News spoke with Bo Li, an AI and security expert from University of Illinois Urbana-Champaign, and Jessica Ji, an AI and cybersecurity expert at Georgetown University’s Center for Security and Emerging Technology in Washington, D.C. This interview has been edited for length and clarity.
SN: What are the risks of using AI models on private and confidential data?
Li: First is data leakage. When you use sensitive data to train or fine-tune the model, it can memorize the information. Say you have patient data trained in the model, and you query the model asking how many people have a particular disease, the model may exactly answer it or may leak the information that [a specific] person has that disease. Several people have shown that the model can even leak credit card numbers, email addresses, your residential address and other sensitive and personal information.
Second, if the private information is used in the model’s training or as reference information for retrieval-augmented generation, then the model could use such information for other inferences [such as tying personal data together].
SN: What are the risks associated with consolidating data from different sources into one large dataset?
Ji: When you have consolidated data, you just make a bigger target for adversarial hackers. Rather than having to hack four different agencies, they can just target your consolidated data source.
In the U.S. context, previously, certain organizations have avoided combining, for example, personally identifiable information and linking someone’s name and address with health conditions that they may have.
On consolidating government data to train AI systems, there are major privacy risks associated with it. The idea that you can establish statistical linkages between certain things in a large dataset, especially containing sensitive information such as financial and medical and health information, just carries civil liberties and privacy risks that are quite abstract. Certain people will be adversely impacted but they may not be able to link the impacts to this AI system.
SN: What cyberattacks are possible?
Li: A membership attack is one, which means if you have a model trained with some sensitive data, by querying the models, you want to know, basically the membership, if a particular person is in this [dataset] or not.
Second is model inversion attack, in which you recover not only the membership but also the whole instance of the training data. For example, there’s one person with a record of their age, name, email address and credit card number, and you can recover the whole record from the training data.
Then, model stealing attack means you actually steal the model weights [or parameters], and you can recover the model [and can leak additional data].
SN: If the model is secure, would it be possible to contain the risk?
Li: You can secure the model in certain ways, like by forming a guardrail model, which identifies the sensitive information in the input and output and tries to filter them, outside the main model as an AI firewall. Or there are strategies for training the model to forget information, which is called unlearning. But it’s ultimately not solving the problem because, for example, unlearning can hurt the performance and also cannot guarantee that you unlearn certain information. And for guardrail models, we will need stronger and stronger guardrails for all kinds of diverse attacks and sensitive information leakage. So I think there are improvements on the defense side, but not a solution yet.
SN: What would your recommendations be for the use of AI with sensitive, public, government data?
Ji: Prioritizing security and thinking about the risks and benefits and making sure that your existing risk management processes can adapt to the nature of AI tools.
What we have heard from various organizations both in government and the private sector is that you have a very strong top-down messaging from your CEO or from your agency head to adopt AI systems right away to keep up with the rivals. It’s the people lower down who are tasked with actually implementing the AI systems and oftentimes they’re under a lot of pressure to bring in systems very quickly without thinking about the ramifications.
Li: Whenever we use the model, we need to pair it with a guardrail model as a defense step. No matter how good or how bad it is, at least you need to get a filter so that we can offer some protection. And we need to continue red teaming [with ethical hackers to assess weaknesses] for these types of applications and models so that we can uncover new vulnerabilities over time.
SN: What are the cybersecurity risks of using AI?
Ji: When you’re introducing these models, there’s a process-based risk where you as an organization have less control, visibility and understanding of how data is being circulated by your own employees. If you don’t have a process in place that, for example, forbids people from using a commercial AI chatbot, you have no way of knowing if your workers are putting parts of your code base into a commercial model and asking for coding assistance. That data could potentially get exposed if the chatbot or the platform that they’re using has policies that say that they can ingest your input data for training purposes. So not being able to keep track of that creates a lot of risk and ambiguity.
Source link
