Research: Bad actors could use a ‘bit flip’ attack to deceive AI systems, causing chaos

If an AI model ain’t broke…you might still need to fix it, or at least check to see if someone has been secretly accessing your model for malicious purposes.

A recent study from George Mason University found that attackers are able to compromise AI systems’ deep neural networks—which are multi-layered and mimic the human brain’s decision-making capabilities—by exploiting a “bit flip” attack to modify a model’s weights, or the numerical parameters that help the model determine its output

A bit flip attack is when a bad actor changes (or “flips”) just one bit (such as from 0 to 1) to secure a patch onto any image and fool a system. The researchers also suspect this technique could work for things like speech recognition. For example, an attacker targeting a fintech AI model could flip a bit so a message reads, “Transfer $10,000,” instead of “Transfer $10,” leading to an illicit payday and a lot of chaos for cybersecurity staffers.

Prior attacks in this area, according to a statement from George Mason University assistant professor Qiang Zeng, looked like attackers patching a single image (such as a stop sign) that an AI tool is attempting to read to something else in order to trick the system.

In the statement, the assistant professor said that selectively flipping just one bit lets an attacker place a patch onto all examples of a particular image (such as any stop sign). The system then interprets the image how the attacker intends.

Now watch this drive. For a self-driving car, constantly understanding visuals such as stop signs and lanes is integral to ensure the vehicle does not hurt a passenger or pedestrian. If an attacker flips a bit on the AI system within the car to see a stop sign as a speed limit sign instead, that can have huge consequences.

According to Zeng, AI consists of weights that are stored in bits. As a result, if a bit is flipped, the AI model will continue to perform as if nothing happened.

“For [an] AI system, what we find is that if you change just one bit, if you damage one of the weights…then you will have something very terrible happen,” Zeng said. “It’s particularly stealthy.”

The stealth of this operation makes its impact severe, according to Zeng—he offered the example of a bad actor attacking an identity recognition system who wants access to sensitive information and flips one bit to patch an image of the company’s CEO over a different image.

How do you solve a problem like this? Zeng said that those looking to defend themselves against these types of attacks should frequently check their systems so that they can detect intrusions. Additionally, professionals could maintain redundant systems (such as a copy of one system so that if an organization suspects an attacker infiltrated one, a backup exists) and compare the two to see if an attacker has compromised one of them.

Limitations, however, exist for these options as well.

ID Tech reported that additional approaches to defending an AI system include hardware safeguards such as error-correcting code memory, which is a type of random-access memory that can identify and fix single-bit memory flaws.

“When you check this weight, the attacker may be attacking another weight,” Zeng said. “The effectiveness really depends on how frequently you check.”

“If they are not identical, then you can report an attack, but of course there could be a false positive,” Zeng said. “Another obvious limitation [is that] you need to double the cost.”