How ‘adversarial’ attacks reveal machine learning’s weakness Posted on : Nov 05 - 2019

The use of computer vision technologies to boost machine learning continues to accelerate, driven by optimism that classifying huge volumes of images will unleash all sorts of new applications and forms of autonomy.

But there’s a darker side to this transformation: These learning systems remain remarkably easy to fool using so-called “adversarial attacks.” Even worse is that leading researchers acknowledge they don’t really have a solution for stopping mischief makers from wreaking havoc on these systems.

“Can we defend against these attacks?” said Nicolas Papernot, a research scientist at Google Brain, the company’s deep learning artificial intelligence research team. “Unfortunately, the answer is no.”

Papernot, who is also an assistant professor at the University of Toronto, was speaking recently in Paris at the annual France is AI conference hosted by France Digitale. He was followed later in the morning by Jamal Atif, a professor at the Université Paris-Dauphine, who also addressed the growing threat of adversarial attacks to disrupt machine learning.

At its most basic, an adversarial attack refers to the notion of introducing some kind of element into a machine learning model designed specifically to incorrectly identify something.

On the left, the machine learning model sees the picture of the panda and correctly identifies it with a moderately high degree of confidence. In the middle, someone has overlaid this pixelated image that is not necessarily visible to the human eye into the panda image. The result is that the computer now is almost certain that it is a gibbon.

The simplicity of this deception highlights a couple of weakness. First, image recognition for machine learning, while it may have greatly advanced, still remains rudimentary. Papernot noted that to “teach” machines to recognize various images of cats and dogs, one needs to keep the parameters and the images fairly basic, introducing quite a bit of bias into the sample set.

Unfortunately, that makes the jobs of hackers much easier. Papernot pointed out that to disrupt these systems, which are often using publicly available images to learn, one doesn’t need to hack into the actual machine learning system. An external party can detect that such a system in searching for such images to learn, and from there it’s fairly easy to reverse-engineer the questions it’s asking and the parameters it has set.

“You can choose the question the model is asking, and you find a way to make the model make the wrong prediction,” he said. “You don’t even need to have internal access. You can send the input, and see what prediction it’s making, and extract the model. You can use that process to replicate the process locally.” View More