Secure computation for a typical training algorithm of a deep neural network - Enhancing AI with cryptography -

Background:

The accumulation, use, and application of cross-sector data in sophisticated data analysis, such as by artificial intelligence (AI), is expected to foster innovation and promote development and economic growth in diverse fields. However, the concern about security breaches and privacy risks has been emphasized, and this often becomes an obstacle for the usage of secret data.

NTT has researched and developed secure computation technology that can remove this barrier. Our secure computation technology enables data processing while keeping the data encrypted by secret sharing technology standardized by ISO/IEC.*4 By using this technology, the original data is never disclosed; therefore, secure computation technology allows the safe use of confidential data.

Here, we developed the first secure computation technology for a typical training algorithm of the deep neural network, a kind of AI algorithm used in practice. It encourages new applications by training of deep neural network with the secret data while maintaining data confidentiality. For example, a restaurant owner can optimize his stock and staff assignment in accordance with a model trained by using individual information, such as location and schedule, and public information, such as weather and surrounding events. We further expect that a doctor can rapidly and accurately diagnose malignancies in accordance with a model trained by X-ray radiography, CT scanning, MRI, and micrograph.

Technical Key Points:

Secure computation is a technology to compute a function on encrypted data; therefore, its procedure is quite different from that of a typical computer. Several functions are quick and easy to calculate for a regular machine but are time-consuming and difficult for secure computation. We refer to these functions as intractable functions. If you want to perform an analysis that includes those, a novel technique to handle that task is required.

In general, training of a deep neural network proceeds as follows. The system has several layers, and some functions are sequentially computed on training data layer-by-layer. Then, a tentative output, which is parameters of the layers continues to the convergence test. If the tentative output is not converged, the parameters are updated by an optimization method, and the above processes will be repeated. The softmax function is a common function computed in the last layer. Adam is an optimization method improved from the stochastic gradient descent (SGD), which tends to incur many repetitions. Both the softmax function and Adam consist of division, exponent, reciprocal, and square root.

For secure computation, those four operations were intractable functions. Therefore, most recent works have focused on the prediction, which is lightweight compared to the training. Even in recent literature handling the training, the complex functions are roughly approximated by different functions, and SGD is used that is an elemental optimization method.

NTT invented novel techniques to rapidly and accurately compute those intractable functions, division, exponent, reciprocal, and square root, on the encrypted data. By using these techniques, we can calculate the softmax function and use Adam as an optimization.

We have two approaches to compute those four functions on encrypted data. One is using secure mapping. We first prepare many pairs of input and output of an intractable function in the clear. On an encrypted input, we can obtain the corresponding (encrypted) output while keeping the input secret. The other is proposing novel techniques that separately compute division, exponent, reciprocal, and square root.
We will present the former approach in FIT2019 and the latter one in CCS2019.

By using these inventions, we can train a model with encrypted data by a capital training algorithm of the deep neural network. For example, in our microbenchmark with the latter approach, training with 60,000 examples of handwritten digit *5 costs about five minutes per epoch.*6