On Numerical Methods for Efficient Deep Neural Networks PDF Download

Author: Chong Li
Publisher:
ISBN:
Category :
Languages : en
Pages : 80

Get Book

Book Description
The advent of deep neural networks has revolutionized a number of areas in machine learning, including image recognition, speech recognition, and natural language processing. Deep neural networks have demonstrated massive generalization power, with which domain-specific knowledge in certain machine learning tasks has become less crucial. However, the impressive generalization power of deep neural networks comes at the cost of highly complex models that are computationally expensive to evaluate and cumbersome to store in memory. The computation cost of training and evaluating neural networks is a major issue in practice. On edge devices such as cell phones and IoT devices, the hardware capability, as well as battery capacity, are quite limited. Deploying neural network applications on edge devices could easily lead to high latency and fast battery drainage. The storage size of a trained neural network is a concern on edge devices as well. Some state-of-the-art neural network models have hundreds of millions of parameters. Even storing such models on edge devices can be problematic. Although we can transfer the input to the neural network to a server and evaluate the neural network on the server-side, the computation cost of network evaluation directly relates to the financial cost of operating the server clusters. More importantly, many neural network applications, such as e-Commerce recommender systems, has stringent delay constraint. Overall speaking, the computation cost network evaluation directly impacts the bottom lines of companies deploying neural network applications. It is highly desirable to reduce the model size and computation cost of evaluating the neural network without degrading the performance of the network. The neural network uses a combination of simple linear operations (such as fully connected layer and convolutional layer) and non-linearities (such as ReLU function) to synthesis elaborated feature extractors. While such automatic feature engineering is among the major driving forces of the recent neural network renaissance, it also contributes to the high computation cost of neural networks. In other words, since we are synthesizing highly complex non-linear functions using very simple building blocks, it is inevitable that a large number of such simple building blocks have to be used for the network to be sufficiently expressive. What if we directly incorporate well-studied classical methods that are known to be helpful for feature extraction in the neural network? Such high-level operations could directly reflect the intent of the network designers so the network does not have to use a large number of simple building blocks. For the network to be end-to-end trainable, we will need to be able to compute the gradient of the operation that we incorporate into the network. The differentiability of the operation could be a limiting factor, since the gradient of operation may not exist, or difficult to compute. We shall demonstrate that incorporating carefully designed feature extractors in the neural network is indeed highly effective. Moreover, if the gradient is difficult to compute, an approximation of the gradient can be used in place of the true gradient without negatively impact the training of the neural network. In this dissertation, we explore applying well-studied numerical methods in the context of deep neural networks for computationally efficient network architectures. In Chapter 2, we present COBLA---Constrained Optimization Based Low-rank Approximation---a systematic method of finding an optimal low-rank approximation of a trained convolutional neural network, subject to constraints in the number of multiply-accumulate (MAC) operations and the memory footprint. COBLA optimally allocates the constrained computation resources into each layer of the approximated network. The singular value decomposition of the network weight is computed, then a binary masking variable is introduced to denote whether a particular singular value and the corresponding singular vectors are used in low-rank approximation. With this formulation, the number of the MAC operations and the memory footprint are represented as linear constraints in terms of the binary masking variables. The resulted 0-1 integer programming problem is approximately solved by sequential quadratic programming. COBLA does not introduce any hyperparameter. We empirically demonstrate that COBLA outperforms prior art using the SqueezeNet and VGG-16 architecture on the ImageNet dataset. Chapter 3 focuses on neural network based recommender systems, a vibrant research area with important industrial applications. Recommender systems on E-Commerce platforms track users' online behaviors and recommend relevant items according to each user's interests and needs. Bipartite graphs that capture both user/item features and user-item interactions have been demonstrated to be highly effective for this purpose. Recently, graph neural network (GNN) has been successfully applied in the representation of bipartite graphs in industrial recommender systems. Response time is a key consideration in the design and implementation of an industrial recommender system. Providing individualized recommendations on a dynamic platform with billions of users within tens of milliseconds is extremely challenging. In Chapter 2, we make a key observation that the users of an online E-Commerce platform can be naturally clustered into a set of communities. We propose to cluster the users into a set of communities and make recommendations based on the information of the users in the community collectively. More specifically, embeddings are assigned to the communities and the user information is decomposed into two parts, each of which captures the community-level generalizations and individualized preferences respectively. The community structure can be considered as an enhancement to the GNN methods that are inherently flat and do not learn hierarchical representations of graphs. The performance of the proposed algorithm is demonstrated on a public dataset and a world-leading E-Commerce company dataset. In Chapter 4, we propose a novel method to estimate the parameters of a collection of Hidden Markov Models (HMM), each of which corresponds to a set of known features. The observation sequence of an individual HMM is noisy and/or insufficient, making parameter estimation solely based on its corresponding observation sequence a challenging problem. The key idea is to combine the classical Expectation-Maximization (EM) algorithm with a neural network, while these two are jointly trained in an end-to-end fashion, mapping the HMM features to its parameters and effectively fusing the information across different HMMs. In order to address the numerical difficulty in computing the gradient of the EM iteration, simultaneous perturbation stochastic approximation (SPSA) is employed to estimate the gradient. We also provide a rigorous proof that the estimated gradient due to SPSA converges to the true gradient almost surely. The efficacy of the proposed method is demonstrated on synthetic data as well as a real-world e-Commerce dataset.