Author: Louis SharrockPublisher:
ISBN:
Category :
Languages : en
Pages : 0
Book Description
On the Theory and Applications of Stochastic Gradient Descent in Continuous Time
Author: Louis SharrockStochastic Gradient Descent in Continuous Time
Author: Justin SirignanoStochastic Gradient Descent for Modern Machine Learning
Author: Rahul KidambiPublisher:
ISBN:
Category :
Languages : en
Pages : 242
Book Description
Tremendous advances in large scale machine learning and deep learning have been powered by the seemingly simple and lightweight stochastic gradient method. Variants of the stochastic gradient method (based on iterate averaging) are known to be asymptotically optimal (in terms of predictive performance). This thesis examines non-asymptotic issues surrounding the use of stochastic gradient descent (SGD) in practice with an aim to achieve its asymptotically optimal statistical properties. Focusing on the stochastic approximation problem of least squares regression, this thesis considers: 1. Understanding the benefits of tail-averaged SGD, and understanding how SGD's non-asymptotic behavior is influenced when faced with mis-specified problem instances. 2. Understand the parallelization properties of SGD, with a specific focus on mini-batching, model averaging and batch size doubling. Can this characterization shed light on algorithmic regimes (for e.g. largest instance dependent batch sizes) that admit linear parallelization speedups over vanilla SGD (with a batch size 1), thus presenting useful prescriptions that make best use of our hardware resources whilst not being wasteful of computation? As a byproduct of these results, can we understand how the learning rate behaves as a function of the batch size? 3. Similar to how momentum/acceleration schemes such as heavy ball momentum, or Nesterov's acceleration improve over standard batch gradient descent, can we formalize improvements achieved by accelerated methods when working with sampled stochastic gradients? Is there an algorithm that achieves this improvement over SGD? How does deterministic accelerated schemes such as heavy ball momentum, or say, Nesterov's acceleration work when used with sampled stochastic gradients? 4. This thesis considers the behavior of the final iterate of SGD (as opposed to a majority of efforts in the stochastic approximation literature which focus on iterate averaging) with varying stepsize schemes, including the standard polynomially decaying stepsizes and the practically preferred step decay scheme, with an aim to achieve minimax rates. The overarching goal of this section is to understand the behavior of SGD's final iterate owing to its widespread use in practical implementations for machine learning applications. Alongside the theory results that focus on the least squares regression, this thesis examines the general applicability of various results (in a qualitative sense) towards the problem of training multi-layer deep neural networks on benchmark datasets, and presents several useful implications when training deep learning models of practical interest.
Understanding Machine Learning
Author: Shai Shalev-ShwartzPublisher: Cambridge University Press
ISBN: 1107057132
Category : Computers
Languages : en
Pages : 415
Book Description
Introduces machine learning and its algorithmic paradigms, explaining the principles behind automated learning approaches and the considerations underlying their usage.
Theory and Applications of Time Series Analysis
Author: Olga ValenzuelaPublisher: Springer Nature
ISBN: 303140209X
Category : Mathematics
Languages : en
Pages : 236
Book Description
This book presents the latest developments in the theory and applications of time series analysis and forecasting. Comprising a selection of refereed papers, it is divided into several parts that address modern theoretical aspects of time series analysis, forecasting and prediction, with applications to various disciplines, including econometrics and energy research. The broad range of topics discussed, including matters of particular relevance for sustainable development, will give readers a modern perspective on the subject. The included contributions were originally presented at the 8th International Conference on Time Series and Forecasting, ITISE 2022, held in Gran Canaria, Spain, June 27-30, 2022. The ITISE conference series provides a forum for scientists, engineers, educators and students to discuss the latest advances and implementations in the foundations, theory, models and applications of time series analysis and forecasting. It focuses on interdisciplinary research encompassing computer science, mathematics, statistics and econometrics.
Machine Learning Refined
Author: Jeremy WattPublisher: Cambridge University Press
ISBN: 1108480721
Category : Computers
Languages : en
Pages : 597
Book Description
An intuitive approach to machine learning covering key concepts, real-world applications, and practical Python coding exercises.
Optimization for Machine Learning
Author: Suvrit SraPublisher: MIT Press
ISBN: 026201646X
Category : Computers
Languages : en
Pages : 509
Book Description
An up-to-date account of the interplay between optimization and machine learning, accessible to students and researchers in both communities. The interplay between optimization and machine learning is one of the most important developments in modern computational science. Optimization formulations and methods are proving to be vital in designing algorithms to extract essential knowledge from huge volumes of data. Machine learning, however, is not simply a consumer of optimization technology but a rapidly evolving field that is itself generating new optimization ideas. This book captures the state of the art of the interaction between optimization and machine learning in a way that is accessible to researchers in both fields. Optimization approaches have enjoyed prominence in machine learning because of their wide applicability and attractive theoretical properties. The increasing complexity, size, and variety of today's machine learning models call for the reassessment of existing assumptions. This book starts the process of reassessment. It describes the resurgence in novel contexts of established frameworks such as first-order methods, stochastic approximations, convex relaxations, interior-point methods, and proximal methods. It also devotes attention to newer themes such as regularized optimization, robust optimization, gradient and subgradient methods, splitting techniques, and second-order methods. Many of these techniques draw inspiration from other fields, including operations research, theoretical computer science, and subfields of optimization. The book will enrich the ongoing cross-fertilization between the machine learning community and these other fields, and within the broader optimization community.
Real and Stochastic Analysis
Author: M. M. RaoPublisher: Springer Science & Business Media
ISBN: 1461220548
Category : Mathematics
Languages : en
Pages : 411
Book Description
As in the case of the two previous volumes published in 1986 and 1997, the purpose of this monograph is to focus the interplay between real (functional) analysis and stochastic analysis show their mutual benefits and advance the subjects. The presentation of each article, given as a chapter, is in a research-expository style covering the respective topics in depth. In fact, most of the details are included so that each work is essentially self contained and thus will be of use both for advanced graduate students and other researchers interested in the areas considered. Moreover, numerous new problems for future research are suggested in each chapter. The presented articles contain a substantial number of new results as well as unified and simplified accounts of previously known ones. A large part of the material cov ered is on stochastic differential equations on various structures, together with some applications. Although Brownian motion plays a key role, (semi-) martingale theory is important for a considerable extent. Moreover, noncommutative analysis and probabil ity have a prominent role in some chapters, with new ideas and results. A more detailed outline of each of the articles appears in the introduction and outline to assist readers in selecting and starting their work. All chapters have been reviewed.
Software Engineering and Management: Theory and Application
Author: Roger LeePublisher: Springer Nature
ISBN: 3031551745
Category :
Languages : en
Pages : 238
Book Description
Adaptive Algorithms and Stochastic Approximations
Author: Albert BenvenistePublisher: Springer Science & Business Media
ISBN: 3642758940
Category : Mathematics
Languages : en
Pages : 373
Book Description
Adaptive systems are widely encountered in many applications ranging through adaptive filtering and more generally adaptive signal processing, systems identification and adaptive control, to pattern recognition and machine intelligence: adaptation is now recognised as keystone of "intelligence" within computerised systems. These diverse areas echo the classes of models which conveniently describe each corresponding system. Thus although there can hardly be a "general theory of adaptive systems" encompassing both the modelling task and the design of the adaptation procedure, nevertheless, these diverse issues have a major common component: namely the use of adaptive algorithms, also known as stochastic approximations in the mathematical statistics literature, that is to say the adaptation procedure (once all modelling problems have been resolved). The juxtaposition of these two expressions in the title reflects the ambition of the authors to produce a reference work, both for engineers who use these adaptive algorithms and for probabilists or statisticians who would like to study stochastic approximations in terms of problems arising from real applications. Hence the book is organised in two parts, the first one user-oriented, and the second providing the mathematical foundations to support the practice described in the first part. The book covers the topcis of convergence, convergence rate, permanent adaptation and tracking, change detection, and is illustrated by various realistic applications originating from these areas of applications.