Webinar Series: Mathematical Foundations of Data Science

Statistical Optimization Methods for Machine Learning

Abstract

Statistics and optimization are two of the foundational pillars of machine learning, and it is well recognized that optimization provides powerful tools for statistics. In recent years, techniques of the complementary nature, i.e., using statistics to improve optimization algorithms, have become increasingly successful. In this talk, we present several recent works on such statistical optimization methods. First, we discuss how to use statistical hypothesis testing to automatically adjust the learning rate of stochastic gradient methods in training deep neural networks. Then we overview some recent advances in structured nonconvex optimization using stochastic variance reduction techniques. Finally we will briefly explain how to leverage statistical preconditioning to accelerate distributed optimization. 

Bio

Lin Xiao is a senior principal researcher at Microsoft Research located in Redmond, Washington, US. He received his PhD from Stanford University, and was a postdoctoral fellow at California Institute of Technology before joining Microsoft. He currently serves as an associate editor for the SIAM Journal on Optimization, and has served as area chairs for several machine learning conferences including NeurIPS and ICML. He was a winner of the Young Researcher Competition at the first International Conference on Continuous Optimization in 2004 for his work on fastest mixing Markov chains, and won the Test of Time Award at NeurIPS 2019 for his work on the regularized dual averaging method for sparse stochastic optimization. His current research interests include theory and algorithms for large-scale optimization and machine learning, reinforcement learning, and parallel and distributed computing.