Optimization for Machine Learning

Algorithmically Faithful and System-Efficient Optimization for Large Language Models

-
The thesis develops an optimizer-aware systems view in which these methods are treated as structured computations with algorithmic invariants, memory behavior, communication objects, and scheduling constraints.

Understanding Practical Optimization in Machine Learning

-
This dissertation studies how practical optimization methods used to train modern machine learning systems can be better understood by analyzing the objectives, updates, assumptions, and approximations that arise in their actual implementation.

Better Methods and Theory for Federated Learning: Compression, Client Selection, and Heterogeneity

-

B5 L5 R5209

Federated learning (FL) is an emerging machine learning paradigm involving multiple clients, e.g., mobile phone devices, with an incentive to collaborate in solving a machine learning problem coordinated by a central server. FL was proposed in 2016 by Konecny et al. and McMahan et al. as a viable privacy-preserving alternative to traditional centralized machine learning since, by construction, the training data points are decentralized and never transferred by the clients to a central server. Therefore, to a certain degree, FL mitigates the privacy risks associated with centralized data collection. Unfortunately, optimization for FL faces several specific issues that centralized optimization usually does not need to handle. In this thesis, we identify several of these challenges and propose new methods and algorithms to address them, with the ultimate goal of enabling practical FL solutions supported with mathematically rigorous guarantees.

Konstantin Burlachenko

Ph.D. Student, Computer Science
Konstantin's research focuses on large-scale mathematical optimization systems for machine learning and artificial intelligence, especially those requiring high-performance computing approaches.