Date & Time:
April 4, 2022 3:00 pm – 4:00 pm
Crerar 390, 5730 S. Ellis Ave., Chicago, IL,
04/04/2022 03:00 PM 04/04/2022 04:00 PM America/Chicago Hao Zhang (UC Berkeley) – Machine Learning Parallelization Could Be Automated, Performant, and Easy-to-use Crerar 390, 5730 S. Ellis Ave., Chicago, IL,

Watch Via Live Stream

As models and data grow bigger, ML parallelization is more essential than ever. However, the amount of engineering effort and domain knowledge required for scaling up ML is often underestimated. The marginal cost for developing specialized systems with hand-tuned parallel strategies is extremely high in the face of emerging models and heterogeneous cluster setups.

In this talk, I will present a better way to build better ML systems. I view ML system building as an optimation over a parallel strategy space, with the objective of maximizing the system “goodput”, conditioned on model and cluster configurations. I show that by formulating each piece in the optimization as math representations, we can make it solvable using existing tools. Unlike specialized systems, this formulation enables building generic ML compilers that automate ML parallelization, generalize to many models, and achieve strong performance, simultaneously. In particular, I’ll describe two compiler systems: Alpa and Cavs, which automate model parallelism on large-scale distributed clusters, and the batching of dynamic neural network computation on accelerators, respectively. My open-source artifacts have been used by organizations such as AI2, Meta, and Google, and parts of my research have been commercialized at multiple start-ups including Petuum and AnyScale.


Hao Zhang

Postdoctoral Researcher, UC Berkeley

Hao Zhang is a postdoc researcher at UC Berkeley working with Ion Stoica. He completed his Ph.D. at CMU where he worked with Eric Xing. His research interests are in the intersection of machine learning and systems, with the focus on improving the performance and ease-of-use of today’s distributed ML systems. Hao’s research has been recognized with an NVIDIA pioneer research award at NeurIPS’17, and the Jay Lepreau best paper award at OSDI’21.

Related News & Events

UChicago CS News

NeurIPS 2023 Award-winning paper by DSI Faculty Bo Li, DecodingTrust, provides a comprehensive framework for assessing trustworthiness of GPT models

Feb 01, 2024

“Machine Learning Foundations Accelerate Innovation and Promote Trustworthiness” by Rebecca Willett

Jan 26, 2024

Nightshade: Data Poisoning to Fight Generative AI with Ben Zhao

Jan 23, 2024
UChicago CS News

UChicago Undergrad Analyzes Machine Learning Models Used By CPD, Uncovers Lack of Transparency About Data Usage

Oct 31, 2023
In the News

In The News: U.N. Officials Urge Regulation of Artificial Intelligence

"Security Council members said they feared that a new technology might prove a major threat to world peace."
Jul 27, 2023
UChicago CS News

UChicago Computer Scientists Bring in Generative Neural Networks to Stop Real-Time Video From Lagging

Jun 29, 2023
UChicago CS News

UChicago Assistant Professor Raul Castro Fernandez Receives 2023 ACM SIGMOD Test-of-Time Award

Jun 27, 2023
Michael Franklin
UChicago CS News

Mike Franklin, Dan Nicolae Receive 2023 Arthur L. Kelly Faculty Prize

Jun 02, 2023
UChicago CS News

PhD Student Kevin Bryson Receives NSF Graduate Research Fellowship to Create Equitable Algorithmic Data Tools

Apr 14, 2023
UChicago CS News

Computer Science Displays Catch Attention at MSI’s Annual Robot Block Party

Apr 07, 2023
UChicago CS News

UChicago, Stanford Researchers Explore How Robots and Computers Can Help Strangers Have Meaningful In-Person Conversations

Mar 29, 2023
UChicago CS News

Postdoc Alum John Paparrizos Named ICDE Rising Star

Mar 15, 2023
arrow-down-largearrow-left-largearrow-right-large-greyarrow-right-large-yellowarrow-right-largearrow-right-smallbutton-arrowclosedocumentfacebookfacet-arrow-down-whitefacet-arrow-downPage 1CheckedCheckedicon-apple-t5backgroundLayer 1icon-google-t5icon-office365-t5icon-outlook-t5backgroundLayer 1icon-outlookcom-t5backgroundLayer 1icon-yahoo-t5backgroundLayer 1internal-yellowinternalintranetlinkedinlinkoutpauseplaypresentationsearch-bluesearchshareslider-arrow-nextslider-arrow-prevtwittervideoyoutube