Date & Time:
November 15, 2019 10:30 am – 11:30 am
Location:
Crerar 390, 5730 S. Ellis Ave., Chicago, IL,
11/15/2019 10:30 AM 11/15/2019 11:30 AM America/Chicago Machine Learning Seminar Series: Greg Ongie (UChicago) & Blake Woodworth (TTIC) Crerar 390, 5730 S. Ellis Ave., Chicago, IL,

A Function Space View of Overparameterized Neural Networks (Ongie)

Contrary to classical bias/variance tradeoffs, deep learning practitioners have observed that vastly overparameterized neural networks with the capacity to fit virtually any labels nevertheless generalize well when trained on real data. One possible explanation of this phenomenon is that complexity control is being achieved by implicitly or explicitly controlling the magnitude of the weights of the network. This raises the question: What functions are well-approximated by neural networks whose weights are bounded in norm? In this talk, I will give some partial answers to this question. In particular, I will give a precise characterization of the space of functions realizable as a two-layer (i.e., one hidden layer) neural network with ReLU activations having an unbounded number of units, but where the Euclidean norm of the weights in the network remains bounded. Surprisingly, this characterization is naturally posed in terms of the Radon transform as used in computational imaging, and I will show how tools from Radon transform analysis yield novel insights about learning with two and three-layer ReLU networks.

The Complexity of Finding Stationary Points in Convex and Non-Convex Optimization (Woodworth)

Non-convex optimization algorithms typically guarantee convergence to approximate stationary points of the objective. However, the fundamental complexity of finding such points is poorly understood, even in the convex setting, and especially in comparison to our very thorough understanding of the complexity of finding points with near optimal function value in the convex case. In this talk, I will discuss two recent papers in which we tightly bound the stochastic first-order oracle complexity of finding an approximate stationary point, first for the convex case and then for the non-convex case. An important implication of our work is that, in a certain sense, plain SGD is an optimal algorithm for stochastic non-convex optimization.

Remote broadcast at TTIC, 6045 S Kenwood Avenue, 5th Floor, Room 526

Host: Nati Srebro and Rebecca Willett

Greg Ongie

Postdoctoral Scholar, Department of Statistics, University of Chicago

I am a postdoc affiliated with the Committee on Compuational and Applied Mathematics (CCAM) within the Department of Statistics at the University of Chicago, supported by Rebecca Willett.

Research interests:

Machine learning, optimization, and compressed sensing, with applications to image reconstruction in MRI, CT, and related inverse problems. My recent projects include:

  • Learning to solve linear inverse problems in imaging
  • Matrix completion with non-linear models
  • Online algorithms for dynamic MRI reconstruction
  • Continuous domain compressed sensing
  • Fast algorithms for large-scale structured low-rank matrix completion
  • Generalizations of total variation regularization
  • Non-convex optimization for large-scale sparsity regularized inverse problems

Blake Woodworth

PhD Student, Toyota Technological Institute at Chicago

I am a fourth-year PhD student in computer science at the Toyota Technological Institute at Chicago (TTIC) advised by Nati Srebro. My research is in machine learning and optimization, and I am interested in both the theoretical analysis of learning and optimization algorithms as well as practically efficient methods for learning from data.

Before coming to TTIC I studied at Yale University where I received a B.S. in computer science, advised by Dan Spielman. At Yale, my coursework was spread evenly across the computer science, mathematics, and statistics departments; I was also a peer tutor for several programming-intensive computer science courses.

From September 2017-July 2019 I was supported by a NSF Graduate Research Fellowship. From July 2019, I am supported by a Google PhD Fellowship in machine learning.

Related News & Events

Video

“Machine Learning Foundations Accelerate Innovation and Promote Trustworthiness” by Rebecca Willett

Jan 26, 2024
Video

Nightshade: Data Poisoning to Fight Generative AI with Ben Zhao

Jan 23, 2024

In The News: U.N. Officials Urge Regulation of Artificial Intelligence

"Security Council members said they feared that a new technology might prove a major threat to world peace."
Jul 27, 2023

UChicago Computer Scientists Bring in Generative Neural Networks to Stop Real-Time Video From Lagging

Jun 29, 2023

Computer Science Displays Catch Attention at MSI’s Annual Robot Block Party

Apr 07, 2023

UChicago, Stanford Researchers Explore How Robots and Computers Can Help Strangers Have Meaningful In-Person Conversations

Mar 29, 2023
Students posing at competition

UChicago Undergrad Team Places Second Overall In Regionals For World’s Largest Programming Competition

Mar 17, 2023

Postdoc Alum John Paparrizos Named ICDE Rising Star

Mar 15, 2023

New EAGER Grant to Asst. Prof. Eric Jonas Will Explore ML for Quantum Spectrometry

Mar 03, 2023

Assistant Professor Chenhao Tan Receives Sloan Research Fellowship

Feb 15, 2023

UChicago Scientists Develop New Tool to Protect Artists from AI Mimicry

Feb 13, 2023

Professors Rebecca Willett and Ben Zhao Discuss the Future of AI on Public Radio

Jan 26, 2023
arrow-down-largearrow-left-largearrow-right-large-greyarrow-right-large-yellowarrow-right-largearrow-right-smallbutton-arrowclosedocumentfacebookfacet-arrow-down-whitefacet-arrow-downPage 1CheckedCheckedicon-apple-t5backgroundLayer 1icon-google-t5icon-office365-t5icon-outlook-t5backgroundLayer 1icon-outlookcom-t5backgroundLayer 1icon-yahoo-t5backgroundLayer 1internal-yellowinternalintranetlinkedinlinkoutpauseplaypresentationsearch-bluesearchshareslider-arrow-nextslider-arrow-prevtwittervideoyoutube