Videos

Landscape Analysis for Overcomplete Tensor and Neural Collapse

Presenter
May 18, 2021
Abstract
Qing Qu - University of Michigan, Center for Data Science In this talk, we provide the first global landscape analysis for overcomplete tensor decomposition and neural collapse. For both problems, we show that landscapes have similar benign global geometric structures. First, overcomplete tensor decomposition relates to many applications in representation learning, such as overcomplete dictionary learning and convolutional dictionary learning. Under tight frame assumptions of the overcomplete components, we show that the nonconvex loss over the sphere has no spurious local minimizers. Second, recent seminal work by Donoho et al. showed a prevalence phenomenon during the terminal phase of network training - neural collapse. By studying the optimization landscape of the training loss under a unconstrained feature model, we provide the theoretical justification for this phenomenon, which could have broad implications for network training, design, and beyond.
Supplementary Materials