Stochastic gradient descent (SGD) forms the core optimization method for deep neural networks, contributing to their resurgence. While some theoretical progress has been made, it remains unclear why SGD leads the learning dynamics in overparameterized networks to solutions that generalize well. Here we show that for overparameterized networks with a degenerate valley in their loss landscape, SGD on average decreases the trace of the Hessian of the loss. We also show that isotropic noise in the non-degenerate subspace of the Hessian decreases its determinant. In addition to explaining SGDs role in sculpting the Hessian spectrum, this opens the door to new optimization approaches that guides the model to solutions with better generalization. We test our results with experiments on toy models and deep neural networks.

Precise scientific analysis in collider-based particle physics is possible because of complex simulations that connect fundamental theories to observable quantities. These simulations have been paired with multivariate methods for many years in search of the smallest distance scales in nature. Deep learning tools hold great promise to qualitatively change this paradigm by allowing for holistic analysis of data in its natural hyperdimensionality with thousands or millions of features instead of up to tens of features. These tools are not yet broadly used for all areas of data analysis because of the traditional dependence on simulations. In this talk, I will discuss how we can change this paradigm in order to exploit the new features of deep learning to explore nature at sub-nuclear distance scales. In particular, I will show how neural networks can be used to (1) overcome the challenge of intractable hypvervariate probability density modeling and (2) learn directly from (unlabeled) data to perform hypothesis tests that go beyond any existing analysis methods. The talk will end with a brief discussion of challenges for hypervariate deep learning analysis. While my examples will be from particle physics, it is likely that these tools have a much broader applicability across fundamental physics and beyond. I will keep the particle physics jargon minimal in order to facilitate discussions about connections to your area of science!

The lecture will review some of the seminal contributions of James Clerk Maxwell, William John Macquorn Rankine and George Biddell Airy to the theory of structures and how those theories can be applied to modern structural engineering design. William F. Baker is a consulting structural engineering partner at Skidmore, Owings & Merrill LLP where he has led the structural engineering practice for more than 20 years. Bill is best known for the development of the “buttressed core” structural system for the Burj Khalifa, the world’s tallest manmade structure. In addition to his work on supertall buildings, Bill’s expertise also extends to long-span roof structures and specialty structures. He has also collaborated with numerous artists, including Jamie Carpenter, Iñigo Manglano-Ovalle, James Turrell, and Jaume Plensa. Bill is an Honorary Professor at the University of Cambridge; he has received honorary doctorates from the University of Stuttgart, Heriot-Watt University, the Illinois Institute of Technology and the University of Missouri; the Gold Medal from the Institution of Structural Engineers (IStructE), the American Society of Civil Engineers (ASCE) Lifetime Award for Design; the Gustav Magnel Gold Medal from the University of Ghent; the Fazlur Rahman Khan Medal from the Council on Tall Buildings and Urban Habitat; and the Fritz Leonhardt Preis (Germany). He is a Fellow of both the ASCE and the IStructE, and a member of the National Academy of Engineering (USA) and an International Fellow of the Royal Academy of Engineering (United Kingdom). Bill is currently collaborating with faculty members from MIT, Cambridge, ETH/Zurich, and EPFL/Lausanne on a book intended to make Maxwell’s structural engineering work accessible to the modern engineer.