Computations in Science Seminars
Feb 2023
8
Wed 12:15
Mary Silber, University of Chicago
Host: Stephanie Palmer ()
Organizer: Daniel Seara ()
Chasing and Channeling the Water: Self-Organized Vegetation in Drylands

A beautiful example of spontaneous pattern formation occurs in certain dryland environments around the globe. Stripes of vegetation alternate with stripes of bare soil, with striking regularity and on a scale readily monitored via satellites. Positive feedbacks, between infiltration of water into the soil and the vegetation itself, help concentrate this essential, but limited, resource into the vegetated zones. These feedbacks play out on the short timescales of the rare and unpredictable rainstorms that sustain life in these dry regions. In contrast, the vegetation may change very little over decades, aside from a gradual upslope colonization. In this talk I will tell you a little bit about the empirical data, the models, their analysis and the results of numerical simulations, focusing on the basic question of what sets the spacing of the vegetation bands. Our work suggests some new questions about how these fragile ecosystems might respond to changes in precipitation characteristics, such as storm strength, frequency, and seasonality, all of which are occurring as a consequence of climate change.

Feb 2023
15
Wed 12:15
Luca Mazzucato, University of Oregon
Host: Stephanie Palmer ()
Organizer: Kabir Husain ()
Neural mechanisms of optimal performance

When we attend a demanding task, our performance is poor at low arousal (drowsy) or high arousal (anxious), but we achieve optimal performance at intermediate arousal, yielding the celebrated Yerkes-Dodson inverted-U law. Despite decades of investigations, the neural mechanisms underlying this inverted-U law are unknown. In this talk, I will elucidate the behavioral and neural mechanisms linking arousal and performance. I will show that mice during auditory and visual decision-making express an array of discrete strategies, including optimal, suboptimal and disengaged, as revealed by an HMM analysis. The optimal strategy occurs at intermediate arousal levels, measured by pupil size, consistent with the inverted-U law. Using neuropixels recordings from neural populations in the auditory cortex, we show that sound encoding is optimal at intermediate arousal level, suggesting that performance modulations occur as early as primary sensory cortex. To explain the computational principle underlying this inverted-U law, we show that in a recurrent network with E/I populations arousal induces a phase transition from a multi-attractor to a single attractor phase, and performance is optimized near the critical region. The model further predicts a monotonic decrease in neural variability induced by arousal, which we confirmed in the empirical data. Our results establish a biologically plausible theory of optimal performance based on phase transitions in attractor networks with E/I balance, whose implications for brain-inspired AI models will be briefly outlined.

Feb 2023
22
Wed 12:15
Ajay Gopinathan, University of California, Merced
Organizer: Daniel Seara ()
Mar 2023
1
Wed 12:15
Peter McMahon, Cornell University
Host: Arvind Murugan ()
Organizer: Yuqing Qiu ()
Computing with Physical Systems

With conventional digital computing technology reaching its limits, there has been a renaissance in analog computing across a wide range of physical substrates. In this talk, I will introduce the concept of Physical Neural Networks [1] and describe a method my group has developed to train any complex physical system to perform as a neural network for machine-learning tasks. We have tested our method experimentally with three different systems – one mechanical, one electronic, and one photonic – and have been able to show MNIST handwritten-digit classification using each of these systems, despite the fact that none of the systems were initially designed to do machine learning.

I will describe several possible future research directions on Physical Neural Networks, including the potential to create large-scale photonic accelerators for server-side machine learning [2], smart sensors that pre-process acoustic, microwave or optical [3] signals in their native domain before digitization, new kinds of quantum neural network that don't require a carefully engineered quantum computer, and generally the prospect to endow analog physical systems with new, unexpected functionality.

[1] L.G. Wright*, T. Onodera* et al. Nature 601, 549-555 (2022)

[2] T. Wang et al. Nature Communications 13, 123 (2022)

[3] T. Wang*, M. Sohoni* et al. arXiv:2207.14293, to appear in Nature Photonics (2023)

Mar 2023
15
Wed 12:15
OPEN
Mar 2023
22
Wed 12:15
OPEN
Mar 2023
29
Wed 12:15
Yuhai Tu, IBM TJ Watson Research
Host: Arvind Murugan ()
Organizer: Kabir Husain ()
Can physicists help understand Deep Learning?

Despite the great success of deep learning, it remains largely a black box. In this seminar, we will describe our recent work in understanding learning dynamics and generalization of deep neural networks based on concepts and tools from statistical physics.

SGD Learning dynamics: The main search engine in deep neural networks is the Stochastic Gradient Descent (SGD) algorithm. However, despite its effectiveness, little is known about how SGD finds ``good" solutions (low generalization error) in the high-dimensional weight space. By studying weight fluctuations in SGD, we find a robust inverse relation between the weight variance in SGD and the landscape flatness, which is the opposite to the fluctuation-dissipation(response) relation in equilibrium statistical physics. We show that the noise strength in SGD depends inversely on the landscape flatness, which explains the inverse variance-flatness relation. Our study suggests that SGD serves as an ``intelligent" annealing strategy where the effective temperature self-adjusts according to the loss landscape, which allows it to find the flat minimum regions that contain generalizable solutions. Time permit, we discuss some application of these insights for solving machine learning problems [1,2].

Geometric determinants of generalization: We first report the discovery of an exact duality relation between changes in activities in a densely connected layer of neurons and the changes in their weights connecting to the next layer. The activity-weight duality leads to an explicit expression for the generalization loss, which can be decomposed into contributions from different directions in weight space. We find that the generalization loss from each direction is the product of two geometric factors (determinants): sharpness of the loss landscape at the solution and the standard deviation of the dual weights, which scales as an activity-weighted norm of the solution. By using the generalization loss decomposition, we uncover how hyperparameters in SGD, different regularization schemes (e.g., weight decay and dropout), training data size, and labeling noise affect generalization by controlling one or both factors [3].

[1] “The inverse variance-flatness relation in Stochastic-Gradient-Descent is critical for finding flat minima”, Y. Feng and Y. Tu, PNAS, 118 (9), 2021.

[2] “Phases of learning dynamics in artificial neural networks: in the absence and presence of mislabeled data”, Y. Feng and Y. Tu, Machine Learning: Science and Technology (MLST), July 19, 2021. https://iopscience.iop.org/article/10.1088/2632-2153/abf5b9/pdf

[3] “The activity-weight duality in feed forward neural networks: The geometric determinants of generalization”, Y. Feng and Y. Tu, https://arxiv.org/abs/2203.10736

Apr 2023
5
Wed 12:15
OPEN
Apr 2023
12
Wed 12:15
OPEN
Apr 2023
19
Wed 12:15
OPEN
Apr 2023
26
Wed 12:15
OPEN
May 2023
3
Wed 12:15
OPEN
May 2023
10
Wed 12:15
William Bialek, Princeton University
Host: Stephanie Palmer ()
May 2023
17
Wed 12:15
Itai Cohen, Cornell University
Host: Stephanie Palmer (), Sid Nagel ()
May 2023
24
Wed 12:15
Benjamin Machta, Yale University
Organizer: Daniel Seara ()
May 2023
31
Wed 12:15
OPEN