AI x Science Seminar: René Vidal

Learning dynamics in the feature-learning regime: implicit bias, robustness, and low-rank adaptation

René Vidal, PIK & Rachleff University Professor Electrical and Systems Engineering (SEAS), and Radiology (PSOM) University of Pennsylvania

Amy Gutman Hall
Room 414

Registration required for lunch

This talk will present new insights into the learning dynamics of gradient flow in the feature-learning regime. For separable data, we show that gradient flow learns a simple ReLU network with two neurons corresponding to the two classes. For data drawn from an orthogonal isotropic Gaussian mixture, we show that while trained ReLU networks are non-robust to adversarial perturbations, using normalized polynomial ReLU activations ensures both generalization and provable robustness without the need for adversarial training. We also analyze Low-Rank Adaptation (LoRA) for matrix factorization, showing that gradient flow converges to a neighborhood of the optimal solution, with an error that depends on the misalignment between pretraining and finetuning tasks. These results highlight how dynamics, data structure, architecture, and initialization jointly determine generalization, robustness, and adaptation.

Tuesday, October 7
Seminar: 12:00 – 1:00 pm
Speaker Lunch: 1:00 – 2:00 pm

Presented by: Penn AI, Innovation in Data Engineering and Science (IDEAS), and the Data Driven Discovery Initiative (DDDI)