Almost 5 years in the making... "Hyperparameter Optimization in Machine Learning" is finally out! 📘
We designed this monograph to be self-contained, covering: Grid, Random & Quasi-random search, Bayesian & Multi-fidelity optimization, Gradient-based methods, Meta-learning.
arxiv.org/abs/2410.22854
🚨 OpenReview might have leaked names, but it won't leak the best hyperparameters, unfortunately! 😅
Tired of the drama? Solve your HPO problems before the ICML deadline with this new monograph by our own Luca Franceschi & Massimiliano Pontil (& colleagues).
arxiv.org/abs/2410.22854
He will also present an entropy-respecting forward–backward learning scheme that mitigates the inherent ill-posedness of stochastic learning problems.
Join us for what promises to be a very insightful session!
In this talk, Arthur Bizzi will introduce Neural Kolmogorov Equations, a deterministic and parallelizable framework for learning continuous-time stochastic processes using Forward and Backward Kolmogorov Equations.
Abstract:
Learning differential equations becomes substantially more challenging in the presence of stochasticity, as Neural SDEs typically require expensive, sequential integration during training.
📢 Upcoming Talk at Our Lab
We’re excited to host Arthur Bizzi from EPFL for a research talk next week!
Title: Towards Neural Kolmogorov Equations: Parallelizable SDE Learning with Neural PDEs
🗓 Date: November 19
⏰ Time: 16:00 CET
📍 Galileo Sala, CHT @iitalk.bsky.social
Excited to share our group’s latest work at #AISTATS2025! 🎓
Tackling concentration in dependent data settings with empirical Bernstein bounds for Hilbert space-valued processes.
📍Catch the poster tomorrow!
🔁 See the original tweet for details!
DeltaProduct is here! Achieve better state tracing through highly parallel execution. Explore more!🚀
P11] (submitted to The Journal of Chemical Physics)
chemrxiv.org/engage/chemr...
Kooplearn library:
kooplearn.readthedocs.io/latest/
For the longer version of the thread, you can take a look at this blog post:
vladi-iit.github.io/posts/2024-1...
Publications:
[P1] NeurIPS 2022
arxiv.org/abs/2205.14027
[P2] NeurIPS2023
arxiv.org/abs/2302.02004
[P3] ICML2024
arxiv.org/abs/2312.13426
[P4] NeurIPS2023
arxiv.org/abs/2306.04520
[P5] ICLR 2024
arxiv.org/abs/2307.09912
[P6] NeurIPS2024
arxiv.org/abs/2405.12940
14/ Looking ahead, we’re excited to tackle new challenges:
• Learning from partial observations
• Modeling non-time-homogeneous dynamics
• Expanding applications in neuroscience, genetics, and climate modeling
Stay tuned for groundbreaking updates from our team! 🌍
🙏 Collaborations with the Dynamic Legged Systems group led by Claudio Semini and the Atomistic Simulations group led by Michele Parrinello enriched our research, resulting in impactful works like [P9, P10] and [P7, P11].
12/ This journey wouldn’t have been possible without the inspiring collaborations that shaped our work.
🌟 Special thanks to Karim Lounici from École Polytechnique, whose insights were a major driving force behind many projects.
Predicting the quantiles for opening/closing of the Chignolin protein in the next simulation step
11/ One of our most exciting results:
[P8] NeurIPS 2024 proposed Neural Conditional Probability (NCP) to efficiently learn conditional distributions. It simplifies uncertainty quantification and guarantees accuracy for nonlinear, high-dimensional data.
10/ [P7] NeurIPS 2024 developed methods to discover slow dynamical modes in systems like molecular simulations. This is transformative for studying rare events and costly data acquisition scenarios in atomistic systems.
9/ Addressing continuous dynamics:
[P6] NeurIPS 2024 introduced a physics-informed framework for learning Infinitesimal Generators (IG) of stochastic systems, ensuring robust spectral estimation.
8/ 🌟 Representation learning takes center stage in:
[P5] ICLR 2024
We combined neural networks with operator theory via Deep Projection Networks (DPNets). This approach enhances robustness, scalability, and interpretability for dynamical systems.
Free energy surface of Chignolin protein folding
7/ 📈 Scaling up:
[P4] NeurIPS 2023 introduced a Nyström sketching-based method to reduce computational costs from cubic to almost linear without sacrificing accuracy. Validated on massive datasets like molecular dynamics, see figure.
Effects of metric distortion in learning eigenvalues (left) and stabilization of forecasting (right) for Ornstein-Uhlenbeck process
6/ [P3] ICML 2024 addressed a critical issue in TO-based modeling: reliable long-term predictions.
Our Deflate-Learn-Inflate (DLI) paradigm ensures uniform error bounds, even for infinite time horizons. This method stabilized predictions in real-world tasks; see the figure.
5/ [P2] NeurIPS 2023 advanced TOs with theoretical guarantees for spectral decomposition—previously lacking finite sample guarantees. We developed sharp learning rates, enabling accurate, reliable models for long-term system behavior.
Koopman Operator Regression Pipeline
4/ 🔑 The journey began with:
[P1] NeurIPS 2022
We introduced the first ML formulation for learning TO, which led to the development of the open-source Kooplearn library. This step laid the groundwork for exploring the theoretical limits of operator learning from finite data.
3/TOs describe system evolution over finite time intervals, while IGs capture instantaneous rates of change. Their spectral decomposition is key for identifying dominant modes and understanding long-term behavior in complex or stochastic systems.
2/ 🌐 Our work revolves around Markov/Transfer Operators (TO) and their Infinitesimal Generators (IG)—tools that allow us to model complex dynamical systems by understanding their evolution in higher-dimensional spaces. Here’s why this matters.
1/ 🚀 Over the past two years, our team, CSML, at IIT, has made significant strides in the data-driven modeling of dynamical systems. Curious about how we use advanced operator-based techniques to tackle real-world challenges? Let’s dive in! 🧵👇
An inspiring dive into understanding dynamical processes through 'The Operator Way.' A fascinating approach made accessible for everyone—check it out! 👇👀
Excited to present
"Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues"
at the M3L workshop at #NeurIPS
https://buff.ly/3BlcD4y
If interested, you can attend the presentation the 14th at 15:00, pass at the afternoon poster session, or DM me to discuss :)
In his book “The Nature of Statistical Learning” V. Vapnik wrote:
“When solving a given problem, try to avoid a more general problem as an intermediate step”
Join us at our posters and talks to connect, share ideas, and explore collaborations. 🚀✨
🔬 Fine-tuning Foundation Models for Molecular Dynamics: A Data-Efficient Approach with Random Features
✍️ @pienovelli.bsky.social, L. Bonati, P. Buigues, G. Meanti, L. Rosasco, M. Pontil | 📅ML4PS Workshop, Dec 15.
🔗 Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues
✍️ R. Grazzi, J. Siems, J. Franke, A. Zela, F. Hutter, M. Pontil
📃https://arxiv.org/abs/2411.12537 | 📅 Oral @ M3L workshop, Dec 14, 15:00 - 15:15.