Do you feel FL research is stuck with methods that do not work well in realistic scenarios? 🤔. 🫵We got you!.Introducing 🚀Generalized Heavy-Ball Momentum (GHBM)🚀, accepted at TMLR:.the FL algorithm with both SOTA theoretical guarantees and much better empirical results. 🧵1/9

Tagging relevant people in Optimization, FL and Distributed Training who might be interested in this work @KairouzPeter @gingsmith @konstmish @aaron_defazio @MatharyCharles @Ar_Douillard @niclane7 @sam_hrvth @_arohan_ @samsja19.

What began as a hunch became a path I’m proud of—this journey through FL & optimization has been rich with growth and purpose. Deep thanks to my PhD advisor @masone_carlo, and to Sai Praneeth Karimireddy & @mciccone_AI—your guidance lit the way. The best is yet to come.

📄 Project&Code: Working on GHBM has unveiled the full interplay between heterogeneous distributions and partial participation, expanding our understanding of momentum in distributed training. We have exciting ideas for the future, stay tuned!.

🚀 Results speak for themselves!. We consider both the common FL benchmarks and large-scale real-world scenarios with complex datasets (Google Landmarks and INaturalist). We always outperform the SOTAs, both in final results and communication/computational efficiency.

🤯 A small change, but massive improvement!. 🤨 Concerned about the additional hparam? Here’s the optimal recipe: just set τ as the inverse of participation ratio, and you’re done. GHBM’s heavy-ball form enables variants (LocalGHBM, FedHBM) without additional comm. w.r.t FedAvg

😎 But no despair—we found a simple solution!.GHBM uses the average of the last τ server pseudo-gradients, including contributions from clients not in the current round. This counters the joint effect of heterogeneity & partial participation, generalizing classical momentum (τ=1)

Momentum can be used to correct local optimization:.✅ provably effective in full participation.⛔️ experimentally loses its advantages under partial participation & heterogeneous clients​. ❓ Why so?.We show that classical momentum is biased towards the last selected clients

🤔 We have a lot of algorithms in our FL toolbox, why do we need another one?. Despite being strong in theory, experimentally most FL algorithms do not work very well under heterogeneous distributions and partial participation.

Federated Learning (FL) is a decentralized learning paradigm, particularly appealing for its privacy preserving nature: data never leaves the device📱!. However, this comes at a cost⚠️: local dataset are heterogeneous, and this affects optimization, e.g. the infamous client-drift

Unbelievable, forward to catch up with recent advances!! More than 9500 submissions overall, crazy numbers!.

What an experience! I.I will always be grateful to #ICVSS 2023 not only for all of that, but also for having met a lot of young researchers and PhD students, and new friends who already have a place in my heart. See you soon at the next conference!.

RT @PaoloCudrano: Great week at #ICVSS 2023!.

RT @bcaputo_iit: grazie all'amico e collega @demartin per questo bellissimo messaggio a tutto l'ateneo, in cui ci ricorda perche' pieni di….

Very proud of our paper presented at #montreal at #ICPR2022, my first experience as an author of a research paper in #federated_learning. Thanks to Professor @bcaputo_iit for the opportunity and to @debcaldarola and @marcuswallacej for the invaluable experience.