A student reached out asking for advice on research directions in optimization, so I wrote a long response with pointers to interesting papers. I thought it'd be worth sharing it here too:. 1. Adaptive optimization. There has been a lot going on in the last year, below are some.

You may think of optimization and linear algebra as established fields, but consider this: until 2008, convergence rate of SGD applied to linear systems wasn't established. I'm sure there are many other discoveries waiting to be made, things we'd be surprised later weren't known.

It's not just my opinion.

A gentle reminder that TMLR is a great journal that allows you to submit your papers when they are ready rather than rushing to meet conference deadlines. The review process is fast, there are no artificial acceptance rates, and you have more space to present your ideas in the.

RT @yenhuan_li: @konstmish @chungentsai It is perhaps helpful to recall Nesterov’s opinion:

Modular duality:

AdamC:

One thing that feels strange about AdamW is that it treats all network parameters identically - norm layers, attention, and dense layers all get the same update rule. Classical optimization, in contrast, uses tricks such as mirror descent with tailored mirror map to significantly.

Several research groups have released papers on the convergence of Muon, mostly looking at either as a Frank-Wolfe method with momentum or a trust region procedure. I find this one to be particularly easy to read.

I believe successful neural network training represents cases of "near convexity": the optimization landscape, while technically non-convex, behaves enough like a convex problem that standard convex optimization is often applicable. At the same time, *in general* neural nets.

To solve variational inequality and nonlinear equations, the implicit update is used because the analogue of gradient descent may diverge even under monotonicity (analogue of convexity for operators). A good source on this is Malitsky & Tam:.5/5

Proximal operator is connected to implicit gradients. This is used in meta-learning, for instance see the iMAML method:.4/

Stochastic prox is also of interest in federated learning, where each worker minimizes a regularized objective. I think this paper by @akhaledv2 and @chijinML is the best source to learn more:.3/

Prox is fundamental for distributed optimization. It can be used to denote the constraint between multiple workers, which is known as "product trick". For instance, we used it this way in our ProxSkip paper (.2/

You'll often encounter the proximal operator in optimization papers. Prox is a fundamental tool that arises in distributed, federated, stochastic optimization, nonlinear equations., and meta learning. For a solid introduction, start with Parikh & Boyd:.1/

I find it particularly nice that we can reproduce the phenomenon using a simple *linear* problem. This immediately rules out nonconvexity as a leading factor.

There are several hypotheses for why Adam outperforms SGD on LLMs: heavy-tailed noise, blowing up curvature, near-constant magnitude of update, etc. The one I find most compelling is label imbalance: Adam specifically improves performance on rare classes, of which there are many.

RT @wazizian: ❓ How long does SGD take to reach the global minimum on non-convex functions?. With @FranckIutzeler, J. Malick, P. Mertikopou….