Many LLMs appear to be widely overparameterized, with no apparent gain in performance. Why is this ? Results on Falcon suggest why. It appears that many text datasets have too many duplicates, effectively lowering the size. This causes large alphas and lower quality.

A new paper by Nagel challenges the “virtue of complexity” in return prediction. Prior work claimed that overparameterized models trained on tiny samples deliver strong out-of-sample performance. Nagel shows they essentially boil down to volatility-timed momentum. The paper

Great news from #UAI that one of our papers got selected for an oral presentation! TL;DR: we analyzed overfitting in overparameterized, heteroskedastic regression models and found a phase transition between two distinct types of overfitting! 🧵

Statistician: You may never cross the interpolation threshold. Your models may never be overparameterized. Deep learner: Hold my beer!

#111論文等共有 1211 [NeurIPS'19 Oral] Overparameterized two-layer NN の教師生徒モデルの理論解析。 activation の違いで違う解に行く事を示した。まず入力次元大で SGD は微分方程式に従うことを証明し、生徒が教師より大きい時 ①1層目だけ訓練すると生徒大で汎化悪化

It's time to reveal the answer to the poll! The correct answer is D. Surprised? I was too. With merely 30 data points in 1D, shouldn't an overparameterized neural network converge exponentially to an interpolating solution? A quick🧵with my explanation.

This is a surprisingly under-discussed paper from a few years ago. Big DNNs are literally memory machines–more than just remembering the data, but with the ability to recall the data as well. If you simply construct an overparameterized autoencoder and train it as the identity

1/3 New paper: We show that leveraging solution space structure via reduction mappings can accelerate convergence in gradient-based optimization. This has practical implications in settings such as low-rank projection, matrix factorisation, and overparameterized neural networks.

"Differentiable All-pass Filters for Phase Response Estimation and Automatic Signal Alignment". Improves the quality of parallel processing with learnable allpass filters optimized via an overparameterized network. abs: web:

#111論文等共有 1096 [ICML'23] Physics-informed NN (PINN) のlossは入力の微分があり難しい。overparameterized PINNの大域解収束を勾配流で証明。勾配法は適当な学習率では大域解収束すると証明。線形2次PDEを含む広い方程式、LeCunUniform等を含む広い初期化で証明。

A little more about Double Descent. Today most AI researchers recognize that in direct contrast to the predictions of Statistical Learning Theory (SLT), AI models can perform very well despite being widely overparameterized (N>P). That is, no one cares anymore about the

Our paper "Symbolic feature engineering of overparameterized Eulerian hyperelasticity models for fast inference time" has been accepted by CMAME. Paper: Data and code: 1/10

#111論文等共有 1079 [ICLR'23] Benign overfitting (BO)はoverparameterizedでは起こるがmildly-overparameterizedだとlabel noiseがあると起きない事を証明。その際はearly stoppingが有効。ResNet on CIFARではBOが起きImageNetでは起きない実験結果とconsistent。

Congratulations to Dr. Mingchen Li on her PhD defense! 🎓 Her research advances the field of AI by guiding overparameterized models to robust generalization using regularization, bilevel optimization, and principled architecture design. #PhDDefense #MachineLearning #ECE #goblue

'DRM Revisited: A Complete Error Analysis', by Yuling Jiao, Ruoxuan Li, Peiying Wu, Jerry Zhijian Yang, Pingwen Zhang. . #overparameterization #overparameterized #deep.

𝐖𝐡𝐞𝐫𝐞 𝐝𝐨𝐞𝐬 𝐃𝐨𝐮𝐛𝐥𝐞 𝐃𝐞𝐬𝐜𝐞𝐧𝐭 𝐜𝐨𝐦𝐞 𝐟𝐫𝐨𝐦 ? 🤷‍♂️ It was actually discovered in the theoretical physics literature in 1989. Looking at a very simple model for a NN--running Linear Regression (LR) on a widely overparameterized binary classification problem.

Here's today's talk on Double Descent from the weightwatcher community discord. WeightWatcher and Double Descent (DD). Or, "Understanding OverParameterized Models." What can we learn about this mysterious, misunderstood phenomenon in from the perspective of theoretical physics,

What are the most important statistical ideas of the past 50 years?.Counterfactual causal inference, overparameterized models and regularization, Bayesian multilevel models, adaptive decision analysis, and more—and how they relate to big data.

The Interpolating Information Criterion for Overparameterized Models. The problem of model selection is considered for the setting of interpolating estimators. Classical information criteria. 🧵 👇

models overparameterized to the universe's generating program itself keeping themselves occupied by looking at light going through dynamic 14 dimensional hyperbolic fractal glass