By @pentagonalize, Lena Strobl, Dana Angluin, and me, on arXiv:

If position embeddings have poly(n) magnitude and the 1st- and 2nd-place attention weight are separated by a constant-size gap, then the required scale is O(log n). If the gap is 1/n^k, then the required scale is O(n^k log n).

It's known that any average-hard attention transformer can be simulated by a softmax-attention transformer by scaling attention logits. We give a new bound on how much they need to be scaled by, and this bound now works for any average-hard attention transformer.

We updated our paper on soft attention simulating hard attention with a more general result. Many theoretical constructions of transformers use hard attention, but what does that say about actual transformers, which use soft attention?

Andy Yang @pentagonalize drove the conceptualization, theory, and experiments of this work. I was just the checker and editor!.

RT @mhahn29: Very excited about this work: deep results from logic shedding light on Transformers and the benefit of depth.

Paper: Code:

Although there is a lot of wiggle room in defining rounding/precision, our theoretical predictions are confirmed by experiments surprisingly well!.

The separating languages are very simple: L_k is the language of k blocks of one or more repetitions of a symbol, e.g., L_3 contains strings aba, aabbbbaaaaaa, etc. More blocks require more depth.

Further, we show that deeper programs/formulas in C-RASP are strictly more expressive than shallower programs/formulas. Together, these results imply that in the above-defined variant, deeper transformers are strictly more expressive than shallower transformers.

C-RASP is a programmer-friendly version of "temporal logic with future-masked counting." We show both are exactly equivalent to soft-attention transformers with fixed precision outside attention but no rounding inside attention (to avoid under/overflow summing over sequence).

New on arXiv: Knee-Deep in C-RASP, by @pentagonalize, Michael Cadilhac and me. The solid stepped line is our theoretical prediction based on what problems C-RASP can solve, and the numbers/colors are what transformers (no position embedding) can learn.

RT @huangxt233: I'll be presenting our paper together with @mhahn29 on Saturday morning poster session. Feel free to reach out!.

RT @AnganaBorah2: Last week, I had a fantastic time presenting our work on belief congruence in LLMs at the Midwest Speech and Language Day….

RT @Ruyuan_Wan: @davidweichiang @aarsri21 @ND_CSE wooohoo! Congratulations Aarohi!!!.

(Out of the papers that Aarohi @aarsri21 has published while at @ND_CSE, 80% have received an award!).

In contrast, on text with variation involving new words or meanings (e.g., "lie" vs. "cap"), far more data is needed, but it leads to a massive breakthrough in performance.

On text with character-level variation (e.g., "strategy" vs. "strat"), out-of-the-box performance improves even with a few additional training examples -- but approaches a plateau, suggesting that more data is not the solution.

Congratulations to @aarsri21 on winning the Best Paper Award at W-NUT at NAACL 2025! This paper applies various interventions simulating noisy text or dialectal variation to discover how different interventions have different effects.