#Tokenization is undeniably a key player in the success story of #LLMs but we poorly understand why. I want to highlight progress we made in understanding the role of tokenization, developing the core incidents and mitigating its problems. 🧵👇.

Check out the full paper here: . 🎓 Work by Jingtong Su, @KempeLab, @NYUDataScience, @AIatMeta.

Plus, we generate importance maps showing where in the transformer the concept is encoded — providing interpretable insights into model internals.

SAMI: Diminishes or amplifies these modules to control the concept's influence. With SAMI, we can scale the importance of these modules — either amplifying or suppressing specific concepts.

SAMD: Finds the attention heads most correlated with a concept. Using SAMD, we find that only a few attention heads are crucial for a wide range of concepts—confirming the sparse, modular nature of knowledge in transformers.

How would you make an LLM "forget" the concept of dog — or any other arbitrary concept? 🐶❓. We introduce SAMD & SAMI — a novel, concept-agnostic approach to identify and manipulate attention modules in transformers.

@EfroniYonathan.

Aligned Multi-Objective Optimization (A-🐮) has been accepted at #ICML2025! 🎉 .We explore optimization scenarios where objectives align rather than conflict, introducing new scalable algorithms with theoretical guarantees. #MachineLearning #AIResearch #Optimization #MLCommunity

RT @buutphan: Our work got accepted to #ICLR2025 @iclr_conf! Learn more about tokenization bias and how to convert your tokenized LLM to by….

work led by @buutphan and w/ @brandondamos @itai_gat @HavasiMarton @mattmucklm.

🎉Our paper just got accepted to #ICLR2025! 🎉. Byte-level LLMs without training and guaranteed performance? Curious how? Dive into our work! 📚✨ . Paper: Github:

RT @brandondamos: 📢 My team at Meta is hiring visiting PhD students from CMU, UW, Berkeley, and NYU! We study core ML, optimization, amorti….

RT @hall__melissa: Excited to release EvalGIM, an easy-to-use evaluation library for generative image models. EvalGIM ("EvalGym") unifies….

Thursday is busy:.9-11am I will be at the Meta AI Booth.12.30-2pm.Mission Impossible: A Statistical Perspective on Jailbreaking LLMs (.OR.End-To-End Causal Effect Estimation from Unstructured Natural Language Data (.

RT @KempeLab: For those into jailbreaking LLMs: our poster "Mission Impossible" today shows the fundamental limits of LLM alignment - and i….

Starting with Fei-Fei Li’s talk 2.30, after that I will mostly be meeting people and wonder the poster sessions.

Folks, I am posting my NeurIPS schedule daily in hopes to see folks, thanks @tkipf the idea ;). 11-12.30 WiML round tables.1.30-4 Beyond Decoding, Tutorial.

RT @NYUDataScience: Researchers at CDS and @AIatMeta prove vulnerabilities in AI language models are unavoidable, but introduce E-RLHF, a m….

What do you think do we need to sharpen our understanding of tokenization? Or will we soon be rid of it by developing models such as "MegaByte" by @liliyu_lili et al (@lukezettlemoyer)? . And add more paper to the threat!

@buutphan et al, found a method to mitigate some of the tokenization problems @karpathy mentioned by projecting tokens into byte space. The key to their method is to develop a map between statistically equivalent token and byte-level models.

In "The Foundations of Tokenization:.Statistical and Computational Concerns", Gastaldi et al. try to make first steps towards defining what a tokenizer should be and define properties it ought to have.