Can a LM that has only ever seen the word “cat” tokenized as ␣cat, understand the token sequence [␣, c, a, t]? In our NeurIPS spotlight ⭐, we show that the answer is surprisingly YES, and in fact, you can even modify the tokenization at inference-time for performance gains!🧵

Paper: https://t.co/zoblxa5SSa Code: https://t.co/otYHURqBf8 This work would not have been possible with the following amazing collaborators @alisawuffles @orevaahia @JonathanHayase @YejinChoinka @nlpnoah

Finally, we study the source of robustness and investigate the contributing factors to this phenomenon. See our paper for more.

We found that alternative tokenizations lead to better performance on several benchmarks that intuitively require character-level understanding, such as constructing acronyms or counting characters. This suggests that we can find more optimal tokenization entirely at inference.

We find that the finer the tokenization (and the farther away from canonical), the less performance is retained.

Yet, we find that LMs remain robust to random or character-level segmentations of input text when evaluated on diverse benchmarks. LMs maintained much of their original performance, with on average 93.4 performance retention on Qwen with random segmentation.

Modern tokenizers map text tokens deterministically, so LMs only see a single tokenization of any string during training. This means that there exists only one canonical token sequence to represent any particular string.

We found that alternative tokenizations lead to better performance on several benchmarks that intuitively require character-level understanding, such as constructing acronyms or counting characters. This suggests that we can find more optimal tokenization entirely at inference.

We find that the finer the tokenization (and the farther away from canonical), the less performance is retained..

Yet, we find that LMs remain robust to random or character-level segmentations of input text when evaluated on diverse benchmarks. LMs maintained much of their original performance, with on average 93.4 performance retention on Qwen with random segmentation.

Modern tokenizers map text tokens deterministically, so LMs only see a single tokenization of any string during training. This means that there exists only one canonical token sequence to represent any particular string.