To Repeat or Not To Repeat: Insights from Scaling LLM under Token-Crisis
Studies what would happen if we train LLM with repeated data and how we can alleviate the LLM mult-epoch degradation.
2
19
128
Replies
@arankomatsuzaki
We need develop sophisticated methods to handle text repetition in LLMs. Tapping into new data sources could redefine the AI foundational model race.
A secondary use would be to solve the issue of *running out* of training data
0
1
3
@arankomatsuzaki
Token crisis 🤣🤣🤣, have researchers become social media influencers or what???
0
0
1