@beyzaermis @mirco_ravanelli @CemSubakan @CgtyYldz For different model families and model sizes we observed similar trend.

@beyzaermis @mirco_ravanelli @CemSubakan @CgtyYldz We conducted our experiments with GPT2 model family, OLMo-1B and LLaMA-7b models. We found out that additional pretraining does not always help and you can get help from dataset similarity measures we introduce.

Check it out: https://t.co/Mpxo6m0Kh4 @beyzaermis @mirco_ravanelli @CemSubakan @CgtyYldz

Excited to share my first PhD paper! Why does additional pretraining sometimes hurt rather than help? In this paper we explore the additional adaptation characteristics of LLMs. We dive deep into the impact of pretraining data similarity and offer insights for future adaptations.