🇪🇺 One month after the AI Action Summit 2025 in Paris, I am thrilled to announce EuroBERT, a family of multilingual encoder exhibiting the strongest multilingual performance for task such as retrieval, classification and regression over 15 languages, mathematics and code. ⬇️ 1/6

EuroBERT is going to @COLM_conf 2025! Can’t wait to be in Montreal with @gisship and @DuarteMRAlves to see all the great research everyone’s bringing!

RT @bclavie: This is excellent science confirming the common wisdom that bidirectional attention is superior for encoder tasks, but that ML….

RT @ManuelFaysse: 🚨Should We Still Pretrain Encoders with Masked Language Modeling? We have recently seen massively trained causal decoders….

RT @gisship: 🚨 New paper drop: Should We Still Pretrain Encoders with Masked Language Modeling?.We revisit a foundational question in NLP:….

More in the paper, go check it out! You can also check out our blog post: Huge thanks to the dream team 💥.@gisship , @ManuelFaysse , @DuarteMRAlves , Emmanuel Malherbe, @andre_t_martins , Céline Hudelot, @PierreColombo6 🙌. 7/7 🧵.

Finally, we show that CLM is more data-efficient — it converges faster early in training — which can be extremely useful in low-resource domains and provides greater stability during downstream fine-tuning. 6/7 🧵

Motivated by strong hybrid training results, we explore a continuous pretraining (CPT) setup. MLM-adapted CLM models consistently outperform. Excitingly, starting from a SOTA decoder can yield even better encoders—unlocking the best of both worlds. 5/7 🧵

First, we study the effect of different pretraining strategies that combine causal language modeling (CLM) and masked language modeling (MLM) objectives. Starting with CLM-only and transitioning to MLM-only setups, consistently outperform pure MLM across downstream tasks. 4/7 🧵

We ran a controlled study with identical model sizes, data, and diverse tasks across two scenarios: pretraining from scratch & continued pretraining. With 30+ models and 15k finetuning runs, results show MLM alone isn’t optimal and starting with CLM boosts downstream perf.3/7 🧵

While encoders usually rely on MLM pretraining, recent work shows decoders trained with causal LM (CLM) can also work well as encoder models. We investigate if CLM’s benefits come from the objective itself or confounding factors like model size and training scale. 2/7 🧵.

🚨 Should you only pretrain encoder models with Masked Language Modeling (MLM)?. Spoiler: definitely not!. Let’s revisit a foundational NLP question: Is MLM still the best way to pretrain encoder models for text representations?. 📄: x @gisship. 1/7 🧵.

So nice to see new distillation libraries emerging around the ULD Loss and other cross-tokenizer distillation methods. Big kudos to @vitransformer and the whole DistillKitPlus team for their work.

• Optimized: Fused ops (Liger Kernel, Flash), distributed training (FSDP, DDP). • Data Processing: Tokenization, packing, subsampling. • Highly Customisable and more. For any large-scale continuous pre-training feel free to contact me or @DuarteMRAlves for extensive support 🤗.

Why EuroBERT training might interest you:.• Resumable: Continue training with different hardware & environments. • Hardware Agnostic: Runs on CPU, @AMD, or @nvidia GPUs. • Hugging Face Friendly: Model, tokenizer.

The EuroBERT training library is live! 🚀.Additionally, as weekends are perfect for experimentation, we’ve released a tutorial on continuous pre-training to add languages to EuroBERT. 🎓Tutorial: 🔨Github:

RT @antoine_chaffin: PyLate 1.1.7 - The big batches update - is out! 🚀.As contrastive learning remains the go-to for large-scale pre-traini….

@Engomar_10 *congratulations: @Engomar_10.

Great to see the community using EuroBERT! As hoped, it’s proving to be an excellent foundation model, especially for information retrieval tasks across multiple languages after just one epoch of finetuning. Check it out: .@Engomar_10