New dataset release: 🌐FineWiki This is an updated and better extracted version of Wikipedia, covering 325+ languages. Unlike the old dataset from 2023, we kept all the math content, tables, properly rendered templates, and extracted key facts. Examples and highlights below.

Hey twitter! I'm releasing the LLM Evaluation Guidebook v2! Updated, nicer to read, interactive graphics, etc! https://t.co/xG4VQOj2wN After this, I'm off: I'm taking a sabbatical to go hike with my dogs :D (back @huggingface in Dec *2026*) See you all next year!

Remi is super knowledgeable and did great things at 🤗 HF, can't wait to see what he'll cool at UMA!

We have just released 📄FinePDFs-Edu, a version of FinePDFs filtered with the FineWeb-Edu approach using ModernBERT and mmBERT. 350B+ tokens of top tier mid-training data in multiple languages. You can also download the classifiers (all 69 of them!)

After ~4 years building SOTA models & datasets, we're sharing everything we learned in ⚡The Smol Training Playbook We cover the full LLM cycle: designing ablations, choosing an architecture, curating data, post-training, and building solid infrastructure. We'll help you

The dataset itself: https://t.co/8VwFzul4fo

We made a small space where you can explore the dataset: https://t.co/Cot9f4BLDV

Filtering Low-resource language Wikipedias have a large prevalence of content from other languages, particularly English. We apply language‑ and script‑aware checks to each wiki. This can be useful if you want to train a language classifier on Wikipedia, for example.

Infoboxes While not keeping infoboxes in the main text content, we extracted them into key-value formats in the metadata so that they can be used as an info summary for each article. You can use these for quick facts/question answering.

Content filtering Sections filled with non-article unnatural content (e.g., “References”, “Notes”, “External links”, localized per language) are excluded. Visual/navigation boilerplate (ToC, navboxes, messageboxes, authority control) are also removed.

Tables A lot of Wikipedia content is locked inside tables. We took extra care to make sure we could extract tables into well formatted markdown. While they may not help on natural language understanding benchmarks, these contain most of the relevant facts in many articles.

Math We carefully extracted math and latex content. Previously, you could train a model on the Wikipedia page of some famous equation and the equation would not actually be on the page. A metadata flag `has_math` also makes it easy to filter for math content.

Templates Wikipedia markdown has a lot of "function calls" (templates) to render specific pieces of text. {{s-|XV}} -> "XVe siècle" (French for 15th century) Most pipelines drop them. We processed HTML dumps directly, where these are are already rendered.

There's a lot of hype around OCR models going around, but how do you actually use them to make a useful dataset? Today, we're releasing our codebase and additional data for 📄FinePDFs!

Did I hear correctly that @gui_penedo announced that we will have free HF merch at our COLM poster session 🤔

We're in 🇨🇦 for @COLM_conf Come talk to me and @HKydlicek on Wednesday at the FineWeb2 poster session (Session 3, Poster #58) @LoubnaBenAllal1 will be on Session 5, Poster #23 (SmolLM2)

Incredible to see FineWeb2 used in this amazing model that will power many many many use cases

The FinePDFs pipeline really shows the massive scale needed for LLM data processing: Raw data: 1.35B PDFs (1.2 PB) → $50k/mo storage Extraction: > rolmOCR: 368M docs → 250k GPUh, $750k > docling: 918M docs → 2.4M vCPUh, $35k Total liberation cost (incl. ablations): ~$1M

We are releasing 📄 FinePDFs: the largest PDF dataset spanning over half a billion documents! - Long context: Documents are 2x longer than web text - 3T tokens from high-demand domains like legal and science. - Heavily improves over SoTA when mixed with FW-EDU&DCLM web copora.