📣 The OSCAR Project and @DFKI are happy to announce the release of Colossal OSCAR 1.0 📚, which is now available on the @huggingface Hub 🤗 at Colossal OSCAR 1.0 was put together by @pjox13 as part of the @OpenGPTX collaboration.

RT @CommonCrawl:

Check out the new multimodal OSCAR by @FuteralMatthieu ! 🚀📚🖼️.

RT @FuteralMatthieu: Announcing mOSCAR, multilingual interleaved text-image corpus as part of @oscarnlp project. Paper: .

👀 We're working on many new features for you, currently we're focusing on improving language identification, so if you want to help or contribute, please join our community 💬 on Discord:

✨ Colossal OSCAR 1.0 has also been made possible thanks to the continuous support of @Inria, @inria_paris, @InriaParisNLP and @CommonCrawl. Specially thanks to the contributions of @Uinelj, @imrua__ ,@sobamchan, @sebnagel and @bensagot.

As Colossal OSCAR 1.0 is based on @CommonCrawl, our annotations are distributed under CC0 (Creative Commons Zero) license, however for the textual content, users agree to the Common Crawl Terms of use 📄.👉

Colossal OSCAR 1.0 is just a partial annotation of the WET files of 10 @CommonCrawl snapshots, the original data is included only for convenience, and specially for researchers looking for data in lower resource languages. 🗣️.

Colossal OSCAR 1.0 is our largest release so far, being almost 10 times as big as previous releases. We're still working on statistics and documentation so please bear with us while we finish these for you in the coming days and weeks. 🤓🧑‍🔬📊.

RT @translation_eu: Everybody is talking about @OpenAI - we should talk more about cool projects like @silo_AI, @oscarnlp (for multilingual….

👀 We're working on many new features for you, currently we're focusing on improving language identification, so if you want to help or contribute, please join our community 💬 on Discord:

OSCAR 23.01 has been made possible thanks to @Uinelj @pjox13 @imrua__ @sobamchan @sebnagel and @bensagot.

🎉 OSCAR 23.01 is for now only available for researchers and academics, but will be available later on 🤗HuggingFace. To access the data, please follow the steps from our documentation:

📄 We also now have a more in-depth, technical documentation available that we will update with tutorials, how-tos, corpus documentation and info about the whole project.

🚨Other changes include metadata naming changes, language naming changes to better respect the BCP47 standard, and a compression change: OSCAR is now compressed using zstandard rather than gzip.

😮 Perplexity scores of the KenLM models are pre-computed, but it is up to the user to set a threshold for selecting the documents. ⚠️ Please use with caution, and do not hesitate to send feedback Please refer to this pre-print for more information: 📝.

👀 KenLM-based Adult Content Filtering. On a select (73) group of languages, computed using a KenLM model trained on harmful content. While being still experimental, this shows promising results in English.

📍Precomputed document-level Locality Sensitive Hashes! This will make both near and exact deduplication easier for you 😁.

💬 OSCAR 23.01 is also the first version ever to introduce a language-specific feature: A new blocklist specifically made for Japanese 🇯🇵. With the help of our community, we hope this will be the first of many language-specific features to come 🌐.

📚Categories! OSCAR 22.01 leveraged the UT1 Blocklists project to attempt to classify some adult content present in OSCAR. The OSCAR 23.01 iterates on this to include all of the categories provided by UT1: blogs, press, etc. Full list (in French) here: