1/Pretraining is hitting a data wall; scaling raw web data alone leads to diminishing returns. Today @datologyai shares BeyondWeb, our synthetic data approach & all the learnings from scaling it to trillions of tokens🧑🏼‍🍳.- 3B LLMs beat 8B models🚀.- Pareto frontier for performance

RT @amrokamal1997: After months of development, we finally share with the world some hard-earned science behind synthetic data. @datologyai….

RT @RicardoMonti9: its a pleasure to show up to work everyday and learn from synthetic data gurus @pratyushmaini , @VineethDorna and team -….

Finalizing the magic seed for synthetic data generation while the data dawg @RicardoMonti9 showers his blessings in the background.

RT @leavittron:

RT @sarahcat21: For years, researchers have known that synthetic data is valuable; but not all synthetic data is created equally. Generatin….

RT @j_mcgraph: @pratyushmaini and @datologyai make synthetic data seem easy, but it's really just how good they are.

RT @acrognali: Great work by the @datologyai team, enjoyed reading this.

RT @RishabhAdiga01: Thrilled to see BeyondWeb launched 🚀 Phenomenal insights and a huge step forward for scaling high-quality synthetic dat….

RT @sjoshi804: As we hit the limits of real web-scale data, @datologyai's synthetic data shows how we can leverage the models we've already….

RT @leavittron: two paths for synthetic data

RT @soldni: OLMo 2 is SOTA web rewriter??

RT @VineethDorna: Big day for the @datologyai team! We introduce BeyondWeb, scaling synthetic data for trillion-scale pretraining! .✨ Colle….

RT @leavittron: Very excited to announce BeyondWeb, @datologyAI’s synthetic pretraining data generation paradigm. BeyondWeb is a rephrasing….

RT @leavittron: The era of "The Era of Pretraining is Over" is over.

RT @code_star: Its here! Checkout our blog and paper on scaling synthetic data 1/n

RT @HaoliYin: Launch day! Take a look at the phenomenal insights into what production-grade synthetic data looks like - driven by @pratyush….

RT @arimorcos: Today, we introduce BeyondWeb, our synthetic data generation approach which significantly outperforms all open synthetic dat….

15/Needless to say, such a massive undertaking could not have been accomplished without a stellar engineering team that helped us scale our work to trillions of tokens. If you are excited about this, join us

14/ Dive into the full post for recipes, benchmarks, and other cool experiments. Thanks to the @datologyai team, especially @leavittron and @VineethDorna, for their massive contributions that helped shape this up. Arxiv: Blog:

13/ Implications: @datologyai is here to democratize high-quality synthetic data for all. We show a pathway for generating synthetic data cheaply at scale. We're part of curating 7T tokens for @arcee_ai 's AFM4.5B, which already shows our real-world wins!