Thank you to everyone who joined us yesterday at the @Databricks Networking Party at Mahony’s Tavern during #ICML2025! 🎉. Attendees connected with fellow conference-goers and the Databricks Research and Engineering team over great conversation, delicious appetizers and drinks,

RT @jefrankle: I'm at ICML 🇨🇦 and I'm hiring at @databricks. Visit our booth if you're interested. My scientific focus: It's 1972 in AI, th….

RT @DbrxMosaicAI: We’re proud to be a platinum sponsor of #MLSys2025 alongside our co-founder & CTO @matei_zaharia serving as general chai….

We’re proud to be a platinum sponsor of #MLSys2025 alongside our co-founder & CTO @matei_zaharia serving as general chair. Stop by our booth to check out the latest projects from the Databricks team and RSVP for our networking event here: See you.

We're kicking off 2025 with another Compound AI System Meetup with @lancedb in Mountain View on Jan 22! 🎉 Join us for a deep dive into AI infrastructure and insights with Lu Qiu, Allison Wang, Holden Karau, and Dr. Sharon Zhou. 🔗Save your spot:

We find that relying on academic benchmarks may be insufficient, and evaluation is best done with sophisticated approaches to domain expertise. S/O to the authors!.@herengoneagn, @ericajiyuen, @KartikSreeni, @andyzhang0, @sam_havens, @matei_zaharia, @mcarbin, and @jefrankle.

3) Developers should choose models based on specific needs. There is no single best model or paradigm. From open-source options to retrieval strategies, different solutions excel in different scenarios. (5/n)

2) There is room for improvement in core capabilities. Some enterprise needs like structured data extraction show clear paths for improvement, while more complex domain-specific tasks require more sophisticated reasoning capabilities. (4/n)

1) Models’ rankings across academic benchmarks do not necessarily map to their rankings across industry tasks. We find discrepancies in performance between academic and enterprise rankings, emphasizing the need for domain-specific testing. (3/n)

We developed the Domain Intelligence Benchmark Suite (DIBS) to help @databricks customers build better AI systems for their use cases. DIBS measures performance on datasets curated to reflect specialized domain knowledge and use cases for enterprises. Our key takeaways? (2/n).

New blog post on Benchmarking Domain Intelligence: Evaluating your #AI solutions should be done with tests that match your actual use case. We observed that the tasks in many academic AI benchmarks don't match what business needs. (1/n).

We're back from Vancouver and #neurips2024—thanks to all the #genai researchers, practitioners and international pop stars who joined us at our @databricks Mosaic AI social event!

Thank you to Brickster/part-time model @mvpatel2000.

Last day of the #neurips2024 expo! Come by the @databricks booth for free toques, t-shirts, and locally-sourced, free-range bricks!

Come for the bricks, stay for the @databricks DSPy demo. #NeurIPS2024

RT @swyx: @dylan522p we have a worthy challenger!!!!!!!. @jefrankle x @dylan522p ."The Thrilla on Chinchilla". Settling the Great Scaling….

RT @NaveenGRao: If you're at NeurIPS, stop by and say hi!.

#TaylorSwift may have wrapped up the Eras Tour but we’re still in our Data and AI era! Stop by our booth at #NeurIPS2024 to chat all things research and meet #Brickster Swifties. For more information on our accepted workshops, see our blog post here.

The results provide solid guidance on how to enhance a small LLM’s general knowledge performance to that of a larger model. Read on for more details, and if you like this post, please follow the authors!.

Finally, we consider how the performance gains from continued pre-training scale with training FLOPS, a measure of the amount of compute used to train the model.