Thank you to everyone who joined us yesterday at the @Databricks Networking Party at Mahony’s Tavern during #ICML2025! 🎉 Attendees connected with fellow conference-goers and the Databricks Research and Engineering team over great conversation, delicious appetizers and drinks,

I'm at ICML 🇨🇦 and I'm hiring at @databricks. Visit our booth if you're interested. My scientific focus: It's 1972 in AI, there's an AI crisis, Dijkstra isn't here to save us, and maybe RL can. Why Databricks? The long road to AGI is being paved here and we have the real evals 🧵

We’re proud to be a platinum sponsor of #MLSys2025 alongside our co-founder & CTO @matei_zaharia serving as general chair. Stop by our booth to check out the latest projects from the Databricks team and RSVP for our networking event here: https://t.co/w4Gs61vaBJ. See you

We’re proud to be a platinum sponsor of #MLSys2025 alongside our co-founder & CTO @matei_zaharia serving as general chair. Stop by our booth to check out the latest projects from the Databricks team and RSVP for our networking event here: https://t.co/w4Gs61vaBJ. See you

We're kicking off 2025 with another Compound AI System Meetup with @lancedb in Mountain View on Jan 22! 🎉 Join us for a deep dive into AI infrastructure and insights with Lu Qiu, Allison Wang, Holden Karau, and Dr. Sharon Zhou. 🔗Save your spot:

We find that relying on academic benchmarks may be insufficient, and evaluation is best done with sophisticated approaches to domain expertise. S/O to the authors! @herengoneagn, @ericajiyuen, @KartikSreeni, @andyzhang0, @sam_havens, @matei_zaharia, @mcarbin, and @jefrankle

3) Developers should choose models based on specific needs. There is no single best model or paradigm. From open-source options to retrieval strategies, different solutions excel in different scenarios. (5/n)

2) There is room for improvement in core capabilities. Some enterprise needs like structured data extraction show clear paths for improvement, while more complex domain-specific tasks require more sophisticated reasoning capabilities. (4/n)

1) Models’ rankings across academic benchmarks do not necessarily map to their rankings across industry tasks. We find discrepancies in performance between academic and enterprise rankings, emphasizing the need for domain-specific testing. (3/n)

We developed the Domain Intelligence Benchmark Suite (DIBS) to help @databricks customers build better AI systems for their use cases. DIBS measures performance on datasets curated to reflect specialized domain knowledge and use cases for enterprises. Our key takeaways? (2/n)

New blog post on Benchmarking Domain Intelligence: https://t.co/QMS6gTxVG4 Evaluating your #AI solutions should be done with tests that match your actual use case. We observed that the tasks in many academic AI benchmarks don't match what business needs. (1/n)

We're back from Vancouver and #neurips2024—thanks to all the #genai researchers, practitioners and international pop stars who joined us at our @databricks Mosaic AI social event!

Thank you to Brickster/part-time model @mvpatel2000

Last day of the #neurips2024 expo! Come by the @databricks booth for free toques, t-shirts, and locally-sourced, free-range bricks!

Come for the bricks, stay for the @databricks DSPy demo. #NeurIPS2024

@dylan522p we have a worthy challenger!!!!!!! @jefrankle x @dylan522p "The Thrilla on Chinchilla" Settling the Great Scaling Debate once and for all see you at @latentspacepod live! https://t.co/FxCMmGlsWN

If you're at NeurIPS, stop by and say hi!

#TaylorSwift may have wrapped up the Eras Tour but we’re still in our Data and AI era! Stop by our booth at #NeurIPS2024 to chat all things research and meet #Brickster Swifties. For more information on our accepted workshops, see our blog post here. https://t.co/wAs5vZuCJ7

The results provide solid guidance on how to enhance a small LLM’s general knowledge performance to that of a larger model. Read on for more details, and if you like this post, please follow the authors!

Finally, we consider how the performance gains from continued pre-training scale with training FLOPS, a measure of the amount of compute used to train the model.