📢New research🧵How can we estimate the quality of our models for AGI tasks?. 1/ My latest paper dives into this fundamental epistemic question. The core finding? Under current data collection practices, we *cannot* know a model's quality for key AI tasks 😬

📑 Get the full paper here: 👋 Make sure to say Hi at NeurIPS if you are around and want to discuss more.🙏 Many thanks to Leon Bottou, Tina Eliassi-Rad, @SmithaMilli, @kchonyc for valuable feedback!.

These results are very related to @ylecun's recent point on the challenge of long tails in advancing AI research. However, the paper looks at it from a validation perspective, not a learning perspective.

6/ To be clear: the paper is not saying that recent advances are not real. Far from it. However, realism is needed on what our validation methods can tell, especially to.- guarantee intended outcomes for deployed AI systems.- advance AI research in a scientifically sound way.

5/ What's next? we need new methods for evaluating AI on complex tasks. Participatory methods and open science could provide promising directions to tackle these challenges.

4/ The main takeaway: The current combination of tasks, validation method, and data collection cannot provide valid validation results—and either naive scaling (collecting more data in the same way) nor limited benchmarks can fix this problem.

3/ The key idea: By modeling current data collection practices via complex social systems, the paper establishes rigorous impossibility results. These results show that, in such settings, no risk estimator—including counterfactual ones—can work with high probability.

2/ Why does it matter? 🤔 The explosive progress in AI has relied heavily on the train-test paradigm. But, as the paper shows, this approach falls apart for various modern AI tasks, including.- Open domain reasoning and QA via LLMs.- Recommender systems in many shapes and forms.

State of the are video generation with.@AIatMeta Movie Gen 😎 Brought to you by Flow Matching 🪭Fantastic work by.@imisra_ and team!.

Make sure to stop by the excellent contributed talks as well as the poster session too!. Co-organized with Alessandro Lazaric, @ArpitAgarwalAI, @HodaHeidari, Nicolas Usunier, and @tinaeliassi . Workshop website:

#ICML workshop on Humans, Algorithmic Decision-Making & Society. Tomorrow, July 27 in Hall A2 with exciting speakers, contributed talks, and 65 accepted papers. Invited talks by @astoica73 , @fariba_k , @weidingerlaura , @manish_raghavan , @MilindTambe_AI , and Sarah Dean.

Excited to share our #icml2024 workshop on "Humans, Algorithmic-Decision Making, and Society"! See below for the amazing group of speakers and call for papers. @astoica73 @fariba_k @manish_raghavan @MilindTambe_AI @weidingerlaura @tinaeliassi @HodaHeidari.

RT @dayvidliu: I had a wonderful time attending and presenting @FAccTConference last week. To reflect on the experience and sum up what I l….

RT @gini_do: I'm very happy that I had the chance to collaborate with @dayvidliu @mnick (and twitterless Nico Usunier, as always!) on a #FA….

RT @lipmanya: 📣 A new #ICML2023 paper investigates the Kinetic Energy of Gaussian Probability Paths which are key in training diffusion/flo….

RT @RickyTQChen: Excited to share our new work on Riemannian Flow Matching. Unlike diffusion-based approaches, it’s. - completely simulati….

RT @FrancoisRozet: @lipmanya @RickyTQChen @helibenhamu @mnick @lematt1991 I wanted to check how Flow Matching-OT worked in practice and . ….

RT @lipmanya: **Flow Matching** (#ICLR2023 spotlight) offers a simple simulation-free method for training flow-based generative models, gen….

This is probably my favorite graph from the #ICLR2023 review release. It's a heavy-tailed distribution that nicely shows 1) our anxiety-level around 10/23 😅 2) the amazing push of ACs, SACs & (emergency) reviewers to submit everything in time thru the bump at the end ❤️

ICLR 2023 reviews are out! Incredible work by the entire community that went into this. 18500+ reviews and 99+% of submissions with 3+ reviews.