New paper: On the unreasonable effectiveness of LLMs for causal inference. GPT4 achieves new SoTA on a wide range of causal tasks: graph discovery (97%, 13 pts gain), counterfactual reasoning (92%, 20 pts gain) & actual causality. How is this possible?🧵.

RT @AniketVashisht8: Extremely happy to have our work on Teaching Transformers Causal Reasoning through Axiomatic Training accepted at ICML….

RT @sirbayes: @amt_shrma Sounds very cool. Here is link to paper (hard to find since it seems to be a TMLR paper, not an official ICLR pape….

PywhyLLM: Creating an API for language models to interact with causal methods and vice versa. v0.1 out, welcome feedback. If you are at #iclr2025, come check out our poster today at 10am-12:30pm.

Podcast link:

What changes for causality research in the age of LLMs and what does not? Enjoyed this conversation with Alex Molak on how LLMs are accelerating causal discovery, how diverse environments can learn help causal agents, and how causality is critical for verifying AI actions. Link👇.

Job Alert: @MSFTResearch India is hiring postdocs! A chance to work with some amazing colleagues while doing world-class research. Apply here: DM me if interested in ML/reasoning/causality.

Excited to present Axiomatic Training at #NeurIPS2024, a new paradigm to teach causal reasoning to language models!.I try to summarize what LLM systems can do today and what new training paradigms we need to improve their causal reasoning. Slides:.

RT @CaLM_Workshop: We are happy 😁 to announce 📢 the First Workshop on Causality and Large Models (C♥️LM) at #NeurIPS2024 . 📜 Submission dea….

RT @IntuitMachine: Teaching Transformers Causal Reasoning through Axiomatic Training. The ability to reason causally is increasingly recogn….

RT @AniketVashisht8: Can we teach Transformers Causal Reasoning?. We propose Axiomatic Framework, a new paradigm for training LMs. Our 67M-….

RT @naga86: We take a closer look at prompt optimization problem & develop UniPrompt for learning LM prompt for a task from scratch. We arg….

PyWhy talk series: DAGitty. This Monday @JohannesTextor will be talking about the amazing dagitty tool and lessons learnt about using causal graphs in science. Monday, Feb 26, 8am PT/9:30pm IST. Join here:

@Lianhuiq @AleksanderMolak @kerstingAIML @devendratweetin @MatejZecevic @tom4everitt @sirbayes Update 2: @AndrewLampinen et al show that it is possible to learn _how_ to discover causal structure using only observational data! Using past experimental data, an agent can learn how to experiment & find new causal structure. h/t @AleksanderMolak

@Lianhuiq @AleksanderMolak @kerstingAIML @devendratweetin @MatejZecevic Update 1: There is now a theoretical proof for this hypothesis! Under some assumptions, Richens & @tom4everitt show that any agent that generalises to a large, diverse set of distribution shifts must have learnt a causal model of the data. h/t @sirbayes

Btw, if you are interested in these topics, @lianhuiq @AleksanderMolak @kerstingAIML @devendratweetin @matejzecevic and I are organizing a.workshop on LLMs and causality at AAAI next week, and hope to have a discussion. Schedule and papers here:

So rather than having to choose one, let generative models be creative (and hallucinate errors!), and combine with causal models/verifiers in the outer loop so that we can autocorrect those errors. This approach has worked for code, wonder if it can improve causal reasoning too?.

So, where does causality fit in such data-rich settings? Maybe we can think of causality as a verifier, not as a solution proposer. The generality that we get from simple training objectives and large datasets is great. And so is the rigor that we get from specific causal models.

Note: Such generalization is only possible where web-scale training data exists. For small-data problems prevalent in science, algorithms to learn causality are as important as before. See @wellingmax response to Bitter Lesson essay (both great reads, btw)

But if you are willing to observe the entire world and a large part of human reflections on those observations, chances are that the same confounding won't be present everywhere. And at this scale, causal-like generalization may be possible.

So from this perspective, I see the value of both @ylecun and @eliasbareinboim viewpoints. True that we cannot learn causal agents simply from observing the world, since observed data is usually confounded.