🔥Our paper PhyloLM got accepted at ICLR 2025 !🔥.In this work we show how easy it can be to infer relationship between LLMs by constructing trees and to predict their performances and behavior at a very low cost with @StePalminteri and @pyoudeyer ! Here is a brief recap ⬇️

RT @ClementRomac: I'm attending ICML 2025 this week in Vancouver where we're presenting our MAGELLAN paper along with @LorisGaven and @Cart….

RT @PourcelJulien: Introducing SOAR 🚀, a self-improving framework for prog synth that alternates between search and learning (accepted to #….

RT @pyoudeyer: New blog post !. What if LLM agents could learn by doing, not just by reading? 🤔. 2024 was the year of "agentic AI"—systems….

Curious about LLM interpretability and understanding ? We borrowed concepts from genetics to map language models, predict their capabilities, and even uncovered surprising insights about their training !. Come see my poster at #ICLR2025 3pm Hall 2B #505 !

If you are interested in this line of research of mapping LLMs you might also want to check the amazing work of @EliahuHorwitz and @momose123456789 10/10.

In short, PhyloLM is a cheap and versatile algorithm that generates useful representations for LLMs that can have creative applications in pratice. 9/10. paper : colab : code : ICLR : Saturday 3pm Poster #505.

A PhyloLM collaborative Huggingface space is available to try the algorithm and visualize maps : .The Model Submit button has been temporarily disabled for technical reasons but you can play with the data while we fix it ! 8/10.

By using code related contexts we can obtain a fairly different map. For example we notice that Qwen and GPT-3.5 have a very different way of coding compared to the other models which was not visible on the reasoning map. 7/10

The contexts choice is important as it reflects different capabilities of LLMs. Here on a general reasoning type of context we can plot a map of models using UMAP. The larger the edge, the closer models are from each other. Models on the same cluster are even closer ! 6/10

It can also measure quantization efficiency by observing the behavioral distance between LLM and quantized versions. In the Qwen 1concequantization could provide additional insights to quantizationould provide additional insights to quantization efficiency. 5/10 @elias_frantar

Aside from plotting trees, PhyloLM similarity matrix is very versatile. For example, running a logistic regression on the distance matrix makes it possible to predict performance of new models even from unseen families with good accuracy. Here is what we got on ARC. 4/10

Not taking into account these requirements can still produce relevant distance vizualisation trees. However it is important to remember they do not represent evolutionary trees. 3/10

Phylogenetic algorithms often require common ancestors to not appear in the objects studied but are clearly able to retrieve the evolution of the family. Here is an example in the richness of open-access model : @docsgptai @Teknium1 @maximelabonne @MistralAI @OpenChatDev 2/10

We build a distance matrix from comparing outputs of LLMs to a hundred of different contexts and build maps and trees from this distance matrix. Because PhyloLM only requires sampling very few tokens after a very short contexts the algorithm is particularly cheap to run. 1/10

RT @pyoudeyer: 🧠 One of the key limitation of LLMs today is their lack of metacognition: they were (mostly) not trained to know what they….

RT @pyoudeyer: Enabling forms of metacognition in LLMs is a frontiers challenge in #AI. We've made progress in this direction: .🧭MAGELLAN a….

RT @CartaThomas2: 🚀 Introducing 🧭MAGELLAN—our new metacognitive framework for LLM agents! It predicts its own learning progress (LP) in vas….

RT @ClementRomac: We just opened a new (engineering) internship position in the @FlowersINRIA team with @pyoudeyer:..

RT @pyoudeyer: 🚀 Exciting Internship Opportunities for AI and CogSci Students🌟. Join @FlowersINRIA and work on these cool topics:. 🔧 Curric….