Stéphane d'Ascoli
@stephanedascoli
Followers
1,048
Following
184
Media
19
Statuses
96
Research Scientist @AIatMeta working on neural decoding. Prev: AI4science fellow @EPFL , PhD & physics @ENS_ULM , astro @NASA .
Paris, France
Joined November 2018
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
Bills
• 142117 Tweets
#WWERaw
• 109587 Tweets
Bengals
• 98344 Tweets
Washington
• 92360 Tweets
Commanders
• 82385 Tweets
sabrina
• 75035 Tweets
Jayden Daniels
• 69124 Tweets
連休明け
• 59408 Tweets
Terry
• 50333 Tweets
John Deere
• 49599 Tweets
Josh Allen
• 44656 Tweets
Jaguars
• 39121 Tweets
津波注意報
• 36779 Tweets
Jey Uso
• 35877 Tweets
呂布カルマ
• 32003 Tweets
うまい棒
• 28429 Tweets
ブロック機能
• 25746 Tweets
Jags
• 24528 Tweets
Dera Sacha Sauda
• 24057 Tweets
Trevor Lawrence
• 20769 Tweets
Bron
• 16727 Tweets
VOTA X KARIME
• 15987 Tweets
Joe Burrow
• 15043 Tweets
#RaiseHail
• 14864 Tweets
ラッパー
• 13685 Tweets
lorde
• 13100 Tweets
ブロック改悪
• 11694 Tweets
Pinned Tweet
Think Transfomers are terrible at logical reasoning? Think again 💥
In this collaboration with Samy Bengio,
@jsusskin
(Apple) & Emmanuel Abbé (EPFL), we show that when trained with Boolean inputs and symbolic outputs, they become very powerful 🧠
🧵⤵️
8
122
612
Thrilled to announce that I will be joining
@MetaAI
next month as a Research Scientist 😍
I will be working in the Brain & AI team on decoding language from neural activity, to hopefully help those which have difficulties to speak or type. Learn more here:
12
12
274
🚨 ODEFormer is on Arxiv!
We show that Transformers can recover the differential equations governing dynamical systems from noisy & irregularly sampled trajectories.
Very fun collaboration with
@SorenBecker
,
@TrackingPlumes
,
@pschwllr
&
@k__niki
!
🧵⤵️
3
52
249
CNNs are more sample-efficient, ViTs are more powerful.
Can we get the best of both worlds?
Check out our paper accepted at
@icmlconf
Thanks to my collaborators
@HugoTouvron
@leavittron
@arimorcos
@leventsagun
@GiulioBiroli
🧵⤵️
3
28
176
After a couple great years at
@MetaAI
and
@ENS_ULM
, I will be starting as
@AI4ScienceEPFL
fellow next month 😍
Can’t wait to leverage modern AI tools with biologists, neuroscientists, chemists and physicists 🧬🧠🧪🔭
If you work at
@EPFL
and want to meet up, please reach out !
4
3
85
1,2,3,5,8,13… What is the next term ? This kind of question is typical in IQ tests, but has received little attention in AI.
We had great fun training Transformers to tackle this problem, check out our paper and our online demo :
Deep Symbolic Regression for Recurrent Sequences -- We show that transformers are great at predicting symbolic functions from values, and can predict the recurrence relation of sequences better than Mathematica. You can try it here:
25
162
733
5
16
74
New preprint : .
When and how should you decay your learning rate ? We give some theoretical insights on this crucial question in our latest work with
@MariaRefinetti
and
@GiulioBiroli
. (1/3)
3
11
40
We hope this work can be applied to other fields in science and spark more research on symbolic reasoning in LLMs.
We release our code & models publicly and provide a pip package & interactive Colab demo!
A few attention maps for the pleasure of the eye:
0
2
40
The so-called "Boolformer" takes as input a set of N (x,y) pairs in {0,1}^D x {0,1}, and tries to predict a Boolean formula which approximates these observations.
Here are two very simple examples: addition and multiplication of 2-bit numbers.
1
6
39
Great video by
@ykilcher
on our symbolic regression paper with
@pa_kamienny
@GuillaumeLample
@f_charton
. Watch until the end to discover his musical skills 😅
📜Paper Video Time!📜Today I'm talking to Stéphane d'Ascoli (
@stephanedascoli
) about Deep Symbolic Regression for Recurrent Sequences. This model is given a sequence of numbers, like 1, 2, 3, 5, 8 and it figures out the *rule behind* the sequence. Insane🤯
4
50
220
0
5
31
Cet après-midi, mon ami Arthur et moi avions l’honneur d’être invités par Étienne Klein sur France Culture dans la Conversation Scientifique. Une émission consacrée pleinement à notre sujet préféré, l’espace-temps et sa courbure : a (ré)-écouter ici !
1
0
24
We apply the Boolformer to a set of classification tasks from PMLB, ranging from predicting chess moves to diagnosing horse colic.
Our model achieves similar performance to classic ML methods, while outputting interpretable Boolean formulas!
1
1
25
Vos semaines surchargées et vos fins de mois difficiles ne sont plus une excuse pour ne pas vous intéresser à l’IA ! Avec ce nouveau livre aussi concis que bon marché, découvrez une nouvelle notion sur l’IA chaque jour entre deux arrêts de métro !
0
3
20
We also applied the Boolformer to the task of gene regulatory network inference, which is central in biology.
On a recent benchmark, our model is competitive with state-of-the-art genetic algorithms for Boolean modelling, while running several orders of magnitude faster!
1
1
19
Double descent has recently become popular in deep learning, but a similar curve was observed in the 1990s for least squares. Wonder if these kinds of overfitting are the same ? Come and see our Spotlight at
#NeurIPS2020
and chat in the poster session !
0
4
13
🧑🔬 We hope our method can guide the intuition of domain experts in many fields of the natural sciences.
To facilitate this, we released ODEFormer & ODEBench publicly and built a pip package & interactive demo to help get started:
0
1
13
Yeah James Webb is nice, but did you know that you can produce these kind of pictures using just… an iPhone (with a perfect night, long exposure and a bit of postprocessing) ?!
Taken in Pumalin national park, southern Chile.
1
1
9
📈 Given the limitations of the "Strogatz" benchmark for this task, we introduce ODEBench, a more extensive collection of dynamical systems curated from the literature.
On both benchmarks, ODEFormer achieves SOTA, with fast inference and impressive robustness to noise!
1
0
10
Could Transformers become the gold standard for symbolic (and even non-symbolic) regression ?
Check out our latest paper to find out !
Our new paper on Symbolic Regression with
@pa_kamienny
@stephanedascoli
@GuillaumeLample
is now on Arxiv !
We achieve performance comparable to SOTA genetic algorithms on SRBench with Transformers, whose inference time is orders of magnitude lower!
1/4
3
27
117
0
1
7
Le livre sur la relativité que nous avons écrit avec Arthur Touati est enfin entre nos mains !
Si vous avez aimé Interstellar et vous voulez vous replonger la tête dans les étoiles, n’hésitez pas à le pré-commander ici :
Sortie officielle le 25 mars 🚀🛰🧑🚀
1
0
7
In convex problems, the best is to decay as 1/time. What about non-convex problems? For random gaussian losses on the sphere, we show that the optimal decay rate is smaller than one (0.5 in plot below). This could explain why the inverse square root schedule is so popular! (2/3)
1
1
7
🌎We open-sourced our code and pretrained models at . Enjoy playing around with it !
0
1
7
🚀The ConViT benefits from a vastly increased sample efficiency, without any sacrifice in terms of maximal performance. We hope this model will spark more exploration of "soft" inductive biases, which make learning easier, but vanish away when not needed!
1
0
5
We then study inference problems, where two phases emerge: a search phase, followed by a convergence phase once the signal is detected. Here, the optimal schedule is to keep a large constant learning rate to speed up the search, then decay as 1/time once in a convex basin. (3/3)
1
1
5
@NicolasToueille
@Aurelie_JEAN
@ylecun
Merci beaucoup ! Un honneur d’être parmi d’aussi grands noms 😇
1
0
4
Très honoré que notre Voyage au Cœur de l’Atome soit récompensé ainsi ! 🤩 Merci
@PrixROBERVAL
,
@badry96
et
@editionsfirst
Retour sur la cérémonie du Prix Roberval: Bravo à la lauréate de la catégorie Grand Public Aline Richard Zivohlava pour son œuvre “La Saga CRISPR” et au coup de cœur des médias de la catégorie Grand Public “Voyage au cœur de l’atome” d’Adrien Bouscal et Stéphane d’Ascoli !
0
2
8
1
1
4
💡The ConViT uses Gated Positional Self-Attention (GPSA) layers, which are initialized to mimick convolutions, then let each attention head learn more complex relationships through a learnable gating parameter.
1
0
4
🧑🔬Hybrid models are a good compromise, but optimal architecture is very task-dependent.
What if we let each layer decide whether to perform convolutions or self-attention? This is the idea behind the ConViT, an “adaptive” hybrid model!
1
0
3
0
0
1
0
0
1
Come visit us at
@icmlconf
on Wednesday evening !
The source code for our ICML 2022 paper Deep Learning for Recurrent Sequences () is now available on .
Spotlight: Wednesday 20, 16:50 ET
Poster session: Wednesday 20, 18:30 ET
@stephanedascoli
@pa_kamienny
@GuillaumeLample
1
2
17
0
0
1
@francoisfleuret
Stochastic method: pick a learning rate eps and initialise m=x_0. Then if x_i > m, m+=eps, otherwise m-=eps. You can decay the learning rate etc.
0
0
1