RT @sharbat_c: Funny how this is true across countries.

RT @sharbat_c: Monsieur le Président, j’ai Choosé France for Science (France m’a Choosé for Science, merci MSCA fellowship) et ça passe bie….

3️⃣ We explain why structural information in PE has proven to be empirically successful. We go back to 1️⃣ and use the content-context connection to show that the higher is the mutual information between data and its positional representation, the better is task performance.

2️⃣ We introduce a new positional encoding method - RoPEPool - that can model causality. How does RoPEPool compare to RoPE and F-StrIPE? Our analysis with a toy example says: RoPEPool isn’t just different, it’s also richer in terms of expressivity.

1️⃣ We show how different families of positional encoding - rotation-based (RoPE) and random fourier features-based (F-StrIPE) - can be compared using kernel methods. It’s not just vibes - we characterize precisely how queries and keys are affected by positional information.

🚨 We just submitted a follow-up to this work, now available as a preprint:

With these two interventions, we obtain better performance at lower cost! 🚀.Curious? Check out the companion webpage:

In our paper, we show that stochastic positional encoding is, in fact, a noisy version of a well-known kernel approximation technique: Random Fourier Features. We also show how prior knowledge (e.g. related to musical structure) can be used in such linear-complexity Transformers.

However, there was a piece missing: how do you handle relative positional encoding in these linear-complexity transformers? 🤔 . Enter Stochastic Positional Encoding! It brings relative positional information back into the picture without going back to quadratic cost.

Luckily, there's a solution: you can think of attention as a kernel function and use kernel approximation techniques to reduce the cost from quadratic to linear. ⚡.This was the idea used by Performers, for example.

Transformers are powerful, but there's a problem: their cost grows quadratically with sequence length. 📈.This makes it really hard to apply them to lengthy sequences, like music, where long-term connections carry important information.

ICASSP 2025 has begun! I will be presenting "F-StrIPE: Fast Structure-Informed Positional Encoding for Symbolic Music Generation" on April 11 in the lecture session "Machine learning for speech, audio and music processing II" at 2 PM. More details below 🧵

RT @sharbat_c: Did Qin Shi Huang know when he united China under a Legalist bureaucratic system 2200 years ago, that he would start a proce….

RT @sharbat_c: Just explained to general disbelief of my Indian parents how @Sorbonne_Univ_ hasn't yet reimbursed me ~1200€ for over 4 year….

I will be in London next week visiting @c4dm ! If you’re around, come say hi 😄.In other news, we’re looking for a Master’s student to carry forward some of this work that I’ll be talking about at next week’s seminar 👇(advert in following tweet).

RT @nikhilsinghmus: Thrilled to join @DartmouthCS as Assistant Professor in Jan 2025! I’m seeking 1-2 PhD students to join in Fall 2025. A….

RT @deeplearnmusic: 👩‍🎓@SonyCSLMusic has an open 3-year PhD position starting in spring 2025!. We are excited to invite applicants who meet….

🙈

*laughs in ICASSP deadline*.