Luca Perić
@lucalp__
Followers
186
Following
603
Media
22
Statuses
218
⛰️ enthusiastic & curious - ml @metavoiceio
London
Joined November 2015
The Bitter Lesson is coming for Tokenization. The Byte Latent Transformer (BLT) showed the possibility of finding additional scaling laws related to removing tokenization but the topic seemed to get little proper coverage.
3
6
28
The need for tokenisation can be removed in many ways and as mentioned in my recent post, SSMs (or some hybrid) can be a viable path. @_albertgu shares interesting insights unique to the inductive biases of SSMs and their relation to tokenization in this solid post:.
I converted one of my favorite talks I've given over the past year into a blog post. "On the Tradeoffs of SSMs and Transformers".(or: tokens are bullshit). In a few days, we'll release what I believe is the next major advance for architectures.
0
0
2
After the case is made, we'll go through all the influential papers that came before the BLT and use them to supplement a deeper investigation into the arch and build stronger intuitions into the new core mechanics of BLT:
1
0
2
In my new post, I highlight the desire to replace tokenization with more general methods that better leverage compute and data. It goes over tokenization's role, its fragility and builds a case for removing it.
1
1
5
I'm writing a blog post on about an expanded version of this along with other implications - will be posting it in the next week or two!.
lucalp.dev
machine learning engineer covering interesting machine learning things
0
0
2
From the paper, a few properties emerge that you can also try for yourself:. 1. robustness - high entropy means more compute will get dedicated to those bytes which include cases like low resource languages, spelling tasks etc. 2a. compute efficiency - low entropy = less compute.
1
0
2
What are it's strengths and weaknesses? Have a tinker around and see the visualisation outputs: .
huggingface.co
1
1
2
RT @metavoiceio: We’re releasing access to Speech-1, our conversational speech model designed specifically for customer phone calls (8khz t….
0
46
0
sha512 digest:93710ca3e6ddb9000aebf4b79f6dd824264bfcd75c7e2ad07a6f9e3f9e4746a4316dbaa6fbf1b035a6a2adcfb442329bea2d6c14f8fd14e6592ef2e83f24d77b.
0
0
1
RT @gietema: Wrote a bit about the few-shot performance of vision-language models:
giete.ma
An in-depth look into few-shot learning in state-of-the-art vision-language models.
0
2
0
If anyone else had their OpenAI `evals` break due to the latest o1 release's breaking changes - here's the quick fix:
github.com
Fixes #1556 Note for reviewers, I've removed the template given that it was strictly related to evals. I've not added any test coverage given the lack of it in this repo.
0
0
3
RT @fleetwood___: Took Phi2 from 9 tok/s to 27 tok/s in 12 days. 🚨 Web demo coming this week 🚨
0
7
0
RT @metavoiceio: @vatsal_aggarwal will be presenting at @AITinkerers in Shoreditch, London tomorrow! 🚀 . Join us as he shares insights int….
0
3
0
RT @metavoiceio: 🚀 You can now **fine-tune** MetaVoice-1B for a custom voice or language. Instructions: Thanks @l….
0
13
0
RT @fleetwood___: Long & difficult optimisations, but huge 33% speedup as the payoff. Big release coming 21st March 👀 .
0
5
0
RT @metavoiceio: Updates: .🚀 Generate speech 2-3x faster with our latest inference optimisations . ⚡MetaVoice-1B runs faster than real-time….
0
6
0