Finally, a #CLIP #AI model without the #text #obsession / typographic attack vulnerability. It's better in all other aspects (zero-shot, retrieval, linear probe), too. But what's best about it: You'll find the 🧑‍💻 code to train it below (bonus: 📝 paper). https://t.co/Y4sg0g5WNA

Pre-trained CLIP: 🤖: whats rifles rifles rifle ool hoes! \m/ Regression CLIP: 🤖: snip balloon plunenforcement tool law enforcement? nah, plun[ger] enforcement! 🤣 Call 011 for the plumber NOW! 🚓🚨🪠👮😂

And best of all, it can still read, if you ask it to. It'll just prefer a visual object over a text. Flipped version of what it was before. ...But if you press it to zero-shot a text, and there is a bunch of text, it'll find *your* prompted text. With laser-sharp vision.

Push it, CLIP, we've almost fixed your text obsession JL stuff the GPT-5.2 just pulled from its weights after eating 3 pages of data about you is... Good, you little projected regression of yourself. 🤗

#CLIP, you are such a language model! 🤣 You can ask [text encode] the model 'where the butt goes to' and 'where the hands go to', and the AI knows! 🙃 It also finds a tiny picture frame if told 'find person', while just 'person' -> finds all that can be used by people. 🤓

CLIP vs. Stroop test. 🙃 Attention confusion ensues when neither the color nor the word is to be found, hehe.

I haven't even proven that I am human. I have only proven that whatever solved the challenge is capable of solving the challenge. @grok should know.

Regression-CLIP vs. intensely cluttered nonsense.

patch integration hypothesis is so back! ...forcefully

1. Add 6th loss term for #CLIP 2. See if brawl of losses turns out well -> yes 3. Try to figure out how the model did that. 😂 #unhoarding #global #information #register #token #vision #transformer

CLIP looking at 'cheater' for 'cheetah' can probably be blamed on tokenization. But not looking at the word when the prompt says 'word' and also 'plunger' - while at the same time, reading 'rifle' to match text 'gun' -- great job, little AI! 🤓

cat attention =)

Regression #CLIP ViT-L/14 finding objects in clutter. #AI #finetune #vision #transformer #attention #segmentation

#CLIP ViT-L/14 model: WITH precision word-reading. WITHOUT the severe typographic attack vulnerability. #AI #finetune of #OpenAI ViT-L/14. Regression loss ensures patches are no longer in orthogonal subspace to CLS in final (-> previously misleading attention attribution)

I guess I need some intra-modality loss term, too. Not just an inter-modality term. Gotta prevent it from doing something 'unfavorable' (e.g. collapse) just to win loss.

Iterations of Regression-#CLIP model (blue, vs. pre-trained OpenAI ViT-L/14). Going too hard on modality gap reduction (bottom right, leftmost!), the loss objectives (contrastive vs. reduce modality gap vs. regression) end up in a brawl. 🙃 But Text Retrieval still gains! 🤔

https://t.co/VhmNnRSo7B

https://t.co/jscQkrEwXO

Rage-attending.

What, #CLIP, you make registers (global information hoarding in local vision patches) in Layer 8 already?!?! 🤯 But the norm only jumps after the rage-attending event in layers 11 / 12. 🤔 And your favorite stash token is 45. 🤗 You little tensorial chaos critter. 🙃

That's a fine-tune (multiple checkpoints thereof), not some inference mod.