Finally, a #CLIP #AI model without the #text #obsession / typographic attack vulnerability. It's better in all other aspects (zero-shot, retrieval, linear probe), too. But what's best about it: You'll find the 🧑‍💻 code to train it below (bonus: 📝 paper).

Proper Attention Heatmaps, too.

Paper. 🙃.

Code for fine-tuning and checking against all the claims I am making:.

SCAM is BLISS.Though SigLIP beat me to it.For a typographic attack diss. I'll happily take 2nd place for my #CLIP, tho, considering. Accuracy 41% -> 78% using $10 of compute (finetune) 😂

Code is done & documented 100% now btw. Now I just need to madly benchmark this "min cos similarity adversarially trained" model around for release, as I'll now be releasing TWO models. 😂. ETA tomorrow.

#CLIP's opinion, max cos sim + adversarial images -> forcing those apart during fine-tuning: Like smashing through embeddings with a baseball bat. But CLIP's weird "NOT-image" min cos sim label?.✅ preserves hallucinwords 🤓👍.✅ 53/60 attack resilience (vs 59/60 w/ max cos sim)

#CLIP's "ah, I figured this pattern out now - holdon, lemme just *memorize* this dataset real quick 🤖" point. Linear probe: ♥️ this (initially).Zero-shot: 🚫 hates this .#overconfidence #overfit #AI #model #memorization

3 - layer 23 (final)

2 - Layer 6

1 - Layer 1

I think I'll have to finish up the code and just release it like "it just happens because of fine-tuning this way!", rather than being able to actually explain WHY exactly this happens. Oh well. It's a complicated and complex thing of tensorial / algorithmic weirdness. 🙃.

Curious: The positions for register patch emergence are similar in NEW finetune #CLIP vs. pre-trained OpenAI CLIP. But my finetune (top) then 'drains' them into [via TE: one-hot CLS for "a dog" / "a cat"], vs. pre-trained: Drains into other register. 🤔.#AI #vision #transformer

Maybe a cosine dissimilarity (min cos sim) from within CLIP can be better than my 'adversarial' classes. What the heck do I know what an anti-cat is in some crazy-dimensional vector space, anyway?

No-Adv model is also only slightly less likely to classify adversarial perturbation (PGD) correctly, vs. with adversarial dataset. Seems like typographic attack vulnerability and vulnerability to adversarial perturbation are just partially, but not fully, overlapping things.