Amit Parekh
@amitkparekh_
Followers
64
Following
439
Media
5
Statuses
42
Nice post on software engineering. "Cognitive load is what matters" https://t.co/eMgxu0YgWw Probably the most true, least practiced viewpoint.
152
824
7K
I will be presenting our paper, Shaking Up VLMs: Comparing Transformers 🤖 and Structured State Space Models 🐍 for Vision & Language Modeling today at #EMNLP24. If you are interested come hang out by our poster (Riverfront Hall 16:00). Details here:
0
3
11
If you are around in #EMNLP2024, come see me talk about our work on discovering minority voices in datasets ( https://t.co/gsxRKdX1VJ). I’ll be on the Ethics, Bias, and Fairness slot in the Ashe auditorium today, but also very open for chats throughout the conference!
0
3
10
Really pleased to say this has been accepted at #EMNLP2024 main
🚨 NEW PAPER ALERT 🚨 Introducing the GlobalBias dataset… We ask Claude 3, GPT 3.5, GPT 4o, and Llama 3 to produce character profiles based on given names from GlobalBias for 40 different gender-by-ethnicity groups. We find that all models displayed stereotypical outputs (1/4)
2
4
20
LLMs are great but they are brittle to minimal prompt perturbations (e.g., typos, indentation, ...). Q: How do we create truly multimodal foundation models? A: Do as we humans do: text as visual perception! Enter PIXAR, our work at #ACL2024NLP! https://t.co/YQFltalAWE
3
17
53
We developed a framework to find robust clusters of diverse minority perspectives, without adding metadata or explicitly training for it!!! Check out the paper for details
So very, very proud to share our new paper “Voices in a Crowd: Searching for Clusters of Unique Perspectives” (arXiv:2407.14259), a novel framework on how to organically find clusters of unique voices (perspectives) in datasets. 🧵 for summary, co-authors @amitkparekh_ @sinantie
0
0
5
🚀 Excited to share our latest paper: "Enhancing Continual Learning in Visual Question Answering with Modality-Aware Feature Distillation"! Paper: https://t.co/x0xYonE8ST (1/5)
1
7
18
We've released everything (w/✨one-liners✨) so take what we did and evaluate your models to make sure that they are not merely guessing how to act. Code:
github.com
A comprehensive framework to explore whether embodied multimodal models are plausibly resilient - amitkparekh/CoGeLoT
0
0
1
Summary: We find that previous findings in robotic manipulation tasks might be evidence of overfitting on instructions and/or spurious correlations. Paper:
arxiv.org
Evaluating the generalisation capabilities of multimodal models based solely on their performance on out-of-distribution data fails to capture their true robustness. This work introduces a...
1
0
0
To see if models simply perform random actions until task completion, we added 2 new difficulty levels (e.g., adding many distractors & changing expected affordances); finding that they significantly impact performance.
1
0
0
To assess which modalities guide performance, we masked each modality separately in the multimodal instruction. Models succeed when given purely visual input, but also when given no instructions whatsoever (although performance deteriorates when language tokens are masked)!
1
0
0
To assess impact of perturbations to multimodal instructions, we “Gobbledygook’d” them: changing how the language modality is given to the model in two distinct ways. When we do this, models find a way to succeed, suggesting that language might not be the most important modality.
1
0
0
We extend VIMA-Bench (@YunfanJiang @DrJimFan) to see how models perform at different generalisation levels and multimodal perturbations (see table). We find that both object-centric or image-patches models are resilient to these perturbations.
1
0
1
Our paper “Investigating the Role of Instruction Variety and Task Difficulty in Robotic Manipulation Tasks” investigates whether models are affected by perturbed instructions or increased task difficulty, and find that performance is generally unaffected (even when it should be!)
1
0
1
What if your multimodal model succeeds when given nonsensical instructions? What if it even succeeds without being told explicitly what to do? Have we finally achieved AGI? We answer at least two of those questions w/ @NikVits @ale_suglia @sinantie (🧵for more)
1
3
10
next year we will have AI job interviewers meeting AI applicants “this meeting could have been an API call”
11
100
945
yeah I'm working on the frontier of AI (googling pytorch errors that only me and one FB engineer have run into)
1
5
83
The Chinchilla scaling paper by Hoffmann et al. has been highly influential in the language modeling community. We tried to replicate a key part of their work and discovered discrepancies. Here's what we found. (1/9)
16
129
902
Semantics at @semdialmeeting "Modelling Disagreement or Modelling Perspectives?" by @NikVits, @amitkparekh_, @t_dinkar, @gavin_does_nlp, @sinantie & @verena_rieser We predict disagreement on subjective data while preserving individual perspectives! https://t.co/NvyAYVWBUz
1
2
5
I cannot get over how beautiful this book is from @francoisfleuret . NeurIPS fashion accessory for the year.
7
21
525