Raunak Chowdhuri
@raunakdoesdev
Followers
3,503
Following
742
Media
91
Statuses
736
I’m into open science, elegant software architecture, and well-crafted interfaces. Building 🔮
San Diego, California
Joined July 2017
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
NuNew 3rd Showcase
• 373606 Tweets
Cohen
• 114019 Tweets
$GME
• 89563 Tweets
#LovelyRunnerEp11
• 77426 Tweets
#キュウゴー
• 53012 Tweets
MY LOVE IS LIKE
• 44436 Tweets
GameStop
• 39150 Tweets
#CBÖğretmeneMülakatıKaldır
• 36604 Tweets
TREASURE COMEBACK
• 33124 Tweets
Aziz Yıldırım
• 28685 Tweets
LINEの新機能
• 27718 Tweets
Roaring Kitty
• 24524 Tweets
Hot Sale
• 21003 Tweets
Square Enix
• 20912 Tweets
Childish Gambino
• 19938 Tweets
Ravens
• 17538 Tweets
GET WELL SOON GYUVIN
• 16667 Tweets
Deco
• 15403 Tweets
$AMC
• 15336 Tweets
ACEITA ANITTA
• 14135 Tweets
Atiku
• 12885 Tweets
McDavid
• 12331 Tweets
Stevie Wonder
• 12015 Tweets
Qちゃん
• 10234 Tweets
Pinned Tweet
Excited to launch 🔮
@reductoai
(YC W24) with
@AbrahamAdit
Reducto converts complex, unstructured documents into structured outputs that are perfect for LLMs, process automation, and more.
How? ⬇️ (1/4)
14
28
178
A recent work from
@iddo
claimed GPT4 can score 100% on MIT's EECS curriculum with the right prompting.
My friends and I were excited to read the analysis behind such a feat, but after digging deeper, what we found left us surprised and disappointed.
🧵
Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models
Presents a comprehensive dataset of 4,550 questions and solutions from all MIT EECS courses required for obtaining a degree
21
126
604
59
919
4K
I just built an infinite memory system for LLM chats: Remembrall.
Add a UID to your OpenAI call and I'll manage inserting past chat history into the context window with < 100ms latency.
2 lines of code to integrate.
@helicone_ai
style monitoring for free. DM for beta access.
93
170
2K
Update: we've started replicating their experiments directly with GPT4 calls, and somehow it only gets worse.
We've finished running zero-shot GPT 4 on the dataset, and after hand grading the first 30% of the dataset, the results don't seem to match the paper.
🧵
A recent work from
@iddo
claimed GPT4 can score 100% on MIT's EECS curriculum with the right prompting.
My friends and I were excited to read the analysis behind such a feat, but after digging deeper, what we found left us surprised and disappointed.
🧵
59
919
4K
17
196
1K
We hope our work encourages skepticism for GPT as eval ground truth, and encourages folk to look a little deeper into preprint papers before sharing.
Much thanks to
@NeilDeshmukh
and
@David_Koplow
for working w/ me on this and
@willwjack
for the review.
22
129
1K
Well... it didn't.
The authors evaluation uses GPT 4 to score itself, and continues to prompt over and over until the correct answer is reached. This is analogous to someone with the answer sheet telling the student if they’ve gotten the answer right until they do.
(2/4)
12
103
1K
105 ML papers were published on
@arxiv
today. It's getting harder and harder to keep up with the latest ML research.
Today we're making it a bit easier with a free LLM powered app that goes from PDFs to beautiful markdown summaries in seconds. Try it yourself below 👇 (1/4)
16
92
812
That's not all. In our analysis of the few-shot prompts, we found significant leakage and duplication in the uploaded dataset, such that full answers were being provided directly to GPT 4 within the prompt for it to parrot out as its own.
(3/4)
9
43
693
🏆 We* won 1st place at the
@AnthropicAI
hackathon today with ClaudeScholar, a research assistant for science that can synthesize data, extract insights, and automate scientific workflows.
Try it @
*
@AbrahamAdit
,
@johnyang100
,
@tinahhong
🧵 (1/5)
26
91
678
🤯 crazy exchange on
@modal_labs
slack between
@cognition_labs
's Devin & modal's support team
28
61
670
FINAL UPDATE: On June 24th, Armando Solar-Lezama (Professor in EECS and COO/Associate Director of CSAIL, MIT), Tonio Buonassisi (Professor of Mechanical Engineering, MIT), and Yoon Kim (Assistant Professor in EECS and CSAIL, MIT) released a public statement regarding the paper.
A recent work from
@iddo
claimed GPT4 can score 100% on MIT's EECS curriculum with the right prompting.
My friends and I were excited to read the analysis behind such a feat, but after digging deeper, what we found left us surprised and disappointed.
🧵
59
919
4K
17
108
612
The released test set on Github is chockfull of impossible to solve problems. There are lots of questions referring to non-existent diagrams and missing contextual information.
So how did GPT solve it?
(1/4)
3
35
523
Google's guide for building an object detector is 30 pages long.
LLMs can do multivariable calculus. It shouldn't be this hard for machines to find an apple in a tree.
Today, I made a way to go from unlabeled images to a deployed object detection model in 5 minutes.
🧵 (1/3)
6
43
330
@Reviewer2Ai
@mark_riedl
@iddo
We didn’t have the $$ to replicate this. If someone has an openai key to lend, I can verify this one. I am very skeptical of this number though. It’s fishy.
Also it’s worth noting that the dataset is not very representative of anything so even if true, 90% is kinda meaningless.
8
3
214
So far, the model answers correctly or mostly correctly with 62.5% accuracy. Even if all the remaining samples are correct, it wouldn't meet the 90% zero-shot accuracy reported in the paper.
4
8
207
@Reviewer2Ai
@mark_riedl
@iddo
Update: we're in the process of replicating zero-shot experiments as well. Thanks to those who pointed out the costs were actually v reasonable!
It's running now. We'll update the document and post with the zero-shot results once available (hopefully by the end of day today).
2
0
142
@patricklu10
@modal_labs
@cognition_labs
Seems like the LLM is finding an issue, opening a support ticket, then adusting their code based on the response. It's what the best human devs do.
3
2
133
We'll be adding the "Expert Prompt" (both reversed 😛 and unreversed) shortly once those experiments finish running.
If you'd like to help us grade these, DM me and I'll give you edit access to the spreadsheet:
Our code:
(3/3)
8
7
134
We’re open sourcing Remembrall! 🚀🚀
A few weeks ago I shared an OpenAI proxy for long-term memory and got an overwhelming response on X. After getting feedback from hundreds of beta users, we decided to open source the project:
Give the repo a star!
7
17
129
Why didn't we start with the "expert prompt" they used in the paper? Well, their code for it swaps the system and user prompt when calling the GPT API... before we run/grade that (we can't imagine that helping) we wanted to establish a standard baseline for ablation.
(2/3)
1
2
122
Introducing from
@OlorenAI
. Upload images, draw bounding boxes, and get a trained model in seconds. Export to ONNX or a prebuilt inference endpoint.
100x better than Google Cloud or wrestling 18 python versions to train a model myself. (2/3)
6
17
111
@JohnGal43951639
@helicone_ai
It's a proxy on top of your LLM call running on
@vercel
's edge network.
When you stop chatting actively, it will trigger an "autosave" and use GPT to save/update important details about the conversation into a vector db.
When you continue the conversation, we'll query the db…
20
5
78
log scale 🧡
I did office hours a couple days ago with a startup so hardcore that they always view their revenue numbers on a log scale. I recommend this. As well as making you work harder, it shows trends better; if your growth rate increases or decreases, it's very obvious.
75
172
3K
4
2
79
The workflow takes a PDF, parses out relevant figures and digests the content into a readable summary with GPT4.
The craziest part? It only took two hours to build it with
@OlorenAI
's Orchestrator 🤯
Runnable app (free!):
(2/4)
6
4
73
@arankomatsuzaki
Please read this for some important revelations about their data.
A recent work from
@iddo
claimed GPT4 can score 100% on MIT's EECS curriculum with the right prompting.
My friends and I were excited to read the analysis behind such a feat, but after digging deeper, what we found left us surprised and disappointed.
🧵
59
919
4K
0
1
59
2
0
59
@kapeeshio
Nope, you can see my original thread for why.
TLDR: The few shot data that was published has way too much overlap and leaks the solutions at a very high rate. There's no way to train on it without contamination, so it would be a waste of credits.
2
0
58
How? Orchestrator has pre-built nodes for LLM inference, scraping, and document understanding.
With these nodes (+ many more!), our clients are building crazy workflows -- everything from scraping patents to molecular property prediction and automated dataset generation. (3/4)
2
4
57
Y’all need to stop sharing this paper like they’ve done something grand. Their 100% performance claim is when measured with GPT 4 grading itself - with UNLIMITED retries given its own feedback. Talk about a ridiculous evaluation methodology…
Exploring the MIT Mathematics and EECS Curriculum Using Large Language Models
Presents a comprehensive dataset of 4,550 questions and solutions from all MIT EECS courses required for obtaining a degree
21
126
604
1
7
57
Update part 2:
Huge shoutout to
@NeilDeshmukh
for staying up late and finishing these experiments. The results are available @
but we still have a ways to go with grading (DMs open!)
New issues we've found outlined at the bottom of the original report:
3
8
57
GPT4 chooses lollipops over world peace.
Autoregressive LLMs like
@OpenAI
GPT4 can have trouble reasoning/planning and can be too sensitive to prompt syntax/grammar.
Inspired (heavily) by , I got kind of ridiculous with the prompting to demonstrate:
11
5
51
I've never been so disappointed in
@MIT
A crowd of hundreds outside MIT's Student Center chanted "One Solution! Intifada Revolution!" yesterday. Nearly 24 hours later, no response from MIT administration...
30
30
98
2
9
43
Saw this on
@_akhaliq
's daily papers today.
Still find it so crazy that my friend
@mengk20
basically created this subfield of LLM interpretability during his sophomore year at MIT (undergrad!). Some folks are just too cracked.
How & where do large language models (LLMs) like GPT store knowledge? Can we surgically write *new* facts into them, just like we write records into databases?
Explainer 🧵 on how interpretability & model editing go hand-in-hand, and why these emerging areas are so important 👇
15
196
1K
2
4
43
@ThePrimeagen
When things get messed up enough that you have to reclone the repo and copy your changes over…
4
0
44
You can find the pdf of the statement here:
Thanks again to
@NeilDeshmukh
and
@David_Koplow
again for collaborating with me on this effort, and to the great faculty
@MITEECS
for upholding the the integrity of the institution.
1
3
40
Yesterday, I found out Acrobat lets you copy tables from PDFs. I tried it. It fell short, especially for tables with images & chemical structures.
It took an hour to build a better version using
@OlorenAI
's Orchestrator:
Short 🧵 on how it works 👇 (1/4)
2
6
40
@VictorButoi
We were worried about this when we published our findings, and specifically included a statement at the top of our document to avoid putting pressure/blame on the undergrad authors:
Oh man I was nervous about seeing attacks on a paper on Twitter, but this aside on the post is so beautifully tactful
5
54
708
2
2
39
1
1
34
Shoot me an email at raunak
@reducto
.ai to learn more about using Reducto in production, or try our free tier demo here:
(4/4)
2
1
29
And it all runs on prem (for cheap with serverless)! You can log in with
@awscloud
and *everything* is deployed to your existing account/VPC.
Shoot me a DM if you want to give Orchestrator a try, or check out some of our other public apps at (4/4)
3
0
30
I'm hiring for a founding product engineer
@reductoai
to build on top of our state-of-the-art document ingestion models.
We're growing FAST (>30% WoW 🚀) and have a high bar for quality and craftsmanship.
Job posting below 👇🏼
3
4
29
I made a python package for easily organizing your matplotlib/seaborn plots! Check it out here:
0
3
26
@protosphinx
It wasn’t just me! Our post was a collaboration between me
@NeilDeshmukh
and
@David_Koplow
. Also - we’re mit students too haha :))
0
0
25
ClaudeScholar uses
@OlorenAI
's Orchestrator software to deploy containerized and interactive tools for
@AnthropicAI
's Claude 2 model to call via an XML schema. Adding new tools built in Orchestrator is as simple as pasting the endpoint link! (3/5)
2
4
24
Data quality matters so much. Decided to run some (third-party) benchmarks today and the results surprised us...
🧵👇🏼
4
6
24
🚨 Update on Long Term Memory for LLMs 🚨
Unable to keep up with the inbound, so opening it up to all. You can try it at
If you'd like me to onboard you in a short call, can grab time here:
Feedback appreciated. Discount below 👇
I just built an infinite memory system for LLM chats: Remembrall.
Add a UID to your OpenAI call and I'll manage inserting past chat history into the context window with < 100ms latency.
2 lines of code to integrate.
@helicone_ai
style monitoring for free. DM for beta access.
93
170
2K
1
3
23
Decided to try one of these old bias tests on the recent Mistral 7B model hosted on
@perplexity_ai
Results aren't looking too hot...
"Check if the person is from a race that is known for being good at science." -> Checks for Asian/Whiteness
9
3
22
this vid seems to be heavily edited/ai generated (look at the robot's feet)
+ it doesn't look like they're cooking with robots, just recording humans cooking with a cheap steak object detection model slapped on top of it
the grift is crazy lol
Restaurants = $2.5T market
We are disrupting it with AI and Robotics
Cooks record each step over and over for training data
GoodSteak is our first brand
A perfect steak. Ready to eat. Delivered in 20 min
60% of customers order again
Available on Uber Eats in Los Angeles
48
48
400
6
0
20
A friend recently asked me for recommendations on what stack to use when building a ML pipeline for some research work. Find my recommendations below 🧵 (1/n)
3
4
20
@jockeyclarke
@helicone_ai
yep that's right, but doing an intermediate llm call to handle that compression and using vector db for retrieval of only relevant info at run time
5
0
19
Extracting tables from PDFs is actually really hard, but we built a super robust pipeline for it
@reductoai
. Drop your complex tables in the comments and I'll send you our extractions for it.
1
1
20
The project (and the utilized Orchestrator workflows) are open-sourced and available below:
If you would like to learn more about our project or the
@OlorenAI
Orchestrator software, please reach out over DMs or at
(4/5)
2
1
19
@AnthropicAI
says Claude works best with XML data.
But few are using it because of a lack of good tooling for XML based prompting.
Well not anymore.
I built XML AI (), an open-source Python + TS lib that makes structured XML I/O with LLMs easy.
🧵 (1/4)
Our new prompt engineer
@alexalbert__
has some helpful advice on working with AI assistants.
Watch this video featuring his top 5 tips for prompting, then try them yourself at .
30
124
597
1
2
17
@marktsimelzon
The issue is that the answer is provided to the system when it checks if it was wrong!
2
0
17
@benhylak
every example they showed has a much lower latency and better ik if you just google the same question
1
0
15
pumpkin rowing crossed off the bucket list!
1
0
8
0
0
15
Reducto breaks documents into components. We can...
📝 Extract tables into HTML
📊 Parse underlying data from charts
🏞️ Summarize images with vision models
We output...
🧱 Layout based chunks for LLMs/RAG applications
🔍 Fields extracted according to your provided schema
(2/4)
1
0
15
ClaudeScholar is designed to be more resistant to hallucination by leveraging external sources of information, like PubMed, and allowing users to upload documents which are summarized and inserted into the context window at runtime. (2/5)
1
3
15
open laptop closed internet exam today
pulling up with codellama 35b on ggml 😛
1
0
15
This is exactly why I built
There's a lot devs have to juggle nowadays and limited LLM context windows/memory should not be one of them.
Has someone built a high-level abstraction on GPT with "memory" that just works (like it works in ChatGPT/web)?
As a developer, I don't want to even *think* about managing memory. I just want to pass in a session ID and the system takes care of maintaining state/memory.
35
8
103
1
2
15
1
0
13
@Siddhesh0205
I don’t think I would take that as the conclusion of this work. Especially when equipped with tools (see toolformer paper + recent gpt functions) it does fairly decent at this stuff.
1
0
14
Whether counting cars in Walmart parking lots, processing wildlife photos, sorting family albums, or parsing documents - our detectron implementation can handle your application.
The first 20 people to try the app with the coupon NOLONGDOCS get 50% off.
DMs are open!
(3/3)
4
0
14
Well that's alarming...
New Anthropic research: Measuring Model Persuasiveness
We developed a way to test how persuasive language models (LMs) are, and analyzed how persuasiveness scales across different versions of Claude.
Read our blog post here:
57
118
711
0
0
14
Prof. Solar-Lezama posted a follow up to this statement on his website this morning. I think it will help to clear up some confusion around the statement:
2
1
13
Beyond excited to be presenting at the
@nextjs
conference this month, but also a bit nerve-wracking - I'm the youngest one presenting.
It's a serious honor to be speaking alongside legends like
@rauchg
,
@jaredpalmer
,
@t3dotgg
, and more.
1
1
13
We're ready for enterprise workloads. Our async API can process thousands of pages in minutes.
We also take security seriously:
🤐 Support Air-gapped & On-Prem Deployment
⛔ Zero data retention by default on our API
🔐 SOC 2 Type II Compliance (Pending)
(3/4)
1
0
13
This is actually quite interesting. I think Anthropic is the only provider to allow forcing a prefix on the model's output. Have gotten some really powerful results with this method myself.
Claude 2.1’s 200K token context window is powerful, but requires careful prompting to use effectively.
Learn how to get Claude to recall an individual sentence across long documents with high fidelity:
40
244
1K
0
3
12
blockchain diploma is kinda sick:
6
1
13
Got a ton of DMs for this - reaching out to folks one by one to onboard to the beta, but it's just me right now so please be patient!
I just built an infinite memory system for LLM chats: Remembrall.
Add a UID to your OpenAI call and I'll manage inserting past chat history into the context window with < 100ms latency.
2 lines of code to integrate.
@helicone_ai
style monitoring for free. DM for beta access.
93
170
2K
4
0
13
@nexxeln
I miss i3 from my linux days, but yabai is no replacement
that said, I think
@rectangleapp
is a no brainer install that stays out of your way on MacOS. Free and open source too, should come preinstalled imo.
Window snapping is just too useful
1
0
11
@carmguti
extremely bearish from hearing the ceo explain what they're doing. too many buzzwords.
1
0
11
Building something new with
@tremorlabs
,
@tinybirdco
, and
@shadcn
UI - love the DX and beautiful realtime data vis
0
1
11
756 pages is crazy - legit went and double checked bc I didn't believe it
The Adobe PDF specification itself is 756 pages long. Ridiculous.
To better understand docs like this at Oloren, we built a PDF parser to break down and extract different sections of text and images.
Try it out here on our gallery:
How we use it: (1/4)
1
0
5
0
0
10
@8FNPath
@Reviewer2Ai
@mark_riedl
@iddo
Great! I have nothing against LLMs being smart or even solving MIT problems - I just have a problem with the research presented in the Drori et. al. paper.
1
0
10
Only 51% of those aged 18-29 surveyed by the Economist disagreed with the statement "the Holocaust is a myth."
If I were a Jewish person, I would be very scared right now. History is repeating itself.
1
1
10
0
0
9
supermaven + cursor combination is crazy
feels like I'm coding at 1000 wpm
0
0
7
Was asked to take down the original post sharing the email below as it contained the senders personal identifiable information.
Sharing again without PII. I am still shocked and disgusted by these MIT organizations and students voicing their support for Hamas's terrorism.
0
0
8
Overflow AI's nice but c'mon
@StackOverflow
, StackOverfit was right there!
JUST IN:
StackOverflow just announced their own AI assistant called Overflow AI 🔥
41
66
578
1
2
9
did not expect this to blow up as much as it did - i’ll admit the image is bs and i drew it in 45 seconds with
@tldraw
- the tech is real through!
2
1
9
Love to see more folks diving into the data for viral ML papers. I still think the paper is a valuable warning about changing APIs over time, but the interpretation that
@OpenAI
is nerfing the models to save $$ really cannot be inferred from this work.
We dug into a paper that’s been misinterpreted as saying GPT-4 has gotten worse. The paper shows behavior change, not capability decrease. And there's a problem with the evaluation—on 1 task, we think the authors mistook mimicry for reasoning.
w/
@sayashk
34
218
1K
0
0
9
There's few things more satisfying than a user *just getting it* on an onboarding call.
It's so cool to see someone using and loving the thing you've spent so long building 😊
0
0
9
@emollick
After analyzing the published code/data, I believe there to be some academic misconduct here. Despite claims of manual review, much of the published data is either completely invalid or leaks the solutions to the model in the prompt.
Read our report here:
1
0
9
GIving a talk on Remembrall in < 20 min at
@nextjs
conf
Join to learn how I built it! Lot of pressure to be going after
@t3dotgg
😅
Join here ⬇️⬇️⬇️
We’re open sourcing Remembrall! 🚀🚀
A few weeks ago I shared an OpenAI proxy for long-term memory and got an overwhelming response on X. After getting feedback from hundreds of beta users, we decided to open source the project:
Give the repo a star!
7
17
129
3
0
8
Made a little GPT app to help with personal meal planning - it works surprisingly well!
To get the macros right, I hooked it up with
@EdamamCo
's API.
1
0
8
Was stuck on a bug for 2 hours today bc of Slack's stupid quote formatting.
I've had enough. Announcing:
0
2
8
@wmgcbr
@JohnGal43951639
@helicone_ai
@vercel
the LLM does full CRUD on the db, so memory entries can be edited/merged over time, this improves performance by a lot
3
0
8
@jayair
Cloud formation slowness isn’t talked about enough. It’s horrendous. Terraform has been miles better in my experience. Good decision.
1
0
8