Hamel Husain
@HamelHusain
Followers
35K
Following
14K
Media
1K
Statuses
13K
AI Evals Course: https://t.co/S9ctbeyvCJ About Me: https://t.co/P6WyeKkyTa
Joined September 2012
Over the last 2 yrs, I've helped 35+ companies improve their AI products. I distilled my approach into this guide, which covers error analysis, synthetic data, eval-human alignment, involving domain experts, optimizing # of experiments & more (1/5).
19
89
546
LLM bullshit knife, to cut through bs. RAG -> Provide relevant context.Agentic -> Function calls that work.CoT -> Prompt model to think/plan.FewShot -> Add examples.PromptEng -> Someone w/good written comm skills. Prompt Optimizer -> For.
100
549
3K
If you remember our Applied LLMs course, you'll love this. Today, we are making all these resources available for free to everyone! 📚 . We did extra work to add learning tracks, resources, and notes to each lesson to maximize your learning. Link in next tweet
43
452
2K
I was doing Leetcode and found it utterly depressing so decided to go back to learning something useful
53
97
2K
"Open AI is not going to make it" @jeremyphoward called it yesterday 🤯. Really prescient. h/t @VanishingData podcast
50
259
2K
I’ve recently struggled to understand Python concurrency:. - How threads & processes work on your OS vs Python.- The role of CPUs, hardware & the GIL.- Understanding beyond rules of thumb. After tons of research, I wrote this w/the answers plus more 👇🧵.
20
386
2K
I cannot be the only person who immediately thought of this today
22
142
2K
My thesis is LLMs significantly boost your motivation to work on something. Getting a shitty prototype fast is immensely motivating . Code and time are often not the bottleneck, motivation is the bottleneck.
there is no market for way shittier versions of apps/games/software no matter how fast you and an LLM can "make" it.
69
128
2K
Google Researchers insulting and getting defensive at genuine user feedback (re:Gemini); is an epic developer relations fail. (Regardless if the feedback is harsh). Everyone is watching. I don’t want to engage or give feedback either. Doesn’t seem welcoming tbh.
48
48
2K
New post re: Devin (the AI SWE). We couldn't find many reviews of people using it for real tasks, so we went MKBHD mode and put Devin through its paces. We documented our findings here. Would love to know if others have had a different experience.
62
173
2K
I've been using this site for years. I don't know how people do regex without something like this.
28
177
1K
New businesses cannot trust @stripe . They literally holding all my money hostage and you cant get through to meaningful support. Just and endless wall of red tape. Ive never had a refund request. So mad about this.
190
61
1K
This talk by @bclavie is the highest value per second talk I have ever watched on RAG . Chapter summaries and additional links in next tweet
15
144
1K
Leetcode is completely dead to me. Absolute waste of time. Even if I wanted FAANG salary, independent consulting easily exceeds that with more autonomy.
Honestly, at this point If you give me a programming interview and don't let me use AI assistance you won't get a very realistic idea of what I'm actually capable of.
31
46
1K
Update: I'm buying a new laptop 🤣🤣.
I’m trying to upgrade my python version in my base conda env. Wish me luck.
32
28
988
I have a new favorite python library: fastcore . I spent a month contributing to it, and want to share why I think this library so wonderful:.
22
225
908
It's peak hubris when VCs argue with Yann about how transformers work.
20
39
830
Whoever wrote this article is my favorite person. I wish I knew who it was . People really need to hear this .
31
88
842
This talk from Apple is the best ad for fine tuning that probably exists. They are also using adapters (and hot swapping IIUC).
16
115
831
I often hear that evals are the most confusing part of creating LLM AI products. It's a shame b/c IMO, domain-specific evals are the most important part of an AI product!. I've written a detailed blog post with real examples on how to do this (1/3).
34
154
812
It’s well known that GitHub offers almost no features for data science. We are trying to change that. Here is a collection of tools, demos, and examples that offer new ways of automating ml/ds workflows on GitHub:. blog: . site:
12
238
786
"Your success in life will be determined largely by your ability to speak, your ability to write, and the quality of your ideas, in that order.".
10
108
796
Math and Programming textbooks that have exercises but no solutions are absolutely maddening. Why would you deprive a learner of the ability to check their work and get immediate feedback?.
36
54
760
Reading this book currently and enjoying it. I only had surface level understanding of how to make python performant before but this book goes really in depth. Highly recommend.
12
68
745
I've been AI consulting for ~ 2 years. Client: "The AI isn't working in XYZ scenario" .Me: "Can we look at a trace together?". ~70%: No traces, no logging.~20%: Log traces, but never look at them.~10%: Actively looking at data . Unbelievable alpha in looking at data.
44
57
761
RAG is another example of bloated jargon. This should just be "provide relevant context".
87
52
739
Here is how you can obtain a massive speedup with llama-v2 models, much faster than anything else I tried. It's so fast that its unreal. I made some additional notes on how to avoid a temporary foot gun as well.
23
118
720
📢 @GitHub is releasing a large dataset for natural language processing and machine learning. It's a large parallel corpus of code and natural language, with benchmarks on IR tasks. Leaderboard is hosted by @weights_biases:
6
221
699
It blows my mind that fine-tuning open models works so well. I've pointed it at all kinds of small data and it works brilliantly. On the order of 1k examples. If you haven't tried fine tuning yet drop everything and do it. It's a magical experience.
29
66
701
I can't be the only one that sees this correspondence between the past and the present.
46
68
708
RAG is dead posts are annoying as F. "R" is retrieval and "AG" is the LLM. This means you think retrieval is dead. Seriously, you think retrieval is dead? Keyword search, metadata filtering (dates, users), grep, and other filtering are retrieval. Good luck without retrieval.
71
72
717
. @sh_reya 's paper confirms what I see in practice. 1) Automated evals don't work (without semi-manual human alignment).2) Most tools don't provide this alignment.3) Automated evals add mostly noise.4) You can only write good evals by looking at data and reacting to failures
19
83
713
We need a MKBHD for AI software. Because there’s tons of bullshit out there to wade through.
46
38
678
Lots of great talks at AI Summit - these are my favorites (links with timestamps). 1. Jane Street - impressive internal dev tools with AI 2. windsurf - The demo has convinced me to give it a try and is a serious cursor competitor
9
78
692
The most common mistakes I see teams make with LLM judges:. • Too many metrics.• Complex scoring systems.• Ignoring domain experts.• Unvalidated measurements. That's why I wrote this guide, w/ detailed examples to help people avoid these issues (1/4).
19
106
676
Wow I hit limits really fast using Anthropic UI even on the Pro Plan. Is this normal?
191
8
611
If someone is consistently making high quality contributions on GitHub to your codebase. And then you believe its a good idea to put them through leetcode because “the process”, you have been brainwashed.
19
20
638
. @johnowhitaker 's talk: Napkin Math For Fine Tuning.was so popular that we ended up doing an encore!. He answers q's like:. - When should I use LoRA? Quantization? GC?.- What’s the cheapest option? most accurate?.- What hardware?.- What batch size / context length . etc?
16
129
616
I can’t begin to tell you how many founders hear the words “retrieval-augmented-generation” and think they need to hire PhDs . It’s kind of impressive that an entire industry is pulling of this heist.
28
44
594
Epic talk on RAG Basics by.@jobergum.from the LLM conf (links to YT, slides, etc in next tweet)
6
77
592
The most annoying gap in AI infra is poor rendering of notebooks on GitHub. Jupyter notebooks are the perfect microblogging medium for technical content, yet are underserved. This is why we are excited to announce nbsanity (link below). 1/7
16
58
580
I almost didn’t write the Devin post. I thought nobody would find it interesting . Yet another lesson that you should just ship it and learn from experience.
27
12
582
Revenge of the python dev. We are back baby! . As much as the typescript dev is pulling ML/data into their stack, the python devs are experimenting with the opposite arrow. Prediction: large wave of small 1-2 person companies building AI products with stuff like this.
Announcing FastHTML. A new way to create modern interactive web apps. Scales down to a 6-line python file; scales up to complex production apps. Auth, DBs, caching, styling, etc built-in & replaceable and extensible. 1-click deploy to @Railway, @vercel, @huggingface, & more.
10
26
396
This is not the right way to use AI for learning/growing as a programmer! You need to ask the AI:. 1) What are other ways we could write this code? What are the pros and cons of those approaches?. 2) Explain this code from the perspective of a <your language> programmer. What.
This is why having AI write code for you sucks. Literally just have the training wheels on forever. Great point by @dhh
33
64
576
Is it just me or is everyone sleeping on the new copilot that just dropped? It works really well in my tests. It does so much more now:. - Plan and write entire projects.- Plan and write notebooks.- Fix bugs, write tests etc. - Write complex terminal commands
20
47
547
Just re-discovered this code from @jeremyphoward how to create schemas from functions w/ 4 lines of code, and no deps except pydantic
8
41
555
This is perhaps the most significant ML product GitHub has released in its history. Meta: only ~1 FT ML person was involved from GitHub, rest was design SWE etc. Modeling work done primarily by OpenAI. Don’t underestimate transfer learning and hosted model hubs/apis.
13
74
539
o1-pro doesn’t accept pdfs or word documents (unlike 4o) . For $200, I expect more from basic UX? Like don’t make me convert to text myself?
39
7
539
Below is an explanation of "Agentic" workflows in ~ 15 LOC. At its core, "Agentic" just means LLMs that can call functions. Another ex of how unnecessary jargon confuses people.
35
62
538
I’m seeing lots of takes that RAG will become irrelevant bc of long context windows. Reasons I think this is BS. - You often want to filter by metadata (dates, # of views, etc). Search/IR is critical in many applied scenarios . - The R in RAG does a lot of work and is.
65
47
542
I learned lots of wonderful things when I worked at GitHub, but the stickiest is scripts-to-rule them-all.
4
44
497
Up until recently, there wasn't an OSS vendorless LLM eval framework I liked. Then I found Inspect, and I love it. (links below). I like it b/c.- VSCode Plugins w/Viz & UI.- Composibility & Devex.- Made by JJ Allaire (cracked eng w/a track record). the demo:
7
74
497
I get emails like this often from VCs. This one went too far. What I don't understand is. 1. Meeting w/people who don't respect their own time isn't likely to work out well for anyone. 2. Zero EQ + entitlement; not offering much value or specificity to the other party. 3. Yes,
63
9
502
Went through this prompt guide carefully Main takeaway for me -> Use markdown. 1. Use markdown tables for compact, token efficient context for ReAct, or generally when you want to enumerate things.2. It helps to delimit context with triple backticks. Why:.
2
60
494
vLLM + awq is the best solution I've found for LLM inference. - Fastest speeds competitive with everything I've seen.- An extremely portable backend that is compatible with many inference servers.- Super easy to use out of the box. The awq part isn't documented that well, but.
22
67
491
New blog post 📰: Understanding what LLM frameworks are doing is difficult (digging through code, docs, etc). I show how intercept API calls with @mitmproxy to quickly understand how popular frameworks work and find interesting results. I walk through examples with Guardrails,.
19
64
480
The biggest mistake in AI Eng hiring is focusing on dumb leetcode exercises when the most important skill is data literacy (cleaning and analyzing data). That’s why l love this talk from @BEBischof where he describes how he actively tests for this.
8
61
487
Is this the official meme for survivorship bias in data science now?. Maybe even better than the WW2 plane one?
11
59
454
Another example of loaded jargon for LLMs. This should be the poster child of that. We should only be saying the first thing. Talk in plain language.
19
42
473
Llama 3 70b function calling works pretty well out of the box with prompting only 🚀💰 . See the below demo (prompt and code in next tweet)
7
61
476
One of the least talked about subjects in AI/ML is how to clean, curate and look at data. Its the least sexiest but most important topic. Good news, @vanstriendaniel & @davidberenstei has an entire talk on this. Links in next tweet
7
56
470
Has anyone created custom-instructions that defeats GPT-4's laziness when writing code? . I thought I had succeeded at first but everything I've tried hasn't helped.
90
20
456
At this point, I'm thinking of enrolling my entire family in the course
28
22
459
Too many people are sleeping on axolotl. You can try most new techniques shortly after they come out, and get your own model. It lets you focus on the data, which is exactly what you should focus on.
16
60
446
📢 Personal announcement: I'm over the moon to share that I'll be exploring commercial opportunities around the @fastdotai ecosystem full-time 🥳. I'm excited to work with @jeremyphoward and @wasimlorgat on this, as they both are brilliant people who I love working with 😍.
33
25
446
It’s definitely worth reading this post for anyone using Claude code. TIL that the word “ultrathink” will result in maximum thinking . Lots of other great tips in here .
12
37
459
I WAS SO FRUSTRATED with the way @github renders Jupyter Notebooks, esp on mobile (You cannot horizontal scroll code, even with nbviewer) . So I made a thing: Source code is here
16
66
439
Used cursor to find this magical terminal command to download any yt video. yt-dlp -v -f mp4 --cookies-from-browser chrome "<url>".
16
20
445
I’m writing a blog post on how to create a LLM-as-a-judge that drives business value, with examples. Would love to hear what kinds of questions you have so I can consider that in my blog post.
51
23
439
If you are on a mac, these settings have given me years back on my life (make keystrokes faster). I learned this from @mussondata
13
25
421
If you use Cursor, this extension is a MUST. It allows you to save your chat history & prompts. You can even reference past conversations with @ to ground conversations in prior interactions. I've enjoyed exchanging prompts w/friends already.
9
37
430
My colleagues and I distilled practical advice re: LLMs into this three-part series. Lot's of bangers. One of my favorite excerpts from this part in the screenshot. Advice from: @eugeneyan, @BEBischof, @charles_irl, @sh_reya, @jxnlco and myself. See:
15
55
425
Maybe OpenAI ran out of training data so they decided to perturb reality as hard as possible to generate out of domain data.
8
23
414
Wrote some notes about tools for optimizing LLM latency.
12
76
416
Did you know you could create infographics with python in @ProjectJupyter ? Neither did I until @PratapVardhan created this amazing chart using Jinja and Pandas DataFrames. Really creative. See his work at @WillingCarol @betatim @jeremyphoward
8
113
404
We’ve always known that software engineering skills are key to unlocking the power of ML. Some large companies (FAANG) have gone as far as adopting a preference of hiring software engineers and teaching them ML to work on applied problems (rather than the reverse).
8
32
411
WTF is “machine teaching”.
21
38
394
This is the prompt they are using for Llama-3 function calling (seems to work well even though it's not specifically fine tuned for that):
📊Delighted to welcome Command-R-Plus, Llama-3, and and Gemini-Pro-1.5 into the Berkeley Function Calling Leaderboard. Check out how they stack up across different categories, P95 latency, and costs at Congratulations to @cohere, @AIatMeta, and
7
59
415
Lately, I've been wondering what the future of ML Eng looks like re: LLMs. - What skills to invest in? (SWE, more ML, etc?).- Is AI Eng going to supplant ML Eng for LLMs?.- What should new people do?.- etc. I have no idea! But @hugobowne has put together this panel with.
23
43
412
Super uncool opinion: For adhoc exploration of data, I think something like Tableau is far superior vs. writing a bunch of maltplotlib code.
60
18
387
I feel like you have to do this when you visit this building 😅?
10
7
389
Why is every AI product announcement gated by waitlist.
200
10
375
This is Huge! 🥳 Half of the conversations around LLMs are “Will this fit on my GPU[s]?” And this calculator makes it much easier. There are some important nuances for LoRA and I took notes on that here:.
Excited to announce a new @huggingface space to help with one of machine learning's biggest questions:. How much space does {X} model take in vRAM? And most importantly: when using `device_map="auto"`.
7
67
391
A distilled 70b-R1 is on Groq now. Kinda hidden in the docs, but its there.
9
36
391
Leaving this here, and then going to hide behind the bush before the python purists come for me.
Greatest minds of my generation making millions of copies of a constructor init method that starts like this and has approximately 157 parameters. self.model = model.self.tokenizer = tokenizer.
11
22
382
Recently, @jeremyphoward, @fastdotai and I launched fastpages. This post goes behind the scenes to explain the magic that makes it work: @github Actions. We show you additional examples of how this can give you new superpowers for ML & Data Science:
3
104
373
At first when I saw xml for Claude I was like "WTF Why XML". Now I LOVE xml so much, can't prompt without it. Never going back.
31
17
367
LOVE how the Meta team went hard on explicitly describing the prompt template.
8
29
374
Only a matter of time before the job title “AI Scientist” emerges . - Better than most AI Engineers at Evals, statistics, Data Analysis, Error Analysis, A/B testing, etc.- Better than most Software Engineers at AI Engineering . (😅 I hope we don’t need another job title ).
Data Scientist (n.): Person who is better at statistics than any software engineer and better at software engineering than any statistician.
36
34
372