Humanloop
@humanloop
Followers
9K
Following
841
Media
332
Statuses
992
Humanloop is the LLM evals platform for enterprises. Trusted by Gusto, Vanta and Duolingo to ship reliable AI products.
Joined April 2020
Humanloop is now generally available!. After 2 years of working closely with early customers, we're opening access to our full evals platform. 🧵 Here's what we've learned and how we can help you build great AI products:
17
15
111
Interact to clean your timeline:. Transformers.LLMs.Prompt engineering.CoT.Constitutional AI.Ray Kurzweil predictions.TPU pods.Attention is all you need.noam shazeer.Scaling laws.Colab Pro.ooms.GPT wrappers.Model distillation.AGI timelines.p doom.NVDA VRT TSMC.open weights.
358
286
6K
Today we're excited to announce that we're partnering with @CarperAI of Stability on bringing the first RLHF-trained GPT-3 like model to the open source community. This will be huge. Let us explain
7
68
446
Prompt Engineering is a thing. Here's Prompt Engineering 101 for working with LLMs like GPT-3.
5
47
444
GPT-4 is now available in Humanloop Playground and API!. You can start prototyping GPT-4 applications now – with the best in-class tools for LLMs.
10
63
326
Today we're excited to announce public access to Humanloop for Large Language Models!. We're making it easier than ever to build incredible products with GPT-3. Sign-up at
6
24
256
TogetherAI have brought the context window of LLaMA-2-7B to 32k tokens. This is potentially the first time an open source model with competitive performance to ChatGPT can be fine-tuned for long-context tasks. This is important for the following reasons 🧵.
9
21
175
The playground may be the first way you interact with GPT-3. but it's also an IDE in disguise. We want to transform it to be best way to build useful apps with LLMs. Here's how ↓
4
23
130
OpenAI on Azure is now generally available. Here's why you may want to use it (and also why you may not).
Just in—new AI models on Azure OpenAI Service empower your business to deliver results at scale. Explore the possibilities → #AzureOpenAI
2
18
98
There are hundreds of different use cases for Large Language Models like GPT-3. Starting today we're going to be showcasing a different awesome start-up and what they're building every day.
3
9
85
RLHF – Reinforcement Learning from Human Preferences. Models are fine tuned using RL from human feedback. They become more helpful, less harmful and they show a huge leap in performance. An RLHF model was preferred over a 100x larger base GPT-3 model.
3
15
78
BREAKING: ChatGPT API is now publicly available!. The price is *10x cheaper* than davinci. Available to use in Humanloop.
1
9
78
Our CEO @RazRazcle sat down with @rowghani of YC to discuss:. • the future of Large Language Models.• the huge opportunities for startups.• and how to build differentiated products in the AI space. Video link below ↓
6
16
74
Cohere has found that data pruning on a pre-training corpus for LLMs, can result in a performance improvement while using just 30% of the training data. Here’s why this is important 🧵
5
10
73
OpenAI is the most important company in the world right now. We sat down with @sama where he revealed their future plans. He shared the ambitious vision for ChatGPT, announced plans for a new stateful API and revealed what's held back multimodal GPT.
6
14
60
The 100k context window Claude is now available in the Humanloop Playground. Try it out, paste in a book. we're excited to see what you build with it!. Model pricing details from Anthropic below
2
5
41
Want to get started with deep learning and NLP? Start by doing. Get stuck in. We've collected the best ML & NLP resources. Here's our guide 🧵👇.
1
11
36
⚡️ Streaming the response from GPT-3 a huge UX win. Feels magical as you watch the AI respond. Now available on the Humanloop Playground and Generate endpoints so you can use it in your app.
0
4
35
Unveiling the new . Humanloop offers you the best way to rapidly build and continuously improve your NLP models. Powered by Active Learning. The best ML teams follow this approach. Here's how you can too.
1
5
33
CarperAI will be building in public, and will be releasing the data, code and weights over the coming months. Follow along in their discord. Like what happened to Stable Diffusion, we can't wait to see the innovation that happens once these models become available for all.
1
4
29
These models show far greater ability to take instruction, which has massively increased their usability. We think RLHF-tuned models will ultimately be applied to every domain and task, and these systems will unlock incredible amounts of value in the real world.
1
1
29
For more prompting guidance have a look at - we hope the tips and tricks help you on your journey of building effective LLM applications. Thank you to @prof_oz for the guest post.
2
3
29
What's the difference between an AI engineer and an ML engineer?. "The AI engineer looks after the 0-1 phase whereas the ML Engineer takes on the 1-N phase". @swyx penned the viral essay 'Rise of the AI Engineer' ( and is the founder of the @aiDotEngineer
2
5
26
OpenAI have just released text-davinci-003 . • higher quality writing.• handles more complex instructions.• better long form content generation. and now supported in @Humanloop playground!
0
1
27
This week we locked ourselves in a country home in South England. Our goal: Launch a unique AI-powered product. • 4 teams.• 48 hours.• 1 rule - everything had to be built on Humanloop. Here's the results 🧵
1
3
24
Big News: We've just added Evaluation Functions to Humanloop!. This gives you a powerful new set of tools to evaluate model performance, allowing you to build LLM applications with much greater confidence than before. Here’s why this is a big deal 🧵
2
6
25
📍PMs in AI Meetup, London 🇬🇧. Yesterday we held a Meetup in the UCL Centre for Artificial Intelligence for product managers working on AI agents and applications. Huges thanks to all who turned up (it was a full house!) and to our speakers:. • @samstphenson (Founder,
3
1
25
New Release: We’ve just added OpenAI Function Calling to Humanloop!. Function Calling has been hailed as one of the most significant OpenAI feature releases of 2023. However it has been tricky to get them into production. Here's how Humanloop helps solve this ⬇️
2
4
25
We're excited to host @ycombinator's London meet-up at @localglobevc's office this summer for future founders to network and learn about startups. We'll be telling the story of starting Humanloop. If you're in London, sign up through @startupschool at
1
7
23
You can now talk to GPT-4 in the Humanloop discord!. The OpenAI live demo inspired us, so we used GPT-4 to create a GPT-4 bot!. With the help of GPT-4 it only took @jordnb about 20 minutes to code this from scratch!
0
4
21
All OpenAI models are down right now. Follow here for updates . (Anthropic, Cohere, AI21 are all running well. An excellent time to toggle on a backup deployment in Humanloop)
1
2
21
Stay tuned and follow us for Prompt Engineering 201 soon 👀. We'll cover chaining, prompt injection, DSLs, hallucination protections and more!.
1
0
21
Want to understand how Humanloop can help you build differentiated products with LLMs?. Join a live demo with @RazRazcle ↓. Today, 6pm GMT / 10am PT March 7th, 5pm GMT / 9am PT March 9th, 6pm GMT / 10am PT
3
2
19
1. Just ask. With the advent of instruction-tuned models, these models are usable without needing to get clever about it. If in doubt, just ask.
1
0
19
Human feedback is critical to aligning these models to do as you want. We've built the specialised tools needed for this crucial component and are delighted to be contributing to this open-source effort.
1
1
15
LLMs are the next big computing platform. At least as important as the internet, perhaps more. We want to empower the next million developers to build AI-first applications. If you want to build successful products and apps with GPT-3, sign up now.
0
0
17
One of the most valuable talks from at @OpenAI DevDay was given by @john__allard and @colintjarvis. Yet many seemed to have missed it. In this blog, @jordnb breaks down its key insights to show how you can start Maximizing the Performance of LLMs.
0
3
18
Sam Altman (@sama) of OpenAI will be doing a fireside chat chaired by our CSO @davidobarber in London. 24th May 2023 1.30-4.15pm. Apply here to join!.
1
5
17
Like any good @ycombinator company, we @humanloop are always trying to listen to our customers and build what they want. Here are 5 recent feature releases driven directly by customer requests and feedback:. 1) Multi-provider chat-focused playground
2
4
16
Building an LLM app has evolved significantly since 2022. If you're working on an AI project, you should familiarize yourself with the anatomy of a modern LLM application. Here's a quick overview:. • LLM model - the core reasoning engine; an API into @OpenAI, @AnthropicAI,
0
2
17
Big product update: We’ve introduced support for multi-modal models!. You can now input images and text to models like GPT-4V in our Editor or API. This presents a new frontier in the range of apps that can now be built with AI. Here’s what GPT-4V looks like in our Editor ⬇️
2
1
15
2. Few shot learning. For extra clarity, provide examples. This is particularly useful if you want an uncommon tone, style or syntax. If accuracy is paramount, stuff the whole context window with examples.
1
0
15
@fastdotai @AndrewYNg @goodfellow_ian @mpd37 @AnalogAldo @ChengSoonOng Check out CS224n from Stanford (best NLP course we know!) The videos and homeworks are online. The course evolved from OGs in NLP: @chrmanning and @RichardSocher. A good amount of background while focusing on the practical and relevant research today.
1
2
15
GPT-3 lets you build apps that feel like science fiction. We now have assistants that can help us. • write better (@writesonic, @peppertype_ai, @sudowrite).• code better (@GitHubCopilot).• imagine better (#dalle, #midjourney).
2
0
15
🚨 Announcement 🚨. Just a few hours ago @AnthropicAI dropped Claude 3 - a new suite of models which outperform GPT-4 and Gemini Ultra. We're excited to announce that Claude 3 is now available on Humanloop. Bring your API Key to test, evaluate and deploy
1
3
16
Really pleased to have been covered in TechCrunch today as one of the startups leading a new wave in NLP!.
1
3
16
Fine tuned models at scale. Instead of a 4-5x higher per-token fee, you pay an hourly rate for hosting fine tuned models. This gets economical at scale. And at scale you'll definitely want to be fine tuning. (though wow at the jump for hosting Davinci vs Curie)
5
4
16
Delighted to be at #AISummit London. Come talk the team about Human in the Loop systems for NLP . @humanloop #AISummit #MLOps
1
3
15
. and now introducing Obsidian AI 🪄 Write faster. Think bigger. Augment your creativity. →
Introducing Notion AI 🪄 Write faster. Think bigger. Augment your creativity.
1
2
15
Will Larger Context Windows Kill Retrieval Augmented Generation?. As model providers continue to extend their context windows, researchers have been looking into whether or not this enhances performance over Retrieval Augmented Generation (RAG).
2
4
15
This is likely the first of many advancements we will see as #opensource continues to level up!. You can check out the model now:
0
2
15
Every ML engineer should consider programmatic labelling. Here's why:.
1
1
15
Ever wanted to build a custom ChatGPT?. Here's a guide to build it with Humanloop, powered by @nextjs, @openai / @AnthropicAI in 15 minutes.
2
2
15
Here’s how you connect an LLM to tools to give it extra capabilities. For example, include google search in your context window with simple syntax like {{ google(query) }}. Read more here:
1
3
15
The HELM benchmark from Stanford is a fantastic resource to understand the relative strengths of large language models. Here’s the overall ranking on the 16 core tests with the win-rate over the other models.
1
2
15
Our feature with @EO__Global is now live on YouTube 🚀. Watch @jordnb and @RazRazcle talk about their journey to co-founding Humanloop, building an MVP at @ycombinator and their advice for AI startups.
1
2
14
If you prefer the TL;DR, we've distilled @jxnlco's 8 core principles for building scalable AI products on the Humanloop blog:
1
3
14
In @siliconrepublic today!. Our CTO Peter Hayes chats to @VishGain about Humanloop's London origins, getting into @ycombinator expanding to San Francisco and how we help enterprises confidently build with AI 🚀.
0
5
14
Subtle update you'll have missed:. Finetuned gpt-3.5-turbo is now only 3x of the cost of the base 3.5-turbo. It was 8x before. Finetuning just got even more compelling.
0
0
13
Higher Rate limit on code. 1200 RPM on the codex models compared to just. 20 with OpenAI. (although still not the full 3k per minute for davinci-003)
1
1
14
GPT-4 usage is free but will have rate limits while in this beta period. Access to Humanloop is available as a free trial. Sign up here →
1
1
11
In fact there is a new job title emerging for generalist engineers who have a strong familiarity with LLMs and AI tooling: "The AI Engineer" @swyx. Despite the claims that 'AI is coming for the jobs of software engineers first’, so far it has only created a greater need for them.
1
1
12
SO excited to share what we've been working on!. We're building the best tool for programmatic labelling. It's a super powerful way to build your datasets for NLP and it's getting great feedback from early users. Sign up for access 👉
@HumanloopML I think it's hard to justify NOT using this as a jumping off point for most NLP projects.
1
4
13
but going from a cool demo ( to a fully-fledged production app is still too hard. • hard to evaluate.• hard to make reliable (hallucinations, repetition).• hard to get real domain knowledge into these models.
1
0
13
Join the Carper discord to follow (or get involved!) Full details of the announcement here
1
1
10
New version of Programmatic dropping today! 🎉 . You can now add ground truth labels directly in the UI! 🤯 . Programmatic uses these to give you instantaneous feedback on precision and recall. 👉 Check it out:
0
1
13
The Allen Institute for AI (@allen_ai) just released a 3T token dataset to support open training of large language models. This is the largest #opensource dataset released this year, with 50% more tokens than Llama2’s training data. Here’s what you need to know 🧵.
5
5
13
3. Plan for your instructions to be taken very literally. These models have been trained to predict the most likely sequences of words. Protect around simplistic pattern matching, and be careful what you wish for.
3
0
12
AI Tinkerers London!. Talks from:.@RazRazcle - projects built on Humanloop.@mysticdotai - fine-tuning #opensource models.@meetcleo - chatting with your bank account.and cool demos from other tinkerers. Congrats @LouisKnightWebb and @bloopdotai on a great event!
1
0
13
Thanks @nasscom or including Humanloop in their Generative AI Landscape Map!. We're building out tooling to support the next generation of applications that will be built on top of large language models. It's great to be recognised here alongside so many amazing companies in AI!
0
4
12
Happy Tuesday: Today we're launching support for @Meta's Llama 2 in the Humanloop Playground!. You can now go to and start testing out the model right away! No need for an API Key or any other configuration. Here's what this means ⬇️
1
2
12
Love machine learning?. Join a YC-backed startup that bridges the gap between people and machine learning, allowing people to truly understand and control their models. ➡ #machinelearning #activelearning #python #PyTorch . Retweets appreciated! 😃.
0
10
11
Had a great discussion of LLM best practices with the @localglobevc portfolio this morning. Thanks for hosting!. Great to share and learn with @bloopdotai, @OrbitalWitness, @SigmaOS and others!
1
2
12
Humanloop takes the guesswork out of prompt engineering and model development with. • Feedback at scale.• Experiment tracking.• One click fine-tuning. all brought together with a simple SDK.
1
0
11
Watch the interview to learn how to build with the huge potential of LLMs and let us know how you're building with AI.
1
1
11
If you're interested in a better way to build apps on top of LLMs:. 1. We're hiring 2. Sign up at
1
0
11
Want #machinelearning in #Zapier? . Humanloop can get to the same level of accuracy as traditional supervised learning with ✨10% ✨ of the labelling requirements. ➡
0
6
10
1. Feedback. Eyeballing a few examples isn't enough for production applications. So, Humanloop lets you collect end-user feedback at scale, unlocking actionable insights on how to improve your models. Discover the issues that you are missing!
1
0
10
One consistent piece of advice that has comes on almost every episode of the High Agency Podcast. • Start building today❗. @sourcegraph is one of the world's most popular AI coding tools. Their CTO and Cofounder @beyang shares his thoughts on this ⬇️
2
1
10
We're excited to be working with @CarperAI, the biggest group in the open-source RLHF space! Their recently released TRLX library makes it easy to adapt LLMs from human feedback. Check it out!.
0
1
9
3. Finetune. Prompts can only take you so far. Finetune GPT-3 with your data to get a custom model at lower latency and lower cost. With Humanloop, you can do it with a single click and no code.
2
0
8
Can @AnthropicAI's Claude 2 model beat OpenAI's GPT-4?!.It has:.- Access to data up to 2023.- 100,000 token context window.- Improved performance in coding, math and reasoning. We've added Claude 2 to Humanloop's Playground so you can compare them side-by-side and see for
2
1
10
(we spun this up at a hackathon last week, using Humanloop as the quickest way to go from idea → prompt template → deployed GPT-3). Install the plugin here to supercharge Obsidian with AI.
0
1
10
AI tool of the week: YC Idea Matcher. It's an open-source semantic search engine for matching ideas with @ycombinator companies!. Just give it an idea and it links you to similar YC backed startups.
3
1
10
An excellent summary of the history and recent progress in NLP by @chrmanning:. Strikingly he dates the present era of NLP as starting in 2018! We are just at the start of what's possible.
0
8
9
Llama 3 on @groqinc vs GPT-4. "Name every country in the United Nations". The difference in speed is no joke ⬇️
0
1
10