Defog.ai (YC W23)
@defogdata
Followers
3,122
Following
13
Media
94
Statuses
276
Deploy fine-tuned LLMs for enterprise analytics. Powered by SQLCoder
San Francisco
Joined November 2021
Don't wanna be here?
Send us removal request.
Explore trending content on Musk Viewer
WIN AT BANGKOK 3024
• 188739 Tweets
#บางกอกคณิกาep1
• 185850 Tweets
Grok
• 131987 Tweets
#WeAreSeriesEP6
• 124390 Tweets
Pablo Marçal
• 120586 Tweets
어버이날
• 67553 Tweets
PondPhuwin WeAre EP6
• 52905 Tweets
Le 8
• 49128 Tweets
Alliés
• 46861 Tweets
#天音かなた爆誕2024
• 43500 Tweets
エンプレス杯
• 29023 Tweets
新シーズン
• 29015 Tweets
#SamPitroda
• 28854 Tweets
Natalie Elphicke
• 28733 Tweets
THE A'TIN OF SB19
• 26223 Tweets
自動車税
• 20016 Tweets
ネズミ混入
• 19119 Tweets
味園ビル
• 17257 Tweets
GOSE
• 17062 Tweets
みどりの窓口
• 16229 Tweets
ダンバイン
• 16071 Tweets
オーサムリザルト
• 12785 Tweets
B'zのGet Wild
• 12341 Tweets
Pinned Tweet
And here it is! SQLCoder-70B outperforms all other models in the market for Postgres text-to-SQL generation by a wide margin.
We got these results by fine-tuning
@AIatMeta
CodeLlama-70B, which was released yesterday, on less than 20,000 hand-curated prompt-completion pairs.
You…
3
4
36
We are open-sourcing SQLCoder, state of the art model for converting natural language questions to SQL queries!
SQLCoder outperforms OpenAI’s gpt-3.5-turbo, and significantly outperforms all major open-source models for generic SQL schemas in Postgres.…
10
43
287
Smaller models are better, cheaper, and faster than large behemoths.
A finetuned SQLCoder-7B costs you just $300/mo to host on AWS, and handles up to 1600 requests/hour on a single machine
At the same scale, GPT-4 costs $18,000/mo
8
34
284
We are thrilled to release SQLCoder-34B today! The 34B model is our largest and most capable model yet.
Even when not fine-tuned, it outperforms GPT-4 turbo for text to SQL conversion on our open-source SQLEval benchmark. With fine-tuning, we have seen SQLCoder-34B reach near…
12
41
255
We did a crazy thing over the last month and built a feature every day!
8
5
123
We are excited to be a part of
@ycombinator
's W23 batch!
@defogdata
is a ChatGPT for data, embedded in your app. Add our insights widget in your app, and let users query data with natural language.
Try our product and sign up here:
4
13
111
You can now run Defog on a consumer-grade Macbook, thanks to the amazing Ollama project (created by
@jmorgan
and
@mchiang0610
)
Ollama is incredible. The video is Defog running on a 16GB RAM Macbook Air, alongside 20 other applications (including a screen recorder). Runs about 3x…
2
5
54
We are excited to announce Defog’s $2.2M funding round to develop open-source LLMs for data analysis.
The round was led by Script Capital (
@ajhodls
) and
@ycombinator
, with participation from Hike Ventures, Pioneer Fund, and notable angels.
Since launch,…
4
7
49
Defog now support multi-lingual queries!
We received many feature requests after our launch yesterday, and will implement them as soon as we can.
We like to move fast. Look out for a new feature added every day for the next month 🛠️
2
3
40
SQLCoder-34b is on Huggingface trending, along with some names that we really look up to! 📈
2
4
36
Defog now has (rudimentary) reasoning abilities!
You can now ask broad questions, like whether higher prices lead to lower sales, and get human-interpretable answers!
This is Day 9 of our daily product pushes. We like moving fast, and we are just getting started 🛠️
2
1
25
We just enabled self-adaptive learning! When Defog gets a query wrong, it now debug itself automatically.
In this example, the initial query had an edge case for a divide by 0 error. Defog saw the error and fixed it - like a human would.
Day 13/30 in our feature-a-day streak!
0
1
23
In the 1970s, Edgar Codd, inventor of the relational database, had a vision. He wanted to build a natural language interface that would allow a user with limited knowledge of a database to query it.
He called it Rendezvous.
1
3
22
We are excited to launch Defog Agents today! These are AI assistants that can handle complex and repetitive workflows in SQL, Python, and R – with human-in-the-loop oversight from data scientists.
Agents allows data teams to focus on high-value tasks, rather than time spent on…
2
5
21
@svpino
We have built exactly this! Much more (causative analysis, data enrichment, statistical modeling) coming soon 🛠️
1
1
19
We're stoked to be onboarding a bunch of new Pro customers this week! Here's how some of them are using Defog:
🧐 A governed analytics platform is letting governments and universities get quick insights from shared data
🛍️ A marketing analytics company is letting e-commerce…
1
2
21
We are thrilled to share Defog's latest capability – statistical modeling! 👨🏽🔬
Until now, Defog could only generate SQL queries to answer data questions. Now, it can also create Python code and run regression models 📷
We can't wait for you to try this!
3
1
19
Defog now works within Slack as a slash command!
Was requested by so many of our users. Microsoft Teams version coming soon.
This was Day 24/30 of our daily product releases. It's been really fun to 🚢 fast. Tons more integrations coming this week!
0
0
18
Our models work with both structured and unstructured data, and can query everything from data warehouses to call transcripts.
We do this with users' privacy in mind. Our model only sees your database schema, and never reads actual data about your customers.
0
1
17
Announcing Defog's MedSQL! 🩺
The MedSQL model lets you query US healthcare billing data in the ICD-10, CPT and HCPCS codes in natural language.
Like the rest of Defog, this works in a completely privacy friendly way – your data never ever leaves your database. 🔐
1
2
17
Stoked to have
@manshar_
join our team! Manas works on our reusable React components and builds the visual interface for Defog's end-users.
1
0
17
We just got our no-code integration up! 🎐
Previously, you had to give us your credentials via an API to let us integrate with your database. Now you can do this without any code at all.
This was day 16/30 of our feature-a-day streak. More coming soon!
0
1
15
We just created AI agents can do analyses and make reports! 🦾
We weren't able to make a "proper" demo because of the need to redact personal data, but this is what a typical report looks like!
We're super excited about what you'll do with this 🚀
0
1
15
We're *finally* launching a free plan 🥳
You'll be able to integrate Defog with a db and run queries via our API and microservices. So far, these have only been available on our paid Pro plan.
We're making it much easier to explore and iterate with Defog. Share your details in…
1
1
15
We did (another) crazy month of releasing a feature every day in May 🤓
⚡️Highlights include:
Our free API plan
Setup in <5 min
New chat UI with visualisations
Model feedback with Guided Teaching mode
1
2
14
New swag in the house! Launching something soon.
1
0
14
Self-hosted models are finally here! 🚀 You can now deploy Defog on-prem, using models fine-tuned on your schema!
This is available to all enterprise customers, starting today. This has been a highly requested enterprise feature, and we can't wait to bring it to more users! 🛠️
2
2
13
Defog now supports MongoDB!
Many of you told us that Mongo is your preferred database. So we implemented this as soon as we could! Python and Node libraries coming soon.
This was Day 10/30 of our feature-a-day streak. We like moving fast - more coming soon 🛠️
0
0
13
Defog is now SOC 2 Type II compliant! We are excited to be providing enterprise-grade security and privacy controls for our customers.
This enables us to serve the needs of global enterprises with the most stringent requirements, utilizing our zero-trust architecture.
0
2
13
Just pushed a major security feature - hard filters 🔐
Hard filters make queries more secure against prompt injections. With them, Defog only return the data that you would want a user to access.
This was day 8/30 of our daily feature pushes for a month. More coming soon! 🛠️
0
0
12
Great to see a review of SQLCoder already up on YouTube!
Happy to see SQLCoder do well on the new questions the reviewer asked!
1
3
11
Defog's unstructured data queries now include citations!
When you ask Defog questions about a list of academic papers or Notion pages, it now gives you the exact sources it is synthesizing from.
This was Day 28/30 of our daily feature releases! More coming in the next 2 days 🛠️
1
2
11
We just enabled self-serve Postgres integrations! Early users can now ask questions of their DB without talking to us, and change parameters anytime they want to
This is the 6th day of our 30 day streak to push a feature every day. Plenty more coming soon 🛠️
0
1
11
Our python library is out! Install it with `pip install defog`. Usage instructions here:
This will help our early users integrate Defog in their apps more easily!
This was day 7/30 of our daily feature pushes for a month. More coming soon! 🛠️
0
1
11
We will push a new feature update every single day of May! 🛠️
Day 1/31: Defog's chat interface is now installable with `npm i defog-components`
We introduced this as an enterprise feature in March, and it's now available to all customers! Much more coming your way this month🚀
1
1
11
Day 3/31: Reinforcement Learning with Human Feedback (RLHF) 🛠️
v0.4 of defog-components has a debug mode, where you can look at the queries behind the results + give feedback on them.
This helps the model get smarter much more quickly, and gives better results over time 🧠
0
1
11
We just released a a guided teaching model for RLHF on Defog! 👨🦯
Instead of just marking a query as good or bad, you can teach Defog exactly what about it is good or bad – leading to much faster learning and improved performance!
Day 25/31 of daily releases – more this week🛠️
1
0
11
Triple feature Saturday! We shipped 3 features this rainy SF day 🌧️ These will be rolled out to all users next week.
1. 🤷♂️ "I don't know" answers – if Defog can't reasonably answer a question, it will now let you know instead of hallucinating a response
1
2
11
Defog is now available on node! Just run `npm i defog` to start using us from your Node.js apps 🍃
This is day 14/30 in our feature-a-day streak! Some really exciting (and visual) ones coming soon 😉
1
2
10
New feature: auto-suggested metadata for SQL tables! When you add tables, we now auto-suggest metadata for you 🪄
Most users found our onboarding process a tad tedious. This should help!
We are continuing to push a new feature everyday. More coming over the next 3 weeks 🛠️
0
0
10
Defog now integrates with
@NotionHQ
out of the box! 🎉
Just paste your Defog link in Notion, and query your database, data warehouse, or text dump from within Notion.
This was Day 22/30 in our feature a day streak! More integrations coming in the next week 🛠️
0
0
10
Defog now integrates with
@awscloud
Redshift!
You can talk to your Redshift cluster in natural language. We can generate queries with complicated joins & aggregations super easily🍃
This was Day 21/30 of our feature-a-day streak. More features and integrations coming soon!
2
1
10
Everything we released in Week 1 of our Launch Month in <1 min!
🤖 Contextual chat interface
📈 Text-to-chart
🧠 Debug mode
🏠 Self-hosted, on-prem models
🪄 CLI integration wizard
🔑 Index and foreign key recognition
✅ Database suitability checker
2
2
9
A case for when to not use an LLM to write.
@PySamarth
is a journalist and editor. He's been thinking about what LLMs mean for the craft of writing and landed on some rules for himself.
The bottom line is that if you want to be a writer, share your ideas or tell your story,…
0
2
9
Day 2/31 of our daily releases: auto-visualizations 📊
If you're using our chat component, you can now ask Defog to create charts just by asking questions in plain English🪄
This was requested by most of our users. We are so glad to finally release it.
More coming this month!
0
2
9
Defog just got a major engine upgrade 🚀
We completely rewrote our engine, and are seeing some incredibly powerful capabilities now – with much better results for most database schemas
Free for anyone to try - just upload a well formed CSV on our website 📊
0
1
9
👋 Say hello to Wendy! She is working on quality assurance of our LLMs
She explores the wild side of models, pushing them to their limits and evaluating their performance.
1
1
9
Over the last month,
@PySamarth
has ben running experiments to see if LLMs can do unbiased data analysis.
He tackled a thorny dataset with an LLM – a list of people by their IQ and gender. There are some great tips for unbiasing your models in here 👇🏼
0
2
8
The epic saga of enterprise LLM adoption
'What sorcery is this?' ➡️ 'We need those model weights stat!' ➡️ 'King of the Server Room'
0
1
8
Defog now integrates with Google Sheets data! This is now available in beta to all early users that requested it.
We like moving fast. We will continue to release a new feature every day for the next 28 days 🛠️
2
0
8
Defog now integrates with
@GoogleCloud
BigQuery! Many of you told us that BQ is your preferred data warehouse. Early users can now query their BigQuery data with natural language
This is feature 11/30 of our feature-a-day streak. More coming very soon – we like moving fast 🛠️
0
0
8
We just implemented a database suitability checker!
You can now check if defog will be a good fit for your database, all from our python library (0.27 and above) 🪄
This was day 7/31 of our daily feature releases. More coming soon!
0
1
8
Day 9/31 of daily feature pushes: we just implemented data insights!
Now, you can get insightful narrative insights about your data, instead of just charts and tables. To use this, just run `npm update defog-components`!
0
1
8
Defog now integrates with
@Azure
SQL+Synapse! With this, we support all major DBs + warehouses 🤝
This was Day 30/30 of our daily releases. It has been incredible to build based on your feedback. We will now shift to weekly releases, with a focus on performance and reliability🛠️
0
0
7
Hindi support is finally here for Defog! 🛠️
Just add language="Hindi" to your run_query call, and you can start using it straight away.
This was Day 21/31 of our daily releases for this month. Our most ambitious work is coming next week!
0
1
7
We just made it easier for you to integrate with Defog!
Your metadata now automatically populates in a Google sheet that you can edit with colleagues. Once done, just upload to Defog via our Python library 🍃
This is Day 12/30 of our feature-a-day streak. More coming soon 🛠️
0
0
7
We just added memoryful querying! With this, Defog remembers the previous questions you asked when generating responses 🎉
This was requested by many users, and makes the experience similar to ChatGPT!
This was Day 19/30 of our feature-a-day streak. Much more coming soon 🛠️
0
0
7
We just introduced a personality field to Defog 🧪
You can now make interactions more engaging by adding personality to your AI assistant, right from the react component!
This is a beta feature available to select users. GA coming soon! 🛠️
0
0
6
Defog's chat mode is now available for all free users via CSV uploads! 🆓
This was Day 15/31 of our daily releases for this month. Feedback very much appreciated!
0
0
6
Defog now supports large databases on the Pro plan! 🎉
Previously, you had to be an enterprise customer if you had more than 15 tables in your database. Now, you can do this on the Pro plan as an add-on 🙌
This was Day 14/31 in our daily releases. More coming next week!
0
1
6
@knowrohit07
We did use a bit of WikiSQL, but mostly had to hand-create a lot of the data. More on that here! The dataset we used for the 34B model is similar.
1
1
6
New feature! You can now fine tune our AI by defining custom terms for your use-case!
Unflashy, but highly requested. Works in any language, and for any domain.
We are still pushing a new feature everyday, and will continue to do so for the next 26 days. More coming soon!🛠️
0
0
6
Defog now supports auto visualization (in beta)! If you ask to plot as a query, Defog can handle that straight away! 📊
This was a really fun feature to build, and was day 18/30 of our one-feature-a-day streak. 2 really fun ones coming up in the next 2 days 🛠️
0
0
6
Week 3 of our launch month was a big one! Here's everything we released 👇
🆓 Free tier upgrades
🇰🇷 Korean
🩺 MedSQL Model
🧪 Pilot Plan
🇧🇷 Portuguese
😎 Personality
🇮🇳 Hindi
2
1
6
Defog is now available in Mandarin! 🇨🇳
Your users can now ask questions in Hanyu Pinyin, and get answers straight away.
This was Day 26/31 of our daily feature releases this month. With this release, we are now support the world's 5 most spoken languages!
0
0
6
New Defog feature – a memoryful interface with query explainability built in! 🛠
Currently available to all enterprise customers, coming to all plans (including free plans) next week 🍃
1
2
6
Defog now works with your chatbots! 🛠️
If you run a Whatsapp or Telegram bot, you can now integrate with Defog and let users ask data questions – instead of bothering your engineers!
This was Day 25/30 of our daily feature updates. More integrations coming this week 👷
0
1
6
Defog now supports websockets! 🔌
If you have long running queries, you can show status updates to users as you wait for the request to finish 🍃
More here: . Props to
@manshar_
for releasing this 🙌🏼
0
0
5
Defog now enables multitable joins! This opens the doors for us to to support causal reasoning and statistical modelling
This is the 5th day of our 30 day streak to push a feature every day. Plenty more features coming up in the next month! 🛠️
1
0
5
Day 11 of our daily updates: better docs! 😅
Our features had outpaced the state of our docs in the last 2 weeks. So we took a break from shipping features and got our docs up to speed today.
Main docs:
Python docs:
React docs:…
0
0
5
We just got brand new docs! 📃 Set up your free defog account in <5 minutes
0
1
5
2. 🔎 Query explainability. Defog can now explain what a generated query does to non-technical users.
Users can understand in plain language what Defog is doing under the hood, instead of just looking at code and trying to figure out what it means.
1
0
5
Defog now supports Japanese! 🇯🇵
Japanese language queries and query explanations have been long requested – we are stoked to finally be releasing them 🗾
This was Day 12/31 of our daily releases. More languages and features up next!
0
0
5
Defog now supports Portuguese! 🇧🇷 🇵🇹
This was requested by many of our Brazilian users – we are excited to see what you will do with this.
Portuguese support was Day 19/31 of our daily updates for this month. More coming your way soon! 🛠️
0
0
5
Defog now supports Arabic! 🇸🇦 🇪🇬 🇦🇪 🇶🇦
This was our last language release for this month, and we now support 80% of the world's population in their native language 🌏
This was Day 28/31 of our daily releases this month. We will close out with 3 big announcements next week!
0
1
5
Bugs are annoying. We *really* appreciate it when you report them! Your reports help us make Defog better.
We fix bugs as soon as we can. If you report a bug, we will drop everything else we are doing and fix it. This one took us about 20 minutes to fix!
@defogdata
@svpino
Looks great, but no matter my input, i keep getting "Error: replace() argument 2 must be str, not None". It also does not show any column descriptions. Perhaps due size constraints, other than 1MB? (my file is less than 47kb).
1
0
1
0
0
5
📣 Announcing our new Pilot Plan
We're excited to launch Defog’s Pilot Program to let you try an AI data assistant in your app, without an investment in a full implementation.
0
0
5
Defog now supports column names and metadata in non-English languages, and we also enabled support for Korean! 🇰🇷
This was Day 16/31 of our daily updates for this month. More coming soon! 🛠️
0
0
4
Day 6 of our daily releases: automated recognition of foreign keys and indexes in your database! 🛠️
When this, integration becomes a lot easier. Just update to `pip install --upgrade defog`, and you'll be good to go!
Props to
@JP_smasher
for building this!
0
0
5
Defog now supports querying unstructured data, in addition to structured data! 🔎
We can now query your notes in Notion/Roam, Slack messages, Google Drive and more. This is an example for
@rishdotblog
's book notes
This was Day 20/30 of our daily shipping streak. More soon! 🛠️
0
0
5
Defog's React components are live! 🎉
You can now visualize JSON from Defog's servers in your front-end applications. Detailed docs coming soon
This was Day 17/30 in our feature-a-day streak. Your feedback help us 🚢 much faster. Please keep it coming!
0
0
5
Sometimes “I don’t know” is just the best answer!
Defog is really good at telling you when it can’t reasonably answer a question from the metadata you’ve given it. Instead of inventing an answer, Defog just tells you “I don’t know”.
This was tricky to build, but incredibly…
1
1
5
Defog now has a rich chart editor, and dark mode! 📊
Early users told us how important dark mode and embeddable chart editing was fro them. So we made it a priority and implemented it! 🛠️
Props to
@manshar_
and
@studiomesmer
for their incredible effort in pushing this!
0
0
4
Everything we released in week✌️of our launch month in 30 sec!
💬 User-level glossaries
📊 Data narratives
🇮🇩 Bahasa Indonesia
📋 Updated docs
🇯🇵 Japanese
🔒 SSO for Enterprise
💪🏽 Large databases
Lots more coming up in the next two weeks!
1
0
4
We shipped 2 things this Sunday! 🛠️
1. A 2-click way to create microservices for calling Defog's servers, w/o exposing your database to us:
2. Our docs (finally):
This was Day 23/30 of our feature a day streak. More next week! 🎉
0
0
4
Defog now integrates with Snowflake! 🎉 You can query your Snowflake schemas in plain English (or Español, عربى, 普通话, or हिंदी)
This was day 26/30 of our feature-a-day release streak. It has been a joy to build based on your feedback. More coming in the next 4 days 🛠️
0
0
4
Defog now has tabbed visualizations, thanks to
@manshar_
! 📈
With v0.15, you can switch between table and chart views – instead of being stuck with one or the other.
This was Day 23/31 in our daily feature releases for this month. More coming over the next week! 🛠️
0
0
4
Day 30/30: Azure and SQL Server integration
Defog now integrates with
@Azure
SQL+Synapse! With this, we support all major DBs + warehouses 🤝
This was Day 30/30 of our daily releases. It has been incredible to build based on your feedback. We will now shift to weekly releases, with a focus on performance and reliability🛠️
0
0
7
2
1
4
Defog's chat-mode is starting support for more languages – starting with Bahasa Indonesia! 🇮🇩
Previously, we were able to generate the right query for any language, but much of the chat experience would be in English. We will start expanding into other languages this month. 🛠️
0
1
4
@DurgeshRathod3
Hi there. We have decided to not open-source our pruning algorithm just yet, and it's only available on our enterprise plans
But writing your own should be fairly straightforward – just use any RAG workflow for your schema
0
1
4
Day 5/31 of our daily updates: a CLI integration wizard! 🧙♂️
This CLI wizard cuts down integration times by 10x. You can now integrate with defog in <1 min. Just run `pip install --upgrade defog`, then `defog init`
Props to
@JP_smasher
for building this. More coming soon! 🛠️
0
0
4
What's new:
⚙️ Integrate with all major databases and data warehouse via our microservices + libraries
❓ Handle complex queries across multiple tables and ask questions on unstructured data like text.
📊 Ask Defog questions via Notion, chatbots and web apps
1
0
3
We're super grateful to all our early users for taking the time to try us out, get on calls and write to us with your use cases and feedback! It helped us prioritise the right features and build super quick.
Stay tuned as we'll soon open up Defog to all users!
0
0
3
We just launched role-specific data definitions! You can now specify instructions by user types on Defog 🛠️
As an example, you can have different hints for "top priorities" for different user types on your app now 🎯
This was day 8/31 of our daily releases, more soon!
0
0
3
This week, we launched 4 updates to make integrations breezy and queries reliable!
🧑🏻🍳 A cookbook for better metadata descriptions
We just published best practices for giving Defog metadata about your database. If you’re a registered user, you will get it in your inbox soon!…
0
0
3
Day 24/30: Slack integration!
Defog now works within Slack as a slash command!
Was requested by so many of our users. Microsoft Teams version coming soon.
This was Day 24/30 of our daily product releases. It's been really fun to 🚢 fast. Tons more integrations coming this week!
0
0
18
1
0
3
Outside of work, Manas is learning carpentry!
After a whole day in carpentry trying to understand hinges...
Some css later – "wow what I was going to make was literally impossible"
A redo awaits ;-;
2
0
5
0
0
3