13 ms·
Vanna.ai: Chat with your SQL database
- arter4 3y agoI'm curious about how this performs with more complex queries, like joins across five tables. Also, does the training phase actually involve writing SELECT queries by hand? In the age of ORMs and so on, many people have probably forgotten how to write raw SQL queries.
- nkozyra 3y agoFrom my experience, GPT-4 will do just as well with joins as without. And that needs no specific, separate SQL training (which I assume tens of thousands of examples are already in).
- teaearlgraycold 3y ago> In the age of ORMs and so on, many people have probably forgotten how to write raw SQL queries. I’ve heard this general sentiment repeated quite a lot - mostly by people that don’t use ORMs. In my experience pretty quickly you reach the limits of even the best ORMs and need to write some queries by hand. And these tend to be the relatively complicated queries. You need to know about all of the different join types, coalescing, having clauses, multiple joins to the same table with where filters, etc. Not that this makes you a SQL expert but you can’t get too far if you don’t know SQL.
- darylteo 3y agoORM abuse are absolutely rife in small-scale/volume build industries i.e. web agencies, outsourced crews 8/10 projects I look into don't have any indexes set up. Use of ORMs with little thought into lazily loaded relations lead to 100s of queries being done per request. It's pretty mad. Do not underestimate the propensity of a developer to stick to the only tool they know how to use. Unfortunately ORMs like Eloquent make it way too easy.
- teaearlgraycold 3y ago> small-scale/volume build industries i.e. web agencies, outsourced crews Well that could explain it. I’ve only worked in companies where everyone working on the app codes with the expectation that they could be dealing with their mistakes for years.
- zainhoda 3y agoAuthor of the package here. Joining 5 tables is not a problem. The training does not necessarily require you to write the queries by hand. A trick that we've seen people do is to just train with DDL statements and then ask "leading" questions if it can't answer on the first try. I've been using the package myself for about 6 months and while I haven't forgotten SQL, what I have forgotten are the fully qualified table names and which tables live in which schemas etc since I never have to think about that.
- account-5 3y agoWhat I'd really be interested in is being able to describe a problem space and have it generate a schema that models it. I'm actually not that bad at generating my own SQL queries.
- CharlesW 3y agoThis works pretty well without a dedicated application today, e.g. "Knowing everything you do about music and music distribution, please define a database schema that supports albums, tracks, and artists". If you have additional requirements or knowledge that the response doesn't address, just add it and re-prompt. When you're done, ask for the SQL to set up the schema in your database of choice.
- account-5 3y agoMaybe my prompting needs to improve, I tried recently to get chatgpt to provide a schema for an sqlite database that implements vcard data in a normalised way. I gave up...
- coder543 3y agoChatGPT-3.5 or ChatGPT-4? There is a big difference. For fun, I just asked ChatGPT-4 to generate a normalized database representation of vcard information: https://chat.openai.com/share/1c88813c-0a50-4ec6-ba92-4d6ff8ee1c9e https://chat.openai.com/share/1c88813c-0a50-4ec6-ba92-4d6ff8... It seems like a reasonable start to me.
- account-5 3y agoChatgpt 3.5. Maybe I should pay for a couple of months access to 4 to see the difference. Is it worth the money?
- coder543 3y agoChatGPT-3.5 isn’t even worth touching as an end-user application. Bard is better (due to having some integrations), but it’s still barely useful. ChatGPT-4 is on an another level entirely compared to either 3.5 or Bard. It is actually useful for a lot. ChatGPT-3.5 can still serve a purpose when you’re talking about API automations where you provide all the data in the prompt and have ChatGPT-3.5 help with parsing or transforming it, but not as a complete chat application on its own. Given the bad experiences ChatGPT-3.5 gives out on a regular basis as a chat application, I don’t even know why OpenAI offers it for free. It seems like a net-negative for ChatGPT/OpenAI’s reputation. I think it is worth paying for a month of ChatGPT-4. Some people get more use out of it than others, so it may not be worth it to you to continue, but it’s hard for anyone to know just how big of a difference ChatGPT-4 represents when they haven’t used it. I provided a sample of ChatGPT-4’s output in my previous response, so you can compare that to your experiences with ChatGPT-3.5.
- jedberg 3y agoThis is awesome. It's a quick turnkey way to get started with RAG using your own existing SQL database. Which to be honest is what most people really want when they say they "want ChatGPT for their business". They just want a way to ask questions in prose and get an answer back, and this gets them a long way there. Very cool!
- zainhoda 3y agoAuthor of the package here. Thank you! That's exactly what we've seen. Businesses spent millions of dollars getting all their structured data into a data warehouse and then it just sits there because the people who need the information don't know how to query the database.
- new_user_final 3y agoDoes it work with Google/Facebook ads data? Can I ask it to show best performing ads from BigQuery Facebook/Google ads data by supermetrics or improvado.
- osigurdson 3y agoI wish we had landed on a better acronym than RAG.
- vinnymac 3y agoEvery single time I see it, I immediately think of Red Amber Green.
- nightski 3y agoIt doesn't matter, RAG is very temporary and will not be around long imho.
- sroecker 3y agoCare to enlighten us why?
- nkozyra 3y agoMost of this stuff is replaced within a calendar year and that will probably accelerate.
- osigurdson 3y agoIt sounds dumb to me.
- ren_engineer 3y agohow else would you get private or recent data into an LLM without some form of RAG? The only aspect that might not be needed is the vector database
- mediaman 3y agoRAG, at its core, is a very human way of doing research, because RAG is essentially just building a search mechanism for a reasoning engine. Much like human research. Your boss asks you to look into something, and you do it through a combination of structured and semantic research. Perhaps you get some books that look relevant, you use search tools to find information, you use structured databases to find data. Then you synthesize it into a response that's useful to answer the question. People say RAG is temporary, that it's just a patch until "something else" is achieved. I don't understand what technically is being proposed. That the weights will just learn everything it needs to know? That is an awful way of knowing things, because it is difficult to update, difficult to cite, difficult to ground, and difficult to precisely manage weights. That the context windows will get huge so retrieval will be unnecessary? That's an argument about chunking, not retrieval. Perhaps people could put 30,000 pages of documents into the context for every question. But there will always be tradeoffs between size and quality: you could run a smarter model with smaller contexts for the same money, so why, for a given budget, would you choose to stuff a dumber model with enormous quantities of unnecessary information, when you could get a better answer from a higher intelligence using more reasonably sized retrievals at the same cost? Likewise, RAG is not just vector DBs, but (as in this case) the use of structured queries to analyze information, the use of search mechanisms to find information in giant unstructured corpuses (i.e., the Internet, corporate intranets, etc). Because RAG is relatively similar to the way organic intelligence conducts research, I believe RAG is here for the long haul, but its methods will advance significantly and the way it gets information will change over time. Ultimately, achieving AGI is not about developing a system that "knows everything," but a system that can reason about anything, and dismissing RAG is to confuse the two objectives.
- teaearlgraycold 3y agoI haven't loaded this up so maybe this has been accounted for, but I think a critical feature is tying the original SQL query to all artifacts generated by Vanna. Vanna would be helpful for someone that knows SQL when they don't know the existing schema and business logic and also just to save time as a co-pilot. But the users that get the most value out of this are the ones without the ability to validate the generated SQL. Issues will occur - people will give incomplete definitions to the AI, the AI will reproduce some rookie mistake it saw 1,000,000 times in its training data (like failing to realize that by default a UNIQUE INDEX will consider NULL != NULL), etc. At least if all distributed assets can tie back to the query people will be able to retroactively verify the query.
- zainhoda 3y agoThis is a good idea. I think what you'd want to do is override the generate_sql function and store the question with the "related" metadata and the generated sql somewhere: https://github.com/vanna-ai/vanna/blob/main/src/vanna/base/base.py#L27-L39 https://github.com/vanna-ai/vanna/blob/main/src/vanna/base/b... We're going to be adding a generic logging function soon and fairly soon what you're talking about could just be a custom logger.
- sighansen 3y agoThis looks really helpful! I'm working a lot on graph databases and am wondering, if there are similar projects working with say neo4j. I guess because you don't have a schema, the complexity goes up.
- jazzyjackson 3y agoneo4j advertises such an integration on their landing page https://neo4j.com/generativeai/ https://neo4j.com/generativeai/
- altdataseller 3y agoWhat's the origin behind the name Vanna?
- booleandilemma 3y agoVanna White? It's the only Vanna I know. https://en.wikipedia.org/wiki/Vanna_White https://en.wikipedia.org/wiki/Vanna_White
- zainhoda 3y agoAuthor of the package here. It was originally because it's both a name and a finance term because I was originally using this to query a financial database. https://www.thebalancemoney.com/vanna-explanation-of-the-options-greek-1031331 https://www.thebalancemoney.com/vanna-explanation-of-the-opt...
- bob1029 3y agoThe most success I had with AI+SQL was when I started feeding errors from the sql provider back to the LLM after each iteration. I also had a formatted error message wrapper that would strongly suggest querying system tables to discover schema information. These little tweaks made it scary good at finding queries, even ones requiring 4+ table joins. Even without any examples or fine tuning data.
- echelon 3y agoPlease turn this into a product. There's enormous demand for that.
- teaearlgraycold 3y agoSomeone get YC on the phone
- bob1029 3y agoI feel like by the time I could turn it into a product, Microsoft & friends will release something that makes it look like a joke. If there is no one on the SQL Server team working on this right now, I don't know what the hell their leadership is thinking. I am not chasing this rabbit. Someone else will almost certainly catch it first. For now, this is a fun toy I enjoy in my free time. The moment I try to make money with it the fun begins to disappear. Broadly speaking, I do think this is approximately the only thing that matters once you realize you can put pretty much anything in a big SQL database. What happens when 100% of the domain is in-scope of an LLM that has iteratively optimized itself against the schema?
- esafak 3y agoThe nitty gritty: https://vanna.ai/blog/ai-sql-accuracy.html https://vanna.ai/blog/ai-sql-accuracy.html
- holoduke 3y agoIt would be fun if you could actually train your raw sql and the llm output is the actual answer and not sql commands. In this way its just another language layer on top/in between of sql. Probably hurts efficiency and performance in the long run.
- pamelafox 3y agoI love that this exists but I worry how it uses the term “train”, even in quotes, as I spend a lot of time explaining how RAG works and I try to emphasize that there is no training/fine-tuning involved. Just data preparation, chunking and vectorization as needed.
- zainhoda 3y agoAuthor of the package here. I would be open to alternative suggestions on terminology! The problem is that our typical user has never encountered RAG before.
- pamelafox 3y agoHm we typically talk about that phase as “data ingestion”. We also use verbs extract, chunk, index, but those are parts of ingestion. Another phrase we use is “data preparation”. The tricky thing is that your train() method is on the vanna object, so if you called it “ingest”, it wouldn’t be clear where the data was headed. But I think it’d be clearer than “train” given that has such a specific meaning in this space. You could also look to LlamaIndex API for naming inspiration.
- zacmps 3y agoI've been using 'build' as in, 'build a dataset'. I think it gets the same idea across without being as easy to confuse with fine-tuning.
- jonahx 3y agoIs the architecture they use in this diagram currently the best way to train LLMs in general on custom data sets? https://raw.githubusercontent.com/vanna-ai/vanna/main/img/vanna-readme-diagram.png https://raw.githubusercontent.com/vanna-ai/vanna/main/img/va... That is, store your trained custom data in vector db and then use RAG to retrieve relevant content and inject that into the prompt of the LLM the user is querying with? As opposed to fine tuning or other methods?
- firejake308 3y agoAll the podcasts I've been listening to recommend RAG over fine-tuning. My intuition is that having the relevant knowledge in the context rather than the weights brings it closer to the outputs, thereby making it much more likely to provide accurate information and avoid hallucinations/confabulations.
- benjaminwootton 3y agoDo you have any podcasts you would reccomend with this type of content?
- firejake308 3y agoI listen to Data Skeptic and Practical AI. It's interesting content, and I find Practical AI has guests who are particularly relevant to what I want to learn, but you should keep in mind that the guests are usually trying to promote their own products, so everything they say should be taken with a grain of salt
- zmmmmm 3y ago> All the podcasts I've been listening to recommend RAG over fine-tuning I'm always suspicious that is just because RAG is so much more accessible (both compute wise and in terms of expertise required). There's far more profit in selling something accessible to the masses to a lot of people than something only a niche group of users can do. I think most people who do actual fine tuning would still probably then use RAG afterwards ...
- luksamer 3y ago[flagged]
- metflex 3y agothat's it, we are going to lose our jobs
- ta8645 3y agoThe extremely-lucrative days in IT for large numbers of people, are drawing to a close. On the other hand, while you won't make more than someone who works at 7/11, there will be rudimentary IT jobs available, if you want them.
- kleiba 3y agoSorry, maybe I'm just too tired to see it, but how much control do you have over the SQL query that is generated by the AI? Is there a risk that it could access unwanted portions or, worse, delete parts of your data? (the AI equivalent of Bobby Tables, so to speak)
- htk 3y agoI guess you could limit that with the correct user permissions.
- thih9 3y agoWhy not give it access to relevant parts of the database only? And read only access too?
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- deleted 3y ago[deleted]
- 8organicbits 3y agoAccess management can be harder than building queries if you start getting fine grained.
- thih9 3y agoYes. Then again, in a project where queries are simpler than granting relevant access to a third party, I guess the AI chat product is just not a good fit.
- bob1029 3y agoIn some SQL providers, your can define rules that dynamically mask fields, suppress rows, etc. based upon connection-specific details (e.g. user or tenant ID). So, you could have all connections from the LLM-enabled systems enforce masking of PII, whereas any back-office connections get to see unmasked data. Doing things at this level makes it very difficult to break out of the intended policy framework.
- ajhai 3y agoWe have recently added support to query data from SingleStore to our agent framework, LLMStack (https://github.com/trypromptly/LLMStack https://github.com/trypromptly/LLMStack). Out of the box performance performance when prompting with just the table schemas is pretty good with GPT-4. The more domain specific knowledge needed for queries, the harder it has gotten in general. We've had good success `teaching` the model different concepts in relation to the dataset and giving it example questions and queries greatly improved performance.
- hrpnk 3y agoPrompts are quite straightforward. - OpenAI: https://github.com/vanna-ai/vanna/blob/a4cdf7593ac0c584f7d745d1be1e5c219129a0a4/src/vanna/openai/openai_chat.py https://github.com/vanna-ai/vanna/blob/a4cdf7593ac0c584f7d74... - Mistral: https://github.com/vanna-ai/vanna/blob/a4cdf7593ac0c584f7d745d1be1e5c219129a0a4/src/vanna/mistral/mistral.py https://github.com/vanna-ai/vanna/blob/a4cdf7593ac0c584f7d74...
- peheje 3y agoMany of these AI "products" - Is it just feeding text into LLMs in a structured manner?
- okwhateverdude 3y agoBasically, yeah. It is shockingly trivial to do, and yet like playing with alchemy when it comes to the prompting, especially if doing inference on the cheap like running smaller models. They can get distracted in your formatting, ordering, CAPITALIZATION, etc.
- wruza 3y agoWhy quotes. Your file manager is just feeding readdir into tableViewDataSource, and your video player is just feeding video files into ffmpeg and then connects its output to a frameBuffer control. Even your restaurant is just feeding vegetables into a dish. Most products "just" do something with existing technologies to remove the tedious parts.
- benjaminwootton 3y agoI built a demo of something similar, using LlamaIndex to query data as it streamed into ClickHouse. I think this has a lot of real world potential, particularly when you move between the query and a GenAI task: https://youtu.be/F3Eup8yQiQQ?si=pa_JrUbBNyvPXlV0 https://youtu.be/F3Eup8yQiQQ?si=pa_JrUbBNyvPXlV0 https://youtu.be/7G-VwZ_fC5M?si=TxDQgi-w5f41xRJL https://youtu.be/7G-VwZ_fC5M?si=TxDQgi-w5f41xRJL I generally found this worked quite well. It was good at identifying which fields to query and how to build where clauses and aggregations. It could pull off simple joins but started to break down much past there. I agree with the peer comment that being able to process and respond to error logs would make it more robust.
- breadwinner 3y agoI have seen good results from just describing the schema to ChatGPT-4 and then asking it to translate English to SQL. Does this work significantly better?
- SOLAR_FIELDS 3y agoThat’s mostly what the products and libraries around this like llamaindex or Langchain are doing. If you look at the Langchain sql agent all it’s doing is chaining together a series of prompts that take the users initial query, attempt to take in a db and discover its schema on the fly and then execute queries against it based on that discovered schema, ensuring the result makes sense. The tough part is doing this at scale as part of a fully automated solution (picture a slack bot hooked up to your data warehouse that just does all of that for you that you converse with). When you have tens or hundreds of tables with relationships and metadata in that schema and you want your AI to be able to unprompted walk all of them, you’re then basically doing some context window shenanigans and building complex state machines to walk that schema Unfortunately that’s kind of what you need if you want to achieve the dream of just having a db that you can ask arbitrary questions to with no other knowledge of sql or how it works. Else the end user has to have some prior knowledge of the schema and db’s to get value from the LLM. Which somewhat reduces the audience for said chatbot if you have to do that
- zainhoda 3y agoThat's basically what happens but the power is that in Python, you can do this: sql = vn.generate_sql(question=...) Which means that now the SQL can be executed and you can get the table, chart, etc in any interface. Flask: https://github.com/vanna-ai/vanna-flask https://github.com/vanna-ai/vanna-flask Streamlit: https://github.com/vanna-ai/vanna-streamlit https://github.com/vanna-ai/vanna-streamlit Chainlit: https://github.com/vanna-ai/vanna-chainlit https://github.com/vanna-ai/vanna-chainlit Slack: https://github.com/vanna-ai/vanna-slack https://github.com/vanna-ai/vanna-slack
- codegeek 3y agoI have been keeping track of a few products like these including some that are YC backed. Interesting space as I am looking for a solution myself: - Minds DB (YC W20) https://github.com/mindsdb/mindsdb https://github.com/mindsdb/mindsdb - Buster (YC W24) https://buster.so https://buster.so - DB Pilot https://dbpilot.io https://dbpilot.io and now this one
- markding00 3y agoalso Brewit.ai (YC W23) https://brewit.ai/ https://brewit.ai/
- bredren 3y agoHave you written up any results of your experience with each? I’m interested in a survey of this field so far and would read it.
- codegeek 3y agoNot yet but not a bad idea if I can get to test them all soon :)
- pylua 3y agoI don’t fully understand the use of business case after reading the documentation. Is it really a time save?
- MattGaiser 3y agoIt would be for people who are not that fluent in SQL. Even as a dev, I find ChatGPT to be easier for writing queries than hand coding them as I do it so infrequently.
- pylua 3y agoYeah, same here. Seems like that approach is much simpler than this. I guess the real benefit here is that you don’t need to understand the schemas so the knowledge is not lost when someone leaves a company. Sort of an abstraction layer for the schemas
- jug 3y agoI wonder if this supports spatial queries as in PostGIS, SpatiaLite, SQL Server Spatial as per the OGC standard? I'm interested in integrating a user friendly natural language query tool for our GIS application. I've looked at LangChain and the SQL chain before but I didn't feel it was robust enough for professional use. You needed to run an expensive GPT-4 backend to begin with and even then, it wasn't perfect. I think a major part of this is that it wasn't actually trained on the data like Vanna apparently does.
- zainhoda 3y agoAuthor of the package here. I think that it probably could handle that but may need more example SQL queries.
- crimbles 3y agoI can't wait until it naively does a table scan on one of our several TB tables...
- swimwiththebeat 3y agoI'm curious to see if people have tried this out with their datasets and seen success? I've been using similar techniques at work to build a bot that allows employees internally to talk to our structured datasets (a couple MySQL tables). It works kind of ok in practice, but there are a few challenges: 1. We have many enums and data types specific to our business that will never be in these foundation models. Those have to be manually defined and fed into the prompt as context also (i.e. the equivalent of adding documentation in Vanna.ai). 2. People can ask many kinds of questions that are time-related like 'how much demand was there in the past year?'. If you store your data in quarters, how would you prompt engineer the model to take into account the current time AND recognize it's the last 4 quarters? This has typically broken for me. 3. It took a LOT of sample and diverse example SQL queries in order for it to generate the right SQL queries for a set of plausible user questions (15-20 SQL queries for a single MySQL table). Given that users can ask anything, it has to be extremely robust. Requiring this much context for just a single table means it's difficult to scale to tens or hundreds of tables. I'm wondering if there's a more efficient way of doing this? 4. I've been using the Llama2 70B Gen model, but curious to know if other models work significantly better than this one in generating SQL queries?
- qiller 3y agoFor 2. we ended up stuffing the prompt with examples for common date ranges "this month", "last year", "this year to date" and some date math, and examples of date fields (we have timestamp, and extracted Year, Month, Day, etc) Current date: `current_date()` 3 days ago: `current_date() - INTERVAL 3 DAY` Beginning of this month: `date_trunc('month', current_date())` ... 4. I get best results with GPT-4, haven't tried Llama yet. 3.5 and 4-turbo tend to "forget" stuff for complex queries, but may be we need more tuning yet.
- totalhack 3y agoHave you considered a semantic layer instead, or does it have to be a natural language interface?
- aussieguy1234 3y agoI've already done this with GPT-4. It goes something like this: Here's the table structure from MySQL cli `SHOW TABLE` statements for my tables I want to query. Now given those tables, give me a query to show me my cart abandonment rate (or, some other business metric I want to know). Seems to work pretty well.
- zainhoda 3y agoAuthor of the package here. That's pretty much what this package does just with optimization around what context gets sent to the LLM about the database: https://github.com/vanna-ai/vanna/blob/main/src/vanna/base/base.py#L27-L39 https://github.com/vanna-ai/vanna/blob/main/src/vanna/base/b... And then of course once you have the SQL, you can put it in an interface where you the SQL can be run automatically and then get a chart etc.
- lopatin 3y agoWhat surprises me is the amount of programmers and analysts that I meet the don’t do this yet. Writing complex, useful SQL queries is probably the most valuable thing that ChatGPT does for me. It makes you look like a god to stakeholders, and “I’m not strong in SQL” is no longer an excuse for analysis tasks.
- zainhoda 3y agoI've noticed this as well. Do you have any theories as to why that is?
- hot_gril 3y agoI already know SQL, and yes it can take time to form exceptionally complex queries, but ChatGPT doesn't seem to do those accurately (yet). CGPT is more useful for general programming languages where there's a lot more boilerplate.
- aussieguy1234 3y agoI've found GPT-4 to be alot better than the free version of ChatGPT (which uses the earlier GPT 3.5).
- miohtama 3y agoHow about instead of making AI wrappers to over 50 years old SQL, we’d make a database query language that’s easier to read an write?
- marginalia_nu 3y agoIn general, if something has been around for a very long time and nobody apparently seems to have thought to improve it, then odds are the reason is it's pretty good and genuinely hard to improve on.
- aae42 3y agoin other words, SQL is a shark, not a dinosaur
- realanswe91 3y agoThe Shark Query Language
- fbdab103 3y agoCodd himself had loads of complaints about SQL.
- oblio 3y agoWell, if you think about it, a huge percentage of dinosaurs survived (birds), and for the rest, can you really fault them for going extinct when a humongous boulder hit the planet?
- runlaszlorun 3y ago> pretty good and genuinely hard to improve on I think that might be a bit more positive than I would be. Broadly speaking, I think you could say that the downsides of the legacy technology in question aren’t larger than the collective switching costs. But I’d definitely agree that when something has been around that long, it’s prob not all bad.
- SoftTalker 3y ago
- neofrommatrix 3y agoI’ve done this with Neo4j. Pretty simple to hook it up with Open AI APIs and have a conversational interface.
- Vosporos 3y agoI can hire a DBA to tell me that my indexes aren't shit, no need for AI.
- l5870uoo9y 3y agoIs there a list of SQL generations to see how it performs? This is a list of SQL examples using GTP-4 and the DVDrental database sample. [1]: https://www.sqlai.ai/sql-examples https://www.sqlai.ai/sql-examples [2]: https://www.postgresqltutorial.com/postgresql-getting-started/postgresql-sample-database/ https://www.postgresqltutorial.com/postgresql-getting-starte...
- kulikalov 3y agoWhile I recognize the efforts in developing natural language to SQL translation systems, I remain skeptical. The core of my concern lies in the inherent nature of natural language and these models, which are approximative and lack precision. SQL databases, on the other hand, are built to handle precise, accurate information in most cases. Introducing an approximative layer, such as a language model, into a system that relies on precision could potentially create more problems than it solves, leading me to question the productivity of these endeavors in effectively addressing real-world needs.
- samstave 3y agoReverse idea: Use this to POPULATE sql based on captured NLP "surveillance" -- for example, build a DB of things I say as my thing listens to me, and categorize things, topics, place, people etc mentioned. Keep count of experiencing the same things.... When I say I need to "buy thing" build table of frequency for "buy thing" etc... Effectively - query anything you've said to Alexa and be able to map behaviors/habits/people/things... If I say - Bob's phone number is BLAH. It add's bob+# to my "random people I met today table" with a note of "we met at the dog park" Narrate yourself into something journaled. Makes it easy to name a trip "Hike Mount Tam" then log all that - then have it create a link in the table to the pics folder on your SpaceDrive9000.ai and then you have a full narration with links to the pics that you take.
- pietz 3y agoI think you have a point. I also think people who share this mindset and keep sticking with it will have a tough future. The technical benefits and potentials clearly outshine the problems and challenges. That's also true in this example. You just have to let go of some principles that were helpful in the past but aren't anymore.
- qiller 3y agoWe did something similar for our reporting service which is based duckdb. Overall it works great, though we've ran into a few things: * Even with low temperature, GPT-4 sometimes deviates from examples or schema. For example, sometimes it forgets to check one or another field... * Our service hosts generic data, but customers ask to generate reports using their domain language (give me top 10 colors... what's a color?). So we need to teach customers to nudge the report generator a bit towards generic terms * Debugging LLM prompts is just tricky... Customers can confuse the model pretty easily. We ended up exposing the "explained" generated query back to give some visibility of what's been used for the report
- ignoramous 3y agoCurious, as we're looking to build / use a similar setup. > Debugging LLM prompts is just tricky... Customers can confuse the model pretty easily. Would a RAG like how Vanna.ai uses, help? > For example, sometimes it forgets to check one or another field Do prompting techniques like CoT improve the outcome? > So we need to teach customers to nudge the report generator a bit towards generic terms. Did you folks experiment with building an Agent-like interface that asks more questions before the LLM finally answers?
- qiller 3y agoOur primary issue is that our DB is a dynamic Entity-Attribute-Value schema, even quite a bit denormalized at that. The model has to remember to do subqueries to retrieve "attributes" based on what's needed for the query and then combine them correctly. NLQ is a somewhat new feature for us, so we don't have a great library to pull from for RAG. Experimenting, I found that having a few-shot examples with some CoT (showing examples of chaining attributes retrieval) sprinkled around did help a lot. Even still, some queries come out quite ugly, but still functional. I'm thankful that DuckDB is a beast when tackling those :D > Did you folks experiment with building an Agent-like interface that asks more questions before the LLM finally answers? That's something I want to figure out next: 1) try to check if a generated query would work but would generate absolutely junk results (cause the model forgot to check something) and ask to rephrase 2) or show results (which may look "real" enough), but give an ability to tweak the prompt. A good example is something like "top 5 products on Cyber Monday" <- which returns 0 products, cause 2024 didn't happen yet, and should trigger a follow up.
- adam_gyroscope 3y agoWe did this at bit.io and people loved it - there a bunch of articles we wrote on what we found during our work: https://innerjoin.bit.io/ https://innerjoin.bit.io/ We’ve since shut down (acquired by databricks) but happy to answer what I can.
- zainhoda 3y agoSuper interesting! Why did you decide to shut down?
- adam_gyroscope 3y agoAcquired by databricks - I can’t really speak about what we’re working on.
- willsmith72 3y agoThe docs don't really work on mobile, the side navbar takes up half the screen and doesn't seem closeable
- kgdiem 3y agoThanks so much for making this and making it under MIT to boot. I’ve been thinking about how to do this for about 6 months now and just started working on a demo for querying just one table // JSON column today. I would feel a lot more comfortable with putting this (and/or my demo) into production if the database had been set up with a schema+db user per account rather than every tenant sharing just the one set of tables.
- hyuuu 3y agoim curious to know how you get around the hallucinations? for example, for the query: "give me <products / sales / rows> that were created yesterday" the llm hallucinates as to what "yesterday" means, there are other instances as well where the generated SQL is valid syntax-wise but not in intent. This is especially dangerous for aggregation queries such as MAX, COUNT, etc because it will spit out a number but is it the right number? and the only way to check is to read the SQL itself and verify, which defeats the whole purpose of it.
- zainhoda 3y agoThat's a fair concern. In actual usage, however, the vast majority of hallucination (>90%) tends to be: - phantom tables and columns, in which case the query will fail - incorrect syntax for date functions (i.e. the wrong flavor of SQL) And we tend to see less of this type of hallucination when there are lots of example SQL queries that have been "trained" into the RAG system.
- thecalebf 3y agoVery neat! I am building something very similar, called ChatDB.ai
- dcreater 3y agoPerhaps an uninformed question, but why do we need an llm rather than a simpler natural language to SQL NLP translator? Wouldn't that be much more efficient and reliable?
- bobbylarrybobby 3y agoPeople have been working on simple natural language processing algorithms since the 50‘s. If it were easy we'd have one by now.
- zainhoda 3y agoWhat would be missing from that is the ability to look up what tables are in the database, how they join together, what terminology the business uses, preferences around whether to exclude nulls from certain columns, etc. This allows you to ask more of a “business question” like “who are the top 10 customers by sales?” and the LLM will be able to construct a query that joins the customers table to an orders table and get the results that you’re looking for. With a simple NL to SQL, you’d have to say “join the customer table with the sales table, aggregate on the sales column from the orders table and limit the results to 10 rows” or something along those lines.
- cpt100 3y agoI have tried things like this, they are good for debugging and asking simple questions but is really hard to train them to be good enough for production. You'll get enough frustrating results, that you'll abandon them soon.
- elietoubi 3y agoIf anyone is interested, I built and open-sourced parse.dev It's a rails app that allows you to talk to your database.
- sagaro 3y agoAll these products that pitch about using AI to find insights from your data always end up looking pretty in demos and fall short in reality. This is not because the product is bad, but because there is enormous amount of nuance in DB/Tables that becomes difficult to manage. Most startups evolve too quickly and product teams generally tries to deliver by hacking some existing feature. Columns are added, some columns get new meaning, some feature is identified by looking at a combination of 2 columns etc. All this needs to be documented properly and fed to the AI and there is no incentive for anyone to do it. If the AI gives the right answer, everyone is like wow AI is so good, we don't need the BAs. If the AI gives terrible answers they are like "this is useless". No one goes "wow, the data engineering team did a great job keeping the AI relevant".
- zurfer 3y agothat is correct. GPT-4 is good on well-modelled data out of the box, but struggles with a messy and incomplete data model. Documenting data definitely helps to close that gap. However the last part you describe is nothing new (BI teams taking credit, and pushing on problems to data engineers). In fact there is a chance that tools like vanna.ai or getdot.ai bring engineers closer to business folks. So more honest conversations, more impact, more budget. Disclaimer: I am a co-founder at getdot.ai :)
- lmeyerov 3y agoAgreed, maybe I wasn't clear enough. I don't view it as BI team vs platform team vs whoever. Maybe a decrease in the need for PhD AI consultants for small projects, or to wait for some privileged IT team for basic tasks, so they can focus on bigger things. Instead of Herculean data infra projects, this is a good time for figuring out new policy abstractions, and finding more productive divisions of labor between different days stakeholders and systems. Machine-friendly abstractions and structure are tools for predictable collaboration and automation. More doing, less waiting. More practically, an increasing part of the Louie.ai stack is helping get the time-consuming quality, guardrails, security, etc parts under easier control of small teams building things. As-is, it takes a lot to give a great experience.
- 3y ago
- FrostKiwi 3y agoA bit ironic, considering SQL statements were designed to read like English sentences.
- BrickTamblan 3y agoAnyone know of any sql management server / GUI software that use AI and your db schema (no data) to aid in SQL generation? I use SQLStudio on a mac, which I love but sadly, no AI
- DoryMinh 3y agoI'm rather confused with the use of chat with database. If we don't know what is in the database, how do we know which questions to ask?
- neodymiumphish 3y agoDoes anyone know of a simple way to use this against LM studio? It mimicks OpenAI's API, but Vanna doesn't seem to allow me to point the OpenAI integration against my own endpoint.
- atomicvibe 3y agogreat tool and as someone mentioned there are quite a few AI products/libraries that accomplish this. The main challenge for production settings 1) soon becomes optimizing the latency for each step and 2) dealing with complex queries at a reasonable cost (GPT-4 is bit expensive at scale) We evaluated multiple options and settled on GPT-3Turbo in the short term. However we are not happy with the latency, especially if you create an agentic solution.
- eek2121 3y agoI don't wanna. Just let me query my data, please.
- jamietanna 3y agoI'll be giving this a go with one of my side projects which has several tables with data that isn't super normalised, as it'd be interesting to see if it can cope with it
- deleted 3y ago[deleted]
- ethanding 3y agodisclaimer: founder of textql.com (yea i know the name is super obvious) I would happily bet a kidney that not a single product that has an LLM of any kind writing SQL using next-character-prediction will ever be useful to get results for a business person in an enterprise setting (~5+ people on the data team of a company). Although it's fine for technical users as advanced autocomplete I'm surprised no one has pointed out that as a building block: english doesn't map 1:1 to SQL at all. "Can you get me a list of users who've ordered more than 10 red products" doesn't make it at all clear whether you're writing - s from u where u.ordered_more_than_10_red_products = True - s from u where u.red_products_ordered > 10 - or any other combination of variants 15 years of VCs have poured well over 10 billion dollars claiming to solve "text to sql" of some kind and 15 year later we're still writing SQL - something something chesterton's fence... at least we don't have training sessions where data eng people sit in a room and hard code synonyms for "revenue" into thoughtspot anymore The only way you can approach something like this at scale is w/ a mature ontology structure for your data. What that means in is a very opinionated data modeling method (handful of valid ones) that is enforced at the level of the data modeling system - dbt / sqlmesh / stored procedures, or otherwise - semantic layers are... kinda ok but mostly doesn't get there on expressiveness either ^lots of ways to do the above, small handful of cool people who are working on it. (yes that kinda includes us - sorry for the implicit self promo) - obvious second disclaimer that if your "database" is 3 tables w/ clearly labeled joins, feel free to disregard everything I just said - oh, also the correct query according to chatgpt is SELECT u.user_id, COUNT(o.product_id) AS total_red_products FROM users u JOIN orders o ON u.user_id = o.user_id JOIN products p ON o.product_id = p.product_id WHERE p.color = 'red' GROUP BY u.user_id HAVING COUNT(o.product_id) > 10; - obvious 4th disclaimer - if i'm wrong, i don't know what i'm talking about
- brnbrdg 3y agoI think some of the doubters here are conflating a few things. Any analyst will struggle if data quality is a mess, and there is a lack of clear semantic layer/useful documentation around how to query metrics/join tables from your warehouse. These are problems that affect human analysts as well as LLM based 'virtual' ones... However, these are separate problems being addressed by other players. An llm powered chatbot that can consume adequate context from company systems should be able to perform on par with a junior analyst with a similar level of context. All else equal, the LLM analyst will be orders of magnitude cheaper and open up data analysis to non-sql-literate people (huge unlock).