7 ms·

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What c

by rgbrgb 3y ago

So it looks like it scores 76.5% on SQL-Eval [0], a bit behind GPT-4 at 83% and sqlcoder-15b at 78%. What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? As a programmer who always has to look stuff up when I SQL, I could definitely see asking something like this for a first draft of a query but it seems like I'm slightly better off asking the bigger models in these one-off cases (and I can run a 15b easily on my 64GB m1). If I'm in a corporate setting I'm not going to leak my schema into OpenAI's training data and there are definitely times when I'd want to run queries offline. Small/local models are great when you want to do a ton of queries (save $$). A mini data scientist that could be queried by non-technical folks would be awesome but I wonder if there's a way to determine whether the query is falling in the 25% "incorrect" case... maybe there's a RAID-like consensus algorithm where you have multiple interrogate each other's answers to get a higher overall success rate. Mostly thinking out loud :) but maybe ya'll have more ideas. Congrats on the release, OP! [0]: https://github.com/defog-ai/sql-eval https://github.com/defog-ai/sql-eval

whimsicalism 3y ago
do you not trust the setting OAI provides to exclude your conversation from training data?
- tempusalaria 3y ago
  1) OpenAI has consistently gone back on commitments it has made 2) Sam Altman has a shady track record publicly, and if you believe the things people say privately he has consistently done business very dishonestly throughout his career. He is the CEO, and virtually the entire executive team are people he brought in from his network. It’s his company. 3) To give one example of many, OpenAI recently changed the terms of ChatGPT so that web users conversations can now be trained on (and if you want to save any chats you must opt in). Presumably this also applies to all conversations you had under the old policy despite saying they would never train on those conversations. I could go on at length…
  - anonylizard 3y ago
    If you don't trust OpenAI, you can choose to trust Microsoft. Azure OpenAI is 3-months behind OpenAI that costs 3x as much, but you get more 'security' and better performance.
    - CharlesW 3y ago
      > Azure OpenAI is 3-months behind OpenAI… How is it 3 months behind if you get access to current OpenAI models?
      - anonylizard 3y ago
        You don't. It took like 2 extra months for Azure OpenAI to get GPT-4 turbo. There's a noticeable time delay between OpenAI deploying their latest model and when Microsoft manages to shove it in Azure.
        CharlesW 3y ago
        Great to know, thank you.
        drittich 3y ago
        And it's not just the models. E.g., the Assistants API is not available yet in Azure, and there is no expected ship date for it. But I'm confident it's coming.
    - tempusalaria 3y ago
      Yes and I recommend that people interested in GPT-4 use this service as it’s isolated from OpenAI and it is the absolute best model right now. That said, there are 3 quite worrying future possibilities, both stemming from the level of investment and commitment by Microsoft in OpenAI - a huge percentage of Azure’s value is bet on an exclusive partnership with them. That gives OpenAI a lot of leverage. And customer data is a very tempting cookie jar to build a long term moat for these companies Possibility 1: someone overtakes OpenAI models and you’re stuck on Azure who won’t offer that model Possibility 2: OpenAI decides to break with Microsoft and you’re stuck with a useless application. AWS Bedrock is far less likely to leave an AI application obsolete. Possibility 3: OpenAI put pressure on Microsoft to loosen the data protections around the service, or go around them entirely. This is a particular concern as the systems in the service become more complex and agentic and more difficult for Microsoft to audit. Model weights are highly opaque and Microsoft cannot trace the exact possible behaviour of these systems. What if GPT-6 changes its own weights during inference for example? How can Microsoft ever understand if that’s a true critical piece of functionality or a proxy method to access customer data etc.
- anonylizard 3y ago
  Your average SQL query is worthless, there no special sauce in SQL queries, their value comes from the dataset they run on. These small SQL LLMs are indeed worthless, since LLM performance on SQL queries is quite limited right now, every % of accuracy matters.
  - whimsicalism 3y ago
    Did you mean to reply to a different comment?
  - rgbrgb 3y ago
    with data privacy decisions I like to think about how the policy/reasoning would sound in a NYT headline or congress testimony. your stance does not imbue trust for me. 0 employment contracts or user facing ToS I've read make a distinction between sharing schema vs data with unvetted third parties. In my opinion, schemas reveal pretty valuable info about an application (and they're absolutely valuable in aggregate because you can use them to train AI data scientists). Maybe I sound like a curmudgeon but since I have the option of running the AI locally with a 5 percentage point accuracy loss, I absolutely will. If GPT-4 was 100% that would be different because you could build totally different things, but 83% has most of the same design problems as 78%.
- rgbrgb 3y ago
  about as much as I trust 23andme to keep my genetic data private i give their intent the benefit of the doubt but I’ve been on the other side of too many data collection systems to trust that theirs is foolproof and more secure than my local machine.
  - whimsicalism 3y ago
    To me there is a significant difference between permanent storing and “keeping private” (23andme is legally prohibited from deleting your genetic data) and not using and deleting after 30 days. fwiw 23andme genetic data (distinct from summary ancestry data) has not leaked afaik
- ren_engineer 3y ago
  it's not about trust, they will always be indirectly controlled by the US government who could force them to leak/release your data. OpenAI is already working with the US military and removed their restrictions on allowing their AI to be used for military purposes https://time.com/6556827/openai-us-military-cybersecurity/ https://time.com/6556827/openai-us-military-cybersecurity/
fifilura 3y ago
1. If you are a a programmer I think you should learn SQL. It will give you a different perspective on programming that I think is invaluable. (I.e. programming without for loops) 2. Combining and slicing data is a craft, and doing it subtly wrong in one step can lead to fatal errors in the outcome. And most importantly, it can be very difficult to notice. Numbers don't smell. That is why I would be very hesitant to give a slightly more than trivial task to an engine that fails 25% of the time. But I guess that is the same as any other programming task. Just that other programming tasks require a lot of boilerplate where an AI can help. SQL is much more straight to it. Maybe it could be useful to ask questions that are similar to writing testcases "how can I verify that my query is doing the right thing"?
- gavinray 3y ago
  What makes you think that SQL doesn't have "for loops"? Ever heard of LATERAL joins/CROSS APPLY? SELECT loop.value, x.squared FROM generate_series(1,5) AS loop(value) CROSS JOIN LATERAL (SELECT loop.value * loop.value AS squared) AS x;
  - neuronexmachina 3y ago
    TIL
    - gavinray 3y ago
      They're incredibly useful -- any time that you want to both: - Reference data from the previous part of the query (the "left-hand side") - Return multiple columns The only way you can achieve it is with LATERAL/CROSS APPLY. Regular correlated subqueries can only return a single column, so something like this doesn't work: SELECT loop.val, (SELECT loop.val * loop.val, 'second column') AS squared FROM (SELECT loop.val FROM generate_series(1,5) AS loop(val)) as loop You'd get: error: subquery must return only one column
      - ttfkam 3y ago
        I love LATERALs, but this still fits within set theory and a bulk application rather than an iterative for-loop. It may even be implemented as a for-loop within the engine, but SQL being declarative abstracts that away from the query interface. It's sets all the way down. A set of f(x) is still a set.
wg0 3y ago
As an end consumer, you should be well versed in SQL (or whatever subject) yourself and should proof read twice all the queries generated before you make a decision based on the data that's pulled from the generated queries. That's the story of the LLMs in general. The hype is free. Startup ecosystem are a bonus.
bongodongobob 3y ago
I can definitely appreciate security concerns but I don't see how providing your schema is of any concern whatsoever.
internet101010 3y ago
> What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? Yeah this is the issue I have with all of the SQL generation stuff. Not only should the SQL be valid, a prompt like "generate a query that pulls sales for the last quarter" should generate the same output for everyone without fail. Vanna's business logic embedding is a good first step but even then it is only correct like 90% of the time with GPT-4. Even then, it will only work if there are strong standards and data governance structures in place that everyone within an organization is aligned on. For example, "sales" can mean different things to different people and all of that needs to be buttoned up as well.
- Closi 3y ago
  As someone who works with lots of analysts, I can guarantee that they also don't make the correct interpretations all the time, and that you have to sense-check the results back against reality. In either case, validation is the key step - you can't just trust that your SQL query is correct regardless of if you have manually written it, you still have to go through the data and check it. That's where the SQL generation stuff can save time - if 50% of the time you can get to an answer in half the time, then it's great! Normally in my experience with current-gen LLM's when they fail they fail quickly, so the other 50% of queries don't take twice as long to write manually. Then there is the other use case - if you aren't sure why a particular SQL query is erroring, these LLM's are great at telling you why and fixing your code.
- tillvz 3y ago
  Having an LLM be in charge of business logic is madness. There cannot be any AI involved when processing the definition of a KPI. Otherwise you'll never be able to roll it out to thousands of users when there's always a 90% (or even 99%) chance that the business logic might not get applied correctly. Check out what we do at Veezoo (https://www.veezoo.com https://www.veezoo.com) with the Knowledge Graph / Semantic Layer to mitigate that.
Closi 3y ago
> What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? The kind of stuff that it is very easy to validate if it works or not :) I am building a warehouse management system at the moment, and it's great to quickly churn out lots of SQL views (particularly as the schema is changing/evolving slightly as I am writing it, so being able to go back to GPT4 to churn through the changes to some of the 'views' of my pages helps, even if it requires a little testing/validation).
davidy123 3y ago
This kind of approach could be a way to make these models reliable. Encode your business restrictions in the database, if they are rigorous enough, if the query passes then it's valid in your business world.
brian_reardon 3y ago
I would wonder how many of the 25% are actually dangerous to run, considering the target audience might be someone who is rusty at SQL and etc.
dimask 3y ago
Imo stuff like this can have applications in a narrower set of problems, where you can also further finetune or simple prompt engineer a model to have a much higher accuracy. I do not think that they can become the bases in which database work is gonna take place in general, but it could be used to automate certain types of tasks that would be too hard/complicated to automate in a classical programmatic way, but not too hard/complicated a real intelligence to be needed to solve.
beefield 3y ago
> What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? I have written a bunch of more or less complicated SQL during my career. And I am pretty sure that if I need to write a SQL statement that's anything but select * from table, my output won't work 75% of time. I may be special case, but typically if I work on a hard problem, it is not a single hard problem but a sh*tload of connected simple problems. If I can get someone to solve the simple problems 75% of the time correctly so that I can spend my time figuring out how those simple problems are to be connected, I'm ore than happy. And that's exactly how I use chatgpt. I have learned not to ask too complex questions from it. But the simple ones, it mostly aces and when it does not , they are easy to spot, as it is not that I could not have solved them myself, I just did not want to spend time for that. Now, if only the chatgpt was not almost as lazy as me to produce long simple stuff, that would be awesome.
bagels 3y ago
I know sql pretty well, but every once in a while doing things like percentiles come up where I don't remember the exact incantations to get the window syntax right. Things like this can save some googling and documentation reading. I can read it to see if it makes sense, and execute it. I can create tests for the output to verify. Can save a lot of time. If it's wrong, I use the fallback of googling for examples.
fijiaarone 3y ago
The 25% of SQL that covers everything in 99% of applications is a subset of the 76.5% of SQL that the LLM got right.
lawxls 3y ago
Yep, just use GPT-4

by rgbrgb 3y ago

whimsicalism 3y ago

do you not trust the setting OAI provides to exclude your conversation from training data?

tempusalaria 3y ago
1) OpenAI has consistently gone back on commitments it has made 2) Sam Altman has a shady track record publicly, and if you believe the things people say privately he has consistently done business very dishonestly throughout his career. He is the CEO, and virtually the entire executive team are people he brought in from his network. It’s his company. 3) To give one example of many, OpenAI recently changed the terms of ChatGPT so that web users conversations can now be trained on (and if you want to save any chats you must opt in). Presumably this also applies to all conversations you had under the old policy despite saying they would never train on those conversations. I could go on at length…
- anonylizard 3y ago
  If you don't trust OpenAI, you can choose to trust Microsoft. Azure OpenAI is 3-months behind OpenAI that costs 3x as much, but you get more 'security' and better performance.
  - CharlesW 3y ago
    > Azure OpenAI is 3-months behind OpenAI… How is it 3 months behind if you get access to current OpenAI models?
    - anonylizard 3y ago
      You don't. It took like 2 extra months for Azure OpenAI to get GPT-4 turbo. There's a noticeable time delay between OpenAI deploying their latest model and when Microsoft manages to shove it in Azure.
      - CharlesW 3y ago
        Great to know, thank you.
      - drittich 3y ago
        And it's not just the models. E.g., the Assistants API is not available yet in Azure, and there is no expected ship date for it. But I'm confident it's coming.
  - tempusalaria 3y ago
    Yes and I recommend that people interested in GPT-4 use this service as it’s isolated from OpenAI and it is the absolute best model right now. That said, there are 3 quite worrying future possibilities, both stemming from the level of investment and commitment by Microsoft in OpenAI - a huge percentage of Azure’s value is bet on an exclusive partnership with them. That gives OpenAI a lot of leverage. And customer data is a very tempting cookie jar to build a long term moat for these companies Possibility 1: someone overtakes OpenAI models and you’re stuck on Azure who won’t offer that model Possibility 2: OpenAI decides to break with Microsoft and you’re stuck with a useless application. AWS Bedrock is far less likely to leave an AI application obsolete. Possibility 3: OpenAI put pressure on Microsoft to loosen the data protections around the service, or go around them entirely. This is a particular concern as the systems in the service become more complex and agentic and more difficult for Microsoft to audit. Model weights are highly opaque and Microsoft cannot trace the exact possible behaviour of these systems. What if GPT-6 changes its own weights during inference for example? How can Microsoft ever understand if that’s a true critical piece of functionality or a proxy method to access customer data etc.
anonylizard 3y ago
Your average SQL query is worthless, there no special sauce in SQL queries, their value comes from the dataset they run on. These small SQL LLMs are indeed worthless, since LLM performance on SQL queries is quite limited right now, every % of accuracy matters.
- whimsicalism 3y ago
  Did you mean to reply to a different comment?
- rgbrgb 3y ago
  with data privacy decisions I like to think about how the policy/reasoning would sound in a NYT headline or congress testimony. your stance does not imbue trust for me. 0 employment contracts or user facing ToS I've read make a distinction between sharing schema vs data with unvetted third parties. In my opinion, schemas reveal pretty valuable info about an application (and they're absolutely valuable in aggregate because you can use them to train AI data scientists). Maybe I sound like a curmudgeon but since I have the option of running the AI locally with a 5 percentage point accuracy loss, I absolutely will. If GPT-4 was 100% that would be different because you could build totally different things, but 83% has most of the same design problems as 78%.
rgbrgb 3y ago
about as much as I trust 23andme to keep my genetic data private i give their intent the benefit of the doubt but I’ve been on the other side of too many data collection systems to trust that theirs is foolproof and more secure than my local machine.
- whimsicalism 3y ago
  To me there is a significant difference between permanent storing and “keeping private” (23andme is legally prohibited from deleting your genetic data) and not using and deleting after 30 days. fwiw 23andme genetic data (distinct from summary ancestry data) has not leaked afaik
ren_engineer 3y ago
it's not about trust, they will always be indirectly controlled by the US government who could force them to leak/release your data. OpenAI is already working with the US military and removed their restrictions on allowing their AI to be used for military purposes https://time.com/6556827/openai-us-military-cybersecurity/ https://time.com/6556827/openai-us-military-cybersecurity/

fifilura 3y ago

1. If you are a a programmer I think you should learn SQL. It will give you a different perspective on programming that I think is invaluable. (I.e. programming without for loops) 2. Combining and slicing data is a craft, and doing it subtly wrong in one step can lead to fatal errors in the outcome. And most importantly, it can be very difficult to notice. Numbers don't smell. That is why I would be very hesitant to give a slightly more than trivial task to an engine that fails 25% of the time. But I guess that is the same as any other programming task. Just that other programming tasks require a lot of boilerplate where an AI can help. SQL is much more straight to it. Maybe it could be useful to ask questions that are similar to writing testcases "how can I verify that my query is doing the right thing"?

gavinray 3y ago
What makes you think that SQL doesn't have "for loops"? Ever heard of LATERAL joins/CROSS APPLY? SELECT loop.value, x.squared FROM generate_series(1,5) AS loop(value) CROSS JOIN LATERAL (SELECT loop.value * loop.value AS squared) AS x;
- neuronexmachina 3y ago
  TIL
  - gavinray 3y ago
    They're incredibly useful -- any time that you want to both: - Reference data from the previous part of the query (the "left-hand side") - Return multiple columns The only way you can achieve it is with LATERAL/CROSS APPLY. Regular correlated subqueries can only return a single column, so something like this doesn't work: SELECT loop.val, (SELECT loop.val * loop.val, 'second column') AS squared FROM (SELECT loop.val FROM generate_series(1,5) AS loop(val)) as loop You'd get: error: subquery must return only one column
    - ttfkam 3y ago
      I love LATERALs, but this still fits within set theory and a bulk application rather than an iterative for-loop. It may even be implemented as a for-loop within the engine, but SQL being declarative abstracts that away from the query interface. It's sets all the way down. A set of f(x) is still a set.

wg0 3y ago

As an end consumer, you should be well versed in SQL (or whatever subject) yourself and should proof read twice all the queries generated before you make a decision based on the data that's pulled from the generated queries. That's the story of the LLMs in general. The hype is free. Startup ecosystem are a bonus.

bongodongobob 3y ago

I can definitely appreciate security concerns but I don't see how providing your schema is of any concern whatsoever.

internet101010 3y ago

> What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? Yeah this is the issue I have with all of the SQL generation stuff. Not only should the SQL be valid, a prompt like "generate a query that pulls sales for the last quarter" should generate the same output for everyone without fail. Vanna's business logic embedding is a good first step but even then it is only correct like 90% of the time with GPT-4. Even then, it will only work if there are strong standards and data governance structures in place that everyone within an organization is aligned on. For example, "sales" can mean different things to different people and all of that needs to be buttoned up as well.

Closi 3y ago
As someone who works with lots of analysts, I can guarantee that they also don't make the correct interpretations all the time, and that you have to sense-check the results back against reality. In either case, validation is the key step - you can't just trust that your SQL query is correct regardless of if you have manually written it, you still have to go through the data and check it. That's where the SQL generation stuff can save time - if 50% of the time you can get to an answer in half the time, then it's great! Normally in my experience with current-gen LLM's when they fail they fail quickly, so the other 50% of queries don't take twice as long to write manually. Then there is the other use case - if you aren't sure why a particular SQL query is erroring, these LLM's are great at telling you why and fixing your code.
tillvz 3y ago
Having an LLM be in charge of business logic is madness. There cannot be any AI involved when processing the definition of a KPI. Otherwise you'll never be able to roll it out to thousands of users when there's always a 90% (or even 99%) chance that the business logic might not get applied correctly. Check out what we do at Veezoo (https://www.veezoo.com https://www.veezoo.com) with the Knowledge Graph / Semantic Layer to mitigate that.

Closi 3y ago

> What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? The kind of stuff that it is very easy to validate if it works or not :) I am building a warehouse management system at the moment, and it's great to quickly churn out lots of SQL views (particularly as the schema is changing/evolving slightly as I am writing it, so being able to go back to GPT4 to churn through the changes to some of the 'views' of my pages helps, even if it requires a little testing/validation).

davidy123 3y ago

This kind of approach could be a way to make these models reliable. Encode your business restrictions in the database, if they are rigorous enough, if the query passes then it's valid in your business world.

brian_reardon 3y ago

I would wonder how many of the 25% are actually dangerous to run, considering the target audience might be someone who is rusty at SQL and etc.

dimask 3y ago

Imo stuff like this can have applications in a narrower set of problems, where you can also further finetune or simple prompt engineer a model to have a much higher accuracy. I do not think that they can become the bases in which database work is gonna take place in general, but it could be used to automate certain types of tasks that would be too hard/complicated to automate in a classical programmatic way, but not too hard/complicated a real intelligence to be needed to solve.

beefield 3y ago

> What kind of applications would this be useful for? What can you build with an AI data science intern that's right 75% of the time? I have written a bunch of more or less complicated SQL during my career. And I am pretty sure that if I need to write a SQL statement that's anything but select * from table, my output won't work 75% of time. I may be special case, but typically if I work on a hard problem, it is not a single hard problem but a sh*tload of connected simple problems. If I can get someone to solve the simple problems 75% of the time correctly so that I can spend my time figuring out how those simple problems are to be connected, I'm ore than happy. And that's exactly how I use chatgpt. I have learned not to ask too complex questions from it. But the simple ones, it mostly aces and when it does not , they are easy to spot, as it is not that I could not have solved them myself, I just did not want to spend time for that. Now, if only the chatgpt was not almost as lazy as me to produce long simple stuff, that would be awesome.

bagels 3y ago

I know sql pretty well, but every once in a while doing things like percentiles come up where I don't remember the exact incantations to get the window syntax right. Things like this can save some googling and documentation reading. I can read it to see if it makes sense, and execute it. I can create tests for the output to verify. Can save a lot of time. If it's wrong, I use the fallback of googling for examples.

fijiaarone 3y ago

The 25% of SQL that covers everything in 99% of applications is a subset of the 76.5% of SQL that the LLM got right.

lawxls 3y ago

Yep, just use GPT-4