6 ms·

Text-to-SQL is dead, long live text-to-SQL

62 points by exagolo 11mo ago

XCSme 11mo ago
It's hit or miss, but for my product type it worked quite well: https://docs.uxwizz.com/guides/ask-ai https://docs.uxwizz.com/guides/ask-ai The problem is that often once it starts failing, it rarely recovers from an error, even given the exact error code. In my dashboard, I added OpenRouter access, so you can quickly swap between models and see which one works best (for your desired accuracy/response time/pricing). Grok 4 Fast is one of the best overall.
deleted 11mo ago
[deleted]
lateforwork 11mo ago
This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential.
- disgruntledphd2 11mo ago
  > This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential. For most businesses, it really really is. In general, people (businesses) are incredibly sensitive about any possibility of data leakage, even just the metadata. There are lots of companies who would pay for this, and they tend to have a lot of money.
  - munk-a 11mo ago
    This would be a (small, granted) red flag for me about a company. As a data architect I work really hard to make sure our model is legible and obvious - the way which the model is adopted to our problem area is certainly a difficult problem that is valuable to have solved. But minor details of that schema should be extremely boring.
    - disgruntledphd2 11mo ago
      Like, I've never really been sure why people care about this so much. However, when I worked for a data observability company where we took in and did work on a lot of this metadata, it became apparent that lots of our customers cared.
memelang 11mo ago
Long term, SQL will be phased out for new LLM-friendly query languages. The cost of AI generating tens of billions of queries per day creates a new incentive for rationality and efficiency that overcomes the human institutional weight of legacy languages. I'm working on such a language now: https://memelang.net/ https://memelang.net/
- throaway54321 11mo ago
  Why would it be phased out? This doesn’t track. If anything SQL is stickier than ever since LLMs are actually pretty good at generating it.
- lee-rhapsody 11mo ago
  That doesn't make sense. LLMs will be used to write SQL, I'm sure (I do that) but replacing SQL?
  - exagolo 11mo ago
    I do agree. SQL is simply an access API for so many systems, and nice as it's a declarative language rather than a normal programming language. LLMs are super powerful to express questions to data that can then be translated into SQL.
- dcreater 11mo ago
  Ah ournfuture where we vibe code memelang
- ghc 11mo ago
  If history is any indication, you've got it backwards: it's more likely that LLMs will be phased out for SQL.
  - Zambyte 11mo ago
    What history are you considering?
dcreater 11mo ago
Horrible title and premise
bob1029 11mo ago
Text-to-SQL is effectively pissing in the wind when we start looking at business value. The trouble is that it works just enough to be dangerous. It is a very interesting idea and can easily consume the resources of a technical team ~indefinitely. One obvious sign we are chasing shiny here is the German-to-SQL example. This is fun, but no one would actually pay for it. I would classify Text-to-SQL as a Schedule I rabbit hole. It has some serious effects, but it otherwise has no acceptable level of value-add in most reasonable enterprises.
- nonethewiser 11mo ago
  >The trouble is that it works just enough to be dangerous. This is a major theme with LLMs. When they first came out you'd see it randomly returning garbage in the middle of an otherwise good output maybe 30% of the time. You knew you had to go through it with a fine tooth comb. Now it's more like 3%. And you just gloss over it. I think you are right about Text-to-SQL being a trap. In this case the deficiencies are unaccaptable. But elsewhere? I think we are going to see the "customer service effect" applied all over the place. I am referring to the downward trend of customer service. Where quality was eschewed for scale. We went from highly competent agents providing individual feedback to hard to reach agents with no agency. It scales, but the bar has been lowered significantly. You get less customer service. I think AI begs for the same tradeoff. Settle for less because it makes some things easier. Which of course makes other things much more complex or even undermines the entire premise.
  - zurfer 11mo ago
    Reliability is the important dimension to focus on, but what is your baseline? I've worked in data and me and my colleagues regularly had to confront bugs (on many levels) that were communicated to end users
- zurfer 11mo ago
  Germany is the 5th biggest economy in the world, then there is Austria and Switzerland. The claim that nobody would pay for German to SQL seems a bit pessimistic ;) I'd also love to understand better why you think that there is no "acceptable level of value-add in most reasonable enterprises".
sschnei8 11mo ago
What is the human need to bash SQL at all costs? At its core, such a simple syntax, yet its so powerful at aggregating/manipulating tabular data and the like. Instead we’d rather declaratively say what we want in a more verbose/disjointed way… fascinating
- reverseindex 11mo ago
  I used to use SQL very extensively, but then I moved into different roles, and now I always have to look up its syntax, stuff like "how did HAVING work again?", "how do I search for a date two days before today". With Text-to-SQL I can express my query in natural language. Because I have a very solid understanding of relational data models I can craft my prompt very precisely, but without constantly having to go back to the documentation. So at least for casual users it's very helpful.
- dvdkon 11mo ago
  I'll give one reason of many: Its fixed pipeline model doesn't match how humans think about data transformation.
- nonethewiser 11mo ago
  I'm with you. The more SQL alternatives I've seen, the more I'm convinced that SQL doesn't have that much fat or disfunction in it's syntax. There is inherent complexity. Follow an SQL-killer's guide far enough and you will see them quietly reintroduce the very same abstractions SQL solved decades ago.
- johtso 11mo ago
  When you're having to work with someone else's godawful convoluted database schema you're sometimes really happy not to have to write the query yourself. Giving the LLM context with previously written queries can be quite effective. The writing SQL experience is a product of both SQL's syntax, the structure of the database you're querying over and the complexity of your query. When things get hairy, and you have a good number of representative queries already written that you can use as context, LLMs can be really nice tool.
ErroneousBosh 11mo ago
But SQL is text, that's the whole point of it. Sure, you have to phrase your question in a way that's a bit like trying to ask a very specific question of an annoying "Self-Diagnosed Internet Autistic" co-worker who can't tell the difference between being "precise" and being "a pedantic pain in the arse", but it is just text. Oh you're upset because SQL isn't in German? Well there's no reason why you can't stick German into the lexer, set your columns up with German names, and get a query like WAHLEN_SIE zeielen_id, benutzer_namen, eingetragen AUS benutzern WO aktiviert = WAHR SORTIEREN NACH registrierungs_datum; people who can really speak German, ja ich weiss meine Deutsch is so schlect, geh schon ;-) But really why would you bother?
- xwowsersx 11mo ago
  Well, of course you're correct that SQL is text, but that's not what the article is arguing about. The point isn't whether SQL is text... it's about the kind of text it is. SQL is a formal language, not a natural one. It's precise, rigid, and requires a specialized understanding of schema, joins, and logic. text-to-sql systems don't exist because people are too lazy to type; they exist because most people can't fluently express analytical intent in sql syntax. They can describe what they want in natural language ("show me all active users who registerd this year"), but translating that into correct, optimized sql requires at least familiarity, and sometimes expertise So the governance challenges discussed in the article aren't about "oh SQL is too hard to type"...they're about trust, validation, and control when you introduce an AI intermediary that converts natural lang into a query that might affect sensitive data
  - munk-a 11mo ago
    I think it's important to reinforce that SQL isn't an COBOL like attempt at building a querying language out of natural expressions (which you could see if you squint really hard). Instead SQL is a refinement of various querying languages (hence being the standard one) that co-evolved with relational algebra. If you have a chance to learn more within an academic environment courses on relational algebra and the abstract theory of set operations can be invaluable to building a basis for more naturally understanding the intent and tools available in SQL.
AvAn12 11mo ago
aren't programming languages like SQL designed to strike a quasi-optimal balance between precision and economy? that is to say, if I want to use plain English to specify some kind of data operations, the prompt will necessarily be more verbose and/or less specific than SQL statements. So why not just learn SQL properly? And then learn the domain-specific considerations of the source data as well as the domain-specific business requirements? Or am I missing something essential?
- zwaps 11mo ago
  Yes you are. I can show you 15 line SQL queries you would be hard pressed to understand, let alone built with precision as a novice And you want everyone to learn this? People who don’t even have time or ability to master excel? Really?
  - da_chicken 11mo ago
    It's hard because relational algebra is hard. It's hard because it's easy to write a query that returns arbitrary data, but it's hard to write a query that returns only the correct data that you really want. The problem is that, IMX, AI is just as shit at that as most humans are. AI can find and connect primary keys and columns with the same name. It doesn't understand cardinality of entities or how to look for data that breaks the data model of the application (which is a critical element of data validation). None of the actual hard parts of writing SQL go away with AI.
- da_chicken 11mo ago
  For all it's flaws, I certainly prefer SQL to something like XQuery.
jinjin2 11mo ago
The original idea idea behind SQL was to create a language that looked like English and allowed regular users to express their queries in something that resembled natur Al language. Naturally it has evolved into something far more complex, but maybe today with LLMs it can get back to its origin, LLMs are getting pretty good at writing SQL. There is so much training material out there, and it is not that hard to validate the results. The real interesting question will be if they will be better at leveraging all the database specific dialects than tool like PowerBI. High-performance databases like Exasol often has a lot of specific features in their SQL dialects that generic tools and ORMs are not able to use, it will be interesting to see if LLMs can make that more accessible.
maxdemarzi 11mo ago
The current benchmark for real world text to sql is https://spider2-sql.github.io/ https://spider2-sql.github.io/ and the SOTA is at 64% Anyone selling you 99% accuracy can prove it there first.
- XCSme 11mo ago
  I don't see Grok there
- zurfer 11mo ago
  Cofounder of one of those analytics agents here (https://getdot.ai https://getdot.ai). The promise of the technology is not that it can deal with any arbitrarily complex Enterprise setup, but rather that you expose it with enough guidance on a controlled and sufficiently good data model. Depending on your use case this can be super valuable as it enables a lot more people to use data and get relevant recommendations. But yeah it's work to make nice roads and put up signs everywhere.
  - macinjosh 11mo ago
    I am working with Databricks' Genies. I have a _very_ complex Enterprise data schema(s). Genies, and from what I can tell, your product work on a set of tables ~20 and expect a well thought out and documented data model. I have hundreds of tables designed by several different teams. I do have decent documentation on the tables but if I had a nice, organized data model I wouldn't need an AI assistant. If I had a perfect data model my team could write simple SQL queries or give chatgpt a schema dump + a natural language query and it would get the answer most of the time. IMHO, the big value in this space will be when these tools can wrangle realistic databases.
    - zurfer 11mo ago
      Well, I can't comment much on Genie, but the core question is always how you scale the complexity. In Dot, it's divide and conquer. If you have several different teams each of them has to maintain their knowledge base. A bunch of our customer have less than 10 tables hooked up to Dot, but this data is core to their business and so the analytics agent is really useful. Our most complex setup is on more than 5000 tables, but that was a lot more work to lay out the structure and guidelines. Also, I don't think all organization are ready for AI. If the data model is a huge mess, data quality is poor and analytics use cases are not mature, it's better to focus on the fundamentals first.
ROSEDOLA 11mo ago
[dead]
fijiaarone 11mo ago
Instead of trying to get LLMs from zero to 100, which is impossible to do, you should concentrate on getting them from 75% to 95%. The idea that someone who has no knowledge of the domain and no understanding of relational data modeling can chat “like to know really good um but don’t make any mistakes” with a GPU an uncover the mysteries of the universe is imbecilic. But… someone what knows approximately what to do and sort of how to do it could work wonders — if we had LLMs trained on a corpus with specific rules. I don’t know how to left join and what table I need to get the aggregate of sales in each region by date and price range, but I can describe it halfway and know how to check if each step is valid. LLMs can do this. They’re trained on English, and they are able to weight definitive rules. But instead we throw a random text at a general purpose transformer. Parsing a response of tokens into grammatical English is the most expensive computation (after the initial scraping and catalog). Instead of wasting all the cycles doing that against the sun total of GitHub, StackOverflow, Reddit, and Wikipedia, create a fuzzy match on a simplification a rigorous specification and train it on your data (just a few million tokens) to teach it that users have primary addresses and are associated to accounts that have regions and region X has roughly 10 times the sales volume of region y. So someone intelligent in the matter with an understanding of logical rigor and a general idea of the data shape can actually become 10x more efficient, instead of trying to lift vibe coders to the level of Spakespearean monkeys, you could be turning mid-level devs into super analysts.
icu5000 11mo ago
Cool! I need this at work XD

62 points by exagolo 11mo ago

XCSme 11mo ago

It's hit or miss, but for my product type it worked quite well: https://docs.uxwizz.com/guides/ask-ai https://docs.uxwizz.com/guides/ask-ai The problem is that often once it starts failing, it rarely recovers from an error, even given the exact error code. In my dashboard, I added OpenRouter access, so you can quickly swap between models and see which one works best (for your desired accuracy/response time/pricing). Grok 4 Fast is one of the best overall.

deleted 11mo ago

[deleted]

lateforwork 11mo ago

This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential.

disgruntledphd2 11mo ago
> This is solving a problem few people have. They assume your table name, column names and relationships is confidential information. For most people such metadata is not confidential. For most businesses, it really really is. In general, people (businesses) are incredibly sensitive about any possibility of data leakage, even just the metadata. There are lots of companies who would pay for this, and they tend to have a lot of money.
- munk-a 11mo ago
  This would be a (small, granted) red flag for me about a company. As a data architect I work really hard to make sure our model is legible and obvious - the way which the model is adopted to our problem area is certainly a difficult problem that is valuable to have solved. But minor details of that schema should be extremely boring.
  - disgruntledphd2 11mo ago
    Like, I've never really been sure why people care about this so much. However, when I worked for a data observability company where we took in and did work on a lot of this metadata, it became apparent that lots of our customers cared.

memelang 11mo ago

Long term, SQL will be phased out for new LLM-friendly query languages. The cost of AI generating tens of billions of queries per day creates a new incentive for rationality and efficiency that overcomes the human institutional weight of legacy languages. I'm working on such a language now: https://memelang.net/ https://memelang.net/

throaway54321 11mo ago
Why would it be phased out? This doesn’t track. If anything SQL is stickier than ever since LLMs are actually pretty good at generating it.
lee-rhapsody 11mo ago
That doesn't make sense. LLMs will be used to write SQL, I'm sure (I do that) but replacing SQL?
- exagolo 11mo ago
  I do agree. SQL is simply an access API for so many systems, and nice as it's a declarative language rather than a normal programming language. LLMs are super powerful to express questions to data that can then be translated into SQL.
dcreater 11mo ago
Ah ournfuture where we vibe code memelang
ghc 11mo ago
If history is any indication, you've got it backwards: it's more likely that LLMs will be phased out for SQL.
- Zambyte 11mo ago
  What history are you considering?

dcreater 11mo ago

Horrible title and premise

bob1029 11mo ago

Text-to-SQL is effectively pissing in the wind when we start looking at business value. The trouble is that it works just enough to be dangerous. It is a very interesting idea and can easily consume the resources of a technical team ~indefinitely. One obvious sign we are chasing shiny here is the German-to-SQL example. This is fun, but no one would actually pay for it. I would classify Text-to-SQL as a Schedule I rabbit hole. It has some serious effects, but it otherwise has no acceptable level of value-add in most reasonable enterprises.

nonethewiser 11mo ago
>The trouble is that it works just enough to be dangerous. This is a major theme with LLMs. When they first came out you'd see it randomly returning garbage in the middle of an otherwise good output maybe 30% of the time. You knew you had to go through it with a fine tooth comb. Now it's more like 3%. And you just gloss over it. I think you are right about Text-to-SQL being a trap. In this case the deficiencies are unaccaptable. But elsewhere? I think we are going to see the "customer service effect" applied all over the place. I am referring to the downward trend of customer service. Where quality was eschewed for scale. We went from highly competent agents providing individual feedback to hard to reach agents with no agency. It scales, but the bar has been lowered significantly. You get less customer service. I think AI begs for the same tradeoff. Settle for less because it makes some things easier. Which of course makes other things much more complex or even undermines the entire premise.
- zurfer 11mo ago
  Reliability is the important dimension to focus on, but what is your baseline? I've worked in data and me and my colleagues regularly had to confront bugs (on many levels) that were communicated to end users
zurfer 11mo ago
Germany is the 5th biggest economy in the world, then there is Austria and Switzerland. The claim that nobody would pay for German to SQL seems a bit pessimistic ;) I'd also love to understand better why you think that there is no "acceptable level of value-add in most reasonable enterprises".

sschnei8 11mo ago

What is the human need to bash SQL at all costs? At its core, such a simple syntax, yet its so powerful at aggregating/manipulating tabular data and the like. Instead we’d rather declaratively say what we want in a more verbose/disjointed way… fascinating

reverseindex 11mo ago
I used to use SQL very extensively, but then I moved into different roles, and now I always have to look up its syntax, stuff like "how did HAVING work again?", "how do I search for a date two days before today". With Text-to-SQL I can express my query in natural language. Because I have a very solid understanding of relational data models I can craft my prompt very precisely, but without constantly having to go back to the documentation. So at least for casual users it's very helpful.
dvdkon 11mo ago
I'll give one reason of many: Its fixed pipeline model doesn't match how humans think about data transformation.
nonethewiser 11mo ago
I'm with you. The more SQL alternatives I've seen, the more I'm convinced that SQL doesn't have that much fat or disfunction in it's syntax. There is inherent complexity. Follow an SQL-killer's guide far enough and you will see them quietly reintroduce the very same abstractions SQL solved decades ago.
johtso 11mo ago
When you're having to work with someone else's godawful convoluted database schema you're sometimes really happy not to have to write the query yourself. Giving the LLM context with previously written queries can be quite effective. The writing SQL experience is a product of both SQL's syntax, the structure of the database you're querying over and the complexity of your query. When things get hairy, and you have a good number of representative queries already written that you can use as context, LLMs can be really nice tool.

ErroneousBosh 11mo ago

But SQL is text, that's the whole point of it. Sure, you have to phrase your question in a way that's a bit like trying to ask a very specific question of an annoying "Self-Diagnosed Internet Autistic" co-worker who can't tell the difference between being "precise" and being "a pedantic pain in the arse", but it is just text. Oh you're upset because SQL isn't in German? Well there's no reason why you can't stick German into the lexer, set your columns up with German names, and get a query like WAHLEN_SIE zeielen_id, benutzer_namen, eingetragen AUS benutzern WO aktiviert = WAHR SORTIEREN NACH registrierungs_datum; people who can really speak German, ja ich weiss meine Deutsch is so schlect, geh schon ;-) But really why would you bother?

xwowsersx 11mo ago
Well, of course you're correct that SQL is text, but that's not what the article is arguing about. The point isn't whether SQL is text... it's about the kind of text it is. SQL is a formal language, not a natural one. It's precise, rigid, and requires a specialized understanding of schema, joins, and logic. text-to-sql systems don't exist because people are too lazy to type; they exist because most people can't fluently express analytical intent in sql syntax. They can describe what they want in natural language ("show me all active users who registerd this year"), but translating that into correct, optimized sql requires at least familiarity, and sometimes expertise So the governance challenges discussed in the article aren't about "oh SQL is too hard to type"...they're about trust, validation, and control when you introduce an AI intermediary that converts natural lang into a query that might affect sensitive data
- munk-a 11mo ago
  I think it's important to reinforce that SQL isn't an COBOL like attempt at building a querying language out of natural expressions (which you could see if you squint really hard). Instead SQL is a refinement of various querying languages (hence being the standard one) that co-evolved with relational algebra. If you have a chance to learn more within an academic environment courses on relational algebra and the abstract theory of set operations can be invaluable to building a basis for more naturally understanding the intent and tools available in SQL.

AvAn12 11mo ago

aren't programming languages like SQL designed to strike a quasi-optimal balance between precision and economy? that is to say, if I want to use plain English to specify some kind of data operations, the prompt will necessarily be more verbose and/or less specific than SQL statements. So why not just learn SQL properly? And then learn the domain-specific considerations of the source data as well as the domain-specific business requirements? Or am I missing something essential?

zwaps 11mo ago
Yes you are. I can show you 15 line SQL queries you would be hard pressed to understand, let alone built with precision as a novice And you want everyone to learn this? People who don’t even have time or ability to master excel? Really?
- da_chicken 11mo ago
  It's hard because relational algebra is hard. It's hard because it's easy to write a query that returns arbitrary data, but it's hard to write a query that returns only the correct data that you really want. The problem is that, IMX, AI is just as shit at that as most humans are. AI can find and connect primary keys and columns with the same name. It doesn't understand cardinality of entities or how to look for data that breaks the data model of the application (which is a critical element of data validation). None of the actual hard parts of writing SQL go away with AI.
da_chicken 11mo ago
For all it's flaws, I certainly prefer SQL to something like XQuery.

jinjin2 11mo ago

The original idea idea behind SQL was to create a language that looked like English and allowed regular users to express their queries in something that resembled natur Al language. Naturally it has evolved into something far more complex, but maybe today with LLMs it can get back to its origin, LLMs are getting pretty good at writing SQL. There is so much training material out there, and it is not that hard to validate the results. The real interesting question will be if they will be better at leveraging all the database specific dialects than tool like PowerBI. High-performance databases like Exasol often has a lot of specific features in their SQL dialects that generic tools and ORMs are not able to use, it will be interesting to see if LLMs can make that more accessible.

maxdemarzi 11mo ago

The current benchmark for real world text to sql is https://spider2-sql.github.io/ https://spider2-sql.github.io/ and the SOTA is at 64% Anyone selling you 99% accuracy can prove it there first.

XCSme 11mo ago
I don't see Grok there
zurfer 11mo ago
Cofounder of one of those analytics agents here (https://getdot.ai https://getdot.ai). The promise of the technology is not that it can deal with any arbitrarily complex Enterprise setup, but rather that you expose it with enough guidance on a controlled and sufficiently good data model. Depending on your use case this can be super valuable as it enables a lot more people to use data and get relevant recommendations. But yeah it's work to make nice roads and put up signs everywhere.
- macinjosh 11mo ago
  I am working with Databricks' Genies. I have a _very_ complex Enterprise data schema(s). Genies, and from what I can tell, your product work on a set of tables ~20 and expect a well thought out and documented data model. I have hundreds of tables designed by several different teams. I do have decent documentation on the tables but if I had a nice, organized data model I wouldn't need an AI assistant. If I had a perfect data model my team could write simple SQL queries or give chatgpt a schema dump + a natural language query and it would get the answer most of the time. IMHO, the big value in this space will be when these tools can wrangle realistic databases.
  - zurfer 11mo ago
    Well, I can't comment much on Genie, but the core question is always how you scale the complexity. In Dot, it's divide and conquer. If you have several different teams each of them has to maintain their knowledge base. A bunch of our customer have less than 10 tables hooked up to Dot, but this data is core to their business and so the analytics agent is really useful. Our most complex setup is on more than 5000 tables, but that was a lot more work to lay out the structure and guidelines. Also, I don't think all organization are ready for AI. If the data model is a huge mess, data quality is poor and analytics use cases are not mature, it's better to focus on the fundamentals first.

ROSEDOLA 11mo ago

[dead]

fijiaarone 11mo ago

Instead of trying to get LLMs from zero to 100, which is impossible to do, you should concentrate on getting them from 75% to 95%. The idea that someone who has no knowledge of the domain and no understanding of relational data modeling can chat “like to know really good um but don’t make any mistakes” with a GPU an uncover the mysteries of the universe is imbecilic. But… someone what knows approximately what to do and sort of how to do it could work wonders — if we had LLMs trained on a corpus with specific rules. I don’t know how to left join and what table I need to get the aggregate of sales in each region by date and price range, but I can describe it halfway and know how to check if each step is valid. LLMs can do this. They’re trained on English, and they are able to weight definitive rules. But instead we throw a random text at a general purpose transformer. Parsing a response of tokens into grammatical English is the most expensive computation (after the initial scraping and catalog). Instead of wasting all the cycles doing that against the sun total of GitHub, StackOverflow, Reddit, and Wikipedia, create a fuzzy match on a simplification a rigorous specification and train it on your data (just a few million tokens) to teach it that users have primary addresses and are associated to accounts that have regions and region X has roughly 10 times the sales volume of region y. So someone intelligent in the matter with an understanding of logical rigor and a general idea of the data shape can actually become 10x more efficient, instead of trying to lift vibe coders to the level of Spakespearean monkeys, you could be turning mid-level devs into super analysts.

icu5000 11mo ago

Cool! I need this at work XD