5 ms·

Prof Murat is calling for examples of how LLMs can help make DBs more usable. I note that Julius.ai is making data analytics easier - just upload some data and

Prof Murat is calling for examples of how LLMs can help make DBs more usable. I note that Julius.ai is making data analytics easier - just upload some data and ask for some charts and it does an ok job. The problem of going from natural language to SQL is still a far from solved problem. The main benchmark for this is Bird-Bench and accuracy is only in the high 60s - https://bird-bench.github.io/ https://bird-bench.github.io/ I have been working on making tables queryable using function calling, where you a fixed number of canned queries based on the primary key and an event-time column. PyData talk on this - https://www.youtube.com/watch?v=dRcjTe5qgwM https://www.youtube.com/watch?v=dRcjTe5qgwM

randomdata 2y ago
> The problem of going from natural language to SQL is still a far from solved problem. Is there any improvement going from natural language to something other than SQL? Certainly SQL can be cut out of the picture if it is what confuses these systems.
- zerodensity 2y ago
  What would that "something other" be? If the goal is to talk to a SQL database the output from the LLM would benefit from being SQL.
  - randomdata 2y ago
    > What would that "something other" be? Depends. Where is the problem? Is it the quirkiness of SQL? Perhaps something like QUEL or Datalog would yield better results? Is it declarative programming that it struggles with? GPT, for example, seems to be optimized for Python in particular. Perhaps an imperative approach is easier for the LLM to understand? It doesn't even have to be a language suitable for humans. Perhaps it would fare better with something like SQLite's byte code language? > If the goal is to talk to a SQL database While being able to talk to an existing SQL database would no doubt simplify the problem in a lot of cases, which is of merit, I doubt that is the actual goal. The user doesn't care about the technology, as they say. Getting the expected results out of the database is undoubtedly the actual goal. SQL as a target is all well and good if it works reliably, but the claim was that it doesn't. If some other target performs better, there is no need to cling to SQL. It is merely an implementation detail.
    - big_whack 2y ago
      I think the problem is the quirkiness on the English side, not the SQL side. You could translate datalog to SQL or vice versa, but understanding intention from arbitrary english is much harder. And often query results must be 100% accurate and reliable.
      - randomdata 2y ago
        > I think the problem is the quirkiness on the English side While likely, the question asked if there was any improvement shown with other targets to validate that assumption. There is no benefit in thinking. > And often query results must be 100% accurate and reliable. It seems that is impossible. Even the human programmers struggle to reliably convert natural language to SQL according to the aforementioned test study. They are slightly better than the known alternatives, but far from perfect. But if another target can get closer to human-level performance, that is significant.
        yuliyp 2y ago
        When I find someone claiming a suspicious data analysis result I can ask them for the SQL and investigate it to see if there's a bug in it (or further investigate where the data being queried comes from). If the abstraction layer between LLM prompt and data back is removed, I'm left with (just like other LLM answers) some words but no way to know if they're correct.
        randomdata 2y ago
        1. How would the abstraction be removed? Language generation is what LLMs do; a language abstraction is what you are getting out, no matter what. There is no magic involved. 2. The language has to represent a valid computer program. That is as true of SQL as any other target. You can know that it is correct by reading it.
        big_whack 2y ago
        Once you have SQL, you have datalog. Once you have datalog, you have SQL. The problem isn't the target, it is getting sufficiently rigorous and structured output from the LLM to target anything.
7thpower 2y ago
This was basically the only reasonable way I found to create a consistent user experience. I think of them as natural language BI, where you have canned reports to answer common categories of questions.
jalcazar 2y ago
Gemini generating SQL queries from natural language could be an example of AI making DBs more usable. There is more people speaking natural language than SQL https://cloud.google.com/bigquery/docs/write-sql-gemini#prompt_to_generate_sql_queries https://cloud.google.com/bigquery/docs/write-sql-gemini#prom...
- jamesblonde 2y ago
  Bird-bench has gemini on 69.03% on the test set. That is a long way from something you can build on.

randomdata 2y ago

> The problem of going from natural language to SQL is still a far from solved problem. Is there any improvement going from natural language to something other than SQL? Certainly SQL can be cut out of the picture if it is what confuses these systems.

zerodensity 2y ago
What would that "something other" be? If the goal is to talk to a SQL database the output from the LLM would benefit from being SQL.
- randomdata 2y ago
  > What would that "something other" be? Depends. Where is the problem? Is it the quirkiness of SQL? Perhaps something like QUEL or Datalog would yield better results? Is it declarative programming that it struggles with? GPT, for example, seems to be optimized for Python in particular. Perhaps an imperative approach is easier for the LLM to understand? It doesn't even have to be a language suitable for humans. Perhaps it would fare better with something like SQLite's byte code language? > If the goal is to talk to a SQL database While being able to talk to an existing SQL database would no doubt simplify the problem in a lot of cases, which is of merit, I doubt that is the actual goal. The user doesn't care about the technology, as they say. Getting the expected results out of the database is undoubtedly the actual goal. SQL as a target is all well and good if it works reliably, but the claim was that it doesn't. If some other target performs better, there is no need to cling to SQL. It is merely an implementation detail.
  - big_whack 2y ago
    I think the problem is the quirkiness on the English side, not the SQL side. You could translate datalog to SQL or vice versa, but understanding intention from arbitrary english is much harder. And often query results must be 100% accurate and reliable.
    - randomdata 2y ago
      > I think the problem is the quirkiness on the English side While likely, the question asked if there was any improvement shown with other targets to validate that assumption. There is no benefit in thinking. > And often query results must be 100% accurate and reliable. It seems that is impossible. Even the human programmers struggle to reliably convert natural language to SQL according to the aforementioned test study. They are slightly better than the known alternatives, but far from perfect. But if another target can get closer to human-level performance, that is significant.
      - yuliyp 2y ago
        When I find someone claiming a suspicious data analysis result I can ask them for the SQL and investigate it to see if there's a bug in it (or further investigate where the data being queried comes from). If the abstraction layer between LLM prompt and data back is removed, I'm left with (just like other LLM answers) some words but no way to know if they're correct.
        randomdata 2y ago
        1. How would the abstraction be removed? Language generation is what LLMs do; a language abstraction is what you are getting out, no matter what. There is no magic involved. 2. The language has to represent a valid computer program. That is as true of SQL as any other target. You can know that it is correct by reading it.
      - big_whack 2y ago
        Once you have SQL, you have datalog. Once you have datalog, you have SQL. The problem isn't the target, it is getting sufficiently rigorous and structured output from the LLM to target anything.

7thpower 2y ago

This was basically the only reasonable way I found to create a consistent user experience. I think of them as natural language BI, where you have canned reports to answer common categories of questions.

jalcazar 2y ago

Gemini generating SQL queries from natural language could be an example of AI making DBs more usable. There is more people speaking natural language than SQL https://cloud.google.com/bigquery/docs/write-sql-gemini#prompt_to_generate_sql_queries https://cloud.google.com/bigquery/docs/write-sql-gemini#prom...

jamesblonde 2y ago
Bird-bench has gemini on 69.03% on the test set. That is a long way from something you can build on.