4 ms·

I’ve seen impressive demos with just LLM direct to SQL. Can’t wait to see what people build with these new tools.

• aazo11 3y ago
  Hi Dylan -- new GPT-4 class LLMs have gotten good at writing correct SQL, but while the SQL they generate almost always executes correctly they often write SQL that generates incorrect answers to the question. Some reasons for this can be the business context or definition is not in the table schema, or if the correct query is complex and requires multiple joins.
  • verdverm 3y ago
    It's also because of the inherent nature and hallucinations in LLMs, likely impossible to remove completely, always double check the LLM's work
    • aazo11 3y ago
      Totally agree. In the hosted version built on top of the engine we block the answer from going to the question asker until an authorized user 'verifies' the answer. However these 'verified' answers are then stored in the context store and retrieved for few shot prompting.
  • thom 3y ago
    Yeah, you really shouldn’t be building this (or really any sort of important or destructive interface) with LLMs unless you have an intermediate representation which users can inspect and understand. Even if the understanding of English were perfect there are still ambiguities you’d need to sort out. You also need to be able to suggest something useful if the user asks a nonsense question (for example data that doesn’t exist in your schema or its contents). When I worked on this a decade ago we were quite good at mapping unknown queries to various suggestions drawn from a canonical subset of English for which we had very explicit semantics. But obviously the Soave of unknowns was huge. I’ve no doubt the AIs will reach human parity at some point (and therefore still make mistakes!) but I’d be terrified deploying any sort of one-shot text-to-results pipeline today.