48 ms·
> This becomes even more dangerous as we're entering the world of LLMs generating SQL[1] I am beginning to push back against the idea of LLMs authoring SQL. De
by bob1029 3y ago
> This becomes even more dangerous as we're entering the world of LLMs generating SQL[1]
I am beginning to push back against the idea of LLMs authoring SQL. Determinism is really important for most realistic use cases. SQL is one of those "all or nothing" experiences. Pretty much the last thing you want to have a temperature/spiciness slider over.
Granted, if your use case is to simply memorize a handful of canonical queries and to provide a UX-friendly way for business administrators to retrieve these queries, then have at it.
But, if you are hoping to ask the LLM to write novel, sophisticated, domain-specific queries involving joins across 10+ tables, aggregations, recursion, windowing, etc., you are going to have a very bad time. In my experience, this is what everyone is dreaming about doing, and I think it is an unproductive fantasy at this point. Fine tuning will not give you what you seek, but don't let me stop you from trying. We burned over 200 hours on it with nothing to show for our time.
For me, LLMs are far more interesting for determining the user's intentions than emitting a final query. Classification seems like the actual superpower here. Classification can feed a very powerful deterministic query building engine that will actually give you what you want and you can prove it will work. It just takes a lot more tedious work than most humans are willing to endure, so here we are talking about various shortcuts.
- Supply5411 3y ago>We burned over 200 hours on it with nothing to show for our time. Can you talk more about this? I'm working in this space now, but not from the "make an LLM give you the right query" perspective, but from the "make the LLM-produced query be safe" perspective[1] 1. https://docs.heimdallm.ai/en/main/blog/posts/safe-sql-execution.html https://docs.heimdallm.ai/en/main/blog/posts/safe-sql-execut...
- bob1029 3y ago> "make an LLM give you the right query" > "make the LLM-produced query be safe" I think these are the same problem.
- Supply5411 3y agoThey're not. A query produced by an LLM may do exactly what the user asks for, but it is not safe to execute.
- bob1029 3y agoYou're right. Now that I'm thinking about it, making a query that works is less difficult than also proving it is safe. Good luck with your project.
- totalhack 3y agoHow about an LLM talking to a semantic layer that then writes the SQL? I added experimental support for that in a project of mine, but using a semantic layer makes most querying so simple that natural language based approaches just seem kinda silly unless you have interface constraints (like on a mobile device). https://github.com/totalhack/zillion https://github.com/totalhack/zillion