3 ms·

In my experience, LLMs are a lot better than my backend engineers who don't even try but not that much better than someone who's skimmed the docs. Typically th

by ellisv 2y ago

In my experience, LLMs are a lot better than my backend engineers who don't even try but not that much better than someone who's skimmed the docs. Typically they suggest 1) adding indexes and 2) refactoring the query. If you only provide the query then the model isn't aware of what indexes already exist. LLMs make assumptions about your data model that often don't hold (i.e. 3NF). Sometimes you have to give the LLM the latest docs because it's unaware of performance improvements and features for newer versions. In my view, it's better to look at the query plan and think yourself about how to improve the query but I also recognize that most people writing queries aren't going to do that. There are other tools (RIP OtterTune) that can tune database parameters. Not something I really see any generative model helping with.

gopalv 2y ago
> LLMs are a lot better than my backend engineers who don't even try but not that much better than someone who's skimmed the docs. LLMs are basically at the skill-set of "Googled something and tried it", which for a lot of basic things is mostly what everyone does. If you can loop back the results of the trial & error back to the model, then it does do a pretty good simulation of someone feeling their way through a problem. The loop is what makes the approach effective, but a model by itself cannot complete the process - it at least needs an agent which can try the recommendation (or a human who will pull the lever).
- sgarland 2y ago
  My main issue with the use of LLMs for problems you’re unfamiliar with is precisely this. Trial and error is a fine way to work through a problem; you learn well from failure. But if you’re copy/pasting from something else the entire time, you’re not the one failing, and you’re not the one reasoning about why something didn’t work. People do not learn this way, full stop. Anyone who thinks they are should try to write down, on paper, their understanding of a subject they think they have learned.
- swasheck 2y ago
  yes. my experience is that an LLM will sometimes suggest an interesting approach that breaks my pattern of thought enough to get me to a better solution. but the ones i primarily use (claude sonnet and chatgpt 4o) also seem to love suggesting CTEs which help readability but it depends on the RDBMS (and other things) as to whether or not they offer any real performance benefit. that’s a long way of saying that LLMs are a tool in this space but not yet a full solution, in my opinion and experience
sgarland 2y ago
> In my view, it's better to look at the query plan and think yourself about how to improve the query but I also recognize that most people writing queries aren't going to do that. That would require people to a. Learn SQL / relational algebra, one of the easiest languages in the world b. Read docs.

by ellisv 2y ago

gopalv 2y ago

> LLMs are a lot better than my backend engineers who don't even try but not that much better than someone who's skimmed the docs. LLMs are basically at the skill-set of "Googled something and tried it", which for a lot of basic things is mostly what everyone does. If you can loop back the results of the trial & error back to the model, then it does do a pretty good simulation of someone feeling their way through a problem. The loop is what makes the approach effective, but a model by itself cannot complete the process - it at least needs an agent which can try the recommendation (or a human who will pull the lever).

sgarland 2y ago
My main issue with the use of LLMs for problems you’re unfamiliar with is precisely this. Trial and error is a fine way to work through a problem; you learn well from failure. But if you’re copy/pasting from something else the entire time, you’re not the one failing, and you’re not the one reasoning about why something didn’t work. People do not learn this way, full stop. Anyone who thinks they are should try to write down, on paper, their understanding of a subject they think they have learned.
swasheck 2y ago
yes. my experience is that an LLM will sometimes suggest an interesting approach that breaks my pattern of thought enough to get me to a better solution. but the ones i primarily use (claude sonnet and chatgpt 4o) also seem to love suggesting CTEs which help readability but it depends on the RDBMS (and other things) as to whether or not they offer any real performance benefit. that’s a long way of saying that LLMs are a tool in this space but not yet a full solution, in my opinion and experience

sgarland 2y ago

> In my view, it's better to look at the query plan and think yourself about how to improve the query but I also recognize that most people writing queries aren't going to do that. That would require people to a. Learn SQL / relational algebra, one of the easiest languages in the world b. Read docs.