4 ms·

I have also seen ChatGPT going berserk yesterday, but in a different way. I have successfully used ChatGPT to convert an ORM query into an actual SQL query for

I have also seen ChatGPT going berserk yesterday, but in a different way. I have successfully used ChatGPT to convert an ORM query into an actual SQL query for performance trouble shooting. It mostly worked until yesterday when it start outputting garbage table names that weren't even present in the code. ChatGPT seemed to think the code is literature and was trying to write the sequel to it. The code style matches the original one so it took some head scratching to find out why those tables didn't exist.

kaptainscarlet 3y ago
Well, wouldn't the sequel be version 2?
rsynnott 3y ago
Okay, so I don’t really _get_ ChatGPT, but I’m particularly baffled by this usecase; why don’t you simply have your ORM tell you the query it is generating, rather than what a black box guesses it might be generating? Depends on the ORM, but generally you’ll just want to raise the log level.
- tomwphillips 3y ago
  Agree. Bizarre to use an LLM to do that. I wouldn’t be surprised if the LLM output wasn’t identical to the ORM-generated SQL.
  - rsynnott 3y ago
    I'd be very surprised if the LLM output is anything _like_ the ORM's, tbh, based on (at this point about a decade old; maybe things have improved) experience. ORMs cannot be trusted.
- mewpmewp2 3y ago
  Maybe didn't have the environment set up locally and did initial investigative work?
  - rsynnott 3y ago
    So, if I wanted to investigate ORM output and didn't have an appropriate environment set up, I would simply set one up. If you just want to see SQL output this should be trivial; clone the repo, install any dependencies, modify an integration test. What I would not do is ask a machine noted for its confident incorrectness to imagine what the ORM's output might be. Like, this is not doing investigative work. That’s not what ‘investigative’ means.
    - mewpmewp2 3y ago
      So imagine there is an urgent performance issue in production and you have a hunch that this SQL code may be the culprit. However before doing all of what you mentioned you want to verify it before following down a bad path. Maybe the environment setup could take few hours, maybe it is not a repo or codebase you are even familiar with. Typical in a large org. But if you know the SQL you will be able to run it raw to see if this causes it. Then maybe you can page the correct team to wake them up etc and fix it themselves.
      - rsynnott 3y ago
        But _you do not_ know the SQL. To be clear, ChatGPT will not be able to tell you what the ORM will generate. At best, it may tell you something that an ORM might plausibly generate. (If it's a _production_ issue, then you should talk to whoever runs your databases and ask them to look at their diagnostics; most DBMs will have a slow query log, for a start. You could also enable logging for a sample of traffic. There are all sorts of approaches likely to be more productive than _guessing_.)
        mewpmewp2 3y ago
        So I don't know what use-case exactly OP had, but all of your suggestions can potentially take hour or more and might depend on other people or systems you might not have access to. While with GPT you can get an answer in 10 seconds, and then potentially try out the query in the database yourself to see if it works or not. If it worked for him so far, it must've worked accurately enough. I would see this some sort of niche solution although OP seemed to indicate it's a recurrent thing they do. I have used ChatGPT for thousands of things, which are on the scale of like this, although I would mostly use if it's potentially an ORM I don't know anything about in a language I don't have experience with, e.g. to see if does some sort of JOIN underneath or does an IN query. If there was a performance issue to debug, then best case is that the query was problematic, and then when I run the GPTs generated query I will see that it was slow, so that's a signal to investigate it further.
- LandR 3y ago
  EVen if your ORM doesn't support this, you can always just turn on profiler on SQL and capture the actual query. SSMS has SQL Server PRofiler, i'm sure others have similar.
- kamray23 3y ago
  No, it's very close to useless. This is exactly the kind of thing that experienced developers talk about when they warn that inexperienced developers using ChatGPT could easily be a disaster. It's the attempt to use a LLM as a crystal ball to retrieve any information they could possibly want - including things it literally couldn't know or good recommendations for which direction to take an architecture. I'm certain there will be people who do stuff exactly like this and will have 'unsolvable' performance issues because of it and massive amounts of useless work as ChatGPT loves suggesting rewrites to convert good code to certain OO patterns (which don't necessarily suit projects) as a response to being asked what it thinks a good solution to a minor issue might be.
- oxfordmale 3y ago
  I am a Data Engineer and it would take me ages to spin up the service with the right log level to grab the query. It is much easier to grab it from the codebase. I used to do this manually from time to time, now I use ChatGPT.

kaptainscarlet 3y ago

Well, wouldn't the sequel be version 2?

rsynnott 3y ago

Okay, so I don’t really _get_ ChatGPT, but I’m particularly baffled by this usecase; why don’t you simply have your ORM tell you the query it is generating, rather than what a black box guesses it might be generating? Depends on the ORM, but generally you’ll just want to raise the log level.

tomwphillips 3y ago
Agree. Bizarre to use an LLM to do that. I wouldn’t be surprised if the LLM output wasn’t identical to the ORM-generated SQL.
- rsynnott 3y ago
  I'd be very surprised if the LLM output is anything _like_ the ORM's, tbh, based on (at this point about a decade old; maybe things have improved) experience. ORMs cannot be trusted.
mewpmewp2 3y ago
Maybe didn't have the environment set up locally and did initial investigative work?
- rsynnott 3y ago
  So, if I wanted to investigate ORM output and didn't have an appropriate environment set up, I would simply set one up. If you just want to see SQL output this should be trivial; clone the repo, install any dependencies, modify an integration test. What I would not do is ask a machine noted for its confident incorrectness to imagine what the ORM's output might be. Like, this is not doing investigative work. That’s not what ‘investigative’ means.
  - mewpmewp2 3y ago
    So imagine there is an urgent performance issue in production and you have a hunch that this SQL code may be the culprit. However before doing all of what you mentioned you want to verify it before following down a bad path. Maybe the environment setup could take few hours, maybe it is not a repo or codebase you are even familiar with. Typical in a large org. But if you know the SQL you will be able to run it raw to see if this causes it. Then maybe you can page the correct team to wake them up etc and fix it themselves.
    - rsynnott 3y ago
      But _you do not_ know the SQL. To be clear, ChatGPT will not be able to tell you what the ORM will generate. At best, it may tell you something that an ORM might plausibly generate. (If it's a _production_ issue, then you should talk to whoever runs your databases and ask them to look at their diagnostics; most DBMs will have a slow query log, for a start. You could also enable logging for a sample of traffic. There are all sorts of approaches likely to be more productive than _guessing_.)
      - mewpmewp2 3y ago
        So I don't know what use-case exactly OP had, but all of your suggestions can potentially take hour or more and might depend on other people or systems you might not have access to. While with GPT you can get an answer in 10 seconds, and then potentially try out the query in the database yourself to see if it works or not. If it worked for him so far, it must've worked accurately enough. I would see this some sort of niche solution although OP seemed to indicate it's a recurrent thing they do. I have used ChatGPT for thousands of things, which are on the scale of like this, although I would mostly use if it's potentially an ORM I don't know anything about in a language I don't have experience with, e.g. to see if does some sort of JOIN underneath or does an IN query. If there was a performance issue to debug, then best case is that the query was problematic, and then when I run the GPTs generated query I will see that it was slow, so that's a signal to investigate it further.
LandR 3y ago
EVen if your ORM doesn't support this, you can always just turn on profiler on SQL and capture the actual query. SSMS has SQL Server PRofiler, i'm sure others have similar.
kamray23 3y ago
No, it's very close to useless. This is exactly the kind of thing that experienced developers talk about when they warn that inexperienced developers using ChatGPT could easily be a disaster. It's the attempt to use a LLM as a crystal ball to retrieve any information they could possibly want - including things it literally couldn't know or good recommendations for which direction to take an architecture. I'm certain there will be people who do stuff exactly like this and will have 'unsolvable' performance issues because of it and massive amounts of useless work as ChatGPT loves suggesting rewrites to convert good code to certain OO patterns (which don't necessarily suit projects) as a response to being asked what it thinks a good solution to a minor issue might be.
oxfordmale 3y ago
I am a Data Engineer and it would take me ages to spin up the service with the right log level to grab the query. It is much easier to grab it from the codebase. I used to do this manually from time to time, now I use ChatGPT.