4 ms·

> it only makes you efficient if you have a good idea what the right answer will be. This very much aligns with my experiments using ChatGPT and Claude as a pa

> it only makes you efficient if you have a good idea what the right answer will be. This very much aligns with my experiments using ChatGPT and Claude as a pair or for conversations about approaches and interactive (lazy) documentation. They're both incorrect a significant amount of the time about things like AWS service features, function signatures, method names, etc. If you don't have existing familiarity or have a very efficient way of verifying what they suggest, etc. you'll wind up wasting lots of time. When you do detect an error, it's also possible to get into a cyclical loop of you correcting it and asking a follow up and it suggesting the initial, wrong solution over again. I also get a kick out of them saying things like, "Okay, the final answer is ..." while they then proceed to provide inaccurate information which results in subsequent final answers.

ktallett 2y ago
For those who actually want a usable output, it's not quicker than just learning the right answer or how to do something. On the other hand, if you just want to sound smart but don't care if you are right, for 75% of the time it's probably good enough.
- lxgr 2y ago
  Sure, instead of asking an LLM and finding a way to verify the solution and iterate towards a correct one if it seems inconsistent, I could also just study to become a domain expert in pretty much any field I'm mildly curious about. If it weren't for the pesky problem of mortality, that does sounds like a viable alternative. There is an infinite number of examples of problems for which LLMs are absolutely useless – as is the case for pretty much any other technology.
  - ktallett 2y ago
    You don't have to become an expert to ask a question and find an answer using existing methods such as the rest of the internet or traditional media such as books. Books and summaries exist already of pretty much any topic you are mildly curious about. You can get a good understanding of any subject in the world in a week or so if you wish to.
    - lxgr 2y ago
      > Books and summaries exist already of pretty much any topic you are mildly curious about. That's absolutely not the case for many questions I've had LLMs answer for me so far. Just as one example, I had one come up with a mathematical model for a specific retirement investment/taxation question I've had for years, and that is too niche to have any prior art publicly available. I've gotten as far as a spreadsheet with a vague approximation myself, with no way of validating my reasoning. I've asked an LLM to model the same thing, once symbolically and once numerically in Python, and to run the numbers for a few sample values, again symbolically and numerically. This I then compared to my own spreadsheet. Sure, I could have spent a few weeks more of studying finance textbooks or paid an accountant a few thousand dollars to do so, but it just wouldn't have been worth it to me. Big qualitative differences matter too sometimes. > You can get a good understanding of any subject in the world in a week or so if you wish to. Yes, but I don't have an infinite number of weeks in my life, nor an infinite number of motivated interns I can send out to search the web and sift through the results for me.

ktallett 2y ago

For those who actually want a usable output, it's not quicker than just learning the right answer or how to do something. On the other hand, if you just want to sound smart but don't care if you are right, for 75% of the time it's probably good enough.

lxgr 2y ago
Sure, instead of asking an LLM and finding a way to verify the solution and iterate towards a correct one if it seems inconsistent, I could also just study to become a domain expert in pretty much any field I'm mildly curious about. If it weren't for the pesky problem of mortality, that does sounds like a viable alternative. There is an infinite number of examples of problems for which LLMs are absolutely useless – as is the case for pretty much any other technology.
- ktallett 2y ago
  You don't have to become an expert to ask a question and find an answer using existing methods such as the rest of the internet or traditional media such as books. Books and summaries exist already of pretty much any topic you are mildly curious about. You can get a good understanding of any subject in the world in a week or so if you wish to.
  - lxgr 2y ago
    > Books and summaries exist already of pretty much any topic you are mildly curious about. That's absolutely not the case for many questions I've had LLMs answer for me so far. Just as one example, I had one come up with a mathematical model for a specific retirement investment/taxation question I've had for years, and that is too niche to have any prior art publicly available. I've gotten as far as a spreadsheet with a vague approximation myself, with no way of validating my reasoning. I've asked an LLM to model the same thing, once symbolically and once numerically in Python, and to run the numbers for a few sample values, again symbolically and numerically. This I then compared to my own spreadsheet. Sure, I could have spent a few weeks more of studying finance textbooks or paid an accountant a few thousand dollars to do so, but it just wouldn't have been worth it to me. Big qualitative differences matter too sometimes. > You can get a good understanding of any subject in the world in a week or so if you wish to. Yes, but I don't have an infinite number of weeks in my life, nor an infinite number of motivated interns I can send out to search the web and sift through the results for me.