4 ms·

> Helping with challenging code problems that would otherwise need hours of searching google or reading the docs. I'm glad this has been working for you -- gen

> Helping with challenging code problems that would otherwise need hours of searching google or reading the docs. I'm glad this has been working for you -- generally any time I actually have a really difficult problem, ChatGPT just makes up the API I wish existed. Then when I bring it up to ChatGPT, it just apologizes and invents new API.

chromanoid 2y ago
Do you provide code examples? In my experience the more specific you get with your problem the more specific are the provided solutions (probably a "natural" occurence in LLMs). Hallucianted APIs are sometimes a problem for me, but then I just specify which API to use.
- brailsafe 2y ago
  Seems like you're just saying if you solve the problem on your own then whatever the LLM produces isn't an issue
- Jensson 2y ago
  Why do you need an LLM if you know what you want it to do? Just write the code rather than wrangling with the LLM, it isn't like writing code take much time when you know what it should do.
  - jacobsimon 2y ago
    I know it’s not a completely fair comparison, but to me this question is kind of missing the point. It’s like asking “Why take a cab if you know where you want to go?”
    - philipwhiuk 2y ago
      It's such a poor comparison it's ridiculous. A better analogy is "why take a cab if you know where you want to go and provide the car and instructions on how to drive"
    - jeffalyanak 2y ago
      No, it's like saying "why take a cab where you have to provide the driver so much guidance on driving as to be equal or greater than the effort of driving yourself."
  - LunicLynx 2y ago
    This is the comment! If so called „prompt engineering“ goes so far that only one solution remains, you don‘t need the LLM.
    - drogus 2y ago
      For me, it depends on the problem. I avoid LLMs for anything complex, cause I prefer to think it through myself. But there are often times when you know exactly what you want and you know how it should look like. Let's say you need a simple web API to help you with a task. These days I'd typically ask an LLM to write the app. It will usually get some stuff wrong, but after a quick glance I can steer it to fix the problems (like: you didn't handle errors etc). That way I can generate a simple few hundred lines of code app in minutes. There is no way I could type that fast even if I exactly know what characters to write and it's not always the case. Like, oftentimes I know exactly what to do and I know if it's OK when I see the code, but writing it would require me to look into the docs here and there.
  - zaptrem 2y ago
    Not op but my response: Because I am lazy and would like to save the 1-5 minutes it would take me to actually write it. When there are dozens of these small things a day the saved time really adds.
    - croes 2y ago
      That's why you spend 10 minutes to write the correct prompt?
      - jeffhuys 2y ago
        If it takes you 10 minutes, you're doing it wrong.
        croes 2y ago
        It's a lot easier in SQL to know the right way than in writing prompts especially with non deterministic models.
  - throwaway_ab 2y ago
    I find LLM's great for: - Getting over the blank canvas hurdle, this is great for kick starting a small project and even if the code isn't amazing, it gets my brain to the "start writing code and thinking about algo/data-structures/interesting-problem" rather than being held up at the "Where to begin?" Metaphorically where to place my first stroke, this helps somewhat. - Sometimes LLM has helped when stuck on issues but this is hit and miss, more specifically it will often show a solution that jogs my brain and gets me there, "oh yeah of course" however I've noticed I'm more in than state when tired and need sleep, so the LLM might let me push a bit longer making up for tired brain. However this is more harmful to be honest without the LLM I go to sleep and then magically like brains do solve 4 hours of issues in 20 minutes after waking up. So LLM might be helping in ways that actually indicate you should sleep as brain is slooooowwwwing down
  - throwaway_ab 2y ago
    I find LLM's great for: - Getting over the blank canvas hurdle, this is great for kick starting a small project and even if the code isn't amazing, it gets my brain to the "start writing code and thinking about algo/data-structures/interesting-problem" rather than being held up at the "Where to begin?" Metaphorically where to place my first stroke, this helps somewhat. - Sometimes LLM has helped when stuck on issues but this is hit and miss, more specifically it will often show a solution that jogs my brain and gets me there, "oh yeah of course" however I've noticed I'm more in than state when tired and need sleep, so the LLM might let me push a bit longer making up for tired brain. However this is more harmful to be honest without the LLM I go to sleep and then magically like brains do solve 4 hours of issues in 20 minutes after waking up. So LLM might be helping in ways that actually indicate you should sleep as brain is slooooowwwwing down
    - andrew-dc 2y ago
      Yes, this. I was skeptical and disgusted at a lot of what was being done or promised by using LLMs, but this was because I initially saw a lot of wholesale: "Make thing for me," being hyped or discussed. In practice, I have found them to be good tools for getting going or un-stuck, and use them more like an inspiration engine, or brain kick-starter.
- actionfromafar 2y ago
  I think it also depends on how common the API is. If it's more unknown to the LLM, it tends to drift away.
rcarmo 2y ago
Has to be something pretty generic. I'm trying to write a little C program that talks to an LCD display via the SPI bus--something I did before a few times, but not with this particular display and MCU. There is no LLM that can even begin to reason this out since they've been mostly trained on web dev content.
- fragmede 2y ago
  is there no documentation about that uc and LCD controller? I'd assume they've been trained on every html, pdf, and video tutorial out there, as well as the pdfs of all of those chip's datasheets, and example microcontroller C code out there in the form of vendor documentation. sure that's maybe less than the amount of html web app tutorials but if we assume it's been fed all of the vendor's documentation about an uc and their library documentation as well, the ones it would fail on are undocumented chips out of china that have no data sheets (which make for a very fun project to reverse engineer, mind you), or something very new. Even without that though, it's still able to dot product (not going to call it "reasoning") its way to at least hallucinate code to talk to an LCD controller chip via SPI for a uc is never even heard of, so I can't agree with "even begin to reason this out".
  - jokethrowaway 2y ago
    Even if it were trained on everything it wouldn't know everything. You need to do some serious RHLF to get something good out of LLMs
  - rcarmo 2y ago
    You don't learn how to program SPI from reading documentation about an LCD controller. You need a lot more context and understand how to string together basic operations, which are quite often not detailed in parts documentation. I think that you have a serious misunderstanding of the capabilities of LLMs - they cannot reason out relationships among documents that easily. They cannot even tell you what they don't know to finish a given task (and I'm not just talking one-shot here, agent frameworks suffer from the same problem).
zaptrem 2y ago
I assume it knows the big stuff like the PyTorch API/major JS and React libs then just paste the docs or even impl code for any libs it needs to know beyond that.
skriticos2 2y ago
I found that ChatGPT needs to be rained in with the prompts, and then it does a very impressive job. E.g. you can create a function prototype (with input and output expectations) and in the body tell the logic you are thinking about in meta-code. Then tell it to write the actual code. It's also good if you want to immerse yourself into a new programming language and outline what kind of program you want, and expect the results to be different from what you throught, but insightful. Now if you throw larger context or more obscure interface expectations at it, it'll start to discard code and hallucinate.
Havoc 2y ago
Phrase the question as a backend problem rather than as a front end problem. Same for lang your asking in. Less JS/TS more Python
sebstefan 2y ago
It's nice that everybody is trying to help with the way you're prompting but just use Bing Copilot or Phind for this, not ChatGPT It'll generate a bunch of queries to Google (well, "to Bing" I guess in that case) based on your question, read the results for you, base its answer on the results and provide you with sources that you can check if it used anything from that webpage. I only use ChatGPT for documentation when I have no idea where I'm going at all, and I need a lay of the land on best practices and the way forward. For specifics, Bing Copilot. Essentially a true semantic web search
barrkel 2y ago
LLMs aren't good when you drift out of the training distribution. You want to be hitting the meat in the middle and leveraging the LLM to blast through it quickly. That means LLMs are great for scaffolding, prototypes, the v0.1 of new code especially when it's very ordinary logic but using a language or library you're not 100% up to speed on. One project I was on recently was translation: converting a JS library into Kotlin. In-editor AI code completion made this really quick: I pasted a snippet of JS for translation in a comment, and the AI completed the Kotlin version. It was frequently not quite right, but it was way faster than without. In particular, when there was repeated blocks of code for different cases that different only slightly, once I got the first block correct, the LLM picked up on the pattern in-context and applied it correctly for the remaining blocks. Even when it's wrong, if it has an opportunity to learn locally, it can do so.
- wanderingstan 2y ago
  I’ve come to the similar conclusion, and realized that I’m slowly learning how to wield this new tool. The sweet spot is when you need something and you’re sure it’s possible but just don’t know (or it’s too time consuming) how. E.G. change the css to to X, rewrite this python code in typescript, use the pattern of this code to do Y, etc. Reminds me of the early days of Google where you had to learn how write a good search query. You learn you need more than a word or two, but don’t write a whole essay, etc.
martin82 2y ago
That's actually an awesome feature! It means that you are working on something that no one else has ever done before... ...or you aren't able to describe your problem correctly.
ericmcer 2y ago
That makes sense, LLM training data probably has tons of common problems in it, but maybe only a few or no instances of really niche, difficult ones. So it just comes up with a bunch of garbage. My experience has been similar, it is amazing for stuff I am a beginner at but kinda useless for my actual work. It was invaluable today when I was trying to grasp CA zoning laws, but its almost useless for coding. This also points to why it will never (imo) be "intelligent". It will never be able to take all its knowledge and use that to solve a problem it doesn't have training data for.

chromanoid 2y ago

Do you provide code examples? In my experience the more specific you get with your problem the more specific are the provided solutions (probably a "natural" occurence in LLMs). Hallucianted APIs are sometimes a problem for me, but then I just specify which API to use.

brailsafe 2y ago
Seems like you're just saying if you solve the problem on your own then whatever the LLM produces isn't an issue
Jensson 2y ago
Why do you need an LLM if you know what you want it to do? Just write the code rather than wrangling with the LLM, it isn't like writing code take much time when you know what it should do.
- jacobsimon 2y ago
  I know it’s not a completely fair comparison, but to me this question is kind of missing the point. It’s like asking “Why take a cab if you know where you want to go?”
  - philipwhiuk 2y ago
    It's such a poor comparison it's ridiculous. A better analogy is "why take a cab if you know where you want to go and provide the car and instructions on how to drive"
  - jeffalyanak 2y ago
    No, it's like saying "why take a cab where you have to provide the driver so much guidance on driving as to be equal or greater than the effort of driving yourself."
- LunicLynx 2y ago
  This is the comment! If so called „prompt engineering“ goes so far that only one solution remains, you don‘t need the LLM.
  - drogus 2y ago
    For me, it depends on the problem. I avoid LLMs for anything complex, cause I prefer to think it through myself. But there are often times when you know exactly what you want and you know how it should look like. Let's say you need a simple web API to help you with a task. These days I'd typically ask an LLM to write the app. It will usually get some stuff wrong, but after a quick glance I can steer it to fix the problems (like: you didn't handle errors etc). That way I can generate a simple few hundred lines of code app in minutes. There is no way I could type that fast even if I exactly know what characters to write and it's not always the case. Like, oftentimes I know exactly what to do and I know if it's OK when I see the code, but writing it would require me to look into the docs here and there.
- zaptrem 2y ago
  Not op but my response: Because I am lazy and would like to save the 1-5 minutes it would take me to actually write it. When there are dozens of these small things a day the saved time really adds.
  - croes 2y ago
    That's why you spend 10 minutes to write the correct prompt?
    - jeffhuys 2y ago
      If it takes you 10 minutes, you're doing it wrong.
      - croes 2y ago
        It's a lot easier in SQL to know the right way than in writing prompts especially with non deterministic models.
- throwaway_ab 2y ago
  I find LLM's great for: - Getting over the blank canvas hurdle, this is great for kick starting a small project and even if the code isn't amazing, it gets my brain to the "start writing code and thinking about algo/data-structures/interesting-problem" rather than being held up at the "Where to begin?" Metaphorically where to place my first stroke, this helps somewhat. - Sometimes LLM has helped when stuck on issues but this is hit and miss, more specifically it will often show a solution that jogs my brain and gets me there, "oh yeah of course" however I've noticed I'm more in than state when tired and need sleep, so the LLM might let me push a bit longer making up for tired brain. However this is more harmful to be honest without the LLM I go to sleep and then magically like brains do solve 4 hours of issues in 20 minutes after waking up. So LLM might be helping in ways that actually indicate you should sleep as brain is slooooowwwwing down
- throwaway_ab 2y ago
  I find LLM's great for: - Getting over the blank canvas hurdle, this is great for kick starting a small project and even if the code isn't amazing, it gets my brain to the "start writing code and thinking about algo/data-structures/interesting-problem" rather than being held up at the "Where to begin?" Metaphorically where to place my first stroke, this helps somewhat. - Sometimes LLM has helped when stuck on issues but this is hit and miss, more specifically it will often show a solution that jogs my brain and gets me there, "oh yeah of course" however I've noticed I'm more in than state when tired and need sleep, so the LLM might let me push a bit longer making up for tired brain. However this is more harmful to be honest without the LLM I go to sleep and then magically like brains do solve 4 hours of issues in 20 minutes after waking up. So LLM might be helping in ways that actually indicate you should sleep as brain is slooooowwwwing down
  - andrew-dc 2y ago
    Yes, this. I was skeptical and disgusted at a lot of what was being done or promised by using LLMs, but this was because I initially saw a lot of wholesale: "Make thing for me," being hyped or discussed. In practice, I have found them to be good tools for getting going or un-stuck, and use them more like an inspiration engine, or brain kick-starter.
actionfromafar 2y ago
I think it also depends on how common the API is. If it's more unknown to the LLM, it tends to drift away.

rcarmo 2y ago

Has to be something pretty generic. I'm trying to write a little C program that talks to an LCD display via the SPI bus--something I did before a few times, but not with this particular display and MCU. There is no LLM that can even begin to reason this out since they've been mostly trained on web dev content.

fragmede 2y ago
is there no documentation about that uc and LCD controller? I'd assume they've been trained on every html, pdf, and video tutorial out there, as well as the pdfs of all of those chip's datasheets, and example microcontroller C code out there in the form of vendor documentation. sure that's maybe less than the amount of html web app tutorials but if we assume it's been fed all of the vendor's documentation about an uc and their library documentation as well, the ones it would fail on are undocumented chips out of china that have no data sheets (which make for a very fun project to reverse engineer, mind you), or something very new. Even without that though, it's still able to dot product (not going to call it "reasoning") its way to at least hallucinate code to talk to an LCD controller chip via SPI for a uc is never even heard of, so I can't agree with "even begin to reason this out".
- jokethrowaway 2y ago
  Even if it were trained on everything it wouldn't know everything. You need to do some serious RHLF to get something good out of LLMs
- rcarmo 2y ago
  You don't learn how to program SPI from reading documentation about an LCD controller. You need a lot more context and understand how to string together basic operations, which are quite often not detailed in parts documentation. I think that you have a serious misunderstanding of the capabilities of LLMs - they cannot reason out relationships among documents that easily. They cannot even tell you what they don't know to finish a given task (and I'm not just talking one-shot here, agent frameworks suffer from the same problem).

zaptrem 2y ago

I assume it knows the big stuff like the PyTorch API/major JS and React libs then just paste the docs or even impl code for any libs it needs to know beyond that.

skriticos2 2y ago

I found that ChatGPT needs to be rained in with the prompts, and then it does a very impressive job. E.g. you can create a function prototype (with input and output expectations) and in the body tell the logic you are thinking about in meta-code. Then tell it to write the actual code. It's also good if you want to immerse yourself into a new programming language and outline what kind of program you want, and expect the results to be different from what you throught, but insightful. Now if you throw larger context or more obscure interface expectations at it, it'll start to discard code and hallucinate.

Havoc 2y ago

Phrase the question as a backend problem rather than as a front end problem. Same for lang your asking in. Less JS/TS more Python

sebstefan 2y ago

It's nice that everybody is trying to help with the way you're prompting but just use Bing Copilot or Phind for this, not ChatGPT It'll generate a bunch of queries to Google (well, "to Bing" I guess in that case) based on your question, read the results for you, base its answer on the results and provide you with sources that you can check if it used anything from that webpage. I only use ChatGPT for documentation when I have no idea where I'm going at all, and I need a lay of the land on best practices and the way forward. For specifics, Bing Copilot. Essentially a true semantic web search

barrkel 2y ago

LLMs aren't good when you drift out of the training distribution. You want to be hitting the meat in the middle and leveraging the LLM to blast through it quickly. That means LLMs are great for scaffolding, prototypes, the v0.1 of new code especially when it's very ordinary logic but using a language or library you're not 100% up to speed on. One project I was on recently was translation: converting a JS library into Kotlin. In-editor AI code completion made this really quick: I pasted a snippet of JS for translation in a comment, and the AI completed the Kotlin version. It was frequently not quite right, but it was way faster than without. In particular, when there was repeated blocks of code for different cases that different only slightly, once I got the first block correct, the LLM picked up on the pattern in-context and applied it correctly for the remaining blocks. Even when it's wrong, if it has an opportunity to learn locally, it can do so.

wanderingstan 2y ago
I’ve come to the similar conclusion, and realized that I’m slowly learning how to wield this new tool. The sweet spot is when you need something and you’re sure it’s possible but just don’t know (or it’s too time consuming) how. E.G. change the css to to X, rewrite this python code in typescript, use the pattern of this code to do Y, etc. Reminds me of the early days of Google where you had to learn how write a good search query. You learn you need more than a word or two, but don’t write a whole essay, etc.

martin82 2y ago

That's actually an awesome feature! It means that you are working on something that no one else has ever done before... ...or you aren't able to describe your problem correctly.

ericmcer 2y ago

That makes sense, LLM training data probably has tons of common problems in it, but maybe only a few or no instances of really niche, difficult ones. So it just comes up with a bunch of garbage. My experience has been similar, it is amazing for stuff I am a beginner at but kinda useless for my actual work. It was invaluable today when I was trying to grasp CA zoning laws, but its almost useless for coding. This also points to why it will never (imo) be "intelligent". It will never be able to take all its knowledge and use that to solve a problem it doesn't have training data for.