4 ms·

I have tried to get useful code out of ChatGPT with GPT-4 and so far been unable to find a case where it was easier for me to verify that chatGPT was right than

I have tried to get useful code out of ChatGPT with GPT-4 and so far been unable to find a case where it was easier for me to verify that chatGPT was right than to just do it myself. This was for toy examples. For more complicated things it's downright useless. Everyone seems to be raving about its ability to write code. What am I doing wrong?

MichaelRazum 3y ago
Depends what you are doing and how much code. Especially if you work with new frameworks or apis ChatGPT is extremly helpful. I think especially on small code snippets it is strong. Guess you would have to give an example, what you tried and what you got...
- skummetmaelk 3y ago
  I can see the argument for using it with frameworks and libraries as it can discover the library functions that do what your prompt describes for you. It seems to have a lot of trouble with building code that works in a certain way. For example if you ask it to build a simple graph container with access functions that give you iterators over neighbourhoods of nodes or whatever it makes a mess. Sometimes it technically works, but the underlying graph datastructure is just ripped out of randomseocodingsite.com using nodes owning pointers to their connected nodes. If you ask it to use a specific underlying structure like a vector of nodes and indexes to represent edges it doesn't quite know what to do. So yes, I guess the initial implementation "works", but not well.
- hutzlibu 3y ago
  Unless it is too new. For wgsl and webgpu, ChatGPT4 is quite useless. To be fair, the spec is still changing, but this means, that ChatGPT4 will mix in various incompatible bits. I hope they handtune it.
  - danielvaughn 3y ago
    I started learning Rust last week, and figured I could let copilot help me out a bit. Spent a while trying to get a very simple example to compile, with no luck. So even for a 5 year old language, it derps it up.
andrewstuart 3y ago
Yeah frankly it’s hard to understand how you can’t get value from ChatGPT as a programming companion. Either you already know how to do everything or you’re not asking it a wide range of questions about the work you’re doing. It’s not there to write your code for you, although it can write snippets and short functions. It’s thereto answer questions like “how do I….”, “how does this work”, “explain this code”, “what’s the correct typescript type for….”, “what’s the maximum decimal of uint16”, “difference between and y”, “give me an example of”, “what are the arguments to”, “what causes this error”, “explain this code”, “given this data structure, how do I get the third row, first item”, “how do I Centre a div”, “create a header file for these functions”, “what’s the command to compile this code”, “what’s the regex to get lines containing foo”, “ffmpeg command to convert X to y”, “given this sql table, write a query to …”, “make 1000 rows of sample json data with fields name address phone”, “write a sed command to search and replace X with y”, “create a cmakelists to compile this”, etc etc and critically important “that didn’t work”, “another approach”, “any other ideas?”. Then you debug the code it gives you.
- laratied 3y ago
  To be fair, it does strike me that not everyone will derive the same value from chatGPT. I have always wanted to rap and it has wrote rap lyrics that are way beyond what I could possibly do myself. If I was Andre 3000 or Tupac though I wouldn't be impressed at all. That is why asking experts in their domain what they think about chatGPT seems to miss the point to me.
  - staunton 3y ago
    Exactly. I've been very frustrated in my attempts to make use of LLMs for programming until I tried using it in the context of languages and technology I wasn't familiar with. There it improves productivity immensely. The primary use-case of LLMs is to reduce the barrier to entry for tasks that used to require a lot of learning. In the past, some people were employed mostly just for knowing how use some library or framework. That will no longer be the case. The value of knowledge that can be found in a book (not the same as "learned from a book") has been going down with digitization (internet, Wikipedia, search engines) and with LLMs its value will be reduced to almost nothing.
onlyrealcuzzo 3y ago
Right, but you're a developer. If you literally cannot code at all, GPT-4 is a solution. Similarly, I cannot draw at all. But I can endlessly prompt Midjourney until I get something acceptable. People who can't code can try GPT over and over again, and hopefully eventually get something that works (even if more by coincidence, who cares as long as it works!)
- pclmulqdq 3y ago
  People have said this about "no-code" tools for decades. It wasn't true for any of them, and I don't see how it is true now. These tools, from labview to copilot, have a 90% problem: they can get you about 90% of the way to a complete project, but without the last 10%, they are functionally useless unless you are an expert. That is very different than art, for example, which does not have the same kind of problem unless you are trying to do something that demands accuracy (like scientific illustration).
  - koheripbal 3y ago
    This time it's different. And I mean that sincerely, not sarcastically. I use it to code in languages I don't know.
    - pclmulqdq 3y ago
      That's cool, but you're still a software engineer, and I bet you're doing plenty of work to debug those programs. Non-programmers don't know how to do that. Your own example is not proof that this time it's different - this is just a no-code tool that you happen to gain productivity by using.
  - hutzlibu 3y ago
    I am pretty sure, people do 100% projects with labview without "code". It surely is not suitable for everything, but it would not be broadly in use, if it would be a toy.
  - cookie_monsta 3y ago
nathan_compton 3y ago
I program in a lot of different programming languages on the regular and there is some cognitive overhead in switching back and forth between them. GPT4 is pretty helpful when I don't want to remind myself for the 10th time how one does X in language Y when I've been programming in language Z for the last 3 months.
CPLX 3y ago
I’ll speak up. I am not a good coder, nowhere near the level of even your average professional developer. I’m OK, and I have been writing code of some kind or another since I was a kid in the late 80’s and have managed many technical projects and I would say I have a pretty deep understanding of how it all works. The thing I’m best at is devops type stuff and rails but even with that I’m wildly out of practice since I’m a founder/CEO type of a non software company and I can go a week or two without even touching code. These tools have completely changed everything for me in a way that is really disconcerting. Previously every time I wanted to change an app I had worked on, add a feature to something, or do something like call an API to grab something interesting from another business resource and pass it along, I was in for a couple hours just getting my head into what I was doing. I’d sort of block out a weekend and by sometime Sunday afternoon I would be getting there. Now I can just start talking to GPT like it’s an experienced developer sitting next to me. Hey so if I wanted to call this service how would I structure the query. Hey look at this curl and tell me some reasons it might be getting this error. It definitely can’t do things for me. Without a pretty deep background in software as a concept and practical experience I wouldn’t even know where to start. But I do have those things. And also I’m the guy who just knows the business logic and what I want. Now I can get what I want like 30-40x faster. To be honest it feels very strange, after being sort of frustrated by translating my requirements into specs for 20 years to work with developers and wasting a lot of time it feels like someone popped a babel fish in my ear. So there’s one anecdote.
agentgumshoe 3y ago
You're just not the variously skilled/unskilled people or marketing reps jumping on the hype train... I have a feeling most of the excitement is coming from the "citizen developer" space.
- rideontime 3y ago
  And from the people trying to sell AI tech.
- lupire 3y ago
  "citizen developer" no longer need to hire a .... patrician(?) developer. Not sure what your classes are.
Geee 3y ago
It's useful for small scripts. For example, I needed to convert a bunch of SVGs to PNGs. GTP-4 made me a node.js script in a few seconds, which saved me from the headache of googling and figuring out how to do it myself. It's also useful for suggesting basic boilerplate code for something you're not very familiar with. If you're not familiar with something, you might get stuck in the phase of how to even get started. In addition, it's useful for short but mathy code. I needed to place a bunch of items in a circle on an arbitrary plane in 3D space, and GPT-4 gave me a perfect answer for it.
poulpy123 3y ago
I use it as a replacement for google because it become more and more shit. I don't just copy/paste the result though and try to check and understand it before
williamcotton 3y ago
Would you consider a CNN that predicts what chord is being played on a guitar based on still video frames of a hand on a fretboard to be complicated? Because GPT-4 helped immensely with what was my very first NN ML project. I didn’t approach the problem with no knowledge as I’ve had a few ML projects over the decades (graphs clustering, PageRank, eigenfaces) and I’ve skimmed a few books on PyTorch and neural nets, worked through Karpasky’s zero to hero, etc. So I knew how to ask very specific questions and I knew the general overview but I had literally never written a single line of PyTorch code. I’m now working on the iOS app to host the model and send MIDI and OSC data to a DAW. I haven’t made an iOS app in about a decade and that was in the Obj-C days so I’ve never written a line of Swift. I have a demo app written in Swift that accesses the camera, converts frames to the model input, and makes predictions, and now I’m ready to use this to assist with labeling many more images!
tomjen3 3y ago
I asked it to create a script to download info from the Hackernews api. It found the right url to get the information from, with me having to look it up.
jasfi 3y ago
It really doesn't work well unless you have a very specific focus, usually a specific function. I'm working on something to make this more manageable, hopefully a lot more. You can get an email if you sign-in at https://inventai.xyz https://inventai.xyz.

MichaelRazum 3y ago

Depends what you are doing and how much code. Especially if you work with new frameworks or apis ChatGPT is extremly helpful. I think especially on small code snippets it is strong. Guess you would have to give an example, what you tried and what you got...

skummetmaelk 3y ago
I can see the argument for using it with frameworks and libraries as it can discover the library functions that do what your prompt describes for you. It seems to have a lot of trouble with building code that works in a certain way. For example if you ask it to build a simple graph container with access functions that give you iterators over neighbourhoods of nodes or whatever it makes a mess. Sometimes it technically works, but the underlying graph datastructure is just ripped out of randomseocodingsite.com using nodes owning pointers to their connected nodes. If you ask it to use a specific underlying structure like a vector of nodes and indexes to represent edges it doesn't quite know what to do. So yes, I guess the initial implementation "works", but not well.
hutzlibu 3y ago
Unless it is too new. For wgsl and webgpu, ChatGPT4 is quite useless. To be fair, the spec is still changing, but this means, that ChatGPT4 will mix in various incompatible bits. I hope they handtune it.
- danielvaughn 3y ago
  I started learning Rust last week, and figured I could let copilot help me out a bit. Spent a while trying to get a very simple example to compile, with no luck. So even for a 5 year old language, it derps it up.

andrewstuart 3y ago

Yeah frankly it’s hard to understand how you can’t get value from ChatGPT as a programming companion. Either you already know how to do everything or you’re not asking it a wide range of questions about the work you’re doing. It’s not there to write your code for you, although it can write snippets and short functions. It’s thereto answer questions like “how do I….”, “how does this work”, “explain this code”, “what’s the correct typescript type for….”, “what’s the maximum decimal of uint16”, “difference between and y”, “give me an example of”, “what are the arguments to”, “what causes this error”, “explain this code”, “given this data structure, how do I get the third row, first item”, “how do I Centre a div”, “create a header file for these functions”, “what’s the command to compile this code”, “what’s the regex to get lines containing foo”, “ffmpeg command to convert X to y”, “given this sql table, write a query to …”, “make 1000 rows of sample json data with fields name address phone”, “write a sed command to search and replace X with y”, “create a cmakelists to compile this”, etc etc and critically important “that didn’t work”, “another approach”, “any other ideas?”. Then you debug the code it gives you.

laratied 3y ago
To be fair, it does strike me that not everyone will derive the same value from chatGPT. I have always wanted to rap and it has wrote rap lyrics that are way beyond what I could possibly do myself. If I was Andre 3000 or Tupac though I wouldn't be impressed at all. That is why asking experts in their domain what they think about chatGPT seems to miss the point to me.
- staunton 3y ago
  Exactly. I've been very frustrated in my attempts to make use of LLMs for programming until I tried using it in the context of languages and technology I wasn't familiar with. There it improves productivity immensely. The primary use-case of LLMs is to reduce the barrier to entry for tasks that used to require a lot of learning. In the past, some people were employed mostly just for knowing how use some library or framework. That will no longer be the case. The value of knowledge that can be found in a book (not the same as "learned from a book") has been going down with digitization (internet, Wikipedia, search engines) and with LLMs its value will be reduced to almost nothing.

onlyrealcuzzo 3y ago

Right, but you're a developer. If you literally cannot code at all, GPT-4 is a solution. Similarly, I cannot draw at all. But I can endlessly prompt Midjourney until I get something acceptable. People who can't code can try GPT over and over again, and hopefully eventually get something that works (even if more by coincidence, who cares as long as it works!)

pclmulqdq 3y ago
People have said this about "no-code" tools for decades. It wasn't true for any of them, and I don't see how it is true now. These tools, from labview to copilot, have a 90% problem: they can get you about 90% of the way to a complete project, but without the last 10%, they are functionally useless unless you are an expert. That is very different than art, for example, which does not have the same kind of problem unless you are trying to do something that demands accuracy (like scientific illustration).
- koheripbal 3y ago
  This time it's different. And I mean that sincerely, not sarcastically. I use it to code in languages I don't know.
  - pclmulqdq 3y ago
    That's cool, but you're still a software engineer, and I bet you're doing plenty of work to debug those programs. Non-programmers don't know how to do that. Your own example is not proof that this time it's different - this is just a no-code tool that you happen to gain productivity by using.
- hutzlibu 3y ago
  I am pretty sure, people do 100% projects with labview without "code". It surely is not suitable for everything, but it would not be broadly in use, if it would be a toy.
- cookie_monsta 3y ago

nathan_compton 3y ago

I program in a lot of different programming languages on the regular and there is some cognitive overhead in switching back and forth between them. GPT4 is pretty helpful when I don't want to remind myself for the 10th time how one does X in language Y when I've been programming in language Z for the last 3 months.

CPLX 3y ago

I’ll speak up. I am not a good coder, nowhere near the level of even your average professional developer. I’m OK, and I have been writing code of some kind or another since I was a kid in the late 80’s and have managed many technical projects and I would say I have a pretty deep understanding of how it all works. The thing I’m best at is devops type stuff and rails but even with that I’m wildly out of practice since I’m a founder/CEO type of a non software company and I can go a week or two without even touching code. These tools have completely changed everything for me in a way that is really disconcerting. Previously every time I wanted to change an app I had worked on, add a feature to something, or do something like call an API to grab something interesting from another business resource and pass it along, I was in for a couple hours just getting my head into what I was doing. I’d sort of block out a weekend and by sometime Sunday afternoon I would be getting there. Now I can just start talking to GPT like it’s an experienced developer sitting next to me. Hey so if I wanted to call this service how would I structure the query. Hey look at this curl and tell me some reasons it might be getting this error. It definitely can’t do things for me. Without a pretty deep background in software as a concept and practical experience I wouldn’t even know where to start. But I do have those things. And also I’m the guy who just knows the business logic and what I want. Now I can get what I want like 30-40x faster. To be honest it feels very strange, after being sort of frustrated by translating my requirements into specs for 20 years to work with developers and wasting a lot of time it feels like someone popped a babel fish in my ear. So there’s one anecdote.

agentgumshoe 3y ago

You're just not the variously skilled/unskilled people or marketing reps jumping on the hype train... I have a feeling most of the excitement is coming from the "citizen developer" space.

rideontime 3y ago
And from the people trying to sell AI tech.
lupire 3y ago
"citizen developer" no longer need to hire a .... patrician(?) developer. Not sure what your classes are.

Geee 3y ago

It's useful for small scripts. For example, I needed to convert a bunch of SVGs to PNGs. GTP-4 made me a node.js script in a few seconds, which saved me from the headache of googling and figuring out how to do it myself. It's also useful for suggesting basic boilerplate code for something you're not very familiar with. If you're not familiar with something, you might get stuck in the phase of how to even get started. In addition, it's useful for short but mathy code. I needed to place a bunch of items in a circle on an arbitrary plane in 3D space, and GPT-4 gave me a perfect answer for it.

poulpy123 3y ago

I use it as a replacement for google because it become more and more shit. I don't just copy/paste the result though and try to check and understand it before

williamcotton 3y ago

Would you consider a CNN that predicts what chord is being played on a guitar based on still video frames of a hand on a fretboard to be complicated? Because GPT-4 helped immensely with what was my very first NN ML project. I didn’t approach the problem with no knowledge as I’ve had a few ML projects over the decades (graphs clustering, PageRank, eigenfaces) and I’ve skimmed a few books on PyTorch and neural nets, worked through Karpasky’s zero to hero, etc. So I knew how to ask very specific questions and I knew the general overview but I had literally never written a single line of PyTorch code. I’m now working on the iOS app to host the model and send MIDI and OSC data to a DAW. I haven’t made an iOS app in about a decade and that was in the Obj-C days so I’ve never written a line of Swift. I have a demo app written in Swift that accesses the camera, converts frames to the model input, and makes predictions, and now I’m ready to use this to assist with labeling many more images!

tomjen3 3y ago

I asked it to create a script to download info from the Hackernews api. It found the right url to get the information from, with me having to look it up.

jasfi 3y ago

It really doesn't work well unless you have a very specific focus, usually a specific function. I'm working on something to make this more manageable, hopefully a lot more. You can get an email if you sign-in at https://inventai.xyz https://inventai.xyz.