4 ms·

Out of curiosity, how do you know that it’s correct?

ThePhysicist 4y ago
Reading the code, and running it. You can ask it for a complete project tree and even test scripts (you have to ask it to continue as it gets interrupted sometimes). I e.g. had it write a simple database engine with an API, and it produced Rust code as well as curl-based examples to test the database. It even correctly added up the data items it sent into the DB via curl to show what the output would probably look like. Pure sci-fi technology.
bipop5000 4y ago
Test and reiterate. I don't think you should take whatever is coming out of chatGPT verbatim. It should be used as an interactive tool to amplify your knowledge.
- mjrpes 4y ago
  GPT-4 for coding is very powerful but can be dangerous. I asked it to write a data retention script that would save X number of backups depending on the number of days/weeks/month/years from the present day. It's first attempt would have worked the first day, but running the script consecutive days would eventually delete all backups. Once I told GPT-4 to take into account the script would be run each day, it "apologized" and produced a script that worked.
- layer8 4y ago
  Testing does not show correctness, it can only show incorrectness.
whatever1 4y ago
Same way we know that any piece of code is correct. Debugging, testing.
YetAnotherNick 4y ago
Similar to how you know someone's stackoverflow answer is correct. Read the code, find the documentation of API calls, think of any missed cases, write tests etc.
- thomastjeffery 4y ago
  So basically the same as writing it yourself: just skipping the writing part. So should we be taking about GPT as an occasional text editor replacement? I honestly think that's more accurate take than most of the ones I have seen.
  - mgfist 4y ago
    Have you seen the GPT4 demo? A few interesting things that make it seem much more than an occasional text editor replacement. One was having it write an entire program. It needed a few prompts to fix some bugs it had, but it worked. The program was then used for the rest of the demo. The program even had an obscure bug that it solved by having the demo-er paste the error + the entire docs page. The second demo was having it write a html page based on a picture drawing on a napkin of a website, and then running the html. It's honestly pretty impressive. I already use chatgpt a lot (doing a lot of google sheets scripting/formula stuff and the docs + syntax are horrendous) and chatgpt helps me find the correct syntax much faster than I'd otherwise be able to
    - thomastjeffery 4y ago
      There is a huge difference between writing a program, and writing the program you want right now. It's tricky to conceptualize because the entire narrative we are familiar with has personified GPT. It may be impressive, but GPT is completely different from humans. If a human is capable of writing a program, that is because they understand conceptually how and why. GPT doesn't. GPT's ability to write a program is entirely dependent on the content of its training corpus: no how, no why, only what. I'll put it another way: If a human is not able to write a program, it is because they don't understand conceptually how. If GPT is not able to write a program, it is because either the desired parts, or the necessary pattern that puts them together, does not exist in the training corpus; or because the prompt didn't yield a continuation that followed that arbitrary pattern. The results are impressive because humans are impressive. GPT doesn't interact with the domain of all possible written text. It deals with the domain of all possible patterns of tokens from the text it was given. It's only given text that was intentionally written by humans. It exists in a world of signal: no noise. It can still only guess, but the guess will always be constructed out of the category of text humans choose to write. Inference models are a completely different approach to language, and confusing them with human behavior is an easy way to make impossible predictions about what they can and cannot accomplish.
eloff 4y ago
The author addressed that rather at length in the article. Don’t. Trust. Code. That it originated from ChatGPT instead of any number of other sources doesn’t change that.

Reading the code, and running it. You can ask it for a complete project tree and even test scripts (you have to ask it to continue as it gets interrupted sometimes). I e.g. had it write a simple database engine with an API, and it produced Rust code as well as curl-based examples to test the database. It even correctly added up the data items it sent into the DB via curl to show what the output would probably look like. Pure sci-fi technology.

bipop5000 4y ago

Test and reiterate. I don't think you should take whatever is coming out of chatGPT verbatim. It should be used as an interactive tool to amplify your knowledge.

mjrpes 4y ago
GPT-4 for coding is very powerful but can be dangerous. I asked it to write a data retention script that would save X number of backups depending on the number of days/weeks/month/years from the present day. It's first attempt would have worked the first day, but running the script consecutive days would eventually delete all backups. Once I told GPT-4 to take into account the script would be run each day, it "apologized" and produced a script that worked.
layer8 4y ago
Testing does not show correctness, it can only show incorrectness.

whatever1 4y ago

Same way we know that any piece of code is correct. Debugging, testing.

YetAnotherNick 4y ago

Similar to how you know someone's stackoverflow answer is correct. Read the code, find the documentation of API calls, think of any missed cases, write tests etc.

thomastjeffery 4y ago
So basically the same as writing it yourself: just skipping the writing part. So should we be taking about GPT as an occasional text editor replacement? I honestly think that's more accurate take than most of the ones I have seen.
- mgfist 4y ago
  Have you seen the GPT4 demo? A few interesting things that make it seem much more than an occasional text editor replacement. One was having it write an entire program. It needed a few prompts to fix some bugs it had, but it worked. The program was then used for the rest of the demo. The program even had an obscure bug that it solved by having the demo-er paste the error + the entire docs page. The second demo was having it write a html page based on a picture drawing on a napkin of a website, and then running the html. It's honestly pretty impressive. I already use chatgpt a lot (doing a lot of google sheets scripting/formula stuff and the docs + syntax are horrendous) and chatgpt helps me find the correct syntax much faster than I'd otherwise be able to
  - thomastjeffery 4y ago
    There is a huge difference between writing a program, and writing the program you want right now. It's tricky to conceptualize because the entire narrative we are familiar with has personified GPT. It may be impressive, but GPT is completely different from humans. If a human is capable of writing a program, that is because they understand conceptually how and why. GPT doesn't. GPT's ability to write a program is entirely dependent on the content of its training corpus: no how, no why, only what. I'll put it another way: If a human is not able to write a program, it is because they don't understand conceptually how. If GPT is not able to write a program, it is because either the desired parts, or the necessary pattern that puts them together, does not exist in the training corpus; or because the prompt didn't yield a continuation that followed that arbitrary pattern. The results are impressive because humans are impressive. GPT doesn't interact with the domain of all possible written text. It deals with the domain of all possible patterns of tokens from the text it was given. It's only given text that was intentionally written by humans. It exists in a world of signal: no noise. It can still only guess, but the guess will always be constructed out of the category of text humans choose to write. Inference models are a completely different approach to language, and confusing them with human behavior is an easy way to make impossible predictions about what they can and cannot accomplish.

eloff 4y ago

The author addressed that rather at length in the article. Don’t. Trust. Code. That it originated from ChatGPT instead of any number of other sources doesn’t change that.