4 ms·

How do you reconcile with the fact that it imagine things?

Philpax 3y ago
GPT-4's solutions are usually correct the first time around, or can be corrected by telling it what it did wrong. The ask-clarify-correct loop is still faster and less effort than doing it yourself.
- verdverm 3y ago
  If people continue to learn outside of LLMs and can tell when it is wrong so they can correct it. The slope can be slippery. For those of us who grew up before the rise of LLMs, this isn't really something we have to worry about. But the next generation will be not know what it was like without LLMs, kind of like the pre/post-internet/phone generational splits
matwood 3y ago
The same way a colleague or myself is often not 100% correct. Neither is Google.
Kiro 3y ago
Not a problem in practice if you're a developer. It's probably worse if you're just copypasting without understanding anything.
jddj 3y ago
Approach it with the same critical eye that you use to consume any other information.
RobCat27 3y ago
I've found that for questions that aren't super complex on things that are extremely likely to be in its training dataset (such as public documentation of popular Python libraries) the error rate is very close to 0%. Even compared to GPT from a few months ago I've found the accuracy of responses has increased dramatically. I still believe anyone using these tools on a day-to-day basis should have a sense of "trust but verify."
karpour 3y ago
As a senior developer: Experience. The output from ChatGPT isn't perfect, but it's good enough to point one in the right direction. The code it outputs is mostly garbage that a good developer would never put into production, but what it does fairly well is finding the relevant bits of documentation. Example: Using a small part of a new, big, unfamiliar library. Rather than digging through the library docs, I can ask ChatGPT about it, which often points me to the relevant parts, which I then can still confirm in the official docs.
- josephg 3y ago
  Yeah. I got chatgpt to write some code using Indexeddb the other day. The code was absolute rubbish, and I rewrote it to be half as long, while being clearer and with better error handling. But the code it gave me was a great starting point. I found it much faster & easier to rewrite the bad code it wrote than pore through documentation and figure out how to solve my problem from scratch.
alright2565 3y ago
I more-or-less know what I expect it to give me, so I can just go somewhere else then. I tend to ask it three broad sets of queries, since it's what it does best at. tl;dr: it can transform text very well, even with lots of ambiguity. also it knows technical vocab - if I'm working in a field that's new to me or that I don't understand, I ask for help understanding the basics and vocabulary of the field. it does very well at this. - if I have a well defined problem, but am simply not familiar with the libraries for a given situation, it tends to do a good job translating my english queries into the right code. I do examine and test the code afterwards to make sure it's correct. You can really see this in action when you ask it to do data analysis; and the REPL loop in that mode is also great at catching bugs. - if I have like a copy-paste from a documentation site, I can ask it to transform that into code or into a better-formatted version. this saves a lot of time, and I don't have to remember regexes or vim keybinds
- ifyoubuildit 3y ago
  > if I'm working in a field that's new to me or that I don't understand, I ask for help understanding the basics and vocabulary of the field. it does very well at this I also do this, but I am careful about being confident that "it does very well at this". We can't actually evaluate what it's putting out, other than that it sounds plausible, which is something LLMs are truly great at.

GPT-4's solutions are usually correct the first time around, or can be corrected by telling it what it did wrong. The ask-clarify-correct loop is still faster and less effort than doing it yourself.

verdverm 3y ago
If people continue to learn outside of LLMs and can tell when it is wrong so they can correct it. The slope can be slippery. For those of us who grew up before the rise of LLMs, this isn't really something we have to worry about. But the next generation will be not know what it was like without LLMs, kind of like the pre/post-internet/phone generational splits

matwood 3y ago

The same way a colleague or myself is often not 100% correct. Neither is Google.

Kiro 3y ago

Not a problem in practice if you're a developer. It's probably worse if you're just copypasting without understanding anything.

jddj 3y ago

Approach it with the same critical eye that you use to consume any other information.

RobCat27 3y ago

I've found that for questions that aren't super complex on things that are extremely likely to be in its training dataset (such as public documentation of popular Python libraries) the error rate is very close to 0%. Even compared to GPT from a few months ago I've found the accuracy of responses has increased dramatically. I still believe anyone using these tools on a day-to-day basis should have a sense of "trust but verify."

karpour 3y ago

As a senior developer: Experience. The output from ChatGPT isn't perfect, but it's good enough to point one in the right direction. The code it outputs is mostly garbage that a good developer would never put into production, but what it does fairly well is finding the relevant bits of documentation. Example: Using a small part of a new, big, unfamiliar library. Rather than digging through the library docs, I can ask ChatGPT about it, which often points me to the relevant parts, which I then can still confirm in the official docs.

josephg 3y ago
Yeah. I got chatgpt to write some code using Indexeddb the other day. The code was absolute rubbish, and I rewrote it to be half as long, while being clearer and with better error handling. But the code it gave me was a great starting point. I found it much faster & easier to rewrite the bad code it wrote than pore through documentation and figure out how to solve my problem from scratch.

alright2565 3y ago

I more-or-less know what I expect it to give me, so I can just go somewhere else then. I tend to ask it three broad sets of queries, since it's what it does best at. tl;dr: it can transform text very well, even with lots of ambiguity. also it knows technical vocab - if I'm working in a field that's new to me or that I don't understand, I ask for help understanding the basics and vocabulary of the field. it does very well at this. - if I have a well defined problem, but am simply not familiar with the libraries for a given situation, it tends to do a good job translating my english queries into the right code. I do examine and test the code afterwards to make sure it's correct. You can really see this in action when you ask it to do data analysis; and the REPL loop in that mode is also great at catching bugs. - if I have like a copy-paste from a documentation site, I can ask it to transform that into code or into a better-formatted version. this saves a lot of time, and I don't have to remember regexes or vim keybinds

ifyoubuildit 3y ago
> if I'm working in a field that's new to me or that I don't understand, I ask for help understanding the basics and vocabulary of the field. it does very well at this I also do this, but I am careful about being confident that "it does very well at this". We can't actually evaluate what it's putting out, other than that it sounds plausible, which is something LLMs are truly great at.