6 ms·

I think people writing all this should stop and try GPT4. It writes reasonable good code and scripts. It is able to perform so many tasks in a reasonable way. A

I think people writing all this should stop and try GPT4. It writes reasonable good code and scripts. It is able to perform so many tasks in a reasonable way. And it doesn't write bullshit in common usage. Yes it might not be AGI and might not be sentient, but does all that matter? GPT4 does make mistakes but the mistakes are not "parroted bullshit", but minor issues here and there. Yes, don't use it to decide extremely important topics, but it is here and it is extremely powerful.

skilled 4y ago
I agree, GPT-4 has a much more "aggressive" confidence, I definitely noticed that, but I haven't seen much of it since the API is still behind closed doors. I'd also like to point out that even though it's better, it still can't reason very well and relies heavily on prompts and instructions, even for very basic projects. It kind of reminds me a pseudo-visual code builder. I was writing a Python script with it to put together documentation summaries, and over the course of an hour it offered me 3 different libraries to connect to an API endpoint I was using. It's distracting.
- SketchySeaBeast 4y ago
  > GPT-4 has a much more "aggressive" confidence Could you expand on this? I only use the unpaid API, but when I ask for, say, song lyrics, it'll be 100% wrong and then it will give me a second version of the wrong lyrics when I say the first was wrong, and then when I ask how confident it is in the new lyrics it'll say 95%, so whatever the current free API is it's undeservedly confident. I must be misunderstanding what you mean by confidence.
  - skilled 4y ago
    I can't really expand because I don't want to share my primary use case, I hope you understand. That said, what I mean is that it feels a lot more comprehensive, as if it really has a grasp on what is being asked, kind of like - its much better at understanding the information it has, so responses feel richer, both in context but also in tone. I am comparing it directly to 3.5. Again, I don't have the API access so I cannot say for sure, this was only my experience from the ChatGPT interface and also a third-party API I spent $20 on just to test its limits, but sadly for the time being those limits are hardcore - it just gets stuck on long responses despite there being a token increase. So, if I really wanted to test it out and make a proper assessment, I'd need access to the API myself so I can do some high-volume token shenanigans.
- spookthesunset 4y ago
  I’ve had it do stuff like that. In between updates to fix issues sometimes it decides to completely rewrite the thing you are working on.
zekica 4y ago
It returns factually incorrect data, and it returns code with subtle but important errors if you ask it anything that's not regurgitated a thousand times in the training dataset. Don't get me wrong, it has emergent properties (more than you would expect from a fancy autocomplete), but factual output was never GPT-4's nor any other LLM's design goal.
- progrus 4y ago
  I’ve started using it as a sort of “shotgun approach” - characterize what you want without trying too hard, let it toss out a jumble of interesting snippets that are often very good leads, and then take over refactoring from there. I do agree that others who don’t understand what it’s doing may not even review the code output, and I predict that this will cause at least one death in the next ten years.
  - musha68k 4y ago
    *weeks/*months
    - progrus 4y ago
      Facts, sadly.
  - bodge5000 4y ago
    I suppose it depends what specifically you're getting it to do, but I've found in my testing the shotgun approach to be more work than just writing it out myself. Honestly I've found its biggest value in learning a new language, it outputs with so many errors that it forces me to get in deep with debugging it. I haven't tried gpt-4 yet though, maybe its a vast improvement, I'm just not using it enough to justify the costs
    - progrus 4y ago
      It really depends on how comfortable you are with the language you’re using, and how recently you switched contexts. For anything unfamiliar, it’s pretty impressive how much it can help. In fact, I bet if you did an experiment comparing people reading a textbook about a somewhat-familiar language, vs. just asking GPT to do things and noticing the bugs, the latter group would reach proficiency faster.
redleggedfrog 4y ago
"I think people writing all this should stop and try GPT4." Have you? I mean, for something that's actually useful beyond just simple stuff, like at your job where it's going to matter? Because for what I'm doing which is custom ERP/Website data flows to/from legacy systems it's mostly a time sink and offers little value.
- throwuwu 4y ago
  Pretty much by definition its not going to be good at highly custom or uncommon programming tasks since it won’t have seen many examples to generalize from. I doubt you could get better results from a freshly hired human either.
  - mdp2021 4y ago
    > from a freshly hired human either But the human will learn, through reasoning and challenges... Those features of which some wonder whether they will emerge are critical pieces.
    - pmoriarty 4y ago
      "But the human will learn" That depends on the human, what they're trying to learn, and how it's taught to them. Some people with brain damage, learning disabilities, other cognitive issues, with poor teachers or learning materials, or with too difficult a task will not learn. It might be useful to view AIs as akin in some ways to brain damaged humans, who might be good at some tasks but deficient at others.
      - hajile 4y ago
        Those people also won’t be hired as coders. There are other things that they will excel at.
  - redleggedfrog 4y ago
    LOL! You got that right! They don't.
oldstrangers 4y ago
It struggles to accurately call its own API when prompted to. It patches together parts of GPT 3 / 3.5 and has no awareness of GPT4 API calls (at least a few weeks ago). I've tried 3 separate projects with it (using different platforms from Firebase to Vercel) and it usually gets me 80% there but fails miserably at the nuance portion (its stuck at a CORS policy issue with my last project). From start to finish I've yet to be able to complete something with it.
- waboremo 4y ago
  How could it have awareness of gpt4 when it wasn't even in existence prior to 2021?
  - oldstrangers 4y ago
    "How could GPT4 have awareness of GPT4" Seems like something they might want to include, especially when its a paid service.
toss1 4y ago
>> but the mistakes are not "parroted bullshit", but minor issues here and there. Nonsense. I'm not saying that it is not useful, but I've easily found ChatGPT produces far worse than "parroted bullshit", and does so reliably, and multiple times (repeated patterns of mistakes). Again, I have found ChatGPT4 (but not 3.5) useful and even educational in some cases, but it's usability requires far more skeptical and error-checked approach. As one example, since I have a top specialist attorney in the household, I started with some broad legal questions about their field, and then asked their opinion. ChatGPT4's answers were impressive to both of us, and especially the solid grammar. When the questions got more particular about their specialist area of the law, ChatGPT4's answers got shaky very fast. Asked about a particular somewhat obscure technique, it parroted some very nice-sounding bullshit, which got the concepts very backwards (but it was still very confident and grammatical). When asked to summarize a published paper about a new innovation in this context, ChatGPT4 very grammatically and confidently completely missed every significant point, again in a very backwards way. Worse yet, when asked to tell us the paper's publication date, author, and author's employer, all written clearly at the top, it confidently got everything massively wrong. Even after being told to look on the line after "Author: ". Even after being told to look after "Author: FName". It eventually settled on the correct date, but even after being told FName, it continuously kept spouting false authors (sometimes including correct FName) and hallucinated middle and surnames and employers. Interestingly enough, most of the hallucinated authors and employers were well known in the region. So, it is literally establishing the regional context and just spitting out plausibly relevant stuff for that region, but not even figuring out how to summarize something that a 6yo kid could easily sort. It was hilariously bad. And only hilarious because it was a test (if someone actually relied upon it, it'd be very costly). Similar results for my field. I got ChatGPT4 to produce some genuinely useful information. But it would also happily hallucinate papers, authors, companies, chemicals, and pretty much anything else. Middle-of-the-road, where ChatGPT4 has ingested insane amounts of data, it probably knows more than we do, and may be useful for some things. As soon as you go even slightly off the centerline, it becomes VERY unreliable. If you take it at face value, you are a fool, being fooled. EDIT: typos, readability
- IanCal 4y ago
  The chat context isn't as big as is available in the API - was the author name in its context window? Papers can be fairly long
barrysteve 4y ago
I mean, I have tried GPT4. I don't think the "it's a parrot" argument means to dismiss benefits of GPT, but apparently the internet discussion has leant into those two warring camps. It doesn't help me writing Unreal C++ code, though it's heavily documented online. It makes up functions that are not accessible from the namespace I'm using, cannot supply valid types to parameters and other basic errors like that. I asked GPT4 to give me specific examples of the meaning of the word "silly" across the past 1000yrs. It gave me the authors and examples I was expecting to read, but the meanings were inaccurate and misleading. If there's GPT Power I'm missing out on, please tell me where it is. It's probably my lack of skill, that the machine cannot give me an experience that would push me into the pro-GPT cheerleading camp.
reso 4y ago
The mistakes it makes are not "minor issues". I suspect that the mistakes are the rule, not the exception. The reason this isn't obvious at first is that the range of questions humans are likely to ask it is relatively constrained, and most are similar to questions that have been asked and answered in its training corpus, which is enormous. I asked it to describe myself recently, by my name and country of residence, and it replied with the correct occupation (founder), and then completely hallucinated a company I have never created. The more you ask it minute tidbits of information, the more it falls over into a bullshitting state.

skilled 4y ago

I agree, GPT-4 has a much more "aggressive" confidence, I definitely noticed that, but I haven't seen much of it since the API is still behind closed doors. I'd also like to point out that even though it's better, it still can't reason very well and relies heavily on prompts and instructions, even for very basic projects. It kind of reminds me a pseudo-visual code builder. I was writing a Python script with it to put together documentation summaries, and over the course of an hour it offered me 3 different libraries to connect to an API endpoint I was using. It's distracting.

SketchySeaBeast 4y ago
> GPT-4 has a much more "aggressive" confidence Could you expand on this? I only use the unpaid API, but when I ask for, say, song lyrics, it'll be 100% wrong and then it will give me a second version of the wrong lyrics when I say the first was wrong, and then when I ask how confident it is in the new lyrics it'll say 95%, so whatever the current free API is it's undeservedly confident. I must be misunderstanding what you mean by confidence.
- skilled 4y ago
  I can't really expand because I don't want to share my primary use case, I hope you understand. That said, what I mean is that it feels a lot more comprehensive, as if it really has a grasp on what is being asked, kind of like - its much better at understanding the information it has, so responses feel richer, both in context but also in tone. I am comparing it directly to 3.5. Again, I don't have the API access so I cannot say for sure, this was only my experience from the ChatGPT interface and also a third-party API I spent $20 on just to test its limits, but sadly for the time being those limits are hardcore - it just gets stuck on long responses despite there being a token increase. So, if I really wanted to test it out and make a proper assessment, I'd need access to the API myself so I can do some high-volume token shenanigans.
spookthesunset 4y ago
I’ve had it do stuff like that. In between updates to fix issues sometimes it decides to completely rewrite the thing you are working on.

zekica 4y ago

It returns factually incorrect data, and it returns code with subtle but important errors if you ask it anything that's not regurgitated a thousand times in the training dataset. Don't get me wrong, it has emergent properties (more than you would expect from a fancy autocomplete), but factual output was never GPT-4's nor any other LLM's design goal.

progrus 4y ago
I’ve started using it as a sort of “shotgun approach” - characterize what you want without trying too hard, let it toss out a jumble of interesting snippets that are often very good leads, and then take over refactoring from there. I do agree that others who don’t understand what it’s doing may not even review the code output, and I predict that this will cause at least one death in the next ten years.
- musha68k 4y ago
  *weeks/*months
  - progrus 4y ago
    Facts, sadly.
- bodge5000 4y ago
  I suppose it depends what specifically you're getting it to do, but I've found in my testing the shotgun approach to be more work than just writing it out myself. Honestly I've found its biggest value in learning a new language, it outputs with so many errors that it forces me to get in deep with debugging it. I haven't tried gpt-4 yet though, maybe its a vast improvement, I'm just not using it enough to justify the costs
  - progrus 4y ago
    It really depends on how comfortable you are with the language you’re using, and how recently you switched contexts. For anything unfamiliar, it’s pretty impressive how much it can help. In fact, I bet if you did an experiment comparing people reading a textbook about a somewhat-familiar language, vs. just asking GPT to do things and noticing the bugs, the latter group would reach proficiency faster.

redleggedfrog 4y ago

"I think people writing all this should stop and try GPT4." Have you? I mean, for something that's actually useful beyond just simple stuff, like at your job where it's going to matter? Because for what I'm doing which is custom ERP/Website data flows to/from legacy systems it's mostly a time sink and offers little value.

throwuwu 4y ago
Pretty much by definition its not going to be good at highly custom or uncommon programming tasks since it won’t have seen many examples to generalize from. I doubt you could get better results from a freshly hired human either.
- mdp2021 4y ago
  > from a freshly hired human either But the human will learn, through reasoning and challenges... Those features of which some wonder whether they will emerge are critical pieces.
  - pmoriarty 4y ago
    "But the human will learn" That depends on the human, what they're trying to learn, and how it's taught to them. Some people with brain damage, learning disabilities, other cognitive issues, with poor teachers or learning materials, or with too difficult a task will not learn. It might be useful to view AIs as akin in some ways to brain damaged humans, who might be good at some tasks but deficient at others.
    - hajile 4y ago
      Those people also won’t be hired as coders. There are other things that they will excel at.
- redleggedfrog 4y ago
  LOL! You got that right! They don't.

oldstrangers 4y ago

It struggles to accurately call its own API when prompted to. It patches together parts of GPT 3 / 3.5 and has no awareness of GPT4 API calls (at least a few weeks ago). I've tried 3 separate projects with it (using different platforms from Firebase to Vercel) and it usually gets me 80% there but fails miserably at the nuance portion (its stuck at a CORS policy issue with my last project). From start to finish I've yet to be able to complete something with it.

waboremo 4y ago
How could it have awareness of gpt4 when it wasn't even in existence prior to 2021?
- oldstrangers 4y ago
  "How could GPT4 have awareness of GPT4" Seems like something they might want to include, especially when its a paid service.

toss1 4y ago

>> but the mistakes are not "parroted bullshit", but minor issues here and there. Nonsense. I'm not saying that it is not useful, but I've easily found ChatGPT produces far worse than "parroted bullshit", and does so reliably, and multiple times (repeated patterns of mistakes). Again, I have found ChatGPT4 (but not 3.5) useful and even educational in some cases, but it's usability requires far more skeptical and error-checked approach. As one example, since I have a top specialist attorney in the household, I started with some broad legal questions about their field, and then asked their opinion. ChatGPT4's answers were impressive to both of us, and especially the solid grammar. When the questions got more particular about their specialist area of the law, ChatGPT4's answers got shaky very fast. Asked about a particular somewhat obscure technique, it parroted some very nice-sounding bullshit, which got the concepts very backwards (but it was still very confident and grammatical). When asked to summarize a published paper about a new innovation in this context, ChatGPT4 very grammatically and confidently completely missed every significant point, again in a very backwards way. Worse yet, when asked to tell us the paper's publication date, author, and author's employer, all written clearly at the top, it confidently got everything massively wrong. Even after being told to look on the line after "Author: ". Even after being told to look after "Author: FName". It eventually settled on the correct date, but even after being told FName, it continuously kept spouting false authors (sometimes including correct FName) and hallucinated middle and surnames and employers. Interestingly enough, most of the hallucinated authors and employers were well known in the region. So, it is literally establishing the regional context and just spitting out plausibly relevant stuff for that region, but not even figuring out how to summarize something that a 6yo kid could easily sort. It was hilariously bad. And only hilarious because it was a test (if someone actually relied upon it, it'd be very costly). Similar results for my field. I got ChatGPT4 to produce some genuinely useful information. But it would also happily hallucinate papers, authors, companies, chemicals, and pretty much anything else. Middle-of-the-road, where ChatGPT4 has ingested insane amounts of data, it probably knows more than we do, and may be useful for some things. As soon as you go even slightly off the centerline, it becomes VERY unreliable. If you take it at face value, you are a fool, being fooled. EDIT: typos, readability

IanCal 4y ago
The chat context isn't as big as is available in the API - was the author name in its context window? Papers can be fairly long

barrysteve 4y ago

I mean, I have tried GPT4. I don't think the "it's a parrot" argument means to dismiss benefits of GPT, but apparently the internet discussion has leant into those two warring camps. It doesn't help me writing Unreal C++ code, though it's heavily documented online. It makes up functions that are not accessible from the namespace I'm using, cannot supply valid types to parameters and other basic errors like that. I asked GPT4 to give me specific examples of the meaning of the word "silly" across the past 1000yrs. It gave me the authors and examples I was expecting to read, but the meanings were inaccurate and misleading. If there's GPT Power I'm missing out on, please tell me where it is. It's probably my lack of skill, that the machine cannot give me an experience that would push me into the pro-GPT cheerleading camp.

reso 4y ago

The mistakes it makes are not "minor issues". I suspect that the mistakes are the rule, not the exception. The reason this isn't obvious at first is that the range of questions humans are likely to ask it is relatively constrained, and most are similar to questions that have been asked and answered in its training corpus, which is enormous. I asked it to describe myself recently, by my name and country of residence, and it replied with the correct occupation (founder), and then completely hallucinated a company I have never created. The more you ask it minute tidbits of information, the more it falls over into a bullshitting state.