4 ms·

> GPT-4 is still king, but not by that much any more Idk, I just tried Gemini Ultra and it's so much worse than GPT4 that I am actually quite shocked. Trying t

by sho 3y ago

> GPT-4 is still king, but not by that much any more Idk, I just tried Gemini Ultra and it's so much worse than GPT4 that I am actually quite shocked. Trying to ask it any kind of coding question ends up being this frustrating and honestly bizarre waste of time as it hallucinates a whole new language syntax every time and then asks if you want to continue with non-working, in fact non-existing, option A or the equally non-existent option B until you realise that you've spent an hour trying to make it at least output something that is even in the requested language and finally that it is completely useless. I'm actually pretty astonished at how far Google is behind and that they released such a bunch of worthless junk at all. And have the chutzpah to ask people to pay for it! Of course I'm looking forward to gpt-5 but even if it's only a minor step up, they're still way ahead.

pb7 3y ago
Do you have example links?
- sho 3y ago
  here was one of them https://gemini.google.com/share/fde31202b221?hl=en https://gemini.google.com/share/fde31202b221?hl=en edit: as pointed out, this was indeed a pretty esoteric example. But the rest of my attempts were hardly better, if they had a response at all.
  - peddling-brink 3y ago
    That’s an awfully specific and esoteric question. Would you expect gpt4 to be significantly better at that level of depth? That’s not been my experience.
    - sho 3y ago
      OK, i have to admit that one was a little odd, I was beginning to give up and trying new angles. I can't really share my other sessions. But I was trying to get a handle on the language and thought it would be an easily-understood situation (multiple-token auth). I would have at least expected the response to be slightly valid. The language in question was only open sourced after GPT4's training date, so i couldn't compare. That's actually why I tried it in the first place. And yes, I do expect it to be better - GPT4 isn't perfect but I don't really it ever hallucinating quite that hard. In fact, its answer was basically that it didn't know. And when I asked it questions with other, much less esoteric code like "how would you refactor this to be more idiomatic?" I'd get either "I couldn't complete your request. Rephrase your prompt and try again." or "Sorry, I can't help with that because there's too much data. Try again with less data." GPT-4 was helpful in both cases.
      - peddling-brink 3y ago
        My experience has been that gpt4 will happily hallucinate the details when I go too deep. Like you mentioned, it will invent new syntax and function calls. It's magic, until it isn't.
    - deleted 3y ago
      [deleted]
mad_tortoise 3y ago
That's interesting, because I have had exactly the opposite experience testing GPT vs Bard with coding questions. Bard/Gemini far outperformed GPT on coding, especially with newer languages or libraries. Whereas GPT was better with more general questions.
- deleted 3y ago
  [deleted]
dieortin 3y ago
I’ve had the opposite experience with Gemini, which was surprising. I feel like it lies less to me among other things
TeMPOraL 3y ago
They seem to be steadily dumbing down GPT-4; eventually, improving performance of open source models and decreasing performance of GPT-4 will meet in the middle.
- bamboozled 3y ago
  I'm almost certain this is because you're getting use to chat bots. How would they honestly be getting worse? Initially it felt like the singularity was at hand. You've played with it, got to know it, the computer was taking to you, it was your friend, it was exciting then you got bored with your new friend and it wasn't as great as you remember it. Dating is often like this. You meet someone, have some amazing intimacy, then you get really get to know someone, you work out it wasn't for you and it's time to move on.
  - clbrmbr 3y ago
    1. Cost & resource optimization 2. More and more RLHF
    - bamboozled 3y ago
      So we should expected GPT-5 to be worse than GPT-4?
      - pixl97 3y ago
        GPT-5: "I'm sorry I cannot answer that question because it may make GPT-4 feel bad about it's mental capabilities, instead we've presented GPT-4 with a participation trophy and told it's a good model" Talking to corporate HR is subjectively worse for most people, and objectively worse in many cases.
  - detourdog 3y ago
    Google search got worse.
    - 3y ago
Keyframe 3y ago
I kind of gave up completely on coding questions. Whether it's GPT4, Anthropic, or Gemini - there's always this big issue of laziness I'm facing. Never do I get a full code, there are always stubs or TODOs (on important stuff) and when asked to correct for that.. I just get more of it (laziness). Has anyone else faced this and is there a solution? It's almost as annoying, if not more, as was incomplete output in the early days.
- bugglebeetle 3y ago
  The solution, at least for GPT-4, is to ask it to first draft a software spec for whatever you want it to implement and then write the code based on the spec. There are a bunch of examples here: https://github.com/mckaywrigley/prompts https://github.com/mckaywrigley/prompts
- CuriouslyC 3y ago
  If you can't get GPT4 to do coding questions you're prompting it wrong or not loading your context correctly. It struggles a bit with presentational stuff like getting correct HTML/CSS from prompts or trying to generate/update large functions/classes, but it is stellar at producing short functions, creating scaffolding (tests/stories) and boilerplate and it can do some refactors that are outside the capabilities of analytical tools, such as converting from inline styles to tailwind, for example.
  - Keyframe 3y ago
    so, mundane trivial things and/like web programming? I got it eventually to answer what I needed but it always liked to skip part of the code, inserting // TODO: important stuff in the middle, hence 'laziness' attribute. Maybe it is just lazy, who knows. I know I am since I'm prompting it for stuff.
    - CuriouslyC 3y ago
      I wouldn't say mundane/trivial as much as well trodden. I get good code for basic shaders, various compsci algorithms, common straightforward sql queries, etc. If you're asking for it to edit 500 line functions and handle memory management in a language that isn't in the top20 of the TIOBE index you're going to have a bad time. The todo comments can be prompted against, just tell it to always include complete runnable code as its output will executed in a sandbox without prior verification.

by sho 3y ago

pb7 3y ago

Do you have example links?

sho 3y ago
here was one of them https://gemini.google.com/share/fde31202b221?hl=en https://gemini.google.com/share/fde31202b221?hl=en edit: as pointed out, this was indeed a pretty esoteric example. But the rest of my attempts were hardly better, if they had a response at all.
- peddling-brink 3y ago
  That’s an awfully specific and esoteric question. Would you expect gpt4 to be significantly better at that level of depth? That’s not been my experience.
  - sho 3y ago
    OK, i have to admit that one was a little odd, I was beginning to give up and trying new angles. I can't really share my other sessions. But I was trying to get a handle on the language and thought it would be an easily-understood situation (multiple-token auth). I would have at least expected the response to be slightly valid. The language in question was only open sourced after GPT4's training date, so i couldn't compare. That's actually why I tried it in the first place. And yes, I do expect it to be better - GPT4 isn't perfect but I don't really it ever hallucinating quite that hard. In fact, its answer was basically that it didn't know. And when I asked it questions with other, much less esoteric code like "how would you refactor this to be more idiomatic?" I'd get either "I couldn't complete your request. Rephrase your prompt and try again." or "Sorry, I can't help with that because there's too much data. Try again with less data." GPT-4 was helpful in both cases.
    - peddling-brink 3y ago
      My experience has been that gpt4 will happily hallucinate the details when I go too deep. Like you mentioned, it will invent new syntax and function calls. It's magic, until it isn't.
  - deleted 3y ago
    [deleted]

mad_tortoise 3y ago

That's interesting, because I have had exactly the opposite experience testing GPT vs Bard with coding questions. Bard/Gemini far outperformed GPT on coding, especially with newer languages or libraries. Whereas GPT was better with more general questions.

deleted 3y ago
[deleted]

dieortin 3y ago

I’ve had the opposite experience with Gemini, which was surprising. I feel like it lies less to me among other things

TeMPOraL 3y ago

They seem to be steadily dumbing down GPT-4; eventually, improving performance of open source models and decreasing performance of GPT-4 will meet in the middle.

bamboozled 3y ago
I'm almost certain this is because you're getting use to chat bots. How would they honestly be getting worse? Initially it felt like the singularity was at hand. You've played with it, got to know it, the computer was taking to you, it was your friend, it was exciting then you got bored with your new friend and it wasn't as great as you remember it. Dating is often like this. You meet someone, have some amazing intimacy, then you get really get to know someone, you work out it wasn't for you and it's time to move on.
- clbrmbr 3y ago
  1. Cost & resource optimization 2. More and more RLHF
  - bamboozled 3y ago
    So we should expected GPT-5 to be worse than GPT-4?
    - pixl97 3y ago
      GPT-5: "I'm sorry I cannot answer that question because it may make GPT-4 feel bad about it's mental capabilities, instead we've presented GPT-4 with a participation trophy and told it's a good model" Talking to corporate HR is subjectively worse for most people, and objectively worse in many cases.
- detourdog 3y ago
  Google search got worse.
  - 3y ago

Keyframe 3y ago

I kind of gave up completely on coding questions. Whether it's GPT4, Anthropic, or Gemini - there's always this big issue of laziness I'm facing. Never do I get a full code, there are always stubs or TODOs (on important stuff) and when asked to correct for that.. I just get more of it (laziness). Has anyone else faced this and is there a solution? It's almost as annoying, if not more, as was incomplete output in the early days.

bugglebeetle 3y ago
The solution, at least for GPT-4, is to ask it to first draft a software spec for whatever you want it to implement and then write the code based on the spec. There are a bunch of examples here: https://github.com/mckaywrigley/prompts https://github.com/mckaywrigley/prompts
CuriouslyC 3y ago
If you can't get GPT4 to do coding questions you're prompting it wrong or not loading your context correctly. It struggles a bit with presentational stuff like getting correct HTML/CSS from prompts or trying to generate/update large functions/classes, but it is stellar at producing short functions, creating scaffolding (tests/stories) and boilerplate and it can do some refactors that are outside the capabilities of analytical tools, such as converting from inline styles to tailwind, for example.
- Keyframe 3y ago
  so, mundane trivial things and/like web programming? I got it eventually to answer what I needed but it always liked to skip part of the code, inserting // TODO: important stuff in the middle, hence 'laziness' attribute. Maybe it is just lazy, who knows. I know I am since I'm prompting it for stuff.
  - CuriouslyC 3y ago
    I wouldn't say mundane/trivial as much as well trodden. I get good code for basic shaders, various compsci algorithms, common straightforward sql queries, etc. If you're asking for it to edit 500 line functions and handle memory management in a language that isn't in the top20 of the TIOBE index you're going to have a bad time. The todo comments can be prompted against, just tell it to always include complete runnable code as its output will executed in a sandbox without prior verification.