4 ms·

Does this guy really believe that GPT-4's answers are of the same quality as GPT-3.5? That's ridiculous.

Zambyte 2y ago
It's pretty dependant on the questions you ask. Obviously when you push them to their limits, GPT 4 goes a lot further. But what about for your average queries though? I don't have the data to make a proper claim about that.
- rafaelero 2y ago
  That's why we have Lmsys for. And spoiler alert, it performs better than 3.5 no matter the complexity of the question.
  - Zambyte 2y ago
    And are you sure people ask the chatbot arena the same caliber of questions they ask a single model? Wouldn't you tend to ask more difficult questions to find where one fails but the other doesn't? I'm not convinced that the average query will get a significantly better result from GPT 4. I only know that is true for more difficult queries.
    - rafaelero 2y ago
      Do you have access to ChatGPT? You can try it yourself (you can query GPT-4o and GPT-3.5 afterwards). The difference is large enough to be noticeable at least 80% of the time in my experience.
- limfop 2y ago
  > It's pretty dependant on the questions you ask. That’s the description of a house of cards rather than some revolutionary new technology.
ado__dev 2y ago
I agree that GPT4 is better, but the amount of times it still hallucinates is still way too high to be useful for any real work. LLMs at present are super impressive, but have very very low trust.(This is true for all models, Claude, Opus, etc.)
- rafaelero 2y ago
  More than 60% of programmers are already using it and finding it useful. Boston Consulting Group conducted a trial and concluded that it increases quality of their consultants work while decreasing the time it took to finish it. So no, you are not correct that hallucination makes it impossible to use.
  - twic 2y ago
    > More than 60% of programmers are already using it Firstly, there is absolutely no way anyone could know this. Secondly, no they aren't.
    - rafaelero 2y ago
      It's what surveys are showing. Why would they lie?
- Terretta 2y ago
  Hallucination, in my opinion, is mostly down to bad prompting. Just as one can say, in response to a perfectly correct answer, "Explain what's wrong with your answer" and it will find something, one can either create a vacuum it feels compelled to fill with hallucinations or ask in a way it stays grounded.
babuskov 2y ago
Recently I got free 30 minute trial access to GPT4 (and was prompted to upgrade if I liked it). I used the opportunity to send some queries which GPT-3.5 was bad at, and while GPT4 was better and closer to the correct answers, it still failed to produce them. When I pointed out the problems, it got even closer, but still ended up insisting that its (wrong) answer was correct in the end. So, yeah, GPT4 is much better than GPT3.5, but it's got a long way to go before it becomes real AI. Currently, I still use it for 3 cases: 1. When exploring something completely new. It saves time reading and learning because you can ask questions with "layman words" and get to learn the specific terms used in the domain quickly. 2. When google search is swamped by SEO spam or google insists on ignoring some of the keywords 3. When I need some boilerplate code that has been written a thousand times. It saves time which would otherwise get spent on concatenating different code segments from 5 different Stackoverflow threads and reading the docs afterwards to ensure it's correct. Now I just copy/paste from ChatGPT and read the docs afterwards to fix any places where it messed up.
- rafaelero 2y ago
  > So, yeah, GPT4 is much better than GPT3.5, but it's got a long way to go before it becomes real AI. I don't disagree. Just wanted to highlight the poor intuition of the guy doing the video.
- Workaccount2 2y ago
  You're doing it wrong if you are not feeding Chat the documentation first. It's not as great a gpt4O, but with gemini and 1M tokens (2M now also avaiable) you can dump in a small library of "writing javascript for X" books before asking it to produce code for X. It dramatically improves output. ChatGPT also can be front loaded with documents, but the context is much smaller (32K in the web app, I believe?)

It's pretty dependant on the questions you ask. Obviously when you push them to their limits, GPT 4 goes a lot further. But what about for your average queries though? I don't have the data to make a proper claim about that.

rafaelero 2y ago
That's why we have Lmsys for. And spoiler alert, it performs better than 3.5 no matter the complexity of the question.
- Zambyte 2y ago
  And are you sure people ask the chatbot arena the same caliber of questions they ask a single model? Wouldn't you tend to ask more difficult questions to find where one fails but the other doesn't? I'm not convinced that the average query will get a significantly better result from GPT 4. I only know that is true for more difficult queries.
  - rafaelero 2y ago
    Do you have access to ChatGPT? You can try it yourself (you can query GPT-4o and GPT-3.5 afterwards). The difference is large enough to be noticeable at least 80% of the time in my experience.
limfop 2y ago
> It's pretty dependant on the questions you ask. That’s the description of a house of cards rather than some revolutionary new technology.

ado__dev 2y ago

I agree that GPT4 is better, but the amount of times it still hallucinates is still way too high to be useful for any real work. LLMs at present are super impressive, but have very very low trust.(This is true for all models, Claude, Opus, etc.)

rafaelero 2y ago
More than 60% of programmers are already using it and finding it useful. Boston Consulting Group conducted a trial and concluded that it increases quality of their consultants work while decreasing the time it took to finish it. So no, you are not correct that hallucination makes it impossible to use.
- twic 2y ago
  > More than 60% of programmers are already using it Firstly, there is absolutely no way anyone could know this. Secondly, no they aren't.
  - rafaelero 2y ago
    It's what surveys are showing. Why would they lie?
Terretta 2y ago
Hallucination, in my opinion, is mostly down to bad prompting. Just as one can say, in response to a perfectly correct answer, "Explain what's wrong with your answer" and it will find something, one can either create a vacuum it feels compelled to fill with hallucinations or ask in a way it stays grounded.

babuskov 2y ago

Recently I got free 30 minute trial access to GPT4 (and was prompted to upgrade if I liked it). I used the opportunity to send some queries which GPT-3.5 was bad at, and while GPT4 was better and closer to the correct answers, it still failed to produce them. When I pointed out the problems, it got even closer, but still ended up insisting that its (wrong) answer was correct in the end. So, yeah, GPT4 is much better than GPT3.5, but it's got a long way to go before it becomes real AI. Currently, I still use it for 3 cases: 1. When exploring something completely new. It saves time reading and learning because you can ask questions with "layman words" and get to learn the specific terms used in the domain quickly. 2. When google search is swamped by SEO spam or google insists on ignoring some of the keywords 3. When I need some boilerplate code that has been written a thousand times. It saves time which would otherwise get spent on concatenating different code segments from 5 different Stackoverflow threads and reading the docs afterwards to ensure it's correct. Now I just copy/paste from ChatGPT and read the docs afterwards to fix any places where it messed up.

rafaelero 2y ago
> So, yeah, GPT4 is much better than GPT3.5, but it's got a long way to go before it becomes real AI. I don't disagree. Just wanted to highlight the poor intuition of the guy doing the video.
Workaccount2 2y ago
You're doing it wrong if you are not feeding Chat the documentation first. It's not as great a gpt4O, but with gemini and 1M tokens (2M now also avaiable) you can dump in a small library of "writing javascript for X" books before asking it to produce code for X. It dramatically improves output. ChatGPT also can be front loaded with documents, but the context is much smaller (32K in the web app, I believe?)