4 ms·

What was the accuracy and the difference between googling the question ?

Toutouxc 4y ago
I can't say anything specific about the accuracy, other than that it's good enough, but the difference between asking GPT-4 and googling the same question is night and day. And that's comparing it to Kagi, with my own search filters with boosted wikipedia, stackoverflow and generally scientific sources. Do you know the feeling that you want to ask a specific question about a specific, technical aspect of something, and you wish you could just tell the search engine somehow that you're just not interested in hand-wavy popular science average Joe half-incorrect answers? And if you actually find the one good answer on page 14 of some god-forgotten thread from 2006, the question differs in some tiny but important detail from what you're after? Well that feeling doesn't exist with GPT-4. So far we've always been able to come up with something, together. If you don't like the first answer, you ask more questions. You can dig deeper. You can tell it what your guess is, what your intuition tells you about the problem, where exactly your uncertainty lies, e.g.: You: I don't understand phenomenon X. GPT: Oh that's easy, X is just [parrots wikipedia]. You: That's fine, but I don't understand how X differs from Y, they sound like the same phenomena. GPT: I see why you'd think that, but Y and X actually differ in this detail called Foo that makes all the difference. You: I still don't get it, compare Foo to something similar that I know from normal life. GPT: Okay, so Foo is like an elephant who's too large to drive a car. You: Ooh, I get it now.
- moonchrome 4y ago
  But here's what happens when I try that : Me : I don't understand this publicly documented AWS service with plenty of OSS examples, can you suggest how I would solve some edge case X GPT : Oh that's easy, invents bullshit that sounds exactly like what I need Me : Googles the shit the GPT came up with and can't find any references Me : Maybe there is something similar - spends more time searching Me : GPT you're wrong - your solution doesn't exist GPT : Oh sorry, here's the correct solution, comes up with more bulshit Me : Googles the shit that GPT came up with and can't find any references Me : Starts googling and solving the problem on my own Me : Finds out X can't be done with AWS service I've had this flow many times now, it never resulted in valuable output from GPT vs just Googling since I need to verify everything anyway. Or Me : Review and come up with improvements to this code : CP code GPT : Suggest bullshit improvements, gives factually wrong reasoning, makes code worse Me : Points out everything wrong GPT : Suggest other bullshit improvements Me : Points out everything wrong GPT : Returns basically identical code to input Or Me : Propose a design to this technical problem GPT : Implements the solution in the most roundabout way and suboptimal way IMO if I'm not sure what the output should be GPT is less than worthless it's actually convincingly misleading.
  - Toutouxc 4y ago
    Is that with 3.5 or with 4? 4 is usually not afraid to tell it doesn’t know.
    - moonchrome 4y ago
      I've upgraded to 4 ASAP, all those examples are with 4. I tried 3.5 buy it was completely useless, 4 actually started giving sane output but still lying and hard to verify. My only use case so far is stuff like "translate this model to openapi schema" because it was faster than setting up the tool, and similar tasks that are easy to verify and boring to type.

I can't say anything specific about the accuracy, other than that it's good enough, but the difference between asking GPT-4 and googling the same question is night and day. And that's comparing it to Kagi, with my own search filters with boosted wikipedia, stackoverflow and generally scientific sources. Do you know the feeling that you want to ask a specific question about a specific, technical aspect of something, and you wish you could just tell the search engine somehow that you're just not interested in hand-wavy popular science average Joe half-incorrect answers? And if you actually find the one good answer on page 14 of some god-forgotten thread from 2006, the question differs in some tiny but important detail from what you're after? Well that feeling doesn't exist with GPT-4. So far we've always been able to come up with something, together. If you don't like the first answer, you ask more questions. You can dig deeper. You can tell it what your guess is, what your intuition tells you about the problem, where exactly your uncertainty lies, e.g.: You: I don't understand phenomenon X. GPT: Oh that's easy, X is just [parrots wikipedia]. You: That's fine, but I don't understand how X differs from Y, they sound like the same phenomena. GPT: I see why you'd think that, but Y and X actually differ in this detail called Foo that makes all the difference. You: I still don't get it, compare Foo to something similar that I know from normal life. GPT: Okay, so Foo is like an elephant who's too large to drive a car. You: Ooh, I get it now.

moonchrome 4y ago
But here's what happens when I try that : Me : I don't understand this publicly documented AWS service with plenty of OSS examples, can you suggest how I would solve some edge case X GPT : Oh that's easy, invents bullshit that sounds exactly like what I need Me : Googles the shit the GPT came up with and can't find any references Me : Maybe there is something similar - spends more time searching Me : GPT you're wrong - your solution doesn't exist GPT : Oh sorry, here's the correct solution, comes up with more bulshit Me : Googles the shit that GPT came up with and can't find any references Me : Starts googling and solving the problem on my own Me : Finds out X can't be done with AWS service I've had this flow many times now, it never resulted in valuable output from GPT vs just Googling since I need to verify everything anyway. Or Me : Review and come up with improvements to this code : CP code GPT : Suggest bullshit improvements, gives factually wrong reasoning, makes code worse Me : Points out everything wrong GPT : Suggest other bullshit improvements Me : Points out everything wrong GPT : Returns basically identical code to input Or Me : Propose a design to this technical problem GPT : Implements the solution in the most roundabout way and suboptimal way IMO if I'm not sure what the output should be GPT is less than worthless it's actually convincingly misleading.
- Toutouxc 4y ago
  Is that with 3.5 or with 4? 4 is usually not afraid to tell it doesn’t know.
  - moonchrome 4y ago
    I've upgraded to 4 ASAP, all those examples are with 4. I tried 3.5 buy it was completely useless, 4 actually started giving sane output but still lying and hard to verify. My only use case so far is stuff like "translate this model to openapi schema" because it was faster than setting up the tool, and similar tasks that are easy to verify and boring to type.