3 ms·

I'm currently trying out Claude 3 (Opus) side by side with ChatGPT (mostly using 4o, but have premium). So far it's pretty much on par, sometimes Claude gets it

by peheje 2y ago

I'm currently trying out Claude 3 (Opus) side by side with ChatGPT (mostly using 4o, but have premium). So far it's pretty much on par, sometimes Claude gets it better sometimes ChatGPT. I will say the ones where Claude did better was technical in nature. But.. still experimenting.

mistrial9 2y ago
what does "better" mean here?
- ramon156 2y ago
  More convincing (;
  - Zambyte 2y ago
    In my case, code that runs is more convincing than code that doesn't. Also it's useful to ask questions that you already know the answer to, in order to understand its limits and how it fails. In that case, "better" means more accurate and appropriate.
    - mistrial9 2y ago
      a teacher once told me, three are three kinds of questions. One is factual, that is a valid answer is maybe a number or details of an event that is documented.. lots of computer things or science knowledge; Second question purely for an opinion .. "Do you like house music?" .. there is no correct answer it is an opinion.. but the Third might be called a "well-reasoned judgement" .. that is often in the realm of decisions.. there are factors, not everything about it is known.. goals or culture outside of the question might shape the acceptable answers.. law certainly.. lots of business things.. extending that to an LLM, perhaps language translation sits as a "3rd type" on top of those three types.. translating a question or answer into another spoken language.. or via an intermediate model of some kind .. but that is going "meta" .. the point is, there are different kinds of questions and answers, and they dont all fit in the same buckets if "testing" an LLM for better..
- peheje 2y ago
  Few examples. One time I asked about reading/filtering JSON in Azure SQL. Claude suggested a feature I didn't know of OPENJSON. ChatGPT did not, but used a more generalize SQL technique - the CTE. Another time I asked about terror attacks in France. Here Claude immediately summarized the motives behind, whereas ChatGPT didn't. Lastly I asked for a summary of the Dune book, as I read it a few years ago and wanted to read Dune Dark Messiah (after watching part 2 of the 2024 movie, which concludes the Dune 1 book). Here ChatGPT was more structured (which I liked) and detailed, whereas Claude's summary was more fluent but left out important details (I specifically said spoilers was ok). Claude don't have access to searching internet or making plots. ChatGPT seems more mature with access to Wolfram alpha, latex for rendering math, matplotlib for making plots etc.
CuriouslyC 2y ago
I find Claude tends to be better at creative writing and to provide more thoughtful answers. Claude also tends to write more elegant code than GPT, but that code tends to be incorrect slightly more often as well. It tends to get confused by questions that aren't clearly worded that GPT handles in stride though.
- crooked-v 2y ago
  I've found Claude useless for writing purposes (even rubber-duck brainstorming), because it eventually but inevitably makes everything more and more sesquipedalian, ignoring all instructions to the contrary, until every response is just a garbage mess of purple prose rephrasing the same thing over and over again. I don't know what the deal is, but it's a failure state I've seen consistently enough that I suspect it has to be some kind of issue at the intersection of training material and the long context window.
  - Aeolun 2y ago
    Huh, my experience is that Claude was hopelessly terse. It’d be nice if I wanted a two paragraph summary.
  - shmeeed 2y ago
    Thank you for introducing me to "sesquipedalian", a word I've never seen before in over 20 years of venturing in the anglosphere, but one which I, as a native speaker of an Awful (and very sesquipedalian) Language, instantly fell in love with. :)
  - digging 2y ago
    I wanted to like Claude but for all my trying I could not get even Opus to understand brevity. I found myself repeating variations of "do not over-explain. just give brief answers until I ask for more" over and over until I cancelled my subscription in frustration. I am sure there is a technical skill in getting Claude to shut the hell up and answer, but I shouldn't have to suss out its arcane secrets. There should be a checkbox.

by peheje 2y ago

mistrial9 2y ago

what does "better" mean here?

ramon156 2y ago
More convincing (;
- Zambyte 2y ago
  In my case, code that runs is more convincing than code that doesn't. Also it's useful to ask questions that you already know the answer to, in order to understand its limits and how it fails. In that case, "better" means more accurate and appropriate.
  - mistrial9 2y ago
    a teacher once told me, three are three kinds of questions. One is factual, that is a valid answer is maybe a number or details of an event that is documented.. lots of computer things or science knowledge; Second question purely for an opinion .. "Do you like house music?" .. there is no correct answer it is an opinion.. but the Third might be called a "well-reasoned judgement" .. that is often in the realm of decisions.. there are factors, not everything about it is known.. goals or culture outside of the question might shape the acceptable answers.. law certainly.. lots of business things.. extending that to an LLM, perhaps language translation sits as a "3rd type" on top of those three types.. translating a question or answer into another spoken language.. or via an intermediate model of some kind .. but that is going "meta" .. the point is, there are different kinds of questions and answers, and they dont all fit in the same buckets if "testing" an LLM for better..
peheje 2y ago
Few examples. One time I asked about reading/filtering JSON in Azure SQL. Claude suggested a feature I didn't know of OPENJSON. ChatGPT did not, but used a more generalize SQL technique - the CTE. Another time I asked about terror attacks in France. Here Claude immediately summarized the motives behind, whereas ChatGPT didn't. Lastly I asked for a summary of the Dune book, as I read it a few years ago and wanted to read Dune Dark Messiah (after watching part 2 of the 2024 movie, which concludes the Dune 1 book). Here ChatGPT was more structured (which I liked) and detailed, whereas Claude's summary was more fluent but left out important details (I specifically said spoilers was ok). Claude don't have access to searching internet or making plots. ChatGPT seems more mature with access to Wolfram alpha, latex for rendering math, matplotlib for making plots etc.

CuriouslyC 2y ago

I find Claude tends to be better at creative writing and to provide more thoughtful answers. Claude also tends to write more elegant code than GPT, but that code tends to be incorrect slightly more often as well. It tends to get confused by questions that aren't clearly worded that GPT handles in stride though.

crooked-v 2y ago
I've found Claude useless for writing purposes (even rubber-duck brainstorming), because it eventually but inevitably makes everything more and more sesquipedalian, ignoring all instructions to the contrary, until every response is just a garbage mess of purple prose rephrasing the same thing over and over again. I don't know what the deal is, but it's a failure state I've seen consistently enough that I suspect it has to be some kind of issue at the intersection of training material and the long context window.
- Aeolun 2y ago
  Huh, my experience is that Claude was hopelessly terse. It’d be nice if I wanted a two paragraph summary.
- shmeeed 2y ago
  Thank you for introducing me to "sesquipedalian", a word I've never seen before in over 20 years of venturing in the anglosphere, but one which I, as a native speaker of an Awful (and very sesquipedalian) Language, instantly fell in love with. :)
- digging 2y ago
  I wanted to like Claude but for all my trying I could not get even Opus to understand brevity. I found myself repeating variations of "do not over-explain. just give brief answers until I ask for more" over and over until I cancelled my subscription in frustration. I am sure there is a technical skill in getting Claude to shut the hell up and answer, but I shouldn't have to suss out its arcane secrets. There should be a checkbox.