2 ms·

What is quality like compared to GPT4?

yieldcrv 3y ago
for the kinds of discussions I have it is very close I often have discussions about political topics that I don’t have a complete history on, things that are hard to get a non-emotional non-accusative response from in a forum or in person, licensing questions about obscure professions, liability questions, roleplays without the preachiness about the topic, coding, brand ideas and naming Lots of stuff that I dont want in any cloud or sent online, but then it just became second nature to primarily use it. the hallucinations are heavier. like it will give you specific links that it made up, and never apologize like chatgpt will, it will say “no, thats a real link” so, much more of a bullshitter LM Studio makes it easy to change the temperament though with a variety of premade system prompts, or allowing custom ones very easily I mainly use ChatGPT4 for multimodal like audio conversations, figuring a DIY problem out by sending it a photo of what I’m looking at, having it render photos on how to use something - I was at a gym and had it look at every piece of equipment there and tell me what it was and show me how a human would use it (I notice this is at odds with another comment, I havent used ChatGPT3.5 in nearly a year, but my experience with ChatGPT4 on the aforementioned topics is similar to my output in mixtral)
moralestapia 3y ago
I can answer that from experience (just built a couple Q&A style sites), although a bit subjectively. Compared with the GPTs, Mixtral (and derivatives) perform: * 5% of the time as good as GPT4 * 75% of the time on par with GPT3.5 * 20% of the time a bit worse than GPT3.5 (too chatty and hallucinates with ease, although one could argue a better prompt could improve things a lot) Advantages of Mixtral, for me: * Cost 10-100x cheaper * Faster completion time * More deterministic output, in the sense that if you run the same query several times you get the same answers back (but they could be wrong). GPT almost always gives me an answer of great quality BUT with a lot of variance across them; even with temperature set to zero. This is a PITA when you want some sort of predictable, structured output like a JSON object.
- sjwhevvvvvsj 3y ago
  Another factor is they monkey around with GPT constantly. The GPT you get on Monday may be totally weird on Friday. Whereas for a local model it doesn’t change unless YOU change it.

for the kinds of discussions I have it is very close I often have discussions about political topics that I don’t have a complete history on, things that are hard to get a non-emotional non-accusative response from in a forum or in person, licensing questions about obscure professions, liability questions, roleplays without the preachiness about the topic, coding, brand ideas and naming Lots of stuff that I dont want in any cloud or sent online, but then it just became second nature to primarily use it. the hallucinations are heavier. like it will give you specific links that it made up, and never apologize like chatgpt will, it will say “no, thats a real link” so, much more of a bullshitter LM Studio makes it easy to change the temperament though with a variety of premade system prompts, or allowing custom ones very easily I mainly use ChatGPT4 for multimodal like audio conversations, figuring a DIY problem out by sending it a photo of what I’m looking at, having it render photos on how to use something - I was at a gym and had it look at every piece of equipment there and tell me what it was and show me how a human would use it (I notice this is at odds with another comment, I havent used ChatGPT3.5 in nearly a year, but my experience with ChatGPT4 on the aforementioned topics is similar to my output in mixtral)

moralestapia 3y ago

I can answer that from experience (just built a couple Q&A style sites), although a bit subjectively. Compared with the GPTs, Mixtral (and derivatives) perform: * 5% of the time as good as GPT4 * 75% of the time on par with GPT3.5 * 20% of the time a bit worse than GPT3.5 (too chatty and hallucinates with ease, although one could argue a better prompt could improve things a lot) Advantages of Mixtral, for me: * Cost 10-100x cheaper * Faster completion time * More deterministic output, in the sense that if you run the same query several times you get the same answers back (but they could be wrong). GPT almost always gives me an answer of great quality BUT with a lot of variance across them; even with temperature set to zero. This is a PITA when you want some sort of predictable, structured output like a JSON object.

sjwhevvvvvsj 3y ago
Another factor is they monkey around with GPT constantly. The GPT you get on Monday may be totally weird on Friday. Whereas for a local model it doesn’t change unless YOU change it.