5 ms·
I've been somewhat disappointed with the performance of the open models. The claims of certain models outperforming GPT-3.5-Turbo and approaching GPT-4 fail to
by drakenot 3y ago
I've been somewhat disappointed with the performance of the open models.
The claims of certain models outperforming GPT-3.5-Turbo and approaching GPT-4 fail to hold up to their benchmark results in real-world scenarios, potentially due to data contamination in assessments, based on my testing.
As noted in the linked survey paper, some models may outperform 3.5-Turbo in specific, narrow areas, depending on the model. Yet, we still lack a general model that definitively exceeds 3.5-Turbo in all respects.
I'm concerned that while we're still striving to reach 3.5-Turbo's performance level, OpenAI may unveil a new next-generation model, further widening the performance gap! Back in the summer, I had higher hopes that we would have surpassed the 3.5 threshold by now.
The performance gap has been surprisingly large. It is especially noticeable in areas requiring consistent structured output or tool use from the LLM. This is where open models particularly falter.
- snowycat 3y agoThere are tools that you can use to force the model to give structured output, such as llama.cpp GBNF grammar for example. They're a bit harder to use than ask gpt4 but they do work pretty well for what I use it for.
- white_dragon88 3y ago[dead]
- raincole 3y ago"OpenAI has no moat" aged so badly that it's almost a satire.
- Tostino 3y agoTry out my model vs gpt4 for the same tasks (I explicitly trained on) and compare. https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 https://huggingface.co/Tostino/Inkbot-13B-8k-0.2 It's a 13b param model that isn't meant to be general purpose, but is meant to excel on the limited tasks I've trained on. You'll see more like this soon.
- LoganDark 3y agoOpenAI has the benefit that it's a hosted service. Even if you can set something up at home, not everybody wants to do that.
- Tostino 3y agoI'm not competing with OpenAI... I did a whole bunch of work, and released it for anyone who wants to use it. It does what I trained it on well. Use it if you want to, or don't. Either way.
- LoganDark 3y agoNever meant to imply anything against your model. The fact that you released one at all is still more than I have to say for myself.
- choxi 3y agoAny suggestions for creating training data? Did you just manually create your own dataset or did you use any synthetic methods?
- Tostino 3y agoAbsolutely, pick a complicated problem and keep breaking it down with an existing model (whatever sota) until you have a consistent output for each step of your problem. And then stitch all the outputs together into a coherent single response for your training pipeline. After that you can do things like create q&a pairs about the input and output values that will help the model understand the relationships involved. With that, your training loss should be pretty reasonable for whatever task you are training. The other thing is, don't try and embed knowledge. Try and train thought patterns when specific knowledge is available in the context window.
- choxi 3y agoInteresting, thank you!
- ben_w 3y agoIt was a cliché almost immediately — "cliché" being just the fancy name we use when humans act like stochastic parrots.