3 ms·
his conclusion is simultaneously not warranted and correct a like-for-like comparison would be GPT-4 against the larger models like LLaMA 65B, but those cannot
by muyuu 4y ago
his conclusion is simultaneously not warranted and correct
a like-for-like comparison would be GPT-4 against the larger models like LLaMA 65B, but those cannot be run on consumer-grade hardware
so one ends up comparing the stuff one can run... against the top stuff from OpenAI running on high-end GPU farms, and this technology clearly benefits a lot still from much larger scale than most people can afford
the great revelation this year is how much does it get better as it get much, much bigger without a clear horizon on where will diminishing returns be hit
but at the same time, some useful stuff can be done on consumer hardware - just not the most impressive stuff
- Szpadel 4y agoif "leaks" are true then even a 65B model is a toy when GPT-4 is supposed to be 1000B I would really like to have an open source model that could compete with even gpt3, but basically all lammas I tried failed to create correct email validation regex, where gpt3 is able to even refer to what rfc it is
- muyuu 4y ago13B on my computer does produce useful output, it's pretty hit and miss though GPT-3.5 OTOH is much better, but it's also much better at producing convincing-sounding but completely incorrect answers
- wokwokwok 4y agoThe 65B model runs fine on a Mac Studio with 64GB of memory. The output is unremarkable; it’s not significantly better than the 13B model for most uses. GPT 3.5 is an order of magnitude better at least.
- muyuu 4y agoYou cannot heavily quantise models and assume they didn't degrade significantly. To run it properly you need a lot more than a Mac Studio, and then comparisons need to be done more or less seriously, not just a few random prompts, because anything in a black box will "cheat" and will be fine tuned to do well at popular benchmarks.