5 ms·
FreeWilly 1 and 2, two new open-access LLMs
- ilaksh 3y agoThe https://freewilly.ai https://freewilly.ai website is just a link to the blog post right now.
- T-A 3y agoMeanwhile, there's https://huggingface.co/stabilityai/FreeWilly2 https://huggingface.co/stabilityai/FreeWilly2
- satvikpendem 3y agoI assume the names are a reference to the Orca model, as well as continuing the theme of naming LLMs after animals, like Falcon and Llama.
- courseofaction 3y agoAccording to my understanding the blog post, FreeWilly2 performs near or above ChatGPT4 for most test-cases. Is this true? Am I misunderstanding this? Is this not a big deal?
- SparkyMcUnicorn 3y agoWhere are you seeing GPT-4? All I see is "compares favorably with GPT-3.5 for some tasks".
- dan-g 3y agoThe post mentions GPT4All benchmarks—maybe that’s where the confusion lies? https://gpt4all.io/ https://gpt4all.io/
- bbatsell 3y agoThe benchmarks towards the bottom are of ChatGPT-4, according to the footnotes.
- deleted 3y ago[deleted]
- rnosov 3y agoThe report cites both GPT-3.5 and GPT-4 scores on page 7 [1]. I've checked the numbers and they compare FreeWilly2 to GPT-3.5. For example, HellaSwag score of 85.5% corresponds to GPT-3.5. [1] https://arxiv.org/pdf/2303.08774v3.pdf https://arxiv.org/pdf/2303.08774v3.pdf
- courseofaction 3y agoThe footnoted papers mention GPT-4 in the title. Unless they're citing GPT3.5 results from papers with GPT-4 the name, which seems confusing.
- lgas 3y agoThere are no actual footnote marks that connect any statements in the post to the footnotes, so there are no specific claims referenced. But if you read the actual text of the page, they say it compares favorably to 3.5 for some tasks. Which means it falls short of 3.5 for the rest and GPT-4 for all of them, or else they surely would have mentioned that as well.
- deleted 3y ago[deleted]
- emadm 3y agoIt beats GPT 3.5 in some benchmarks, the first open model to do so I believe. Versions being worked on now will do much better. GPT 4 is far better and will likely not be beaten by any current open models and approaches but maybe an ensemble of them.
- coder543 3y agoDoes this mean Stability gave up on StableLM? I notice that the repo hasn’t been updated since April, and a question asking for an update has been ignored for at least a month: https://github.com/Stability-AI/StableLM/issues/83 https://github.com/Stability-AI/StableLM/issues/83
- emadm 3y agoNot yet, update soon on that. Just had to change tack with LLaMA 2, we trained and released OpenLLaMA 13b in the meantime.
- theolivenbaum 3y agoIs the fine-tuning code also available?
- armatav 3y agoAre these models aligned?
- nomel 3y ago> Limitations and bias > Although the aforementioned dataset helps to steer the base language models into "safer" distributions of text, not all biases and toxicity can be mitigated through fine-tuning.
- gaogao 3y agoHuh, kind of weird that they don't have the chat-tuned Llama 2 in the comparison mix.
- lolinder 3y agoThey probably had this all put together before Llama 2 was available.
- turnsout 3y agoFreeWilly 2 is based on Llama 2
- lolinder 3y agoI stand corrected!
- turnsout 3y agoIt's just wild to me how fast they turned this around! Llama 2 was released 3 days ago, and we're already seeing fine-tuned variants.
- emadm 3y agoIt outperforms chat tuned LLaMA handily https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- deleted 3y ago[deleted]
- ozr 3y agoThe llama 2 chat models are completely neutered. They are practically unusable. I'm convinced it was joke from the Meta engineers.
- ssabev 3y agoGreat name. 11/10. Expect this to be really popular in the UK ;)
- dev_daftly 3y agoFree Willy 1 is a classic and holds up, Free Willy 2 doesn't
- victor9000 3y agoSo Free, as in 4 random letters that were strung together.
- version_five 3y agoGratisWilly?
- jojobas 3y agoCC-BY-NC 4.0 is pretty free?
- victor9000 3y agoA more thoughtful discussion on the subject: https://opensource.stackexchange.com/questions/1717/why-is-cc-by-sa-discouraged-for-code https://opensource.stackexchange.com/questions/1717/why-is-c...
- IanCal 3y agoIt's not code, that's also a discussion about a license not used here.
- avereveard 3y ago-NC is the pretty restrictive part.
- IanCal 3y agoIt is. I think people are perhaps leaping to the usual discussions when models are called "open source" but here it seems they've called it free. Wasn't the whole foundational part of the open source discussion about distinguishing open and free? Edit - someone will no doubt bring up "open access" as a term. This is a common term for academic work and the license here easily meets the criteria usually applied. Open access is not the same as open source.
- jrflowers 3y agoIt’s so good to see a non-commercial fork of llama 2
- c0brac0bra 3y agoWhy?
- jrflowers 3y agoIt’s good to see companies putting effort into making models that nobody will use, it protects us from skynet happening
- ilaksh 3y agoSomeone in the Stable Foundation Discord told me that FreeWilly1 codes better than FreeWilly2. Anyone can confirm?
- emadm 3y agoYes that would be the case
- monlockandkey 3y agoThis is impressive. It tops the hugging face leaderboard. I think we are inching closer and closer to chatgpt levels in 70B models. https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- davidkunz 3y agoGreat work, Stability! Note: It's "Llama 2", not "LLaMA 2", they changed the capitalization.
- swfsql 3y agoI'm out of context, but shouldn't it be possible to train a LLM-like model for images? (as an alternative to the stable diffusion process) If you rearrenge all pixels from square-sized images using the Hilbert curve, you should end-up with pixels arranged in 1D, and that shouldn't be much different from "word tokens" that LLMs are used to deal with, right? Like a LLM that only "talks" in pixels. This would have the benefit that you may be able to use various resolutions during training with the model still "converging" (since the Hilbert curve stabilizes towards infinite resolution). I'm not sure if the pixels would also need to be linearized, then maybe it could work to represent the RGB values as a 3D cube and also apply a 3D Hilbert curve on it, then you would have a 1D representation of all of the colors. I don't really know the subject but I guess something like that should be possible.
- famouswaffles 3y agohttps://sites.research.google/parti/ https://sites.research.google/parti/
- singhrac 3y agoNo need for a Hilbert curve, you can just flatten pixels the usual way (ie X = img.reshape(-1)). The main issue is that attention doesn’t scale that well, and with a 512x512 img the attended region is now 262k tokens, which is a lot. The other issue is that you’d throw away data linearizing colors (why not keep them 3-dimensional?). The corresponding work you’re looking for is Vision Transformers (ViT) - they work well, but not as great as LLMs, I think, for generation. Also I think people like that diffusion models are comparatively small and expensive - they’d rather wait than OOM.
- famouswaffles 3y agohttps://sites.research.google/parti/ https://sites.research.google/parti/