7 ms·
Falcon 2
- tosh 3y agohuggingface model card: https://huggingface.co/tiiuae/falcon-11B https://huggingface.co/tiiuae/falcon-11B
- BryanLegend 3y ago"When tested against several prominent AI models in its class among pre-trained models, Falcon 2 11B surpasses the performance of Meta’s newly launched Llama 3 with 8 billion parameters (8B), and performs on par with Google’s Gemma 7B at first place, with a difference of only 0.01 average performance (Falcon 2 11B: 64.28 vs Gemma 7B: 64.29) according to the evaluation from Hugging Face." From: https://falconllm.tii.ae/falcon-2.html https://falconllm.tii.ae/falcon-2.html
- Hugsun 3y ago> New Falcon 2 11B Outperforms Meta’s Llama 3 8B, and Performs on par with leading Google Gemma 7B Model I was strongly under the impression that Llama 3 8B outperformed Gemma 7B on almost all metrics.
- JimDabell 3y agoAnecdotal, I know, but in my experience Gemma is absolutely worthless and Llama 3 8b is exceptionally good for its size. The idea that Gemma is ahead of Llama 3 is bizarre to me. Surely there’s some contamination or something if Gemma is showing up ahead in some benchmarks‽
- freedomben 3y agoAdding more anecdata, but this has been exactly my experience as well. I haven't dug into details about the benchmarks, but just trying to use the things for basic question asking, Llama 3 is so much better it's like comparing my Milwaukee drill to my sons Fisher Price plastic toy drill.
- hehdhdjehehegwv 3y agoYeah, Llama3 is also smoking Mistral/Mixtral. It’s my new model of choice.
- 7thpower 3y agoI found that curious too. I don’t stay up on the benchmarks much these days though; I’ve fully dedicated myself to b-ball. I’m actually a bit better than Lebron btw, who is nowhere near as good as my 3 year old daughter. I occasionally beat her. At basketball.
- coder543 3y agoKeep in mind that this is a comparison of base models, not chat tuned models, since Falcon-11B does not have a chat tuned model at this time. The chat tuning that Meta did seems better than the chat tuning on Gemma. Regardless, the Gemma 1.1 chat models have been fairly good in my experience, even if I think the Llama3 8B chat model is definitely better. CodeGemma 1.1 7B is especially underrated compared to my testing of other relevant coding models. The base CodeGemma 7B base model is one of the best models I’ve tested for code completion, and the chat model is one of the best models I’ve tested for writing code. Some other models seem to game the benchmarks better, but in real world use, don’t hold up as well as CodeGemma for me. I look forward to seeing how CodeLlama3 does, but it doesn't exist yet.
- Hugsun 3y agoThe model type is a good point. It's hard to track all the variables in this very fast paced field. Thank you for sharing your CodeGemma experience. I haven't found a emacs setup I'm satisfied with, using a local llm, but it will surely happen one day. Surely.
- attentive 3y agofor me, CodeGemma is super slow. I'd say 3-4 times slower than llama3. I am also looking forward to CodeLlama3 but I have a feeling Meta can't improve on llama3 it. Was there anything official from Meta?
- coder543 3y agoCodeGemma has fewer parameters than Llama3, so it absolutely should not be slower. That sounds like a configuration issue. Meta originally released Llama2 and CodeLlama, and CodeLlama vastly improved on Llama2 for coding tasks. Llama3-8B is okay at coding, but I think CodeGemma-1.1-7b-it is significantly better than Llama3-8B-Instruct, and possibly a little better than Llama3-70B-Instruct, so there is plenty of room for Meta to improve Llama3 in that regard. > Was there anything official from Meta? https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ "The text-based models we are releasing today are the first in the Llama 3 collection of models." Just a hint that they will be releasing more models in the same family, and CodeLlama3 seems like a given to me.
- latchkey 3y agoIt would be interesting to hear more about the compute used to build this.
- mmoskal 3y agoFrom model card: Falcon2-11B was trained on 1024 A100 40GB GPUs for the majority of the training, using a 3D parallelism strategy (TP=8, PP=1, DP=128) combined with ZeRO and Flash-Attention 2. Doesn't say how long though.
- kouteiheika 3y agoIt does say how long on Huggingface: > The model training took roughly two months.
- wg0 3y agoTime is coming when you'll be able to select out of hundreds of models and they'll be downloaded on demand if not already, would totally infer locally and offline even on your phone and I guess no more than 2030, we'll be there. I am not so up to date with the hardware landscape but don't think smart people would let not be noticing the need.
- treprinum 3y agoCloud models will always have edge compared to local models. Maybe in 2030 your iPhone would run GPT-4 on your phone but cloud GPT-9 will solve all your kids homework, do 95% of your job and manage your household.
- wg0 3y agoI'm pretty sure models are at their end game already. Now the law of diminishing returns is at play.
- treprinum 3y agoTransformers still didn't hit their scaling limits wrt number of tokens trained on and the number of layers. Model size limits are now purely given by finances ($100M/training run still seems to be too excessive).
- wg0 3y agoThat is true. And that financial limits might change however.
- simonw 3y agoThe license is not good: https://falconllm-staging.tii.ae/falcon-2-terms-and-conditions.html https://falconllm-staging.tii.ae/falcon-2-terms-and-conditio... It's a modified Apache 2 license with extra clauses that include a requirement to abide by their acceptable use policy, hosted here: https://falconllm-staging.tii.ae/falcon-2-acceptable-use-policy.html https://falconllm-staging.tii.ae/falcon-2-acceptable-use-pol... But... that modified Apache 2 license says the following: "The Acceptable Use Policy may be updated from time to time. You should monitor the web address at which the Acceptable Use Policy is hosted to ensure that your use of the Work or any Derivative Work complies with the updated Acceptable Use Policy." So no matter what you think of their current AUP they reserve the right to update it to anything they like in the future, and you'll have to abide by the new one! Great example of why I don't like the trend of calling licenses like this "open source" when they aren't compatible with the OSI definition.
- cs702 3y agoWell, that really sucks. Thanks But No Thanks.
- tantalor 3y agoIt's probably unenforceable
- Perz1val 3y agoMaybe, but would you risk trying?
- SiempreViernes 3y agoWhen it is backed by the UAE the muscle you have to contend with is not simply legal muscle, it also includes armed muscle of questionable moral fibre (see support for the RSF).
- CityOfThrowaway 3y agoDo you have any examples of the UAE using its military force to force foreign companies to abide a contract?
- dongobread 3y agoTheir benchmark results seem roughly on par with Mistral 7B and Llama 3 8B, which hardly seems that great given the increase in model size. https://huggingface.co/tiiuae/falcon-11B https://huggingface.co/tiiuae/falcon-11B https://huggingface.co/meta-llama/Meta-Llama-3-8B https://huggingface.co/meta-llama/Meta-Llama-3-8B https://mistral.ai/news/announcing-mistral-7b/ https://mistral.ai/news/announcing-mistral-7b/
- hehdhdjehehegwv 3y agoSeems to be the case with all their models - really huge in size, no actual performance gains for the effort. Their refined web dataset is heavily censored so maybe that has something to do with it. It’s very morally conservative - total exclusion of pornography and other topics. So I’d not be surprised if some of the issues are they are just filtering out too much content and adding more of the same instead.
- kaetemi 3y agoWhat? Falcon-7B base model is pretty much one of the only few small models that'll happily write a whole coherent fanfic all the way to the end without getting stuck in a loop right before the explicit content. Ignore instruct tunes.
- nabakin 3y agoExactly. Falcon-180b had a lot of hype at first but the community soon realized it was nearly worthless. Easily outperformed by smaller LLMs in the general case. Now they are back and claiming their falcon-11b LLM outperforms Llama 3 8b. I already see a number of issues with this: - falcon-11b is like 40% larger than Llama 3 8b so how can you compare them when they aren't in the same size class - their claim seems to be based on automated benchmarks when it has long been clear that automated benchmarks are not enough to make that claim - some of their automated benchmarks are wildly lower than Llama 3 8b's scores. It only beats Llama 3 8b on one benchmark and just barely. I can make an LLM does the best anyone has ever seen on one benchmark, but that doesn't mean my LLM is good. Far from it - clickbait headline with knowingly premature claims because there has been zero human evaluation testing - they claim their LLM is better than Llama 3 but completely ignore Llama 3 70b Honestly, it annoys me how much attention tiiuae get when they haven't produced anything useful and continue this misleading clickbait.
- iLoveOncall 3y ago> Outperforming Meta’s New Llama 3 I know it's hard to objectively rank LLMs, but those are really ridiculous ways to keep track of performance. If my reference of performance is (like the vast majority of users) ChatGPT-3.5, I have to first know how Llama 3 compares to that to then understand how that new models compare to what I'm using at the moment. Now, if I look for the performance of Llama 3 compared to ChatGPT-3.5, I don't find it on the official launch page https://ai.meta.com/blog/meta-llama-3/ https://ai.meta.com/blog/meta-llama-3/ where it is compared to Gemma 7B it, Mistral 7B Instruct, Gemini Pro 1.5 and Claude 3 Sonnet. How does Gemma 7B perform? Well you can only find out how it compares to Llama 2 on the official launch page https://blog.google/technology/developers/gemma-open-models/ https://blog.google/technology/developers/gemma-open-models/. Let's look at the Llama 2 performance on its launch announcement: https://llama.meta.com/llama2/ https://llama.meta.com/llama2/ No GPT-3.5 turbo again. I get that there are multiple aspects and that there's probably not one overall "performance" metric across all tasks, and I get that you can probably find a comparative between two specific models relatively easily, but there absolutely needs to be a standard by which those performances are communicated. The number of hoops to jump through is ridiculous.
- coder543 3y agoHuman preference data from side by side, anonymous comparisons of models: https://leaderboard.lmsys.org/ https://leaderboard.lmsys.org/ Llama3 8B significantly outperforms ChatGPT-3.5, and LLama3 70B is significantly better than that. These are ELO ratings, so it would not be accurate to try to say X is 10% better than Y because the score is 10% higher. Obviously Falcon 2 is too new to be on the leaderboard yet. Honestly, I don't think anybody should be using ChatGPT-3.5 as a chatbot at this point. Google and Meta both offer free chatbots that are significantly better than ChatGPT-3.5, among other options.
- iLoveOncall 3y ago> Honestly, I don't think anybody should be using ChatGPT-3.5 as a chatbot at this point. Google and Meta both offer free chatbots that are significantly better than ChatGPT-3.5, among other options. Yet I guarantee you that ChatGPT-3.5 has 95% of the "direct to consumer" marketshare. Unless you're a technical user, you haven't even heard about any alternative, let alone used them. Now onto the ranking, I perfectly recognized in my original comment that those comparisons exist, just that they're not highlighted properly in any launch announcement of any new model. I haven't used Llama, only ChatGPT and the multiple versions of Claude 2 and 3. How am I supposed to know if this Falcon 2 thing is even worth looking at beyond the first paragraph if I have to compare it to a specific model that I haven't used before?
- vessenes 3y agoI welcome open models, although the Falcon model is not super open, as noted here. I will say that the original Falcon did not perform as well as its benchmark stats indicated -- it was pushed out as a significant leap forward, and I didn't find it outperformed competitive open models at release. The PR stating an 11B model outperforms 7B and 8B models 'in the same class' feels like it might be stretching a bit. We'll see -- I'll definitely give this a go for local inference. But, my gut is that finetuned llama 3 8B is probably best in class...this week.
- htrp 3y ago> I will say that the original Falcon did not perform as well as its benchmark stats indicated Yea I saw that as well. I believe it was undertrained in terms of parameters vs tokens because they really just wanted to have a 40bn parameter model (like pre chinchilla optimal)
- vessenes 3y agoIt's hard to know if there's any special sauce here, but the internet so far has decided "meh" on these models. I think it's an interesting choice to put it out as tech competitive. Stats say this one was trained on 5T tokens. For reference, Llama 3 so far was reported at 15T. There is no way you get back what you lost in training by expanding parameters 3B. If I were in charge of UAE PR and this project, I'd a) buy a lot more H100s and get the training budget up b) compete on a regional / messaging / national freedom angle c) fully open license it I guess I'm saying I'd copy Zuck's plan, with oil money instead of social money and play to my base. Overstating capabilities doesn't give you a lot of benefit out of a local market, unfortunately.
- database_lost 3y agoFirst headline in bold: "Next-Gen Falcon 2 Series launches [...] and is only AI Model with Vision-to-Language Capabilities" ...
- artninja1988 3y ago11b model outperforms 8b model, news at 11
- jl6 3y ago"only AI Model with Vision-to-Language Capabilities" What do they mean by this? Isn't this roughly what GPT-4 Vision and LLaVA do?
- tictacttoe 3y agoAnd all Claude models…
- Me1000 3y agoAnd Gemini.
- Hugsun 3y agoAt first I thought they were playing some semantic game. Something like LLaVA being a language to vision model but I can't steelman the idea so it makes sense. Maybe they're just lying?
- bbor 3y agoAbsurdly biased article, cmon UAE be more subtle! “Beats llama 3” is a dubiously helpful summary, and this is just baffling: and is only AI Model with Vision-to-Language Capabilities
- pimlottc 3y agoFor a moment, I thought this might be related to the classic flight sim: https://en.wikipedia.org/wiki/Falcon_4.0 https://en.wikipedia.org/wiki/Falcon_4.0
- nordsieck 3y agoAlso, SpaceX has the Falcon 1 and Falcon 9 rockets, as well as the proposed but never developed Falcon 5.
- rationalfaith 3y ago[dead]
- j-pb 3y agoThese reminders that AI will not only be wielded by democracies with (at least partial attempts at) ethical oversight, but also by the worst of the worst autocrats, are truly chilling.
- logicchains 3y ago>but also by the worst of the worst autocrats MBZ (note MBZ is not MBS; Saudia Arabia and UAE are two different countries!) is one of the most popular leaders in the world and his people among the wealthiest. His country is one of the few developed countries in the world where the economy is still growing steadily, and one of the safest countries in the world outside of East Asia, in spite of having one of the world's most liberal immigration policies. Much more a contender for the best of the best autocrats than the worst of the worst.
- interpjinx 3y ago[dead]
- redleader55 3y agoI want to understand something: the model was trained on mostly a public dataset(?), with hardware from AWS, using well-known algorithms and techniques. How is it different from other models that anyone that has the money can train? My skeptic/hater(?) mentality, sees this as only a "flex" and an effort to try be seen as relevant. Is there more to this kind of effort that I'm not seeing?
- andy99 3y agoA lot of models are in this category. Sovereignty (whether national or corporate) has some value. And the threat of competition is a good thing for everyone. I'm glad people are working on these even if the end result in most cases isn't anything particularly interesting.
- adt 3y ago"With the release of Falcon 2 11B, we've introduced the first model in the Falcon 2 series." https://lifearchitect.ai/models-table/ https://lifearchitect.ai/models-table/
- GuB-42 3y agoI am a bit disappointed that it isn't about a new, small rocket from SpaceX with two first-stage engines.
- jwblackwell 3y agoThe speed things are moving, it feels like we'll get a GPT-4 level "small" model really soon.
- cedws 3y agoI guess that explains why OpenAI are rushing to make their models free despite having paying users. They don't want to lose market share to local LLMs just yet.
- renonce 3y agoSo Falcon 2 with 11B params outperform Llama 3 8B? With more parameters that doesn’t make a fair comparison. The strongest open source model seems to be Llama 3 70B, why claim outperforming Llama 3 when you didn’t outperform the best model?
- hypertexthero 3y agoSigh, I thought this was going to be about Spectrum Holobyte’s Falcon AT. From MyAbandonware.com: > Essentially Falcon 2 but somehow marketed differently, Falcon AT is the second release in Spectrum Holobyte's revolutionary hard-core flight sim Falcon series. Despite popular belief that Falcon 3.0 was THE dawn of modern flight sims, Falcon AT actually is already a huge leap over Falcon, sporting sharp EGA graphics, and a lot of realistic options and greatly expanded campaigns. The game is still the simulation of modern air combat, complete with excellent tutorials, varied missions, and accurate flight dynamics that Falcon fans have come to know and love. Among its host of innovations is the amazingly playable multiplayer options -- including hotseat and over the modem. Largely forgotten now, Falcon AT serves to explain the otherwise inexplicable gap between Falcon and Falcon 3.0.
- jhbadger 3y agoThere seems to be a trend of people naming new things (perhaps unintentionally) after classic computer games. We just had a post here on a system called Loom which apparently isn't the classic adventure game. I'm half expecting someone to come up with an LLM or piece of networking software and name it Zork.
- mdaniel 3y agoIt doesn't help any that currently "F-16 Strike Eagle II reverse engineering" <https://news.ycombinator.com/item?id=40347662 https://news.ycombinator.com/item?id=40347662> is also on the front page, "priming" one to think similarly