10 ms·
Falcon 40B LLM (which beats Llama) now Apache 2.0
- amstan 3y ago40B is pretty large, right? I expect it would take 70GB or so of RAM to run it. That's some expensive hardware ($10,000 or more).
- esperent 3y agoI briefly looked at prices a few days ago, I think you can rent an 80gb GPU for about $2.50 an hour. Then you just pay while you're using it. Someone else might be better able to confirm the pricing, but in any case you don't need to purchase the hardware.
- amstan 3y agoI'm less interested in using someone else's computer (not as much, but similar to how i'm disinterested in an API from someone like OpenAI), would rather pay the upfront hardware cost than worry about how many tokens i generate (kind of hinders creativity and excitment about it).
- gymbeaux 3y ago70GB of RAM would cost around $150 these days depending on how you get there. 64GB (2x32GB) of DDR4 is around $140 then another 8GB stick would be around $15. Used DDR3 ECC would be roughly half that.
- Semaphor 3y agoI think they are talking about VRAM
- gymbeaux 3y agooop you're right... I know once the model is trained they run on the CPU/regular RAM
- drdaeman 3y agoAnything that runs on GPU can be run on CPU, the only question is how much slower it's gonna be.
- Semaphor 3y agoI know, I have a weak GPU but 64 GB RAM. Using those models works, but it’s more "ask a question, then do something else for a while, while your fans spin up" ;)
- amstan 3y agoI've been running vicunda-13B on a workstation that can be aquired from ebay for about $500. It's slow compared to online services, but probably slightly faster than text to speech would recite its output, so plenty. Falcon 40B is probably too much for it, but apparently there's similar cheap hardware that could work.
- GaggiX 3y agoWith 4-bit quantization should take less than 40GB of VRAM/RAM.
- valvar 3y agoThat much good RAM in itself isn't super expensive. So does the rest of the hardware have to be particularly powerful?
- esquire_900 3y agoSome people use second hand P40 GPUs, which go for around 200-300$. Combine 3 of them with SLI and you've got 72GB of VRAM for less then $1000
- gengolas 3y agoI do use a P40 for my machine learning box, but I'm curious how you put three on the same system, given they need a CPU power plug and a pci-e port. Then, to cool them, you need to plug your own cooling system, requiring more specific power plugs to be available. What kind of chassis, motherboard, power unit you use to do that? It'll certainly will cost more than $1000 anyway, especially since you also need a decent amount of RAM to preload the models before you move them to the GPUs.
- esquire_900 3y agohttp://nonint.com/ http://nonint.com/ has some interesting posts about how he build a custom server to house 8 GPU's (3090's in this case). You're right that that will set you back more than $1000, though I was only referring to the GPU's themselves.
- amstan 3y agoWoah, that's a cool direction. Thank you! I'll explore this.
- washadjeffmad 3y agoP40s are kind of a meme. Using ggmls has roughly the same performance at a fraction of the wattage on a dual-channel DDR5 system. I still use GPTQ for 30B, but even CPU generates quickly enough at q5_1 on modern hardware.
- deleted 3y ago[deleted]
- Kelamir 3y ago> https://github.com/ggerganov/llama.cpp/issues/1602#issuecomment-1570827592 https://github.com/ggerganov/llama.cpp/issues/1602#issuecomm... Here somebody quantized it down to 29929.56MB .
- dheera 3y agoStupid question but for feed-forward models why do we not yet have some kind of CPU RAM memory swap mechanism? Why is Pytorch still trying to load the whole damn model into GPU RAM at once and then complaining when it can't, instead of swapping portions of the model to CPU RAM, or hell, even SSD? Sure, it might be a lot slower, but that's a lot better than "I give up, go buy $20K worth of hardware"
- coolspot 3y agoThat’s what llama.cpp does, including offload to a disk. It allows you to run models as big as any combination of your VRAM, RAM or disk. But in the end, if it doesn’t fit into GPU VRAM, it will be slow. For example Guanaco-33B generates ~10 token per second running fully from VRAM of my 3090, the ~1 token/second running from DDR4 RAM of my Ryzen. I would imagine it would do like a token per minute from NVM SSD.
- anonymousDan 3y agoIs it memory/disk bandwidth bound or latency bound?
- mike_hearn 3y agoMemory bandwidth bound.
- sebzim4500 3y agoI haven't run the numbers, but I would expect that doing that would make it slower than just running on the CPU.
- convexstrictly 3y agohttps://huggingface.co/tiiuae/falcon-40b https://huggingface.co/tiiuae/falcon-40b
- LrnByTeach 3y agoThis Falcon-40B royalty free license may force Meta ... that LLama-7B/13B may soon be fully open sourced as Meta wants open source LLM advancements and contributions on its own LLM architecture.
- convexstrictly 3y agoYann LeCun: "No. But it's not because we don't want to. It's because of complicated legal issues." https://twitter.com/ylecun/status/1651782621540524032 https://twitter.com/ylecun/status/1651782621540524032
- jacooper 3y agoComplicated legal issues of having the cake and eating it too.
- smoldesu 3y agoAnd also the potential, unprecedented legal issues that accompany releasing and defending a free/open model. The brownie points they'd receive aren't worth it, at least yet.
- az226 3y agoGPT2 was released open. It’s been years. Microsoft’s CELA has vetted it. If they are thumbs up, I don’t see why others would be averse. Unless it’s not the reason. Methinks it isn’t.
- brucethemoose2 3y agoMeta's eventual, internal LLM architecture will be totally different than the open source LLMs, right? They don’t need to run on commodity hardware. Maybe I am cynical, but I dont see the incentive for Meta to contribute an open model.
- version_five 3y ago
- version_five 3y agoI saw this, I still am highly suspicious about the future of these models and later attempts at monetization. Did they really just drop their 10% royalty thing and decide they'll just open source all their models now? Another comment mentions llama may get an open license, and there are other emerging alternatives. In six months there will be lots of options. I would not spend my time building anything around a model that started in such a sketchy way. It would be interesting to hear about why they decided to change their license and what their plans are for the future. I admit Falcon has really pissed my off because they pretended they had an open source model when they really released a sleazy freemium thing.
- convexstrictly 3y agoDo you consider the Unity model "sleazy"? But I agree that it was poor form to call it open-source at the time. Sadly, it seems to be standard practice in the LLM space to release models with all kinds of restrictions on use and call them "open source". Well now Falcon is open source. Given they are giving away something that was very expensive to train, I am grateful.
- version_five 3y agoI don't know unity's model. I call Falcon sleazy because the royalty stuff (plus a bunch of absurd related terms) were buried in the license while the headline said open source. I'm not against commercial products, nor products with free and premium tiers. This otoh felt, like I say, just sleazy.
- jrflowers 3y agoHere is a link to an explanation of the Unity model. https://gamedevbeginner.com/is-unity-free/ https://gamedevbeginner.com/is-unity-free/
- generalizations 3y agoThe explanation is less interesting than the fact they chose not to be upfront in their PR about the limitations of their original license. Personally, I don't care what terms they choose - but I do care if they misrepresent them. Honesty is important, and they weren't. But all that's void now that they've gone Apache.
- mikeravkine 3y ago4bit Quantized versions that run on an A100-40G or 2x3090/4090 24GB: https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-GPTQ Inference is very slow right now but it works!
- takeiteasyy 3y agoThis will require a cpp port first to run on e.g. apple silicon?
- wsgeorge 3y agoNo, but to benefit from the ggml ecosystem, yes. Someone's taking a stab at it: https://github.com/nikisalli/falcon.cpp https://github.com/nikisalli/falcon.cpp
- rovr138 3y agoThis is using huggingface. It's python. There's no reason it wouldn't run on arm. https://huggingface.co/tiiuae/falcon-40b https://huggingface.co/tiiuae/falcon-40b I'll also add that the fact something is in C++ doesn't mean it will run on arm or that it can be compiled in it.
- bongobingo1 3y agoThe hardware requirements on these models is basically at a fixed floor, and the democratisation will come from cheaper, possibly specialised, hardware, not reduced requirements, right?
- airgapstopgap 3y agoMulti-Query Attention, used here, should make 40B inference viable on systems where even 33B LLaMA with Multi-Head is basically unusable, so sometimes improvements still come from software optimization (it's no free lunch though).
- BonoboIO 3y agoWill be interesting to see if someone comes up with an ASIC or FPGA.
- fareesh 3y agoIn terms of building something that's usable (considering cost, speed, scale, etc) if comparing an OpenAI API call to these, it's difficult for me to see a current path where these have any viable application outside some niche scenario. From what I understand, even to run locally you/your team needs to be able to afford a machine with a 4090. These are super expensive in some countries. I played around with the smaller Llama/Alpaca models and it wasn't really viable to build anything with. Not really seeing a use-case for fine-tuning either compared to just few-shot prompting. Can someone fill me in on what I'm missing? It feels like I'm out of the loop
- smoldesu 3y agoI'm running Vicuna on a free 4core Oracle VPS, and it's perfectly usable for a Discord bot. Responses rarely take more than 15 seconds with <256 max token limit, and the responses are much more entertaining than GPT 3.5. I'm not using the streaming API my server software[0] offers, but if I did it would probably load somewhere between the speeds of GPT-3.5 and GPT-4. It's more or less the same time a human would take to compose the same message. So... not exactly a serious use-case. But it's what I'm using, and now I'm saving 10s of dollars on inferencing costs per month! [0] https://github.com/go-skynet/LocalAI https://github.com/go-skynet/LocalAI I'm also using this to improve acceleration - https://cloudmarketplace.oracle.com/marketplace/en_US/adf.task-flow?tabName=O&adf.tfDoc=%2FWEB-INF%2Ftaskflow%2Fadhtf.xml&application_id=125935163&adf.tfId=adhtf https://cloudmarketplace.oracle.com/marketplace/en_US/adf.ta...
- paulgb 3y agoNow I’m curious what your bot does!
- smoldesu 3y agoIt's one of those "say a keyword with a question, get a response" type bots. I added in a few other "prompt sources" though, where it grabs the first part of an RSS entry or HN comment and tries to autocomplete the rest. Mostly just a boring testbed for me to play with models, for free, with friends.
- gjreda 3y agoNot specific to this model, but beyond the large players (OpenAI, Cohere, etc) are there any free hosted versions of the open(ish) LLMs? Even the smaller 7B parameter ones? I'm prototyping out a project and using OpenAI for now, but it feels like there has to be a hosted alternative somewhere. I spent some time today exploring HuggingFace's Inference API but if the model is sufficiently large (> 10gb), HF requires you to use their commercial offerings.
- dzhiurgis 3y ago> HF requires you to use their commercial offerings Some of which are quite affordable ($80 per month). Larger ones can be like 2000 a month which is still ok to prototyping phase. You're basically paying for aws/gcp infrastructure. I quite liked the UX of it, very intuitive. My trouble was finding a model that executes out-of-the-box tho. All of the GPT ones crash on startup.
- YetAnotherNick 3y agohttps://chat.lmsys.org/ https://chat.lmsys.org/
- fnordpiglet 3y agoSerious question - why doesn’t someone crowd source the funds to train a GPT scale model for open source? I assume it’s not just a matter of a ton of GPU instances?
- djbusby 3y agoDoesn't one need to have a bunch of "very good" data to train on? I'm under the impression that sourcing costs are large.
- throwoutway 3y agoYes, and the labor of the Reinforcement Learning from Human Feedback (RLHF) .. is anyone crowdsourcing this?
- jahewson 3y agoAnthropic and Databricks have made datasets available. OpenAssistant is crowdsourcing one.
- fnordpiglet 3y agoThese seem like things affordable with money and crowd sourcing of volunteer labor. In some ways it feels too valuable to leave in the hands of megacorps.
- klysm 3y agoIf it turns out that there is a consistent architecture which works really well, how long before we see an ASIC?
- fennecfoxy 3y agoAs with everything, follow the money, pretty much. There will be an ASIC as soon as serious money is being made from LLMs, most use cases atm seem to be in prototype/toy stage, but I imagine we'll start seeing that change.
- yewenjie 3y agoFor people who directly want to check the benchmark - https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderb...
- kossTKR 3y agoVery cool but how does this compare to GPT-4 (before it was nerfed)? I feel like the best benchmark atm is the orig gpt-4 version.
- valine 3y agoGPT-4 wasn’t nerfed. https://news.ycombinator.com/item?id=36155267 https://news.ycombinator.com/item?id=36155267
- fbrncci 3y agoVery very hard to believe.
- Taek 3y ago*the api wasn't nerfed. ChatGPT4 was most certainly changed
- kossTKR 3y agoThe comments seem to conclude that it was actually nerfed just not the API.
- johnwheeler 3y agoIt most certainly was, despite what anyone from openai says. It’s not working the same.
- az226 3y agoIt most definitely was. The chat output of GPT4 is much faster now and much worse quality. If you go in the playground and use the March 14 api (as opposed to the default GPT4 API) it is high quality and slow. Yes, there are two GPT4 API endpoints .
- FeepingCreature 3y agoArchive: https://web.archive.org/web/20230000000000/https://www.tii.ae/news/uaes-falcon-40b-now-royalty-free https://web.archive.org/web/20230000000000/https://www.tii.a...
- stan_kirdey 3y agoChatting with Falcon 40B feels like chatting with GPT4, very capable model.
- azinman2 3y agoCan you give examples?
- low_tech_punk 3y ago"As an AI model, I cannot give an example of me chatting with myself"
- barefeg 3y agoI’m curious to know the real reason for the change. Is it related to the data they used for training?
- gandalfff 3y agoAmy implementation for this akin to llama.cpp?
- Kelamir 3y agoSame question, so far I've found this thread https://github.com/ggerganov/llama.cpp/issues/1602 https://github.com/ggerganov/llama.cpp/issues/1602 where people work on it.
- low_tech_punk 3y agoHypothetically speaking, if Falcon 40B could out-perform GPT-3.5, would that force OpenAI to open-source GPT-3.5?
- nc 3y agoIt would as open source improvements would start to exceed performance of 3.5 for specific use-cases. At the very least they would have to make it fine-tunable.
- sebzim4500 3y agoSam Altman said recently that they are already working on making GPT-3.5/GPT-4 finetunable, they are just limited by the availability of compute (partially since none of their SFT infrastructure uses LoRa). I had previously assumed it was safety concerns, since I don't see what stops someone from finetuning away all guard rails.
- janekm 3y agoDid Stable Diffusion "force" OpenAI or Google to open-source their imaging models? No. Don't see what the mechanism would be for that
- wokwokwok 3y agoNo. There no reason, particularly, to believe either that it does, or it would. For openai to scramble and try to “catch up” with a competitor and make such a massive change in strategy would require someone to be offering an equivalent service (hosted inference) that was either orders of magnitude cheaper than their offering and just as good, or significantly better than it. Or legal compulsion. This is none of those things. They won’t care.
- Havoc 3y agoOh wow. That’s great news. Falcon seemed good till I read the license fine print about pre approvals and what not. This seems to fix that
- version_five 3y agoAm I the only one who finds this very sketchy? They had the whole license things, there's been some loud complaining by the HF CTO on social media that this model is not getting enough attention, and there are also press releases about how Falcon tops the "leaderboard": https://www.morningstar.com/news/business-wire/20230529005045/uaes-falcon-40b-dominates-leaderboard-ranks-1-globally-in-latest-hugging-face-independent-verification-of-open-source-ai-models https://www.morningstar.com/news/business-wire/2023052900504... I've never seen this kind of "strategy" with an ML model before. Maybe I'm seeing something that isn't there... It could be a question of not being used to see blatantly commercial advertising in places we're used to being about software. Feels like we're moving more towards the bro-ification of AI.
- throwaway290 3y agoProbably PR to attract subsidies/investment. UAE might be throwing more money on tech/innovation branded projects than China.
- jerpint 3y agomaybe attention IS all you need
- version_five 3y agoI wish I'd thought of that. Seriously though, I think it's mostly the opposite. How much advertising did Georgi Gerganov do for ggml / llama.cpp and it's super popular. Maybe other people are just being more subtle, but I feel generally merit stands out on it's own and advertising is a poor substitute.
- lbotos 3y agoFor the segment of HN that isn't up to speed on ML this is a double entendre from a famous ML paper: https://towardsdatascience.com/attention-is-all-you-need-discovering-the-transformer-paper-73e5ff5e0634 https://towardsdatascience.com/attention-is-all-you-need-dis...
- Havoc 3y ago
- sidcool 3y agoI am an amateur when it comes to these models. What can I do with this model and how?
- muttled 3y agoAt the most basic, you give it text and it can guess at what comes next. This means you could type "10 types of ferns" and it will build a list of 10 ferns. Or you could type it out how a transcript of a conversation would look like and it will basically fill in the other "side" of the conversation to make a chatbot (all the complicated chatbots are basically abstracting this). Think of it like a text box with a super-smart (arguably) person also having access where you can type one thing and then it'll type whatever it thinks would be the next thing to type.
- barrenko 3y agoIs this multilingual?
- convexstrictly 3y agoFor fast inference, the HuggingFace cofounder, Thom Wolf recommends their text-generation-inference library https://github.com/huggingface/text-generation-inference https://github.com/huggingface/text-generation-inference https://twitter.com/Thom_Wolf/status/1664613366751526914 https://twitter.com/Thom_Wolf/status/1664613366751526914
- randomname9191 3y agohttps://medium.com/towards-artificial-intelligence/falcon-40b-a-fully-opensourced-foundation-llm-945dd9824157 https://medium.com/towards-artificial-intelligence/falcon-40...
- lostmsu 3y agoI wish these releases had short but meaningful descriptions for their models. E.g. 10 layers x 2048 tokens x 1024 embedding model using full attention.
- lostmsu 3y agoIt's interesting, that intruction-tuned model has lower performance on ARC and HellaSwag.