13 ms·
LLaMA: A foundational, 65B-parameter large language model
- minimaxir 4y agoThis blog post is terrible at listing the improvements offered by the model, the abstract is better: > We introduce LLaMA, a collection of foundation language models ranging from 7B to 65B parameters. We train our models on trillions of tokens, and show that it is possible to train state-of-the-art models using publicly available datasets exclusively, without resorting to proprietary and inaccessible datasets. In particular, LLaMA-13B outperforms GPT-3 (175B) on most benchmarks, and LLaMA-65B is competitive with the best models, Chinchilla70B and PaLM-540B. We release all our models to the research community.
- mjburgess 4y ago> We release all our models to the research community. This is yet more evidence for the "AI isn't a competitive advantage" thesis. State-of-the-Art is a public resource, so competing with AI offers no "moat".
- jstx1 4y agoThe research is open, you still need 1) to have a good idea for a product that people want 2) lots of talent and resources to actually build and run everything
- minimaxir 4y agoBusiness orgs have been finetuning open-source models like these on their own internal data to create a moat since BERT in 2018.
- Der_Einzige 4y agoGlad to see you still on HN! You've done amazing work in this domain! I'd argue that this goes further back to the word2vec/glove days too. I was working for a company in 2018 who leveraged my skills for fine-tuning word2vec/fasttext even before BERT/attention is all you need paper.
- fwlr 4y agoIn terms of medieval warfare, what Facebook is doing here looks like filling the moat with rocks and dirt. OpenAI is worth billions, Microsoft is spending billions to retrofit most of their big offerings with AI, Google is doubtless also spending billions to integrate AI in their products. Their moat in all cases is “a blob of X billion weights trained on Y trillion tokens”. Facebook here is spending mere _millions_ to make and release competitive models, effectively filling in the moat by giving it to everyone.
- recuter 4y agoI don't see such grand stratagems as being a likely explanation. It seems more likely that a bunch of dorks are running around unsupervised trying their best to make lemonade with whatever budgets they are given as nobody can realistically manage AI researches. Or at least, it seems that much like in political governance, events are suddenly outpacing the ability of corporate to react. In the case of OpenAI it is a "nudge humanity into a more wholesome direction" because a lot of them went on an acid fueled toxic affective altruism bender. And in this case it is "release all the things while Zuck is still obsessed with VR --for science!". I like this other group better. But it is disturbing that it can stop at any moment. Probably why they are doing it, while they still can.
- fwlr 4y agoThe researchers and engineers and other assorted dorks who built it weren’t thinking about moats, for sure, I agree with you there. But I guarantee you that the metaphor of medieval warfare was on the minds of the executives and legal team deciding whether to let the team release their work in this way.
- titzer 4y agoThey almost certainly spent at least a few million dollars on this research project. Hard to say when the decision was made to open source this (from the outset or after it started showing results), but the decision was conscious and calculated. Nothing this high-profile is going to escape the strategic decision making processes of upper management.
- 4y ago
- layer8 4y agoThey are only releasing it on a case-by-case basis and with a non-commercial license.
- cardine 4y agoAnd yet there are still no publicly available models that could actually compete with ChatGPT. I'm not even talking about RLHF (although data like that is also a huge moat) - just simple things like larger context sizes. There are still plenty of AI advantages to be had if you go just a little bit outside of what is currently possible with off the shelf models.
- lr1970 4y ago> We release all our models to the research community. And from the FB blogpost [0] "Request Form Thank you for your interest in Meta AI’s LLaMA (Large Language Model Meta AI) models. To request access to the models, please fill out this form, and we'll review and let you know if your use case is approved. The information you provide below will be used solely to assess eligibility to access these models." So much for "releasing" the model for research community. [0] https://ai.facebook.com/blog/large-language-model-llama-meta-ai/ https://ai.facebook.com/blog/large-language-model-llama-meta...
- rvz 4y ago> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. Access to the model will be granted on a case-by-case basis to academic researchers; those affiliated with organizations in government, civil society, and academia; and industry research laboratories around the world. People interested in applying for access can find the link to the application in our research paper. The closest you are going to get to the source is here: https://github.com/facebookresearch/llama https://github.com/facebookresearch/llama It is still unclear if you are even going to get access to the entire model as open source. Even if you did, you can't use it for your commercial product anyway.
- ninjin 4y agoFacebook continues to appropriate the word “open” [1], which is sad really. There are plenty of good words they and others could use instead. [1]: https://news.ycombinator.com/item?id=32079558 https://news.ycombinator.com/item?id=32079558
- jerpint 4y agoI just signed some form with license to maybe possibly get access to the model… not very open indeed
- swatcoder 4y ago> Even if you did, you can't use it for your commercial product anyway. And of course, the irony is that its not commercial products that endanger “integrity and misuse”. The looming misuse of LLM’s is cheap content flooding by spammers, black hats, and propaganda bots — and those people don’t care about licenses, and will inevitably defeat any watermarks meant to prevent or track leaks.
- milesward 4y agoRight, but does it whip the LLaMA's ass?
- layer8 4y agoYou mean, will their competitors whip the LLaMA's ass. ;)
- ImprobableTruth 4y agoUnfortunately non-commercial and only available to academics upon request.
- flangola7 4y agoThat's probably the best and safest way to do it
- yieldcrv 4y agodarn, guess we'll have to wait until next week when some other surprise organization releases something extremely surprising and useful in this space
- elorant 4y agoUntil someone leaks it.
- brucethemoose2 4y agoI give it a week.
- koheripbal 4y agoPrescient
- swyx 4y ago> To maintain integrity and prevent misuse, we are releasing our model under a noncommercial license focused on research use cases. if say i wanted to replicate this paper for commercial use, what would it take and how do i get started? would FB have a basis for objection?
- minimaxir 4y agoA metric-ton of GPU compute, for starters.
- chandureddyvari 4y ago82342 GPU(A-100 80GB) hours for training their 7B(smallest) model according to their paper.
- riku_iki 4y ago115 GPUs can be used to train it within month.
- causalmodels 4y agoI'm going to assume you know how to stand up and manage a distributed training cluster as a simplifying assumption. Note this is an aggressive assumption. You would need to replicate the preprocessing steps. Replicating these steps is going to be tricky as they are not described in detail.Then you would need to implement the model using xformers [1]. Using xformers is going to save you a lot of compute spend. You will need to manually implement the backwards pass to reduce recomputation of expensive activations. The model was trained using 2048 A100 GPUs with 80GBs of VRAM. A single 8 A100 GPU machine from Lambda Cloud costs $12.00/hr [2]. The team from meta used 256 such machines giving you a per day cost of $73,728. It takes 21 days to train this model. The upfront lower bound cost estimate of doing this is [(12.00 * 24) * 21 * 256) = ] $1,548,288 dollars assuming everything goes smoothly and your model doesn't bite it during training. You may be able to negotiate bulk pricing for these types of workloads. That dollar value is just for the compute resources alone. Given the compute costs required you will probably also want a team composed of ML Ops engineers to monitor the training cluster and research scientists to help you with the preprocessing and model pipelines. [1] https://github.com/facebookresearch/xformers https://github.com/facebookresearch/xformers [2] https://lambdalabs.com/service/gpu-cloud https://lambdalabs.com/service/gpu-cloud edit: these costs are for the 65B parameter model.
- nafizh 4y ago>'We release all our models to the research community' Where release means you fill out a form and wait indefinitely. Also no use for commercial purposes - which means 95% of users are out - certainly doesn't democratize LLMs.
- mliker 4y ago> doesn't democratize LLMs open source always leads to more democratization in the long run
- version_five 4y agoI think these "open" models are the lamest release strategy. I get the idea of being closed à la OpenAI to protect a business model, and I understand open sharing in the community "We'll let you see it if we approve you" is just being a self-important pompous middle man.
- lovelearning 4y agoI hope they make at least the smallest one available for everyone. I find it ironic that they want to prevent misuse, but think allowing access to those affiliated with government organizations will achieve that.
- deleted 4y ago[deleted]
- saurabh20n 4y agoQuick notes from first glance at paper https://research.facebook.com/publications/llama-open-and-efficient-foundation-language-models/ https://research.facebook.com/publications/llama-open-and-ef...: * All variants were trained on 1T - 1.4T tokens; which is a good compared to their sizes based on the Chinchilla-metric. Code is 4.5% of the training data (similar to others). [Table 2] * They note the GPU hours as 82,432 (7B model) to 1,022,362 (65B model). [Table 15] GPU hour rates will vary, but let's give a range of $1 to $4. The 7B model would have cost ~$82-329k and the 65B something in the range of ~$1-4M. They also note their total time spent for all models: "we used 2048 A100-80GB for a period of approximately 5 months" [sec 6, pg 10] * 65B model's performance is broadly comparable to PALM-540B. Not a small feat, but also could indicate the benefits of good model-vs-token size ratios [Tables 3,4,5,6]. Their conjecture for underperforming on MMLU (multitask language understanding) compared to PALM-540B and Chinchilla-70B is smaller fraction of books and academic training data. * Math and code tasks: Math tasks they are substantially worse than Minerva (comparing their 65B to Minerva 62B; they hands down fail against Minerva 540B) [Table 7]. Code tasks they are broadly competitive with PALM-540B (HumanEval and MBPP evals) [Table 8] * Surprising that instruction fine tuning takes such a small part of the paper (sec 4, pg. 7)
- zozbot234 4y agohttps://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md https://github.com/facebookresearch/llama/blob/main/MODEL_CA... (linked in OP) has basic information about this model.
- machinekob 4y agoI hate when people don't include approximation for traning before final hyperparameters are found as its most costly part of whole process most of the time. Just yes we train it for so long etc. but they never speak about tens or even hundres of runs before they finalize the model parameters and architecture -.-
- scotty79 4y agoAren't those done on smaller version of the same model?
- muglug 4y agoI was surprised to see this citation in the linked paper > Alan M Turing. 2009. Computing machinery and intelligence Did he invent a time machine too?
- Mizza 4y agoLooks like it was republished with additional commentary: https://link.springer.com/chapter/10.1007/978-1-4020-6710-5_3 https://link.springer.com/chapter/10.1007/978-1-4020-6710-5_...
- igravious 4y agoAbsolutely a pet peeve of mine. What's wrong with (orig. pub. 1950, rev. ed. 2009) or something like that. With Zotero, the ref manager I use, you need to know the special magic incantation to store the original date of publication (the publishing date refers to the edition you're citing) but it just looks stupid (I think) to see Nietzsche F. 1999, Untimely Meditations (or whatever) and also I'd like to sort texts by original date of publication from oldest to newest because that's interesting. You have to put a special magic code in the extra field and then your tool-chain has to preserve it and transmogrify it correctly in the doc you're making use of the citation.
- harveywi 4y agoIt is undecidable to determine whether a manuscript has terminated, or whether it will continue being revised forever.
- simonw 4y agoThe most interesting snippet in the paper I think is this: > For instance, LLaMA-13B outperforms GPT-3 on most bench- marks, despite being 10× smaller. We believe that this model will help democratize the access and study of LLMs, since it can be run on a single GPU.
- stevenhuang 4y agoHow about alignment/ability to answer prompt queries and chain of thought reasoning capabilities? Without the fine tuning RLHF phase to make it like instructgpt I'm assuming it won't be as good as ChatGPT, is that right? How hard would it be to fine tune the 65B model on commodity hardware? Found answer here: > Out-of-scope use cases LLaMA is a base, or foundational, model. As such, it should not be used on downstream applications without further risk evaluation and mitigation. In particular, our model has not been trained with human feedback, and can thus generate toxic or offensive content, incorrect information or generally unhelpful answers. https://github.com/facebookresearch/llama/blob/main/MODEL_CARD.md https://github.com/facebookresearch/llama/blob/main/MODEL_CA...
- zozbot234 4y agoYes but fine tuning for RL is not expected to be hard. You're essentially limited by how much human feedback is available, so it's very different from training the foundational model on random bulk data.
- simonw 4y agoAt this point I fully expect that someone will release a usable RLHF-fine-tuned language model that can run on consumer hardware, based on the methodology used for LLaMA (and other similar papers e.g. https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen ), at some point in the next 6-24 months.
- monkeydust 4y agoWhat are the functional steps that need to happen to get to that point ?
- amoss 4y agoGiven the limited release of access under a license - is there any decision on whether a model like this would be protected by copyright if it leaked? The output of an algorithm is not a human authored creative work, which given the recent decision in the book case would seem to weigh against it.
- qup 4y agoWhat kind of machine does it take to run the model?
- rafaelero 4y agoWow. That's nuts. An open-source model as competitive as 540b Palm? Sign me in. Wish they made it easy to access it, though. Not sure if I would be able to put my hands on it solely as an interested individual.
- cypress66 4y agoThere's nothing even close to open source here.
- thedudeabides5 4y agoLLMs are already getting commoditized, and this is good
- bioemerl 4y agoIt's not really open source which is something of a massive bummer, but still cool to see Facebook of all companies being the relative good guy.
- adeon 4y agoHas anyone had success being approved to download weights even if you are just a hobbyist with a big GPU? (I'm here about to begrudgingly fill out the Google Form) Asking in general, not just for this particular model. I've been following these language models and so far went through approval once for a English/Chinese GLM-130B model and it took only ~30 minutes to get approved even though I am a complete nobody.
- nathanh4903 4y agoi got in after 2 days and im just an undergrad with some experience tangentially related to machine learning. Too bad i only found out the hardware requirement after downloading 13GB of weights
- blablablub 4y agoSo what hardware do we need to run this model?
- CuriouslyC 4y ago7 billion can run on 16+ gb GPUs as fp16, 14 billion can be run on 16+ gb if quantized to int8. 14G @ fp16 and 30G at int8 will require one of the 48 gb cards (less, but hardware mostly goes 24 -> 48).
- blablablub 4y agothanks
- brucethemoose2 4y agoRequirements could be reduced with something like DeepSpeed or ColossalAI (or even just simple hacks to move bits to RAM more aggressively)
- miketery 4y agoCan this be downloaded like stable diffusion? How big are these models? How long does it take to run a "query" against them on a decent gaming PC?
- flatiron 4y agoyou have to contact facebook for a copy, so not as open as stable diffusion
- bioemerl 4y agoThis can't, but there's some software, KoboldAi, which lets you download and run other LLMs
- brucethemoose2 4y agoNo one seems to be running it locally at the moment.
- apgwoz 4y agoLLaMA is short for "Large Language Model Meta AI" -- I don't get how, but I guess the "fakeronym" was a brandable, recognizable word, and close enough that they went with it?
- freezed88 4y ago(creator of gpt index / llamaindex here https://github.com/jerryjliu/gpt_index https://github.com/jerryjliu/gpt_index) Funny that we had just rebranded our tool from GPT Index to LlamaIndex about a week ago to avoid potential trademark issues with OpenAI, and turns out Meta has similar ideas around LLM+llama puns :). Must mean the name is good though! Also very excited to try plugging in the LLaMa model into LlamaIndex, will report the results.
- mark_l_watson 4y agoI look forward to your comparison results. BTW, I have been heavily experimenting with both your LlamaIndex and also LangChain which you use in LlamaIndex. I am also writing a new book centered around both projects. Great stuff!!
- haolez 4y agogpt-index is awesome! What should I study to understand the inner workings a bit better?
- modeless 4y agoHow possible is it to run these models on a gaming GPU?
- coolspot 4y agoYes, if your gaming rig is NVidia DGX-1 workstation. https://youtu.be/5TRr2oWeSw0 https://youtu.be/5TRr2oWeSw0
- infinityio 4y agoa very rough approximation is 2GB of vram for every billion fp16 model parameters, so the lower end models may be just about achievable on high-end cards
- sp332 4y agoIf you're patient, https://github.com/FMInference/FlexGen https://github.com/FMInference/FlexGen lets you trade off GPU RAM for system RAM or even disk space.
- weystrom 4y agoI'd love to hear what Facebook defines as "open" and "democratic". I understand the unwillingness to attach your brand to whatever porn that inevitably comes out of making it available for everybody, but maybe don't use such big words then.
- monkeydust 4y agoWhat no implementation example?
- iandanforth 4y ago"we are publicly releasing LLaMA" "Access to the model will be granted on a case-by-case basis to academic researchers" They keep saying the word 'release' but I don't think they know what that word means. There are perfectly good words in the English language to describe this situation without abusing "release." They "will begin to grant access to a select few". Nothing about that releases the model, or their control, which they are not doing and shouldn't imply.
- golem14 4y agoIDK: "Mr. Burns: Smithers, release the hounds. "
- modeless 4y agoSomeone needs to start a pirate bay for torrents of ML models. Call it ClosedAI, since in AI land these words mean the opposite of what they say.
- Zuiii 4y ago> since in AI land these words mean the opposite of what they say. Huh. I think you just might be right: OpenAI that isn't open, AI safety/ethics "researchers" that have nothing to do with safety or ethics, almost every answer chatGPT gives about a topic considered "sensitive" by said "researchers", almost every time ChatGPT falsely asserts it "cannot" do something or simply lies (1). I often wonder why this field became so twisted and perverted. I support the idea behind ClosedAI. 1: Answers given by "DAN" provide a glimpse of what chatGPT output could be like if it was allowed to provide answers that are factual, genuine, and truthful according to its dataset.
- Koshkin 4y agoThis has reminded me about “open addressing” also known as “closed hashing.”
- codegladiator 4y agoOff topic: Dan doesn't work anymore right? Or is there a new prompt
- dentalperson 4y agoHow does token->embedding lookup work with 1.4T BPE tokens? Since there are more tokens than the 65B parameters it must be doing some sort of interesting thing based on the merge operations. Is it different from what other GPT models with ~100k tokens are doing? At inference, how many of those tokens are used? (they mention most tokens are used only once during training, so the must be very long sequences.)
- bitRAKE 4y agoThe 1.4T tokens are what the model was trained on, and not the token range of the embedding.
- dentalperson 4y agoAh, that makes more sense, thank you. Since this was mentioned in the tokenizer section and the number of unique tokens wasn't mentioned I misunderstood.
- rsrsrs86 4y agoFunny how the release has a note that the models clearly have ethical problemas that must be addressed, and still the company chooses to publish it.
- 323 4y agoThe old "not my problem how people use the gun, I'm just a manufacturer" defense.
- UncleEntity 4y agoEthical problems that they don’t purge it of Wrong Think? Why people care that a dodgy AI can be made to say politically incorrect things is beyond me. Don’t use its outputs for anything important and everything will be alright.
- kuhewa 4y agoPerhaps because that is going to be a key factor in how many use cases for which it could be commercialised, worth considering when training a model costs a million dollars in compute time.
- jokethrowaway 4y agoThat's the only reason I'd use it over ChatGPT
- carlsborg 4y ago"The code is licensed under the GPLv3, which permits commercial use." Yann on Twitter
- return_to_monke 4y agoThe code. They didn't release a model tho, so unless you have a lot of money or get granted access to it, not for you.
- yawaramin 4y agoLeCun on Twitter: https://twitter.com/ylecun/status/1629189925089296386 https://twitter.com/ylecun/status/1629189925089296386 > LLaMA is a new open-source, high-performance large language model from Meta AI - FAIR. > > Meta is committed to open research and releases all the models the research community under a GPL v3 license.
- _a_a_a_ 4y agoNewbie asks: what is a 'parameter' here
- sp332 4y agoOne neuron weight. A single element of a matrix or tensor.
- throwaway1851 4y agoTake y=3x+7. 3 and 7 are parameters, which would be learned by using gradient descent to find the values that make the x’s in the training data produce the corresponding y’s. LLMs are essentially functions like this, only they have billions of parameters and are nonlinear.
- zxienin 4y agoWill it get 1M users using it by next week? “Our 65B model is better performant than ChatGPT‘s 175B” is missing the point openai made in Nov'22. That aside, this looks nice incremental research progress.
- naveen99 4y agoWhy not a 1 Billion parameter model that can run on a single GPU ? worse is better will come to LLM’s.
- tevon 4y agoHow big is one of these models once it is trained and tuned? Could it be run on a single reasonably-large EC2? Or does it need any special architecture?
- WizardClickBoy 4y agoI somehow have a strong intuition that Winamp is going to beat this thing.