11 ms·

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model

343 points by sillysaurusx 4y ago

lumost 4y ago
Curious what the ultimate enforcability of a restrictive license is. If I fine-tune a model, is it still covered? what if I randomize and retrain a layer - or remove a layer? It seems like it will be impossible to verify that someone did not just train the model from scratch.
swyx 4y ago
thanks for doing this, honestly your writeup seems more valuable than the model weights lol > But for what it's worth, my personal opinion is that LLaMA probably isn't OpenAI-grade -- there's a big difference between training a model in an academic setting vs when your entire company depends on it for wide-scale commercial success. I wasn't impressed that 30B didn't seem to know who Captain Picard was. im new to benchmarking shenanigans but how is it that facebook was able to proclaim that it matched GPT3 performance on presumably standard LLM benchmarks? is there a good survey paper or blogpost on how to think about known deficiencies in benchmarks?
- sillysaurusx 4y ago
  Because loss != quality. This was one of the most counterintuitive discoveries in ML for me. People treat the two as interchangeable, and to a certain extent — a controlled extent — they are. But if your dataset doesn’t include a word about Captain Picard, no amount of training will get it to know about the USS enterprise. Yet your loss metrics will still reach that magical 2.1 value with time. (2.1 is pretty much “excellent” quality; below that means you’re probably overfitting and need a bigger dataset.) Thanks for the comment friendo. I wasn’t sure if this would get any attention at all, but that made it worth it. Be sure to DM me on Twitter if you’d like to chat about anything ML related: basic questions are one of my favorite things to assist with too, so feel free.
  - nl 4y ago
    This isn't really correct. Loss is a training-time measurement based on performance on the training objective. The training objective is rarely the same as an end user task that is being benchmark. For example, classically language models are training on next token prediction. The closest benchmark for that is perplexity[1], often reported on the WikiText-103 dataset. Until around 2019 this was often reported, but since then most large language model papers have moved onto reporting more useful benchmarks. Some examples of this are question answering performance or maybe embedding performance. Unfortunately there aren't great benchmarks (yet?) for generative tasks. Quality is quite hard to measure here in a systematic way (see, eg the issues with BLEU benchmarks in summarization benchmarks). [1] https://en.wikipedia.org/wiki/Perplexity https://en.wikipedia.org/wiki/Perplexity
- rnosov 4y ago
  You can read the original LLaMA paper which is pretty accessible[1]. For example, they claim to outperform GPT-3 on HellaSwag benchmark ( finishing sentences ). You can find examples of unfinished sentences in the HellaSwag paper [2] on page 13. Unfortunately for LLaMA, most people would be probably just asking questions about Captain Picard and so on, and on this benchmark LLaMA significantly underperforms compared to OpenAI models (thats's from their paper). [1] https://research.facebook.com/file/1574548786327032/LLaMA--Open-and-Efficient-Foundation-Language-Models.pdf https://research.facebook.com/file/1574548786327032/LLaMA--O... [2] https://arxiv.org/pdf/1905.07830.pdf https://arxiv.org/pdf/1905.07830.pdf
  - yunyu 4y ago
    Hellaswag is also a deeply flawed benchmark, I wouldn't read too much into it: https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this-popular-llm-benchmark-contains-errors https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this...
- nl 4y ago
  Because there are many benchmarks that measure different things. You need to look at the benchmark that reflects your specific interest. So in this case ("I wasn't impressed that 30B didn't seem to know who Captain Picard was") the closest relevant benchmark they performed is MMLU (Massive Multitask Language Understanding"[1]. In the LLAMA paper they publish a figure of 63.4% for the 5-shot average setting without fine tuning on the 65B model, and 68.9% after fine tuning. This is significantly better that the original GPT-3 (43.9% under the same conditions) but as they note: > "[it is] still far from the state-of-the-art, that is 77.4 for GPT code-davinci-002 on MMLU (numbers taken from Iyer et al. (2022))" InstructGPT[2] (which OpenAI points at as most relevant ChatGPT publication) doesn't report MMLU performance. [1] https://github.com/hendrycks/test https://github.com/hendrycks/test [2] https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155
  - JonathanFly 4y ago
    The capability of a language model I care about most is probably its ability to represent or simulate Captain Picard. In the sense of being good at creative tasks but also Captain Picard, specifically. Is OpenAI deliberately doing something different on purpose that makes their models better for this, or is just that OpenAI has a lot more copyrighted data in their dataset, as I noticed just now when skimming the Facebook paper for MMLU section and seems be what the Facebook folks think? "A potential explanation is that we have used a limited amount of books and academic papers in our pre-training data, i.e., ArXiv, Gutenberg and Books3, that sums up to only 177GB, while these models were trained on up to 2TB of books. This large quantity of books used by Gopher, Chinchilla and PaLM may also explain why Gopher outperforms GPT-3 on this benchmark, while it is comparable on other benchmarks."
    - nl 4y ago
      It's unclear exactly why it doesn't work as well for you. I have two comments that may be useful: 1) It's very unclear how good the generative capabilities of LLAMA are generally. It benchmarks well for code generation, but for English there aren't really any good benchmarks around. There's good chance the larger model performs much better since generative capabilities seem to be a partially emergent capability. 2) If you just want to "make it work" I'd suggest downloading all the Star Trek scripts you can that include Captain Picard and fine tuning LLAMA using them. It's unclear how well this will work, but that is probably about as good as you can get. If you care about this probably deeply, it's probably worth trying the same with some of the other open GPT-3 models (GPTJ, GPT-NEOX etc)
sillysaurusx 4y ago
Update: FB disabled the download link, so I mirrored everything to R2 and updated the script to use it. It should be working now (though the speed is "only" around 50MB/s).
- toomuchtodo 4y ago
  Have you dropped the artifacts in the Internet Archive yet by chance?
  - sillysaurusx 4y ago
    I'm surprised Internet Archive is appropriate for a 220GB model weight dump. Please feel free; it seems like a good idea. I'm not sure I have enough weekend left to figure out yet another upload service today.
- e12e 4y ago
  I've read the readme - but I'm not sure why this is any faster than just adding seeds to the torrent? More people downloading via torrent than http?
EMIRELADERO 4y ago
I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.
- kuroguro 4y ago
  That definition would apply to almost anything software produces ^^; We can already have different licenses for compiled binaries vs the source. Also the output of ML seems to belong to whoever pressed the generate button atm.
  - cthalupa 4y ago
    >Also the output of ML seems to belong to whoever pressed the generate button atm. So far the rulings in the US, at least, do not support this. https://arstechnica.com/information-technology/2023/02/us-copyright-office-withdraws-copyright-for-ai-generated-comic-artwork/ https://arstechnica.com/information-technology/2023/02/us-co... In this case, it was images generated via Midjourney and not the output of an LLM, but my layman's understanding of the result here would be equally applicable to LLM output. Effectively, the copyright office does not consider putting in a prompt enough for there to be "human authorship" of the work. In this specific case, that resulted on the images in the comic being considered uncopyrightable. The broader comic, in the organization of the images, the plot and dialogue, etc., still enjoys copyright protection. But in the US, I could just directly take the images in the comic that Midjourney produced and use them for another purpose without violating copyright.
  - EMIRELADERO 4y ago
    > That definition would apply to almost anything software produces Not really. The reason software can be copyrighted at all is because the actual code (and resulting object code) is creative. Courts have named this threshold the "Structure, sequence and organization" of the work. ML models don't follow any creative SSO the way actual code does. > Also the output of ML seems to belong to whoever pressed the generate button atm. The output, it seems to me, is uncopyrightable. Copyright only cares about who provides the creativity for the work at issue, not who put in the effort to make it happen. You may own the copyright to your prompt, but the result is generated entirely by the AI and thus lacks human autorship.
bitL 4y ago
How does LLaMA handle fast fine-tuning? Are they using transformer adapters for it?
- loufe 4y ago
  It's already been adapted for hugging face transformers[1]. Apparently that should unlock its full potential. Oobabooga integrated the change into text-generation-webui[2] meaning we can already access a large chunk of its potential (from what I understand). [1] https://github.com/huggingface/transformers/pull/21955 https://github.com/huggingface/transformers/pull/21955 [2] https://github.com/oobabooga/text-generation-webui/commit/90206204aa41a73fe754e580ae1913dc2b01a384 https://github.com/oobabooga/text-generation-webui/commit/90...
  - bitL 4y ago
    That's absolutely fantastic! Thanks for the links!
arjvik 4y ago
How big is this model? (i.e. disk space to store it)
- kuroguro 4y ago
  65B is ~120GB. All of them combined with the smaller versions is ~220GB.
  - nextaccountic 4y ago
    After converting to int8, does it become smaller? Also, can this be further compressed? Like, is there some redundancy a special-purpose compressor could exploit?
    - rfoo 4y ago
      Converting to int8 halves the size.
linearalgebra45 4y ago
It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?
- v64 4y ago
  The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWeave. [1] https://news.ycombinator.com/item?id=35028738 https://news.ycombinator.com/item?id=35028738
  - linearalgebra45 4y ago
    Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/ https://www.oracle.com/cloud/compute/gpu/
    - v64 4y ago
      > Are you sure about that? I'm not. The only way to know it is to try :) thank you for the link!
      - linearalgebra45 4y ago
        You only get a single month-long window to spend the credit! And I'm sure not going to spend any of my own money on prompting experiments. I might be suffering from FOMO to some degree, I've just got to tell myself that this won't have been the only time model weights get leaked!
v64 4y ago
If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software." I'm very thankful we have researchers like this further democratizing access to this data and prying it out of the hands of the gatekeepers who wish to monetize it. [1] https://github.com/tloen/llama-int8 https://github.com/tloen/llama-int8 [2] https://arxiv.org/abs/2208.07339 https://arxiv.org/abs/2208.07339
- rnosov 4y ago
  Hmmm, the Github repo suggests that you might be able to run the 65B model on a single A100 80gb card. At the moment, the spot price on Google cloud for this card is $1.25/hour which makes it not so crazy expensive...
  - nabla9 4y ago
    $1.25/hour is roughly a year of GPU time until it exceeds the price of A100 80GB card.
    - metadat 4y ago
      I think OP meant that $1.25/hr makes this accessible for people try it out themselves cost effectively, without having to spend thousands or tens of thousands up front to obtain a capable hardware rig. Obviously $1.25/hr 24/7 does add up quickly, after one month the bill would come to $900.
      - deleted 4y ago
        [deleted]
- downvotetruth 4y ago
  Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https://github.com/markasoftware/llama-cpu https://github.com/markasoftware/llama-cpu So, an int8 patch could allow the 65B to run on a standard 128 GB setup assuming the 65B model's cache bursts fit, which if I were to speculate is why the released models stop @ 65B & meta likely already has larger unreleased internal ones.
notpushkin 4y ago
For even better speeds, perhaps use the link from this script (if it ever goes back up) as a webseed for torrent?
ur-whale 4y ago
Is this the full model or just the weights? [EDIT]: are there checksums available? [EDIT2]: MD5 signatures seem to be included for all models in checklist.chk files next to them And there's also what the author mentions: the magnet file he provides in his README does seed immediately on the download when loaded in a bt app which is usually a good sign that the files are correct.
- rany_ 4y ago
  MD5 signatures don't mean much now that hash collisions could be created instantly on consumer hardware. MD5 is only good for checking for unintentional data corruption.
  - deleted 4y ago
    [deleted]
  - ur-whale 4y ago
    Bittorrent uses SHA-1 https://en.wikipedia.org/wiki/BitTorrent https://en.wikipedia.org/wiki/BitTorrent Collisions are possible but not exactly trivial
  - charcircuit 4y ago
    >hash collisions could be created instantly on consumer hardware Collisions can be created, but MD5 is still preimage resistant. As long as someone with the actual model made the hash and Meta didn't generate colliding models themselves. You can trust it.
    - deleted 4y ago
      [deleted]
m3kw9 4y ago
If an AI model like this isn’t able to evolve and improve is it really useful? Example is code generation or questions that more recent training data can teach the AI
version_five 4y ago
Thanks for doing what Facebook should have been mature / humble enough to have done on their own. The best outcome of this would be for FB to stop the silliness and just release the weights openly themselves.
- ALittleLight 4y ago
  This may be a good compromise for plausible deniability. Facebook can be "responsible" and release only to "researchers" and the public can get the model from torrents shortly after. So long as the models keep going public I'm satisfied.
vanillax 4y ago
How or what can someone do with this who isn't a ML expert? Is there some docker app that leverages this? To the average dev, is this useful to me? I know there's lots of "plug and play" style docker apps to get started with Stable Diffusion. I'm curious if I can do something fun with this.
- swalsh 4y ago
  Not an expert, but i've downloaded the model, and used it. What you get is pretty raw, and not super useful by itself. There are some projects to try and do some RLHF on it, and with that we might start to get something you can do some useful stuff with.
- deleted 4y ago
  [deleted]
- kkielhofner 4y ago
  You can shortcut a lot of the steps in these various guides by using the Pytorch container from Nvidia[0]. It shouldn't be too hard for someone (me?) to create a Dockerfile and Docker hub container FROM this image to get it up and running easily. [0] - https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc...
cloudking 4y ago
What's up with the domain in the script? PRESIGNED_URL="https://agi.gpt4.org https://agi.gpt4.org...
- arjvik 4y ago
  It's pointed at Cloudflare storage right now
yumraj 4y ago
What’s the minimum single GPU that’ll work for the smallest model?
- downvotetruth 4y ago
  3060 12GB
- zargon 4y ago
  This reddit post says that the 7B model consumes about 9.7GB of VRAM (using int8). I'm sure very soon people will add support for using system RAM as swap space, which will allow you to use it on an 8GB card, though with a fairly hefty performance penalty. https://www.reddit.com/r/MachineLearning/comments/11h3p2x/d_facebooks_llama_leaks_via_torrent_file_in_pr/jb0wots/ https://www.reddit.com/r/MachineLearning/comments/11h3p2x/d_...
ahahahahah 4y ago
Are we celebrating theft from tech companies now?
- tjranagk 4y ago
  Their model was trained on vast amounts of copied data. Is making a copy of their model really much different?
- anaganisk 4y ago
  I mean highseas, adblockers, bypassing paywalls, each one of them is theft. But on the flipside, companies are constantly trying to keep the ownership of data we paid for full price, scooping up personal data, selling low quality work behind paywall.
- antibasilisk 4y ago
  Copying isn't theft. If you bought the ssd it belongs to you in its entirety regardless what state you decide to configure it into.
- 7to2 4y ago
  This isn't theft by any common definition. End of argument. Now you could try to argue that it's copyright infringement but there are many solid arguments as to why these model weights don't meet the threshold of copyrightability. You could also try to argue distribution of trade secrets, but facebook doesn't seem to view them as such - shared with little restrictions to anyone with an accedemic email, no vetting or ndas, etc. I personally think that facebook planned all of this (sans the childish behavior occurring on their github repo, maybe). They probably wanted to release a capable language model publicly but didn't want the legal and social liabilities associated with it. Facebook is no stranger to keeping things secret. I simply refuse to believe that they didn't see this happening. (Thank you, Facebook!)
- squokko 4y ago
  Yes

343 points by sillysaurusx 4y ago

Curious what the ultimate enforcability of a restrictive license is. If I fine-tune a model, is it still covered? what if I randomize and retrain a layer - or remove a layer? It seems like it will be impossible to verify that someone did not just train the model from scratch.

swyx 4y ago

thanks for doing this, honestly your writeup seems more valuable than the model weights lol > But for what it's worth, my personal opinion is that LLaMA probably isn't OpenAI-grade -- there's a big difference between training a model in an academic setting vs when your entire company depends on it for wide-scale commercial success. I wasn't impressed that 30B didn't seem to know who Captain Picard was. im new to benchmarking shenanigans but how is it that facebook was able to proclaim that it matched GPT3 performance on presumably standard LLM benchmarks? is there a good survey paper or blogpost on how to think about known deficiencies in benchmarks?

sillysaurusx 4y ago
Because loss != quality. This was one of the most counterintuitive discoveries in ML for me. People treat the two as interchangeable, and to a certain extent — a controlled extent — they are. But if your dataset doesn’t include a word about Captain Picard, no amount of training will get it to know about the USS enterprise. Yet your loss metrics will still reach that magical 2.1 value with time. (2.1 is pretty much “excellent” quality; below that means you’re probably overfitting and need a bigger dataset.) Thanks for the comment friendo. I wasn’t sure if this would get any attention at all, but that made it worth it. Be sure to DM me on Twitter if you’d like to chat about anything ML related: basic questions are one of my favorite things to assist with too, so feel free.
- nl 4y ago
  This isn't really correct. Loss is a training-time measurement based on performance on the training objective. The training objective is rarely the same as an end user task that is being benchmark. For example, classically language models are training on next token prediction. The closest benchmark for that is perplexity[1], often reported on the WikiText-103 dataset. Until around 2019 this was often reported, but since then most large language model papers have moved onto reporting more useful benchmarks. Some examples of this are question answering performance or maybe embedding performance. Unfortunately there aren't great benchmarks (yet?) for generative tasks. Quality is quite hard to measure here in a systematic way (see, eg the issues with BLEU benchmarks in summarization benchmarks). [1] https://en.wikipedia.org/wiki/Perplexity https://en.wikipedia.org/wiki/Perplexity
rnosov 4y ago
You can read the original LLaMA paper which is pretty accessible[1]. For example, they claim to outperform GPT-3 on HellaSwag benchmark ( finishing sentences ). You can find examples of unfinished sentences in the HellaSwag paper [2] on page 13. Unfortunately for LLaMA, most people would be probably just asking questions about Captain Picard and so on, and on this benchmark LLaMA significantly underperforms compared to OpenAI models (thats's from their paper). [1] https://research.facebook.com/file/1574548786327032/LLaMA--Open-and-Efficient-Foundation-Language-Models.pdf https://research.facebook.com/file/1574548786327032/LLaMA--O... [2] https://arxiv.org/pdf/1905.07830.pdf https://arxiv.org/pdf/1905.07830.pdf
- yunyu 4y ago
  Hellaswag is also a deeply flawed benchmark, I wouldn't read too much into it: https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this-popular-llm-benchmark-contains-errors https://www.surgehq.ai/blog/hellaswag-or-hellabad-36-of-this...
nl 4y ago
Because there are many benchmarks that measure different things. You need to look at the benchmark that reflects your specific interest. So in this case ("I wasn't impressed that 30B didn't seem to know who Captain Picard was") the closest relevant benchmark they performed is MMLU (Massive Multitask Language Understanding"[1]. In the LLAMA paper they publish a figure of 63.4% for the 5-shot average setting without fine tuning on the 65B model, and 68.9% after fine tuning. This is significantly better that the original GPT-3 (43.9% under the same conditions) but as they note: > "[it is] still far from the state-of-the-art, that is 77.4 for GPT code-davinci-002 on MMLU (numbers taken from Iyer et al. (2022))" InstructGPT[2] (which OpenAI points at as most relevant ChatGPT publication) doesn't report MMLU performance. [1] https://github.com/hendrycks/test https://github.com/hendrycks/test [2] https://arxiv.org/abs/2203.02155 https://arxiv.org/abs/2203.02155
- JonathanFly 4y ago
  The capability of a language model I care about most is probably its ability to represent or simulate Captain Picard. In the sense of being good at creative tasks but also Captain Picard, specifically. Is OpenAI deliberately doing something different on purpose that makes their models better for this, or is just that OpenAI has a lot more copyrighted data in their dataset, as I noticed just now when skimming the Facebook paper for MMLU section and seems be what the Facebook folks think? "A potential explanation is that we have used a limited amount of books and academic papers in our pre-training data, i.e., ArXiv, Gutenberg and Books3, that sums up to only 177GB, while these models were trained on up to 2TB of books. This large quantity of books used by Gopher, Chinchilla and PaLM may also explain why Gopher outperforms GPT-3 on this benchmark, while it is comparable on other benchmarks."
  - nl 4y ago
    It's unclear exactly why it doesn't work as well for you. I have two comments that may be useful: 1) It's very unclear how good the generative capabilities of LLAMA are generally. It benchmarks well for code generation, but for English there aren't really any good benchmarks around. There's good chance the larger model performs much better since generative capabilities seem to be a partially emergent capability. 2) If you just want to "make it work" I'd suggest downloading all the Star Trek scripts you can that include Captain Picard and fine tuning LLAMA using them. It's unclear how well this will work, but that is probably about as good as you can get. If you care about this probably deeply, it's probably worth trying the same with some of the other open GPT-3 models (GPTJ, GPT-NEOX etc)

sillysaurusx 4y ago

Update: FB disabled the download link, so I mirrored everything to R2 and updated the script to use it. It should be working now (though the speed is "only" around 50MB/s).

toomuchtodo 4y ago
Have you dropped the artifacts in the Internet Archive yet by chance?
- sillysaurusx 4y ago
  I'm surprised Internet Archive is appropriate for a 220GB model weight dump. Please feel free; it seems like a good idea. I'm not sure I have enough weekend left to figure out yet another upload service today.
e12e 4y ago
I've read the readme - but I'm not sure why this is any faster than just adding seeds to the torrent? More people downloading via torrent than http?

EMIRELADERO 4y ago

I womder, could Facebook take legal action here? While some (most of) the data used to train the model is copyrighted, I don't think the model is. It's the result of a mathematical process applied to a series of facts and works with no more creativity put onto them.

kuroguro 4y ago
That definition would apply to almost anything software produces ^^; We can already have different licenses for compiled binaries vs the source. Also the output of ML seems to belong to whoever pressed the generate button atm.
- cthalupa 4y ago
  >Also the output of ML seems to belong to whoever pressed the generate button atm. So far the rulings in the US, at least, do not support this. https://arstechnica.com/information-technology/2023/02/us-copyright-office-withdraws-copyright-for-ai-generated-comic-artwork/ https://arstechnica.com/information-technology/2023/02/us-co... In this case, it was images generated via Midjourney and not the output of an LLM, but my layman's understanding of the result here would be equally applicable to LLM output. Effectively, the copyright office does not consider putting in a prompt enough for there to be "human authorship" of the work. In this specific case, that resulted on the images in the comic being considered uncopyrightable. The broader comic, in the organization of the images, the plot and dialogue, etc., still enjoys copyright protection. But in the US, I could just directly take the images in the comic that Midjourney produced and use them for another purpose without violating copyright.
- EMIRELADERO 4y ago
  > That definition would apply to almost anything software produces Not really. The reason software can be copyrighted at all is because the actual code (and resulting object code) is creative. Courts have named this threshold the "Structure, sequence and organization" of the work. ML models don't follow any creative SSO the way actual code does. > Also the output of ML seems to belong to whoever pressed the generate button atm. The output, it seems to me, is uncopyrightable. Copyright only cares about who provides the creativity for the work at issue, not who put in the effort to make it happen. You may own the copyright to your prompt, but the result is generated entirely by the AI and thus lacks human autorship.

bitL 4y ago

How does LLaMA handle fast fine-tuning? Are they using transformer adapters for it?

loufe 4y ago
It's already been adapted for hugging face transformers[1]. Apparently that should unlock its full potential. Oobabooga integrated the change into text-generation-webui[2] meaning we can already access a large chunk of its potential (from what I understand). [1] https://github.com/huggingface/transformers/pull/21955 https://github.com/huggingface/transformers/pull/21955 [2] https://github.com/oobabooga/text-generation-webui/commit/90206204aa41a73fe754e580ae1913dc2b01a384 https://github.com/oobabooga/text-generation-webui/commit/90...
- bitL 4y ago
  That's absolutely fantastic! Thanks for the links!

arjvik 4y ago

How big is this model? (i.e. disk space to store it)

kuroguro 4y ago
65B is ~120GB. All of them combined with the smaller versions is ~220GB.
- nextaccountic 4y ago
  After converting to int8, does it become smaller? Also, can this be further compressed? Like, is there some redundancy a special-purpose compressor could exploit?
  - rfoo 4y ago
    Converting to int8 halves the size.

linearalgebra45 4y ago

It's been enough time since this leaked, so my question is why aren't there blog posts already of people blowing their $300 of starter credit with ${cloud_provider} on a few hours' experimentation running inference on this 65B model? Edit: I read the linked README. > I was impatient and curious to try to run 65B on an 8xA100 cluster Well?

v64 4y ago
The compute necessary to run 65B naively was only available on AWS (and perhaps Azure, I don't work with them) and the required instance types have been unavailable to the public recently (it seems everyone had the same idea to hop on this and try to run it). In my other post here [1], the memory requirements have been lowered through other work, and it should now be possible to run the 65B on a provider like CoreWeave. [1] https://news.ycombinator.com/item?id=35028738 https://news.ycombinator.com/item?id=35028738
- linearalgebra45 4y ago
  Are you sure about that? I can't remember where I saw the table of memory requirements, but I'm sure some of the larger instances here [1] will surely be able to cope (assuming they're available!) Oracle gives you a $300 free trial, which equates to running BM.GPU4.8 for over 10 hours - enough for a focused day of prompting [1] https://www.oracle.com/cloud/compute/gpu/ https://www.oracle.com/cloud/compute/gpu/
  - v64 4y ago
    > Are you sure about that? I'm not. The only way to know it is to try :) thank you for the link!
    - linearalgebra45 4y ago
      You only get a single month-long window to spend the credit! And I'm sure not going to spend any of my own money on prompting experiments. I might be suffering from FOMO to some degree, I've just got to tell myself that this won't have been the only time model weights get leaked!

v64 4y ago

If anyone is interested in running this at home, please follow the llama-int8 project [1]. LLM.int8() is a recent development allowing LLMs to run in half the memory without loss of performance [2]. Note that at the end of [2]'s abstract, the authors state "This result makes such models much more accessible, for example making it possible to use OPT-175B/BLOOM on a single server with consumer GPUs. We open-source our software." I'm very thankful we have researchers like this further democratizing access to this data and prying it out of the hands of the gatekeepers who wish to monetize it. [1] https://github.com/tloen/llama-int8 https://github.com/tloen/llama-int8 [2] https://arxiv.org/abs/2208.07339 https://arxiv.org/abs/2208.07339

rnosov 4y ago
Hmmm, the Github repo suggests that you might be able to run the 65B model on a single A100 80gb card. At the moment, the spot price on Google cloud for this card is $1.25/hour which makes it not so crazy expensive...
- nabla9 4y ago
  $1.25/hour is roughly a year of GPU time until it exceeds the price of A100 80GB card.
  - metadat 4y ago
    I think OP meant that $1.25/hr makes this accessible for people try it out themselves cost effectively, without having to spend thousands or tens of thousands up front to obtain a capable hardware rig. Obviously $1.25/hr 24/7 does add up quickly, after one month the bill would come to $900.
    - deleted 4y ago
      [deleted]
downvotetruth 4y ago
Eagerly awaiting the int8 vs 4 benchmarks. Also, it can run on CPU https://github.com/markasoftware/llama-cpu https://github.com/markasoftware/llama-cpu So, an int8 patch could allow the 65B to run on a standard 128 GB setup assuming the 65B model's cache bursts fit, which if I were to speculate is why the released models stop @ 65B & meta likely already has larger unreleased internal ones.

notpushkin 4y ago

For even better speeds, perhaps use the link from this script (if it ever goes back up) as a webseed for torrent?

ur-whale 4y ago

Is this the full model or just the weights? [EDIT]: are there checksums available? [EDIT2]: MD5 signatures seem to be included for all models in checklist.chk files next to them And there's also what the author mentions: the magnet file he provides in his README does seed immediately on the download when loaded in a bt app which is usually a good sign that the files are correct.

rany_ 4y ago
MD5 signatures don't mean much now that hash collisions could be created instantly on consumer hardware. MD5 is only good for checking for unintentional data corruption.
- deleted 4y ago
  [deleted]
- ur-whale 4y ago
  Bittorrent uses SHA-1 https://en.wikipedia.org/wiki/BitTorrent https://en.wikipedia.org/wiki/BitTorrent Collisions are possible but not exactly trivial
- charcircuit 4y ago
  >hash collisions could be created instantly on consumer hardware Collisions can be created, but MD5 is still preimage resistant. As long as someone with the actual model made the hash and Meta didn't generate colliding models themselves. You can trust it.
  - deleted 4y ago
    [deleted]

m3kw9 4y ago

If an AI model like this isn’t able to evolve and improve is it really useful? Example is code generation or questions that more recent training data can teach the AI

version_five 4y ago

Thanks for doing what Facebook should have been mature / humble enough to have done on their own. The best outcome of this would be for FB to stop the silliness and just release the weights openly themselves.

ALittleLight 4y ago
This may be a good compromise for plausible deniability. Facebook can be "responsible" and release only to "researchers" and the public can get the model from torrents shortly after. So long as the models keep going public I'm satisfied.

vanillax 4y ago

How or what can someone do with this who isn't a ML expert? Is there some docker app that leverages this? To the average dev, is this useful to me? I know there's lots of "plug and play" style docker apps to get started with Stable Diffusion. I'm curious if I can do something fun with this.

swalsh 4y ago
Not an expert, but i've downloaded the model, and used it. What you get is pretty raw, and not super useful by itself. There are some projects to try and do some RLHF on it, and with that we might start to get something you can do some useful stuff with.
deleted 4y ago
[deleted]
kkielhofner 4y ago
You can shortcut a lot of the steps in these various guides by using the Pytorch container from Nvidia[0]. It shouldn't be too hard for someone (me?) to create a Dockerfile and Docker hub container FROM this image to get it up and running easily. [0] - https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorch https://catalog.ngc.nvidia.com/orgs/nvidia/containers/pytorc...

cloudking 4y ago

What's up with the domain in the script? PRESIGNED_URL="https://agi.gpt4.org https://agi.gpt4.org...

arjvik 4y ago
It's pointed at Cloudflare storage right now

yumraj 4y ago

What’s the minimum single GPU that’ll work for the smallest model?

downvotetruth 4y ago
3060 12GB
zargon 4y ago
This reddit post says that the 7B model consumes about 9.7GB of VRAM (using int8). I'm sure very soon people will add support for using system RAM as swap space, which will allow you to use it on an 8GB card, though with a fairly hefty performance penalty. https://www.reddit.com/r/MachineLearning/comments/11h3p2x/d_facebooks_llama_leaks_via_torrent_file_in_pr/jb0wots/ https://www.reddit.com/r/MachineLearning/comments/11h3p2x/d_...

ahahahahah 4y ago

Are we celebrating theft from tech companies now?

tjranagk 4y ago
Their model was trained on vast amounts of copied data. Is making a copy of their model really much different?
anaganisk 4y ago
I mean highseas, adblockers, bypassing paywalls, each one of them is theft. But on the flipside, companies are constantly trying to keep the ownership of data we paid for full price, scooping up personal data, selling low quality work behind paywall.
antibasilisk 4y ago
Copying isn't theft. If you bought the ssd it belongs to you in its entirety regardless what state you decide to configure it into.
7to2 4y ago
This isn't theft by any common definition. End of argument. Now you could try to argue that it's copyright infringement but there are many solid arguments as to why these model weights don't meet the threshold of copyrightability. You could also try to argue distribution of trade secrets, but facebook doesn't seem to view them as such - shared with little restrictions to anyone with an accedemic email, no vetting or ndas, etc. I personally think that facebook planned all of this (sans the childish behavior occurring on their github repo, maybe). They probably wanted to release a capable language model publicly but didn't want the legal and social liabilities associated with it. Facebook is no stranger to keeping things secret. I simply refuse to believe that they didn't see this happening. (Thank you, Facebook!)
squokko 4y ago
Yes