5 ms·

You can try the model on HuggingFace: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom Though it requires an account, unlike oth

You can try the model on HuggingFace: https://huggingface.co/bigscience/bloom https://huggingface.co/bigscience/bloom Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). Bloom looks pretty exciting. It's reportedly as performant as GPT-3 (I haven't tested it enough to confirm, but what little testing I did gave okay results), but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil. I'm expecting the coming 12 months to be pretty interesting for text generation.

dekhn 4y ago
If you create a hugginface account, can you download the model and weights and run inference locally (I have a nvidia 3080 Ti on a 32-core machine with 64GB of RAM)? Or do they just host the inference server for you? IIUC the model design is open so I assume the weights are too.
- moyix 4y ago
  You can actually download the weights without an account: git clone https://huggingface.co/bigscience/bloom You will need Git LFS and ~330GB of free space though. It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then: import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained('./bloom') inputs = tokenizer("Hacker News is", return_tensors="pt") model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16) output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True) print(tokenizer.decode(output[0].tolist())) After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk): print(model.hf_device_map) Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs: https://twitter.com/moyix/status/1546961979566325760 https://twitter.com/moyix/status/1546961979566325760
  - dekhn 4y ago
    Thanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working. Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec
    - dekhn 4y ago
      Comment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.
      - moyix 4y ago
        I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB: $ du -csh pytorch_model*.bin | tail -n 1 329G total $ du -sh .git 329G .git Not sure if there's a way to get git-lfs not to do that...
        dekhn 4y ago
        I haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).
  - epups 4y ago
    If I understand correctly, it took you 7 minutes to generate a single prompt in your machine?
    - moyix 4y ago
      Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU. I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issuecomment-1182480255 https://github.com/huggingface/accelerate/issues/512#issueco...
- amrrs 4y ago
  You can use the model without Hugging Face account - https://www.youtube.com/watch?v=bGBY3K-_bio https://www.youtube.com/watch?v=bGBY3K-_bio 13:23
minimaxir 4y ago
> Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). This model is larger than anything HF has done, so resource usage likely has to be constrained a bit (GPT-3 was restricted at first for similar reasons, in addition to the potential abuse issues)
schoen 4y ago
> but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil It seems that the developers' position is that it's not fully open-source, and that some potentially evil or unethical uses are actually restricted: https://huggingface.co/spaces/bigscience/license https://huggingface.co/spaces/bigscience/license
- PoignardAzur 4y ago
  Right, it's open source with a license that forbids using it for evil. Practically speaking, though, you can just download it on a private server and do evil and afaict nobody can stop you or know you're using BLOOM.
  - schoen 4y ago
    > Right, it's open source with a license that forbids using it for evil. That's not considered open source, though. https://opensource.org/osd-annotated https://opensource.org/osd-annotated Also compare https://web.archive.org/web/20130203112329/http://dev.hasenj.org/post/3272592502 https://web.archive.org/web/20130203112329/http://dev.hasenj... (although IBM wasn't asking specifically about the open source nature of the license).
    - PoignardAzur 4y ago
      Alright, fine. It's open-source-with-an-asterisk, which is completely different from open-source in every way, and happens to involve having access to the source of the thing. The asterisk says "I'm not allowed to say it's open source because some website I don't care about says it's not the right definition".

dekhn 4y ago

If you create a hugginface account, can you download the model and weights and run inference locally (I have a nvidia 3080 Ti on a 32-core machine with 64GB of RAM)? Or do they just host the inference server for you? IIUC the model design is open so I assume the weights are too.

moyix 4y ago
You can actually download the weights without an account: git clone https://huggingface.co/bigscience/bloom You will need Git LFS and ~330GB of free space though. It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then: import torch from transformers import AutoTokenizer, AutoModelForCausalLM tokenizer = AutoTokenizer.from_pretrained('./bloom') inputs = tokenizer("Hacker News is", return_tensors="pt") model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16) output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True) print(tokenizer.decode(output[0].tolist())) After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk): print(model.hf_device_map) Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs: https://twitter.com/moyix/status/1546961979566325760 https://twitter.com/moyix/status/1546961979566325760
- dekhn 4y ago
  Thanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working. Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec
  - dekhn 4y ago
    Comment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.
    - moyix 4y ago
      I noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB: $ du -csh pytorch_model*.bin | tail -n 1 329G total $ du -sh .git 329G .git Not sure if there's a way to get git-lfs not to do that...
      - dekhn 4y ago
        I haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).
- epups 4y ago
  If I understand correctly, it took you 7 minutes to generate a single prompt in your machine?
  - moyix 4y ago
    Yep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU. I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issuecomment-1182480255 https://github.com/huggingface/accelerate/issues/512#issueco...
amrrs 4y ago
You can use the model without Hugging Face account - https://www.youtube.com/watch?v=bGBY3K-_bio https://www.youtube.com/watch?v=bGBY3K-_bio 13:23

minimaxir 4y ago

> Though it requires an account, unlike other models on the hub (I guess they're hoping to get new users from the hype of this model). This model is larger than anything HF has done, so resource usage likely has to be constrained a bit (GPT-3 was restricted at first for similar reasons, in addition to the potential abuse issues)

schoen 4y ago

> but the model is completely open-source; if you can afford some cloud compute, you can just upload the model on your preferred cloud provider and just generate whatever you want from it, good or evil It seems that the developers' position is that it's not fully open-source, and that some potentially evil or unethical uses are actually restricted: https://huggingface.co/spaces/bigscience/license https://huggingface.co/spaces/bigscience/license

PoignardAzur 4y ago
Right, it's open source with a license that forbids using it for evil. Practically speaking, though, you can just download it on a private server and do evil and afaict nobody can stop you or know you're using BLOOM.
- schoen 4y ago
  > Right, it's open source with a license that forbids using it for evil. That's not considered open source, though. https://opensource.org/osd-annotated https://opensource.org/osd-annotated Also compare https://web.archive.org/web/20130203112329/http://dev.hasenj.org/post/3272592502 https://web.archive.org/web/20130203112329/http://dev.hasenj... (although IBM wasn't asking specifically about the open source nature of the license).
  - PoignardAzur 4y ago
    Alright, fine. It's open-source-with-an-asterisk, which is completely different from open-source in every way, and happens to involve having access to the source of the thing. The asterisk says "I'm not allowed to say it's open source because some website I don't care about says it's not the right definition".