3 ms·
You can actually download the weights without an account: git clone https://huggingface.co/bigscience/bloom You will need Git LFS and ~330GB of free space
by moyix 4y ago
You can actually download the weights without an account:
git clone https://huggingface.co/bigscience/bloom
You will need Git LFS and ~330GB of free space though.
It is possible to run inference locally, even without very much RAM/VRAM (though it will be very slow). Get the most recent versions of "transformers" and "accelerate" from git, and then:
import torch
from transformers import AutoTokenizer, AutoModelForCausalLM
tokenizer = AutoTokenizer.from_pretrained('./bloom')
inputs = tokenizer("Hacker News is", return_tensors="pt")
model = AutoModelForCausalLM.from_pretrained('./bloom', device_map="auto", offload_folder='offload', torch_dtype=torch.bfloat16)
output = model.generate(inputs["input_ids"].to(0), min_length=30, max_length=30, do_sample=True)
print(tokenizer.decode(output[0].tolist()))
After the model is loaded you can also look at where it put each part of the model (GPU, CPU RAM, or on disk):
print(model.hf_device_map)
Here's a screenshot of it in action on my machine, which has 256GB RAM and 2xA6000 GPUs:
https://twitter.com/moyix/status/1546961979566325760 https://twitter.com/moyix/status/1546961979566325760
- dekhn 4y agoThanks, that's what I was looking for (double thanks for the example inference code). I'll queue up the download and try to get the dependencies working. Comment 1: git-lfs is now close to saturating my gigabit ethernet connection, around 990Mb/sec
- dekhn 4y agoComment 2: filled up the disk on my first computer and had to find an actual hard drive since none of my SSDs have 330GB. It seems more like 500+GB on disk. Haven't gotten to inference yet.
- moyix 4y agoI noticed this too -- I think it's a git LFS issue? The weights themselves are only 330GB, but the .git directory adds another 330GB: $ du -csh pytorch_model*.bin | tail -n 1 329G total $ du -sh .git 329G .git Not sure if there's a way to get git-lfs not to do that...
- dekhn 4y agoI haven't used LFS before but it seems to buffer incompletely downloaded files in .git/lfs/incomplete and a copy of "objects" in .git/lfs/objects (I am one of those people who absolutely no interest in how git's DB works, nor how its blob DB works).
- epups 4y agoIf I understand correctly, it took you 7 minutes to generate a single prompt in your machine?
- moyix 4y agoYep. The model is too big to fit on the two GPUs I have (even though each one is 48GB). So instead it gets split between the GPUs, CPU RAM, and some parts even have to sit on disk (SSD). This makes it much slower than if everything were on the GPU. I believe there are performance enhancements planned in Accelerate that will allow layers to be preloaded in parallel as other layers are executing inference, which should make things faster: https://github.com/huggingface/accelerate/issues/512#issuecomment-1182480255 https://github.com/huggingface/accelerate/issues/512#issueco...
- zxcb1 4y agoWould NVLink and more RAM improve performance?
- dekhn 4y agoI wasn't going to ask about this until I had it running on my machine but... its very interesting that you can do out of core models with weights in various memory hierarchies. It feels almost like swap/paging. This is exciting!
- lostmsu 4y agoSimple swap via memory-mapped files would have worked probably. The issue is the lack of prefetch.
- dekhn 4y agoAlmost there. I've got everything installed and want to run your inference script. Model loading fails, TypeError: Got unsupported ScalarType BFloat16 File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/utils/offload.py", line 25, in offload_weight array = weight.numpy() I verified my numpy knows nothing of bfloat16: >>> import torch >>> t=torch.arange(0,6).resize(2,3).to(dtype=torch.bfloat16) /home/dek/miniconda3/lib/python3.9/site-packages/torch/_tensor.py:586: UserWarning: non-inplace resize is deprecated warnings.warn("non-inplace resize is deprecated") >>> t tensor([[0., 1., 2.], [3., 4., 5.]], dtype=torch.bfloat16) >>> t.numpy() Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: Got unsupported ScalarType BFloat16 Seems par for the course that I'd be able to get the world's most advanced NLP model downloaded over gigabit to my home supercomputer only to be stymied by a data type.
- dekhn 4y agoComment three: after hacking my way around that, the script ran for another few hours doing disk IE (writing the offload) such that my directory is now 1.3T (330GB .git/lfs/objects, 330GB sharded weights, and 607G of "offload". Then it failed again: File "/home/dek/miniconda3/lib/python3.9/site-packages/accelerate/big_modeling.py", line 188, in dispatch_model main_device = [d for d in device_map.values() if d not in ["cpu", "disk"] [0] IndexError: list index out of range Now I'm curious just how long it will take to repro this error (IE, running again, with the offload files aalready written). It's also puzzling since I set the device_map to 'auto'. Again, all par for the course and stuff I expected (having worked in HPC/ML/science for 3 decades, you get used to research codes).
- moyix 4y agoHuh. That one looks like it wasn't able to place any part of the model on the GPU. I know there is currently an issue where if the first layer of the model is too big to fit on the GPU it will put all layers on the CPU/disk (rather than trying to see if a later layer would fit). But the 3080Ti has 12GB so I'd be surprised if that's happening here?
- deleted 4y ago
- lostmsu 4y agoWhat does 'offload_folder' parameter do? Why is it even needed? Does the inference produce intermediate results so large they have to be stored on disk? o-O